3.3.1. Process Monitoring and States

💡 First Principle: Process state tells you what a process is waiting for. Running means it has CPU. Sleeping means it's waiting for an event (usually I/O). Stopped means it received SIGSTOP. Zombie means it's finished but the parent hasn't called wait(). Diagnosing performance problems starts with understanding which state processes are in and why.

Process States:

A process stuck in D state (uninterruptible sleep) usually means it's waiting on slow or blocked I/O — often a sign of a storage or NFS problem. These processes cannot be killed until the I/O completes or times out.

Process Inspection Tools:

# ps — snapshot of current processes
ps aux                           # All processes, user-oriented format
ps aux | grep nginx              # Filter for nginx
ps -ef                           # All processes, full format (shows PPID)
ps -eo pid,ppid,user,%cpu,%mem,state,cmd   # Custom columns
pstree                           # Visual parent/child hierarchy
pstree -p alice                  # Process tree for user alice

# top — live process monitor
top                              # Interactive; press q to quit
top -b -n 1 > snapshot.txt       # Non-interactive batch output

# htop — enhanced top (color, mouse, F-key shortcuts)
htop

# atop — advanced resource monitor (disk I/O, network per process)
atop

# Per-process statistics
pidstat -u -p 1234 1             # CPU usage for PID 1234, update every 1s
mpstat -P ALL 1                  # CPU usage per core
strace -p 1234                   # Trace system calls for running process
strace -e openat cmd             # Trace only file-open syscalls

# Open files
lsof -p 1234                     # Files opened by PID 1234
lsof -u alice                    # All files opened by user alice
lsof -i TCP:80                   # Processes using TCP port 80

Priority and Niceness:

Linux schedules processes partly based on their nice value (range: -20 to +19). Lower nice = higher priority. Only root can lower nice values (increase priority).

nice -n 10 ./backup.sh           # Start process with nice +10 (lower priority)
nice -n -5 ./realtime.sh         # Start with nice -5 (higher priority; root only)
renice -n 15 -p 1234             # Change nice of running process to +15
renice -n -5 -u alice            # Renice all of alice's processes

⚠️ Exam Trap: A process in D (uninterruptible sleep) state cannot be killed — not even with kill -9. SIGKILL is processed by the kernel scheduler, but a process in D state is waiting inside the kernel for I/O to complete and isn't checking signals. The only resolution is to fix the underlying I/O issue (reconnect NFS, replace failing disk, etc.).

Reflection Question: ps aux shows a process in Z (zombie) state. It has been there for hours. What does this indicate about the parent process, and what is the correct remediation?