6.1.2. Process, Memory, and Performance Troubleshooting

💡 First Principle: Every performance problem is a resource contention problem. Something needs a resource (CPU cycles, memory pages, disk IOPS, network bandwidth) faster than the system can supply it. Diagnosis means measuring each resource and identifying which one is the bottleneck — then either reducing demand (optimize the process) or increasing supply (upgrade hardware, add nodes).

CPU Troubleshooting Workflow:

# Step 1: Is CPU actually the bottleneck?
uptime                           # Check load average vs CPU count
vmstat 1 5                       # Look at 'r' (run queue) and 'wa' (iowait)
# If r > CPU count → CPU queuing
# If wa > 20% → I/O causing CPU to idle

# Step 2: Which process is consuming CPU?
top                              # Interactive; press 'P' to sort by CPU
ps aux --sort=-%cpu | head -20   # Top 20 CPU consumers
pidstat -u 1 5                   # Per-process CPU usage

# Step 3: What is the process doing?
strace -p <PID>                  # System calls being made
strace -cp <PID>                 # Summary of system calls + counts + time
perf top -p <PID>                # CPU profiling (which functions)
lsof -p <PID>                    # What files/sockets are open

Memory Troubleshooting Workflow:

# Step 1: What is memory state?
free -h                          # Total, used, free, cached, available
cat /proc/meminfo                # Detailed memory breakdown
vmstat -s                        # Memory stats summary

# Key distinction: 'free' vs 'available'
# free = pages truly idle (often low — Linux caches disk data in RAM)
# available = memory that can be made available for new processes (free + reclaimable cache)
# Use 'available' to assess actual memory pressure

# Step 2: Who is using memory?
ps aux --sort=-%mem | head -20   # Top memory consumers
top                              # Press 'M' to sort by memory
smem -tk                         # More accurate RSS/PSS/USS metrics

# Step 3: Swap usage
swapon --show                    # Swap devices
vmstat 1 5                       # 'si' and 'so' = swap-in and swap-out
# High swap I/O (si/so > 0 constantly) = memory exhaustion

# Step 4: Memory leaks
valgrind --leak-check=full ./myapp   # Memory leak detection (development)
/proc/<PID>/status               # VmRSS = physical memory; VmSize = virtual
watch -n 1 'ps -o pid,rss,vsz -p <PID>'  # Watch RSS growing over time

Disk I/O Troubleshooting:

iostat -xz 1                     # r/s, w/s, await (avg I/O wait ms), %util
# %util near 100% = device saturated
# await > 10-20ms for SSD, > 50ms for spinning disk = contention

iotop                            # Which process is doing the most I/O
blktrace -d /dev/sda -o trace && blkparse trace  # Detailed block tracing
fio --name=randread --rw=randread --bs=4k --size=1G  # Disk benchmark

# Filesystem issues
fsck -n /dev/sda1                # Read-only check (don't repair)
dmesg | grep -i "i/o error"      # Disk errors in kernel log
smartctl -a /dev/sda             # SMART health data (read failure counts, etc.)

Identifying Zombie Processes:

ps aux | grep Z                  # Show zombie processes
ps -eo pid,ppid,state,cmd | grep "^.\{15\}Z"  # Detailed zombie view

# Remedy: kill the parent process
# The zombie will be reaped when parent dies (re-parented to init)
kill -9 <PPID>                   # Kill parent → zombie is reaped

⚠️ Exam Trap (M10): %cpu in top/ps shows CPU utilization, not CPU wait. A server can show near-zero %cpu while being completely I/O-bound — processes are sleeping (D state) waiting for disk, so CPU is idle. The correct indicator for I/O saturation is %iowait in mpstat or wa in vmstat. Confusing these leads to wrong root-cause analysis.

Reflection Question: A web application becomes intermittently slow. top shows CPU at 15%, but vmstat shows wa at 70%. What is the bottleneck and what tool would you use next to identify which process is causing the I/O?