5.5.1. System Performance Monitoring

💡 First Principle: System performance problems fall into four categories: CPU-bound (process needs more compute), memory-bound (not enough RAM — swap thrashing), I/O-bound (waiting for disk/network), and capacity (system simply doesn't have enough resources for the workload). The diagnostic goal is to identify which category applies before reaching for a solution.

# CPU and Load
uptime                           # Load averages: 1, 5, 15 minute
top                              # Live all-in-one view: load average, CPU split (us/sy/wa/id), memory/swap, per-process table; press 1 for per-CPU
vmstat 1 5                       # Virtual memory stats: 1-second intervals, 5 samples
mpstat -P ALL 1                  # Per-CPU utilization
sar -u 1 10                      # CPU utilization (10 samples)

# Memory
free -h                          # Memory and swap usage
vmstat -s                        # Memory statistics summary
swapon --show                    # Swap devices and usage

# Disk I/O
iostat -xz 1                     # Extended disk I/O stats per device
iotop                            # I/O by process (like top for disk)
blktrace /dev/sda                # Low-level block device tracing
sar -d 1 10                      # Disk activity via sar

# Network
sar -n DEV 1 5                   # Network interface stats
nethogs                          # Network usage by process (like top for network)
iftop                            # Network traffic by connection

# Comprehensive tools
sar                              # System Activity Reporter — collects everything
atop                             # Advanced: CPU, memory, disk, network per process
dstat                            # Combined stats (replaces vmstat/iostat/netstat)

Reading vmstat columns: r = runnable processes (running or waiting for a CPU — the run queue); si/so = memory swapped in from / out to disk per second (sustained non-zero = memory pressure); in = interrupts per second; cs = context switches per second; us/sy/id/wa = CPU time in user, system, idle, and I/O wait. vmstat and iostat -c report CPU averaged across all cores — use mpstat -P ALL or top (press 1) for a per-CPU view.

Historical data with sar: sar belongs to the sysstat package. Once its collector is enabled (e.g. systemctl enable --now sysstat), it samples the system every 10 minutes into daily files (/var/log/sa/ on RHEL, /var/log/sysstat/ on Debian/Ubuntu), kept for the number of days set by HISTORY in the sysstat config. sar with no arguments shows today's history and sar -f <file> an earlier day — the tool for trends, where top and vmstat only show the present.

Understanding Load Average:

Load average represents the average number of processes in the run queue (running or waiting for CPU) over the last 1, 5, and 15 minutes. On a 4-core system:

• Load average of 4.0 = 100% utilized (one process per core)
• Load average of 8.0 = 200% — system is overloaded
• Load average > CPU count = performance problem
• Load average < CPU count = capacity available

Note: I/O-waiting processes also count toward load average in Linux, so high load can mean disk/network saturation, not just CPU saturation.

Read the three figures as a trend: a 1-minute value above the 15-minute value means load is rising; below it, load is falling.