6.1.1. Boot and Kernel Troubleshooting

💡 First Principle: Linux boot failures have a defined failure sequence: firmware can't find boot device → GRUB can't load → kernel panics → initramfs fails → root filesystem fails to mount → systemd target fails. Each stage has specific symptoms and specific recovery tools. Knowing which stage failed narrows the fix immediately.

Systemd Boot Analysis:

systemd-analyze                      # Total boot time
systemd-analyze blame                # Per-unit time (find slow services)
systemd-analyze critical-chain       # The slowest dependency path
systemd-analyze plot > boot.svg      # Visual timeline (open in browser)

# Units that failed during boot
systemctl list-units --state=failed  # Failed units
systemctl --failed                   # Same
journalctl -b -p err                 # All errors from current boot
journalctl -b -1 -p err             # All errors from previous boot
journalctl -b -1                     # All logs from previous boot (useful after crash)

GRUB Recovery:

# At GRUB menu: press 'e' to edit boot entry
# Common recovery edits:
# - Append 'single' or '1' to kernel line → boot to single-user mode
# - Append 'rd.break' → break into initramfs shell before pivot_root
# - Append 'systemd.unit=rescue.target' → rescue mode with more services
# - Remove 'quiet' and 'rhgb' → verbose boot messages

# From GRUB rescue prompt:
grub> set root=(hd0,gpt2)
grub> linux /boot/vmlinuz root=/dev/sda2
grub> initrd /boot/initramfs.img
grub> boot

# After fixing, regenerate GRUB config
grub2-mkconfig -o /boot/grub2/grub.cfg    # RHEL
update-grub                                 # Debian

Kernel Panic Causes:

dmesg — Kernel Ring Buffer:

dmesg                            # All kernel messages since boot
dmesg -T                         # With human-readable timestamps
dmesg -l err,crit,alert,emerg    # Only serious messages
dmesg | grep -i "error\|fail\|warn"   # Filter for problems
dmesg | grep -i "oom"            # Out-of-memory killer events
dmesg | grep -i "i/o error"      # Disk I/O errors → hardware failure
dmesg | grep -i "drm\|nvidia"    # GPU driver messages
journalctl -k                    # Kernel messages via journalctl (same data)

OOM (Out of Memory) Killer:

When the system runs out of memory and swap, the kernel OOM killer selects and kills processes to free memory:

# Detect OOM kills
dmesg | grep -i "oom"
journalctl | grep -i "killed process"

# /proc/sys/vm/overcommit_memory
# 0 = heuristic overcommit (default)
# 1 = always overcommit (no checks)
# 2 = never overcommit (strict)

# Adjust OOM score for a critical process (protect from OOM killer)
echo -1000 > /proc/$(pgrep -f myapp)/oom_score_adj  # -1000 = never kill
echo 1000 > /proc/$(pgrep -f badproc)/oom_score_adj # 1000 = kill first

⚠️ Exam Trap: journalctl -b -1 shows logs from the previous boot — essential for diagnosing crashes and panics that caused the reboot, since journalctl -b only shows the current boot's logs. After a kernel panic, this is usually the first command to run.

Reflection Question: A server rebooted unexpectedly overnight. No one was working on it. What are the first three commands you run to determine the cause, and what specific log indicators would you look for?