6.1.1. Boot and Kernel Troubleshooting

💡 First Principle: Linux boot failures have a defined failure sequence: firmware can't find boot device → GRUB can't load → kernel panics → initramfs fails → root filesystem fails to mount → systemd target fails. Each stage has specific symptoms and specific recovery tools. Knowing which stage failed narrows the fix immediately.

Systemd Boot Analysis:

systemd-analyze                      # Total boot time
systemd-analyze blame                # Per-unit time (find slow services)
systemd-analyze critical-chain       # The slowest dependency path
systemd-analyze plot > boot.svg      # Visual timeline (open in browser)

# Units that failed during boot
systemctl list-units --state=failed  # Failed units
systemctl --failed                   # Same
journalctl -b -p err                 # All errors from current boot
journalctl -b -1 -p err             # All errors from previous boot
journalctl -b -1                     # All logs from previous boot (useful after crash)

Common blame offenders:

• NetworkManager-wait-online.service (or systemd-networkd-wait-online.service) holds network-online.target until a connection is fully configured — which can add tens of seconds to boot when DHCP is slow or a link is absent. Services that only need interfaces up should order on network.target; if nothing genuinely needs network-online.target, disabling the wait-online service removes the delay.
• systemd-udev-settle.service blocks until udev has processed every queued device event. It is deprecated and runs only because some unit pulls it in: watch device events with udevadm monitor to find the slow device, and fix the dependent unit rather than masking settle (masking can let services start before their devices exist).

GRUB Recovery:

# At GRUB menu: press 'e' to edit boot entry — the edit applies to THIS boot only (Ctrl+X boots it)
# Persistent changes: GRUB_CMDLINE_LINUX in /etc/default/grub + regenerate, or grubby --update-kernel=ALL --args="..."
# Common recovery edits:
# - Append 'single' or '1' to kernel line → boot to single-user mode
# - Append 'rd.break' → break into initramfs shell before pivot_root
# - Append 'systemd.unit=rescue.target' → rescue mode with more services
# - Remove 'quiet' and 'rhgb' → verbose boot messages

# From the grub> command line (at grub rescue>, first set prefix/root and insmod normal):
grub> set root=(hd0,gpt2)
grub> linux /boot/vmlinuz root=/dev/sda2
grub> initrd /boot/initramfs.img
grub> boot

# After fixing, regenerate GRUB config
grub2-mkconfig -o /boot/grub2/grub.cfg    # RHEL
update-grub                                 # Debian

grub> vs grub rescue>: at grub> GRUB's normal shell has loaded (core image and modules available) but grub.cfg was missing or unusable, so you can boot by hand with linux, initrd and boot. At grub rescue> GRUB could not load its modules from its /boot/grub prefix (partition moved, deleted or renumbered), so only a few built-ins (ls, set, insmod) work until you point prefix/root at the right partition and insmod normal.

Kernel Panic Causes:

dmesg — Kernel Ring Buffer:

dmesg                            # All kernel messages since boot
dmesg -T                         # With human-readable timestamps
dmesg -l err,crit,alert,emerg    # Only serious messages
dmesg | grep -i "error\|fail\|warn"   # Filter for problems
dmesg | grep -i "oom"            # Out-of-memory killer events
dmesg | grep -i "i/o error"      # Disk I/O errors → hardware failure
dmesg | grep -i "drm\|nvidia"    # GPU driver messages
journalctl -k                    # Kernel messages via journalctl (same data)

OOM (Out of Memory) Killer:

When the system runs out of memory and swap, the kernel OOM killer selects and kills processes to free memory:

# Detect OOM kills
dmesg | grep -i "oom"
journalctl | grep -i "killed process"

# /proc/sys/vm/overcommit_memory
# 0 = heuristic overcommit (default)
# 1 = always overcommit (no checks)
# 2 = never overcommit (strict)

# Adjust OOM score for a critical process (protect from OOM killer)
echo -1000 > /proc/$(pgrep -f myapp)/oom_score_adj  # -1000 = never kill
echo 1000 > /proc/$(pgrep -f badproc)/oom_score_adj # 1000 = kill first

⚠️ Exam Trap: journalctl -b -1 shows logs from the previous boot — essential for diagnosing crashes and panics that caused the reboot, since journalctl -b only shows the current boot's logs. After a kernel panic, this is usually the first command to run.

Reflection Question: A server rebooted unexpectedly overnight. No one was working on it. What are the first three commands you run to determine the cause, and what specific log indicators would you look for?