6.2.1. Filesystem and Disk Diagnostics

# Capacity
df -h                            # Block space usage by mount point
df -i                            # Inode usage (can fill independently of blocks)
du -sh /*                        # Space by top-level directory
du -sh /var/* | sort -h          # Find large directories in /var
find / -size +500M               # Files larger than 500 MB

# Filesystem integrity
# MUST unmount before checking (or use read-only root for /)
fsck -n /dev/sda1                # Dry run — report errors without fixing
fsck -y /dev/sda1                # Auto-fix all errors (use carefully)
xfs_repair -n /dev/sdb1          # XFS read-only check
xfs_repair /dev/sdb1             # XFS repair

# Force fsck on next boot (ext4)
touch /forcefsck                 # Some distros honor this file
tune2fs -C 100 /dev/sda1        # Set mount count to trigger fsck threshold

# Disk health
smartctl -a /dev/sda             # Full SMART data
smartctl -H /dev/sda             # Health summary only
smartctl -t short /dev/sda       # Run short self-test
badblocks -v /dev/sdb            # Scan for bad blocks (read-only mode)

# LVM diagnostics
pvdisplay                        # Physical volume status
vgdisplay                        # Volume group status  
lvdisplay                        # Logical volume status
lvs -o +devices                  # Show which PVs each LV spans
vgck                             # Check VG consistency

Common Storage Error Patterns:

• Reserved blocks (ext2/3/4): mkfs.ext4 reserves 5% of blocks for root by default (tune2fs -l <dev> | grep -i 'reserved block'; change with tune2fs -m). df leaves them out of Avail and computes Use% as used ÷ (used + avail), so when non-root writes start failing for lack of blocks, df already shows 100% — reserved space never hides behind a lower Use%.
• Core dumps filling a filesystem: systemd-coredump writes dumps under /var/lib/systemd/coredump/ (list them with coredumpctl list). Cap them in /etc/systemd/coredump.conf (MaxUse=, KeepFree=, ProcessSizeMax=, or Storage=none) rather than disabling dumps outright — they are the crash evidence — and keep /var/log on its own filesystem so a runaway writer elsewhere cannot stop logging.

Recovering from Read-Only Remount:

When a filesystem encounters I/O errors, Linux automatically remounts it read-only to prevent further corruption. Fix the underlying hardware issue first, then:

# Check for errors
dmesg | tail -50                 # I/O errors, SMART alerts
smartctl -a /dev/sda             # SMART status

# If hardware is healthy — filesystem corruption
umount /data
fsck -y /dev/sda1
mount /data

# If hardware is bad — replace disk first
# Do NOT try to repair filesystem on a failing drive — you'll lose data

⚠️ Exam Trap: Running fsck on a mounted filesystem is dangerous and will corrupt data. The only exception is the root filesystem during boot (when a special journal-aware check can run). For all other filesystems: unmount first, then fsck. If you can't unmount (root), boot to rescue mode.

Reflection Question: A server shows df -h at 60% for /var/log, but every new log write fails with "No space left on device." What command reveals the actual problem, and what could cause this discrepancy?