5.5.2. Observability Stacks and Alerting
Modern Observability: Three Pillars:
| Pillar | What It Is | Linux Tools |
|---|---|---|
| Metrics | Numerical measurements over time (CPU %, requests/sec) | Prometheus + Node Exporter |
| Logs | Timestamped text records of events | journald, rsyslog, ELK Stack |
| Traces | Request paths through distributed systems | Jaeger, Zipkin |
Prometheus + Grafana:
# Node Exporter — exposes Linux metrics for Prometheus
# Installed on every monitored node; listens on port 9100
systemctl enable --now node_exporter
# Prometheus scrapes metrics from exporters
# grafana visualizes metrics from Prometheus
# alertmanager handles alerting rules
# Check metrics endpoint
curl http://localhost:9100/metrics | grep node_cpu
Log Aggregation (ELK Stack):
- Elasticsearch — stores and indexes log data
- Logstash — processes and transforms log data
- Kibana — visualizes logs from Elasticsearch
- Filebeat — lightweight log shipper (installed on each node)
Alerting Concepts:
# Nagios/Icinga — traditional monitoring
# Alert conditions: check_disk, check_load, check_http
check_disk -w 20% -c 10% -p / # Warn at 20% free, critical at 10% (bare numbers would mean MB, not %)
# systemd watchdog — restart failing services automatically
# In unit file:
# WatchdogSec=30s — systemd kills/restarts if no keepalive for 30s
# Restart=on-failure — restart if service exits non-zero
# SNMP — Simple Network Management Protocol
# For monitoring network devices and legacy systems
snmpwalk -v2c -c public 192.168.1.1 .1.3.6.1.2.1.1.1 # Get sysDescr
SNMP works two ways: polling, where the manager asks the device for values (GET, snmpwalk; UDP 161), and traps, where the device's agent sends an unsolicited notification to the manager as soon as an event occurs — link down, threshold crossed (UDP 162). A trap means the device detected the event and pushed it; polling is how the manager notices a device that has gone silent.
Threshold vs. predictive alerts: a threshold alert fires when a metric crosses a fixed value (disk > 90% used). A predictive alert extrapolates the current rate of change — Prometheus's predict_linear() — and fires when the trend will reach the limit within a window, so it can fire while plenty of space is still free if usage is growing fast.
SLIs, SLOs, and error budgets:
| Term | Meaning | Example |
|---|---|---|
| SLI (indicator) | The measured value | 0.4% of requests failed this month |
| SLO (objective) | The internal target for that SLI | Error rate stays below 1% |
| SLA (agreement) | The contractual promise to customers, with penalties | Service credit if availability falls below 99.5% |
The error budget is the unreliability an SLO allows: (1 − SLO) × period. A 99.95% availability SLO over a 30-day month (43,200 minutes) allows 0.0005 × 43,200 = 21.6 minutes of downtime. While budget remains, the team can keep shipping changes; once it is spent, the usual policy is to freeze risky releases and put engineering effort into reliability until the service is back within its objective — not to lower the SLO to fit the outage.
⚠️ Exam Trap: A high load average alone doesn't identify the cause — you need to distinguish CPU saturation (%usr and %sys in mpstat) from I/O wait (%iowait). vmstat column wa shows the percentage of time CPUs are idle waiting for I/O. High wa with low us/sy means I/O is the bottleneck, not CPU — adding more CPUs won't help.
Reflection Question: uptime shows a load average of 24 on a 4-core server. top shows %iowait at 85% and CPU %us at 5%. What is the bottleneck and what diagnostic tools would you use next?
