4.3.1. CloudWatch Metrics, Logs, and Alarms

💡 First Principle: CloudWatch is the unified monitoring nervous system for all AWS services. Every service publishes metrics to CloudWatch automatically — Glue job duration, Lambda invocation errors, Kinesis iterator age, Redshift query throughput. Alarms turn passive metrics into active notifications, ensuring problems are detected before users notice. Monitoring (metrics and alarms) tells you a known failure happened; observability adds logs and traces (AWS X-Ray) so you can diagnose failures you didn't anticipate — the full stack is CloudWatch metrics + CloudWatch Logs + X-Ray traces, and it complements monitoring rather than replacing it.

CloudWatch Metrics — built-in metrics for every AWS service. Key data engineering metrics: Glue job run status and duration (plus glue.driver.aggregate.bytesRead — if duration climbs week over week while bytes read climb with it, the input is growing, not the job degrading), Lambda errors and throttles, Kinesis IteratorAge (how far behind a consumer is — critical for detecting slow consumers), DynamoDB consumed capacity and throttled requests, and Redshift query duration.

CloudWatch Logs — centralized log storage. Glue jobs, Lambda functions, and EMR clusters write logs to CloudWatch Log Groups. Log Insights provides a SQL-like query language for searching and analyzing logs interactively. Subscription filters stream log events to Lambda, Kinesis, or OpenSearch for real-time processing.

CloudWatch Alarms — trigger actions when metrics cross thresholds. Alarm → SNS → email/PagerDuty for alerting. Alarm → Lambda for automated remediation (e.g., restart a failed Glue job). Composite alarms combine multiple alarms with AND/OR logic and notify only when the combined condition holds — the fix for alert fatigue from individual alarms firing on transient, self-recovering failures.

Amazon Managed Grafana provides advanced dashboarding with CloudWatch as a data source. Use Grafana when teams need rich operational dashboards beyond what CloudWatch natively offers; it unifies many data sources in one dashboard — CloudWatch, Amazon Managed Service for Prometheus, X-Ray, OpenSearch and third-party tools — across multiple AWS accounts.

Key data engineering monitoring patterns:

For centralized monitoring across pipelines, create a CloudWatch dashboard that aggregates metrics from all pipeline components — Glue job status, Lambda error rates, Kinesis throughput, and Redshift query duration — providing a single pane of glass for the data engineering team. When pipelines span several accounts, CloudWatch cross-account observability links source accounts to a central monitoring account that can view and alarm on their metrics, logs and traces, so on-call engineers diagnose from one place.

⚠️ Exam Trap: CloudWatch custom metrics require explicit PutMetricData API calls — they're not automatic. If a question describes monitoring a business metric (e.g., "records processed per batch"), you need to publish it as a custom metric from your Lambda or Glue code. Built-in metrics only cover infrastructure-level data.

Reflection Question: A Kinesis Data Streams consumer processes records for a real-time dashboard. The dashboard occasionally shows stale data. Which CloudWatch metric would you alarm on, and what does a high value indicate?