3.2.1.9. Cost-Effective Observability (Log Filtering, Metric Aggregation, Retention)
3.2.1.9. Cost-Effective Observability (Log Filtering, Metric Aggregation, Retention)
💡 First Principle: Observability spend grows with what you ingest, how many distinct metric series you create, and how long you keep data in a premium store. Decide what you must alert on, what you must be able to search, and what you only must keep, and pay the premium price only for the first two.
Scenario: A platform team's "CloudWatch" bill has passed its "EC2" bill. Applications log at DEBUG level in production, every log group is set to never expire, a latency metric is published with a per-request ID dimension, and a 7-year audit requirement is being met by keeping everything in "CloudWatch Logs".
| Lever | Mechanism | What it saves |
|---|---|---|
| Filter at the source | "CloudWatch agent" filters (include/exclude regular expressions per log file); "Lambda" advanced logging controls (JSON log format with ApplicationLogLevel/SystemLogLevel, such as WARN or ERROR in production); application log levels; "VPC Flow Logs" with only the fields and traffic type you need, delivered to "S3" | Ingestion, the largest log cost, before it is incurred |
| Log class | Infrequent Access log groups cost about half as much per GB ingested; storage and Logs Insights prices are the same | Ingestion for logs kept for ad hoc queries and forensics |
| Retention | Set a retention period on every log group (the default is never expire); archive long-term copies to "S3" with a lifecycle to "Glacier" classes | Storage of logs nobody queries |
| Metric aggregation | Publish at the dimension you alarm on (AutoScalingGroupName, service) instead of per instance or per request; the agent's aggregation_dimensions rolls metrics up | Custom metric count |
| Resolution and sampling | Keep standard 60-second resolution unless an alarm needs seconds; sample traces with "X-Ray" sampling rules | Metric and trace volume |
Key details:
- The Infrequent Access log class is a subset: It supports managed ingestion and storage,
"KMS"encryption, cross-account analytics, most Logs Insights queries and export to"S3", but not metric filters, subscription filters, Live Tail, anomaly detection, the embedded metric format, or"Container Insights"/"Lambda Insights"ingestion, and it cannot be read withGetLogEvents/FilterLogEvents. The class is chosen when the log group is created and cannot be changed, so moving a log stream means creating a new log group. Logs that drive alarms or stream to another system stay in Standard. (A third,"Delivery"class exists only for"Lambda"logs sent on to"S3"or"Firehose".) - Custom metrics are billed per unique series: Each combination of metric name and dimension values is a separate custom metric. A dimension with unbounded values (request ID, user ID, container ID, or instance ID on a large fleet) turns one metric into thousands. Aggregate to the dimensions you alarm and graph on, and put per-request detail in logs where Logs Insights can query it. The embedded metric format extracts metrics from structured logs without separate
PutMetricDatacalls, but each extracted series is still a custom metric and the log line is still ingested. - Long retention belongs in
"S3": For multi-year compliance, set a short"CloudWatch Logs"retention (for example 30 to 90 days for investigations) and send a copy to"S3"through a subscription filter to"Firehose"(continuous) or an export task (batch), then lifecycle it to"Glacier Flexible Retrieval"or"Deep Archive". Setting a shorter retention on an existing log group deletes the older events, so create the archive first. - Query cost: Logs Insights is billed per GB scanned, so narrow queries by time range and log group, and alarm on metric filters or metrics rather than scheduling repeated scans of raw logs.
⚠️ Exam Trap: Moving every log group to the Infrequent Access class to cut ingestion. Any log group that feeds a metric filter, alarm, subscription filter or Live Tail session stops working there, and the class cannot be switched back, so only logs used for after-the-fact querying belong in it.