3.2.1.8. Creating CloudWatch Metric Streams (Amazon S3 or Amazon Kinesis Data Firehose options)

3.2.1.8. Log Lifecycle Management & Search

Logs accumulate fast. Without lifecycle management, CloudWatch Logs costs grow unbounded and search performance degrades.

Retention policies: Set per log group (1 day to 10 years, or indefinite). After the retention period, log events are automatically deleted.

Cost-effective tiering:

1. CloudWatch Logs (hot): Recent logs for real-time queries. $0.50/GB ingestion + $0.03/GB storage/month.
2. S3 Standard (warm): Exported logs for Athena analysis. $0.023/GB/month.
3. S3 Glacier (cold): Compliance archive. $0.004/GB/month.
4. S3 Glacier Deep Archive (frozen): Legal hold. $0.00099/GB/month.

Log export to S3:

# Export last 24 hours of logs to S3
aws logs create-export-task \
  --log-group-name "/prod/app/api" \
  --from $(date -d "yesterday" +%s)000 \
  --to $(date +%s)000 \
  --destination "log-archive-bucket" \
  --destination-prefix "prod/api"

CloudWatch Logs Insights provides SQL-like queries across log groups:

fields @timestamp, @message
| filter @message like /ERROR/
| stats count(*) as errorCount by bin(5m)
| sort errorCount desc
| limit 20

Amazon OpenSearch (successor to Elasticsearch) provides full-text search, visualization (Kibana/OpenSearch Dashboards), and anomaly detection on log data. Stream logs via Kinesis Firehose for near-real-time analysis.

Exam Trap: CloudWatch Logs CreateExportTask only exports to S3 — and it's a one-time operation, not continuous. For continuous export to S3, use a subscription filter with Kinesis Data Firehose. The exam may present both options; choose CreateExportTask for ad-hoc/historical exports and Firehose for continuous streaming.