3.2.1.7. Creating CloudWatch Metrics from Log Events (Metric Filters)
3.2.1.7. Streaming & Subscription-Based Log Processing
Sometimes you need logs processed in real-time — for alerting, transformation, or forwarding to external systems. CloudWatch Logs subscriptions enable this.
Subscription filters stream matching log events to a destination in real-time:
| Destination | Use Case | Latency |
|---|---|---|
| Lambda | Transform, filter, or enrich log events | Seconds |
| Kinesis Data Streams | High-volume streaming to multiple consumers | Seconds |
| Kinesis Data Firehose | Batch delivery to S3, OpenSearch, Splunk | 60-900 seconds |
| OpenSearch | Full-text search and visualization | Seconds (via Lambda/Firehose) |
# Stream ERROR logs to a Lambda function for alerting
aws logs put-subscription-filter \
--log-group-name "/prod/app/api" \
--filter-name "ErrorsToLambda" \
--filter-pattern '"ERROR"' \
--destination-arn "arn:aws:lambda:us-east-1:123456789012:function:ErrorAlertHandler"
Cross-account log aggregation uses subscription filters to stream logs from application accounts to a centralized logging account. In the logging account you create a CloudWatch Logs destination — a resource that wraps a Kinesis Data Stream or Firehose stream — whose access policy names the source accounts (or organization) allowed to call PutSubscriptionFilter against it; each source account then points its subscription filter at the destination ARN.
Architecture pattern: Centralized logging
Account A → CW Logs → Subscription Filter → Kinesis Firehose → S3 (Log Archive Account)
Account B → CW Logs → Subscription Filter ↗
Account C → CW Logs → Subscription Filter ↗
Exam Trap: Each log group supports only two subscription filters maximum. If you need to stream logs to three destinations, use a single Kinesis Data Stream as the subscription target and then fan out to multiple consumers (Lambda, Firehose, custom app) from the stream. The two-filter limit is a hard limit that can't be increased.
