2.2.5. Normalizing, Parsing, and Correlating Logs
First Principle: Raw logs from different sources have different formats, timestamps, and field names. Without normalization and correlation, it's impossible to reconstruct a complete attack timeline from disparate log types.
Amazon OpenSearch Service provides full-text search and analytics:
- Ingest logs from CloudWatch Logs, Kinesis, or direct API
- Powerful query language for complex searches and aggregations
- Visualization dashboards (OpenSearch Dashboards, formerly Kibana)
- Best for: high-volume log search, pattern detection, security dashboards
AWS Lambda for custom log processing:
- Transform log formats between services (e.g., convert VPC Flow Logs to OCSF)
- Enrich logs with additional context (e.g., add account name from Organization metadata)
- Filter and route logs to different destinations based on content
- Triggered by CloudWatch Logs subscription filters, S3 events, or Kinesis
Amazon Managed Grafana (new in C03) for security visualization:
- Create unified dashboards across multiple AWS data sources
- Query CloudWatch, Athena, OpenSearch, and Prometheus from a single interface
- Share dashboards with security teams using AWS IAM Identity Center integration
- Best for: real-time security operations dashboards and trend analysis
⚠️ Exam Trap: Managed Grafana is for visualization and dashboarding — it doesn't store or process logs itself. It connects to data sources (CloudWatch, OpenSearch, Athena) and presents them visually.
Scenario: A SOC team needs a real-time dashboard showing GuardDuty finding trends, VPC Flow Log anomalies, and CloudTrail error rates — all on one screen. They deploy Managed Grafana with data sources connected to CloudWatch (metrics), OpenSearch (Flow Logs), and Athena (CloudTrail).
Reflection Question: Why is a unified visualization layer critical for security operations, and how does Managed Grafana solve the "too many dashboards" problem?
