4.3.2. CloudTrail, CloudTrail Lake, and Log Analysis
💡 First Principle: CloudTrail records every API call in your AWS account — who did what, when, from where, and what happened. If CloudWatch answers "is the system healthy?", CloudTrail answers "who touched the system?" This is the foundation of security auditing, compliance, and incident investigation for data pipelines.
CloudTrail captures management events (API calls that create, modify, or delete AWS resources) by default. Data events (S3 object-level operations, Lambda invocations, DynamoDB item-level operations) require explicit configuration and incur additional cost. For data engineering, data events are critical — they reveal who accessed which S3 objects and when. Data events are not retroactive: they are recorded only from the moment you enable them, and management events alone never show DeleteObject. S3 server access logs also record requests but must be enabled beforehand and are delivered best-effort; AWS Config records bucket configuration changes, not object-level deletes.
CloudTrail Lake (closed to new customers since May 31, 2026 — existing event data stores keep working, and AWS recommends Amazon CloudWatch for new deployments) stores CloudTrail events in a managed, queryable data store. Instead of sending trail logs to S3 and querying them with Athena (the traditional approach), CloudTrail Lake provides a built-in SQL query interface with faster performance and simpler setup. Use it for centralized security queries across accounts and regions.
Log analysis patterns tested on the exam:
| Log Source | Analysis Service | Use Case |
|---|---|---|
| Application logs | CloudWatch Logs Insights | Debugging pipeline failures |
| API audit trails | CloudTrail Lake or Athena | Security investigation, compliance |
| High-volume logs | Amazon OpenSearch | Interactive search, dashboards |
| Large-scale log ETL | Amazon EMR | Processing TB+ of log data |
⚠️ Exam Trap: CloudTrail logs API calls, not application-level data access. If a question asks "who queried this Redshift table?", the answer is Redshift audit logging (not CloudTrail). CloudTrail would show who called the Redshift API (created a cluster, modified settings), but not the SQL queries executed within Redshift.
Reflection Question: Security needs to investigate whether any IAM user accessed a specific S3 bucket containing PII data during the last 30 days. Which service provides this information, and what event type must be enabled?
