4.2.2. Event-Driven Automation: Lambda and S3 Event Notifications
💡 First Principle: Polling is expensive and slow. Instead of checking "has anything changed?" every 60 seconds, event-driven architecture inverts this: resources announce when they change, and automation responds immediately. This makes systems faster, cheaper, and more responsive than any polling approach.
AWS services emit events when things happen: an S3 object is uploaded, a DynamoDB stream record is written, an EC2 instance changes state. By connecting these events to Lambda functions, you create automation that responds in near real-time without any infrastructure to manage.
S3 Event Notifications:
S3 can publish events when objects are created, deleted, tagged, restored from Glacier, or replicated. Destinations:
| Destination | Use Case |
|---|---|
| SNS | Fan-out notification to multiple subscribers |
| SQS | Queued processing (handles backpressure for high-volume events) |
| Lambda | Direct invocation for immediate processing |
| EventBridge | Advanced routing with filtering and cross-account delivery |
Example pipeline: Images uploaded to S3 → S3 event → Lambda → generate thumbnail → store in different S3 prefix.
EventBridge as the Central Event Bus:
EventBridge (covered in Phase 2) is the preferred integration point for operational automation because it supports:
- Pattern matching on event content
- Multiple simultaneous targets
- Cross-account and cross-region routing
- Scheduled rules (cron-based automation)
- Dead-letter queues for failed deliveries
When events flow through an SQS queue to Lambda, failures are handled by the queue: its redrive policy names a dead-letter queue and a maxReceiveCount, and a message received that many times without being deleted moves to the DLQ instead of expiring, so it can be inspected and redriven. Longer retention or more retries alone still lose a message that can never succeed.
AWS Step Functions orchestrates multi-step workflows where the overall process must be tracked, steps may have human approval requirements, or individual steps can fail and need retry logic:
Lambda Limitations for Automation:
| Constraint | Value | Implication |
|---|---|---|
| Max execution timeout | 15 minutes | Long operations need Step Functions or SSM Automation |
| Max memory | 10,240 MB | Memory-intensive operations may need EC2 or Fargate |
| Max package size | 250 MB (unzipped) | Large dependencies need Lambda layers or container images |
| Concurrent executions | 1,000 (default, adjustable) | High-volume event processing needs throttling consideration |
Common Event-Driven Automation Patterns:
| Trigger | Event | Lambda Action |
|---|---|---|
| S3 upload | s3:ObjectCreated | Parse, transform, validate, route |
| EC2 state change | EC2 Instance State-change | Update DNS, notify, deregister from monitoring |
| Config rule violation | Config Rules Compliance Change | Auto-remediate, notify, create OpsItem |
| CloudTrail API call | AWS API Call via CloudTrail | Detect suspicious activity, block access, alert |
| DynamoDB stream | Record insert/modify/delete | Replicate to Elasticsearch, audit, trigger downstream |
⚠️ Exam Trap: S3 event notifications and EventBridge are two separate systems for S3 events. S3 Event Notifications deliver directly to SNS/SQS/Lambda with no filtering beyond event type. EventBridge receives all S3 events (when enabled) and supports rich filtering by prefix, suffix, event type, and metadata — plus cross-account delivery. For complex filtering or cross-account routing, enable S3 → EventBridge integration and use EventBridge rules.
Event-Driven Investigation with AWS DevOps Agent and EventBridge:
AWS DevOps Agent (see 2.3.4) plugs into event-driven automation in both directions, and the two directions use different mechanisms:
- Into the agent: investigations are started by built-in integrations (a ServiceNow ticket, a Dynatrace Workflow), by webhooks (PagerDuty, Grafana alerts, or a generic HMAC or API-key webhook that any tool can POST to), or manually from the web app.
- Out of the agent: every investigation and mitigation state change is published to the EventBridge default event bus with
source=aws.aidevops. No extra permission is needed for delivery; your rule targets only need the usual target permissions.
| Event Family | detail-type Values |
|---|---|
| Investigation | Investigation Created, Priority Updated, In Progress, Completed, Failed, Timed Out, Cancelled, Pending Triage, Linked, Skipped |
| Mitigation | Mitigation In Progress, Completed, Failed, Timed Out, Cancelled |
Rule pattern — page the team when an investigation cannot finish:
{
"source": ["aws.aidevops"],
"detail-type": ["Investigation Failed", "Investigation Timed Out"]
}
Documented patterns include invoking Lambda to process results when an investigation completes, sending an SNS notification when one fails or times out, updating a ticketing system when one is created, and starting a Step Functions workflow when a mitigation completes. To scope a rule to one Agent Space, match detail.metadata.agent_space_id; to match every investigation event, use a prefix match on detail-type ("prefix": "Investigation").
Four behaviors break consumers that assume otherwise:
| Behavior | Consequence |
|---|---|
A Completed event carries detail.data.summary_record_id but not the summary text | Call ListJournalRecords with the event's agent_space_id and execution_id and a recordType of investigation_summary_md or mitigation_summary_md, then pick the record whose recordId matches. Needs aidevops:ListJournalRecords. There is no call that fetches a single record by ID |
| Empty fields are omitted, not set to null | Treat execution_id and summary_record_id as optional. Failed, Timed Out and Cancelled events never include summary_record_id |
detail.data.task_type is INVESTIGATION for both investigation and mitigation events | Branch on detail-type, not task_type |
time is when the investigation or mitigation was last updated | It is not the time EventBridge received the event |
Incident prevention evaluations and custom agent runs do not emit these events — only investigations, including their mitigation phase.
⚠️ Exam Trap: The aws.aidevops events tell you what the agent did; they are not a remediation engine. A Lambda or Step Functions target that acts on a Mitigation Completed event runs with its own IAM role and your own logic, exactly like any other EventBridge-driven automation — the agent's approval model does not extend to it.
Reflection Question: Every time a new object is uploaded to an S3 bucket, you need to: (1) immediately trigger a Lambda to validate the file format, (2) send a notification to an SQS queue for async processing, and (3) route events matching *.csv to a different Lambda than events matching *.json. Which approach — S3 Event Notifications or EventBridge — best handles requirement 3, and why?
