2.6.3. Event-Driven Architectures with SNS and SQS

💡 First Principle: SNS and SQS decouple pipeline components by replacing direct invocations with message passing. Instead of Service A directly calling Service B (tight coupling — if B is down, A fails), Service A publishes a message to a topic or queue, and Service B consumes it independently. This decoupling makes pipelines more resilient, scalable, and maintainable.

Amazon SNS (Simple Notification Service) implements the publish/subscribe (pub/sub) pattern. A producer publishes a message to a topic, and all subscribers receive it — fan-out. Subscribers can be SQS queues, Lambda functions, HTTP endpoints, email addresses, or SMS. In data pipelines, SNS is commonly used for: alerting on pipeline failures (SNS → email), distributing events to multiple consumers (SNS → multiple SQS queues), and triggering Lambda functions from pipeline events.

Amazon SQS (Simple Queue Service) implements the producer/consumer pattern with a queue. Messages are stored until a consumer processes them. Key features for data engineering: visibility timeout (prevents two consumers from processing the same message), dead letter queues (DLQ) (captures messages that repeatedly fail processing — critical for debugging), and FIFO queues (guarantee message ordering and exactly-once delivery). SQS buffers between pipeline components, absorbing bursts without overwhelming downstream services.

SNS + SQS pattern (fan-out). A single SNS topic fans out to multiple SQS queues, each feeding a different consumer. This is the standard pattern when one event (e.g., "new file arrived in S3") needs to trigger multiple independent processing paths — one queue for ETL, another for data quality checks, a third for notifications.

⚠️ Exam Trap: Standard SQS queues deliver messages at least once and don't guarantee ordering. If a question requires exactly-once processing and strict ordering, the answer is FIFO SQS (with deduplication ID and message group ID). Standard SQS is sufficient for most data pipeline use cases where idempotent processing handles duplicates.

Reflection Question: A data pipeline must notify three different teams when processing completes: the analytics team (email), the data science team (Lambda trigger for model retraining), and the operations team (Slack webhook). What architecture handles this?