2.2.1. Amazon S3 as an Ingestion Layer

💡 First Principle: S3 serves as the universal landing zone for batch data because it decouples producers from consumers. Any source can write to S3 — applications, databases, third-party feeds — and any consumer can read from it. This decoupling means producers and consumers can evolve independently, and data is durable (99.999999999%) from the moment it lands.

In most data lake architectures, S3 is where raw data first arrives. Application logs are shipped via agents, database exports land as CSV or Parquet files, and third-party data feeds drop files on a schedule. S3's event notification system then triggers downstream processing — a new file in the raw/ prefix fires an S3 Event Notification to Lambda or EventBridge, which kicks off the transformation pipeline.

Key S3 ingestion patterns:

Direct upload. Applications write directly to S3 using the AWS SDK. For large files (>100 MB), multipart upload splits the file into parts uploaded in parallel. S3 Transfer Acceleration uses CloudFront edge locations to speed up uploads from distant locations — look for this when questions mention geographically distributed data sources.

S3 Batch Operations. Perform operations on billions of existing S3 objects — copy, invoke Lambda, restore from Glacier, or apply tags. If a question describes needing to process or transform a large number of existing S3 objects, Batch Operations is the answer, not writing a custom script.

S3 Event Notifications. When objects are created, deleted, or restored, S3 can notify Lambda, SQS, SNS, or EventBridge. This is the glue for event-driven batch architectures: "when a new file arrives, process it." EventBridge integration (the newer option) supports filtering, routing to multiple targets, and cross-account delivery.

⚠️ Exam Trap: S3 Event Notifications can send to Lambda, SQS, SNS, or EventBridge — but only EventBridge supports advanced filtering (e.g., matching on object key prefix AND suffix AND size). If a question requires complex event filtering, the answer is S3 → EventBridge → target, not S3 → Lambda directly.

Reflection Question: A partner company drops 500 CSV files daily into your S3 bucket at unpredictable times. You need to process each file within 15 minutes of arrival. Which S3 feature triggers the processing, and what target service would you use?