1.1.2. The Five Stages of a Data Pipeline

💡 First Principle: Every data pipeline — regardless of complexity — moves through the same five stages: ingest, transform, store, serve, and orchestrate. Understanding these stages gives you a framework for mapping any exam scenario to the right AWS services.

Ingest is about getting data from its source into your pipeline. This could be reading records from a streaming source like Kinesis, pulling batch files from S3, or replicating a database via DMS. The key decisions here are how often (real-time vs batch), how much (throughput), and how reliably (replayability, exactly-once delivery).

Transform is where raw data becomes useful. You clean dirty records, convert formats (CSV to Parquet), join data from multiple sources, aggregate values, and apply business logic. AWS Glue, EMR with Spark, Lambda, and Redshift stored procedures are the primary tools here — and the exam frequently tests which one to choose.

Store means placing transformed data where it can be efficiently queried. S3 for data lakes, Redshift for warehousing, DynamoDB for low-latency key-value lookups, and RDS/Aurora for relational needs. Storage selection depends on access patterns, query types, cost tolerance, and latency requirements.

Serve is how consumers access the data — through SQL queries in Athena or Redshift, dashboards in QuickSight, APIs via API Gateway, or direct S3 reads. The serve layer is where pipeline value is realized.

Orchestrate ties the other four stages together. Step Functions, MWAA (Airflow), EventBridge, and Glue workflows coordinate when each stage runs, handle dependencies between stages, and manage retries when things fail. Without orchestration, a pipeline is just a collection of disconnected scripts.

On this exam, most questions implicitly test one or two stages. When you see a scenario, immediately ask yourself: "Which stage is this question really about?" That narrows the service options dramatically.

⚠️ Exam Trap: Orchestration is a stage, not an afterthought. Questions that describe multi-step pipelines failing inconsistently are usually testing whether you understand Step Functions error handling or Airflow retry logic — not the individual services.

Reflection Question: A company ingests clickstream data in real time, transforms it hourly, and loads it into Redshift for dashboards. Which of the five stages is the clickstream consumer responsible for? Which stage is the hourly Glue job?