5.1. The Modern Data Warehouse Architecture
💡 First Principle: The modern data warehouse isn't a single product—it's a pipeline of specialized services, each optimized for one job. This is the Data Lifecycle from Phase 1 in action: Ingest → Store → Process → Serve. Think of it like an assembly line: raw materials enter (ingestion), get processed through stations (transformation), stored in warehouses (data lakes), and finally delivered as finished products (reports). No single service does everything well, so we chain specialists together. The magic happens in how they connect.
What breaks without this architecture? Store raw data directly in a relational database, and you'll pay premium prices for storage that doesn't need SQL capabilities. Try to transform data in your BI tool, and it will crash under the load. Run analytics on your production database, and transactions grind to a halt. The architecture exists because each stage has fundamentally different requirements.
Consider the data journey: raw JSON arrives from web APIs (ingestion via Data Factory), lands in cheap object storage (Data Lake Gen2), gets cleaned and modeled (Synapse or Databricks), and finally appears as executive dashboards (Power BI). Each handoff is intentional.
Visual: Modern Analytics Pipeline
The Stage-to-Service Map
| Stage | Question it answers | Azure services |
|---|---|---|
| Ingest | How does data get in? | Data Factory (batch/scheduled), Event Hubs and IoT Hub (streaming) |
| Store | Where does it rest? | Data Lake Storage Gen2, Blob Storage, Azure SQL, Cosmos DB |
| Process | How is it transformed? | Synapse Analytics, Azure Databricks, Stream Analytics |
| Serve | How do people consume it? | Power BI, APIs, applications |
The Second Axis: Batch or Streaming
Naming the stage narrows the field; the batch-versus-streaming split usually finishes the job, and it is the most frequently tested distinction in this domain.
- Batch — data accumulates and is processed on a schedule or a trigger. Data Factory ingests it; Synapse or Databricks process it. Latency is minutes to hours and that is acceptable.
- Streaming — each event is handled as it arrives. Event Hubs or IoT Hub ingest it; Stream Analytics processes it. Latency is seconds because something must react.
⚠️ Exam Trap: the big services deliberately straddle stages — Synapse stores and processes, Fabric does all four — so "this service could do it" is never sufficient. Find the stage the requirement names, then apply the batch/streaming test. Two clues, and almost every question in this phase resolves to one service.
