5.1.1. Data Ingestion with Azure Data Factory
💡 First Principle: Data Factory is an orchestration service—it moves and transforms data but doesn't store or analyze it. Think of it as the logistics company that picks up packages, routes them through distribution centers, and delivers them to destinations. It coordinates the work; others do the heavy lifting.
ETL vs. ELT (Critical Concept!)
| Approach | Process | When to Use |
|---|---|---|
| ETL | Extract → Transform → Load | Traditional; transform before loading |
| ELT | Extract → Load → Transform | Modern; load raw data first, transform with cloud compute |
Modern preference: ELT. Load raw data into cheap storage (Data Lake), then use powerful cloud compute (Synapse, Databricks) to transform. Storage is cheap; compute is flexible.
The Five Building Blocks of a Pipeline
💡 First Principle: Data Factory separates where the data is, what it looks like, what to do, when to do it, and what runs it. Almost every ADF exam question is really asking which of these five you need.
| Object | What it is | Think of it as |
|---|---|---|
| Linked Service | Connection information and credentials for a data store or compute resource | The connection string |
| Dataset | The structure and location of the data within that store — a table, a folder, a file | The address inside the building |
| Activity | A single step, such as Copy, Lookup, or a notebook call | One instruction |
| Pipeline | A logical grouping of activities that together perform a task, run in sequence or in parallel | The recipe |
| Trigger | What causes a pipeline to run | The alarm clock |
Trigger types: a schedule trigger fires on a clock (nightly at 2 AM); a storage event trigger fires when a blob is created or deleted; a tumbling window trigger fires over fixed, non-overlapping time slices and can backfill history.
Integration Runtime: the compute that actually does the work
💡 First Principle: A pipeline is only a definition. The Integration Runtime (IR) is the compute that executes it — and which IR you need is decided by where the data lives.
| Integration Runtime | Use it when |
|---|---|
| Azure IR | Both source and sink are reachable over the public cloud network |
| Self-hosted IR | Data sits on-premises or inside a private network — you install the IR on a machine there so it can reach the source |
| Azure-SSIS IR | You are lifting existing SQL Server Integration Services packages into Azure |
Reasoning Tool: if a question says "on-premises SQL Server" and asks what must be installed, the answer is the Self-hosted Integration Runtime. ExpressRoute or a VPN may also exist, but they are network plumbing — the IR is the Data Factory component.
Mapping Data Flows
Mapping data flows let you build transformations visually, with no code; Data Factory executes them on a managed Spark cluster behind the scenes. They transform data — they do not train models or schedule anything.
