2.5.1. Choosing: Dataflow Gen2 vs. Pipeline vs. Notebook
💡 First Principle: Each orchestration tool has distinct strengths, like specialized vehicles: Dataflows are the family sedan (easy, reliable, limited cargo), Pipelines are the logistics truck (coordinates deliveries), and Notebooks are the custom-built race car (powerful but requires a skilled driver). The right choice depends on your team's skills and requirements.
Scenario: Your team needs to: (1) Extract customer data from Salesforce, (2) Transform it by merging with product data, (3) Load into a lakehouse, (4) Run a quality check notebook, (5) Trigger a Power BI refresh. This requires multiple tools working together.
Decision Framework
| Requirement | Best Tool |
|---|---|
| No-code/low-code ETL | Dataflow Gen2 |
| Orchestrate multiple activities | Data Pipeline |
| Complex transformations (code) | Notebook |
| Real-time event routing | Eventstream |
| Simple data copy | Copy Data activity (in pipeline) |
Visual: Tool Selection Decision Tree
Dataflow Gen2
- Engine: Power Query (M)
- Interface: Visual, no-code
- Best For: Business users, simple transformations
- Limitations: Not suitable for massive datasets or complex logic
Data Pipeline
- Engine: Azure Data Factory core
- Interface: Visual orchestration canvas
- Best For: Orchestrating multiple activities, scheduling, error handling
- Activities: Copy Data, Dataflow, Notebook, Stored Procedure, etc.
Pipeline Activities and Control Flow:
| Need | Activity / Setting |
|---|---|
| Run a notebook, dataflow or stored procedure as a step | Notebook / Dataflow / Stored procedure activity |
| Call an external REST endpoint (e.g., send a notification) | Web activity |
| Repeat the same steps for each item in an array | ForEach (inside it, @item() is the current element) |
| Branch on a true/false expression | If Condition (Switch for many values) |
| Loop until a condition becomes true | Until |
| Hold a value that changes during the run | Set Variable (variables are mutable; parameters are fixed at start) |
| Reuse another pipeline as a step | Invoke pipeline (ADF calls it Execute Pipeline): a parent calls a child. A schedule or trigger instead starts a pipeline on its own |
| Refresh a semantic model after loading | Semantic model refresh activity. A model's own refresh schedule is separate from the pipeline's |
| Stop with a clear error | Fail activity (see 4.2.1) |
Dependencies (the arrows between activities): each connection has a condition: on success, on failure, on completion or on skip. An activity with several incoming dependencies waits for all of them, so C runs only after both A and B succeed. If A fails, activities that depend on A's success are skipped. Activities with no dependency between them run in parallel. Because of the "all" rule, one alert activity wired to five on-failure arrows fires only if all five fail, so give each branch its own failure path. Retry, timeout and failure handling are covered in 4.2.1.
Notebook
- Engine: Apache Spark
- Interface: Code (PySpark, Spark SQL)
- Best For: Complex transformations, ML, large-scale processing
- Integration: Can be called from pipelines
- vs. Spark Job Definition: a Spark job definition submits a prebuilt Spark application or script as a non-interactive batch job. Choose a notebook when the work is interactive, iterative or co-authored
⚠️ Exam Trap: The Copy Data activity moves data from source to destination with basic mapping but does not transform data. For transformations, use Dataflow Gen2 or Notebook. Questions that mention needing transformations with Copy Data are testing whether you recognize this limitation.
