2.2.2. Data Workflow Workspace Settings
💡 First Principle: An orchestrator is only useful if it is running when work arrives. Fabric's Apache Airflow runtime can be cheap-and-sleepy or always-on — choosing between them is the whole decision.
Scenario: Your team's Apache Airflow DAGs work fine in development, but in production a DAG triggered after a quiet evening sometimes waits minutes before starting, and on busy nights DAGs queue behind each other.
Apache Airflow Runtime Settings
Apache Airflow Jobs (formerly Data Workflows) run on the workspace's Apache Airflow runtime. Admins configure it in workspace settings → Data Factory → Apache Airflow Runtime Settings, where the Default Data Workflow Setting is either the starter pool or a custom pool; an individual Apache Airflow Job can also manage its own runtime settings.
| Property | Starter pool (default) | Custom pool |
|---|---|---|
| Size | Compute node size: Large (fixed) | You choose compute node size, extra nodes and autoscale |
| Startup | Instantaneous | Starts in the stopped state |
| Resume after idle | Up to 5 minutes | Up to 5 minutes |
| Uptime | Shuts down after 20 minutes of inactivity | Always on until manually paused |
| Suggested for | Development | Production |
Custom pool options:
- Compute node size —
Largefor complex or production DAGs,Smallfor simpler DAGs. - Enable autoscale — lets the pool scale nodes up or down as needed.
- Extra nodes — let the pool run more DAGs concurrently; each node provides capacity for three more workers.
If the setting for customizing compute configurations for items is disabled, the starter pool is used for every environment in the workspace.
⚠️ Exam Trap: These settings belong to Apache Airflow Jobs only. They do not throttle Dataflow Gen2 — a dataflow's concurrency is configured on the dataflow itself — and they are separate from the Spark pools used by notebooks.
Dataflow Gen2 settings live on each dataflow, not in workspace settings: in the dataflow's Power Query editor, Options → Scale sets the maximum number of parallel query evaluations (concurrency), Fast Copy and the modern query evaluation engine; staging is turned on per query (Enable staging). Overall resources come from the capacity.
Key Trade-Offs:
- Starter vs. custom: the starter pool costs nothing while idle but can make an early run wait; a custom pool is ready immediately but runs (and consumes capacity) until paused.
- Bigger nodes vs. more nodes: a larger node helps complex DAGs; extra nodes (or autoscale) help when many simple DAGs are queuing.
