2.2.1. Spark Workspace Settings
💡 First Principle: Apache Spark distributes work across multiple "executors" (worker processes). Think of it like a restaurant kitchen: having 20 chefs when you only have 5 orders wastes money, but having 2 chefs during dinner rush means customers wait forever. Dynamic allocation adjusts your chef count based on actual orders.
Scenario: Your data engineering team runs Spark jobs with varying complexity—some process 1 GB of data, others process 1 TB. Fixed resource allocation means either wasting money on small jobs or failing on large ones.
Starter Pool vs. Custom Pools
- Starter Pool:
- Pre-warmed Spark cluster for fast session start—best-effort: sessions usually start in seconds, but when no pre-warmed capacity is free Fabric provisions on demand and startup takes longer
- Limited customization: Medium nodes only; a workspace admin can adjust autoscale (maximum nodes, up to the capacity SKU's limit) and dynamic executor allocation
- Best for: Development, small workloads, quick testing
- Custom Spark Pool:
- User-defined node sizes and counts
- Supports autoscaling
- Session start: about 3 minutes, because clusters are provisioned on demand—unless the pool is configured as a custom live pool, which keeps clusters warm during a scheduled active window so sessions start in about 5 seconds
- Best for: Production workloads, specific resource requirements
Key Spark Configuration Options
| Setting | Purpose | When to Enable |
|---|---|---|
| Dynamic Allocation | Adjusts executor count based on workload | Always for variable workloads |
| Autoscale | Adjusts node count based on demand | Production environments with varying loads |
| Memory-Optimized Nodes | Higher memory ratio for data-intensive operations | Large dataset processing, complex transformations |
| Compute-Optimized Nodes | Better for CPU-bound operations | ML training, complex calculations |
spark.sql.shuffle.partitions | Partition count after a shuffle (joins, aggregations); default 200 | Tune for data volume, or let AQE coalesce |
| Adaptive Query Execution (AQE) | Uses runtime statistics to coalesce shuffle partitions, switch join strategies and split skewed partitions (on by default) | Leave enabled; it removes most manual partition tuning |
Driver vs. executors: the driver plans the job and collects results (collect(), toPandas()); executors process the partitions. An out-of-memory error on the driver is not fixed by adding executor memory. Spark also evaluates lazily: transformations only build a plan, and nothing runs until an action (count(), write, show()), which lets the optimizer see the whole chain before executing it.
Environments: the starter pool is every workspace's default—no setup needed. A Fabric Environment item bundles the Spark runtime, libraries (public PyPI/Conda packages pinned to versions, plus custom .whl/.jar files) and Spark properties; changes take effect only after you Publish it. Attach it to notebooks and Spark job definitions, or set it as the workspace default in the workspace's Spark settings, so every notebook gets the same versions instead of ad-hoc %pip install. After a runtime upgrade, update or remove incompatible libraries and republish. Libraries don't travel inside the notebook: a notebook promoted to another workspace fails (for example with ClassNotFoundException) if the environment it runs with there doesn't supply the same library.
Visual: Dynamic vs. Static Allocation
⚠️ Exam Trap: More partitions don't always improve performance. Excessive partitions create overhead from task scheduling and shuffle operations. Partition count should align with executor count and data volume—hundreds of tiny partitions can be slower than dozens of right-sized ones.
Session Tags for Session Reuse
- Concept: Reuse existing Spark sessions across pipeline activities to eliminate cold start latency (30-60 seconds per session)
- Implementation: Tag sessions in Data Factory pipeline Spark activities
- Use Case: Multiple notebook activities in a single pipeline run
- High concurrency mode: a workspace Spark setting that lets several notebooks share one running session instead of each starting its own; in pipelines, notebook activities that carry the same session tag share that session
Reflection Question: Your Spark jobs consistently use only 20% of allocated resources. What combination of settings would you adjust to optimize cost without risking job failures?
