Copyright (c) 2026 MindMesh Academy. All rights reserved. This content is proprietary and may not be reproduced or distributed without permission.
4.3.3. Spark Performance Optimization
💡 First Principle: Spark performance depends on parallelism (partition count), data locality (caching), and resource utilization (executor configuration). Optimizing one dimension while ignoring others produces limited improvement.
Key Optimization Techniques
| Technique | Purpose | When to Use |
|---|---|---|
| Dynamic Allocation | Right-size executor count | Variable workloads |
| Caching | Keep frequently accessed data in memory | Iterative processing |
| Broadcast Join | Avoid shuffle for small tables | Joining large + small tables |
| Filter and select early | Spark pushes predicates into the scan and fuses consecutive filters into one pass | Always filter before aggregating or joining |
Lazy evaluation + cache() | Every action (count(), write) re-runs the whole lineage from source unless the DataFrame is cached (memory, spilling to disk) | Cache a DataFrame reused by 2+ actions and unpersist() after. Caching a single-use result only adds cost |
coalesce(n) vs. repartition(n) | coalesce merges partitions with no full shuffle (reduce only); repartition does a full shuffle | Reduce partitions or output files before writing |
| Partitions = parallelism | One task per partition, run concurrently across executors | Too few leaves cores idle; too many adds overhead |
| Shuffle spill | Shuffle data larger than executor memory spills to disk, so runtime can jump non-linearly as data grows | More memory or shuffle partitions; shuffle less data |
| Partition Pruning | Skip irrelevant partitions | Filtered queries |
Native Execution Engine
- Concept: Fabric's optimized Spark engine for common operations
- Benefit: Significant performance improvement for supported operations
- Limitation: Does not support User-Defined Functions (UDFs)
⚠️ Exam Trap: Using UDFs with native execution engine causes fallback to traditional Spark. If using UDFs, expect performance regression compared to native-supported operations. Questions about "slow UDF performance" are testing this knowledge.
Shuffle, skew, and memory tuning:
| Symptom | Cause | Fix |
|---|---|---|
| Most time in shuffle when joining a big table to a small one | Both sides shuffled (sort-merge join) | Broadcast the small side—automatic below spark.sql.autoBroadcastJoinThreshold (10 MB default), or forced with a broadcast() hint. Every executor holds a full copy, so a side of hundreds of MB suits broadcast only if executor and driver memory can take it; otherwise keep the sort-merge join |
| A few tasks take far longer than the rest, all on the same key | Data skew—uneven data across partitions | AQE skew-join handling (spark.sql.adaptive.enabled + spark.sql.adaptive.skewJoin.enabled) splits oversized partitions; for severe or aggregation skew, salt the hot key (add a random suffix, aggregate, then merge the partial results) |
| Slow or spilling shuffle stage with large partitions | Too few shuffle partitions (spark.sql.shuffle.partitions, default 200) | Raise it—but not blindly: thousands of tiny tasks add scheduling overhead (see 2.2.1) |
| Executor out-of-memory, "container killed for exceeding memory," high GC time | Memory pressure | Increase spark.executor.memory (or use memory-optimized nodes); reduce data per task |
- Adaptive Query Execution (AQE) re-plans at runtime from actual shuffle sizes: it coalesces small shuffle partitions, switches a sort-merge join to a broadcast join when one side turns out small, and splits skewed partitions.
- Dynamic allocation adds and releases executors with demand to save cost; scaling up adds startup latency, and jobs that need executors to stay put (streaming state, long-lived cached DataFrames) often run with a fixed executor count instead.
- Python UDFs process rows one at a time across the JVM–Python boundary and fall back from the native engine; prefer built-in functions, or vectorized Pandas UDFs when custom logic is unavoidable.
- Capacity-wide slowdowns: before tuning one engine, check the Fabric Capacity Metrics app for CU utilization and throttling, then Spark pool settings, Delta table health (small files, statistics), and warehouse workload isolation (custom SQL pools, preview).
Enroll to unlock the 39 practice questions written for this section, so you can test what you just read while it is fresh.
Enroll to unlock the 6 flashcards for this section and review them on a spaced-repetition schedule.
Written byAlvin Varughese
Founder•20 professional certifications