4.4.2. Data Sampling and Skew Mechanisms

💡 First Principle: Not all data needs equal treatment. Sampling lets you validate quality on a statistically representative subset instead of scanning every record — like a quality inspector checking 100 items per batch rather than every item off the assembly line. Data skew, conversely, is when data is unevenly distributed across partitions, causing some workers to be overloaded while others sit idle.

Sampling techniques: random sampling (select N% of records randomly), stratified sampling (maintain proportional representation of subgroups), and systematic sampling (select every Nth record). For quality validation, random sampling with sufficient sample size provides high confidence at a fraction of the cost.

Data skew occurs when a partition key concentrates data unevenly. In Spark (Glue, EMR), skew manifests as one task taking far longer than others — the job is only as fast as the slowest task. Common causes: a single customer with millions of records while most have hundreds, null values in a partition key (all nulls land in one partition), or temporal skew (holiday sales creating massive partitions).

Skew mitigation strategies: salting (append a random suffix to the skewed key to distribute across partitions), broadcast joins (replicate the smaller table to all workers when joining with a skewed large table), adaptive query execution (Spark 3.x+ dynamically coalesces skewed partitions), and key redesign (use a more evenly distributed partition key).

⚠️ Exam Trap: If a Spark job has 99 tasks completing in 5 minutes and 1 task taking 45 minutes, the answer is data skew — not "add more workers." Adding workers doesn't help because the bottleneck is a single overloaded partition. The fix is skew mitigation: salting, repartitioning, or broadcasting the join table.

Reflection Question: A Glue Spark job joins a 100 GB transactions table with a 500 MB stores table. The job takes 2 hours, but profiling shows one task processing 40% of the data. What's happening, and what Spark technique fixes it?