Copyright (c) 2026 MindMesh Academy. All rights reserved. This content is proprietary and may not be reproduced or distributed without permission.
3.3.2. PySpark Transformations
💡 First Principle: PySpark provides distributed data processing for large-scale transformations. When data exceeds memory limits or transformations require complex logic, Spark distributes the work across multiple nodes.
Common PySpark Operations
# Read Delta table
df = spark.read.format("delta").load("Tables/sales")
# Filter
df_filtered = df.filter(col("Region") == "West")
# Group and aggregate
df_summary = df.groupBy("Region").agg(
sum("SalesAmount").alias("TotalSales"),
count("OrderID").alias("OrderCount")
)
# Handle nulls
df_clean = df.na.fill({"Discount": 0.0}).na.drop(subset=["CustomerID"])
# Add calculated column
df_enriched = df.withColumn("SalesWithTax", col("SalesAmount") * 1.08)
# Write to Delta table
df_enriched.write.format("delta").mode("overwrite").save("Tables/sales_enriched")
PySpark Building Blocks
| Need | Code | Note |
|---|---|---|
| Conditional column | when(col("t") > 100, "High").when(col("t") > 50, "Medium").otherwise("Low") | Evaluated in order, like CASE WHEN |
| Parse timestamp string | to_timestamp(col("s"), "yyyy-MM-dd'T'HH:mm:ss'Z'") | Pattern must match the string, literals in quotes |
| Sequential row number | row_number().over(Window.orderBy("ts")) | monotonically_increasing_id() is unique, not gap-free |
| Running total | sum("Amount").over(Window.partitionBy("CustomerID").orderBy("TransactionDate").rowsBetween(Window.unboundedPreceding, Window.currentRow)) | Without orderBy the window is the whole partition |
| Fill null with group average | coalesce(col("price"), avg("price").over(Window.partitionBy("category"))) | Keeps every row, unlike groupBy |
| Nested struct fields | df.select("*", "address.city", "address.zip") | Dot notation; explode is for arrays |
| Query with SQL | df.createOrReplaceTempView("t"); spark.sql("SELECT ...") | Session-scoped; saveAsTable writes a real table |
| Partitioned write | .withColumn("year_month", date_format("d", "yyyy-MM")).write.partitionBy("year_month") | Low-cardinality key; date-range filters prune partitions |
| Delta vs. Parquet | .write.format("delta") | Delta adds the _delta_log transaction log (ACID, time travel) |
Enroll to unlock the 9 practice questions written for this section, so you can test what you just read while it is fresh.
Enroll to unlock the 8 flashcards for this section and review them on a spaced-repetition schedule.
Written byAlvin Varughese
Founder•20 professional certifications