2.4.1. AWS Glue ETL: DynamicFrames, Spark, and Job Bookmarks
💡 First Principle: Glue ETL is the serverless Spark engine optimized for data lake transformations. It runs Apache Spark under the hood but adds features that Spark alone doesn't provide: DynamicFrames for schema flexibility, job bookmarks for incremental processing, and native integration with the Glue Data Catalog. When the exam says "serverless ETL" or "least operational overhead for transformation," Glue is almost always the answer.
DynamicFrames vs DataFrames. Standard Spark DataFrames require a consistent schema — every record must have the same columns and types. Glue's DynamicFrames handle schema inconsistency gracefully: records can have different columns, and the resolveChoice method lets you decide how to handle type conflicts (cast, project, or make_struct). This is critical for data lake workloads where source schemas drift. For nested JSON, the Relationalize transform flattens structs and pivots arrays into separate child tables linked by generated join keys — ready for Redshift.
Reading and writing in a script. glueContext.create_dynamic_frame.from_catalog(database=..., table_name=...) reads a cataloged table; from_options(connection_type="s3", ...) reads a path directly. To write partitioned output, pass partitionKeys inside connection_options of write_dynamic_frame.from_options(...); Glue then writes Hive-style year=/month=/ folders. For thousands of tiny input files, the S3 options "groupFiles": "inPartition" and groupSize read them in larger groups; on output, coalesce(n) cuts the file count by merging partitions without a full shuffle, while repartition(n) forces one.
Job bookmarks. Glue can track which data has already been processed. On subsequent runs, only new or modified data is processed — enabling incremental ETL. Bookmarks work with S3 (tracking new files by timestamp) and JDBC sources (tracking new rows by a monotonically increasing column); Glue streaming jobs reading Kinesis or Kafka use checkpoints instead. This eliminates reprocessing and dramatically reduces cost for pipelines that run frequently. Turn them on with the job parameter --job-bookmark-option job-bookmark-enable; job-bookmark-disable ignores the bookmark and processes everything, and job-bookmark-pause reads from the last saved bookmark without ever advancing it.
Glue job types for the exam:
| Job Type | Engine | Use Case | DPU |
|---|---|---|---|
| Spark | Apache Spark | Heavy ETL, joins, aggregations, format conversion | 2–100 DPUs |
| Spark Streaming | Spark Structured Streaming | Near-real-time from Kinesis/Kafka | 2–100 DPUs |
| Python Shell | Python | Lightweight scripts, API calls, simple transforms | 0.0625–1 DPU |
Glue Studio provides a visual ETL designer — drag-and-drop sources, transforms, and targets. It generates PySpark code that you can customize. For the exam, Glue Studio is the signal when questions mention "visual pipeline design" or "no-code ETL."
Cost optimization. Glue charges per DPU-second (1-minute minimum), so cost tracks total DPU-hours = DPUs × runtime: 10 DPUs for 1 hour and 20 DPUs for 30 minutes both use 10 DPU-hours and cost the same. Key optimizations: use Flex execution (pre-empt-able capacity, ~34% cheaper) for non-urgent jobs with slack before their deadline (not SLA-bound ones, since Flex runs may start later or take longer), enable auto-scaling to avoid over-provisioning (Flex lowers the price per DPU; auto-scaling lowers how many DPUs you hold by adding and removing workers as each stage needs them), use column projection and predicate pushdown to read only necessary data, and convert CSV to Parquet early in the pipeline to reduce downstream scan costs.
⚠️ Exam Trap: Glue job bookmarks only work if the data source supports them. For S3, bookmarks compare object last-modified times, so a file overwritten under the same name is picked up again on the next run. For JDBC, they track a bookmark key column (by default a sequentially increasing primary key) and read only rows with new key values — rows updated in place are missed unless you choose a key that changes on update. If a question mentions "reprocessing data that was already processed," check whether job bookmarks were properly configured.
Reflection Question: A Glue job reads CSV files from S3, converts them to Parquet, and writes to an output bucket. The job runs hourly but reprocesses all historical files every time. What Glue feature would fix this?
