2.2.4. AWS Transformation Tools: Glue, DataBrew, EMR, and Data Wrangler
💡 First Principle: AWS provides four main data transformation tools, each optimized for a different combination of scale, user skill level, and integration depth. The exam tests your ability to match the tool to the scenario—not your knowledge of any single tool's API.
Imagine four different workshops: a professional industrial kitchen (Glue), a home kitchen with pre-set recipes (DataBrew), a fully custom restaurant kitchen (EMR), and a recipe testing station that feeds directly into the dining room (Data Wrangler). Each produces food, but you'd choose based on volume, customization needs, and where the food is going next.
| Tool | Scale | User | Integration | Best For |
|---|---|---|---|---|
| AWS Glue | Petabytes | Data engineers | ETL catalog, S3, databases | Large-scale ETL, data lake pipelines, scheduled transforms |
| AWS Glue DataBrew | Terabytes | Analysts, no code | Visual profiling + recipe transforms | Data profiling, visual transformations, data quality rules |
| Amazon EMR | Petabytes | Engineers (Spark/Hive) | Full Spark/Hadoop ecosystem | Custom Spark jobs, complex transformations, existing Spark code |
| SageMaker Data Wrangler | Gigabytes | Data scientists | Direct to SageMaker training | Feature engineering, visual exploration, quick prototyping for ML |
Key distinctions the exam tests:
Glue vs. DataBrew: Glue is for engineers writing ETL jobs (PySpark or Scala). DataBrew is the no-code/visual interface for profiling data and applying transformation "recipes." If the question mentions "visual interface," "data profiling," or "no-code," the answer is DataBrew. If it mentions "ETL pipeline," "PySpark," or "data catalog," the answer is Glue.
Data Wrangler vs. Glue: Data Wrangler is SageMaker-native—it integrates directly with SageMaker training, Feature Store, and Pipelines. It's for data scientists doing feature engineering interactively. Glue is for data engineers building production ETL pipelines. If the scenario describes an ML workflow where the transformed data feeds directly into model training, Data Wrangler is likely the answer. If the scenario describes building a reusable data pipeline serving multiple consumers, Glue is the answer.
EMR vs. Glue: Both can run Spark jobs at scale. EMR gives you full control over the cluster (Spark version, libraries, instance types) and is the choice when teams have existing Spark code or need specific Hadoop ecosystem tools. Glue is serverless and managed—you don't manage clusters. If the question mentions "serverless" or "no cluster management," it's Glue. If it mentions "custom Spark configuration" or "existing Hadoop," it's EMR.
Streaming transformations: For real-time data, AWS Lambda processes individual records (low throughput, simple transforms), while Amazon Managed Service for Apache Flink handles high-throughput streaming transformations with windowed aggregations. If the question describes computing rolling averages or time-windowed features from a stream, Flink is the answer.
⚠️ Exam Trap: Data Wrangler and Glue DataBrew sound similar—both offer visual interfaces for data transformation. The key differentiator is where the output goes. Data Wrangler outputs flow into SageMaker (training, Feature Store, Pipelines). DataBrew outputs flow into S3 or other data stores. If the question mentions "SageMaker pipeline" or "Feature Store," pick Data Wrangler.
Reflection Question: A company has a data engineer maintaining a nightly ETL pipeline in PySpark and a data scientist who wants to interactively explore features for a new model. Which AWS tools should each person use, and how would you connect their workflows?
