4.1.2. Processing with Glue, EMR, Lambda, and Athena

💡 First Principle: Each processing service occupies a different point on the volume-complexity-latency spectrum. Lambda processes individual events in milliseconds; Athena queries terabytes of S3 data in seconds; Glue transforms datasets from gigabytes to terabytes in minutes; and EMR handles petabyte-scale jobs with custom frameworks. The exam tests whether you can match the processing need to the right tool.

Athena for ad-hoc processing. Athena is serverless SQL for S3 — no infrastructure to manage, pay per TB scanned. Use Athena when analysts need to query data without loading it into a warehouse, or when pipelines need SQL-based transformations on S3 data. Key optimization: partitioned Parquet data with partition projection reduces scan costs dramatically.

Glue DataBrew is a no-code visual data preparation tool. 250+ built-in transforms for cleaning, normalizing, and formatting data. DataBrew is the answer when questions mention "data preparation for analysts" or "visual data cleaning without code."

SageMaker Unified Studio (v1.1) provides an integrated environment for data preparation, analytics, and ML. It consolidates DataBrew, notebooks, and SageMaker workflows into a unified interface with domain-based governance.

Processing service selection decision framework: If the data is already in Redshift or heading there, use Redshift SQL. If the task is a simple event-driven transform on small data, use Lambda. If it's a scheduled ETL on S3 data with no custom libraries, use Glue. If it requires custom Spark configurations or non-Spark frameworks, use EMR. If it's ad-hoc SQL on S3, use Athena. If it's visual data preparation for analysts, use DataBrew.

For SDK and API integration, the exam tests understanding of when pipeline steps should call AWS APIs programmatically (using boto3 in Python or AWS SDK in Java) versus using native service integrations (Step Functions' direct Glue integration, for example). Native integrations are preferred because they eliminate custom code.

⚠️ Exam Trap: Athena is serverless and requires no setup, but it charges per TB scanned. For frequently repeated queries on the same data, Redshift (with caching and compiled queries) is more cost-effective. The exam tests this distinction: infrequent ad-hoc queries → Athena; frequent dashboard queries → Redshift.

Reflection Question: A data science team runs exploratory Spark queries interactively during business hours and batch feature engineering jobs overnight. What processing services serve each pattern with minimal cost?