4.3.3. Troubleshooting Pipeline Failures
💡 First Principle: Pipeline failures follow predictable patterns. Most failures fall into five categories: resource limits (memory, timeout, throttling), data issues (schema changes, nulls, corrupt files), permission errors (IAM role missing permissions), network issues (VPC endpoints, security groups), and logic errors (wrong transformation, incorrect joins). Knowing the category narrows the investigation immediately.
Glue job troubleshooting: Check CloudWatch Logs for Spark executor errors. Common failures: out-of-memory (first cut the data each executor holds — see the trap below — and move to a larger worker type such as G.2X only once the volume is confirmed legitimate), timeout (job exceeded maximum execution time), and data issues (schema mismatch, corrupt files). Job bookmarks can cause issues if the bookmark state becomes corrupted — reset by deleting the bookmark.
EMR troubleshooting: Check the EMR step logs in S3 (s3://aws-logs-{account}-{region}/elasticmapreduce/). Each step has controller, syslog, stdout and stderr logs; the failed step's stderr (with the Spark driver log) holds the stack trace, while the console only shows that the step failed. Common issues: cluster sizing (under-provisioned instances), YARN resource allocation, and Spark executor configuration. EMR logs are stored hierarchically by cluster ID and step ID.
Lambda troubleshooting: Check CloudWatch Logs for the function. Common issues: timeout (15-min limit exceeded), memory exhaustion, cold starts causing latency spikes, and permission errors (execution role missing required policies). A timeout on one large file is a design problem, not a setting — 15 minutes is the ceiling, so stream the object in chunks (ranged S3 reads) or split it upstream and process the pieces in parallel. A cold start is the one-time initialization of a new execution environment (loading code, starting the runtime, running init code) after the function has been idle or scales out; provisioned concurrency keeps environments warm. X-Ray tracing provides distributed tracing for Lambda-based pipelines: it follows each request across service boundaries, and its service map shows which downstream call (for example DynamoDB or S3) adds the latency — something per-service metrics viewed one at a time cannot show.
Performance tuning patterns: right-size compute (Glue DPUs, EMR instance types), optimize data formats (CSV → Parquet), improve partitioning, use caching (Redshift result cache, Athena query results), and eliminate unnecessary data movement (use Redshift Spectrum instead of COPY for infrequent queries).
⚠️ Exam Trap: When a Glue job fails with "out of memory," the answer is usually NOT to increase the number of workers. First, check if the job is processing more data than expected (missing job bookmark), reading unnecessarily wide datasets (lack of column projection), or creating excessive shuffle (repartition or coalesce the data). Architectural fixes before capacity increases.
Reflection Question: A Glue ETL job that ran successfully for months suddenly fails with "Exit Code: 1." CloudWatch Logs show a Java heap space error. The source data volume hasn't changed. What could have changed, and how would you investigate?
