4.1.1. Orchestration with Step Functions and MWAA

💡 First Principle: The operational difference between Step Functions and MWAA comes down to where you want complexity: Step Functions manages state and retries natively but requires careful state machine design; MWAA manages complex dependency graphs naturally but requires Airflow expertise and an always-on environment.

For data operations, Step Functions excels at multi-service coordination: start a Glue crawler, wait for completion, run a Glue ETL job, check results, conditionally branch to success or failure handling, and send notifications via SNS. The built-in error handling (Retry and Catch blocks) makes pipelines resilient without custom code.

MWAA excels at dependency-heavy workflows: DAGs naturally express "Task C depends on both Task A and Task B," sensor operators wait for external conditions ("wait until this S3 file exists"), and the Airflow UI provides task-level visibility, log access, and manual re-triggers. For troubleshooting, the Airflow UI is far richer than Step Functions' execution history.

Operational patterns for the exam: triggering on schedule (EventBridge → Step Functions), triggering on data arrival (S3 event → EventBridge → Step Functions), and combining orchestrators (Airflow DAG that triggers individual Step Functions workflows for each processing stage). A common production pattern is using Airflow as the "outer loop" scheduler with Step Functions handling the "inner loop" of each pipeline's execution logic — this separates scheduling concerns from execution concerns.

For troubleshooting managed workflows, MWAA provides Airflow's built-in logging to CloudWatch Logs (scheduler, worker, webserver, and DAG processing logs), while Step Functions provides a visual execution history showing which state succeeded or failed and why. When debugging, start with the execution history to identify which step failed, then check CloudWatch Logs for the why.

⚠️ Exam Trap: Step Functions Standard workflows charge per state transition — a Map state iterating over 10,000 items creates 10,000+ transitions. For high-volume iteration, use Distributed Map (batches items into parallel child executions) or move the iteration inside a Lambda function. The exam may present a cost optimization scenario targeting this.

Reflection Question: An existing Airflow DAG orchestrates 15 data processing tasks with complex dependencies. The team wants to reduce the $400/month MWAA cost. Under what conditions would migrating to Step Functions be appropriate, and when should they keep MWAA?