2.6.2. Amazon MWAA (Apache Airflow) and Glue Workflows
💡 First Principle: MWAA is for teams that need maximum orchestration flexibility — complex dependency graphs, custom operators, integration with non-AWS systems, and task-level visibility. Glue Workflows is for teams that only need to orchestrate Glue jobs and crawlers. The decision is a spectrum of complexity: don't bring an orchestra conductor when a metronome will do.
Amazon MWAA runs Apache Airflow as a managed service. You write Directed Acyclic Graphs (DAGs) in Python that define tasks, dependencies, schedules, and retry behavior. MWAA manages the Airflow web server, scheduler, workers, and metadata database — you focus on DAG development. Cost follows the environment class (mw1.micro through mw1.2xlarge) plus any workers above the minimum, so right-size the class and cap the maximum worker count before considering a re-platform.
Airflow's strength is its operator ecosystem — pre-built integrations for AWS services (GlueJobOperator, RedshiftSQLOperator, and S3KeySensor — a sensor that pauses the DAG until a given S3 key exists), databases (PostgresOperator, MySqlOperator), and external systems (SlackWebhookOperator, HttpOperator). Complex pipelines that coordinate AWS services, check external APIs, and send notifications to Slack are natural Airflow territory.
Glue Workflows provide native orchestration for Glue resources. You define triggers (schedule or on-demand), connect them to crawlers and ETL jobs, and Glue manages execution ordering. Workflows support conditional triggers ("start Job B only if Job A succeeds") and parallel execution. The visual editor in the Glue console shows workflow status in real time.
| Aspect | Step Functions | MWAA (Airflow) | Glue Workflows |
|---|---|---|---|
| Pricing | Per state transition | Per environment-hour (always-on) | Free (pay for jobs/crawlers) |
| Startup cost | None (serverless) | Always-on hourly charge (~$350/month for mw1.small) | None |
| Scope | Any AWS service | Any system (AWS, external, custom) | Glue jobs and crawlers only |
| Language | JSON (States Language) | Python (DAGs) | Visual / API |
| Complexity | Medium | High (but most flexible) | Low |
| Best for | Multi-service AWS workflows | Complex, cross-system orchestration | Simple Glue-only pipelines |
⚠️ Exam Trap: MWAA bills for every hour the environment exists (about $350/month for mw1.small; mw1.micro costs less but also runs 24/7) because the Airflow web server and scheduler never stop. If a question describes a cost-sensitive workload with a simple Glue pipeline, MWAA is the wrong answer — Glue Workflows or Step Functions are cheaper. The exam punishes overly complex solutions.
Reflection Question: A small startup has a single Glue ETL job that runs nightly, followed by a Glue crawler. They want to add monitoring but minimize costs. Which orchestration service is the best fit?
