3.1.3.4. Testing Failover of Multi-AZ and Multi-Region Workloads (RDS, Aurora, Route 53, CloudFront)

3.1.3.4. Testing Failover of Multi-AZ and Multi-Region Workloads (RDS, ECS)

DR testing that only exists in documentation provides zero confidence. Regular, automated failover tests are the only way to validate recovery capability.

Testing approaches:

Work up that ladder: a team that has never tested DR starts with a tabletop walk-through of the runbook, which finds missing steps and unclear ownership at no risk to production, then moves to component tests and live failovers.

RDS failover testing:

# Force RDS Multi-AZ failover (causes 1-2 min downtime)
aws rds reboot-db-instance \
  --db-instance-identifier prod-db \
  --force-failover
# Monitor: RDS event "Multi-AZ instance failover started"

AWS Fault Injection Service (FIS) provides managed chaos experiments:

• Terminate EC2 instances in a specific AZ
• Inject CPU/memory stress on instances
• Disrupt network connectivity between services
• Throttle API calls to simulate service degradation
• Cut a dependency without code changes: aws:network:disrupt-connectivity with scope dynamodb or s3 blocks a subnet's traffic to that service's Regional endpoint, and the ECS/EKS network latency and packet-loss actions can target the DYNAMODB/S3 endpoints

Game Day pattern:

1. Schedule a maintenance window
2. Run the DR playbook against the production environment
3. Measure actual RTO (time from failover trigger to full recovery)
4. Compare actual RTO vs. target RTO
5. Document gaps and fix before next test

Metrics to capture during DR tests:

• Time to detect failure
• Time to execute failover
• Data loss during failover (actual RPO)
• Application error rate during transition
• Time to full recovery (actual RTO)

Exam Trap: Testing RDS failover in production causes a brief outage (1-2 minutes). Schedule it during a maintenance window and ensure the application handles the connection drop gracefully. If the exam asks "how to test DR without affecting production," the answer is to run tests in a separate environment that mirrors production architecture, after a tabletop walk-through has shaken out a runbook that has never been exercised.