3.1.3.4. Testing Failover of Multi-AZ and Multi-Region Workloads (RDS, Aurora, Route 53, CloudFront)

3.1.3.4. Testing Failover of Multi-AZ and Multi-Region Workloads (RDS, ECS)

DR testing that only exists in documentation provides zero confidence. Regular, automated failover tests are the only way to validate recovery capability.

Testing approaches:

RDS failover testing:

# Force RDS Multi-AZ failover (causes 1-2 min downtime)
aws rds reboot-db-instance \
  --db-instance-identifier prod-db \
  --force-failover
# Monitor: RDS event "Multi-AZ instance failover started"

AWS Fault Injection Simulator (FIS) provides managed chaos experiments:

• Terminate EC2 instances in a specific AZ
• Inject CPU/memory stress on instances
• Disrupt network connectivity between services
• Throttle API calls to simulate service degradation

Game Day pattern:

1. Schedule a maintenance window
2. Run the DR playbook against the production environment
3. Measure actual RTO (time from failover trigger to full recovery)
4. Compare actual RTO vs. target RTO
5. Document gaps and fix before next test

Metrics to capture during DR tests:

• Time to detect failure
• Time to execute failover
• Data loss during failover (actual RPO)
• Application error rate during transition
• Time to full recovery (actual RTO)

Exam Trap: Testing RDS failover in production causes a brief outage (1-2 minutes). Schedule it during a maintenance window and ensure the application handles the connection drop gracefully. If the exam asks "how to test DR without affecting production," the answer is to run tests in a separate environment that mirrors production architecture.