Black Friday sale: an extra 40% off every course until Monday, November 30.New Year sale: an extra 30% off every course until Friday, January 1.Memorial Day sale: an extra 25% off every course until Monday, May 31.SysAdmin Day sale: an extra 25% off every course until Sunday, August 1.Back to School sale: an extra 25% off every course until Sunday, August 22.Labor Day sale: an extra 25% off every course until Monday, September 6.Black Friday sale: an extra 40% off every course until Monday, November 29.New Year sale: an extra 30% off every course until Saturday, January 1.Choose your certification →

Copyright (c) 2026 MindMesh Academy. All rights reserved. This content is proprietary and may not be reproduced or distributed without permission.

3.1.1.6. Failover Testing and Recovery-Time Measurement (FIS, Route 53 Health Checks, ARC)

3.1.1.6. Failover Testing and Recovery-Time Measurement (FIS, Route 53 Health Checks, ARC)

💡 First Principle: Recovery time is a measurement, not a design property: it runs from the moment the fault starts to the moment users are served normally again, and it can only be known by causing the fault and timing the result.

Scenario: A retailer runs active/passive across two Regions behind "Route 53" failover records. The business asks for evidence each quarter that a Region failover completes within 15 minutes, and that losing one "Availability Zone" causes no customer-visible errors.

What the clock includes: detection (health checks and alarms) + decision (automatic, or a human approval) + traffic shift (DNS, routing control or load balancer) + recovery of the standby (scale-out, database promotion, cache warm-up) + return to the steady-state metric. Start the clock at fault injection, and stop it when the steady-state metric (error rate, latency, successful synthetic transactions) is back within its threshold, not when the last command returns.

"Route 53" health check timing
  • A health check fails after failure threshold consecutive failed checks (default 3) at its request interval (30 seconds standard, 10 seconds fast). Detection therefore takes roughly interval x threshold: about 90 seconds with defaults, about 30 seconds with fast checks.
  • After the record flips, resolvers keep the old answer until the record's TTL expires, so keep TTLs on failover records short (60 seconds or less), and remember that some clients cache longer than the TTL.
  • To test without breaking production, make the primary's health check fail on purpose: an "FIS" action that stops the target instances or blocks their subnet traffic in a test copy of the stack, or the health check's invert health check status setting. Measure when the secondary starts receiving traffic, then restore the primary and measure failback.
  • Fail over with the data plane. "Route 53" health checks and ARC routing control state changes are data-plane operations. Editing DNS records or creating resources during an incident depends on control planes that may be impaired; design so that failover needs no record edits (static stability).
"FIS" for failover tests
  • AZ Availability: Power Interruption and Cross-Region: Connectivity scenarios reproduce realistic zonal and Regional failures, and aws:rds:failover-db-cluster exercises a database failover on its own.
  • Put the steady-state alarms in as stop conditions, attach a "CloudWatch" dashboard to the experiment report, and the report becomes the timed evidence auditors ask for.
  • Run the Resilience Hub SOP experiments to measure each SOP's actual recovery time against the policy.
"Amazon Application Recovery Controller (ARC)"
CapabilityScopeWhat it doesTesting angle
Zonal shiftOne AZ, one resourceYou shift traffic for an "ALB", "NLB", "EC2 Auto Scaling group" or "EKS" cluster away from an impaired AZ; temporary, with an expiry of up to three days that you can extendPractice an AZ evacuation on demand
Zonal autoshiftOne AZAWS starts the shift for you when its telemetry shows an AZ impairment; the resource must be pre-scaled to run without that AZPractice runs are mandatory: about weekly, ARC shifts traffic away from one AZ for 30 minutes. At least one outcome alarm is required (ALARM ends the run as FAILED); optional blocking alarms and blocked windows prevent runs at bad times
Routing controlRegionOn/off switches tied to "Route 53" routing-control health checks, changed through a cluster whose data plane spans five Regions; safety rules prevent turning off every replica at onceFlip a test cell, time the cutover
Region switchRegion, many accountsOrchestrated plan (see 3.1.1.5), triggered manually or by alarms, with dashboards of each stepExecute the plan in a test, measure each step against the RTO
Readiness checkRegionMonitors quotas, capacity and routing configuration of replicasCatches a standby that could not take the load; not used in the failover path

⚠️ Exam Trap: Reading the runbook, confirming replication is running, or checking that the health check exists does not measure recovery time. The answer that proves an RTO injects the fault (FIS, a practice run, a test execution of the plan) and times recovery against a steady-state metric.

See how it connects
Alvin Varughese
Written byAlvin Varughese
Founder•26 professional certifications earned