3.1.1.6. Failover Testing and Recovery-Time Measurement (FIS, Route 53 Health Checks, ARC)
3.1.1.6. Failover Testing and Recovery-Time Measurement (FIS, Route 53 Health Checks, ARC)
💡 First Principle: Recovery time is a measurement, not a design property: it runs from the moment the fault starts to the moment users are served normally again, and it can only be known by causing the fault and timing the result.
Scenario: A retailer runs active/passive across two Regions behind "Route 53" failover records. The business asks for evidence each quarter that a Region failover completes within 15 minutes, and that losing one "Availability Zone" causes no customer-visible errors.
What the clock includes: detection (health checks and alarms) + decision (automatic, or a human approval) + traffic shift (DNS, routing control or load balancer) + recovery of the standby (scale-out, database promotion, cache warm-up) + return to the steady-state metric. Start the clock at fault injection, and stop it when the steady-state metric (error rate, latency, successful synthetic transactions) is back within its threshold, not when the last command returns.
"Route 53" health check timing
- A health check fails after failure threshold consecutive failed checks (default 3) at its request interval (30 seconds standard, 10 seconds fast). Detection therefore takes roughly interval x threshold: about 90 seconds with defaults, about 30 seconds with fast checks.
- After the record flips, resolvers keep the old answer until the record's TTL expires, so keep TTLs on failover records short (60 seconds or less), and remember that some clients cache longer than the TTL.
- To test without breaking production, make the primary's health check fail on purpose: an
"FIS"action that stops the target instances or blocks their subnet traffic in a test copy of the stack, or the health check's invert health check status setting. Measure when the secondary starts receiving traffic, then restore the primary and measure failback. - Fail over with the data plane.
"Route 53"health checks and ARC routing control state changes are data-plane operations. Editing DNS records or creating resources during an incident depends on control planes that may be impaired; design so that failover needs no record edits (static stability).
"FIS" for failover tests
- AZ Availability: Power Interruption and Cross-Region: Connectivity scenarios reproduce realistic zonal and Regional failures, and
aws:rds:failover-db-clusterexercises a database failover on its own. - Put the steady-state alarms in as stop conditions, attach a
"CloudWatch"dashboard to the experiment report, and the report becomes the timed evidence auditors ask for. - Run the Resilience Hub SOP experiments to measure each SOP's actual recovery time against the policy.
"Amazon Application Recovery Controller (ARC)"
| Capability | Scope | What it does | Testing angle |
|---|---|---|---|
| Zonal shift | One AZ, one resource | You shift traffic for an "ALB", "NLB", "EC2 Auto Scaling group" or "EKS" cluster away from an impaired AZ; temporary, with an expiry of up to three days that you can extend | Practice an AZ evacuation on demand |
| Zonal autoshift | One AZ | AWS starts the shift for you when its telemetry shows an AZ impairment; the resource must be pre-scaled to run without that AZ | Practice runs are mandatory: about weekly, ARC shifts traffic away from one AZ for 30 minutes. At least one outcome alarm is required (ALARM ends the run as FAILED); optional blocking alarms and blocked windows prevent runs at bad times |
| Routing control | Region | On/off switches tied to "Route 53" routing-control health checks, changed through a cluster whose data plane spans five Regions; safety rules prevent turning off every replica at once | Flip a test cell, time the cutover |
| Region switch | Region, many accounts | Orchestrated plan (see 3.1.1.5), triggered manually or by alarms, with dashboards of each step | Execute the plan in a test, measure each step against the RTO |
| Readiness check | Region | Monitors quotas, capacity and routing configuration of replicas | Catches a standby that could not take the load; not used in the failover path |
⚠️ Exam Trap: Reading the runbook, confirming replication is running, or checking that the health check exists does not measure recovery time. The answer that proves an RTO injects the fault (FIS, a practice run, a test execution of the plan) and times recovery against a steady-state metric.