3.1.1.5. DR Assessment and Recovery Procedures (AWS Resilience Hub, Systems Manager Runbooks)
3.1.1.5. DR Assessment and Recovery Procedures (AWS Resilience Hub, Systems Manager Runbooks)
💡 First Principle: A recovery objective is only a promise until two things exist: an assessment that measures the deployed architecture against it, and a written, automated and tested procedure that actually performs the recovery.
Scenario: A payments platform has a board-approved target of "RTO" 1 hour and "RPO" 15 minutes for a Region outage. Nobody can say whether the current design meets it, the recovery steps live in a wiki, and every release changes the infrastructure.
"AWS Resilience Hub": the assessment loop
- Describe the application by importing its resources from
"CloudFormation"stacks, Terraform state files,"AWS Resource Groups","Amazon EKS"clusters, or an existing myApplications application. Related resources that fail together (a primary database and its replica) form one Application Component. - Define a resiliency policy: an
"RTO"and"RPO"target for each of four disruption types: application (bad deployment, configuration error, corrupted data), cloud infrastructure (a failed component), Availability Zone and Region. - Assess. Resilience Hub analyzes the configuration against Well-Architected practices and estimates the workload RTO and RPO each component can achieve, reporting Policy met or Policy breached. A cross-Region target with no cross-Region replica or backup copy is breached before anything is tested. The report gives resiliency recommendations (configuration changes such as Multi-AZ, cross-Region copies, global tables) and operational recommendations:
"CloudWatch"alarms that detect the outage, standard operating procedures (SOPs) that recover from it, and"FIS"experiments that test them. - Implement the operational recommendations as code: Resilience Hub generates
"CloudFormation"templates (stored in"S3") for the alarms, SOPs and experiments, which you add to the application's repository and pipeline. - Track. Re-run assessments (on a schedule or from the CI/CD pipeline for every build). Drift detection reports application policy drift (a component that met the policy in the last assessment no longer does) and resource drift (resources added or removed from the input sources). The resiliency score shows how far the application follows the recommendations for policy, alarms, SOPs and tests.
Recovery procedures as "Systems Manager" runbooks
- An SOP is a prescriptive set of recovery steps. Resilience Hub bases its SOPs on
"Systems Manager"Automation runbooks (SSM documents): each SOP recommendation comes with the automation document, an IAM role to run it, an"FIS"experiment that exercises it, and an SSM parameter recording which resources it applies to. - A runbook turns the wiki page into steps that run the same way every time: call AWS APIs (promote a replica, restore a snapshot, scale an
"Auto Scaling group"), run commands on instances, invoke a"Lambda"function, branch on results, and pause at anaws:approvestep when a human must authorize a failover. Runbooks can run across multiple accounts and Regions with rate and error controls, and can start from an"EventBridge"rule or a"CloudWatch"alarm. - For a multi-Region recovery that spans many resources and accounts,
"Amazon Application Recovery Controller (ARC)"Region switch is the orchestrated procedure: a plan made of workflows and execution blocks (scale"EC2 Auto Scaling","ECS"tasks or"EKS"pods in the Region being activated, fail over or switch over an"Aurora"global database, redirect traffic with"Route 53"health checks or ARC routing controls, run a custom"Lambda"action, wait for manual approval, or call a child plan). It runs from a data plane in each Region, so executing it does not depend on the impaired Region, and it can be started by hand or by a"CloudWatch"alarm.
| Need | Tool |
|---|---|
| "Does the design meet our RTO/RPO per disruption type, and has a change broken it?" | "AWS Resilience Hub" assessment, drift detection |
| Repeatable recovery steps for one component or one account | "Systems Manager" Automation runbook (the Resilience Hub SOP) |
| Coordinated multi-Region failover and failback across services and accounts | ARC Region switch plan |
| Proof the procedure works and how long it takes | "FIS" experiment that runs the SOP (see 3.1.1.6) |
⚠️ Exam Trap: "Policy met" in Resilience Hub is an estimate from configuration, not a measured recovery. The SOP still has to be tested and timed, and a policy breach is fixed by changing the architecture (adding the cross-Region copy or replica), not by editing the policy until it passes.