3.5.1.9. Controlled Remediation, Scaling Policies, and Automated Runbooks
💡 First Principle: Automation that fixes things can also break things at machine speed. Every automated action needs a blast-radius limit (rate, concurrency, error threshold), a way to stop itself, and a human checkpoint where the action is irreversible.
Safety constraints on automated remediation:
"SSM Automation"rate controls: target resources by tag or resource group and set concurrency (number or percentage running at once) and an error threshold (stop sending the automation to more targets after N or N% failures). Concurrency 1 with error threshold 0 is the most cautious rollout. The same controls exist on Run Command, State Manager associations, Maintenance Windows and patch policies. Addaws:approvesteps for destructive actions. Automation can run across accounts and Regions from a management or delegated administrator account."AWS Config"automatic remediation retries a failed remediation a set number of times within a time window; stale compliance snapshots can re-trigger it, so remediation documents must be idempotent.- Deployment circuit breakers: the ECS deployment circuit breaker (rolling-update deployment controller) counts tasks that fail to reach
RUNNINGor fail ELB, Cloud Map or container health checks; at the threshold (default 50% of desired count, bounded 3 to 200) the deployment is markedFAILEDand, with rollback on, ECS redeploys the lastCOMPLETEDdeployment. CodeDeploy alarm-based rollback does the same for EC2, Lambda and ECS blue/green. - Application-level patterns (circuit breaker, retries with exponential backoff and jitter, bulkheads, rate limiting) are covered in 3.1.1.2; use them so a remediation storm or retry storm cannot overwhelm a recovering dependency.
Scaling policies:
| Policy | Use when |
|---|---|
| Target tracking | Default for most workloads: keep a metric (CPU, ALBRequestCountPerTarget, SQS backlog per instance) at a target; AWS creates the alarms |
| Step scaling | You need different adjustments for different breach sizes |
| Scheduled actions | Load changes at known times (business hours, a planned sale) |
| Predictive scaling | Recurring daily or weekly patterns and slow-starting instances: forecasts 48 hours ahead from up to 14 days of history (needs at least 24 hours), launches capacity before the hour, optionally earlier with a scheduling buffer |
- Predictive scaling starts in forecast only mode so you can check accuracy; in forecast-and-scale mode it only scales out, so pair it with target tracking for scale-in and unexpected spikes (the group uses the highest desired capacity any policy asks for). Max-capacity breach behavior can raise the group maximum when the forecast exceeds it.
- ECS services, DynamoDB, Aurora replicas and Lambda provisioned concurrency scale through Application Auto Scaling with the same target-tracking, step and scheduled policy types.
Automated failure recovery:
- Auto Scaling replaces instances that fail EC2 status checks, or ELB health checks when the group's health check type is ELB; ECS replaces tasks that fail container or load balancer health checks; Lambda asynchronous invocations retry twice by default (configurable 0 to 2, with a maximum event age) and send failures to an on-failure destination or DLQ.
Automated runbooks and incident response:
- Pattern: alarm or finding, then
"EventBridge"rule, then SSM Automation runbook (AWS-providedAWS-runbooks or your own; check opt-in tags inside the runbook, and use rate controls whenever it targets many resources), then notify via"SNS"and open an OpsItem in OpsCenter where human judgment is needed. - Patching: Quick Setup patch policies (created from the Organizations management account) define scan and install schedules, patch baselines, rate controls and log storage once for all accounts and Regions or chosen OUs, instead of per-account Maintenance Windows.
- Scaling runbooks: for a known event, schedule scaling actions or run an Automation runbook that raises Service Quotas and capacity ahead of time.
⚠️ Common Pitfall: Pointing an auto-remediation runbook at "all non-compliant resources" with unlimited concurrency. If the runbook has a bug, it now breaks the whole fleet in minutes. Set concurrency and an error threshold, test in one OU, then widen.