Black Friday sale: an extra 40% off every course until Monday, November 30.New Year sale: an extra 30% off every course until Friday, January 1.Memorial Day sale: an extra 25% off every course until Monday, May 31.SysAdmin Day sale: an extra 25% off every course until Sunday, August 1.Back to School sale: an extra 25% off every course until Sunday, August 22.Labor Day sale: an extra 25% off every course until Monday, September 6.Black Friday sale: an extra 40% off every course until Monday, November 29.New Year sale: an extra 30% off every course until Saturday, January 1.Choose your certification →

Copyright (c) 2026 MindMesh Academy. All rights reserved. This content is proprietary and may not be reproduced or distributed without permission.

3.5.1.9. Controlled Remediation, Scaling Policies, and Automated Runbooks

💡 First Principle: Automation that fixes things can also break things at machine speed. Every automated action needs a blast-radius limit (rate, concurrency, error threshold), a way to stop itself, and a human checkpoint where the action is irreversible.

Safety constraints on automated remediation:
  • "SSM Automation" rate controls: target resources by tag or resource group and set concurrency (number or percentage running at once) and an error threshold (stop sending the automation to more targets after N or N% failures). Concurrency 1 with error threshold 0 is the most cautious rollout. The same controls exist on Run Command, State Manager associations, Maintenance Windows and patch policies. Add aws:approve steps for destructive actions. Automation can run across accounts and Regions from a management or delegated administrator account.
  • "AWS Config" automatic remediation retries a failed remediation a set number of times within a time window; stale compliance snapshots can re-trigger it, so remediation documents must be idempotent.
  • Deployment circuit breakers: the ECS deployment circuit breaker (rolling-update deployment controller) counts tasks that fail to reach RUNNING or fail ELB, Cloud Map or container health checks; at the threshold (default 50% of desired count, bounded 3 to 200) the deployment is marked FAILED and, with rollback on, ECS redeploys the last COMPLETED deployment. CodeDeploy alarm-based rollback does the same for EC2, Lambda and ECS blue/green.
  • Application-level patterns (circuit breaker, retries with exponential backoff and jitter, bulkheads, rate limiting) are covered in 3.1.1.2; use them so a remediation storm or retry storm cannot overwhelm a recovering dependency.
Scaling policies:
PolicyUse when
Target trackingDefault for most workloads: keep a metric (CPU, ALBRequestCountPerTarget, SQS backlog per instance) at a target; AWS creates the alarms
Step scalingYou need different adjustments for different breach sizes
Scheduled actionsLoad changes at known times (business hours, a planned sale)
Predictive scalingRecurring daily or weekly patterns and slow-starting instances: forecasts 48 hours ahead from up to 14 days of history (needs at least 24 hours), launches capacity before the hour, optionally earlier with a scheduling buffer
  • Predictive scaling starts in forecast only mode so you can check accuracy; in forecast-and-scale mode it only scales out, so pair it with target tracking for scale-in and unexpected spikes (the group uses the highest desired capacity any policy asks for). Max-capacity breach behavior can raise the group maximum when the forecast exceeds it.
  • ECS services, DynamoDB, Aurora replicas and Lambda provisioned concurrency scale through Application Auto Scaling with the same target-tracking, step and scheduled policy types.
Automated failure recovery:
  • Auto Scaling replaces instances that fail EC2 status checks, or ELB health checks when the group's health check type is ELB; ECS replaces tasks that fail container or load balancer health checks; Lambda asynchronous invocations retry twice by default (configurable 0 to 2, with a maximum event age) and send failures to an on-failure destination or DLQ.
Automated runbooks and incident response:
  • Pattern: alarm or finding, then "EventBridge" rule, then SSM Automation runbook (AWS-provided AWS- runbooks or your own; check opt-in tags inside the runbook, and use rate controls whenever it targets many resources), then notify via "SNS" and open an OpsItem in OpsCenter where human judgment is needed.
  • Patching: Quick Setup patch policies (created from the Organizations management account) define scan and install schedules, patch baselines, rate controls and log storage once for all accounts and Regions or chosen OUs, instead of per-account Maintenance Windows.
  • Scaling runbooks: for a known event, schedule scaling actions or run an Automation runbook that raises Service Quotas and capacity ahead of time.

⚠️ Common Pitfall: Pointing an auto-remediation runbook at "all non-compliant resources" with unlimited concurrency. If the runbook has a bug, it now breaks the whole fleet in minutes. Set concurrency and an error threshold, test in one OU, then widen.

See how it connects
Alvin Varughese
Written byAlvin Varughese
Founder•26 professional certifications earned