3.1.1.7. Multi-Cluster Container Strategies for High Availability and DR (Amazon EKS, Amazon ECS)
3.1.1.7. Multi-Cluster Container Strategies for High Availability and DR (Amazon EKS, Amazon ECS)
💡 First Principle: A container cluster is itself a failure domain: its control plane is Regional, and one bad upgrade, add-on or configuration change can take down every workload on it. Availability above that level comes from running more than one cluster and keeping them interchangeable.
Scenario: A logistics company runs 60 microservices on one "Amazon EKS" cluster in one Region. A failed cluster upgrade caused a two-hour outage, and the business now requires that a Region outage be survivable with an "RTO" of 15 minutes.
Layers of container resilience
- Within one cluster (AZ failures): spread replicas across three
"AZs"(pod topology spread constraints,"ECS"AZ spread placement), use PodDisruptionBudgets, and enable ARC zonal shift / zonal autoshift for the"EKS"cluster or the load balancers in front of"ECS"services. Stateful pods backed by"EBS"are tied to the volume's AZ; use"EFS"or a managed data store when pods must reschedule anywhere. - Several clusters in one Region (blast radius): run two clusters and move traffic between them with weighted
"ALB"target groups or weighted"Route 53"records. This also enables blue/green cluster upgrades: build the new-version cluster, shift traffic gradually, keep the old one for rollback. - Clusters in two Regions (Regional DR): an
"EKS"or"ECS"cluster cannot span Regions, so DR means a second cluster in the second Region, either warm standby (running at reduced scale) or active/active.
What keeps two clusters interchangeable
| Concern | Design |
|---|---|
| Configuration and manifests | GitOps (Argo CD or Flux syncing every cluster from the same repository) or one pipeline ("CodePipeline", "CDK"/"CloudFormation") deploying the same task definitions and services to both Regions. Manual kubectl changes are drift. |
| Images | "Amazon ECR" cross-Region replication rules, so the standby pulls from its own Region instead of the failed one |
| Secrets and keys | "Secrets Manager" replica secrets, multi-Region "KMS" keys |
| State | Outside the cluster: "Aurora Global Database", "DynamoDB" global tables, "S3" replication |
| Service discovery | Regional: "ECS Service Connect" and "Cloud Map" namespaces exist per Region, so each Region has its own |
| Traffic | "Route 53" failover or latency records with health checks, or "Global Accelerator" endpoint groups per Region with traffic dials |
| Orchestration | ARC Region switch plans scale "EKS" pods and "ECS" tasks in the Region being activated, fail over the database, then move traffic |
| Backup | "AWS Backup" for "EKS" captures cluster state plus persistent volumes on "EBS", "EFS" and "S3" as one composite recovery point, can be copied to another Region or account, and restores a whole cluster, namespaces or single volumes, including into a new cluster. Velero is the self-managed alternative. Container images are not included (that is "ECR"'s job). |
Choosing the pattern: backup and restore of cluster state suits hours of "RTO"; a warm-standby cluster with replicated images and GitOps-synced configuration suits minutes; active/active clusters behind "Global Accelerator" or latency routing suit near-zero "RTO" but require the data layer to accept writes in both Regions (or a single-writer design that routes writes to one Region).
⚠️ Exam Trap: A second cluster is not a recovery plan if it depends on the failed Region: images pulled from the primary Region's "ECR", secrets in one Region only, or manifests applied by hand. The correct answer replicates every dependency into the standby Region and keeps both clusters in sync from one source of truth.