Boring Failover: Predictable Region Recovery Across 5,000 Microservices

At Intuit, 5,000+ services run at peak 1M+ TPS across TurboTax, QuickBooks, Credit Karma, and Mailchimp. Eighty percent are multi-region. Before EWOK — our Ecosystem Wide Orchestrator Kit — disaster recovery meant thousands of non-standard DR scripts, per-team runbooks, and 8,000+ engineers each solving the same problems differently. Game days were feared. MTTR was unpredictable. DR was treated as “the database snapshot” — not the full stack.

This talk is the story of how we made regional failover boring — predictable, repeatable, automated, no heroics. I will walk through the architecture and the hard lessons:

• A declarative YAML DSL for DR plans — stages, parallel blocks, per-stage agent versioning (armador/v1, database/v1, route53/v1), inline IAM role assumption.

• AWS Step Functions as the DAG orchestrator — durable state for long-running promotions (Aurora global cluster failover, Redis replication switch), and a visual audit log that doubles as the incident timeline.

• A Golang control plane on Kubernetes running goroutine-parallel mutations across thousands of namespaces.

• A Python Agent Framework with an ABC contract — PreCheck, Failover, PostCheck — encapsulating IAM, logging, metrics. Product teams shipped a Redis agent in days without platform bottleneck.

• Specialized agents per layer: armador (compute/capacity), Route53 (DNS cutover), Database (Aurora global failover), Redis (flush + replication).

• The multi-workload problem everyone skips — cron jobs, async consumers, stateful tiers, caches. Parallel suspend, scale, resume, dial.

• Progressive Dial — incremental traffic shift with error-gated automatic rollback.

• Auto Failover via our Alert2Incident framework.

• Same machinery for migrations — same-region failover as an upgrade feature.

You leave with: concrete patterns for declarative DR, a replicable agent contract for inner-sourcing reliability, and a checklist of the workload types your DR plan probably does not cover yet.