Boring Failover: Predictable Region Recovery Across 5,000 Microservices

At Intuit, 5,000+ services run at peak 1M+ TPS across TurboTax, QuickBooks, Credit Karma, and Mailchimp. Eighty percent are multi-region. Before EWOK — our Ecosystem Wide Orchestrator Kit — disaster recovery meant thousands of non-standard DR scripts, per-team runbooks, and 8,000+ engineers each solving the same problems differently. Game days were feared. MTTR was unpredictable. DR was treated as "the database snapshot" — not the full stack. This talk is the story of how we made regional failover boring — predictable, repeatable, automated, no heroics. I will walk through the architecture and the hard lessons: • A declarative YAML DSL for DR plans — stages, parallel blocks, per-stage agent versioning (armador/v1, database/v1, route53/v1), inline IAM role assumption. • AWS Step Functions as the DAG orchestrator — durable state for long-running promotions (Aurora global cluster failover, Redis replication switch), and a visual audit log that doubles as the incident timeline. • A Golang control plane on Kubernetes running goroutine-parallel mutations across thousands of namespaces. • A Python Agent Framework with an ABC contract — PreCheck, Failover, PostCheck — encapsulating IAM, logging, metrics. Product teams shipped a Redis agent in days without platform bottleneck. • Specialized agents per layer: armador (compute/capacity), Route53 (DNS cutover), Database (Aurora global failover), Redis (flush + replication). • The multi-workload problem everyone skips — cron jobs, async consumers, stateful tiers, caches. Parallel suspend, scale, resume, dial. • Progressive Dial — incremental traffic shift with error-gated automatic rollback. • Auto Failover via our Alert2Incident framework. • Same machinery for migrations — same-region failover as an upgrade feature. You leave with: concrete patterns for declarative DR, a replicable agent contract for inner-sourcing reliability, and a checklist of the workload types your DR plan probably does not cover yet.