World Congress 2026 North America

Boring Failover: Predictable Region Recovery Across 5,000 Microservices

September 24, 2026 17:30 – 18:00 Β· 30 min Stage 4

What this session covers

At Intuit, 5,000+ services run at peak 1M+ TPS across TurboTax, QuickBooks, Credit Karma, and Mailchimp. Eighty percent are multi-region. Before EWOK β€” our Ecosystem Wide Orchestrator Kit β€” disaster recovery meant thousands of non-standard DR scripts, per-team runbooks, and 8,000+ engineers each solving the same problems differently. Game days were feared. MTTR was unpredictable. DR was treated as β€œthe database snapshot” β€” not the full stack.

This talk is the story of how we made regional failover boring β€” predictable, repeatable, automated, no heroics. I will walk through the architecture and the hard lessons:

β€’ A declarative YAML DSL for DR plans β€” stages, parallel blocks, per-stage agent versioning (armador/v1, database/v1, route53/v1), inline IAM role assumption.

β€’ AWS Step Functions as the DAG orchestrator β€” durable state for long-running promotions (Aurora global cluster failover, Redis replication switch), and a visual audit log that doubles as the incident timeline.

β€’ A Golang control plane on Kubernetes running goroutine-parallel mutations across thousands of namespaces.

β€’ A Python Agent Framework with an ABC contract β€” PreCheck, Failover, PostCheck β€” encapsulating IAM, logging, metrics. Product teams shipped a Redis agent in days without platform bottleneck.

β€’ Specialized agents per layer: armador (compute/capacity), Route53 (DNS cutover), Database (Aurora global failover), Redis (flush + replication).

β€’ The multi-workload problem everyone skips β€” cron jobs, async consumers, stateful tiers, caches. Parallel suspend, scale, resume, dial.

β€’ Progressive Dial β€” incremental traffic shift with error-gated automatic rollback.

β€’ Auto Failover via our Alert2Incident framework.

β€’ Same machinery for migrations β€” same-region failover as an upgrade feature.

You leave with: concrete patterns for declarative DR, a replicable agent contract for inner-sourcing reliability, and a checklist of the workload types your DR plan probably does not cover yet.

World Congress 2026 North America

Related talks

All sessions at this congress