> Markdown version of [/events/world-congress-2026-north-america/sessions/1698-boring-failover](https://www.wearedevelopers.com/events/world-congress-2026-north-america/sessions/1698-boring-failover). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Boring Failover: Predictable Region Recovery Across 5,000 Microservices


- **Event:** World Congress 2026 North America

## Description

At Intuit, 5,000+ services run at peak 1M+ TPS across TurboTax, QuickBooks, Credit Karma, and Mailchimp. Eighty percent are multi-region. Before EWOK — our Ecosystem Wide Orchestrator Kit — disaster recovery meant thousands of non-standard DR scripts, per-team runbooks, and 8,000+ engineers each solving the same problems differently. Game days were feared. MTTR was unpredictable. DR was treated as "the database snapshot" — not the full stack.

This talk is the story of how we made regional failover boring — predictable, repeatable, automated, no heroics. I will walk through the architecture and the hard lessons:

• A declarative YAML DSL for DR plans — stages, parallel blocks, per-stage agent versioning (armador/v1, database/v1, route53/v1), inline IAM role assumption.

• AWS Step Functions as the DAG orchestrator — durable state for long-running promotions (Aurora global cluster failover, Redis replication switch), and a visual audit log that doubles as the incident timeline.

• A Golang control plane on Kubernetes running goroutine-parallel mutations across thousands of namespaces.

• A Python Agent Framework with an ABC contract — PreCheck, Failover, PostCheck — encapsulating IAM, logging, metrics. Product teams shipped a Redis agent in days without platform bottleneck.

• Specialized agents per layer: armador (compute/capacity), Route53 (DNS cutover), Database (Aurora global failover), Redis (flush + replication).

• The multi-workload problem everyone skips — cron jobs, async consumers, stateful tiers, caches. Parallel suspend, scale, resume, dial.

• Progressive Dial — incremental traffic shift with error-gated automatic rollback.

• Auto Failover via our Alert2Incident framework.

• Same machinery for migrations — same-region failover as an upgrade feature.

You leave with: concrete patterns for declarative DR, a replicable agent contract for inner-sourcing reliability, and a checklist of the workload types your DR plan probably does not cover yet.

## Speakers

### [Garvit Kataria](https://www.wearedevelopers.com/@garvit-kataria)

Senior Software Engineer
### [Sahil Sabharwal](https://www.wearedevelopers.com/@sahil-sabharwal)

Senior Staff Software Engineer at Intuit


## Related talks at this congress

- [Your Evals Passed. Your Agent Just Emptied a Database.](https://www.wearedevelopers.com/events/world-congress-2026-north-america/sessions/1684-your-evals-passed) — Tejas Pravinbhai Patel
- [Agents Can't Iterate Against Tests That Lie](https://www.wearedevelopers.com/events/world-congress-2026-north-america/sessions/1700-agents-can-t-iterate) — Rocky Warren
- [It passed auth, then production caught fire](https://www.wearedevelopers.com/events/world-congress-2026-north-america/sessions/1703-it-passed-auth-then) — Alex Olivier
- [Run your agents in Kubernetes: Build once, deploy anywhere. But really?](https://www.wearedevelopers.com/events/world-congress-2026-north-america/sessions/1676-run-your-agents-in) — Michal Salanci

## Watch remotely

Can’t make it to San José? Watch this session live with Pro. You also get:

- All full videos, bookmarks, and playlists
- World Congress livestreams

[See pricing](https://www.wearedevelopers.com/pricing)

## Links

- [Get tickets](https://www.wearedevelopers.com/world-congress-north-america/tickets)