> Markdown version of [/videos/746-swapping-low-latency-data-storage-under-high-load?t=141](https://www.wearedevelopers.com/videos/746-swapping-low-latency-data-storage-under-high-load?t=141). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Swapping Low Latency Data Storage Under High Load

How do you migrate 140 million accounts at 100,000 requests per second with zero downtime? Uncover the DynamoDB strategy Miro used to slash core latency by 4x.

- **Speakers:** George Asafev
- **Event:** World Congress 2023
- **Published:** October 6, 2023
- **Duration:** 22:32
- **URL:** https://www.wearedevelopers.com/videos/746-swapping-low-latency-data-storage-under-high-load

## Summary

Miro faced the challenge of migrating over 140 million user accounts from a legacy Redis datastore to a dedicated, persistent database under a daily load of over 100,000 requests per second. Originally integrated when the platform relied on an online game engine, Redis acted as a rapid but ultimately transient data solution. As the engineering organization grew, the sheer scale of the monolith hindered concurrent deployments, blurred responsibility borders, and complicated horizontal scaling. Transitioning to a microservices architecture required safely extracting this massive dataset from the monolithic application's toolbox without introducing any downtime or data inconsistencies.

Evaluating potential replacements, the team initially considered PostgreSQL but determined it would struggle with sharding demands and read speeds. They ultimately selected AWS DynamoDB, favored for its fully managed scaling and out-of-the-box key-value access patterns. However, DynamoDB alone proved slower and more expensive for their read-heavy service. By integrating DynamoDB Accelerator (DAX), a transparent in-memory caching layer, they successfully fit their entire rarely-changed dataset directly into cache. This architectural shift accelerated data retrieval, plummeting database access times from milliseconds to tenths of microseconds—an up to tenfold performance boost that drastically minimized direct DynamoDB read costs.

Executing a zero-downtime migration required a sequential deployment approach heavily reliant on feature flags designed for five-second rollbacks. The transition began with dual-writes utilizing database inserts rather than upserts to specifically prevent race conditions alongside background migration jobs. Following the initial migration, the engineering team utilized shadow reads—asynchronously querying DAX to catch exceptions, perform state consistency checks against Redis, and validate metrics before shifting production traffic. The team also uncovered obscure DAX operational quirks, encountering connection-creation CPU spikes entirely masked by CloudWatch's one-second metric resolution, as well as an AWS bug causing severe replication lag when expanding past three DAX nodes. By navigating these pitfalls, the migration successfully decoupled the user registry from the monolith and slashed core endpoint latency by 4x.

**Keywords:** redis to dynamodb migration, zero-downtime database migration, DAX cluster scaling, dynamodb accelerator, dual-write migration patterns, shadow read validation, monolith to microservices extraction, high load data storage, low latency caching strategies, AWS cloudwatch resolution limits, eventual consistency debugging, database sharding challenges, feature flag rollbacks, read-heavy database optimization, microsecond query performance

## Chapters

1. **Migrating millions of user accounts at scale without downtime** (00:00) — An overview of the challenge to orchestrate a seamless migration of over 140 million user profiles out of Redis in a high-load environment.
1. **Evaluating data storage limitations and monolithic architectural challenges** (02:21) — Why adopting microservices resolves the difficult deployment processes and unclear responsibility borders that complicate a monolithic setup.
1. **Comparing PostgreSQL and DynamoDB for read-heavy storage** (04:35) — Evaluating the trade-offs in speed and cost when deciding between a relational database and a fully-managed NoSQL solution for specific query patterns.
1. **Accelerating DynamoDB performance with DAX memory caching** (06:54) — Utilizing an in-memory cache accelerator layer to push read query times down from milliseconds to microseconds.
1. **Load testing production traffic for database selection** (07:54) — Simulating high-load scenarios using synthetic traffic tools to confidently guide the core database infrastructure decision.
1. **Navigating the limitations of DynamoDB and DAX caching** (10:01) — Working around access pattern differences, cache invalidation delays, and rigid vertical scaling constraints when provisioning clusters.
1. **Executing a zero-downtime database migration with dual writes** (12:00) — Utilizing sequential queries and protected background data inserts to migrate systems securely while avoiding race conditions.
1. **Validating data consistency through asynchronous shadow reads** (14:18) — Protecting data integrity during switchovers by conducting offline evaluations matched against live interactions with the original store.
1. **Managing gradual rollouts and internal testing for UI impact** (15:33) — Minimizing external disruption by internally testing features to prevent eventual consistency logic from degrading application interfaces.
1. **Troubleshooting connection throttling and hidden CPU spikes in DAX** (16:28) — Diagnosing undocumented processing surges caused by rapid request influxes that initially bypassed standard system observability thresholds.
1. **Resolving DAX replication lag and measuring final performance improvements** (17:58) — Understanding cache performance degradation caused by AWS synchronization delays when manually expanding node counts.
1. **Q&A on database vendor lock-in and alternative architectural choices** (19:15) — Addressing long-term engineering choices, potential vendor constraints, and evaluating in-memory cluster alternatives like Elasticache.

## Related Moments

- [Solving architectural challenges during rapid public cloud migrations](https://www.wearedevelopers.com/videos/238-the-journey-from-developer-to-devops-what-i-ve-learnt-along-the-way) (from "The journey from developer to devops - what i've learnt along the way")
- [Audience questions on database performance, deployments, and data migrations](https://www.wearedevelopers.com/videos/254-tomorrow-s-cloud-data-platforms-fully-managed-database-as-a-service-dbaas) (from "Tomorrow's cloud data platforms - fully managed database-as-a-service (DBaaS)")
- [Executing massive cloud network migrations while maintaining live systems](https://www.wearedevelopers.com/videos/100128-the-golden-age-of-email-owning-the-inbox-in-the-age-of-ai) (from "The Golden Age of Email: Owning the Inbox in the Age of AI")
- [The database outage that inspired the creation of Dynamo](https://www.wearedevelopers.com/videos/100214-20-years-of-solving-unsolvable-problems) (from "20 Years of Solving Unsolvable Problems")
- [Final performance results and key architectural takeaways](https://www.wearedevelopers.com/videos/733-accelerating-authentication-architecture-taking-passwordless-to-the-next-level) (from "Accelerating Authentication Architecture: Taking Passwordless to the Next Level")
- [Debugging serverless timeouts and replacing databases for lookups](https://www.wearedevelopers.com/videos/1413-ip-authentication-a-tale-of-performance-pitfalls-and-challenges-in-prod) (from "IP Authentication: A Tale of Performance Pitfalls and Challenges in Prod")

## Related Articles

- [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story)
- [Why Event-Driven Architecture Isn’t About Speed (and When You Actually Need It)](https://www.wearedevelopers.com/magazine/745-why-event-driven-architecture-isn-t-about-speed-and-when-you-actually-need-it)
- [Dev Digest 139 - Soft and hard queries](https://www.wearedevelopers.com/magazine/487-dev-digest-139-soft-and-hard-queries)
- [Dev Digest 120 - Apple and peers](https://www.wearedevelopers.com/magazine/455-dev-digest-120-apple-and-peers)

## Related Jobs

- [Lead Software Engineer - Data Engineering](https://www.wearedevelopers.com/jobs/ext/2000968-lead-software-engineer-data-engineering) at **Dynatrace**
- [Senior Data Engineer](https://www.wearedevelopers.com/jobs/ext/1589390-senior-data-engineer) at **Douglas GmbH**
- [Principal Software Engineer, Database Infrastructure](https://www.wearedevelopers.com/jobs/ext/1465908-principal-software-engineer-database-infrastructure) at **GitHub**
- [Senior Software Engineer, Data](https://www.wearedevelopers.com/jobs/48273-senior-software-engineer-data) at **Sportradar Media Services GmbH**
- [Data Scientist](https://www.wearedevelopers.com/jobs/ext/1351648-data-scientist) at **Almedia**
- [Principal Engineer - AI Search & Vector Infrastructure](https://www.wearedevelopers.com/jobs/ext/381484-principal-engineer-ai-search-vector-infrastructure) at **Redis**