George Asafev

Swapping Low Latency Data Storage Under High Load

Miro's Redis instance couldn't scale for 700 engineers. Learn how they moved 140 million accounts to DynamoDB while handling 100,000 requests per second.

Swapping Low Latency Data Storage Under High Load
#1about 4 minutes

Migrating 140M users from a Redis-based monolith

Miro's user registry team faced migrating 140 million users from Redis, which was unsuitable as a persistent store and was tied to a difficult-to-maintain monolith.

#2about 3 minutes

Evaluating PostgreSQL vs DynamoDB for the new database

The team compared PostgreSQL's flexibility with DynamoDB's managed scaling, ultimately choosing DynamoDB paired with DAX for its performance on read-heavy workloads.

#3about 2 minutes

Load testing the database solutions under production load

To validate their choice, they load-tested PostgreSQL using pgbench and DynamoDB with a k6 and EC2 setup to simulate production traffic.

#4about 2 minutes

Understanding the limitations of DynamoDB and DAX

Adopting DynamoDB with DAX required navigating limitations like primary key access patterns, separate item and query caches, and difficult vertical scaling.

#5about 4 minutes

The phased migration strategy from Redis to DynamoDB

The migration was executed in controlled phases using feature flags, including dual writes, a background data copy, shadow calls for validation, and a gradual rollout.

#6about 2 minutes

Unexpected DAX performance issues and learnings

After going live, the team discovered that rapid connection spikes caused DAX throttling and that adding a fourth node to their cluster unexpectedly increased replication lag.

#7about 4 minutes

Results of the migration and Q&A

The successful migration reduced endpoint latency up to four times and enabled the team to move forward with their microservice architecture, followed by a Q&A on alternative solutions.

Related jobs
Jobs that call for the skills explored in this talk.

Featured Partners

Related Videos

Single Server, Global Reach: Running a Worldwide Marketplace on Bare Metal in a Cloud-Dominated World50:06

Single Server, Global Reach: Running a Worldwide Marketplace on Bare Metal in a Cloud-Dominated World

Jens Happe

In-Memory Computing - The Big Picture30:36

In-Memory Computing - The Big Picture

Markus Kett

Building Real-Time AI/ML Agents with Distributed Data using Apache Cassandra and Astra DB28:20

Building Real-Time AI/ML Agents with Distributed Data using Apache Cassandra and Astra DB

Dieter Flick

Scaling: from 0 to 20 million users28:12

Scaling: from 0 to 20 million users

Josip Stuhli

Database Magic behind 40 Million operations/s24:22

Database Magic behind 40 Million operations/s

Jürgen Pilz

From 0 to 1.000.000: How to build a serverless raffle service for hyperscale30:03

From 0 to 1.000.000: How to build a serverless raffle service for hyperscale

Marco Plaul & Martin Sakowski

Reliable scalability: How Amazon.com scales on AWS22:24

Reliable scalability: How Amazon.com scales on AWS

Florian Mair

How building an industry DBMS differs from building a research one30:34

How building an industry DBMS differs from building a research one

Markus Dreseler

From learning to earning

Jobs that call for the skills explored in this talk.