George Asafev

Swapping Low Latency Data Storage Under High Load

Miro's Redis instance couldn't scale for 700 engineers. Learn how they moved 140 million accounts to DynamoDB while handling 100,000 requests per second.

Swapping Low Latency Data Storage Under High Load
#1about 4 minutes

Migrating 140M users from a Redis-based monolith

Miro's user registry team faced migrating 140 million users from Redis, which was unsuitable as a persistent store and was tied to a difficult-to-maintain monolith.

#2about 3 minutes

Evaluating PostgreSQL vs DynamoDB for the new database

The team compared PostgreSQL's flexibility with DynamoDB's managed scaling, ultimately choosing DynamoDB paired with DAX for its performance on read-heavy workloads.

#3about 2 minutes

Load testing the database solutions under production load

To validate their choice, they load-tested PostgreSQL using pgbench and DynamoDB with a k6 and EC2 setup to simulate production traffic.

#4about 2 minutes

Understanding the limitations of DynamoDB and DAX

Adopting DynamoDB with DAX required navigating limitations like primary key access patterns, separate item and query caches, and difficult vertical scaling.

#5about 4 minutes

The phased migration strategy from Redis to DynamoDB

The migration was executed in controlled phases using feature flags, including dual writes, a background data copy, shadow calls for validation, and a gradual rollout.

#6about 2 minutes

Unexpected DAX performance issues and learnings

After going live, the team discovered that rapid connection spikes caused DAX throttling and that adding a fourth node to their cluster unexpectedly increased replication lag.

#7about 4 minutes

Results of the migration and Q&A

The successful migration reduced endpoint latency up to four times and enabled the team to move forward with their microservice architecture, followed by a Q&A on alternative solutions.

Related jobs
Jobs that call for the skills explored in this talk.

Matching moments

Achieving massive scale on AWS for Prime Day
02:31 MIN

Achieving massive scale on AWS for Prime Day

Reliable scalability: How Amazon.com scales on AWS

Answering questions on data volume, challenges, and databases
07:21 MIN

Answering questions on data volume, challenges, and databases

Remote Driving on Plant Grounds with State-of-the-Art Cloud Technologies

Q&A on migration strategy and stakeholder management
10:39 MIN

Q&A on migration strategy and stakeholder management

AWS Migration within 3 months

Building a dynamic LRU cache with DynamoDB and CDC
05:40 MIN

Building a dynamic LRU cache with DynamoDB and CDC

Advanced Caching Patterns used by 2000 microservices

Merging data engineering and DevOps for scalability
01:55 MIN

Merging data engineering and DevOps for scalability

Software Engineering Social Connection: Yubo’s lean approach to scaling an 80M-user infrastructure

Q&A on implementation details and technology choices
03:43 MIN

Q&A on implementation details and technology choices

Challenges for omnichannel applications at ALDI: Data distribution and offline capabilities

Supporting hyperscale workloads with a single database
03:24 MIN

Supporting hyperscale workloads with a single database

Fault Tolerance and Consistency at Scale: Harnessing the Power of Distributed SQL Databases

Key learnings and results from the MLOps transformation
03:01 MIN

Key learnings and results from the MLOps transformation

The Road to MLOps: How Verivox Transitioned to AWS

Featured Partners

Related Videos

See all videos
Single Server, Global Reach: Running a Worldwide Marketplace on Bare Metal in a Cloud-Dominated World50:06

Single Server, Global Reach: Running a Worldwide Marketplace on Bare Metal in a Cloud-Dominated World

Jens Happe

In-Memory Computing - The Big Picture30:36

In-Memory Computing - The Big Picture

Markus Kett

Building Real-Time AI/ML Agents with Distributed Data using Apache Cassandra and Astra DB28:20

Building Real-Time AI/ML Agents with Distributed Data using Apache Cassandra and Astra DB

Dieter Flick

From 0 to 1.000.000: How to build a serverless raffle service for hyperscale30:03

From 0 to 1.000.000: How to build a serverless raffle service for hyperscale

Marco Plaul & Martin Sakowski

Scaling: from 0 to 20 million users28:12

Scaling: from 0 to 20 million users

Josip Stuhli

Database Magic behind 40 Million operations/s24:22

Database Magic behind 40 Million operations/s

Jürgen Pilz

Reliable scalability: How Amazon.com scales on AWS22:24

Reliable scalability: How Amazon.com scales on AWS

Florian Mair

The Road to MLOps: How Verivox Transitioned to AWS25:48

The Road to MLOps: How Verivox Transitioned to AWS

Elisabeth Günther

Related Articles

View all articles
DC
Daniel Cranney
Why Attend a Developer Event?
Modern software engineering moves too fast for documentation alone. Attending a world-class event is about shifting from tactical execution to strategic leadership. Skill Diversification: Break out of your specific tech stack to see how the industry...
Why Attend a Developer Event?

From learning to earning

Jobs that call for the skills explored in this talk.

Database Migration Architect

Database Migration Architect

Amazon.com, Inc
Cambridge, United Kingdom

Senior
MySQL
PostgreSQL
Amazon DynamoDB
Microsoft SQL Server
Amazon Web Services (AWS)