> Markdown version of [/videos/1143-scaling-databases](https://www.wearedevelopers.com/videos/1143-scaling-databases). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Scaling Databases

Is your database crashing under the weight of a rapidly growing application? Learn to avoid premature architectural sprawl by mastering indexing, replication, and caching before tackling complex sharding.

- **Speakers:** [Tobias Petry](https://www.wearedevelopers.com/@tobias-petry)
- **Event:** World Congress 2024
- **Published:** August 20, 2024
- **Duration:** 21:09
- **URL:** https://www.wearedevelopers.com/videos/1143-scaling-databases

## Summary

Growing applications inevitably hit performance limits, and the database—rather than the programming language or framework—is almost always the primary bottleneck. Using a hypothetical, rapidly expanding e-commerce startup, this narrative explores the escalating challenges of database availability and the architectural patterns used to alleviate application crashes under heavy load.

The progression of scaling strategies must begin with the simplest, most effective steps: optimizing slow queries through proper indexing and scaling up server hardware (vertical scaling). When a single node is no longer sufficient, organizations often explore multi-master replication to balance loads, though this can introduce severe write conflicts under high concurrency. Alternatively, primary-secondary read replication offers a proven way to offload read-heavy operations, provided the engineering team explicitly handles asynchronous replication lag without relying on the severe performance penalties of synchronous data transfer.

For extreme write loads and massive scale, database sharding provides near-infinite scalability by distributing distinct data contexts across multiple servers. However, sharding introduces immense engineering complexity, such as managing cross-shard joins and tenant routing, sometimes necessitating specialized routing middleware like Vitess or Citus. Ultimately, development teams must delay distributed database architectures for as long as possible. By prioritizing index optimization, caching, and simple hardware upgrades, organizations can avoid premature architectural sprawl and unmanageable operational overhead.

**Keywords:** database scaling, query optimization, database indexing, cache invalidation, vertical scaling, horizontal scaling, multi-master replication, write conflicts, read replication, asynchronous replication, synchronous replication, replication lag, database sharding, cross-shard joins, vitess, citus, architectural complexity

## Chapters

1. **Recognizing the database as the primary scaling bottleneck** (01:01) — Programming languages and frameworks rarely cause system failures compared to underlying database limitations.
1. **Optimizing database queries and implementing application caching strategies** (03:15) — Fixing slow queries through better indexes and applying caching should precede multi-server scaling.
1. **Scaling up by increasing single server hardware capacity** (04:34) — Upgrading server memory and CPU cores provides a simple performance boost before adopting complex topologies.
1. **Evaluating multi-master replication for database architectures** (05:31) — Synchronous dual-primary setups can introduce data conflict errors and reduce overall write performance during traffic spikes.
1. **Implementing read replication and managing asynchronous lag** (09:07) — Routing read operations to secondary servers increases capacity but requires application logic to handle asynchronous replication delays.
1. **Understanding the latency costs of synchronous read replication** (11:30) — Waiting for multiple distributed read replicas to acknowledge writes synchronously significantly degrades query execution speeds.
1. **Distributing data across multiple database servers with sharding** (13:57) — Splitting workloads across identical databases allows infinite scaling but complicates contextual data aggregation and cross-table joins.
1. **Managing sharding complexity with specialized database proxy solutions** (17:20) — Abstraction layers like Vitess and Citus manage internal routing complexities but restrict the availability of standard SQL features.
1. **Prioritizing architectural simplicity over infinite scalability solutions** (18:47) — Avoiding premature scaling optimizations and maintaining minimal database topologies prevents costly architecture rewrites and unexpected bugs.

## Related Moments

- [The challenges of scaling single-node database architectures](https://www.wearedevelopers.com/videos/100117-tidb-one-layer-at-a-time-how-distributed-sql-became-an-agentic-ai-backbone) (from "TiDB, One Layer at a Time: How Distributed SQL Became an Agentic AI Backbone")
- [Overcoming scaling bottlenecks caused by commercial database limitations](https://www.wearedevelopers.com/videos/1389-building-systems-that-last) (from "Building Systems that Last")
- [Scaling infrastructure for peak application traffic patterns](https://www.wearedevelopers.com/videos/100360-developing-the-backend-with-stefan-lingler-cto-at-shpock) (from "Developing the Backend with Stefan Lingler, CTO at Shpock")
- [Scaling MySQL databases for massive user growth](https://www.wearedevelopers.com/videos/350-branch-your-database-like-your-code-how-schema-changes-and-pull-requests-go-hand-in-hand) (from "Branch your database like your code:  How schema changes and pull requests go hand in hand")
- [Deployment architectures for scalable multi-tenant database environments](https://www.wearedevelopers.com/videos/1552-building-multi-tenant-asp-net-core-applications-best-practices-and-real-world-solutions) (from "Building Multi-Tenant ASP.NET Core Applications: Best Practices and Real-World Solutions")
- [Evaluating scalability trade-offs and complexity in distributed databases](https://www.wearedevelopers.com/videos/1167-maximising-cassandra-s-potential-tips-on-schema-queries-parallel-access-and-reactive-programming) (from "Maximising Cassandra's Potential: Tips on Schema, Queries, Parallel Access, and Reactive Programming")

## Related Articles

- [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story)
- [Why Event-Driven Architecture Isn’t About Speed (and When You Actually Need It)](https://www.wearedevelopers.com/magazine/745-why-event-driven-architecture-isn-t-about-speed-and-when-you-actually-need-it)
- [Graph and AI Trends 2026: Why Is AI Running but Not Yet Delivering?](https://www.wearedevelopers.com/magazine/680-graph-and-ai-trends-2026-why-is-ai-running-but-not-yet-delivering)
- [What does the history of data storage tell us about the future?](https://www.wearedevelopers.com/magazine/495-what-does-the-history-of-data-storage-tell-us-about-the-future)

## Related Jobs

- [Principal Software Engineer, Database Infrastructure](https://www.wearedevelopers.com/jobs/ext/1465908-principal-software-engineer-database-infrastructure) at **GitHub**
- [Staff Software Engineer, Database Infrastructure](https://www.wearedevelopers.com/jobs/ext/1470125-staff-software-engineer-database-infrastructure) at **GitHub**
- [Principal Engineer - AI Search & Vector Infrastructure](https://www.wearedevelopers.com/jobs/ext/353953-principal-engineer-ai-search-vector-infrastructure) at **Redis**
- [Principal Engineer - AI Search & Vector Infrastructure](https://www.wearedevelopers.com/jobs/ext/319507-principal-engineer-ai-search-vector-infrastructure) at **Redis**
- [Lead Software Engineer - Data Engineering](https://www.wearedevelopers.com/jobs/ext/2000968-lead-software-engineer-data-engineering) at **Dynatrace**
- [Principal Engineer - AI Search & Vector Infrastructure](https://www.wearedevelopers.com/jobs/ext/381484-principal-engineer-ai-search-vector-infrastructure) at **Redis**