> Markdown version of [/videos/676-scaling-from-0-to-20-million-users?t=1293](https://www.wearedevelopers.com/videos/676-scaling-from-0-to-20-million-users?t=1293). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# Scaling: from 0 to 20 million users

To survive 1.7 million concurrent connections, SofaScore abandoned the cloud. Discover how migrating from AWS to bare-metal servers slashed their infrastructure costs to just 0.8% of revenue.

- **Speakers:** [Josip Stuhli](https://www.wearedevelopers.com/@josip-stuhli)
- **Event:** World Congress 2023
- **Published:** September 21, 2023
- **Duration:** 28:12
- **URL:** https://www.wearedevelopers.com/videos/676-scaling-from-0-to-20-million-users

## Summary

SofaScore's backend evolution from a single crashing PHP server to handling 23 million monthly active users and 1.7 million concurrent real-time connections required a systematic approach to aggressive optimization. Initially mitigating server failure via proactive Memcached workers and static FTP uploads during massive traffic spikes, the engineering team quickly discovered the limits of basic application-layer caching. To prevent persistent "cache stampede" states and minimize backend database requests, they adopted Varnish in front of their PHP stack, enabling request coalescing and achieving a remarkable 99.3% cache hit rate that shielded raw infrastructure from unpredictable live-sports traffic.

Scaling iteratively often means recognizing when a specific tool no longer fits the workload. After heavily relying on AWS and MongoDB, crippling replication issues and the lack of foreign keys prompted a zero-downtime hot migration to PostgreSQL for better locking capability and analytical rigor. More radically, as cloud egress traffic costs eclipsed core compute expenses, the team fully repatriated their infrastructure out of AWS to a hybrid on-premise bare-metal ecosystem. By shifting high-performance tiers—including PostgreSQL and Varnish—to physical data centers, they slashed infrastructure operational costs to just 0.8% of top-line revenue. To safeguard against explosive demand, they established a hybrid failover cluster that bursts into cloud virtual machines only when physical capacity is fully exhausted.

Managing global traffic extremes—essentially large-scale self-administered DDoS events triggered by automated push notifications—required further refinement of the network and real-time event layers. By deploying geo-routed caching instances globally via Cloudflare, average global response latencies dropped from 500 milliseconds to an imperceptible 80 milliseconds. The team transitioned away from inefficient HTTP client polling by implementing NATS for high-speed pub/sub real-time updates directly to end devices. They also corralled a petabyte-scale analytics workload into ClickHouse to fuel high-velocity machine learning models. Ultimately, their scaling philosophy centers on keeping application instances stateless across an on-prem Kubernetes cluster, leveraging off-thread asynchronous queues, and aggressively profiling performance with APM tools to systematically target bottlenecks without succumbing to premature optimization.

**Keywords:** scaling PHP applications, cache stampede prevention, varnish request coalescing, cloud infrastructure repatriation, on-prem AWS hybrid failover, mongodb to postgresql hot-migration, stateless application deployment, high-concurrency push notifications, NATS pub/sub messaging, latency optimization geo routing, clickhouse analytics workloads, bare metal kubernetes management, asynchronous queuing patterns, cloudflare DDoS traffic mitigation, APM bottleneck profiling

## Chapters

1. **Overview of application scale and exponential growth** (00:04) — How an exponentially growing sports app handles billions of requests with a PHP backend.
1. **Surviving traffic spikes on a single shared server** (01:54) — A basic shared server setup crashes when overwhelmed by thousands of live users.
1. **Adding Memcached and experiencing cache stampede problems** (02:38) — Adding memory caching improved user capacity but led to backend crashes during cache expiration.
1. **Using proactive caching to compute data before expiration** (04:06) — Background workers recalculate and cache live data before expiration to prevent extreme database load.
1. **Scaling huge traffic spikes with static HTML files** (05:28) — Rendering pages as static files over FTP handles overwhelming server load before major sporting events.
1. **Migrating to AWS and utilizing Varnish for caching** (07:56) — Moving to AWS involves deploying stateless application images and using Varnish for request coalescing.
1. **Realizing the limitations of MongoDB for live statistics** (10:08) — Data type errors, poor analytics capabilities, and lack of locking plague the initial MongoDB deployment.
1. **Live migration to PostgreSQL for relational analytics features** (11:29) — Switching the primary database to PostgreSQL on the fly grants advanced SQL capabilities, foreign keys, and stability.
1. **Shifting caching layers in front of backend servers** (13:01) — Deploying load balancers and sharding techniques ensures cache efficiency and protects backend databases during upgrades.
1. **Moving from AWS to on-premise servers for cost reduction** (14:26) — Migrating out of the public cloud to private data center servers eliminates excessively high bandwidth pricing.
1. **Mitigating traffic spikes generated by real-time push notifications** (15:35) — Custom Cloudflare rules whitelist massive inbound traffic waves caused by millions of devices opening simultaneous push notifications.
1. **Monitoring application load and supplementing with virtual machines** (17:02) — Application performance monitoring tools reveal bottlenecks while dynamic virtual machines handle temporary capacity shortages.
1. **Utilizing global edge nodes and geo-routing strategies** (18:27) — Caching content at international edge nodes drastically decreases global load times for users outside the primary data center.
1. **Setting up bare-metal Kubernetes for multi-datacenter redundancies** (20:36) — On-premise Kubernetes deployments ease infrastructure updates and ensure failover safety after datacenters experience severe outages.
1. **Broadcasting lightning fast application updates using NATS messaging** (21:33) — Transitioning clients from long polling to NATS pub/sub networks delivers lightning fast dashboard updates.
1. **Powering machine learning models using ClickHouse database scale** (23:06) — The ClickHouse database efficiently stores and analyzes petabytes of analytical sports data locally at minimal hardware costs.
1. **Emphasizing statelessness, caching, and delayed processing architectures** (24:23) — Core scaling takeaways include keeping infrastructure stateless, personalizing cached data effectively, and actively queuing slow tasks.
1. **Handling metrics pipelines, geo-routing, and Kubernetes workload isolation** (26:22) — Monitoring live systems requires pipelines to InfluxDB, while stateful endpoints like databases remain outside the bare-metal Kubernetes cluster.

## Related Moments

- [Managing high-traffic infrastructure without chasing technology hype](https://www.wearedevelopers.com/videos/1817-how-to-avoid-tech-hype-traps-josip-stuhli) (from "How to Avoid Tech Hype Traps - Josip Stuhli")
- [Boosting scale and lowering latency natively with NoSQL](https://www.wearedevelopers.com/videos/806-leveraging-real-time-data-in-fsis) (from "Leveraging Real time data in FSIs")
- [Handling unexpected traffic spikes without proper load testing](https://www.wearedevelopers.com/videos/937-sleek-swift-and-sustainable-optimizations-every-web-developer-should-consider) (from "Sleek, Swift, and Sustainable: Optimizations every web developer should consider")
- [Hiring passionate software engineers to tackle unprecedented scaling challenges](https://www.wearedevelopers.com/videos/100128-the-golden-age-of-email-owning-the-inbox-in-the-age-of-ai) (from "The Golden Age of Email: Owning the Inbox in the Age of AI")
- [Overcoming traffic burst challenges in hyperscale web applications](https://www.wearedevelopers.com/videos/793-from-0-to-1-000-000-how-to-build-a-serverless-raffle-service-for-hyperscale) (from "From 0 to 1.000.000: How to build a serverless raffle service for hyperscale")
- [Overcoming scaling bottlenecks caused by commercial database limitations](https://www.wearedevelopers.com/videos/1389-building-systems-that-last) (from "Building Systems that Last")

## Related Articles

- [Making Data Warehouses Fast: A Developer’s Story](https://www.wearedevelopers.com/magazine/107-making-data-warehouses-fast-a-developer-s-story)
- [Dev Digest 120 - Apple and peers](https://www.wearedevelopers.com/magazine/455-dev-digest-120-apple-and-peers)
- [Dev Digest 139 - Soft and hard queries](https://www.wearedevelopers.com/magazine/487-dev-digest-139-soft-and-hard-queries)
- [Dev Digest 103 - Superb Owl Trafficking](https://www.wearedevelopers.com/magazine/388-dev-digest-103-superb-owl-trafficking)

## Related Jobs

- [Staff Software Engineer](https://www.wearedevelopers.com/jobs/ext/1425755-staff-software-engineer) at **GitHub**
- [Senior Software Engineer, Client Apps Platform](https://www.wearedevelopers.com/jobs/ext/1773893-senior-software-engineer-client-apps-platform) at **GitHub**
- [Principal Software Engineer, Database Infrastructure](https://www.wearedevelopers.com/jobs/ext/1465908-principal-software-engineer-database-infrastructure) at **GitHub**
- [Staff Software Engineer, Database Infrastructure](https://www.wearedevelopers.com/jobs/ext/1470125-staff-software-engineer-database-infrastructure) at **GitHub**
- [Cloud Foundations Team](https://www.wearedevelopers.com/jobs/ext/1483289-cloud-foundations-team) at **GitHub**
- [Senior Software Engineer, Data](https://www.wearedevelopers.com/jobs/48273-senior-software-engineer-data) at **Sportradar Media Services GmbH**