> Markdown version of [/videos/418-how-gatsby-cloud-s-real-time-streaming-architecture-drives-5-second-builds](https://www.wearedevelopers.com/videos/418-how-gatsby-cloud-s-real-time-streaming-architecture-drives-5-second-builds). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# How Gatsby Cloud's real-time streaming architecture drives <5 second builds

By modeling HTML generation after database indexes, Gatsby eliminates slow static rebuilds and complex SSR caches. See how its reactive streaming architecture drives sub-five-second publish times.

- **Speakers:** Kyle Mathews
- **Event:** World Congress 2022
- **Published:** June 15, 2022
- **Duration:** 42:10
- **URL:** https://www.wearedevelopers.com/videos/418-how-gatsby-cloud-s-real-time-streaming-architecture-drives-5-second-builds

## Summary

Web development has long struggled with efficiently generating and maintaining HTML for scalable content sites. HTML is essentially "derived data"—the result of passing inputs like markdown or headless CMS entries through a template function. Historically, developers have relied on batch processing via static site generation (SSG) or on-demand application caching through server-side rendering (SSR). SSG guarantees strong consistency but creates overwhelming node queues and painfully slow rebuilds for large platforms. Conversely, SSR handles immediate updates on demand but introduces complex TTL (time-to-live) cache invalidation puzzles and risks bankrupting compute resources during traffic spikes.

Instead of treating page generation as a pure batch or on-demand task, Gatsby models HTML generation after database indexes and reactive programming. By treating the framework's data layer as a first-class citizen, Gatsby ingests CMS metadata as generic nodes that pages explicitly declare dependencies on via GraphQL queries. When a content piece changes, Gatsby utilizes a reactive data pipeline to instantly trace the updated node to its dependent pages for a targeted rebuild. This framework-level cache validation completely bypasses manual cache purging, cutting median publish times for content updates from 60 seconds down to under 5 seconds.

To eliminate infrastructure bottlenecks on massive platforms, Gatsby evolved to aggressively scale its reactive engine both vertically and horizontally. Overcoming Node.js single-thread limitations, Gatsby isolates HTML generation and GraphQL polling into parallel worker pools backed by an embedded LMDB database. Moving forward, Gatsby horizontally distributes page-building jobs across separated compute nodes. This transforms traditional caching from a localized build hurdle into a cloud-native processing layer, providing near-infinite HTML computation to enterprise content platforms.

**Keywords:** HTML page generation, derived data architecture, SSG vs SSR trade-offs, application cache invalidation, reactive programming patterns, database index mapping, GraphQL data dependencies, framework-level cache validation, Node.js parallel processing, embedded LMDB architecture, horizontal compute scaling, headless CMS deployments, static site generation, server-side rendering, real-time streaming builds

## Chapters

1. **Building and updating HTML content as derived data** (00:05) — Exploring ways to transform inputs into updated site documents without constant recomputation.
1. **Tradeoffs of batch processing and application caching** (05:37) — Evaluating historical caching methods and their impacts on computational load and data freshness.
1. **Comparing static site generation and server-side rendering** (11:26) — Breaking down cold start speeds and deployment times for pre-rendered and on-demand web architectures.
1. **Using database indexes and reactive processing patterns** (18:35) — How explicit data pipelines use targeted transformations to react immediately to single input updates.
1. **Implementing reactive data flows for incremental builds** (24:09) — Using dependency tracking to immediately rebuild only the components affected by a content change.
1. **Managing content scale and cache invalidation bottlenecks** (31:33) — Overcoming the severe performance limits of traditional static and server-rendered frameworks at massive scale.
1. **Scaling build systems vertically and horizontally** (36:15) — Distributing component rendering and query executions across multiple cores to accelerate initial site constructions.
1. **Concluding thoughts on infinite background computation architectures** (40:35) — Envisioning a future of infinitely scalable computing for resolving traditional site generation dependencies.

## Related Moments

- [Leveraging Gatsby for React-based static site generation](https://www.wearedevelopers.com/videos/421-next-nest-nuxt-nust) (from "Next, Nest, Nuxt… Nust?")
- [Strategies for scaling frontend application rendering speeds](https://www.wearedevelopers.com/videos/685-angular-unleashed-mastering-modern-web-development-with-angular) (from "Angular Unleashed: Mastering Modern Web Development with Angular")
- [Optimizing system scalability and real-time push capabilities](https://www.wearedevelopers.com/videos/494-the-rise-of-reactive-microservices) (from "The Rise of Reactive Microservices")
- [Modern application stacks and real-time data requirements](https://www.wearedevelopers.com/videos/806-leveraging-real-time-data-in-fsis) (from "Leveraging Real time data in FSIs")
- [Comparing client, server, and static frontend rendering strategies](https://www.wearedevelopers.com/videos/236-microfrontends-at-scale) (from "Microfrontends at Scale")
- [Leveraging server-side streaming for performance and interactivity](https://www.wearedevelopers.com/videos/1302-wearedevelopers-live-dishonest-charts-britcss-debugging-with-ai) (from "WeAreDevelopers LIVE - Dishonest Charts, BritCSS, Debugging with AI")

## Related Articles

- [Web Development Best Practices: What The Data Tells us](https://www.wearedevelopers.com/magazine/84-web-development-best-practices-what-the-data-tells-us)
- [Dev Digest 120 - Apple and peers](https://www.wearedevelopers.com/magazine/455-dev-digest-120-apple-and-peers)
- [Dev Digest 139 - Soft and hard queries](https://www.wearedevelopers.com/magazine/487-dev-digest-139-soft-and-hard-queries)
- [Dev Digest 132 - Binging WADFlix?](https://www.wearedevelopers.com/magazine/473-dev-digest-132-binging-wadflix)

## Related Jobs

- [Staff Frontend Engineer](https://www.wearedevelopers.com/jobs/48313-staff-frontend-engineer) at **SMG Swiss Marketplace Group**
- [Mid/Senior Full-Stack Engineer (Web-first)](https://www.wearedevelopers.com/jobs/ext/1210833-mid-senior-full-stack-engineer-web-first) at **SMG Swiss Marketplace Group**
- [Remote Senior Full-Stack Engineer](https://www.wearedevelopers.com/jobs/ext/646086-remote-senior-full-stack-engineer) at **Edge Impulse**
- [Staff Frontend Engineer (Expert+/Lead equivalent) - Hybrid working model, 100%, Ho Chi Minh City](https://www.wearedevelopers.com/jobs/48314-staff-frontend-engineer-expert-lead-equivalent-hybrid-working-model-100-ho-chi-minh-city) at **SMG Swiss Marketplace Group**
- [Remote Senior Full-Stack Engineer](https://www.wearedevelopers.com/jobs/ext/643147-remote-senior-full-stack-engineer) at **Edge Impulse**
- [Remote Senior Full-Stack Engineer](https://www.wearedevelopers.com/jobs/ext/356601-remote-senior-full-stack-engineer) at **Edge Impulse**