> Markdown version of [/videos/2057-the-memory-leak-that-ate-our-cluster-a-postmortem](https://www.wearedevelopers.com/videos/2057-the-memory-leak-that-ate-our-cluster-a-postmortem). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# The Memory Leak That Ate Our Cluster: A Postmortem

A persistent Puppeteer memory leak was silently eating our Node.js cluster. Discover how we finally eliminated OOM crashes and zombie processes using disposable contexts and OS-level process management.

- **Speakers:** [Dina Matveev](https://www.wearedevelopers.com/@dina-matveev)
- **Event:** World Congress 2026 Europe - Virtual Stage
- **Published:** July 2, 2026
- **Duration:** 17:55
- **URL:** https://www.wearedevelopers.com/videos/2057-the-memory-leak-that-ate-our-cluster-a-postmortem

## Summary

A Node.js cluster generating PDFs and screenshots experienced frequent Out of Memory (OOM) crashes. Initial attempts to scale by adding more pods and RAM failed, as the memory leak was persistent and gradual. By utilizing Datadog dashboards to correlate memory spikes with specific endpoint calls, the engineering team traced the root cause to the screenshot generation endpoint, which relied on Puppeteer to drive headless Chrome. The team discovered that their initial singleton approach—reusing a single browser instance for all requests to save startup time—caused Chrome to bloat to 10GB by accumulating cache, cookies, and unclosed resources. Pivoting to a one-browser-per-request model isolated the state but inadvertently spawned zombie processes because Node's garbage collector could not terminate external OS-level Chromium processes when connections dropped. The definitive fix was a hybrid approach using Puppeteer's incognito contexts. This maintained a single lightweight browser instance while opening isolated, disposable contexts for each request, delivering both performance and complete state separation. To ensure long-term cluster stability, the team shifted from treating Puppeteer as a simple library to managing it as an OS-level process manager. Critical proactive measures included enabling init in the Docker configuration—making PID 1 responsible for adopting and destroying zombie child processes—and enforcing strict timeouts on all browser interactions to prevent frozen pages from hogging pod resources. Additionally, applying strategic Chrome flags, specifically bypassing shared memory limits with disable-dev-shm-usage and completely disabling graphics pipelines with disable-gpu and disable-software-rasterizer, proved vital for running slim, reliable headless browsers in server containers.

**Keywords:** node.js memory leak profiling, kubernetes out of memory crashes, puppeteer headless chrome automation, datadog endpoint memory correlation, browser automation singleton pattern, zombie processes linux collection, docker init process management, puppeteer incognito browser contexts, chromium memory state isolation, chrome devtools protocol scaling, docker container shared memory, disable-dev-shm-usage flag, headless browser graphics pipeline, stateless api resource cleanup

## Chapters

1. **Architecture context and out of memory cluster crashes** (00:00) — A document generation API experiences severe instability as pods unexpectedly crash from out-of-memory errors.
1. **Testing naive solutions with more pods and memory** (01:23) — Increasing pod counts and doubling RAM limits fail to resolve persistent memory leaks.
1. **Visualizing memory limits and isolating suspicious endpoints** (02:25) — Creating a Datadog dashboard reveals gradual memory hoarding and correlates spikes to specific API endpoints.
1. **Investigating database connections and node heap dumps** (04:01) — Checking open database connections and profiling the node heap fails to explain the memory magnitude.
1. **Analyzing screenshot generation and puppeteer library versions** (05:47) — Upgrading Puppeteer does not resolve the memory climb caused by underlying code implementation flaws.
1. **Identifying memory accumulation in singleton browser patterns** (07:25) — Reusing a single Chrome instance to save overhead leads to massive accumulations of cache and unclosed resources.
1. **Attempting browser isolation with one instance per request** (08:59) — Launching a fresh Chromium instance for every request prevents shared state but still results in memory errors.
1. **Uncovering unclosed zombie processes in the operating system** (10:01) — Treating Puppeteer as a process manager reveals that failed requests leave unused Chromium zombies hoarding memory.
1. **Implementing incognito contexts for performance and state isolation** (12:16) — Using a single Chrome process with separate disposable window contexts resolves the memory leak entirely.
1. **Destroying zombie processes by enabling init in Docker** (13:32) — Adding the init process to Docker Compose automatically adopts and collects exit statuses for orphaned child processes.
1. **Preventing resource exhaustion by defining browser task timeouts** (14:26) — Setting strict timeouts on page navigations and interactions stops frozen browsers from permanently holding memory slots.
1. **Customizing Chrome launch flags for headless server environments** (15:09) — Disabling extensions, bypassing shared memory limits, and dropping GPU rendering drastically improves stability.

## Related Moments

- [Browser developer tools for detecting memory leaks](https://www.wearedevelopers.com/videos/2011-the-lord-of-the-memory-the-return-of-the-leak) (from "The Lord of the Memory: The Return of the Leak")
- [Managing browser memory usage across Chromium environments](https://www.wearedevelopers.com/videos/1938-wearedevelopers-live-the-shape-of-svg-to-come) (from "WeAreDevelopers LIVE - The Shape of SVG to Come!")
- [Isolating metaspace memory leaks with a minimal reproducer](https://www.wearedevelopers.com/videos/100321-swapping-code-losing-memory-a-jvm-deep-dive) (from "Swapping Code, Losing Memory: A JVM Deep Dive")
- [Containerizing Puppeteer and Chromium within Docker environments](https://www.wearedevelopers.com/videos/322-automate-everything-via-nodejs-and-puppeteer) (from "Automate everything via NodeJS and Puppeteer")
- [Debugging isolated contexts and managing dynamic browser memory](https://www.wearedevelopers.com/videos/1587-web-fragments-incremental-micro-frontends-migration-approach-for-enterprise) (from "Web-Fragments: incremental micro-frontends migration approach for enterprise")
- [Understanding Puppeteer and its underlying architectural design](https://www.wearedevelopers.com/videos/322-automate-everything-via-nodejs-and-puppeteer) (from "Automate everything via NodeJS and Puppeteer")

## Related Articles

- [Dev Digest 128 - Do not Google Monopoly](https://www.wearedevelopers.com/magazine/465-dev-digest-128-do-not-google-monopoly)
- [Dev Digest 129 - Now that's what I call private data!](https://www.wearedevelopers.com/magazine/468-dev-digest-129-now-that-s-what-i-call-private-data)
- [Dev Digest 106 - DHH in HD](https://www.wearedevelopers.com/magazine/394-dev-digest-106-dhh-in-hd)
- [Dev Digest 127 - putting 5 rings on it](https://www.wearedevelopers.com/magazine/464-dev-digest-127-putting-5-rings-on-it)

## Related Jobs

- [Remote Senior Full-Stack Engineer](https://www.wearedevelopers.com/jobs/ext/645320-remote-senior-full-stack-engineer) at **Edge Impulse**
- [Remote Senior Full-Stack Engineer](https://www.wearedevelopers.com/jobs/ext/644637-remote-senior-full-stack-engineer) at **Edge Impulse**
- [Remote Senior Full-Stack Engineer](https://www.wearedevelopers.com/jobs/ext/639235-remote-senior-full-stack-engineer) at **Edge Impulse**
- [Remote Senior Full-Stack Engineer](https://www.wearedevelopers.com/jobs/ext/679408-remote-senior-full-stack-engineer) at **Edge Impulse**
- [Remote Senior Full-Stack Engineer](https://www.wearedevelopers.com/jobs/ext/646086-remote-senior-full-stack-engineer) at **Edge Impulse**
- [Remote Senior Full-Stack Engineer](https://www.wearedevelopers.com/jobs/ext/356601-remote-senior-full-stack-engineer) at **Edge Impulse**