nodejs / debugging / memory-leaks / performance

What I Learned Debugging a Node.js Memory Leak

A service was crashing every three days with memory climbing to 1.5 GB. The leak turned out to be in a trusted dependency, not our own code.

2 min read
Cover image for What I Learned Debugging a Node.js Memory Leak

What I Learned Debugging a Node.js Memory Leak

A few months ago my team noticed that one of our services was crashing every three days. The pattern was consistent. Memory would climb to 1.5 GB and then the process would die. The health check would restart it and the cycle would start over.

We tried the usual things first. Check for unclosed database connections. Look for unbounded array pushes. Scan for event listeners that never got cleaned up. Nothing obvious showed up.

The tool that finally helped was node --inspect with the Chrome DevTools memory tab. We took a heap snapshot right after startup and another one after the memory had grown. Then we compared them.

The difference was full of strings. Thousands of them. All from the same source. We were using a library that cached log entries in memory before writing them to disk. The cache had no upper limit. Under normal traffic it stayed small, but during a burst it would grow until the process ran out of memory.

The fix was a one-line configuration change to set a max cache size. But finding that line took the better part of a week because we were looking in the wrong places. We assumed the leak was in our code, not in a dependency we trusted.

I learned a few things from this.

Heap snapshots are better than guessing. If you are debugging a memory issue, take a snapshot before you change anything. Take another one after the problem reproduces. Compare them. The diff will show you what is growing.

Do not assume the leak is in your code. Dependencies can leak too. The npm ecosystem has thousands of packages and most of them are maintained by small teams or single people. Bugs happen.

Add memory limits to production services. Node.js has a --max-old-space-size flag. Use it. It is better for a process to restart cleanly than to crawl toward an OOM kill while latency spikes.

We added a health check that watches RSS memory and sends an alert before the process gets close to the limit. We also wrote a test that loads the service with traffic and checks that memory stays flat. It is not perfect but it catches the kind of regression we missed the first time.

Keep reading
Cover image for Why I Reach for Map Over Object in JavaScriptjavascript

Why I Reach for Map Over Object in JavaScript

Plain objects have prototype quirks and limited key types. Map handles dynamic collections more cleanly and eliminates a whole class of bugs.

Cover image for The Art of Reading Someone Else's Codeproductivity

The Art of Reading Someone Else's Code

You spend more time reading code than writing it. Here is how I approach an unfamiliar codebase without drowning in details.

Cover image for CSS Container Queries Are Finally Herecss

CSS Container Queries Are Finally Here

Container queries let you style elements based on their parent size instead of the viewport. After years of waiting, they are finally ready for production.