r/programming • u/mttd • 6h ago
r/programming • u/Lopsided-Delivery710 • 10h ago
Go Was Burning 60% CPU Just Running GC. The Fix Was a Memory Limit.
buildopsy.comA Go service can burn CPU collecting garbage without having a memory leak.
That is the counterintuitive part of Zomato's Go garbage collection story.
The reported symptoms were striking: garbage collection consuming 25–60% of CPU in some services, alongside containers getting OOM-killed without an obvious memory leak.
These look like separate problems. But they can share a common cause: the garbage collector's pacing decisions don't automatically account for the container's actual memory budget.
The problem with doubling the live heap
Go's garbage collector uses GOGC to control when the next collection should happen.
With the default GOGC=100, the GC target is roughly based on the live heap growing by 100% relative to the previous cycle, with additional accounting for GC roots and other implementation details.
In simplified terms:
Next GC target ≈ live heap × (1 + GOGC/100)
At GOGC=100:
Live heap: 500 MiB
Approx. target: 1,000 MiB
That target is reasonable as a starting point when the process has enough memory available.
But containers introduce another constraint: the process has a memory budget imposed from outside the Go runtime.
The GC's heap-growth target and the container's memory ceiling are two different things.
When they aren't coordinated, two failure modes become possible.
Failure mode 1: OOM kills without a leak
Suppose a container has a memory limit of 1 GiB.
The Go process has a live heap of 600 MiB. A simplified GOGC=100 calculation puts the next target around 1,200 MiB.
But the container cannot safely accommodate that much memory. The process also needs room for goroutine stacks, runtime metadata, native allocations, and other memory outside the live heap.
The container can hit its limit before the collector's normal heap target becomes appropriate.
The result can be an OOM kill even when reachable memory isn't growing uncontrollably.
That doesn't mean every OOM without an obvious leak is a GC pacing issue. Transient allocation spikes, native memory, and other causes still need investigation.
The important point is that a heap-growth target alone isn't a container memory budget.
Failure mode 2: Excessive GC CPU
Now consider the opposite situation.
A service has a relatively small live heap and plenty of memory available in its container.
The GC keeps using a target derived from that live heap. Depending on the allocation rate and workload, it may collect frequently even though the container has significant unused headroom.
Every collection consumes CPU.
At fleet scale, unnecessary GC work can become a meaningful compute cost.
This is why the following combination deserves attention:
- High GC CPU.
- A relatively flat live-heap graph.
- No convincing evidence of sustained heap growth.
It is a diagnostic clue, not proof. A service can allocate and discard large volumes of temporary objects while maintaining a flat live heap. Allocation rate, GC frequency, and allocation profiles matter too.
But before treating every high-GC-CPU incident as a memory leak, inspect the collector's pacing and the container's actual memory constraints.
The fix: Give Go a realistic memory limit
Go provides GOMEMLIMIT to let applications communicate a soft memory limit to the runtime.
Unlike GOGC, which controls GC aggressiveness in relation to heap growth, GOMEMLIMIT helps the runtime account for its overall memory footprint when deciding how aggressively to collect.
Conceptually, the runtime must balance two competing costs:
- Collect too late, and the process risks exhausting its container memory budget.
- Collect too aggressively, and the application wastes CPU reclaiming memory it could have used safely.
A memory limit gives the runtime another constraint to consider.
In the Zomato case, the reported approach used container metadata to derive a per-container limit, refreshed it periodically, and combined memory tuning with CPU scheduling adjustments.
A limit around 90% of the container ceiling can be a starting point for investigation, not a universal safe value. The correct margin depends on the application's non-heap memory, traffic spikes, allocation behavior, and operational requirements.
The goal is to give the runtime a realistic budget while preserving enough headroom for the rest of the process.
Why GOMEMLIMIT isn't a hard wall
This distinction matters in production.
GOMEMLIMIT is a soft limit, not a strict process-memory cap. The runtime can exceed it when necessary, and it does not guarantee that a container will avoid OOM kills.
The runtime also has to balance memory pressure against GC CPU overhead.
If the limit is too high, it may offer little protection against container memory pressure. If it is too low, the runtime can spend excessive CPU collecting garbage in an attempt to stay within the target.
That is why blindly setting GOMEMLIMIT equal to the container limit is not a complete solution.
A production rollout should account for:
- The container's actual memory limit.
- Heap and non-heap memory usage.
- Peak allocation rates and traffic bursts.
- GC CPU and pause behavior.
- OOM events and memory headroom after deployment.
Start with representative workloads, observe the results, and tune based on measurements rather than assuming a single multiplier works everywhere.
r/programming • u/fykup • 10h ago
Spring Boot Connection Pools: 20 vs 2 Idle PostgreSQL Connections
pvrlabs.xyzr/programming • u/derjanni • 12h ago
From Pixel Art to Packet Sniffing: A Programmer’s Guide to Flipper Zero Programming and App…
programmers.fyir/programming • u/actually_NOLAN • 16h ago
Cooldown or compromise: the three-day rule for npm and PyPI
research.eye.securityr/programming • u/correctMeIfImcorrect • 19h ago
Layered vs Hexagonal vs Onion vs Clean Architecture | Visually Explained
youtube.comI hope you find this resource useful and that it helps you better understand things when working on new projects!
r/programming • u/Ok_Stomach6651 • 21h ago
How Stripe uses graph search and state machines to auto-remediate a global database fleet
stripe.devr/programming • u/dfbaggins • 22h ago
First Steps into Frames in Python
tech.daniellbastos.com.brr/programming • u/lelanthran • 1d ago
yet another place where a linux man page is inaccurate
sebsite.pwr/programming • u/Efficient_File • 1d ago
Do you still need to drain the response body in Go 1.27?
importstatic.comr/programming • u/Ok_Stomach6651 • 1d ago
How Netflix Simplified Batch Compute with Kueue
netflixtechblog.comr/programming • u/MegaGreenLightning • 1d ago
A C++ Trick Question - Blog Post
greenlightning.eur/programming • u/sgr_nz • 1d ago
Can I offer you a nice CI Pipeline in this trying time?
practicalai.co.nzTL; DR - A bone dry technical article about continuous integration, of little interest to non-programmers.
r/programming • u/nightcracker • 1d ago
Black-Box HashDoS: Attacking Hash Tables Without Knowing the Hash Function
orlp.netr/programming • u/IntelligentDay4375 • 1d ago
Das UI-Framework für Java & Spring
youtube.comr/programming • u/masaischool_in • 1d ago
Where do you go for help when you're stuck now? Survey says 64% of developers visit Stack Overflow less for simple questions
stackoverflow.blogThe 2026 Stack Overflow survey reports that 83% of developers search online for answers and 70% ask an AI agent, and 64% say they visit Stack Overflow less for simple questions.
When you're stuck as a learner, where do you start: search, an AI tool, a forum, a friend? Has the order changed in the past year, and what do you do to make sure you actually learn the answer?
Source (first comment): Stack Overflow 2026 survey post. Opened and verified: 83%, 70%, 64%.
r/programming • u/Iamsodarncool • 1d ago
Why I Love AI (No, Not That Kind)
possibilityspace.orgr/programming • u/CircumspectCapybara • 1d ago
Anthropic designs new in-house monorepo source control system "Scion", code review tool "Antique", designed for massive scale
anthropic.comr/programming • u/Ok_Stomach6651 • 1d ago
Ensuring Similar Computational Resources to Stateful Workloads Running on a Diverse Server Fleet
uber.comr/programming • u/pollop-12345 • 1d ago
Zxc 0.15.0 - decompression 2x faster than Zstandard
libzxc.orgr/programming • u/BrewedDoritos • 1d ago
kcgi: minimal CGI and FastCGI library for C/C++
kristaps.bsd.lvr/programming • u/alefteris • 2d ago
Built-in alternatives to pre-commit-hooks in prek and hk
thanoslefteris.comr/programming • u/jessillions • 2d ago