Hunting a Memory Leak in GoQuorum with Flame Graphs

How every RPC transaction quietly became immortal — and the 13-line fix, proven live.

GoQuorum nodes have a memory problem. RSS climbs steadily with transaction traffic — on the IBFT validators and on the reader nodes that only serve RPC — and it never comes back down. No crashes, no errors. Just a slow, patient climb toward the OOM killer.

This is the story of finding the root cause: a static sweep that surfaced five unbounded code paths, deployment facts that killed four of them, memory flame graphs in the style of Brendan Gregg’s memory leak flame graphs, a two-node reproduction network that pinned the fifth — and finally a 13-line patch, verified by watching 4,000 permanently stuck transactions get evicted in a single sweep.

The numbers that ended the investigation:

• 4,000 transactions pinned in the reader’s queue — 3.9× the 1,024-slot GlobalQueue cap
• 4,001 “Setting new local account” log lines — one per unique RPC sender, never pruned
• +27 MB vs +9 MB of heap growth — reader (RPC entry point) vs validator (p2p only), identical chain traffic
• Goroutines flat at 109/138 across every phase — whatever this was, it wasn’t a goroutine leak

Step 1: Read the code like a leak hunter

GoQuorum is a fork of go-ethereum frozen at geth v1.10.3. We ran five parallel sweeps — over raft/, consensus/istanbul/, private/ + extension/, qlight/ + plugin/ + permission/, and core/ + eth/ + p2p/ — looking for the classic Go leak shapes: maps that only grow, caches without eviction, goroutines that never exit, subscriptions never unsubscribed.

Five paths came back genuinely unbounded.

#1 — The IBFT/QBFT future-message backlog. Every future-view message that passes signature checks is pushed onto a per-validator priority queue with no size cap and no dedup, so the same message enqueued N times occupies N slots. The only eviction pops messages once the chain reaches their view — so if the sequence stalls (one validator down, a network partition), nothing is ever evicted while round-change traffic keeps arriving. QBFT amplifies it: backlogged messages carry full PreparedBlock payloads and justification sets, making each entry kilobytes to megabytes. This is the one that matches "RSS climbs during instability and never comes back." (consensus/istanbul/qbft/core/backlog.go:107, ibft/core/backlog.go:80)

#2 — The raft speculative chain. The minter keeps minting on every new-transaction tick and appends each block, plus every transaction hash, to the speculative chain; entries are only removed when a ChainHeadEvent confirms the block. Lose quorum while transactions keep flowing and unappliedBlocks accumulates full *types.Block objects indefinitely. The authors knew — the code carries a TODO: "we're going to want to put an upper limit on our speculative mining chain length." (raft/minter.go:167)

#3 — Raft log entries live in RAM until a snapshot. Every raft entry is a full RLP-encoded block appended to etcd’s in-memory MemoryStorage. Compaction only runs after 250 applied entries, and can only discard up to appliedIndex — so proposals that never commit are never compactable. Healthy cost: up to 250 blocks of RAM between snapshots. Stalled cost: unbounded, one entry per proposed block. (raft/handler.go:870, raft/snapshot.go:183)

#4 — Contract extension leaks an RPC client and a goroutine per event. service.client(psi) is called twice in the extension-created handler: the first client is closed via defer, the second — the one actually used for the TransactionInBlock call — never is. Each unclosed in-process RPC client pins a dispatch goroutine forever, one per contract-extension event, per PSI, for the life of the node. Bonus defect found alongside it: the watcher loop's break on subscription error only exits the select, leaving the goroutine spinning at 100% CPU on a closed channel. (extension/backend.go:119–121, extension/subscriptions.go:47)

#5 — Plugin restart orphans the old subprocess. basePlugin.Start() overwrites its plugin client without killing the previous one, orphaning the old subprocess, its gRPC connection, and its broker goroutines. The qlight token manager calls Start() directly — once at startup, and again on every qlight_reloadPlugin RPC call. Each reload leaks one OS process, which shows up as a growing child-process count rather than heap. (plugin/base.go:102–127, qlight/token_holder.go:50)

Cleared as properly bounded: per-peer p2p metrics, the tx-pool’s own bounds, filter eviction, Tessera response handling, the 5-minute TTL private-payload caches, permission LRU caches, qlight client caches, and RPC notifier backpressure.

Two non-leak bugs fell out of the sweep anyway. eth/handler.go:593–602 duplicates its broadcast loop verbatim — every transaction is queued to every peer twice. And GoQuorum force-enables trie preimage recording with a literal if true || at cmd/utils/flags.go:2577, growing the secure-key-* LevelDB table without bound on every node. That one costs disk and compaction working set rather than Go heap, which is exactly why it hides from a heap profile.

Then the symptoms killed four of the five

We don’t run raft, so #2 and #3 were out. We don’t use the privacy features, so #4 and #5 were out. And the decisive fact: the reader nodes leak at the same rate as the validators — but readers never even start the IBFT consensus core, so #1 couldn’t be it either.

Whatever this was, it lived in the code path every full node shares, and it scaled with transaction traffic. Four of five findings died on deployment facts alone, before a single profile was captured.

Step 2: Make memory visible

Go’s runtime tracks allocation stacks natively, so pprof heap profiles stand in for the malloc/brk/mmap tracing on Gregg’s page: inuse_space is his "memory in use" view, alloc_space the allocation-rate view, and the difference between two snapshots is his red/blue leak-hunting workflow. Rendering uses his flamegraph.pl and difffolded.pl unchanged, fed by a ~90-line pprof-to-folded converter.

# nodes run with: --pprof --pprof.addr 127.0.0.1 --pprof.port 6060
./heapflame.sh capture http://127.0.0.1:6060 t0     # baseline after warm-up + forced GC
./heapflame.sh capture http://127.0.0.1:6060 t1     # hours later
./heapflame.sh diff t0.heap.pb.gz t1.heap.pb.gz growth.svg
./heapflame.sh goroutines t1.goroutine.pb.gz goroutines.svg

Three views do most of the work. The examples below are smoke tests rather than evidence: they come from a throwaway single-node network — 8,000 transactions, 187 blocks — that existed only to confirm the pipeline produced sane output before it was aimed at anything real. It happened to be a raft node, which is not what we run in production; the raft frames in these stacks are scenery. What matters here is the shape of each view.

Growth-only: what gained bytes between two captures. Width is bytes added, and nothing else is drawn. This is the leak-hunting view; start here.

Two-sided differential: red grew, blue shrank, white was unchanged. This is how you separate churn from retention.

Goroutines by stack: the leak that never shows up in the heap.

Two more round out the set: inuse_space at t1 tells you what is resident right now, and alloc_space tells you everything ever allocated. The pair is how you tell a leak from noise — a leak is large in both, while pure churn is large only in alloc_space.

Step 3: Reproduce it

To turn suspicion into proof we built the production topology in miniature: a two-node IBFT network compiled from the same source, one validator and one reader, with transactions entering through the reader’s RPC exactly like our applications do.

Phase A, the control. 8,000 transactions from a single sender. Both nodes grew about 15 MB, all of it in bounded caches that plateau. Boring — which is what you want from a control.

Phase B, the trap. In the geth 1.10.3 era, every transaction submitted through a node’s RPC marks its sender as a “local” account (core/tx_pool.go:708–712). The locals set is an unbounded map with no pruning path — and locals' pooled transactions are exempt from every eviction mechanism: the 3-hour lifetime sweep, the 1,024-slot queue cap, and pool-overflow truncation. It is active even at gas price 0.

So we flooded the reader with 4,000 fresh senders, each submitting one executable transaction and one future-nonce transaction carrying 2 KB of calldata — a transaction that can never execute, and therefore can never leave.

Get Lou Ciampanelli’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

The result:

• The reader ended with 4,000 permanently queued transactions — 3.9× the supposed queue cap. Remotes can’t exceed 1,024; only eviction-exempt locals can.
• Its log showed exactly 4,001 “Setting new local account” lines: one per unique sender, never removed.
• The reader gained +27 MB of heap against the validator’s +9 MB under identical chain traffic — and the validator, which received the same transactions over p2p as evictable “remotes,” held zero queued transactions afterward.

That’s the root cause, and it explains every symptom we had: growth proportional to traffic (every new sender is retained forever), on every node type (anything accepting RPC submissions is affected), that never recovers (nothing ever prunes it).

Step 4: Fix it, then watch the fix work

Right now, with no code change: add --txpool.nolocals to every node that accepts RPC transaction submissions. Senders become ordinary remotes, so lifetime eviction and queue caps apply to them. The side effect is that the pool journal no longer preserves pending transactions across restarts.

The proper patch is 13 lines: make lifetime eviction apply to locals’ queued transactions, and prune the locals set once an account holds nothing in the pool.

// core/tx_pool.go — evict ticker in func (pool *TxPool) loop()
 for addr := range pool.queue {
-    // Skip local transactions from the eviction mechanism
-    if pool.locals.contains(addr) {
-        continue
-    }
-    // Any non-locals old enough should be removed
+    // Quorum: locals are NOT exempt from queue lifetime eviction.
+    // Every RPC-submitted sender is marked local (AddLocal), so the
+    // upstream exemption lets stale queued transactions accumulate
+    // without bound on RPC-serving nodes.
+    // Any account old enough should be removed
     if time.Since(pool.beats[addr]) > pool.config.Lifetime {
         ...
     }
 }
+// Quorum: stop tracking accounts as local once they have nothing
+// left in the pool, so the locals set cannot grow without bound.
+for _, addr := range pool.locals.flatten() {
+    if pool.pending[addr] == nil && pool.queue[addr] == nil {
+        pool.locals.remove(addr)
+    }
+}
 pool.mu.Unlock()

// accountSet
+// remove deletes an address from the set. Quorum: used to prune local
+// markers for accounts that no longer have transactions in the pool.
+func (as *accountSet) remove(addr common.Address) {
+    delete(as.accounts, addr)
+    as.cache = nil
+}

There is a behavior change here, by design: senders that submit via RPC and then go idle lose their pool-eviction immunity. The one upstream test that asserted the old immunity (TestTransactionQueueTimeLimiting) is updated in the same patch, and the tx-pool suite passes. Modern go-ethereum later deleted the blanket locals exemption entirely — this moves the fork in the same direction.

Verification was the satisfying part. We rebuilt geth with the patch and restarted the flooded reader. Its journal dutifully reloaded all 4,000 previously-immortal transactions — and the first eviction sweep past their lifetime removed every single one. queued: 4000 → 0, watched live.

While you’re in there, three cheap wins from the same sweep: delete the duplicated broadcast loop at eth/handler.go:593–602, which currently sends every transaction to every peer twice; remove the forced-preimages if true || at cmd/utils/flags.go:2577 and :2083 to stop unbounded secure-key-* disk growth; and backport geth #23556 if your dapps use WebSocket subscriptions, because severed WS clients are never detected in the 1.10.3 base.

How to replicate this

Everything here runs on public sources. One caveat up front: Consensys archived GoQuorum on 5 June 2026, and the documentation site went offline with it — so the repositories below are read-only and the docs links point at the Internet Archive.

1. Get the source. Clone Consensys/quorum and build with make geth. This investigation used commit 5ffacc482 — GoQuorum 24.4.1, on the geth 1.10.3 base.

2. Stand up a network. The Quorum Developer Quickstart (npx quorum-dev-quickstart) generates a Docker Compose IBFT network of four validators plus a non-validator node — validator-plus-reader is exactly the topology this investigation needed. Swap in your own build, or hand-roll two nodes from the genesis it generates. The original getting-started documentation and its quickstart tutorial survive only as Archive snapshots; the Markdown source of the docs is still on GitHub if you'd rather read it there.

3. Turn on profiling. Start every node with --pprof --pprof.addr 127.0.0.1 --pprof.port 6060. Keep it on loopback — the pprof endpoint is unauthenticated and will happily dump your heap to anyone who asks.

4. Get the flame graph tooling. Clone brendangregg/FlameGraph: flamegraph.pl does the rendering, difffolded.pl the red/blue differential. The method comes straight from Memory Leak (and Growth) Flame Graphs. The only custom piece here is a converter from pprof's protobuf into folded stacks — about 90 lines; if you'd rather not write one, go tool pprof -raw piped through the repo's stackcollapse-go.pl gets you most of the way.

5. Run both phases. Phase A is your control: N transactions from a single funded sender, submitted through the reader’s RPC. Phase B is the trap: N fresh senders, each submitting one executable transaction and one with a deliberate nonce gap, so the second can never execute. Capture the heap before and after each phase, and force a GC first (debug_freeOSMemory) so you're measuring retention rather than garbage.

6. Read the result. On the reader, grep -c "Setting new local account" should equal your unique sender count plus one, txpool_status should report queued above 0x400, and the growth-only flame graph should show the SendRawTransaction tower. On the validator, all three should come back clean.

Point this at your own nodes

One-minute triage first, no restart required — this tells you whether you’re even looking at a code leak:

curl -s -X POST -H 'Content-Type: application/json' \
  --data '{"jsonrpc":"2.0","method":"debug_memStats","params":[],"id":1}' http://NODE:RPCPORT
ps -o rss= -p GETH_PID
# RSS growing while HeapInuse stays flat = allocator behaviour, not a leak in your code

grep -c "Setting new local account" geth.log
# each line is one permanently retained RPC sender
curl -s -X POST -H 'Content-Type: application/json' \
  --data '{"jsonrpc":"2.0","method":"txpool_status","params":[],"id":1}' http://NODE:RPCPORT
# "queued" above 0x400 (1,024) is the smoking gun — remotes cannot exceed the cap

Then take two captures a few hours apart and read the red:

• A SendRawTransaction tower, with queued above 1,024 — the tx-pool locals leak. --txpool.nolocals now, the pruning patch after.
• backlog / prque / istanbul frames — finding #1. Cap the per-validator queue, dedup by view, prune on commit.
• speculative_chain / minter — finding #2. Bound the chain length; stop minting when unconfirmed blocks exceed N.
• MemoryStorage / serveLocalProposals — finding #3. Lower snapshotPeriod, and find out why apply stalled.
• rpc.(*Client).dispatch climbing in the goroutine graph — finding #4. Reuse the already-open client and return on subscription error.
• Child geth-plugin processes climbing — finding #5. Call Stop() before Start() in the reload path.
• RSS climbing while HeapInuse is flat — not a code leak at all. That's GC and allocator tuning (GOGC, memory limit), not a patch.

Takeaways

• Symptoms triage candidates better than code reading alone. “Both node types, proportional to traffic” eliminated four of our five static findings in one stroke, before any profiling.
• Differential heap profiles are the memory-leak workhorse. One flame graph of what grew beats hours of staring at absolute numbers.
• Reproduce before you patch. The two-node repro took an afternoon and turned “plausible candidate” into “watched it happen, watched the fix kill it.”
• Frozen forks inherit upstream’s old bugs. If you run a fork pinned to an old base, upstream’s later fixes are a ready-made checklist of what’s still wrong in yours.

Tooling: pprof plus Brendan Gregg’s flamegraph.pl and difffolded.pl, with a ~90-line converter between them. GoQuorum 24.4.1 (geth 1.10.3 base) at commit 5ffacc482, darwin/arm64 — the project is archived by Consensys, so patches live downstream.