You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Zstd.compress and Zstd.decompress leak native memory when interrupted (Thread#raise, Timeout, Thread#kill) #160
While reviewing the C extension I found that Zstd.compress and Zstd.decompress leak native memory when a call is interrupted mid-flight.
Both release the GVL around the libzstd call via rb_thread_call_without_gvl. When that call returns, Ruby delivers any pending interrupt by raising from inside the C function. Neither function has an ensure/cleanup path, so the ZSTD_CCtx (compress) or the ZSTD_DCtx plus its 128 KB scratch buffer (decompress) is abandoned rather than freed.
In practice this is reachable via Timeout.timeout, Rack::Timeout, Sidekiq's shutdown handling (which calls Thread#raise on busy job threads), and plain Thread#kill.
Measured on macOS with the leaks tool and RUBY_FREE_AT_EXIT=1, 20 interrupted calls per scenario, current main (a4873ac). Ruby's own baseline for this harness is 1 leak / 192 bytes:
39 leaks, 1,533,665,472 bytes (roughly 75-85 MB per call)
Zstd.decompress of a 256 MiB-output frame, interrupted
60 leaks, 52,543,680 bytes (about 2.6 MB per call)
The decompress figures are a regression from my own #156, which released the GVL in Zstd.decompress. Before #156, decompress held the GVL throughout, so an interrupt could only be delivered after the call returned, with nothing left to leak. I rebuilt with just that one line reverted and confirmed it gives 0 leaks for the interrupted-decompress case. Flagging this plainly since it's my own regression to own.
The same root cause — a native context created, then a Ruby exception raised with nothing to free it — also shows up on argument errors raised after the context already exists: Zstd.compress with an unknown keyword, Zstd.compress with a level that doesn't fit in an int (e.g. 2**40), and StreamingCompress.new with an unknown keyword. Each of these leaks about 6 KB per call (21 leaks / 123,072 bytes over 20 calls).
The decompress side is already covered by @Watson1978's open PR #148, which wraps the decompress loop in rb_ensure. It currently has a few conflicts with main after #154 and #156 touched the same lines, but they're mechanical — three hunks. I rebased #148 locally (haven't pushed it anywhere) to check, and it clears every decompress-side leak above, including the interrupted-decompress one, with the suite passing. Happy to share the conflict resolution if that's useful, no pressure either way.
Side note: the Valgrind CI job added in #152 has been failing on main since it merged, and as far as I can tell it's solely down to the rb_decompressDCtx leak that #148 fixes.
I've put together a fix for the compress side in a separate PR.
Reproduction (compress side; decompress is analogous with a large output frame):
Activity