Repository navigation
Conversation
Motivation: ZstdDecompressor creates a bare ZstdDecompressCtx, so a small frame can make it allocate a window of up to 128 MiB before any data is decompressed. zstd-jni's ZstdDecompressCtx.setLongMax (luben/zstd-jni#431) lets the window be limited, but no zstd-jni release has it yet. Modification: Add ZstdDecompressionSettings.withWindowLogMax, where 0 keeps the libzstd limit, mirroring ZstdCompressionSettings.withWindowLog. ZstdDecompressor looks up setLongMax reflectively, because zstd-jni is an optional dependency that users supply. Without it, a configured limit fails materialization with an UnsupportedOperationException instead of being ignored. Document the option and the RFC 9659 limit. Result: Users can set the window limit now, and it takes effect as soon as they use a zstd-jni release with setLongMax. A later change can bump zstd-jni and call the method directly. Tests: - sbt "stream-tests/testOnly org.apache.pekko.stream.io.compression.ZstdSpec org.apache.pekko.stream.io.compression.CompressionWithoutZstdSpec" with zstd-jni 1.5.7-24: 42 passed References: Refs apache#3674
He-Pin
reviewed
Oct 26, 2026
Motivation: Review feedback on apache#3710 asked for a MethodHandle instead of reflection. Modification: Look up ZstdDecompressCtx.setLongMax with MethodHandles.publicLookup and call it with invokeExact. Exceptions from setLongMax are no longer wrapped, so the InvocationTargetException unwrapping is gone. Result: The optional zstd-jni method is called through a MethodHandle. Tests: - sbt "stream-tests/testOnly org.apache.pekko.stream.io.compression.ZstdSpec org.apache.pekko.stream.io.compression.CompressionWithoutZstdSpec" with zstd-jni 1.5.7-24: 42 passed - sbt "++3.3.8" stream/compile References: Refs apache#3674
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
ZstdDecompressorcreates a bareZstdDecompressCtx. libzstd allocates the window buffer from the frame header before any decompressed-size limit applies, so a small input can make the decoder allocate up to 128 MiB per stream. Server-side users such as pekko-http'szstdcontent-coding need a lower limit; RFC 9659 caps it at a window log of 23 (8 MiB).zstd-jni's
ZstdDecompressCtx.setLongMax(luben/zstd-jni#431) sets that limit, but no zstd-jni release includes it yet (the latest is 1.5.7-24). This PR adds the setting for 2.0.0, so finishing the work later only needs a dependency bump.Modification
ZstdDecompressionSettings.withWindowLogMax(Int)andwindowLogMax. 0, the default, keeps libzstd's limit. This mirrorsZstdCompressionSettings.withWindowLog, and the validation range is the same.ZstdDecompressorlooks upsetLongMaxreflectively, because zstd-jni is an optional dependency that users add themselves. If the method is missing, a configured limit fails materialization with anUnsupportedOperationExceptionrather than being silently ignored. The default path is unchanged.zstdDecompresspage.Result
On 2.0.0, users can configure the window limit, and it takes effect as soon as they use a zstd-jni release with
setLongMax. The follow-up (2.0.1) bumpszstdJniVersionand replaces the reflection with a direct call; there is a TODO at the lookup. The new test already covers the real path: it checks whether the method exists, and once it does, it asserts that a 2^24-window frame is rejected at 23 and decodes at 24.The settings class and the zstd API are new in 2.0.0 and
@ApiMayChange, so there is no binary compatibility impact.ZstdDecompressoris@InternalApi.Tests
sbt "stream-tests/testOnly org.apache.pekko.stream.io.compression.ZstdSpec org.apache.pekko.stream.io.compression.CompressionWithoutZstdSpec"with zstd-jni 1.5.7-24: 42 passed. The new window test exercised theUnsupportedOperationExceptionbranch.sbt stream/scalafmt stream-tests/Test/scalafmtReferences
Refs #3674