Skip to content

TEZ-4762: UnorderedPartitionedKVWriter: size each partition's codec stream to the partition - #547

Open
deniskuzZ wants to merge 2 commits into
apache:masterfrom
deniskuzZ:TEZ-4762
Open

deniskuzZ wants to merge 2 commits into
apache:masterfrom
deniskuzZ:TEZ-4762

Conversation

@deniskuzZ

Copy link
Copy Markdown
Member

A spill of UnorderedPartitionedKVWriter opens an IFile.Writer per partition, and for Snappy and LZ4 each one goes
through codec.createOutputStream: the codec parses its buffer size from the Configuration (under the codec's conf
lock), and BlockCompressorStream allocates and zeroes a 256 KB buffer plus the compressed buffer, while a partition
often holds a few KB. With 1,009 partitions that is ~336 MB allocated per spill. In Hive on Tez (LLAP), TPC-DS 10 TB,
the part of this setup that the patch removes is 2.5 % (q75) and 2.8 % (q64) of LLAP CPU (profile in TEZ-4762).

The patch sizes each partition's stream to its data:

  • CodecUtils.getOutputBufferSize(codec) reads the buffer size once per output (-1 for codecs other than Snappy and
    LZ4), and CodecUtils.createOutputStream(codec, out, compressor, bufferSize, dataLength) builds their
    BlockCompressorStream directly, with no Configuration lookup and no lock. A stream known to fit in one block gets a
    block of its length and a buffer of its compression bound, and writes the same bytes as the configured stream;
    otherwise the configured stream is used.
  • IFile.Writer takes the codec buffer size and an upper bound of its raw length (Writer.maxRawLength(records, keyValueBytes)), and sizes its write buffer to the stream too.
  • UnorderedPartitionedKVWriter passes each partition's bound from the buffers' record and byte counts on a spill, and
    the sum of the spilled raw lengths on the final merge. Other codecs and other writers keep codec.createOutputStream.

Microbenchmark (JMH, one spill of 1,009 partitions, 22 B records, JDK 21, Apple M-series, avgt of 10; before = the
configured stream, as on master):

Codec Per partition Time Allocated per spill
Snappy 22 B -76 % (2,963 -> 703 us) 335.8 MB -> 5.2 MB
Snappy 2.2 KB -48 % (4,878 -> 2,535 us) 335.8 MB -> 10.9 MB
Snappy 22 KB -10 % (15,898 -> 14,310 us) 335.8 MB -> 62.1 MB
LZ4 22 B -59 % (3,249 -> 1,346 us) 335.8 MB -> 5.2 MB
LZ4 2.2 KB -41 % (5,153 -> 3,032 us) 335.8 MB -> 10.4 MB
LZ4 22 KB -12 % (15,198 -> 13,431 us) 335.8 MB -> 57.8 MB

Tests: TestCodecUtils (a sized stream writes the same bytes as the configured one and round-trips, also when the
length is underestimated), TestIFile (a sized writer writes the bytes of an unsized one),
TestUnorderedPartitionedKVWriter (a spill's streams are sized to their partitions, bounded allocation, and every
partition is written as the unsized path would); with the existing tests of the three classes, 401 run, 0 failures.
Nine mutations of the sizing (no sizing, buffer without the compression bound, block one byte short, bound without the
vints, spill unsized, spill reading the conf, merge ignoring the spills, write buffer unsized, spill bound from the first
buffer) each fail at least one of them.

…tream to the partition

- A spill opens a codec stream per partition: SnappyCodec and Lz4Codec parse the buffer size from the Configuration
  and allocate and zero a 256 KiB buffer for each, and IFile.Writer a 64 KiB write buffer, while a partition holds a
  few KB (Hive LLAP, TPC-DS 10 TB q75 / q64: writer setup in spills is 3.2 / 3.5 % of LLAP CPU, Integer.parseInt
  1.5 / 1.5 % and these buffers 1.0 / 1.4 % of it)
- CodecUtils reads the buffer size once per output and builds these codecs' BlockCompressorStream itself: a stream
  known to fit one block gets a block of its length and a buffer of its compression bound, which writes the same
  bytes; IFile.Writer sizes its write buffer to the stream too
- The spill passes each partition's raw length bound from the buffers' record and byte counts, the final merge the
  sum of the spilled raw lengths; other codecs and writers keep codec.createOutputStream
- JMH, one spill of 1,009 partitions (Snappy, 22 B / 2.2 KB / 22 KB per partition): -76 % / -48 % / -10 %, 336 MB
  allocated -> 5 / 11 / 62 MB; LZ4 -59 % / -41 % / -12 %
@tez-yetus

Copy link
Copy Markdown

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 16m 7s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 3 new or modified test files.
_ master Compile Tests _
-1 ❌ mvninstall 0m 19s /branch-mvninstall-root.txt root in master failed.
-1 ❌ compile 0m 19s /branch-compile-root.txt root in master failed.
-0 ⚠️ checkstyle 0m 18s /buildtool-branch-checkstyle-tez-runtime-library.txt The patch fails to run checkstyle in tez-runtime-library
-1 ❌ javadoc 0m 9s /branch-javadoc-tez-runtime-library.txt tez-runtime-library in master failed.
-1 ❌ spotbugs 0m 18s /branch-spotbugs-tez-runtime-library.txt tez-runtime-library in master failed.
_ Patch Compile Tests _
-1 ❌ mvninstall 0m 20s /patch-mvninstall-root.txt root in the patch failed.
+1 💚 codespell 0m 11s No new issues.
-1 ❌ compile 0m 19s /patch-compile-root.txt root in the patch failed.
-1 ❌ javac 0m 19s /patch-compile-root.txt root in the patch failed.
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 0m 17s /buildtool-patch-checkstyle-tez-runtime-library.txt The patch fails to run checkstyle in tez-runtime-library
-1 ❌ javadoc 0m 19s /patch-javadoc-tez-runtime-library.txt tez-runtime-library in the patch failed.
-1 ❌ spotbugs 0m 20s /patch-spotbugs-tez-runtime-library.txt tez-runtime-library in the patch failed.
_ Other Tests _
-1 ❌ unit 0m 13s /patch-unit-root.txt root in the patch failed.
+0 🆗 asflicense 0m 20s ASF License check generated no output?
20m 32s
Subsystem Report/Notes
Docker ClientAPI=1.56 ServerAPI=1.56 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-547/1/artifact/out/Dockerfile
Optional Tests dupname compile unit asflicense javac javadoc spotbugs checkstyle codespell detsecrets
uname Linux 3c0a6765eafb 5.15.0-190-generic #200-Ubuntu SMP Fri Aug 7 15:06:04 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality tez-personality.sh
git revision master / b434f5c
Default Java Eclipse Adoptium-21.0.13+9-LTS
Test Results https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-547/1/testReport/
Max. process+thread count 40 (vs. ulimit of 5500)
modules C: tez-runtime-library U: tez-runtime-library
Console output https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-547/1/console
versions git=2.43.0 maven=3.9.15 codespell=2.4.1
Powered by Apache Yetus 0.15.1 https://yetus.apache.org

This message was automatically generated.

- spillInfoList is a ConcurrentLinkedQueue: spill threads add without a lock, and the loops that run after close()
  waits for the spills iterate without one
- Tests: drop the checks that repeat the byte equality with the unsized writer or stream (read-backs, decompression,
  the stream buffer read by reflection)
- lz4-java at Hadoop's version in test scope, so the Lz4Codec cases run instead of skipping
@tez-yetus

Copy link
Copy Markdown

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 9m 27s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+0 🆗 xmllint 0m 0s xmllint was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 3 new or modified test files.
_ master Compile Tests _
+0 🆗 mvndep 0m 43s Maven dependency ordering for branch
+1 💚 mvninstall 4m 32s master passed
+1 💚 compile 4m 6s master passed
+1 💚 checkstyle 1m 40s master passed
+1 💚 javadoc 2m 10s master passed
+0 🆗 spotbugs 1m 15s tez-runtime-library in master has 143 extant spotbugs warnings.
+0 🆗 spotbugs 6m 37s root in master has 1054 extant spotbugs warnings.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 15s Maven dependency ordering for patch
+1 💚 mvninstall 3m 49s the patch passed
+1 💚 codespell 1m 36s No new issues.
+1 💚 compile 4m 5s the patch passed
-1 ❌ javac 4m 5s /results-compile-javac-root.txt root generated 5 new + 705 unchanged - 1 fixed = 710 total (was 706)
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 0m 35s /results-checkstyle-tez-runtime-library.txt tez-runtime-library: The patch generated 2 new + 243 unchanged - 0 fixed = 245 total (was 243)
-0 ⚠️ checkstyle 1m 2s /results-checkstyle-root.txt root: The patch generated 2 new + 243 unchanged - 0 fixed = 245 total (was 243)
+1 💚 javadoc 2m 8s the patch passed
-1 ❌ spotbugs 1m 27s /new-spotbugs-tez-runtime-library.html tez-runtime-library generated 3 new + 138 unchanged - 5 fixed = 141 total (was 143)
-1 ❌ spotbugs 7m 2s /new-spotbugs-root.html root generated 3 new + 1049 unchanged - 5 fixed = 1052 total (was 1054)
_ Other Tests _
-1 ❌ unit 76m 39s /patch-unit-root.txt root in the patch passed.
+1 💚 asflicense 1m 24s The patch does not generate ASF License warnings.
133m 34s
Reason Tests
SpotBugs module:tez-runtime-library
Shared primitive variable "compressOutput" in one thread may not yield the value of the most recent write from another thread At IFile.java:one thread may not yield the value of the most recent write from another thread At IFile.java:[line 389]
Exception thrown in class org.apache.tez.runtime.library.common.sort.impl.IFile$Writer at new org.apache.tez.runtime.library.common.sort.impl.IFile$Writer(Serialization, Serialization, FSDataOutputStream, Class, Class, CompressionCodec, int, long, TezCounter, TezCounter) will leave the constructor. The object under construction remains partially initialized and may be vulnerable to Finalizer attacks. At IFile.java:at new org.apache.tez.runtime.library.common.sort.impl.IFile$Writer(Serialization, Serialization, FSDataOutputStream, Class, Class, CompressionCodec, int, long, TezCounter, TezCounter) will leave the constructor. The object under construction remains partially initialized and may be vulnerable to Finalizer attacks. At IFile.java:[line 348]
Exception thrown in class org.apache.tez.runtime.library.common.sort.impl.IFile$Writer at new org.apache.tez.runtime.library.common.sort.impl.IFile$Writer(Serialization, Serialization, FSDataOutputStream, Class, Class, CompressionCodec, int, long, TezCounter, TezCounter, boolean) will leave the constructor. The object under construction remains partially initialized and may be vulnerable to Finalizer attacks. At IFile.java:at new org.apache.tez.runtime.library.common.sort.impl.IFile$Writer(Serialization, Serialization, FSDataOutputStream, Class, Class, CompressionCodec, int, long, TezCounter, TezCounter, boolean) will leave the constructor. The object under construction remains partially initialized and may be vulnerable to Finalizer attacks. At IFile.java:[line 361]
SpotBugs module:root
Shared primitive variable "compressOutput" in one thread may not yield the value of the most recent write from another thread At IFile.java:one thread may not yield the value of the most recent write from another thread At IFile.java:[line 389]
Exception thrown in class org.apache.tez.runtime.library.common.sort.impl.IFile$Writer at new org.apache.tez.runtime.library.common.sort.impl.IFile$Writer(Serialization, Serialization, FSDataOutputStream, Class, Class, CompressionCodec, int, long, TezCounter, TezCounter) will leave the constructor. The object under construction remains partially initialized and may be vulnerable to Finalizer attacks. At IFile.java:at new org.apache.tez.runtime.library.common.sort.impl.IFile$Writer(Serialization, Serialization, FSDataOutputStream, Class, Class, CompressionCodec, int, long, TezCounter, TezCounter) will leave the constructor. The object under construction remains partially initialized and may be vulnerable to Finalizer attacks. At IFile.java:[line 348]
Exception thrown in class org.apache.tez.runtime.library.common.sort.impl.IFile$Writer at new org.apache.tez.runtime.library.common.sort.impl.IFile$Writer(Serialization, Serialization, FSDataOutputStream, Class, Class, CompressionCodec, int, long, TezCounter, TezCounter, boolean) will leave the constructor. The object under construction remains partially initialized and may be vulnerable to Finalizer attacks. At IFile.java:at new org.apache.tez.runtime.library.common.sort.impl.IFile$Writer(Serialization, Serialization, FSDataOutputStream, Class, Class, CompressionCodec, int, long, TezCounter, TezCounter, boolean) will leave the constructor. The object under construction remains partially initialized and may be vulnerable to Finalizer attacks. At IFile.java:[line 361]
Failed junit tests tez.test.TestSecureShuffle
tez.auxservices.TestShuffleHandler
tez.auxservices.TestShuffleHandlerJobs
tez.runtime.library.common.shuffle.orderedgrouped.TestFetcher
tez.http.TestHttpConnection
Subsystem Report/Notes
Docker ClientAPI=1.57 ServerAPI=1.57 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-547/2/artifact/out/Dockerfile
Optional Tests dupname compile unit asflicense javac javadoc codespell detsecrets xmllint spotbugs checkstyle
uname Linux 375bf57f8139 5.15.0-190-generic #200-Ubuntu SMP Fri Aug 7 15:06:04 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality tez-personality.sh
git revision master / 6441bd5
Default Java Eclipse Adoptium-21.0.13+9-LTS
Test Results https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-547/2/testReport/
Max. process+thread count 1410 (vs. ulimit of 5500)
modules C: tez-runtime-library . U: .
Console output https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-547/2/console
versions git=2.43.0 maven=3.9.15 spotbugs=4.9.3 codespell=2.4.1
Powered by Apache Yetus 0.15.1 https://yetus.apache.org

This message was automatically generated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants