Skip to content

TEZ-4763: UnorderedPartitionedKVWriter: concurrent spills can lose a spill event or have a spill rejected - #550

Open
deniskuzZ wants to merge 1 commit into
apache:masterfrom
deniskuzZ:TEZ-4763
Open

deniskuzZ wants to merge 1 commit into
apache:masterfrom
deniskuzZ:TEZ-4763

Conversation

@deniskuzZ

Copy link
Copy Markdown
Member

With tez.runtime.enable.final-merge.in.output=false (and pipelined shuffle off), each spill's callback
(SpillCallback.onSuccess -> mayBeSendEventsForSpill) adds the spill's CompositeDataMovementEvent to finalEvents
on its spill thread, and close() returns that list. When spills finish at the same time:

  • finalEvents is a plain LinkedList, and two addAlls at once can corrupt it. If a node is lost but the size is
    kept, close() returns a null and TezEvent fails the attempt ("Unknown event"). If the node and the size
    increment are both lost, the list is one event short and the consumer waits forever for that spill: a silent hang.
  • The spill pool hands its tasks over through a SynchronousQueue. A callback releases its availableSlots permit while
    it still runs on its pool thread, and the maxThreads - 1 permits leave room for one such thread, not several. When
    every thread is busy, scheduleSpill gets a RejectedExecutionException and the task fails.
  • The pool's core thread never times out, so a writer that is never closed keeps it: we counted 125 idle
    UnorderedOutSpiller threads, 1 to 2.6 hours old, in an LLAP daemon.

Two spills in flight are routine: in Hive on Tez (LLAP), TPC-DS 10 TB, one q78 run had 218 of 1,377 writers of a map
vertex with two pool spills in flight, as close() schedules a spill of the filled buffers while a mid-task one runs.
A corruption needs two appends to overlap, so it is rare per writer.

The patch:

  • makes finalEvents a synchronized list. close() already waits for every pending spill under spillLock before it
    reads the list.
  • gives the pool core = max = maxThreads, a LinkedBlockingQueue and allowCoreThreadTimeOut(true).
    availableSlots still bounds the spills in flight; the queue only absorbs the moment between a callback releasing
    its permit and its thread returning to the pool. Idle threads exit after 60 s.

Test: testConcurrentSpillsWithFinalMergeDisabled runs 40 writers, 4 at a time, each with 64 buffers of 4 KiB (32
spill threads), 16 partitions and 200,000 records, final merge off. It checks that no spill is rejected, that close()
returns no null, and that it returns one spill event per spill. It is a stress test, so it can pass on a broken
writer: on master it failed 6 of 6 runs (5 RejectedExecutionException, 1 null event), and with other sizes it also
caught the silent case (spill events ==> expected: <316> but was: <315>). With the patch it passed 8 of 8 runs, ~28 s
each (14-core laptop). TestUnorderedPartitionedKVWriter: 371 run, 0 failures; tez-runtime-library: 824 run, 0 failures (2 async-http tests error with a netty NoSuchFieldError in my local build, on master too).

…spill event or have a spill rejected

- finalEvents is a synchronized list: with final merge off, spill callbacks add their events from the spill threads
- The spill pool queues instead of rejecting: core = max = maxThreads, LinkedBlockingQueue, idle threads time out
- testConcurrentSpillsWithFinalMergeDisabled: 40 writers, 4 at a time, 64 buffers of 4 KiB each, final merge off
@tez-yetus

Copy link
Copy Markdown

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 4m 25s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 1 new or modified test files.
_ master Compile Tests _
+1 💚 mvninstall 5m 6s master passed
+1 💚 compile 4m 9s master passed
+1 💚 checkstyle 0m 40s master passed
+1 💚 javadoc 0m 40s master passed
+0 🆗 spotbugs 1m 17s tez-runtime-library in master has 143 extant spotbugs warnings.
_ Patch Compile Tests _
+1 💚 mvninstall 3m 48s the patch passed
+1 💚 codespell 1m 37s No new issues.
+1 💚 compile 4m 4s the patch passed
-1 ❌ javac 4m 4s /results-compile-javac-root.txt root generated 1 new + 706 unchanged - 0 fixed = 707 total (was 706)
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 0m 33s the patch passed
+1 💚 javadoc 0m 37s the patch passed
+1 💚 spotbugs 1m 25s the patch passed
_ Other Tests _
-1 ❌ unit 74m 56s /patch-unit-root.txt root in the patch passed.
+1 💚 asflicense 0m 32s The patch does not generate ASF License warnings.
105m 34s
Reason Tests
Failed junit tests tez.test.TestSecureShuffle
tez.auxservices.TestShuffleHandler
tez.auxservices.TestShuffleHandlerJobs
tez.http.TestHttpConnection
tez.runtime.library.common.shuffle.orderedgrouped.TestFetcher
tez.dag.app.TestSpeculation
Subsystem Report/Notes
Docker ClientAPI=1.57 ServerAPI=1.57 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-550/1/artifact/out/Dockerfile
Optional Tests dupname compile unit asflicense javac javadoc spotbugs checkstyle codespell detsecrets
uname Linux 1fa1c8a05f5a 5.15.0-190-generic #200-Ubuntu SMP Fri Aug 7 15:06:04 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality tez-personality.sh
git revision master / 6441bd5
Default Java Eclipse Adoptium-21.0.13+9-LTS
Test Results https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-550/1/testReport/
Max. process+thread count 1860 (vs. ulimit of 5500)
modules C: tez-runtime-library U: tez-runtime-library
Console output https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-550/1/console
versions git=2.43.0 maven=3.9.15 spotbugs=4.9.3 codespell=2.4.1
Powered by Apache Yetus 0.15.1 https://yetus.apache.org

This message was automatically generated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants