Repository navigation
Conversation
…spill event or have a spill rejected - finalEvents is a synchronized list: with final merge off, spill callbacks add their events from the spill threads - The spill pool queues instead of rejecting: core = max = maxThreads, LinkedBlockingQueue, idle threads time out - testConcurrentSpillsWithFinalMergeDisabled: 40 writers, 4 at a time, 64 buffers of 4 KiB each, final merge off
|
💔 -1 overall
This message was automatically generated. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
With
tez.runtime.enable.final-merge.in.output=false(and pipelined shuffle off), each spill's callback(
SpillCallback.onSuccess->mayBeSendEventsForSpill) adds the spill'sCompositeDataMovementEventtofinalEventson its spill thread, and
close()returns that list. When spills finish at the same time:finalEventsis a plainLinkedList, and twoaddAlls at once can corrupt it. If a node is lost but the size iskept,
close()returns anullandTezEventfails the attempt ("Unknown event"). If the node and the sizeincrement are both lost, the list is one event short and the consumer waits forever for that spill: a silent hang.
SynchronousQueue. A callback releases itsavailableSlotspermit whileit still runs on its pool thread, and the
maxThreads - 1permits leave room for one such thread, not several. Whenevery thread is busy,
scheduleSpillgets aRejectedExecutionExceptionand the task fails.UnorderedOutSpillerthreads, 1 to 2.6 hours old, in an LLAP daemon.Two spills in flight are routine: in Hive on Tez (LLAP), TPC-DS 10 TB, one q78 run had 218 of 1,377 writers of a map
vertex with two pool spills in flight, as
close()schedules a spill of the filled buffers while a mid-task one runs.A corruption needs two appends to overlap, so it is rare per writer.
The patch:
finalEventsa synchronized list.close()already waits for every pending spill underspillLockbefore itreads the list.
maxThreads, aLinkedBlockingQueueandallowCoreThreadTimeOut(true).availableSlotsstill bounds the spills in flight; the queue only absorbs the moment between a callback releasingits permit and its thread returning to the pool. Idle threads exit after 60 s.
Test:
testConcurrentSpillsWithFinalMergeDisabledruns 40 writers, 4 at a time, each with 64 buffers of 4 KiB (32spill threads), 16 partitions and 200,000 records, final merge off. It checks that no spill is rejected, that
close()returns no
null, and that it returns one spill event per spill. It is a stress test, so it can pass on a brokenwriter: on master it failed 6 of 6 runs (5
RejectedExecutionException, 1nullevent), and with other sizes it alsocaught the silent case (
spill events ==> expected: <316> but was: <315>). With the patch it passed 8 of 8 runs, ~28 seach (14-core laptop).
TestUnorderedPartitionedKVWriter: 371 run, 0 failures; tez-runtime-library: 824 run, 0 failures (2 async-http tests error with a nettyNoSuchFieldErrorin my local build, on master too).