Skip to content

Fix: Propagate terminal worker failures and stop queued DAG work - #6140

Merged
nicksspirit merged 1 commit into
SQLMesh:mainfrom
mday-io:fix/concurrency-terminal-errors
Oct 15, 2026
Merged

nicksspirit merged 1 commit into
SQLMesh:mainfrom
mday-io:fix/concurrency-terminal-errors

Conversation

@mday-io

@mday-io mday-io commented Oct 13, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Worker cancellation could leave callers waiting indefinitely. Propagate the original exception and stop queued work after a terminal failure.

- unfinished future or queued work after failure
+ propagate failure, stop queued work, drain active workers

Test Plan

  • Reproduced the original hangs and queued-work failure.
  • Tested both error modes, worker draining, exception identity and executor reuse.
  • Ran make style and all make fast-test batches: 2,862 passed.
  • Confirm current-head CI, including Windows.

@mday-io
mday-io marked this pull request as ready for review October 14, 2026 19:27
Signed-off-by: mday-io <mdaytn@gmail.com>
@mday-io
mday-io force-pushed the fix/concurrency-terminal-errors branch from edfb13b to 8be00c3 Compare October 14, 2026 19:34

@nicksspirit nicksspirit left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Took a look. Catching BaseException so a Ctrl-C or SystemExit in a worker actually reaches the caller instead of hanging it, plus having queued nodes bail once fail-fast trips, makes sense. Looks good.

@nicksspirit
nicksspirit merged commit 3bd5972 into SQLMesh:main Oct 15, 2026
34 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants