Repository navigation
B3: fix the image-browser segfault and the 99% hang after a worker crash - #1184
Merged
Merged
Conversation
…t worker ends the session Repin pyqt-reactive 0.3.30 (10ba5b4). Its flash coordinator kept registrations for widgets outside a flash window forever and called window() on a scroll-area viewport after Qt had freed it. sip had not marked that wrapper deleted, so the call segfaulted in test_image_browser_streaming_config.py. pytest.ini: --max-worker-restart=0. After a crash, xdist's loadfile scheduler requeues every file the dead worker had already finished. It hands the replacement worker one of those files per completed test, so the workers run out of tests with files still queued, and the controller waits forever at 99%. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Symptom
pytest tests/unit tests/core -n 2 --dist loadfilesegfaulted an xdist worker at about 96%, intest_image_browser_streaming_config.py::test_streaming_enableable_is_title_only_and_targets_its_config_group. The session then hung at 99% for hours.Cause 1: the segfault (pyqt-reactive flash coordinator)
_GlobalFlashCoordinatorkeeps a flash registration "pending" while its widget is outside aQMainWindow/QDialog. Nothing removed these registrations except a successful attach, so they piled up for the whole process. In this reproducer, 300+ were pending by the image-browser file. Every local flash calledwidget.isVisible()andwidget.window()on each one and caughtRuntimeErrorto skip deleted widgets.test_step_code_widget_roundtrip.pyleaves a registration whose widget is aReflowingVerticalScrollAreaviewport. Qt creates that viewport, and the code reaches it throughparentWidget(). Its owner is deleted later, during the image-browser test's firstQTest.qWait. I attached a watcher to the viewport: itsdestroyedsignal fired, butsip.isdeleted(viewport)stayedFalse. The nextParameterFormManagerflush therefore calledwindow()on freed memory, and the crash stack goes through_process_pending_registrations→get_for_window.Fix (OpenHCSDev/PyQT-reactive#19, pinned here as 0.3.30 / 10ba5b4): a pending registration now ends when Qt destroys its widget or any widget its element closes over. It uses the
_DestructionCallbackmechanism the mixin already uses for its own records. TheRuntimeErrorprobe is gone.Cause 2: the 99% hang (pytest-xdist loadfile scheduler)
When a worker crashes,
LoadScopeScheduling.remove_nodeputs every file assigned to the dead worker back in the work queue, including files it had already finished. The replacement worker is handed one queued file per test it completes, and an already-finished file contains no tests. With ~470 finished files queued, the workers run out of tests while files are still queued.tests_finishedthen never becomes true, and the controller waits forever without sending shutdown. TheOSError: cannot send (already closed?)lines at the end of the logs come from the workers'pytest_sessionfinishrunning only when the session was killed. Our conftest is not involved. A 41-file synthetic suite whose last file segfaults hangs the same way: it hittimeout 120with xdist 3.8.0.Fix:
pytest.iniadds--max-worker-restart=0, so a crashed worker ends the session with the crashed test reported. The same synthetic suite now exits 1 in under a second (worker 'gw0' crashed while running ...).Reproduction
Without the fix,
tests/unit/test_napari_reset_transitions.py tests/unit/pyqt_gui/test_step_code_widget_roundtrip.py tests/unit/pyqt_gui/test_image_browser_streaming_config.pysegfaults every time at the same frame. Without the napari file it does not crash. That three-file set came from bisecting the crashed worker's 48 GUI and viewer files, and the full 48-file sequence also segfaulted on both of its unfixed runs.Verification (QT_QPA_PLATFORM=offscreen, DISPLAY unset, worktree basetemp)
test_async_failure_finalizes_before_pipeline_editor_bridge_presentation(5 s poller timeout under load). That test passed 3/3 alone and passed in thetests/unit/pyqt_guirun.test_image_browser_streaming_config.py: 20/20 serially, 20/20 with-n 2.-n 2): 309 passed. 11 failed and 7 errored, all in the existing main failure list.tests/unit/pyqt_gui -n 2 --dist loadfile: no crash. 953 passed, 10 failed, all in the existing main failure list.tests -n 2: 610 passed.🤖 Generated with Claude Code