Skip to content

Don't let a slow queue feeder make the runner skip tests - #263

Open
fcostaoliveira wants to merge 1 commit into
RedisLabsModules:masterfrom
fcostaoliveira:fix/serial-runner-queue-race
Open

fcostaoliveira wants to merge 1 commit into
RedisLabsModules:masterfrom
fcostaoliveira:fix/serial-runner-queue-race

Conversation

@fcostaoliveira

@fcostaoliveira fcostaoliveira commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

The serial runner puts every test on a multiprocessing.Queue and reads them back with get(timeout=0.1), breaking on Empty. put() is flushed by a background feeder thread. If that thread is starved for more than 100 ms on a loaded machine, the runner runs no tests, prints Total Tests Run: 0 and exits 0, which is a silent pass. This was introduced in #148.

Fix: the serial path now iterates the test list directly. In the parallel path, workers do a blocking get() and stop on one None sentinel per worker, instead of relying on a timeout.

Repro: q = Queue(); q.put(1, block=False); q.get(timeout=0.1) raised Empty in 2 of 30 runs at nice 19 under 28 busy loops, and in 0 of 200 idle. Under the same load, RLTest once reported Total Tests Run: 0 with exit 0. The new tests/unit/test_serial_runner_queue.py fails on master (no tests run) and passes with this change.

🤖 Generated with Claude Code


Note

Medium Risk
Changes core test scheduling for serial and parallel modes; incorrect sentinel or iteration logic could skip or hang tests, though behavior is covered by new unit tests.

Overview
Fixes a silent false pass where serial RLTest runs could report Total Tests Run: 0 and exit 0 when a multiprocessing.Queue feeder thread was slow to flush.

Serial execution no longer round-trips tests through a queue with get(timeout=0.1) and an early break on Empty; it materializes list(self.loader) and runs tests by index on the main thread.

Parallel workers use blocking get() and exit on a per-worker None sentinel (tests + [None] * parallelism) instead of timing out on an apparently empty queue before the feeder has flushed.

Adds tests/unit/test_serial_runner_queue.py to lock in serial behavior under a mocked starved queue and to assert all 20 tests run under parallelism 3.

Reviewed by Cursor Bugbot for commit 687bee2. Bugbot is set up for automated code reviews on this repo. Configure here.

The serial runner put every test on a multiprocessing.Queue and read it
back with get(timeout=0.1), stopping on Empty. Queue.put() is flushed by
a background feeder thread, so when that thread was starved for 100 ms
the runner ran zero tests and exited 0.

Iterate the test list directly in the serial path. In the parallel path,
end the queue with one None sentinel per worker and use a blocking get().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fcostaoliveira added a commit to redis/memtier_benchmark that referenced this pull request Oct 26, 2026
RLTest's serial runner read tests from a multiprocessing.Queue with
get(timeout=0.1) and stopped on a timeout, so under load it could run no
tests and exit 0. Pin the RLTest fork to a6d741e, which carries
RedisLabsModules/RLTest#263 (iterate the test list; parallel workers
stop on an end marker), and make run_tests.sh fail when RLTest prints no
"Total Tests Run" summary or reports 0, keeping RLTest's own exit code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant