Skip to content

serve: queue HTTP connection bursts before accept (#267) - #290

Merged
gspark-etri merged 1 commit into
optimus-rmfrom
fix/267-listen-backlog
Nov 3, 2026
Merged

gspark-etri merged 1 commit into
optimus-rmfrom
fix/267-listen-backlog

Conversation

@gspark-etri

Copy link
Copy Markdown
Collaborator

Background

During concurrent Qwen3-4B HTTP sweeps (clients=16), the serving process stayed alive but clients got connection resets. serve_stage uses ThreadingHTTPServer, whose default listen backlog on Python 3.10 is 5. The original failing runs showed host listen-drop counters rising (+36 on the GPU route at 2048/128/c16, +82 on the ATOM SSE route at 512/128/c16).

Changes

  • accel/serve/llmd.py: the serving listener subclasses ThreadingHTTPServer with request_queue_size = 128.
  • tests/test_http_listener_backlog.py: holds the accept thread back and opens 16 real TCP connections to it.

This carries the code change from #289 onto optimus-rm. The experiment evidence in #289 (JSON summaries, experiment record) stays out of the repository.

Verification

OMP_NUM_THREADS=2 TORCH_DEVICE_BACKEND_AUTOLOAD=0 PYTHONPATH=opt_prime python3 -m pytest -q \
  opt_prime/tests/test_http_listener_backlog.py opt_prime/tests/test_pd_http_cancel.py \
  opt_prime/tests/test_pd_stage_metrics.py opt_prime/tests/test_pd_llmd.py
  • With the change: 15 passed.
  • Without the llmd.py change: the new test fails (connection timeout).
  • From fix: queue HTTP serving connection bursts before accept #289 (hardware, measured on base f52c413 + this change): 264 conditions over Qwen3-4B RNGD→GPU, Gemma3-1B RNGD→GPU and Qwen3-4B ATOM routes, with zero host listen-drop increments and outputs exact-matched within each route.

Compatibility and limits

Only the listen backlog changes; admission, timeouts and KV APIs do not. The retries support the backlog as the cause but don't prove it was the only cause of resets. No throughput claim.

Related Issue

Related to #267 (supersedes the code part of #289).

🤖 Generated with Claude Code

Python 3.10's ThreadingHTTPServer listens with backlog 5, so 16 clients
arriving before the accept loop runs get connection resets.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@gspark-etri
gspark-etri merged commit 52ff463 into optimus-rm Nov 3, 2026
1 check passed
@gspark-etri
gspark-etri deleted the fix/267-listen-backlog branch November 3, 2026 17:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant