Skip to content

fix: queue HTTP serving connection bursts before accept - #289

Closed
gspark-etri wants to merge 8 commits into
perf/265-rngd-streamfrom
fix/267-http-listen-backlog
Closed

gspark-etri wants to merge 8 commits into
perf/265-rngd-streamfrom
fix/267-http-listen-backlog

Conversation

@gspark-etri

@gspark-etri gspark-etri commented Oct 30, 2026 •

Copy link
Copy Markdown
Collaborator

Background

Concurrent Qwen3-4B HTTP sweeps reset connections while their servers remained alive. Python 3.10's default listen backlog is 5. A real-socket regression reproduces failed connections when 16 clients arrive before the accept loop runs.

Changes

Set the shared serving listener backlog to 128 before bind/listen. Add a regression that pauses accept while 16 clients connect. Preserve failed runs, independently audited retries, all timing conditions and numerical limitations in the experiment records.

This PR is stacked on perf/265-rngd-stream (PR266). Relative to that base, the runtime change is confined to the HTTP listener; engine admission, timeouts and KV APIs are unchanged.

Verification

The real-socket regression fails with TimeoutError before the change and passes afterward. The focused command below passed 15 tests:

OMP_NUM_THREADS=2 MKL_NUM_THREADS=2 TORCH_DEVICE_BACKEND_AUTOLOAD=0 PYTHONPATH=opt_prime python -m pytest -q opt_prime/tests/test_http_listener_backlog.py opt_prime/tests/test_pd_http_cancel.py opt_prime/tests/test_pd_stage_metrics.py opt_prime/tests/test_pd_llmd.py

Hardware grid: inputs 128/512/2048/8192, outputs 32/128, concurrent clients 1/4/16, one timing batch per condition. Measured base f52c41305f6bb2ee65d3c0fe8166acbca51dcc72 plus listener source SHA 9f3aaf0902f7b1958f32be12b1f276384f20cc3e0070571af10bb7bce148c1f0.

Completed candidate scope Conditions Responses Output check
Qwen3-4B RNGD→GPU, whole/stream × cache off/on 96 1376 Same-route exact; all four reference sets equal
Gemma3-1B RNGD→GPU, whole/stream × cache off/on 96 1376 Same-route exact; all four reference sets equal
Qwen3-4B ATOM native/whole/stream, cache on 72 1032 Same-route exact; whole/stream 8/8 references equal, native/PD 6/8

All completed candidate runs above have zero host listen-drop/overflow increment. The original GPU reset at 2048/128/c16 had +36 host listen drops; the original native ATOM SSE reset at 512/128/c16 had +82 and zero cgroup OOM counters. Both failed results remain failures; retries do not replace them.

GPU geometry: one A10080GB decoder plus one RNGD research producer, BF16, four decoder slots, context9216, page/chunk256, producerwidth8192. ATOM geometry: one execution lane, B1/k3/context9216, one additional RNGD for PD. Concurrent ATOM clients queue; this is not hardware B16.

Full raw/config/source fingerprints, package versions, every timing row and reproduction commands are under opt_prime/examples/pd/http_matrix/experiments/2026-10-30-http-backlog267.md and evidence267/ (qwen4-full-pd-cache-summary.json, gemma-full-pd-cache-summary.json, qwen4-atom-full-summary.json). Evidence auditors check actual outputs, coverage, timing arithmetic and counters. Sensitive-data and whitespace checks pass. No CI checks were reported on the stacked base; local verification is not CI certification.

Compatibility and limits

Successful retries support the backlog mitigation but do not prove it was the sole reset cause or that all resets are eliminated. Listen counters are host-wide. Failed originals do not provide complete timing controls, so this PR claims no backlog throughput speedup.

Transfer-mode/cache timings are single ordered runs. Cache benefits are not universal: streamed cache-on improves only 14/24 Qwen4 and 10/24 Gemma conditions. For Qwen4 ATOM at 8192/128/c4, native/whole/stream delivers 23.07/15.04/16.52 tok/s: PD remains slower than native in that condition. Native/PD pre-EOS differences at 512/128 and 2048/128 already exist in historical cache-off references; their numerical cause remains unresolved. No broad engine equivalence or answer-quality claim.

Producer full-prefill and CPU-copy work remain despite decoder cache hits. Logical KV counters do not establish physical PCIe savings or DMA overlap. KONI GPU and TP2 tests are still running; keep Draft pending remaining verification. No merge performed.

Related Issue

Related to #267 and #201; does not close either. Depends on PR266.

@gspark-etri

Copy link
Copy Markdown
Collaborator Author

Landed on optimus-rm via #290. Closing; the branch is archived locally (experiment records and evidence included).

@gspark-etri gspark-etri closed this Nov 3, 2026
@gspark-etri
gspark-etri deleted the fix/267-http-listen-backlog branch November 3, 2026 17:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant