Repository navigation
fix: queue HTTP serving connection bursts before accept - #289
Closed
gspark-etri wants to merge 8 commits into
Closed
gspark-etri wants to merge 8 commits into
gspark-etri wants to merge 8 commits into
Conversation
Collaborator
Author
|
Landed on optimus-rm via #290. Closing; the branch is archived locally (experiment records and evidence included). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background
Concurrent Qwen3-4B HTTP sweeps reset connections while their servers remained alive. Python 3.10's default listen backlog is 5. A real-socket regression reproduces failed connections when 16 clients arrive before the accept loop runs.
Changes
Set the shared serving listener backlog to 128 before bind/listen. Add a regression that pauses accept while 16 clients connect. Preserve failed runs, independently audited retries, all timing conditions and numerical limitations in the experiment records.
This PR is stacked on
perf/265-rngd-stream(PR266). Relative to that base, the runtime change is confined to the HTTP listener; engine admission, timeouts and KV APIs are unchanged.Verification
The real-socket regression fails with TimeoutError before the change and passes afterward. The focused command below passed 15 tests:
Hardware grid: inputs 128/512/2048/8192, outputs 32/128, concurrent clients 1/4/16, one timing batch per condition. Measured base
f52c41305f6bb2ee65d3c0fe8166acbca51dcc72plus listener source SHA9f3aaf0902f7b1958f32be12b1f276384f20cc3e0070571af10bb7bce148c1f0.All completed candidate runs above have zero host listen-drop/overflow increment. The original GPU reset at 2048/128/c16 had +36 host listen drops; the original native ATOM SSE reset at 512/128/c16 had +82 and zero cgroup OOM counters. Both failed results remain failures; retries do not replace them.
GPU geometry: one A10080GB decoder plus one RNGD research producer, BF16, four decoder slots, context9216, page/chunk256, producerwidth8192. ATOM geometry: one execution lane, B1/k3/context9216, one additional RNGD for PD. Concurrent ATOM clients queue; this is not hardware B16.
Full raw/config/source fingerprints, package versions, every timing row and reproduction commands are under
opt_prime/examples/pd/http_matrix/experiments/2026-10-30-http-backlog267.mdandevidence267/(qwen4-full-pd-cache-summary.json,gemma-full-pd-cache-summary.json,qwen4-atom-full-summary.json). Evidence auditors check actual outputs, coverage, timing arithmetic and counters. Sensitive-data and whitespace checks pass. No CI checks were reported on the stacked base; local verification is not CI certification.Compatibility and limits
Successful retries support the backlog mitigation but do not prove it was the sole reset cause or that all resets are eliminated. Listen counters are host-wide. Failed originals do not provide complete timing controls, so this PR claims no backlog throughput speedup.
Transfer-mode/cache timings are single ordered runs. Cache benefits are not universal: streamed cache-on improves only 14/24 Qwen4 and 10/24 Gemma conditions. For Qwen4 ATOM at 8192/128/c4, native/whole/stream delivers 23.07/15.04/16.52 tok/s: PD remains slower than native in that condition. Native/PD pre-EOS differences at 512/128 and 2048/128 already exist in historical cache-off references; their numerical cause remains unresolved. No broad engine equivalence or answer-quality claim.
Producer full-prefill and CPU-copy work remain despite decoder cache hits. Logical KV counters do not establish physical PCIe savings or DMA overlap. KONI GPU and TP2 tests are still running; keep Draft pending remaining verification. No merge performed.
Related Issue
Related to #267 and #201; does not close either. Depends on PR266.