Repository navigation
feat(scheduler): support mixed prefill and decode batching - #624
Open
Simon12345777 wants to merge 3 commits into
Open
Simon12345777 wants to merge 3 commits into
Simon12345777 wants to merge 3 commits into
Conversation
ma-hang
reviewed
Nov 19, 2026
Add opt-in decode-first mixed batching with chunked prefill support and scheduling regression tests.
Simon12345777
force-pushed
the
feat/mixed-prefill-decode
branch
from
November 19, 2026 21:13
ca9f461 to
f4473fb
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add opt-in decode-first mixed batching with chunked prefill support and scheduling regression tests.
PR description
Summary
enable_mixed_batching/--enable-mixed-batchingfor the standard local paged text inference path.python/infinilm/llm/scheduler.py, select running decode requests first, then admit whole or chunked prefills using the remaining request slots and token budget.SchedulerOutput.num_decode_requestsand use it inpython/infinilm/processors/basic_llm_processor.pyto pack each row according to its phase.Motivation
Separate prefill and decode batches can require two forwards when a short prompt arrives during ongoing generation. With spare request slots, the paged attention input can contain both kinds of work. This option lets existing generation advance during a prefill forward and reduces the number of forwards in those workloads.
The scheduler charges one token per decode row and the actual scheduled token count per prefill row. Chunked prefills use page-aligned intermediate slices so their KV state can be resumed through the existing allocation path. Whole-prefill mode retains whole prompts and gives a prompt deferred by decode token usage a retry before selecting more decode work.
Type of Change
feat— new configurable scheduling behaviorfix— bug fixperf— performance improvement without behavioral changerefactor— code restructuring without behavior changetest— tests onlydocs— documentation onlybuild/ci— build system or CI configurationchore— other changesTest Results of Involved Models on Supported Platforms (Please attach screenshots)
python -m unittest discover -s test/runtime -p test_chunked_prefill_sched.py -vAdded tests cover mixed input packing and resumed chunk budgets, whole-prefill retry, and KV admission after selecting decode rows. The connector compatibility test also checks that mixing is disabled for that path.
The recorded validation used focused scheduler tests, the service benchmark harness, and controlled engine runs. A fresh build and the standard single-request, offline-performance, and sanity scripts have not been rerun for this final submission. Screenshots should be attached in this section before requesting review.
Benchmark / Performance Impact
Hardware and model configuration
9g_8b_thinkingFM9G 8B checkpoint. Its checkpoint configuration declarestorch_dtype=bfloat16; the backend uses the checkpoint dtype and default KV precision.flash-attn, paged cache, 192 blocks, block size 256, maximum batch size 8, token budget 4096.chunked_prefill_size=0in both policies for the comparisons below. CUDA Graph and prefix caching disabled.BurstGPT service benchmark
64 fixed length records sampled without replacement from valid rows using seed 0, concurrency 8, 8 warmup requests per run, uncapped recorded output lengths. Synthetic text approximates the recorded lengths and requests use saturated arrivals. Each measured run processes 41,282 input and 17,305 output tokens. Original trace arrival timestamps are not replayed.
The four paired throughput gains are +0.70%, +1.19%, +1.28%, and +1.27%.
Controlled short-output arrivals
Each case runs 16 arrival waves with background contexts of 128 tokens. New requests finish and background requests advance by the same number of output tokens before the next wave. Both policies execute equal token counts. Background seed prefill is excluded from timing. These are direct engine measurements, with identical request-progress observation in both policies. Nine cases were screened once; the five highest gains received four new balanced pairs, excluding screening runs from the table below.
All 20 confirmation pairs improved throughput. Independent batch-count runs changed each case from 16 prefill + 64 decode forwards to 16 mixed + 48 decode forwards. Performance runs excluded the batch-count wrapper.
Notes for Reviewers
CI / ChatOps
Manual CI and broader platform validation are pending. Trigger the CI workflow for the final head branch or request a maintainer
/retestbefore completing review.Checklist
Title, Branch, and Commits
feat(nvidia): …,fix(cuda/gemm): …).<type>/xxx-yyyy-zzzzwhere<type>matches the PR title's Conventional Commits type and words are joined with hyphens (seeCONTRIBUTING.md§Branches).CONTRIBUTING.md§Pull Requests).main— the branch is rebased cleanly on top of the currentmain.fixup!/squash!/wipcommits remain.Scope and Design
CONTRIBUTING.md§Code/General).printf/std::cout/print(...)left behind, orTODOwithout an owner and issue link.General Code Hygiene (applies to all languages)
CONTRIBUTING.md§Code/General).CONTRIBUTING.md§Code/General).the `seqlens_k` tensor) (CONTRIBUTING.md§Code/General).CONTRIBUTING.md§Code/General).CONTRIBUTING.md§Code/General; §Python).C++ Specific
N/A — the diff contains Python changes.
Python Specific (if Python files changed)
CONTRIBUTING.md§Python).CONTRIBUTING.md§Python).scripts/format.py.python/infinilm/auto_config.py.Testing
examples/test_infer.py), or specify the reason for skipping.examples/bench.py), or specify the reason for skipping.test/bench/test_benchmark.py), or specify the reason for skipping.python/infinilm/server/inference_server.py+scripts/test_perf.py), or specify the reason for skipping.Build, CI, and Tooling
/retestwas requested.Documentation
README.md,CONTRIBUTING.md, or inline docs updated when behavior, build flags, or developer workflow changed.!orBREAKING CHANGE:footer.Security and Safety