Skip to content

feat(scheduler): support mixed prefill and decode batching - #624

Open
Simon12345777 wants to merge 3 commits into
InfiniTensor:mainfrom
Simon12345777:feat/mixed-prefill-decode
Open

Simon12345777 wants to merge 3 commits into
InfiniTensor:mainfrom
Simon12345777:feat/mixed-prefill-decode

Conversation

@Simon12345777

Copy link
Copy Markdown

Add opt-in decode-first mixed batching with chunked prefill support and scheduling regression tests.

PR description

Summary

  • Add opt-in enable_mixed_batching / --enable-mixed-batching for the standard local paged text inference path.
  • In python/infinilm/llm/scheduler.py, select running decode requests first, then admit whole or chunked prefills using the remaining request slots and token budget.
  • Add SchedulerOutput.num_decode_requests and use it in python/infinilm/processors/basic_llm_processor.py to pack each row according to its phase.
  • Reserve KV headroom for decode requests already selected for the current batch, and retry whole prefills deferred by decode token usage.
  • Wire the option through the Python, server, and example entry points and add scheduler regression coverage.

Motivation

Separate prefill and decode batches can require two forwards when a short prompt arrives during ongoing generation. With spare request slots, the paged attention input can contain both kinds of work. This option lets existing generation advance during a prefill forward and reduces the number of forwards in those workloads.

The scheduler charges one token per decode row and the actual scheduled token count per prefill row. Chunked prefills use page-aligned intermediate slices so their KV state can be resumed through the existing allocation path. Whole-prefill mode retains whole prompts and gives a prompt deferred by decode token usage a retry before selecting more decode work.

Type of Change

  • feat — new configurable scheduling behavior
  • fix — bug fix
  • perf — performance improvement without behavioral change
  • refactor — code restructuring without behavior change
  • test — tests only
  • docs — documentation only
  • build / ci — build system or CI configuration
  • chore — other changes
  • Breaking change

Test Results of Involved Models on Supported Platforms (Please attach screenshots)

Platform / model Validation Result
NVIDIA A100 PCIe 80GB, local FM9G 8B checkpoint python -m unittest discover -s test/runtime -p test_chunked_prefill_sched.py -v 26 tests passed
Same platform and checkpoint Service benchmark, 64 fixed BurstGPT requests × 2 policies × 4 runs 512 requests completed, 0 failures
Same platform and checkpoint Controlled engine workloads Matched token counts, completed requests, cleared active owners and KV references
Other supported devices / additional models Platform regression Pending access and validation on those platforms

Added tests cover mixed input packing and resumed chunk budgets, whole-prefill retry, and KV admission after selecting decode rows. The connector compatibility test also checks that mixing is disabled for that path.

The recorded validation used focused scheduler tests, the service benchmark harness, and controlled engine runs. A fresh build and the standard single-request, offline-performance, and sanity scripts have not been rerun for this final submission. Screenshots should be attached in this section before requesting review.

Benchmark / Performance Impact

Hardware and model configuration

  • One NVIDIA A100 PCIe 80GB, local 9g_8b_thinking FM9G 8B checkpoint. Its checkpoint configuration declares torch_dtype=bfloat16; the backend uses the checkpoint dtype and default KV precision.
  • flash-attn, paged cache, 192 blocks, block size 256, maximum batch size 8, token budget 4096.
  • chunked_prefill_size=0 in both policies for the comparisons below. CUDA Graph and prefix caching disabled.
  • Four paired runs with alternating policy order. Values below are medians of the measured runs. Gains are the ratio of mixed to separate median throughput minus one.

BurstGPT service benchmark

64 fixed length records sampled without replacement from valid rows using seed 0, concurrency 8, 8 warmup requests per run, uncapped recorded output lengths. Synthetic text approximates the recorded lengths and requests use saturated arrivals. Each measured run processes 41,282 input and 17,305 output tokens. Original trace arrival timestamps are not replayed.

Metric Separate Mixed Value change
Total tokens/s 1566.11 1585.44 +1.23%
Mean TTFT, ms 116.48 122.35 +5.04%
P99 TTFT, ms 516.79 482.54 -6.63%
Mean TPOT, ms 15.59 15.41 -1.13%

The four paired throughput gains are +0.70%, +1.19%, +1.28%, and +1.27%.

Controlled short-output arrivals

Each case runs 16 arrival waves with background contexts of 128 tokens. New requests finish and background requests advance by the same number of output tokens before the next wave. Both policies execute equal token counts. Background seed prefill is excluded from timing. These are direct engine measurements, with identical request-progress observation in both policies. Nine cases were screened once; the five highest gains received four new balanced pairs, excluding screening runs from the table below.

Workload Separate total tokens/s Mixed total tokens/s Gain
1 decode, one 32-input / 4-output arrival per wave 603.3 726.4 +20.39%
4 decodes, one 32-input / 4-output arrival per wave 771.2 919.8 +19.27%
6 decodes, two 32-input / 4-output arrivals per wave 1343.7 1600.8 +19.13%
7 decodes, one 256-input / 4-output arrival per wave 3257.1 3621.8 +11.20%
4 decodes, four 128-input / 4-output arrivals per wave 5242.1 5642.0 +7.63%

All 20 confirmation pairs improved throughput. Independent batch-count runs changed each case from 16 prefill + 64 decode forwards to 16 mixed + 48 decode forwards. Performance runs excluded the batch-count wrapper.

Notes for Reviewers

  • Review the decode-row boundary, shared token-budget accounting, page-aligned intermediate chunks, whole-prefill retry ownership, and KV reservations for decode requests removed from the running queue.
  • The supported domain is standard paged text inference. Compatibility checks restrict MLA, speculative execution, Mamba cache, and NSA owner paths. The scheduler disables mixing when a KV connector is active. Performance validation currently covers the A100 path.

CI / ChatOps

Manual CI and broader platform validation are pending. Trigger the CI workflow for the final head branch or request a maintainer /retest before completing review.


Checklist

Every contributor must verify every item below before requesting
review. Tick each box only after the check has actually been performed —
do not tick speculatively. If an item truly does not apply, replace the
checkbox with N/A and briefly explain why in an inline comment.

Title, Branch, and Commits

  • PR title follows Conventional Commits (e.g. feat(nvidia): …, fix(cuda/gemm): …).
  • Branch name follows <type>/xxx-yyyy-zzzz where <type> matches the PR title's Conventional Commits type and words are joined with hyphens (see CONTRIBUTING.md §Branches).
  • Each commit message follows Conventional Commits.
  • Small PR is a single squashable commit; or, for a large PR, every commit is meaningful, well-formed, and independently reviewable (see CONTRIBUTING.md §Pull Requests).
  • No stray merge commits from main — the branch is rebased cleanly on top of the current main.
  • No fixup! / squash! / wip commits remain.
  • Existing PR/branch/commit that followed the legacy issue format.

Scope and Design

  • Changes are minimal — nothing unrelated to the stated motivation was added (CONTRIBUTING.md §Code/General).
  • No dead code, commented-out blocks, debug prints, printf/std::cout/print(...) left behind, or TODO without an owner and issue link.
  • No unrelated formatting churn that would obscure the diff.
  • Public API changes (if any) are intentional, documented, and reflected in affected callers/tests.

General Code Hygiene (applies to all languages)

  • The code is self-explanatory; comments were added only where the why is non-obvious (CONTRIBUTING.md §Code/General).
  • Every modified or added file ends with a single trailing newline (CONTRIBUTING.md §Code/General).
  • No trailing whitespace, tab/space mixing, or stray BOMs.
  • Identifiers in comments and error messages are wrapped in backticks (e.g. the `seqlens_k` tensor) (CONTRIBUTING.md §Code/General).
  • All comments and error messages are in English (CONTRIBUTING.md §Code/General).
  • Comments and error messages are complete sentences — capitalized first letter, terminal punctuation — unless the language/framework convention says otherwise (CONTRIBUTING.md §Code/General; §Python).

C++ Specific

N/A — the diff contains Python changes.

Python Specific (if Python files changed)

  • Code is PEP 8 compliant.
  • Comments are complete English sentences, starting with a capital letter and ending with punctuation; Markdown backticks are used for code references (CONTRIBUTING.md §Python).
  • Docstrings (if any) follow PEP 257 (CONTRIBUTING.md §Python).
  • Changed files are formatted by scripts/format.py.
  • No changes/reference to python/infinilm/auto_config.py.

Testing

  • For any platform that could not be tested, an explicit reason is given in the table and a reviewer with access has been tagged.
  • Passed single request test (examples/test_infer.py), or specify the reason for skipping.
  • Passed offline performance test (examples/bench.py), or specify the reason for skipping.
  • Passed sanity test (test/bench/test_benchmark.py), or specify the reason for skipping.
  • Passed service test (python/infinilm/server/inference_server.py + scripts/test_perf.py), or specify the reason for skipping.

Build, CI, and Tooling

  • The project builds cleanly from a fresh directory on at least one affected platform.
  • CI has been triggered manually (Actions → CI on this branch), or /retest was requested.

Documentation

  • README.md, CONTRIBUTING.md, or inline docs updated when behavior, build flags, or developer workflow changed.
  • Any user-visible breaking change is called out explicitly under "Motivation" and in the commit/PR title with a ! or BREAKING CHANGE: footer.

Security and Safety

  • No secrets, access tokens, internal URLs, customer data, or personal hardware identifiers have been committed.
  • Third-party code is license-compatible and attributed.
  • No unsafe pointer arithmetic, uninitialized reads, or missing bounds checks were introduced.

@Simon12345777
Simon12345777 requested a review from a team November 17, 2026 20:29
@wooway777
wooway777 requested a review from ma-hang November 18, 2026 02:05
Comment thread python/infinilm/llm/scheduler.py Resolved
Add opt-in decode-first mixed batching with chunked prefill support and scheduling regression tests.
@Simon12345777
Simon12345777 force-pushed the feat/mixed-prefill-decode branch from ca9f461 to f4473fb Compare November 19, 2026 21:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants