[h-guo18]

What does this PR do?

Type of change: new example

Adds examples/specdec_aiperf/, a standalone benchmark that measures the acceptance length and throughput of a speculative-decoding model served by vLLM, using AIPerf as the client. It neither imports nor changes examples/specdec_bench. It follows specdec_bench's interface and output format, so the two can be compared.

• bench.py benchmarks a running server once for every temperature × concurrency and writes specdec_bench's files to <save_dir>/t<T>_c<C>/:

  • acceptance_rate.json: Average_AL, Category_AL, Request_AL, the histogram, and conditional and joint rates.
  • timing.json: Output TPS, Output TPS/gpu, time to first token, per-request decode speed.
  • *_responses.jsonl.
  • configuration.json, including container_image, so upload_to_s3.py accepts the run.

  AIPerf's own exports stay in aiperf/.

  Before the first run, bench.py sends one short request and stops if the server returns no per-request spec-decode stats. A point that fails is reported and skipped, the rest of the sweep still runs, and the script exits non-zero.

• Datasets are MT-Bench and Spec-Bench JSONL, and SPEED-Bench splits prepared by specdec_bench's prepare_data.py. Each dataset is one entry in DATASETS, giving its reader, responses file and Request_AL key. --num_requests uses the same stratified selection as specdec_bench, so the prompts match. --runtime_params takes specdec_bench's runtime YAML (sampling_kwargs, chat_template_args).

• tools/launcher/common/specdec_aiperf/run.sh starts vllm serve $HF_MODEL_CKPT with the args before --, adding --per-request-spec-decode-metrics summary, and runs bench.py with the args after it. This is the same convention as common/vllm/query.sh. examples/Qwen/Qwen3-8B/specdec_aiperf_dflash.yaml is an example.

Why. specdec_bench builds an in-process engine for every configuration. For an 816B target that is about 8 minutes of engine start per run, for a benchmark of about 2 minutes. A throughput-vs-concurrency curve also needed a separate vllm bench serve run. Against one long-lived server, each additional temperature, concurrency or dataset costs only its benchmark time, and the same run reports acceptance length together with per-user and per-GPU throughput.

How the numbers stay comparable with specdec_bench:

• Acceptance length comes from vLLM's per-request spec-decode histograms (vLLM v0.29.0 or later). It is counted like specdec_bench's streamed chunks: the prefill token, then 1 + accepted drafts per verify step, pooled over a request's turns and averaged over requests. A unit test checks the acceptance and timing fields against specdec_bench's own metric classes on the same input.
• Sampling (temperature, top_p, top_k) is sent explicitly. Otherwise vllm serve fills unset fields from the model's generation_config.json; for example, one target we tested sets top_k: 64.
• Turn-2 history replays the turn-1 reply with special tokens kept (skip_special_tokens: false), as specdec_bench's tokenizer.decode() does.
• AIPerf 0.13.0 needs a few workarounds:
  • A multi_turn input needs --num-conversations; otherwise AIPerf sends 10 requests, all first turns.
  • HF_HUB_OFFLINE and TRANSFORMERS_OFFLINE break a local --tokenizer directory, so they are dropped for the AIPerf subprocess.
  • DCGM GPU telemetry is turned off.

Known differences:

• vllm serve picks its own default max_num_batched_tokens: 16384 on GB300, versus 2048 for specdec_bench's in-process engine. This changes the KV cache size, time to first token and throughput, but not acceptance length. The README says to set it when comparing throughput.
• When a model stops on a token other than tokenizer.eos_token, specdec_bench replays that stop token's text in turn 2 and this tool does not.

Usage

python3 -m venv aiperf-venv && aiperf-venv/bin/pip install -r examples/specdec_aiperf/requirements.txt

vllm serve <target> --tensor-parallel-size 4 --per-request-spec-decode-metrics summary \
    --speculative-config '{"method": "dflash", "model": "<draft>", "num_speculative_tokens": 7}' &

aiperf-venv/bin/python examples/specdec_aiperf/bench.py --url http://localhost:8000 \
    --mtbench question.jsonl --runtime_params runtime.yaml \
    --temperature 0 1 --concurrency 1 8 32 --num_gpus 4 --save_dir results/mtbench

Testing

Tool comparison on JHB. This used a hand-written harness and motivated this PR. Setup:

• 1 node, 4× GB300, TP4 with expert parallel.
• Target: an internal 816B NVFP4 MoE model. Drafter: a DFlash2 drafter with 7 speculative tokens.
• MT-Bench, 80 questions × 2 turns, 32 concurrent conversations, max 4096 output tokens, t=0 and t=1.

Arms, both on the same engine configuration with token-identical prompts:

• specdec_bench in process: 2 runs per temperature.
• AIPerf against vllm serve: 4 runs per temperature.

Results:

• Under the same acceptance-length definition, the two agree within run-to-run noise. Paired by question, AIPerf − specdec_bench is −0.011 ± 0.042 at t=0 and −0.001 ± 0.037 at t=1 (95% interval). Between two runs of the same tool the difference is about ±0.07.
• Per-user decode speed matched within 1%.
• Getting one t=0 and one t=1 number took 22 min with specdec_bench and 13 min against one server.

This branch end to end on JHB. common/specdec_aiperf/run.sh ran in the launcher's directory layout, but not through slurm.py. Same target and drafter, and the server used the same engine settings as the comparison above. The script ran t=0/t=1 × concurrency 16/32 against one server:

Temperature	Concurrency	Average_AL	Output tok/s	per GPU	per-user decode tok/s	TTFT
0	16	3.826	1952	488	153.8	149 ms
0	32	3.821	2542	636	123.6	204 ms
1	16	3.445	1793	448	140.6	148 ms
1	32	3.437	2500	625	112.1	205 ms

The server took the speculative config in the YAML's dotted form (--speculative-config.method dflash ...), and max_num_batched_tokens was pinned to 2048 (KV cache 3.66M tokens, as in process). All 640 requests succeeded. upload_to_s3.py --dry-run accepted the four result directories, and timing.json matches AIPerf's own profile_export_aiperf.json.

How this compares with specdec_bench in process at concurrency 32:

• At t=0, specdec_bench gave 3.793 and 3.829; this run gave 3.821.
• At t=1, specdec_bench gave 3.462; this run gave 3.437. The gap comes from turn 1, whose prompts are identical in every run: this run's turn-1 AL is 3.484, against 3.503–3.530 for the comparison's AIPerf runs. So it is sampling noise, not a method difference.

The second commit (dataset registry, per-turn results, server preflight, sweep isolation) was run again the same way at concurrency 32: Average_AL was 3.840 at t=0 and 3.499 at t=1, within the spread of the runs above. The preflight check passed against the real server.

Unit tests. Added tests/examples/specdec_aiperf/test_bench.py, with three tests:

• bench.py end to end with a stand-in for aiperf profile, including a failed point that leaves the rest of the sweep intact;
• a server without spec-decode stats, which must fail before AIPerf runs;
• parity with specdec_bench: its MTBench and Timing metric classes on the same steps and timestamps, and its SPEED-Bench subsampling.

Removing the failure isolation, the preflight check, or the stratified subsampling each makes one of these tests fail.

Not run:

• The Qwen3-8B example YAML (vllm/vllm-openai:v0.29.0) on a cluster. tools/launcher/tests/test_examples_resolve.py passes.
• A TensorRT-LLM server. AIPerf reads its per-request stats too, but this is untested here.

Before your PR is "Ready for review"

• Is this change backward compatible?: ✅ New files only, plus a CHANGELOG entry and one sentence in the speculative-decoding skill.
• If you copied code from any other sources or added a new PIP dependency, did you follow guidance in CONTRIBUTING.md: ✅ AIPerf (Apache-2.0) is a dependency of this example only, installed in its own venv.
• Did you write any new necessary tests?: ✅
• Did you update Changelog?: ✅
• Did you get Claude approval on this PR?: ❌ Not yet.

Additional Information

Reopens #2973 as a draft. That PR changed examples/specdec_bench itself; this version is standalone.

🤖 Generated with Claude Code

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.04%. Comparing base (ace9fd9) to head (30eabe3).
⚠️ Report is 16 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2974      +/-   ##
==========================================
- Coverage   79.53%   79.04%   -0.50%     
==========================================
  Files         660      661       +1     
  Lines       74720    76053    +1333     
==========================================
+ Hits        59430    60117     +687     
- Misses      15290    15936     +646     

Flag	Coverage Δ
unit	61.11% <ø> (+0.05%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.