Skip to content

accel/pd: KV transport on hetero-kv - #299

Merged
gspark-etri merged 1 commit into
optimus-rmfrom
feat/298-hetero-kv
Nov 4, 2026
Merged

gspark-etri merged 1 commit into
optimus-rmfrom
feat/298-hetero-kv

Conversation

@gspark-etri

@gspark-etri gspark-etri commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

Background

accel/pd carried its own KV transport pieces (torch_cache.py, layer_progress.py, the hetero_p2p.py CopySession bridge). KV transport now lives in the standalone library ai-computing/hetero-kv (Rust core, C ABI, Python). Layering: Optimus keeps its engine contracts and owns the glue; hetero-kv never imports Optimus (its former adapters/optimus is removed in ai-computing/hetero-kv#6, which this PR does not depend on).

Changes

  • New opt_prime/accel/pd/transport.py (Optimus glue on hetero-kv sessions: reserve -> transfer -> fence -> admit -> finish, poison/recover):
    • TorchCacheTransport: same constructor, supports, receipt and error messages; copies the source prefix of every layer and batch entry with hetero-kv's TorchCopyEngine over bhsd buffers (one group per distinct layer shape), fences, then publishes seq_len as the admission step. recover() settles every session.
    • SessionTransport + TransferPlan: explicit route over any hetero-kv engine (replaces HeteroP2PTransport; fenced = CopySession "ready", admitted = "decoding").
    • p2p_transport + P2PBackendEngine: the RNGD -> ATOM path. The existing hetero-npu-p2p backend (RngdPdmaBackend) runs as a hetero-kv engine; the hetero_p2p TransferPlan from build_plan is validated, mapped onto flat 4 KiB-row hetero-kv buffers with one selection per DMA span, and each destination's spans run as a validated hetero_p2p plan with the original descriptors_per_command. Receipt checks are the same as CopySession's. On failure the session is poisoned before notify, and recover() returns False (no quiesce, as before).
  • LayerProgress is now hetero_kv.LayerProgress (same semantics and messages) in pd_batching.py and prefill_server.py.
  • Removed torch_cache.py, layer_progress.py and hetero_p2p.py. TorchCache, LayeredKVCache and fence move to torch_engine.py, which still re-exports TorchCacheTransport. KVCacheManager.storage(layer) (new, public) and LayeredKVCache.storage(layer) return the full-capacity tensors the transport wraps, so it no longer reads _cache.
  • contracts.py and npu/shared_kv.py are unchanged.
  • Updated callers: npu/factory.py, examples (validate_koni.py, validate_checkpoint.py, benchmark_torch.py, validate_moe_pd.py), docs (HETEROGENEOUS_PD.md, CONTRIBUTING_PD.md, NPU_BACKEND_EXTENSIONS.md, NATIVE_SHARED_DIAGNOSTIC.md, CODE_REVIEW_GUIDE_KO.md).
  • Tests: test_pd_layer_progress.py runs against hetero_kv.LayerProgress. test_pd.py replaces the CopySession bridge test with SessionTransport fence/notify, uncertain-completion poisoning, and a p2p_transport test against real hetero_p2p plan validation (skips without hetero_p2p). New test_pd_transport.py covers TorchCacheTransport with batch > 1, bf16, layered caches, rejections and recovery.
  • Dependency: new pd extra, hetero-kv @ git+https://github.com/ai-computing/hetero-kv@d4625f28ba24d7c8bbe4ffab95c13f852fe35922#subdirectory=crates/hetero-kv-py (hetero-kv main, P0). The pin builds with pip and the maturin backend; no hetero-kv change was needed. import opt_prime and opt_prime.accel.pd (contracts, runtime) still work without it. The transports, torch_engine, pd_batching and streaming prefill_server need it.
  • CI (cpu-tests.yml): dtolnay/rust-toolchain@stable + pip install -e "opt_prime[test,pd]". The hetero-kv wheel builds in about 20 s, and the Install step took 66 s.

Verification

Commits: base fff45de (optimus-rm), PR a3f4d5c; hetero-kv d4625f2. Host: 2x A100 80GB PCIe, torch 2.10.0+cu128, transformers 5.1.0, Python 3.10. Same venv for both sides (system site packages + hetero-kv).

Run base fff45de PR
CUDA_VISIBLE_DEVICES= python3 -m pytest -q opt_prime/tests 576 passed, 34 skipped, 135 subtests 580 passed, 34 skipped, 135 subtests
same, GPUs visible 582 passed, 28 skipped, 135 subtests 586 passed, 28 skipped, 135 subtests
python -m unittest discover -s opt_prime/examples/pd/native_kv -p 'test_*.py' (CPU) 51 tests OK (17 skipped) 51 tests OK (17 skipped)
PD torch + layer streaming subset, GPU visible (test_pd_torch, test_pd_stream_admission, test_pd_stream_recurrent, test_prefill_source_progress, test_atom_stream_progress, test_pd_layer_progress, test_pd_families, test_pd_sampling) 68 passed, 4 skipped 68 passed, 4 skipped (+4 in test_pd_transport)
hetero_p2p-dependent tests with hetero-npu-p2p on PYTHONPATH (test_pd, test_pd_npu, test_pd_compact, test_pd_atom_reuse, test_pd_factory) 24 passed 26 passed
CI cpu-tests 560 passed, 46 skipped; native_kv 51 OK 564 passed, 46 skipped; native_kv 51 OK

Test-ID diff (collect-only): -3 test_import_order cases for the removed modules, +1 for transport, +2 in test_pd.py, +4 test_pd_transport.py. Every test that existed before still passes; the skip sets are identical apart from the replaced bridge test.

CUDA A/B for TorchCacheTransport (the suite runs it on CPU only): a 3-layer bf16 Llama through PDRuntime, prompts of 2/19/130 tokens, full and prefix prefill, routes cuda->cuda, cpu->cuda and cuda->cpu (GPU 1). Base and PR give bit-identical tokens, receipts, and all 333 captured decode logit vectors (18/18 cases). The token mismatches against the no-cache reference on the cross-device n=130 cases are identical on both sides: they come from bf16 CPU-vs-GPU prefill numerics, not from the transport.

Not run: RNGD -> ATOM hardware (no ATOM on this host). p2p_transport is checked against hetero_p2p's own TransferPlan.validate() and a fake backend only.

Compatibility and limits

  • New install requirement for PD: pip install -e './opt_prime[pd]' (needs a Rust toolchain). An SDK venv that runs pd.prefill_server with layer streaming also needs hetero-kv.
  • opt_prime.accel.pd.torch_cache, .layer_progress and .hetero_p2p are gone. torch_engine.TorchCacheTransport / LayeredKVCache / TorchCache / fence keep working; new code imports transports from opt_prime.accel.pd.transport.
  • HeteroP2PTransport(backend, plan_builder, notify, source=..., destination=..., spec=...) becomes p2p_transport(...) with the same arguments. source must be the RNGD device path (/dev/rngd/npuNchM), as create_qwen_p2p and validate_koni.py already pass. Engine failures now surface as hetero_kv.TransferError (a RuntimeError) instead of the backend's own exception, and session.state names are hetero-kv's.
  • With several destination DMA-BUFs, one hetero_p2p plan becomes one backend call per destination. The command grouping per DMA-BUF is unchanged, which CopySession already required.
  • Direct GPU-to-GPU tensor copies behave as before (Tensor.copy_ instead of .to()). They are subject to the same P2P caveat on PCIe hosts as before; not exercised here.

Related Issue

Closes #298. Related: ai-computing/hetero-kv#5, ai-computing/hetero-kv#6.

🤖 Generated with Claude Code

accel/pd/transport.py is Optimus' glue to the standalone hetero-kv library:
TorchCacheTransport (TorchCopyEngine over KVCacheManager/LayeredKVCache storage),
SessionTransport (any hetero-kv engine) and p2p_transport (hetero-npu-p2p RNGD
backend as a hetero-kv engine, one selection per DMA span). LayerProgress is
hetero_kv.LayerProgress. torch_cache.py, layer_progress.py and hetero_p2p.py are
removed; cache storage moves to torch_engine.py, KVCacheManager gains storage().

hetero-kv is the new `pd` extra, pinned to d4625f2; CPU CI installs Rust and it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@gspark-etri
gspark-etri merged commit 8b550a6 into optimus-rm Nov 4, 2026
1 check passed
@gspark-etri
gspark-etri deleted the feat/298-hetero-kv branch November 4, 2026 07:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant