Repository navigation
accel/pd: KV transport on hetero-kv - #299
Merged
Merged
Conversation
accel/pd/transport.py is Optimus' glue to the standalone hetero-kv library: TorchCacheTransport (TorchCopyEngine over KVCacheManager/LayeredKVCache storage), SessionTransport (any hetero-kv engine) and p2p_transport (hetero-npu-p2p RNGD backend as a hetero-kv engine, one selection per DMA span). LayerProgress is hetero_kv.LayerProgress. torch_cache.py, layer_progress.py and hetero_p2p.py are removed; cache storage moves to torch_engine.py, KVCacheManager gains storage(). hetero-kv is the new `pd` extra, pinned to d4625f2; CPU CI installs Rust and it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background
accel/pdcarried its own KV transport pieces (torch_cache.py,layer_progress.py, thehetero_p2p.pyCopySession bridge). KV transport now lives in the standalone library ai-computing/hetero-kv (Rust core, C ABI, Python). Layering: Optimus keeps its engine contracts and owns the glue; hetero-kv never imports Optimus (its formeradapters/optimusis removed in ai-computing/hetero-kv#6, which this PR does not depend on).Changes
opt_prime/accel/pd/transport.py(Optimus glue on hetero-kv sessions: reserve -> transfer -> fence -> admit -> finish, poison/recover):TorchCacheTransport: same constructor,supports, receipt and error messages; copies the source prefix of every layer and batch entry with hetero-kv'sTorchCopyEngineoverbhsdbuffers (one group per distinct layer shape), fences, then publishesseq_lenas the admission step.recover()settles every session.SessionTransport+TransferPlan: explicit route over any hetero-kv engine (replacesHeteroP2PTransport; fenced = CopySession "ready", admitted = "decoding").p2p_transport+P2PBackendEngine: the RNGD -> ATOM path. The existing hetero-npu-p2p backend (RngdPdmaBackend) runs as a hetero-kv engine; the hetero_p2pTransferPlanfrombuild_planis validated, mapped onto flat 4 KiB-row hetero-kv buffers with one selection per DMA span, and each destination's spans run as a validated hetero_p2p plan with the originaldescriptors_per_command. Receipt checks are the same as CopySession's. On failure the session is poisoned before notify, andrecover()returns False (no quiesce, as before).LayerProgressis nowhetero_kv.LayerProgress(same semantics and messages) inpd_batching.pyandprefill_server.py.torch_cache.py,layer_progress.pyandhetero_p2p.py.TorchCache,LayeredKVCacheandfencemove totorch_engine.py, which still re-exportsTorchCacheTransport.KVCacheManager.storage(layer)(new, public) andLayeredKVCache.storage(layer)return the full-capacity tensors the transport wraps, so it no longer reads_cache.contracts.pyandnpu/shared_kv.pyare unchanged.npu/factory.py, examples (validate_koni.py,validate_checkpoint.py,benchmark_torch.py,validate_moe_pd.py), docs (HETEROGENEOUS_PD.md,CONTRIBUTING_PD.md,NPU_BACKEND_EXTENSIONS.md,NATIVE_SHARED_DIAGNOSTIC.md,CODE_REVIEW_GUIDE_KO.md).test_pd_layer_progress.pyruns againsthetero_kv.LayerProgress.test_pd.pyreplaces the CopySession bridge test with SessionTransport fence/notify, uncertain-completion poisoning, and ap2p_transporttest against real hetero_p2p plan validation (skips without hetero_p2p). Newtest_pd_transport.pycovers TorchCacheTransport with batch > 1, bf16, layered caches, rejections and recovery.pdextra,hetero-kv @ git+https://github.com/ai-computing/hetero-kv@d4625f28ba24d7c8bbe4ffab95c13f852fe35922#subdirectory=crates/hetero-kv-py(hetero-kv main, P0). The pin builds with pip and the maturin backend; no hetero-kv change was needed.import opt_primeandopt_prime.accel.pd(contracts, runtime) still work without it. The transports,torch_engine,pd_batchingand streamingprefill_serverneed it.cpu-tests.yml):dtolnay/rust-toolchain@stable+pip install -e "opt_prime[test,pd]". The hetero-kv wheel builds in about 20 s, and the Install step took 66 s.Verification
Commits: base
fff45de(optimus-rm), PRa3f4d5c; hetero-kvd4625f2. Host: 2x A100 80GB PCIe, torch 2.10.0+cu128, transformers 5.1.0, Python 3.10. Same venv for both sides (system site packages + hetero-kv).fff45deCUDA_VISIBLE_DEVICES= python3 -m pytest -q opt_prime/testspython -m unittest discover -s opt_prime/examples/pd/native_kv -p 'test_*.py'(CPU)test_pd_torch,test_pd_stream_admission,test_pd_stream_recurrent,test_prefill_source_progress,test_atom_stream_progress,test_pd_layer_progress,test_pd_families,test_pd_sampling)test_pd_transport)PYTHONPATH(test_pd,test_pd_npu,test_pd_compact,test_pd_atom_reuse,test_pd_factory)cpu-testsTest-ID diff (collect-only): -3
test_import_ordercases for the removed modules, +1 fortransport, +2 intest_pd.py, +4test_pd_transport.py. Every test that existed before still passes; the skip sets are identical apart from the replaced bridge test.CUDA A/B for TorchCacheTransport (the suite runs it on CPU only): a 3-layer bf16 Llama through
PDRuntime, prompts of 2/19/130 tokens, full and prefix prefill, routes cuda->cuda, cpu->cuda and cuda->cpu (GPU 1). Base and PR give bit-identical tokens, receipts, and all 333 captured decode logit vectors (18/18 cases). The token mismatches against the no-cache reference on the cross-device n=130 cases are identical on both sides: they come from bf16 CPU-vs-GPU prefill numerics, not from the transport.Not run: RNGD -> ATOM hardware (no ATOM on this host).
p2p_transportis checked against hetero_p2p's ownTransferPlan.validate()and a fake backend only.Compatibility and limits
pip install -e './opt_prime[pd]'(needs a Rust toolchain). An SDK venv that runspd.prefill_serverwith layer streaming also needs hetero-kv.opt_prime.accel.pd.torch_cache,.layer_progressand.hetero_p2pare gone.torch_engine.TorchCacheTransport/LayeredKVCache/TorchCache/fencekeep working; new code imports transports fromopt_prime.accel.pd.transport.HeteroP2PTransport(backend, plan_builder, notify, source=..., destination=..., spec=...)becomesp2p_transport(...)with the same arguments.sourcemust be the RNGD device path (/dev/rngd/npuNchM), ascreate_qwen_p2pandvalidate_koni.pyalready pass. Engine failures now surface ashetero_kv.TransferError(aRuntimeError) instead of the backend's own exception, andsession.statenames are hetero-kv's.Tensor.copy_instead of.to()). They are subject to the same P2P caveat on PCIe hosts as before; not exercised here.Related Issue
Closes #298. Related: ai-computing/hetero-kv#5, ai-computing/hetero-kv#6.
🤖 Generated with Claude Code