Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .github/workflows/cpu-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,11 +20,12 @@ jobs:
with:
python-version: "3.10"
cache: pip
- uses: dtolnay/rust-toolchain@stable # builds hetero-kv (pd extra); pip caches the wheel by commit
- name: Install
run: |
python -m pip install --upgrade pip
python -m pip install torch==2.10.0 --index-url https://download.pytorch.org/whl/cpu
python -m pip install -e "opt_prime[test]" "transformers==5.1.0" triton
python -m pip install -e "opt_prime[test,pd]" "transformers==5.1.0" triton
- name: opt_prime tests
working-directory: opt_prime
run: python -m pytest -q tests
Expand Down
2 changes: 1 addition & 1 deletion opt_prime/docs/CODE_REVIEW_GUIDE_KO.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
| 3 | `examples/pd/validate_native_no_primer.py` | 실제 ATOM 검증. 첫 요청 전 prefill 금지, 이후 참조 방식과 전체 vocabulary logits 비교 |
| 4 | `examples/pd/native_rngd_prefix_worker.py` | 별도 Furiosa SDK 프로세스에서 요청마다 실제 RNGD prefill. 현재는 NPZ 파일을 이용한 호스트 전달 검증 도구 |
| 5 | `opt_prime/pd/contracts.py`, `runtime.py` | 모델 revision·물리 KV 규격, 수명, prefill→reserve→transfer→decode→finish 계약 |
| 6 | `opt_prime/pd/npu/engines.py`, `hetero_p2p.py` | 기존 커스텀 NPU 엔진과 P2P 전송을 공통 런타임에 연결하는 어댑터 |
| 6 | `opt_prime/pd/npu/engines.py`, `transport.py` | 기존 커스텀 NPU 엔진과 P2P 전송을 공통 런타임에 연결하는 어댑터 |
| 7 | `opt_prime/pd/llmd.py` | llm-d stage 서비스, 요청/취소 상태, 소유 스레드 실행과 HTTP 경계 |

## 제조사 제공 기능과 우리 코드
Expand Down
11 changes: 5 additions & 6 deletions opt_prime/docs/CONTRIBUTING_PD.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,16 +14,15 @@ dependencies, see [NPU backend extensions](NPU_BACKEND_EXTENSIONS.md).
| `contracts.py` | Request, continuation, physical KV identity, engine/transport protocols | Import SDKs or infer device addresses |
| `runtime.py` | Single-flight execution and poisoned-lane owner retention | Model-specific layout conversion or service routing |
| `torch_models.py` | Explicit family support and cache geometry constraints | Load weights or silently accept hybrid/sliding caches |
| `torch_cache.py` | Optimus KVCacheManager storage and explicit tensor-copy transport | Choose a fallback after DMA failure |
| `torch_engine.py` | Model forward calls, continuation handling, greedy decoding | HTTP request handling or vendor driver calls |
| `torch_engine.py` | Model forward calls, continuation handling, greedy decoding; per-layer cache storage | HTTP request handling or vendor driver calls |
| `transport.py` | Optimus glue to hetero-kv: tensor-copy and P2P transports on hetero-kv sessions | Choose a fallback after DMA failure; bypass completion validation |
| `koni.py` | Existing RNGD prefix and owned ATOM worker adaptation | Reimplement allocation/export ABI |
| `hetero_p2p.py` | Bridge to checked CopySession | Bypass completion validation |
| `llmd.py` (feature branch) | HTTP/control-plane ownership queue | Run DMA inside HTTP worker threads |

Earlier `opt_prime.accel.pd.runtime.KVSpec` and
`opt_prime.accel.pd.torch_engine.TorchCacheTransport` imports remain compatibility
exports. New code should import contracts from `opt_prime.accel.pd`, and the Torch
transport from `opt_prime.accel.pd.torch_cache`.
exports. New code should import contracts from `opt_prime.accel.pd`, and transports
from `opt_prime.accel.pd.transport`.

## Adding a Torch model

Expand Down Expand Up @@ -124,5 +123,5 @@ A native checkpoint success does **not** register it with `KoniRngdEngine` or
`KoniAtomWorkerEngine`. Qwen requires Q/K/V projection biases, tied embedding
handling, checkpoint-specific layer/head dimensions and cache layouts in both
compiled graphs. Each new physical layout needs export/import/packing and fence
validation before it can use `HeteroP2PTransport`. Keep vendor-native baseline
validation before it can use `p2p_transport`. Keep vendor-native baseline
support and end-to-end heterogeneous PD support as separate evidence columns.
38 changes: 26 additions & 12 deletions opt_prime/docs/HETEROGENEOUS_PD.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,18 +15,23 @@ Caller / future llm-d service adapter
\ /
Transport
|
optional HeteroP2PTransport
accel/pd/transport.py (Optimus glue)
|
hetero_p2p.CopySession + checked backend
hetero-kv Session + engine (TorchCopyEngine,
P2PBackendEngine over a checked hetero_p2p backend)
|
supported driver / device pair
```

`from opt_prime import PDRuntime` and `from opt_prime.accel.pd import ...` do not
initialize PyTorch, CUDA, or a vendor SDK. Existing `Optimus_Inference` imports
and behavior remain unchanged. Run from the repository with `PYTHONPATH=opt_prime`.
The optional P2P bridge imports `hetero_p2p` only when constructed. Install that
separate project in the same environment; no vendor implementation is vendored.
KV transports (`opt_prime.accel.pd.transport`) run on the separate library
[hetero-kv](https://github.com/ai-computing/hetero-kv) (`pd` extra): it owns the
reserve -> transfer -> fence -> admit -> finish lifecycle, poisoning and explicit
routes; Optimus keeps its engine contracts and maps its caches onto hetero-kv
buffers. The RNGD -> ATOM path imports `hetero_p2p` only when a transfer runs.
Install that separate project in the same environment; no vendor implementation is vendored.

## Interface and lifetime contract

Expand Down Expand Up @@ -64,12 +69,12 @@ object garbage collection is not a recovery procedure.

```python
from opt_prime import PDRuntime
from opt_prime.accel.pd.hetero_p2p import HeteroP2PTransport
from opt_prime.accel.pd.transport import p2p_transport

# Real endpoint adapters must implement the contracts above.
transport = HeteroP2PTransport(
transport = p2p_transport(
checked_backend, build_plan_from_live_leases, mark_atom_cache_visible,
source="rngd:3", destination="atom:2", spec=prefill_engine.spec,
source="/dev/rngd/npu3ch0", destination="atom:2", spec=prefill_engine.spec,
)
runtime = PDRuntime(prefill_engine, decode_engine, transport)
tokens = runtime.generate(request)
Expand All @@ -79,8 +84,13 @@ The snippet illustrates dependency injection; use the bundled KONI endpoint
adapters and explicit validation runner to bind existing hardware resources.
`build_plan_from_live_leases` must produce the existing validated `TransferPlan`:
RNGD-owned source allocation, ATOM-owned exported DMA-BUF destinations and bounded
copy spans. The bridge checks the physical format and delegates completion receipt,
owner retention and decode sequencing to the real `CopySession`. Hardware backend
copy spans. `p2p_transport` checks the physical format, maps the plan onto
hetero-kv buffers (one selection per DMA span) and runs it through a hetero-kv
`Session`: `P2PBackendEngine` executes each destination's spans as a validated
`TransferPlan` with the original command grouping and checks the completion receipt
as `CopySession` did; the session fences before `notify`, retains owners and
poisons on uncertainty (fenced = `CopySession` "ready", admitted = "decoding"). Other
device pairs use `SessionTransport` with any hetero-kv engine. Hardware backend
construction must retain its driver/SDK/helper ABI checks. The transfer adapter
never discovers raw addresses or attempts reverse import.

Expand Down Expand Up @@ -119,13 +129,17 @@ process; do not kill it as a routine timeout cleanup.

```bash
python -m pip install -e ./opt_prime # SDK-free contracts
# KV transports (hetero-kv; builds from source, needs a Rust toolchain):
python -m pip install -e './opt_prime[pd]'
# Optional isolated Torch validation environment (not the NPU SDK environment):
python -m pip install -e './opt_prime[torch-pd]'
python -m pip install -e './opt_prime[torch-pd,pd]'
```

The Torch validation extra pins the versions actually tested. Keep vendor SDK
Python environments separate; the core package has no mandatory torch dependency.
KONI uses the existing RNGD Python environment and a separate ATOM owning process.
An SDK environment that streams layers (`pd.prefill_server`) or runs a transport
also needs hetero-kv installed.
The hardware runner accepts `--config` and `--hetero-repo` explicitly rather than
embedding local artifact paths in the engine package.

Expand All @@ -151,14 +165,14 @@ explicit, independently validated adapter rather than a transport side effect.

```bash
PYTHONPATH=opt_prime python3 -m unittest discover -s opt_prime/tests -v
# Also exercise the real optional CopySession with a fake CPU backend:
# Also exercise the hetero_p2p plan path with a fake CPU backend:
PYTHONPATH=opt_prime:/path/to/hetero-npu-p2p/src \
python3 -m unittest discover -s opt_prime/tests -v
```

Tests include 24 actual GPT-2/Llama parity cases and cover request reuse, output-token accounting, incompatible revisions,
unsupported routes, transfer/decode failures, owner retention, lazy imports and
P2P completion-before-notify ordering. The small randomly initialized CPU models test execution parity, not task quality.
P2P completion-before-notify ordering on the hetero-kv session. The small randomly initialized CPU models test execution parity, not task quality.
Contract fixtures and CPU tests do not measure NPU performance, MFU or power.

Measured results and exact scope: [validation record](validation/RESULTS.md).
Expand Down
2 changes: 1 addition & 1 deletion opt_prime/docs/NATIVE_SHARED_DIAGNOSTIC.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ Initial0.5B r1/r2 lacked dedicated warmup and are preserved, not headline result

`hetero-npu-p2p/src/hetero_p2p/atom_experimental.py` exports verified live eager
allocations at pinned thunk ABI. `qwen_atom_worker.py` supplies external KV tensor
addresses; `npu/engines.py` builds the plan and `pd/hetero_p2p.py` uses CopySession.
addresses; `npu/engines.py` builds the plan and `pd/transport.py` runs it on a hetero-kv session.
RNGD writes ATOM-owned exported buffers directly, then the owner marks KV updated.

Installed `rebel/kv_cache.py` native `set_kv_cache_for_layer` accepts **host** data.
Expand Down
4 changes: 2 additions & 2 deletions opt_prime/docs/NPU_BACKEND_EXTENSIONS.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,13 +10,13 @@ serving/control plane. Neither branch is the upstream main branch.
| Layer | Files | Responsibility | New device/model work |
|---|---|---|---|
| Common lifecycle | `contracts.py`, `runtime.py` | Requests, continuation boundary, physical KV identity, fenced stages, retained owners on error | Usually none |
| Torch backend | `torch_models.py`, `torch_cache.py`, `torch_engine.py` | Dense HuggingFace model execution and explicit tensor copy | Add family policy and numerical tests; actual GPU validation still required |
| Torch backend | `torch_models.py`, `torch_engine.py`, `transport.py` | Dense HuggingFace model execution and explicit tensor copy | Add family policy and numerical tests; actual GPU validation still required |
| Dense NPU model contract | `npu/model.py` | Immutable checkpoint geometry, full-attention policy, QKV bias/tied-weight checks, per-head DMA alignment | Add explicit semantics; reject unsupported architectures |
| RNGD graph | `npu/rngd.py` | Model-sized biased attention/MLP prefill; device-resident BF16 KV | Implement supported compiler operations and precision policy |
| ATOM graph | `npu/atom.py` | Model-sized resident paged decode | Compile and validate exact input/cache contract |
| ATOM runtime | `npu/atom_runtime.py`, `npu/atom_device.py` | Pinned SDK runtime, physical-device binding, cache visibility | Replace version-specific implementation for a new SDK/device |
| Endpoint adapters | `npu/engines.py` | Adapt these graphs/worker to the common prefill/decode protocols | Preserve lease ownership and explicit completion fences |
| Transport | `hetero_p2p.py`, external `hetero-npu-p2p` library | Validate physical route/layout, export/import/copy completion | Implement a separately negotiated route; no implicit fallback |
| Transport | `transport.py` on hetero-kv, external `hetero-npu-p2p` backend | Validate physical route/layout, export/import/copy completion | Implement a separately negotiated route; no implicit fallback |
| Serving | `llmd.py` on feature branch | Request validation, source preparation, serial owner, HTTP | Independent of model dimensions and vendor graph implementation |
| Experiments | `examples/pd/*qwen*.py` | Build artifacts, compare references, finite test runs and diagnostics | Extend measured cases; do not put test traffic into the serving engine |

Expand Down
2 changes: 1 addition & 1 deletion opt_prime/examples/pd/benchmark_torch.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
MistralConfig, MistralForCausalLM)
from opt_prime.accel.pd import PDRequest, PDRuntime
from opt_prime.accel.pd.torch_engine import TorchCausalLMEngine
from opt_prime.accel.pd.torch_cache import TorchCacheTransport
from opt_prime.accel.pd.transport import TorchCacheTransport


class Timed:
Expand Down
2 changes: 1 addition & 1 deletion opt_prime/examples/pd/validate_checkpoint.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
from transformers import AutoModelForCausalLM, AutoTokenizer
from opt_prime.accel.pd import PDRequest, PDRuntime
from opt_prime.accel.pd.torch_engine import TorchCausalLMEngine
from opt_prime.accel.pd.torch_cache import TorchCacheTransport
from opt_prime.accel.pd.transport import TorchCacheTransport


def main():
Expand Down
6 changes: 3 additions & 3 deletions opt_prime/examples/pd/validate_koni.py
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ def main():
from hetero_p2p import RngdPdmaBackend
from opt_prime.accel.pd import PDRuntime, PDRequest
from opt_prime.accel.pd.koni import KoniRngdEngine, KoniAtomWorkerEngine
from opt_prime.accel.pd.hetero_p2p import HeteroP2PTransport
from opt_prime.accel.pd.transport import p2p_transport
device = cfg['atom_devices'][0]
if cfg['atom_devices'] != [device] or device not in (2, 3):
raise ValueError('one authorized ATOM2/3 required')
Expand Down Expand Up @@ -137,8 +137,8 @@ def _next_token(self, x, P):
print('REFERENCES_READY', flush=True)
source = KoniRngdEngine(engine, packer, binding, revision=model_dir.name, max_prefix=capacity)
decoder = KoniAtomWorkerEngine(owner, spec=source.spec, device=device, batch_size=batch)
transport = HeteroP2PTransport(backend, decoder.build_plan, lambda lease: None,
source=binding.device_path, destination=f'atom:{device}', spec=source.spec)
transport = p2p_transport(backend, decoder.build_plan, lambda lease: None,
source=binding.device_path, destination=f'atom:{device}', spec=source.spec)
runtime = PDRuntime(source, decoder, transport)
rows = []
for repeat in range(cfg.get('repeats', 2)):
Expand Down
2 changes: 1 addition & 1 deletion opt_prime/examples/pd/validate_moe_pd.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
from transformers import AutoModelForCausalLM, AutoTokenizer

from opt_prime.accel.pd import PDRequest, PDRuntime
from opt_prime.accel.pd.torch_cache import TorchCacheTransport
from opt_prime.accel.pd.transport import TorchCacheTransport
from opt_prime.accel.pd.torch_engine import TorchCausalLMEngine

PROMPTS = ("The capital of France is",
Expand Down
32 changes: 0 additions & 32 deletions opt_prime/opt_prime/accel/pd/hetero_p2p.py

This file was deleted.

63 changes: 0 additions & 63 deletions opt_prime/opt_prime/accel/pd/layer_progress.py

This file was deleted.

4 changes: 2 additions & 2 deletions opt_prime/opt_prime/accel/pd/npu/factory.py
Original file line number Diff line number Diff line change
Expand Up @@ -174,7 +174,7 @@ def create_qwen_p2p(config_path, engine, binding, packer, backend, directory):
config explicitly enables the harness; it is never inferred or auto-loaded.
"""
from opt_prime.accel.pd.npu.engines import QwenRngdEngine, QwenAtomWorkerEngine
from opt_prime.accel.pd.hetero_p2p import HeteroP2PTransport
from opt_prime.accel.pd.transport import p2p_transport
from opt_prime.accel.pd.runtime import PDRuntime

cfg = json.loads(Path(config_path).read_text())
Expand Down Expand Up @@ -205,7 +205,7 @@ def create_qwen_p2p(config_path, engine, binding, packer, backend, directory):
from opt_prime.accel.pd.capabilities import negotiate

negotiate(source.capabilities, decoder.capabilities, "p2p")
transport = HeteroP2PTransport(
transport = p2p_transport(
backend,
decoder.build_plan,
lambda _: None,
Expand Down
2 changes: 1 addition & 1 deletion opt_prime/opt_prime/accel/pd/pd_batching.py
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@

from opt_prime.accel.pd.batching import ContinuousBatcher, prefill_chunk
from opt_prime.accel.pd.runtime import RequestCancelled
from opt_prime.accel.pd.layer_progress import LayerProgress
from hetero_kv import LayerProgress
from opt_prime.accel.pd.sampling import select_token
from opt_prime.accel.pd.torch_models import describe_model

Expand Down
Loading
Loading