Skip to content

Thor port: build and run on NVIDIA Jetson (non-AMD hosts) - #1

Closed
dcharlot65-openie wants to merge 1 commit into
mainfrom
thor-port
Closed

dcharlot65-openie wants to merge 1 commit into
mainfrom
thor-port

Conversation

@dcharlot65-openie

Copy link
Copy Markdown

What

Makes the workspace build and run correct decode inference on an NVIDIA Jetson AGX Thor (Tegra T264, Blackwell CC 11.0, Vulkan 1.4, driver 595.78), while changing nothing on the Strix Halo box.

  • npu: builds as a no-op stub when AMD XRT headers are absent (npu_stub cfg emitted by build.rs). With XRT present, byte-identical behavior.
  • kitten: portable dot4 / sdpa_rows fallbacks for non-x86_64 (multi-accumulator scalar, LLVM autovec to NEON).
  • vkrt: enable maintenance4 + vulkanMemoryModel(+DeviceScope) device features. Our SPIR-V already requires both (LocalSizeId from glslc, GL_KHR_memory_scope_semantics); RADV tolerated the omission, NVIDIA device-losts. Spec-compliance fix for all platforms.
  • jenergy (new crate): jrecord, a std-only Rust energy recorder for Jetson INA3221/INA238 hwmon rails with the same CLI and JSON shape as tools/energy/record.py.
  • bench/devinfo (new bin): prints vkrt DeviceInfo including the driver coopmat config table.
  • PORT_NOTES_THOR.md: lab notebook for the port — first numbers and open gaps, failures included.

Measured (Thor, decode, differential 256−16; details + logs in PORT_NOTES_THOR.md)

Nemotron 3 Nano 30B-A3B Q4_K_M strix-rt (untuned) llama.cpp CUDA
decode tok/s 24.6 61.6
J/token module / GPU rail 2.07 / 0.83 1.14 / 0.49

Correctness smoke: "The capital of France is" -> " Paris" top-1 by 4 logits (4B model).

Known gaps (documented, not fixed here)

  • Batched prefill (PF=1) device-losts on NVIDIA; PF_PROBE bisect pins it to coopMatLoad-fed MulAdd (register-only MMA is fine). Standalone repro is the next step.
  • int8 coopmat shaders use 16x16x16; NVIDIA only exposes K=32 for s8/u8 (ternary path needs a KSize-32 variant).
  • One shader declares 62592 B shared vs NVIDIA 49152 max.
  • Host needs glslc with GL_EXT_integer_dot_product (Ubuntu 24.04 package is too old; conda-forge shaderc 2026.4 works).

🤖 Generated with Claude Code

- npu: build as a no-op stub when AMD XRT headers are absent (npu_stub cfg); unchanged where XRT exists
- kitten: portable dot4/sdpa_rows fallbacks for non-x86_64 (NEON via autovec)
- vkrt: enable maintenance4 + vulkanMemoryModel(DeviceScope) device features (spec-required by our shaders; RADV tolerated the omission, NVIDIA does not)
- jenergy: Rust energy recorder for Jetson INA3221/INA238 hwmon rails, same CLI/JSON as tools/energy/record.py
- bench/devinfo: print vkrt DeviceInfo incl. supported coopmat configs
- PORT_NOTES_THOR.md: lab notes - first Thor numbers (Nemotron 4B + 30B-A3B decode J/token vs llama.cpp CUDA) and known gaps

Decode works and is correct on Thor; batched prefill (PF=1) still device-lost, narrowed by PF_PROBE bisect to coopMatLoad-fed MulAdd. See PORT_NOTES_THOR.md.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

Thank you for taking the time to open this! strix-rt is published read-only for now, so we don't merge pull requests and this one was closed automatically. It isn't a judgment of your change: we just can't review and maintain outside code yet. Issues are open for questions, reproduction reports and feedback, and forks are very welcome. Take whatever helps you save energy.

@github-actions github-actions Bot closed this Oct 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant