Repository navigation
Rust 1.100.0, GPUI Kit 0.9.0, faster Q8 decode at long contexts - #34
Merged
Merged
Conversation
Pin the GUI and development toolchain to Rust 1.100.0 and GPUI Kit to 0.9.0. The headless MSRV stays at Rust 1.92. The GPUI Kit update needs no source change. Clippy 1.100 finds more redundant clones and index loops, which the workspace denies. Remove nine of them. The k_parity loop keeps its strictness: a short reference run still fails the test. The Python bindings now inherit the workspace lints, as Cargo 1.100 asks. The crate passes them with no finding. Against Rust 1.99.1, captures, outputs and tokenization are byte-identical in the reference, planned and planned-fused modes, and Q8 decode and prefill speed is within noise. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On aarch64, the Q8_0 fast decode path now uses the register-tiled attention kernel that prefill and planned decode already use. The kernel computes each element with the same arithmetic in the same order, so the output is bit-identical. The reference path keeps the generic body for single-row decode, as an independent check. Paired decode A/B, Llama-3.2-1B Q8_0, M1 Pro, 4 and 8 threads: +1.8% / +0.7% at 32 tokens of context, +7.2% / +5.0% at 512, +12.5% / +11.3% at 1024, +21.9% / +15.5% at 2048. A per-head bit check on real model data, a 96-token real-text run and the golden morphology check all match. Known cost: Q6_K prefill is 6-8% lower and Q4_K prefill about 2% lower, although this branch is not on that route and the Q6 kernel's instructions do not change. Loop alignment, fat LTO, PGO, allocator and prompt-size changes do not remove it. The cause is not known. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The audio step ran all 675 library tests a second time, but no unit test uses the audio code. Build and link the tests with the feature instead. The step keeps its purpose: it still compiles and links the audio code, which clippy alone does not link. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cargo audit fails on quinn-udp 0.5.15. In that version, a remote sender can make the receive loop spin on targets that read one datagram per recvmsg call, Apple platforms among them (denial of service). The fix is in 0.5.16. The crate comes in through GPUI Kit's HTTP client (gpui-kit-assets -> gpui-pre-reqwest -> quinn). main has the same version; the advisory was published after main's last audit run. The update also changes which windows-sys version five crates resolve to, for Windows only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This branch moves Ember to Rust 1.100.0 and GPUI Kit 0.9.0, makes Q8 decode faster at long contexts, and removes a duplicate test run from the gate. Each part is a separate commit, so you can review or revert it alone.
e4c381ebuilddd3966fperf3f41a92ci--no-run, so that it no longer runs the 675 library tests a second time.The headless MSRV stays at Rust 1.92. The Ember version stays at
1.0.0.Toolchain and GPUI Kit (
e4c381e)Tab::ixwas checked again in that version.redundant_cloneandneedless_range_loopcases, and the workspace denies them. This commit fixes nine.tests/k_parity.rs: A new assert keeps the test as strict as before, so a short reference run still fails.Cargo.lockchanges a lot, because GPUI Kit moves its own dependency tree. The headless MSRV build still passes.Q8 decode speedup (
dd3966f)The tiled kernel does the same arithmetic in the same order for each element, so the output is bit-identical. Single-row reference decode keeps the generic body, so it stays an independent check of the kernel.
Paired decode A/B, Llama-3.2-1B Q8_0, M1 Pro, 8 alternating rounds:
Warning
Known cost, cause not found: Q6_K prefill is 6–8% lower, and Q4_K prefill is about 2% lower. This change is not on that route, and the Q6 kernel's instructions do not change. Loop alignment, fat LTO, PGO, allocator settings, and prompt size do not remove the drop. Q4_K and Q6_K decode do not change.
Tests (
3f41a92)Verification
scripts/ci_local.sh fullon Rust 1.100.0,e4c381etreecargo auditwas skipped, because it is not installed locally)scripts/ci_local.sh quickon the final treeexact-semanticvalidate_k_parity.sh)Not verified
🤖 Generated with Claude Code