From 51f666c3d97bde88e4d2646b7f8349192f060d7e Mon Sep 17 00:00:00 2001 From: Kevaundray Wedderburn Date: Tue, 17 Nov 2026 19:27:21 +0000 Subject: [PATCH 1/2] doc: AVX2 single-thread performance ideas from riscv-exploration Ideas from riscv-exploration (06e5c9cd6) for the experimental prover on AVX2 without AVX-512, on one thread. Nothing changes in the code. Measured on clifv-remote-04 (EPYC 9454P, Zen 4), one pinned core, LEANVM_NUM_THREADS=1, cargo 1.99.1. Four builds of 6cac93d7f: native (AVX-512), x86-64-v3 +pclmulqdq,+vpclmulqdq, the same +gfni,+aes, and target-cpu=haswell. No zmm in the AVX2 binaries. aggregate --xmss 100, last of three passes, ms (avx512 / avx2+vpclmul / +gfni / haswell): Prove 542 / 979 / 930 / 1340 Keccak reduction 154 / 383 / 342 / 612 Skip 49 / 174 / 175 / 319 Product 50 / 116 / 77 / 149 Tables 20 / 49 / 49 / 81 Tail 15 / 21 / 21 / 38 Commit (encode+Merkle) 121 / 185 / 185 / 222 PCS open 151 / 237 / 230 / 287 level-0 Project 36 / 60 / 52 / 83 commits L1-L3 49 / 81 / 81 / 94 Bus GKR 33 / 61 / 61 / 71 Bus leaves 20 / 30 / 30 / 50 Bus grind 22 / 34 / 34 / 34 Execute program 19 / 20 / 20 / 20 Prove, ms, same order: --sphincs 40 876 / 1410 / 1340 / 1870; --xmss 400 1790 / 3310 / 3110 / 4630. In every workload Skip is the span furthest behind AVX-512, by more than three times. PCS throughput bench, PCS_LOG_N=22, rate 1/2, one thread: the commit's encode alone takes 32.1 ms here on AVX2+VPCLMULQDQ, 33.8 ms on riscv-exploration's AVX2 build, 24.5 ms on AVX-512. The whole commit takes 94.7 ms on AVX2 (riscv-exploration: 63.8 ms with BLAKE2s), so Keccak leaf hashing is most of it. keccak256 hash_throughput, AVX2, one thread: 10.91 Mhash/s at 64 bytes (one permutation), 2.75 Mhash/s at 512 bytes (four): the same time per permutation, so absorbing is not the limit. bit_transpose_64bytes, 256 blocks in L1, one core: the portable gather 7.25 ns per block, riscv's AVX2 form 4.34 ns, its AVX2+GFNI form 1.40 ns; all three agree on random input. --- doc/perf/avx2-single-thread.md | 79 ++++++++++++++++++++++++++++++++++ 1 file changed, 79 insertions(+) create mode 100644 doc/perf/avx2-single-thread.md diff --git a/doc/perf/avx2-single-thread.md b/doc/perf/avx2-single-thread.md new file mode 100644 index 0000000000..f34065f7fd --- /dev/null +++ b/doc/perf/avx2-single-thread.md @@ -0,0 +1,79 @@ +# AVX2 single-thread performance ideas from `riscv-exploration` + +Ideas for the experimental prover on x86 machines with AVX2 and no AVX-512, proving on one thread (`LEANVM_NUM_THREADS=1`), taken from the `riscv-exploration` branch (`06e5c9cd6`). Each idea keeps the proof bytes, so `bins/leanvm/tests/identical_proofs.rs` checks it. Paths prefixed `riscv:` are in that branch; unprefixed ones are under `crates/experimental_prover/`. + +The ranking comes from the span trees of `cargo leanvm --prover experimental aggregate` at 100 and 400 leanXMSS and 40 leanSPHINCS signatures, built for one AVX-512 and three AVX2 targets on one Zen 4 core. The measurements are in the commit message. AVX2 targets come in three tiers, and some ideas need the third: + +|tier|flags|machines| +|---|---|---| +|AVX2 + PCLMULQDQ|`-C target-cpu=haswell`|Haswell to Skylake, Zen 1 and 2, CI's `haswell` testbed| +|+ VPCLMULQDQ|`-C target-cpu=x86-64-v3 -C target-feature=+pclmulqdq,+vpclmulqdq`|Zen 3| +|+ GFNI|the above with `+gfni`|Alder Lake and later Intel client cores, whose AVX-512 is off| + +## Where the time goes on AVX2 + +In order of share of the proof: + +1. The Keccak reduction (`keccak_gkr`), the largest stage and the furthest behind AVX-512. Within its layers, Skip falls furthest behind, then Tables and Product. Tail stays closest to AVX-512. +2. The PCS opening: level 0's lane projection (`Project`) and the extension commits of levels 1 to 3. +3. The commit: the base encode and the Keccak Merkle leaves it hashes as rows finish. The additive NTT alone already runs as fast on AVX2 as `riscv-exploration`'s, so most of the AVX2 cost here is Keccak (below). +4. The bus: GKR, then the grind and the leaves. + +Execution and witness building run at the same speed on every target. + +## Ideas, by expected gain + +### 1. An AVX2+GFNI arm for the Keccak skip pass + +- Today: `keccak_gkr/src/kernels.rs:255` selects `skip_gfni::SkipBlock` only with AVX-512F, AVX-512BW and GFNI. Every AVX2 target falls back to embedding each position and multiplying by CLMUL, which is the AVX2 kernel furthest behind AVX-512. +- In riscv: every byte-field kernel has an AVX2 arm alongside its AVX-512 one, the same algorithm two registers wide (`riscv:crates/flock/src/zerocheck/round1.rs:711`, `product_bytes_avx2`, beside `product_bytes_gfni` at `:665`). +- The port: `skip_gfni.rs` uses only `gf2p8affineqb`, `gf2p8mulb`, XOR, loads, stores and broadcasts, all of which exist at 256 bits under GFNI with AVX. Each `__m512i` becomes a pair of `__m256i`. The 24 residues are three ZMM registers, so six YMM; keep the per-slot loop's live set within 16 registers, as riscv's eight-byte output tiles do (`riscv:round1.rs:930`). +- Gain: the largest of this list on the GFNI tier. None on the other two tiers. + +### 2. An AVX2+GFNI arm for the Keccak tables + +- Today: `keccak_gkr/src/prover.rs:415` builds `table_gfni::RowMaps` only on AVX-512 with GFNI. Elsewhere each position is embedded, multiplied by the fold weights, written to a `6 * BITS`-word scratch and reduced from there (`prover.rs:460`). +- In riscv: the lift and weighting of packed bytes are GF(2)-linear maps applied 32 lanes at a time with `_mm256_gf2p8affine_epi64_epi8`, eight output bytes at a time so the accumulators stay in registers (`riscv:round1.rs:930` `convert`, `riscv:crates/primitives/src/bit_fold/avx2.rs:55`). +- The port: `table_gfni.rs`'s AVX-512 operations are the affine map, unpacks, `shuffle_i64x2` (`vperm2i128` or `vpermq` at 256 bits) and one `ternarylogic` (two XORs). The scratch pass goes away with it. +- Gain: medium, GFNI tier only. + +### 3. The bus GKR's vertical kernel on AVX2+VPCLMULQDQ + +- Today: `lean_vm_core/src/gkr.rs:44` (`products_into`) and `:518` (`fold_and_message`) use the transposed eight-lane kernel only on AVX-512, through `field::x8::F128x8`. AVX2 runs the scalar-shaped row loops, calling `mul2` per pair. +- In riscv: the GKR is written once over `field::lanes::Lane`, which on x86 is the AVX2+VPCLMULQDQ type even where AVX-512 exists (`riscv:crates/primitives/src/field/lanes.rs:19`). Products are gathered into lanes, kept there across the layer, and accumulated unreduced (`riscv:crates/leanvm_core/src/gkr/mod.rs:48`, `gkr/layer.rs:231`). +- The port: `field::f128x8::F128x8` already has an AVX2+VPCLMULQDQ arm (`primitives/src/field/f128x8.rs:20`), two `x4` halves. Write `gkr.rs`'s `vertical` module over it instead of `x8`, gate it on AVX2 with VPCLMULQDQ, and check for spills, since one AVX-512 plane register becomes two YMM registers. `products_into` alone, which also builds the leaves, is the smaller first step. +- Gain: medium on the VPCLMULQDQ and GFNI tiers. + +### 4. An AVX2 bit transpose, with a GFNI form + +- Today: `primitives/src/bits.rs:16` has an AVX-512 VBMI+GFNI arm, a NEON arm, and a portable gather for everything else, AVX2 with GFNI included. It runs per eight words in the level-0 lane projection (`pcs/src/stack_open/basis.rs:422`) and in the Keccak openings (`keccak_gkr/src/open.rs:129`, `:278`). +- In riscv: `riscv:crates/primitives/src/bits.rs:182` gathers the byte columns with two loads, two unpack rounds, a `vpermq` and a `vpshufb`. Then it applies either one `gf2p8affineqb` per register (`:203`) or the three masked-swap rounds on four words at once (`:217`). +- The port: the same 64-byte contract and bit order, so copy the functions and add the two `cfg` arms. Both forms beat the portable gather in a microbenchmark on one core, the GFNI form by far. +- Gain: low to medium, on every tier. The projection also spends time in `fold_columns` and the pattern dots, which this leaves unchanged. + +### 5. Row-major `BitFold::fold_block` on AVX2 + +- Today: with or without GFNI, AVX2 folds row-major blocks through scalar subset-sum lookups, one 256-entry `F128` table per input byte (`primitives/src/bit_fold.rs:178`, and `:753` under GFNI). Only column-major input reaches the AVX2 nibble and GFNI-256 kernels. +- In riscv: rows are transposed into 32-byte columns, then go through the same byte maps (`riscv:crates/primitives/src/bit_fold/avx2.rs:198` `transpose_rows`, `:308` `fold_block`). Without GFNI, each nibble table is stored once as 16 bytes and broadcast to both lanes (`:98`), not stored twice. +- The port: put an AVX2 row transpose in front of the existing `fold_columns`. The consumer is the ring switch's dense lift (`pcs/src/ring_switch.rs:313`). +- Gain: low on these workloads, where the ring switch's spans are small. It grows with ring-heavy openings. + +### 6. Two sumcheck rounds per pass over a table + +- Today: the Keccak tail (`keccak_gkr/src/prover.rs:547`) and the bus GKR (`lean_vm_core/src/gkr.rs:446`) already fuse each fold with the next round's message. Each round still writes a half-size table that the next round reads. +- In riscv: a quadratic round computes the shared terms for two rounds and folds two pending challenges in one read of the table (`riscv:crates/flock/src/zerocheck/multilinear.rs:232`, `:562`). The quartic GKR does the same with a degree-(4, 4) grid (`riscv:crates/leanvm_core/src/gkr/layer.rs:82` `fold`, `:199` `second_round`). +- Gain: low to medium, and the effort is high. On AVX2 the quartic version's extra accumulators may spill, so try the quadratic Keccak tail first. + +### 7. Byte-field products without GFNI + +- Today: on the two tiers without GFNI, Product and Skip multiply by CLMUL. +- In riscv: the same byte-field kernels run without GFNI by shift-and-add byte products (`riscv:round1.rs:749`, `gf8_mul_vec32`). +- Whether residues with shift-and-add products beat the CLMUL path is open: try it on the PCLMULQDQ tier, where the CLMUL path is slowest. + +## Looked at and not proposed + +- Explicit AVX2 absorb and digest transposes for four-way Keccak (`riscv:crates/primitives/src/hash/x86.rs:85`): `keccak256`'s AVX2 batch takes the same time per permutation for 64-byte and 512-byte inputs, so absorbing is not what limits it. Keccak on AVX2 is bound by the permutation: no 64-bit rotate and no ternary logic. That is the main AVX2 cost in the commit, the extension commits and the bus grind, and none of it is in `riscv-exploration`, which hashes with BLAKE2s. +- Fewer live registers in the radix-8 NTT tile (`riscv:crates/pcs/src/ntt/kernels.rs:103`): on one core, the commit's encode alone already runs as fast here as `riscv-exploration`'s. +- Incremental twiddles (`riscv:crates/pcs/src/ntt/mod.rs:44`), SIMD for WHIR's deeper-level affine rounds (`pcs/src/whir.rs:572`), and composed ring-weight maps (`riscv:crates/pcs/src/ring_switch.rs:134`): the spans they would speed up are small on these workloads. +- Karatsuba with delayed reduction, VPCLMULQDQ-256 batching, split equality tables in the bus GKR, Gruen's equality factoring, and packed Keccak witnesses: this tree already has them. +- Flock's fixed small challenges and univariate skip: these change the protocol, so they are out of scope here. From 950333f4634d065778f5023c32120fa79fb2ac1e Mon Sep 17 00:00:00 2001 From: Kevaundray Wedderburn Date: Tue, 17 Nov 2026 22:13:59 +0000 Subject: [PATCH 2/2] doc: record which AVX2 ideas were done or rejected Each idea now says what came of it in kw/avx2-single-thread, and a new section lists what was done beyond the list and what limits the rest. Single thread, aggregate, minimum pass over runs, local Ryzen 9 7950X3D core 14, base 6cac93d7f -> kw/avx2-single-thread, s: --xmss 100 haswell 1.002 -> 0.799, vp 0.732 -> 0.607, gfni 0.704 -> 0.509, native 0.420 -> 0.376 --sphincs 40 vp 1.059 -> 0.887, gfni 1.015 -> 0.778 --xmss 400 vp 2.524 -> 2.134, gfni 2.374 -> 1.697 --- doc/perf/avx2-single-thread.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/doc/perf/avx2-single-thread.md b/doc/perf/avx2-single-thread.md index f34065f7fd..82cc9b73fc 100644 --- a/doc/perf/avx2-single-thread.md +++ b/doc/perf/avx2-single-thread.md @@ -29,6 +29,7 @@ Execution and witness building run at the same speed on every target. - In riscv: every byte-field kernel has an AVX2 arm alongside its AVX-512 one, the same algorithm two registers wide (`riscv:crates/flock/src/zerocheck/round1.rs:711`, `product_bytes_avx2`, beside `product_bytes_gfni` at `:665`). - The port: `skip_gfni.rs` uses only `gf2p8affineqb`, `gf2p8mulb`, XOR, loads, stores and broadcasts, all of which exist at 256 bits under GFNI with AVX. Each `__m512i` becomes a pair of `__m256i`. The 24 residues are three ZMM registers, so six YMM; keep the per-slot loop's live set within 16 registers, as riscv's eight-byte output tiles do (`riscv:round1.rs:930`). - Gain: the largest of this list on the GFNI tier. None on the other two tiers. +- Outcome: done (`kw/avx2-single-thread`). The direct two-register port spilled, so the AVX2 arm splits a row into three passes through memory. The other tiers got a cofactor formulation instead (below). ### 2. An AVX2+GFNI arm for the Keccak tables @@ -36,6 +37,7 @@ Execution and witness building run at the same speed on every target. - In riscv: the lift and weighting of packed bytes are GF(2)-linear maps applied 32 lanes at a time with `_mm256_gf2p8affine_epi64_epi8`, eight output bytes at a time so the accumulators stay in registers (`riscv:round1.rs:930` `convert`, `riscv:crates/primitives/src/bit_fold/avx2.rs:55`). - The port: `table_gfni.rs`'s AVX-512 operations are the affine map, unpacks, `shuffle_i64x2` (`vperm2i128` or `vpermq` at 256 bits) and one `ternarylogic` (two XORs). The scratch pass goes away with it. - Gain: medium, GFNI tier only. +- Outcome: done, now level with AVX-512 on GFNI. Without GFNI the same maps run as two `vpshufb` nibble lookups, which also beat the scratch pass. ### 3. The bus GKR's vertical kernel on AVX2+VPCLMULQDQ @@ -43,6 +45,7 @@ Execution and witness building run at the same speed on every target. - In riscv: the GKR is written once over `field::lanes::Lane`, which on x86 is the AVX2+VPCLMULQDQ type even where AVX-512 exists (`riscv:crates/primitives/src/field/lanes.rs:19`). Products are gathered into lanes, kept there across the layer, and accumulated unreduced (`riscv:crates/leanvm_core/src/gkr/mod.rs:48`, `gkr/layer.rs:231`). - The port: `field::f128x8::F128x8` already has an AVX2+VPCLMULQDQ arm (`primitives/src/field/f128x8.rs:20`), two `x4` halves. Write `gkr.rs`'s `vertical` module over it instead of `x8`, gate it on AVX2 with VPCLMULQDQ, and check for spills, since one AVX-512 plane register becomes two YMM registers. `products_into` alone, which also builds the leaves, is the smaller first step. - Gain: medium on the VPCLMULQDQ and GFNI tiers. +- Outcome: done. Once `x4` built with plain PCLMULQDQ, the same kernels and the leaf fill also ran on the Haswell tier. ### 4. An AVX2 bit transpose, with a GFNI form @@ -50,6 +53,7 @@ Execution and witness building run at the same speed on every target. - In riscv: `riscv:crates/primitives/src/bits.rs:182` gathers the byte columns with two loads, two unpack rounds, a `vpermq` and a `vpshufb`. Then it applies either one `gf2p8affineqb` per register (`:203`) or the three masked-swap rounds on four words at once (`:217`). - The port: the same 64-byte contract and bit order, so copy the functions and add the two `cfg` arms. Both forms beat the portable gather in a microbenchmark on one core, the GFNI form by far. - Gain: low to medium, on every tier. The projection also spends time in `fold_columns` and the pattern dots, which this leaves unchanged. +- Outcome: done. ### 5. Row-major `BitFold::fold_block` on AVX2 @@ -57,18 +61,21 @@ Execution and witness building run at the same speed on every target. - In riscv: rows are transposed into 32-byte columns, then go through the same byte maps (`riscv:crates/primitives/src/bit_fold/avx2.rs:198` `transpose_rows`, `:308` `fold_block`). Without GFNI, each nibble table is stored once as 16 bytes and broadcast to both lanes (`:98`), not stored twice. - The port: put an AVX2 row transpose in front of the existing `fold_columns`. The consumer is the ring switch's dense lift (`pcs/src/ring_switch.rs:313`). - Gain: low on these workloads, where the ring switch's spans are small. It grows with ring-heavy openings. +- Outcome: done, with the nibble tables stored once and broadcast. The no-GFNI column kernel was also retuned. ### 6. Two sumcheck rounds per pass over a table - Today: the Keccak tail (`keccak_gkr/src/prover.rs:547`) and the bus GKR (`lean_vm_core/src/gkr.rs:446`) already fuse each fold with the next round's message. Each round still writes a half-size table that the next round reads. - In riscv: a quadratic round computes the shared terms for two rounds and folds two pending challenges in one read of the table (`riscv:crates/flock/src/zerocheck/multilinear.rs:232`, `:562`). The quartic GKR does the same with a degree-(4, 4) grid (`riscv:crates/leanvm_core/src/gkr/layer.rs:82` `fold`, `:199` `second_round`). - Gain: low to medium, and the effort is high. On AVX2 the quartic version's extra accumulators may spill, so try the quadratic Keccak tail first. +- Outcome: rejected. The fused two-round block kernel needs 32 accumulators and spills on 16 YMM registers, so it ran slower than the rounds it replaces. The tail got faster another way: cheaper products by shifted multipliers, with B left unreduced. ### 7. Byte-field products without GFNI - Today: on the two tiers without GFNI, Product and Skip multiply by CLMUL. - In riscv: the same byte-field kernels run without GFNI by shift-and-add byte products (`riscv:round1.rs:749`, `gf8_mul_vec32`). - Whether residues with shift-and-add products beat the CLMUL path is open: try it on the PCLMULQDQ tier, where the CLMUL path is slowest. +- Outcome: residues without GFNI were not built, since a shift-and-add byte product costs dozens of AVX2 operations against one. Instead, Skip on these tiers takes cofactors once per pack, reduced modulo the minimal polynomial of `s`, which needs fewer carry-less products. For Product, Haswell takes `ones * a` by nibble lookups, and all AVX2 tiers pack two products per lane in registers and fold bucket sums by nibbles. ## Looked at and not proposed @@ -77,3 +84,12 @@ Execution and witness building run at the same speed on every target. - Incremental twiddles (`riscv:crates/pcs/src/ntt/mod.rs:44`), SIMD for WHIR's deeper-level affine rounds (`pcs/src/whir.rs:572`), and composed ring-weight maps (`riscv:crates/pcs/src/ring_switch.rs:134`): the spans they would speed up are small on these workloads. - Karatsuba with delayed reduction, VPCLMULQDQ-256 batching, split equality tables in the bus GKR, Gruen's equality factoring, and packed Keccak witnesses: this tree already has them. - Flock's fixed small challenges and univariate skip: these change the protocol, so they are out of scope here. + +## Outcomes outside the list + +- The absorb and digest transposes for four-way Keccak were done anyway, for a small gain. The PoW grind gained more: it now searches nonces with `keccak256::first_nonce`, which builds states directly in lanes and hoists the first round's nonce-independent work. That is shared, so both provers use it, and the result is the same smallest nonce. +- The NTT tile change gave nothing on the VPCLMULQDQ tiers, where the butterfly is bound by carry-less product throughput. The register-tiled kernels now also build with plain PCLMULQDQ, which helps the Haswell tier. +- WHIR's deeper-level affine rounds now run two lanes a register, and so does level 0's lane fold. Later level-0 rounds project whole `E` lanes by Karatsuba dots, and project `X F` through `X^T pi` where no ring folds it. +- Product on GFNI now sums values at six points by `gf2p8mulb` and converts back once per layer. +- Zero-padded Merkle leaves are hashed from their rows, with no staging copy. The bytecode share skips the stacked bytecode's zero words. +- What is left on AVX2 sits close to the carry-less multiplier's throughput (NTT, bus, Skip and Product without GFNI) or to the Keccak permutation's (commits, grind). On Zen 4 a 256-bit VPCLMULQDQ costs the same as a 512-bit one, so AVX2 gets half of AVX-512's product throughput.