diff --git a/doc/perf/avx2-single-thread.md b/doc/perf/avx2-single-thread.md new file mode 100644 index 0000000000..82cc9b73fc --- /dev/null +++ b/doc/perf/avx2-single-thread.md @@ -0,0 +1,95 @@ +# AVX2 single-thread performance ideas from `riscv-exploration` + +Ideas for the experimental prover on x86 machines with AVX2 and no AVX-512, proving on one thread (`LEANVM_NUM_THREADS=1`), taken from the `riscv-exploration` branch (`06e5c9cd6`). Each idea keeps the proof bytes, so `bins/leanvm/tests/identical_proofs.rs` checks it. Paths prefixed `riscv:` are in that branch; unprefixed ones are under `crates/experimental_prover/`. + +The ranking comes from the span trees of `cargo leanvm --prover experimental aggregate` at 100 and 400 leanXMSS and 40 leanSPHINCS signatures, built for one AVX-512 and three AVX2 targets on one Zen 4 core. The measurements are in the commit message. AVX2 targets come in three tiers, and some ideas need the third: + +|tier|flags|machines| +|---|---|---| +|AVX2 + PCLMULQDQ|`-C target-cpu=haswell`|Haswell to Skylake, Zen 1 and 2, CI's `haswell` testbed| +|+ VPCLMULQDQ|`-C target-cpu=x86-64-v3 -C target-feature=+pclmulqdq,+vpclmulqdq`|Zen 3| +|+ GFNI|the above with `+gfni`|Alder Lake and later Intel client cores, whose AVX-512 is off| + +## Where the time goes on AVX2 + +In order of share of the proof: + +1. The Keccak reduction (`keccak_gkr`), the largest stage and the furthest behind AVX-512. Within its layers, Skip falls furthest behind, then Tables and Product. Tail stays closest to AVX-512. +2. The PCS opening: level 0's lane projection (`Project`) and the extension commits of levels 1 to 3. +3. The commit: the base encode and the Keccak Merkle leaves it hashes as rows finish. The additive NTT alone already runs as fast on AVX2 as `riscv-exploration`'s, so most of the AVX2 cost here is Keccak (below). +4. The bus: GKR, then the grind and the leaves. + +Execution and witness building run at the same speed on every target. + +## Ideas, by expected gain + +### 1. An AVX2+GFNI arm for the Keccak skip pass + +- Today: `keccak_gkr/src/kernels.rs:255` selects `skip_gfni::SkipBlock` only with AVX-512F, AVX-512BW and GFNI. Every AVX2 target falls back to embedding each position and multiplying by CLMUL, which is the AVX2 kernel furthest behind AVX-512. +- In riscv: every byte-field kernel has an AVX2 arm alongside its AVX-512 one, the same algorithm two registers wide (`riscv:crates/flock/src/zerocheck/round1.rs:711`, `product_bytes_avx2`, beside `product_bytes_gfni` at `:665`). +- The port: `skip_gfni.rs` uses only `gf2p8affineqb`, `gf2p8mulb`, XOR, loads, stores and broadcasts, all of which exist at 256 bits under GFNI with AVX. Each `__m512i` becomes a pair of `__m256i`. The 24 residues are three ZMM registers, so six YMM; keep the per-slot loop's live set within 16 registers, as riscv's eight-byte output tiles do (`riscv:round1.rs:930`). +- Gain: the largest of this list on the GFNI tier. None on the other two tiers. +- Outcome: done (`kw/avx2-single-thread`). The direct two-register port spilled, so the AVX2 arm splits a row into three passes through memory. The other tiers got a cofactor formulation instead (below). + +### 2. An AVX2+GFNI arm for the Keccak tables + +- Today: `keccak_gkr/src/prover.rs:415` builds `table_gfni::RowMaps` only on AVX-512 with GFNI. Elsewhere each position is embedded, multiplied by the fold weights, written to a `6 * BITS`-word scratch and reduced from there (`prover.rs:460`). +- In riscv: the lift and weighting of packed bytes are GF(2)-linear maps applied 32 lanes at a time with `_mm256_gf2p8affine_epi64_epi8`, eight output bytes at a time so the accumulators stay in registers (`riscv:round1.rs:930` `convert`, `riscv:crates/primitives/src/bit_fold/avx2.rs:55`). +- The port: `table_gfni.rs`'s AVX-512 operations are the affine map, unpacks, `shuffle_i64x2` (`vperm2i128` or `vpermq` at 256 bits) and one `ternarylogic` (two XORs). The scratch pass goes away with it. +- Gain: medium, GFNI tier only. +- Outcome: done, now level with AVX-512 on GFNI. Without GFNI the same maps run as two `vpshufb` nibble lookups, which also beat the scratch pass. + +### 3. The bus GKR's vertical kernel on AVX2+VPCLMULQDQ + +- Today: `lean_vm_core/src/gkr.rs:44` (`products_into`) and `:518` (`fold_and_message`) use the transposed eight-lane kernel only on AVX-512, through `field::x8::F128x8`. AVX2 runs the scalar-shaped row loops, calling `mul2` per pair. +- In riscv: the GKR is written once over `field::lanes::Lane`, which on x86 is the AVX2+VPCLMULQDQ type even where AVX-512 exists (`riscv:crates/primitives/src/field/lanes.rs:19`). Products are gathered into lanes, kept there across the layer, and accumulated unreduced (`riscv:crates/leanvm_core/src/gkr/mod.rs:48`, `gkr/layer.rs:231`). +- The port: `field::f128x8::F128x8` already has an AVX2+VPCLMULQDQ arm (`primitives/src/field/f128x8.rs:20`), two `x4` halves. Write `gkr.rs`'s `vertical` module over it instead of `x8`, gate it on AVX2 with VPCLMULQDQ, and check for spills, since one AVX-512 plane register becomes two YMM registers. `products_into` alone, which also builds the leaves, is the smaller first step. +- Gain: medium on the VPCLMULQDQ and GFNI tiers. +- Outcome: done. Once `x4` built with plain PCLMULQDQ, the same kernels and the leaf fill also ran on the Haswell tier. + +### 4. An AVX2 bit transpose, with a GFNI form + +- Today: `primitives/src/bits.rs:16` has an AVX-512 VBMI+GFNI arm, a NEON arm, and a portable gather for everything else, AVX2 with GFNI included. It runs per eight words in the level-0 lane projection (`pcs/src/stack_open/basis.rs:422`) and in the Keccak openings (`keccak_gkr/src/open.rs:129`, `:278`). +- In riscv: `riscv:crates/primitives/src/bits.rs:182` gathers the byte columns with two loads, two unpack rounds, a `vpermq` and a `vpshufb`. Then it applies either one `gf2p8affineqb` per register (`:203`) or the three masked-swap rounds on four words at once (`:217`). +- The port: the same 64-byte contract and bit order, so copy the functions and add the two `cfg` arms. Both forms beat the portable gather in a microbenchmark on one core, the GFNI form by far. +- Gain: low to medium, on every tier. The projection also spends time in `fold_columns` and the pattern dots, which this leaves unchanged. +- Outcome: done. + +### 5. Row-major `BitFold::fold_block` on AVX2 + +- Today: with or without GFNI, AVX2 folds row-major blocks through scalar subset-sum lookups, one 256-entry `F128` table per input byte (`primitives/src/bit_fold.rs:178`, and `:753` under GFNI). Only column-major input reaches the AVX2 nibble and GFNI-256 kernels. +- In riscv: rows are transposed into 32-byte columns, then go through the same byte maps (`riscv:crates/primitives/src/bit_fold/avx2.rs:198` `transpose_rows`, `:308` `fold_block`). Without GFNI, each nibble table is stored once as 16 bytes and broadcast to both lanes (`:98`), not stored twice. +- The port: put an AVX2 row transpose in front of the existing `fold_columns`. The consumer is the ring switch's dense lift (`pcs/src/ring_switch.rs:313`). +- Gain: low on these workloads, where the ring switch's spans are small. It grows with ring-heavy openings. +- Outcome: done, with the nibble tables stored once and broadcast. The no-GFNI column kernel was also retuned. + +### 6. Two sumcheck rounds per pass over a table + +- Today: the Keccak tail (`keccak_gkr/src/prover.rs:547`) and the bus GKR (`lean_vm_core/src/gkr.rs:446`) already fuse each fold with the next round's message. Each round still writes a half-size table that the next round reads. +- In riscv: a quadratic round computes the shared terms for two rounds and folds two pending challenges in one read of the table (`riscv:crates/flock/src/zerocheck/multilinear.rs:232`, `:562`). The quartic GKR does the same with a degree-(4, 4) grid (`riscv:crates/leanvm_core/src/gkr/layer.rs:82` `fold`, `:199` `second_round`). +- Gain: low to medium, and the effort is high. On AVX2 the quartic version's extra accumulators may spill, so try the quadratic Keccak tail first. +- Outcome: rejected. The fused two-round block kernel needs 32 accumulators and spills on 16 YMM registers, so it ran slower than the rounds it replaces. The tail got faster another way: cheaper products by shifted multipliers, with B left unreduced. + +### 7. Byte-field products without GFNI + +- Today: on the two tiers without GFNI, Product and Skip multiply by CLMUL. +- In riscv: the same byte-field kernels run without GFNI by shift-and-add byte products (`riscv:round1.rs:749`, `gf8_mul_vec32`). +- Whether residues with shift-and-add products beat the CLMUL path is open: try it on the PCLMULQDQ tier, where the CLMUL path is slowest. +- Outcome: residues without GFNI were not built, since a shift-and-add byte product costs dozens of AVX2 operations against one. Instead, Skip on these tiers takes cofactors once per pack, reduced modulo the minimal polynomial of `s`, which needs fewer carry-less products. For Product, Haswell takes `ones * a` by nibble lookups, and all AVX2 tiers pack two products per lane in registers and fold bucket sums by nibbles. + +## Looked at and not proposed + +- Explicit AVX2 absorb and digest transposes for four-way Keccak (`riscv:crates/primitives/src/hash/x86.rs:85`): `keccak256`'s AVX2 batch takes the same time per permutation for 64-byte and 512-byte inputs, so absorbing is not what limits it. Keccak on AVX2 is bound by the permutation: no 64-bit rotate and no ternary logic. That is the main AVX2 cost in the commit, the extension commits and the bus grind, and none of it is in `riscv-exploration`, which hashes with BLAKE2s. +- Fewer live registers in the radix-8 NTT tile (`riscv:crates/pcs/src/ntt/kernels.rs:103`): on one core, the commit's encode alone already runs as fast here as `riscv-exploration`'s. +- Incremental twiddles (`riscv:crates/pcs/src/ntt/mod.rs:44`), SIMD for WHIR's deeper-level affine rounds (`pcs/src/whir.rs:572`), and composed ring-weight maps (`riscv:crates/pcs/src/ring_switch.rs:134`): the spans they would speed up are small on these workloads. +- Karatsuba with delayed reduction, VPCLMULQDQ-256 batching, split equality tables in the bus GKR, Gruen's equality factoring, and packed Keccak witnesses: this tree already has them. +- Flock's fixed small challenges and univariate skip: these change the protocol, so they are out of scope here. + +## Outcomes outside the list + +- The absorb and digest transposes for four-way Keccak were done anyway, for a small gain. The PoW grind gained more: it now searches nonces with `keccak256::first_nonce`, which builds states directly in lanes and hoists the first round's nonce-independent work. That is shared, so both provers use it, and the result is the same smallest nonce. +- The NTT tile change gave nothing on the VPCLMULQDQ tiers, where the butterfly is bound by carry-less product throughput. The register-tiled kernels now also build with plain PCLMULQDQ, which helps the Haswell tier. +- WHIR's deeper-level affine rounds now run two lanes a register, and so does level 0's lane fold. Later level-0 rounds project whole `E` lanes by Karatsuba dots, and project `X F` through `X^T pi` where no ring folds it. +- Product on GFNI now sums values at six points by `gf2p8mulb` and converts back once per layer. +- Zero-padded Merkle leaves are hashed from their rows, with no staging copy. The bytecode share skips the stacked bytecode's zero words. +- What is left on AVX2 sits close to the carry-less multiplier's throughput (NTT, bus, Skip and Product without GFNI) or to the Keccak permutation's (commits, grind). On Zen 4 a 256-bit VPCLMULQDQ costs the same as a 512-bit one, so AVX2 gets half of AVX-512's product throughput.