Skip to content

doc: AVX2 single-thread performance ideas from riscv-exploration - #769

Closed
kevaundray wants to merge 2 commits into
leanEthereum:mainfrom
kevaundray:kw/avx2-single-thread-ideas
Closed

kevaundray wants to merge 2 commits into
leanEthereum:mainfrom
kevaundray:kw/avx2-single-thread-ideas

Conversation

@kevaundray

Copy link
Copy Markdown
Collaborator

Adds doc/perf/avx2-single-thread.md: performance ideas from riscv-exploration (06e5c9cd6) for the experimental prover on x86 with AVX2 and no AVX-512, proving on one thread. It only lists them; no code changes. Every idea keeps the proof bytes.

Ideas, by expected gain

  1. An AVX2+GFNI arm for the Keccak skip pass. skip_gfni is gated on AVX-512, but it only uses gf2p8affineqb, gf2p8mulb and XOR, which all have 256-bit forms. riscv ships the same byte-field kernel two YMM wide (flock/src/zerocheck/round1.rs, product_bytes_avx2).
  2. An AVX2+GFNI arm for the Keccak tables (table_gfni, also AVX-512 only). riscv applies the same affine-map conversion 32 lanes at a time, in 8-byte output tiles.
  3. The bus GKR's vertical kernel on AVX2+VPCLMULQDQ. Build it on field::f128x8::F128x8, which already has an AVX2 arm, instead of the AVX-512-only x8. riscv's GKR is written once over its AVX2 Lane.
  4. An AVX2 bit transpose, with a GFNI form, for the level-0 lane projection and the Keccak openings. Today AVX2 falls back to the portable gather.
  5. Row-major BitFold::fold_block on AVX2 (the ring switch's dense lift). Today it is scalar table lookups.
  6. Two sumcheck rounds per pass in the Keccak tail and the bus GKR.
  7. Byte-field products without GFNI (riscv's shift-and-add), as an experiment for the Haswell tier.

Proposals 1, 2 and the GFNI half of 4 need GFNI: Alder Lake and later Intel client cores. Proposal 3 needs VPCLMULQDQ: Zen 3 and those Intel cores.

Looked at and not proposed: Keccak absorb transposes (the time per permutation does not change with input length), NTT register pressure (on one core the encode already matches riscv's), and ideas already in main or ones that change the protocol.

Measurements

On an EPYC 9454P (Zen 4), one pinned core, LEANVM_NUM_THREADS=1. These are four builds of 6cac93d7f; the AVX2 binaries contain no zmm.

aggregate --xmss 100, ms, last of three passes:

span AVX-512 (native) AVX2+VPCLMULQDQ +GFNI haswell
Prove 542 979 930 1340
Keccak reduction 154 383 342 612
Skip 49 174 175 319
Product 50 116 77 149
Tables 20 49 49 81
Tail 15 21 21 38
Commit (encode + Merkle) 121 185 185 222
PCS open 151 237 230 287
level-0 Project 36 60 52 83
commits L1 to L3 49 81 81 94
Bus GKR 33 61 61 71
Bus leaves 20 30 30 50
Bus grind 22 34 34 34
Execute program 19 20 20 20

Prove, ms, same build order:

workload AVX-512 AVX2+VPCLMULQDQ +GFNI haswell
--xmss 100 542 979 930 1340
--sphincs 40 876 1410 1340 1870
--xmss 400 1790 3310 3110 4630

Skip trails AVX-512 by more than three times in every workload.

Supporting checks, one thread:

  • PCS throughput (PCS_LOG_N=22, rate 1/2):
    • The encode alone: 32.1 ms on main's AVX2, 33.8 ms on riscv's AVX2 build and 24.5 ms on AVX-512.
    • The whole commit on main's AVX2: 94.7 ms. riscv's, which hashes with BLAKE2s, takes 63.8 ms.
  • keccak256 hash_throughput on AVX2: 10.91 Mhash/s at 64 bytes (one permutation) and 2.75 Mhash/s at 512 bytes (four). That is the same time per permutation.
  • bit_transpose_64bytes, per 64-byte block: 7.25 ns for the portable gather, 4.34 ns for riscv's AVX2 form and 1.40 ns for its AVX2+GFNI form. All three outputs agree.

Ideas from riscv-exploration (06e5c9c) for the experimental prover on AVX2 without AVX-512, on one thread. Nothing changes in the code.

Measured on clifv-remote-04 (EPYC 9454P, Zen 4), one pinned core, LEANVM_NUM_THREADS=1, cargo 1.99.1. Four builds of 6cac93d: native (AVX-512), x86-64-v3 +pclmulqdq,+vpclmulqdq, the same +gfni,+aes, and target-cpu=haswell. No zmm in the AVX2 binaries.

aggregate --xmss 100, last of three passes, ms (avx512 / avx2+vpclmul / +gfni / haswell):
  Prove                   542 /  979 /  930 / 1340
  Keccak reduction        154 /  383 /  342 /  612
    Skip                   49 /  174 /  175 /  319
    Product                50 /  116 /   77 /  149
    Tables                 20 /   49 /   49 /   81
    Tail                   15 /   21 /   21 /   38
  Commit (encode+Merkle)  121 /  185 /  185 /  222
  PCS open                151 /  237 /  230 /  287
    level-0 Project        36 /   60 /   52 /   83
    commits L1-L3          49 /   81 /   81 /   94
  Bus GKR                  33 /   61 /   61 /   71
  Bus leaves               20 /   30 /   30 /   50
  Bus grind                22 /   34 /   34 /   34
  Execute program          19 /   20 /   20 /   20

Prove, ms, same order: --sphincs 40  876 / 1410 / 1340 / 1870; --xmss 400  1790 / 3310 / 3110 / 4630. In every workload Skip is the span furthest behind AVX-512, by more than three times.

PCS throughput bench, PCS_LOG_N=22, rate 1/2, one thread: the commit's encode alone takes 32.1 ms here on AVX2+VPCLMULQDQ, 33.8 ms on riscv-exploration's AVX2 build, 24.5 ms on AVX-512. The whole commit takes 94.7 ms on AVX2 (riscv-exploration: 63.8 ms with BLAKE2s), so Keccak leaf hashing is most of it.

keccak256 hash_throughput, AVX2, one thread: 10.91 Mhash/s at 64 bytes (one permutation), 2.75 Mhash/s at 512 bytes (four): the same time per permutation, so absorbing is not the limit.

bit_transpose_64bytes, 256 blocks in L1, one core: the portable gather 7.25 ns per block, riscv's AVX2 form 4.34 ns, its AVX2+GFNI form 1.40 ns; all three agree on random input.
TomWambsgans added a commit that referenced this pull request Nov 17, 2026
From PR #769's first idea ("An AVX2+GFNI arm for the Keccak skip pass"): four points per YMM register, one factor pair extended at a time; aggregate 1400 leanXMSS on Arrow Lake -0.9% (all cores), -0.8% (8 threads), -1.6% (4 threads), -1.0% (1 thread), Skip -16%; the AVX-512 kernel is unchanged (Zen 4 flat).

Co-authored-by: Kevaundray Wedderburn <kevtheappdev@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@TomWambsgans

Copy link
Copy Markdown
Collaborator

1e67491

Each idea now says what came of it in kw/avx2-single-thread, and a new section lists what was done beyond the list and what limits the rest.

Single thread, aggregate, minimum pass over runs, local Ryzen 9 7950X3D core 14, base 6cac93d -> kw/avx2-single-thread, s:
  --xmss 100   haswell 1.002 -> 0.799, vp 0.732 -> 0.607, gfni 0.704 -> 0.509, native 0.420 -> 0.376
  --sphincs 40 vp 1.059 -> 0.887, gfni 1.015 -> 0.778
  --xmss 400   vp 2.524 -> 2.134, gfni 2.374 -> 1.697
@kevaundray

Copy link
Copy Markdown
Collaborator Author

The ideas are implemented in #775. The doc now records the outcome of each one, including the rejected ones.

@kevaundray kevaundray closed this Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants