Skip to content

experimental: AVX2 single-thread speedups (taken from riscv branch) - #775

Closed
kevaundray wants to merge 18 commits into
leanEthereum:mainfrom
kevaundray:kw/avx2-single-thread
Closed

kevaundray wants to merge 18 commits into
leanEthereum:mainfrom
kevaundray:kw/avx2-single-thread

Conversation

@kevaundray

@kevaundray kevaundray commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Single-thread speedups for the experimental prover on x86 with AVX2 and no AVX-512, from the ideas in #769, which records what came of each one. Proof bytes are unchanged. AVX-512 and NEON arms keep their behaviour.

The branch keeps only the changes with a large gain. Eleven smaller commits, each worth a few ms at most, are on kevaundray:kw/avx2-single-thread-full if any are wanted back.

Results

Single thread (LEANVM_NUM_THREADS=1), aggregate --xmss 100, Ryzen 9 7950X3D pinned to one core, minimum pass over two runs of three, upstream main 28dcbbcbf against this PR, s:

tier upstream this PR change all 27 commits
haswell 1.043 0.864 −17% 0.855
x86-64-v3 + VPCLMULQDQ 0.774 0.694 −10% 0.672
+ GFNI 0.748 0.571 −24% 0.552

The three builds ran back to back on the same core, on a busier machine than the earlier runs, so compare within a row only. Each commit message has its own same-core A/B, taken on an EPYC 9454P (also Zen 4).

What changed, by stage

  • Keccak product, the largest gain on GFNI:
    • GFNI sums values at six points and converts back once per layer.
    • Without GFNI: whole-position parts, two products per lane in registers, and nibble-folded bucket sums.
    • Haswell takes ones * a by lookup.
  • Keccak tables: byte maps on every AVX2 target, by GFNI or vpshufb nibbles. The scratch pass is gone, and the maps run inside Exp/keccak tail fold #771's Round::pass.
  • Keccak skip pass without GFNI: cofactors once per pack, reduced modulo the minimal polynomial of s. With GFNI, upstream's 1e674918a kernel is kept.
  • Bus GKR: the AVX-512-only vertical kernels and the lane-wise leaf fill now run on AVX2. On Haswell they run through 128-bit pairs.
  • PCS level 0: the lane fold and the X products run two coefficients a register, Haswell included.
  • Bit kernels:
    • AVX2 and AVX2+GFNI bit_transpose_64bytes.
    • Row-major BitFold::fold_block goes through the column kernels.
  • PoW grind (shared keccak256): searches with first_nonce, which gives the same smallest nonce.
  • NTT on Haswell: the register-tiled sweeps also build with plain PCLMULQDQ.
  • Base encode lane padding (x86):
    • Rows are padded to 4 lanes instead of 8. That is 9 to 10% off Commit / NTT when the data lanes are not a multiple of 8, for example at --xmss 200 with 42 data lanes, where rows go from 48 to 44.
    • L0 Merkle leaves are hashed from their data lanes only, which saves a permutation at some shapes: --xmss 150 Commit / NTT 176 → 152 ms on GFNI.
    • Hashed bytes are unchanged. Every committed-cell reduction now pays at a 4-lane granularity instead of 8.

Tried and rejected, with numbers in #769:

  • Two sumcheck rounds per pass on the old tail. It spilled on 16 YMM registers, and Exp/keccak tail fold #771 has since landed a different version.
  • Residues without GFNI.
  • Reshaping the NTT tiles on VPCLMULQDQ.
  • A two-group Keccak in assembly.

Checks

  • Release tests pass on haswell, x86-64-v3 + VPCLMULQDQ, + GFNI and native Zen 4:
    • keccak256, fiat_shamir_prover, the experimental crates, stable_lean_vm_core, and leanvm-cli, which includes identical_proofs.
  • clippy --workspace --all-targets -D warnings is clean on CI's x86 targets (x86-64, alderlake, znver3, x86-64-v4, znver4) and on haswell.
  • On aarch64, neoverse-n2 and apple-m1, only the library crates were clippy-checked, because jemalloc needs a cross compiler that wasn't available.
  • cargo fmt is clean.
  • NEON code was not run.

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Benchmarks: base against this PR, on one runner

08209e9 merged into its base, against the base 6a828ec: each benchmark run in turns on one runner, base and PR 5 times each. A size is shown when it changed; a time or a peak memory when it moved the same way in every round and its median by at least 1% (the change is the median ratio, with the rounds' range).

Threads shows the named configuration; default means no thread count is encoded in the benchmark ID.

Keccak GKR

benchmark threads runner measure base this PR change
keccak-hash-batch-262144 default arm64 Proving time 2.70 s 2.66 s -1.63% (-3.21% to -1.36%)
keccak-hash-batch-262144 default x86-64 Proving time 3.77 s 2.82 s -25.58% (-25.65% to -24.07%)
Every result

Keccak GKR

benchmark threads runner CPU measure base this PR change
keccak-hash-batch-262144 default arm64 Neoverse-N2 Proving time 2.70 s 2.66 s -1.63% (-3.21% to -1.36%)
keccak-hash-batch-262144 default x86-64 AMD EPYC 9V45 96-Core Processor Proving time 3.77 s 2.82 s -25.58% (-25.65% to -24.07%)

Without AVX-512 VBMI, gather each byte column with two unpack rounds, a qword
permute and a byte shuffle, then transpose the bits of four columns per YMM:
with GFNI one affine map (rows reversed by the gather), without it the three
masked-swap rounds. The consumers are the PCS level-0 projection and the
Keccak openings.

Throwaway microbench, one 64-byte block, single thread on an EPYC 9454P
(Zen 4) core: portable 7.43 ns, AVX2 (haswell) 4.48 ns, AVX2+GFNI 1.33 ns.
On AVX2 without the AVX-512 GFNI kernel, the row-major fold transposes its
rows into byte columns (16x16 in-lane byte transposes, rows of 8 bytes paired
across the two halves) and runs the column kernel, instead of one 256-entry
subset-sum lookup per byte, so the byte tables are no longer built there.

The nibble kernel (AVX2 without GFNI) stores each 16-entry table once and
broadcasts it to both lanes, which halves its table footprint, and writes its
output through the same 16x16 byte transpose as the GFNI-256 kernel instead of
a word buffer. Rows of 128 bytes, whose tables outgrow L1, take four output
bytes of both 32-row halves at a time so each table is read once per block.

Throwaway microbench, ns per 64-row block, single thread, EPYC 9454P core 40,
min of runs:
  fold_columns, nibble (haswell)  C=16 256 -> 205, C=32 634 -> 386,
                                  C=64 1502 -> 899, C=128 2775 -> 2024
  fold_block, nibble (haswell)    C=16 349 -> 228, C=32 1229 -> 437
  fold_block, GFNI-256            C=16 327 -> 130, C=32 2619 -> 245

aggregate --xmss 100, LEANVM_NUM_THREADS=1, core 40, both commits against
upstream main, min over six alternating runs, ms (base -> new):
  vp:   Prove 979 -> 971, Project 59.4 -> 54.4, Openings 7.6 -> 5.4,
        Lift 1.5 -> 1.2
  gfni: Prove 931 -> 923, Project 52.1 -> 49.0, Openings 5.6 -> 3.9,
        Lift 1.5 -> 1.0
  hsw:  Prove 1330 -> 1330, Project 82.1 -> 77.0, Openings 7.5 -> 5.5,
        Lift 2.0 -> 1.7
The table pass's rows come from 32 x 16 byte maps per register of positions on every AVX2 target, not only AVX-512 with GFNI: one 256-bit affine product per map with GFNI, two vpshufb nibble lookups without. The 6 * BITS scratch and its reduction pass remain only off x86. The module is renamed table_gfni -> table_maps; the AVX-512 arm is unchanged but for sharing the unpack rounds.

Single thread, aggregate --xmss 100, last pass, Tables span, same core (16), base then candidate:
- gfni (x86-64-v3 + gfni): 49.1 -> 22.6, and 49.1 -> 23.3 on a repeat
- vp (x86-64-v3 + vpclmulqdq): 49.1 -> 33.7
- hsw (haswell): 81.9 -> 43.9
- native (Zen 4, AVX-512 arm): 19.5 -> 19.6
Rows alone, microbenchmark per 1600-position block: 11.3 us with GFNI, 22.3 us with nibble lookups.
The proof-of-work grind permuted 64 whole states through keccak_f_many, transposing each into lanes and back, and kept 25 output lanes to read one. `first_nonce` builds the states in lanes from the template, takes the nonce lane as a constant so LLVM hoists the first round's work on the untouched lanes out of the loop, and computes only the output lane in the last round. It returns the same smallest passing nonce, so proofs are unchanged; both provers grind through it.

Single thread, AMD EPYC 9454P core 12, same core, back to back.

Grind kernel, ns per permutation, keccak_f_many on 64 states before, first_nonce after: hsw 97.9 / 98.1 -> 85.6 / 81.4.

aggregate --xmss 100, last pass, ms, base -> this, two runs each:
- hsw: prove 1344 / 1355 -> 1320 / 1323; Bus grind 34.1 -> 27.5; PCS open 286 -> 272; Final level 12.9 -> 10.8
- vp: prove 983 / 992 -> 958 / 960; Bus grind 34.0 -> 27.5; PCS open 238 -> 223; Final level 13.1 -> 10.8
- gfni: prove 942 / 939 -> 915 / 915
- native: prove 551 / 552 -> 522 / 529; Bus grind 22.5 -> 15.7; PCS open 151 -> 135; Final level 8.6 -> 5.9

The prove totals include the next commit's AVX2 absorb, which moves only Commit.
@kevaundray
kevaundray force-pushed the kw/avx2-single-thread branch from a3361b4 to 4cd49cc Compare November 17, 2026 22:31
The transposed eight-lane kernels of the bus GKR (`products_into`, the round message and the fused fold) ran only with AVX-512. They now also run with AVX2 and VPCLMULQDQ, on `f128x8`'s AVX2 `F128x8`: two `x4` halves of four lanes. The message runs one half after the other on `x4::F128x4`, so its live set fits sixteen YMM registers; interleaving the halves, or looping over them, measured slower. `x4` gains `Sum4` (a lane sum of unreduced products, Karatsuba's terms summed before one fold), `F128x4::mul_sum` and `Mul`; `f128x8`'s AVX2 arm gains `load_runs`, `store_interleaved` and `halves`. The AVX-512 path is unchanged.

Single thread, core 32 of an EPYC 9454P, `-C target-cpu=x86-64-v3 +pclmulqdq,+vpclmulqdq` (vp), upstream main vs this, back to back:
- a 2^18-row layer of three trees (min of 8): first message 10.36 -> 6.20 ms, first fused fold 11.03 -> 7.18, second 5.24 -> 4.84, the rest 5.31 -> 3.80; `products_into` over 2^20 elements 1.75 -> 1.10 ms.
- `aggregate --xmss 100`, three pairs: Bus GKR 61.3 / 61.7 / 61.1 -> 40.4 / 40.0 / 41.4 ms; Bus leaves 29.9 -> 27.8 ms (the four- and sixteen-tuple products).
The radix-8 and radix-4 register kernels now also build where AVX2 has only
128-bit PCLMULQDQ (the haswell tier, and AVX-512 without VPCLMULQDQ), with
one four-lane vector per row in flight instead of two. Before, that tier ran
each butterfly through memory, a row pair at a time. The product is shared
with the row-pair kernel as `mul_xor_ymm`; the VPCLMULQDQ and AVX-512 arms
are unchanged, and the transposed butterfly keeps its old arms.

Single thread, `aggregate --xmss 100`, AMD EPYC 9454P core 28, same-core
A/B against upstream main, three alternating passes each, ms (base / new):

hsw (-C target-cpu=haswell):
  proving time      1346-1349 / 1318-1319
  Commit / NTT        208 / 195
  Commit L1          57.3 / 53.8
  Commit L2          25.6 / 24.5
  Commit L3          11.2 / 10.9
  Open L0            18.3 / 14.4-14.8
  Open L1             6.5 / 5.3

NTT alone (no hashing, jemalloc), hsw: base encode (2^18 x 48, 44 lanes)
86 -> 73, extension encodes 2^17/2^16/2^15 x 32 words 21.0/7.5/2.15 ->
17.5/6.3/1.87; kernel 2.86-3.0 -> 2.44-2.49 ns per four-lane butterfly,
against 2.43 for the same butterflies kept in registers.

vp and gfni compile to the same kernels: proving time 982/982-984 and
937-938/937, Commit / NTT 173/173-174 and 174/174.
Without GFNI the skip pass embedded and extended every position, then
took per slot and position a product of two embeddings and two products
of it with the weight's words: fifteen carry-less products a slot. Those
products bound it on these tiers.

`skip_clmul` takes a row's sum as `e[2] A + e[4] B + e[0] C`, the
cofactors `A = w[0] e[1] + w[1] e[3]`, `B = w[2] e[3] + w[3] e[0]` and
`C = w[4] e[1]` multiplied out on the 32 packs (ten products a pack) and
extended like them. A cofactor has three words, so a slot takes nine
products, each of an embedding and a word: 1490 a row instead of 1950.
The extension's binary points of new vectors are stored, and its low
mixed points summed as the products read them. With VPCLMULQDQ a
register's two lanes are the two groups at the same output; with
PCLMULQDQ alone the groups take a pass each. NEON and the portable form
keep the row kernel.

Measured on one core of an EPYC 9454P (Zen 4), single thread,
`aggregate --xmss 100`, last of three passes, ms, base and new run back
to back twice on the same core:

vp tier (x86-64-v3 + VPCLMULQDQ):
  span                     base    new   base2   new2
  Prove                     990    941     977    940
  Keccak reduction          397    348     384    348
  Keccak / Layer / Skip   174.4  139.1   174.2  138.8

hsw tier (haswell):
  span                     base    new   base2   new2
  Prove                    1330   1280    1330   1270
  Keccak reduction          611    553     612    551
  Keccak / Layer / Skip   318.5  258.1   318.9  258.0

`SkipBlock::run` on one random block (microbenchmark, best of 4 runs):
vp 194 us before, 152 us after; hsw 356 us before, 292 us after.

Tried and rejected on the vp tier (us per block, same core):
- The extension stored to a slot array, field by field, as 128-bit lanes
  (the products then reading two slots a register): 262 against 194.
- The extension stored to an output array a register at a time: 216
  against 194. Storing only the new vectors and summing the low points
  as the products read them is the kept 152.
- Residues without GFNI (nibble lookups for the affine maps, shift and
  add for the byte products): not built; a byte product costs about
  sixty AVX2 operations against one, and the products dominate.
The GFNI outer products staged each lane byte's products from CLMUL, a
word per pack, then transposed them to column bytes. A product of two
embeddings has degree below 48, so its values at six points of GF(2^8)
with distinct minimal polynomials fix it: the staging now takes the
embeddings' values by affine byte maps of the witness bytes, the
products' values by GF2P8MULB, and one 8x32 byte transpose per point.
The sums run over the value bits, and the inverse of the evaluation map
sends them back to coefficients once a layer.

Single thread, `aggregate --xmss 100`, Zen 4 core 24, same core back to
back, last pass, ms:

  span      gfni base  gfni  native base  native
  Product        89.0  49.6         50.7    39.3
  Keccak        399.0 352.0        160.0   144.0

Per block (pack_products microbench, n = 4096, gfni): 87.4 -> 51.2 us;
staging 49.7 -> 13.2 us, accumulation unchanged at about 30 us.
The digit buckets summed each block position's products in two parts of
three AVX2 registers, so each bucket's loop ran twice per digit. A part
of the whole position, six registers, runs it once.

Single thread, `aggregate --xmss 100`, Zen 4 core 24, two runs each,
same core back to back, Product span, ms:

  vp:  base 115.7 / 115.8      this 110.3 / 109.7
  hsw: with the next commit's PCLMULQDQ row products only 135.6 / 135.5,
       with this commit as well 127.7 / 127.9
The digit buckets' products are `ones * (a + c) + b * c`, two carry-less
products a pack. `ones * a` is a linear map of the pack, so with
PCLMULQDQ alone (a product a lane) a row now takes it once a position by
nibble lookups, an 8x32 byte transpose to a word a pack, and the products
of the row XOR two of them. That halves the carry-less products. With
VPCLMULQDQ the lookups cost more than they save (row products per block,
microbenchmark: 62.3 us with them, 54.4 us without), so that tier keeps
the products. `kernels::row_products` hands the five products of a row to
a closure, so the other arms keep their shape.

Single thread, `aggregate --xmss 100`, Zen 4 core 24, two runs each,
same core back to back, Product span, ms:

  hsw: base 148.9 / 148.9      lookups 135.6 / 135.5
       with the previous commit as well 127.7 / 127.9
  vp:  unchanged code path
The digit buckets' payloads hold a position's products 48 bits each. On
AVX2 every product went through memory as a word, then four at a time
through scalar shifts into three words, then into the payload. Now a
lane's two products come out of the carry-less products already packed
(the second shifted up six bytes), and each lane's twelve bytes are
stored straight into the payload, twelve bytes apart, the last lane's
without the four bytes past it (`kernels::row_packed`). With PCLMULQDQ
alone the nibble lookups of `ones * a` produce that packed layout too.
A part is a whole position on x86, so nothing else is copied; the other
arms keep products and `Part::pack`.

Single thread, `aggregate --xmss 100`, Zen 4 core 24, same core back to
back, Product span, ms:

  vp:  before 109.9 / 110.0           after 101.8 / 101.5
  hsw: before 129.5 / 128.4 / 135.4   after 119.7 / 119.3 / 119.5

pack_products microbench (n = 4096) per block: vp 122.7 -> 114.2 us,
hsw 143.3 -> 130.5 us; the row kernel alone per block: vp 49.7 -> 40.5
us, hsw 82.0 -> 63.8 us.
…e made

Each digit's 256 bucket sums were stored, then halved eight times into
the digit's rows, which with whole-position parts on AVX2 spills past
L1. Now each bucket's sum, made in registers, adds into the sum of its
value's high nibble (kept in registers across the 16 values that share
it) and into that of its low nibble (16 parts), and the two sets of 16
make the rows of the digit's high and low bits. Shared by every arm that
uses the digit buckets.

Single thread, `aggregate --xmss 100`, Zen 4 core 24, two runs each,
same core back to back, Product span, ms:

  vp:  before 101.2 / 103.1   after 93.2 / 93.5
  hsw: before 119.3 / 119.5   after 108.8 / 109.1

pack_products microbench (n = 4096) per block: vp 113.8 -> 104.8 us,
hsw 131.4 / 134.2 -> 121.9 / 121.4 us.
… AVX2

With VPCLMULQDQ and no AVX-512, the level-0 rounds wrote each folded lane
(`P = G + alpha F + beta X F`) and multiplied it by `X` through the plane form
of `F128x8`, which transposes eight elements to two coefficient planes on load
and back on store. `primitives::field::x4` gains `F128x2` and `F64x2`: two
elements a register in memory order, with Karatsuba (`Prod2`) and mixed
(`Mixed2`) products that reduce straight back to that order. `Folded::block`
and `mul_x_block` run on them, and an `E` block's `X` product multiplies only
its even columns, the odd columns' constant being one; a `K` block's gathers
its four even sums into one register.

Single thread, `aggregate --xmss 100`, last pass, same core, back to back, ms:

  vp:   base / new
    PCS open                      237 238 / 230
    Fold round L0 / Project       59.3 59.8 / 51.9
  gfni:
    PCS open                      233 / 222 224
    Fold round L0 / Project       52.8 / 44.2 44.2

hsw and AVX-512 compile the unchanged arms.
…CLMULQDQ

`primitives::field::x4` now builds with AVX2 and PCLMULQDQ, a 256-bit
carry-less product being two 128-bit ones where VPCLMULQDQ is missing
(`x4::clmul`). The level-0 lane fold (`Folded::block`) and the `X` products
(`mul_x_block`) take their `F128x2` arms on the haswell tier too, in
place of the portable eight-lane path. The leaf builder and `f128x8` keep
their VPCLMULQDQ gates.

Single thread, `aggregate --xmss 100`, last pass, same core, back to back,
ms (base is upstream main):

  hsw:  base / new
    PCS open                      289 288 / 279 282
    Fold round L0 / Project       82.7 82.2 / 74.0 75.1   (78.5 78.7 before)
…minimal polynomial

Only a block's skip sums at `s` matter: the prover evaluates them there.
So `skip_clmul` reduces each cofactor modulo the minimal polynomial of
`s` (Barrett's reduction from `PowerField`'s modulus, three carry-less
products) from three words to two, and a slot takes six products instead
of nine: per row 32 * 19 + 130 * 6 = 1388 products instead of 1490.
`SkipBlock::new` takes the `PowerField`; the other arms keep exact sums,
and the kernel test compares sums modulo the minimal polynomial.

`SkipBlock::run` on one random block (microbenchmark, best of 6 rounds,
one core of an EPYC 9454P, against kw/avx2-single-thread at e557ff7):
vp 154.9 us before, 145.0 us after; hsw 319.4 us before, 301.9 us after.
On an earlier tip of the branch: vp 152.3 to 142.8, hsw 291.9 to 275.7.

`aggregate --xmss 100`, single thread, last pass, Skip span in ms, ref
and new back to back twice on core 8 (host load average about 50):
  hsw: 262.7, 267.9 before; 255.1, 250.5 after.
  vp: unusable, the reference alone swung from 142 to 272.
…LMULQDQ

With `x4` building on AVX2 and PCLMULQDQ alone (a 256-bit carry-less product as two 128-bit ones), `f128x8`'s AVX2 arm, the bus GKR's vertical kernels (`products_into`, the round message, the fused fold) and the lane-wise leaf fill now run on the haswell tier too, in place of the portable eight-lane types and the scalar row loops. They are gated on AVX2 and PCLMULQDQ, with VPCLMULQDQ where AVX-512 is present (the `x8` path), so an AVX-512 machine without VPCLMULQDQ keeps the portable path.

Single thread, core 32 of an EPYC 9454P, `aggregate --xmss 100`, `target-cpu=haswell`, the previous commit vs this, two quiet pairs back to back, ms:
  Bus leaves          49.5 49.3 -> 35.5 34.5
  Bus GKR             72.8 72.4 -> 57.9 57.7
  Prove bus           156 156   -> 127 126
  Prove constraints   22.0 21.8 -> 19.8 20.3
  proving time        1138 1134 -> 1135 (noisy) 1103
The other stages are unchanged. vp, gfni and native compile to the same code as before.
@kevaundray kevaundray changed the title experimental: AVX2 single-thread speedups experimental: AVX2 single-thread speedups (taken from riscv branch) Nov 17, 2026
@kevaundray
kevaundray force-pushed the kw/avx2-single-thread branch from 4cd49cc to c528130 Compare November 17, 2026 22:46
The base commit encoded n_lanes data lanes as next_multiple_of(8), so the NTT only shrank when the committed cells crossed an eighth of the stack. Rows are now padded to `ntt::ROW_LANES`: four on x86 with AVX2 (AVX-512 included), eight on NEON and the portable build, where the kernels' tail goes word by word.

- The lane-major first gathered pass and the opening's rebuild of it move whole octets of lanes and then a quad where the row ends in one (`stage::lane_groups`), with a four-line stage tile.
- The AVX2 register kernels take the count of vectors that hold lanes, so a last chunk of one vector no longer multiplies a vector of zeros with VPCLMULQDQ (two vectors in flight).
- Leaf images are unchanged: the padding is the image's leading zeros, and only the data lanes ride the proof. A narrower row also absorbs more of the leading zeros as the shared zero-prefix state, so a leaf can take one Keccak permutation fewer.
- The base encode's span reports the data lanes beside the row width.

Single thread, EPYC 9454P core 44, `aggregate --xmss N`, three passes, A/B back to back against origin/kw/avx2-single-thread (ms):

| tier | N | data / row lanes | Commit/NTT | Open L0 | Commit L1..L3 NTT | total |
| gfni | 200 | 42: 48 -> 44 | 356 -> 323 | 13.5 -> 13.0 | 168 -> 169 | 1276 -> 1244 |
| gfni | 1300 | 43: 48 -> 44 | 1470 -> 1340 | 16.5 -> 15.7 | 693 -> 695 | 5811 -> 5666 |
| gfni | 1440 | 38: 40 -> 40 | 2480 -> 2490 | 370 -> 374 | 1412 -> 1424 | 8744 -> 8761 |
| hsw | 200 | 42: 48 -> 44 | 401 -> 364 | 18.7 -> 17.6 | 186 -> 186 | 2115 -> 2000 |
| hsw | 1300 | 43: 48 -> 44 | 1670 -> 1510 | 23.7 -> 21.7 | 775 -> 775 | 10321 -> 10179 |
| hsw | 1440 | 38: 40 -> 40 | 2840 -> 2830 | 512 -> 510 | 1588 -> 1592 | 14297 -> 14370 |
| native (AVX-512) | 200 | 42: 48 -> 44 | 227 -> 220 | 9.6 -> 9.6 | 102 -> 103 | 932 -> 928 |
| native (AVX-512) | 1300 | 43: 48 -> 44 | 941 -> 913 | 11.9 -> 12.1 | 425 -> 424 | 4422 -> 4382 |
| native (AVX-512) | 1440 | 38: 40 -> 40 | 1640 -> 1640 | 301 -> 301 | 866 -> 867 | 6555 -> 6568 |

At 1440 the row is 40 lanes either way, the no-regression shape; its differences are within the runs' spread (hsw ref 200 had 11% spread). On AVX-512 the same tree with eight-lane rows measured 225 ms at 200 and 942 ms at 1300, so four lanes is kept there too. The extension commits encode 16 lanes of data each and need no padding.
A base codeword row leads with zero lanes up to the NTT's lane quantum, and the leaf hasher counted only the leaf's own padding as the shared zero-prefix state. It now takes the data lanes (`MerkleBuilder::new` and `CutTree::open` gain `data_words`), so the whole zero prefix, the row's padding included, is one sponge state, and a leaf takes one Keccak permutation fewer whenever the padding crosses a rate block. The hashed bytes are unchanged; the extension commits pass their full row.

Single thread, EPYC 9454P core 44, `aggregate --xmss 150` (46 data lanes in rows of 48, leaf images 512 -> 376 bytes), two A/B rounds back to back against the previous commit (Commit/NTT is the last pass's span; total the three-pass mean):

| tier | Commit/NTT ms | total s |
| gfni | 176 -> 152 | 0.849, 0.835 -> 0.813, 0.801 |
| hsw | 196 -> 174 | 1.376, 1.371 -> 1.360, 1.352 |
| native (AVX-512) | 112 -> 98.4 | 0.609, 0.611 -> 0.596, 0.594 |

At `--xmss 200` (42 data lanes in rows of 44) both hash the same image; the totals matched within spread (gfni 1.253 both, hsw 2.091 -> 2.026 at 6% spread, native 0.935 -> 0.942 at 6% spread).
@TomWambsgans

Copy link
Copy Markdown
Collaborator
Machine All threads 8 threads 4 threads 1 thread
Zen 4 (8c/16t, AVX-512) −4.7% −4.8% −5.0% −6.1%
Arrow Lake (20c, AVX2) −4.0% −4.7% −6.5% −7.8%
Zen 5 (32c/64t, AVX-512) noisy −3.3% −2.3% (noisy) −5.4%
Apple M4 Max −3.3% −3.8% −4.2% not run

@TomWambsgans TomWambsgans mentioned this pull request Nov 18, 2026
TomWambsgans added a commit that referenced this pull request Nov 18, 2026
From #775: the grind permutes its candidates in lanes and reads the one lane it tests, rather than transposing whole states through keccak_f_many; same smallest nonce. 1400 leanXMSS at 8 / 4 threads: Zen 4 -1.3% / -1.2%, Arrow Lake -0.7% / -1.1%.

Co-authored-by: Kevaundray Wedderburn <kevtheappdev@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
TomWambsgans added a commit that referenced this pull request Nov 18, 2026
From #775: where AVX-512 VBMI with GFNI is missing, gather the byte columns with unpacks and a shuffle, then bit-transpose them by one GFNI affine map, or by three masked-swap rounds without GFNI. 1400 leanXMSS at 8 / 4 threads: Arrow Lake -0.7% / -1.2%.

Co-authored-by: Kevaundray Wedderburn <kevtheappdev@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
TomWambsgans added a commit that referenced this pull request Nov 18, 2026
From #775: the table pass builds its rows by byte maps, GFNI affine or vpshufb nibble lookups, on every AVX2 target rather than only on AVX-512 with GFNI; the x86 fold_groups arms it leaves unused go. 1400 leanXMSS at 8 / 4 threads: Arrow Lake -0.8% / -1.4%.

Co-authored-by: Kevaundray Wedderburn <kevtheappdev@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
TomWambsgans added a commit that referenced this pull request Nov 18, 2026
From #775: products_into, the round message and the fused fold run eight lanes at a time with VPCLMULQDQ on AVX2 too, as two x4 halves, with x8 and f128x8 sharing their method names. 1400 leanXMSS at 8 / 4 threads: Arrow Lake -0.5% / -2.0%.

Co-authored-by: Kevaundray Wedderburn <kevtheappdev@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
TomWambsgans added a commit that referenced this pull request Nov 18, 2026
From #775: the product sums are taken on each product's values at six points of GF(2^8), by byte maps and GF2P8MULB, rather than on its carry-less coefficients, which the inverse evaluation map recovers once a layer. 1400 leanXMSS at 8 / 4 threads: Zen 4 -2.4% / -3.2%, Arrow Lake -2.1% / -3.8%.

Co-authored-by: Kevaundray Wedderburn <kevtheappdev@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
TomWambsgans added a commit that referenced this pull request Nov 18, 2026
From #775: a digit-bucket part is a whole block position on AVX2 too, six registers, so each bucket's loop runs once per digit rather than twice and the half-position arm goes. Only AVX2 without GFNI (Zen 3, Haswell) runs the buckets; elsewhere it is a simplification.

Co-authored-by: Kevaundray Wedderburn <kevtheappdev@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
TomWambsgans added a commit that referenced this pull request Nov 18, 2026
…e made

From #775: each bucket's sum adds into the sums of its value's two nibbles as it is made, rather than into 256 sums halved eight times, so a digit's scratch is 32 parts on the stack. It pays where the buckets are the product path; 1400 leanXMSS on an Apple M4 Max at 8 / 4 threads: -0.8% / -0.2%.

Co-authored-by: Kevaundray Wedderburn <kevtheappdev@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
TomWambsgans added a commit that referenced this pull request Nov 18, 2026
* keccak256: search grinding nonces with first_nonce

From #775: the grind permutes its candidates in lanes and reads the one lane it tests, rather than transposing whole states through keccak_f_many; same smallest nonce. 1400 leanXMSS at 8 / 4 threads: Zen 4 -1.3% / -1.2%, Arrow Lake -0.7% / -1.1%.

Co-authored-by: Kevaundray Wedderburn <kevtheappdev@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* experimental primitives: AVX2 arms for bit_transpose_64bytes

From #775: where AVX-512 VBMI with GFNI is missing, gather the byte columns with unpacks and a shuffle, then bit-transpose them by one GFNI affine map, or by three masked-swap rounds without GFNI. 1400 leanXMSS at 8 / 4 threads: Arrow Lake -0.7% / -1.2%.

Co-authored-by: Kevaundray Wedderburn <kevtheappdev@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* experimental keccak_gkr: table rows by byte maps on AVX2

From #775: the table pass builds its rows by byte maps, GFNI affine or vpshufb nibble lookups, on every AVX2 target rather than only on AVX-512 with GFNI; the x86 fold_groups arms it leaves unused go. 1400 leanXMSS at 8 / 4 threads: Arrow Lake -0.8% / -1.4%.

Co-authored-by: Kevaundray Wedderburn <kevtheappdev@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* experimental lean_vm_core: bus GKR vertical kernels on AVX2+VPCLMULQDQ

From #775: products_into, the round message and the fused fold run eight lanes at a time with VPCLMULQDQ on AVX2 too, as two x4 halves, with x8 and f128x8 sharing their method names. 1400 leanXMSS at 8 / 4 threads: Arrow Lake -0.5% / -2.0%.

Co-authored-by: Kevaundray Wedderburn <kevtheappdev@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* experimental keccak_gkr: GFNI product sums on values at six points

From #775: the product sums are taken on each product's values at six points of GF(2^8), by byte maps and GF2P8MULB, rather than on its carry-less coefficients, which the inverse evaluation map recovers once a layer. 1400 leanXMSS at 8 / 4 threads: Zen 4 -2.4% / -3.2%, Arrow Lake -2.1% / -3.8%.

Co-authored-by: Kevaundray Wedderburn <kevtheappdev@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* experimental keccak_gkr: whole-position parts on AVX2

From #775: a digit-bucket part is a whole block position on AVX2 too, six registers, so each bucket's loop runs once per digit rather than twice and the half-position arm goes. Only AVX2 without GFNI (Zen 3, Haswell) runs the buckets; elsewhere it is a simplification.

Co-authored-by: Kevaundray Wedderburn <kevtheappdev@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* experimental keccak_gkr: fold digit bucket sums by nibbles as they are made

From #775: each bucket's sum adds into the sums of its value's two nibbles as it is made, rather than into 256 sums halved eight times, so a digit's scratch is 32 parts on the stack. It pays where the buckets are the product path; 1400 leanXMSS on an Apple M4 Max at 8 / 4 threads: -0.8% / -0.2%.

Co-authored-by: Kevaundray Wedderburn <kevtheappdev@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Tom Wambsgans <TomWambsgans@users.noreply.github.com>
Co-authored-by: Kevaundray Wedderburn <kevtheappdev@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@TomWambsgans

Copy link
Copy Markdown
Collaborator

merged here #782

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants