fastaggregation: count before allocating in FastAnd

[tamirms]

Description

FastAnd with three or more inputs intersects them pairwise, allocating an intermediate container for each input and key, even when the final result is empty. This checks each key before allocating anything for it, and counts the result before building it.

This depends on #566, so the first commit here is #566's. It uses clearBitmapGaps, containerFromWords and the allocation-free run intersects from #566, and GitHub cannot base a pull request on a branch in a fork. Only the second commit, fastaggregation: count before allocating in FastAnd, is new here; the diff shrinks to it once #566 merges.

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Performance improvement
• Code refactoring
• Documentation update
• Test improvements
• Build/CI changes

Changes Made

What was changed?

• With three or more inputs, FastAnd walks the keys of the input with the fewest keys and looks each one up in the first four inputs, with a cursor per input on the stack. A key that any of them lacks is skipped without allocating.
• If that key's container is an array, it is also checked for a value it shares with each of those inputs, which usually stops at the first value. Runs and bitmaps skip this check, since for them it would cost about as much as the intersection.
• The containers of a key that survives are intersected without allocating until the result is known. Bitmaps are ANDed and counted in a stack scratch bitmap, and runs clear the values they don't cover. Arrays, and keys of runs only, use the existing and and iand, smallest array first. A container holding the whole key is skipped.
• Any further inputs are then intersected into the result one at a time, as before. Two inputs keep the existing And path.
• bitmapContainer.andArray is a quarter faster on amd64 and arm64 with its slice headers hoisted out of the loop, which And(bitmap, array) gets too.
• ParAnd's worker ran the same loop as the new andKChain and now calls it.
• Added BenchmarkFastAndShapes and BenchmarkRealDataFastAnd.

Why was it changed?

• A key with an empty result still allocated up to 8 KiB per step, and a run among bitmaps fell back to the same pairwise path.
• For dense inputs whose intersection is empty, the old path did all of the work and all of the allocation for nothing.

How was it changed?

• A key the first four inputs rule out costs no allocation. If they rule out every key, the call allocates only the result bitmap, however many inputs there are.
• Only four inputs are walked key by key: walking every input for every key made calls with many inputs whose keys survive up to 1.9x slower than pairwise And.
• The checks compare each input with the walked key's container, not all inputs together, so a key can pass them and still turn out empty. On the bitmap path that costs nothing; on the array path it costs one intersection.
• The result's slices are allocated once, at its first key, for up to 64 keys.

Testing

• fastand_kway_test.go compares FastAnd with pairwise And, which intersects the first two inputs and then each next input into the result, on 400 random cases of 3 to 12 inputs mixing array, bitmap and run containers, including whole-key ranges, over keys the inputs share but each sometimes lacks. About a third have a non-empty result, and the test requires more than a quarter.
• With up to 12 inputs, the input that drives the walk and the keys that die land both inside and past the first four.
• Edge cases, each kept because it catches a mutation the random cases miss: ranges that end on a word boundary on one side only, a many-interval run clipped by a narrower range, two ranges that overlap only where the bitmap is empty, ranges with no common value, two ranges spanning whole words, and whole-key runs.
• Results must pass Validate, use arrays and bitmaps by the usual size rule, and share no storage with the inputs.
• An empty intersection must allocate only the result, including for 1,000 single-value inputs.

go test ./...
go test -tags appengine ./...
GOOS=linux GOARCH=386 go build ./...    # also arm, arm64, ppc64

All pass on linux/amd64 (Xeon 8375C, with AVX-512 VBMI2), darwin/arm64, and amd64 without AVX-512.

Formatting

gofmt -s -l . and go tool unconvert ./... report nothing. go vet ./... reports only the existing ReadFrom signature warning from master.

Fuzzing

The FuzzSmat target in the template does not exist on master (smat.go refers to a missing smat_fuzz_test.go), so -fuzz=FuzzSmat matches nothing. The smat corpus runs and passes, but none of its sequences calls FastAnd:

go test -tags=gofuzz -run 'TestGenerateSmatCorpus|TestSmatHits'
ok  github.com/RoaringBitmap/roaring/v2

The randomized comparison under Testing is what exercises the change.

Performance Impact

BenchmarkFastAndShapes (added) compares FastAnd with pairwise And, which is what FastAnd did before, on the same inputs and in the same binary, so both use the same kernels. One key unless noted; the first seven rows add a whole-key range, like a query window, as the last input. Xeon 8375C:

	pairwise And	FastAnd	speedup
array30-bitmap	283 ns, 200 B, 6 allocs	277 ns, 188 B, 6 allocs	1.02x
array4000-bitmap	7.3 µs, 8328 B, 6 allocs	7.3 µs, 8316 B, 6 allocs	1.00x
array30-runs	482 ns, 168 B, 3 allocs	276 ns, 80 B, 1 alloc	1.75x
array4000-runs	4.3 µs, 8328 B, 6 allocs	4.4 µs, 8316 B, 6 allocs	0.99x
bitmap-bitmap	3.3 µs, 8336 B, 6 allocs	3.3 µs, 8324 B, 6 allocs	0.98x
runs-bitmap	4.0 µs, 4232 B, 6 allocs	3.5 µs, 4220 B, 6 allocs	1.12x
runs-runs	3.2 µs, 2176 B, 13 allocs	3.1 µs, 2168 B, 13 allocs	1.00x
jointly-empty	13.7 µs, 33200 B, 18 allocs	1.9 µs, 80 B, 1 alloc	7.1x
eight-keys	1.5 µs, 1056 B, 24 allocs	1.4 µs, 936 B, 20 allocs	1.04x
singletons-10000	26.0 µs, 106 B, 3 allocs	12.2 µs, 80 B, 1 alloc	2.1x
equal-x128	49.8 µs, 1056 B, 24 allocs	50.4 µs, 936 B, 20 allocs	0.99x
surviving-1024x256	13.1 ms, 33456 B, 535 allocs	13.3 ms, 32272 B, 522 allocs	0.98x

BenchmarkRealDataFastAnd (added), master against this PR. Each dataset's full intersection is empty, so this measures how quickly FastAnd finds that out:

	master	this PR	speedup
census-income	1.2 µs, 336 B, 12 allocs	270 ns, 80 B, 1 alloc	4.6x
census-income_srt	2.2 µs, 1400 B, 20 allocs	306 ns, 80 B, 1 alloc	7.3x
census1881	550 ns, 80 B, 1 alloc	259 ns, 80 B, 1 alloc	2.1x
census1881_srt	558 ns, 80 B, 1 alloc	262 ns, 80 B, 1 alloc	2.1x
dimension_003	51.2 µs, 80 B, 1 alloc	27.5 µs, 80 B, 1 alloc	1.86x
dimension_008	14.4 µs, 520 B, 32 allocs	7.7 µs, 520 B, 32 allocs	1.87x
dimension_033	24.1 µs, 2237 B, 93 allocs	248 ns, 80 B, 1 alloc	97x
uscensus2000	556 ns, 80 B, 1 alloc	251 ns, 80 B, 1 alloc	2.2x
weather_sept_85	17.6 µs, 15632 B, 43 allocs	374 ns, 112 B, 3 allocs	47x
weather_sept_85_srt	2.6 µs, 1024 B, 44 allocs	244 ns, 80 B, 1 alloc	11x
wikileaks-noquotes	653 ns, 128 B, 3 allocs	286 ns, 80 B, 1 alloc	2.3x
wikileaks-noquotes_srt	616 ns, 112 B, 3 allocs	282 ns, 80 B, 1 alloc	2.2x

Performance Analysis

• Empty intersections are 1.7x to 7x faster than pairwise And, and every real dataset is 1.9x to 97x faster than master. All but two datasets allocate only the result; dimension_008 and weather_sept_85 have keys that only a later input rules out, and allocate no more than before.
• When every key survives, FastAnd does the same intersections plus the walk and the checks, so those rows stay within a few percent of pairwise And.
• On an M1 the jointly-empty gain is smaller, 13.9 µs to 5.0 µs, because andCardSlice has no NEON version.
• BenchmarkIntersection*, BenchmarkUnion* and BenchmarkAndAny/* are unchanged.

Breaking Changes

None. Results are the same as with pairwise And.

Related Issues

Related to #566.

Additional Notes

arrayContainer.iandBitmap has the loop bitmapContainer.andArray had, and the same change would speed it up; that is left for a separate change.

🤖 Generated with Claude Code

[tamirms]

@guymolinari @lemire thanks for the quick turnaround on #570. Could you take a look at #566 and this PR when you have time? This one builds on #566 (its first commit is #566's), so #566 is the place to start. We use roaring in Stellar RPC, the RPC server for the Stellar network, and these two PRs significantly reduce allocations in our event queries, so I'd love to get them upstream rather than maintain a fork.

[guymolinari]

Thanks @tamirms. These look useful. I checked both changes against the base commit on Linux/amd64, using Go 1.24.4 on an AMD EPYC 9V74. I have two requests before approval:

#572: Avoid the large cursor allocation on early-empty intersections

I reproduced the dimension_003 allocation increase. The dataset contains 15,482 input bitmaps, so make([]int, len(bitmaps)) requests 123,856 bytes, rounded by Go to a 131,072-byte allocation. Together with the empty result, this explains the reported increase from 80 to 131,152 B/op.

The first two inputs already have an empty intersection, but the cursor array is allocated before discovering that. In three isolated runs, median time was approximately 37.5 us/op on the base implementation versus 55.0 us/op with #572. Allocation measurements were 80 B/op versus approximately 131,154 B/op.

Could you avoid allocating the full cursor workspace on this early-empty path and add a regression benchmark covering a large input count with an empty first-pair intersection?

#566: Benchmark fresh inputs for the in-place cases

The current in-place benchmark clones dense once and repeatedly intersects the same result. After the first iteration, it measures an already-filtered bitmap.

I reran both cases with a fresh clone before every iteration, excluding cloning from the timed section and checking each result against the expected intersection. Medians of three runs:

• Four long ranges: 2.05 us/op on base versus 0.577 us/op with runcontainer: count or test before allocating in run-bitmap intersections #566.
• 1,024 short intervals: 6.80 us/op on base versus 4.53 us/op with runcontainer: count or test before allocating in run-bitmap intersections #566.
• Both cases: temporary allocations fell from 8,224 B/op and 2 allocs/op to zero.

The gains hold up with fresh inputs. However, the four-range case measured approximately 258 ns/op with the reused input versus 577 ns/op with fresh inputs.

Could you update the in-place benchmarks and reported results to cover fresh inputs? A separate clone-plus-intersection benchmark would also help show the end-to-end cost.

The targeted run-container intersection and FastAnd tests passed in my checks.

[tamirms]

Thanks @guymolinari, both are addressed, and I fixed a few more things while I was there. Both PRs are rebased on current master, and the tables are re-measured on a Xeon 8375C.

#572, the cursor allocation. FastAnd now walks only the first four inputs key by key, with their cursors in a fixed array on the stack, and intersects the rest into the result one at a time, so there is nothing left to allocate. dimension_003 is back to 80 B/op and 1 alloc, as are 9 of the other 11 datasets in BenchmarkRealDataFastAnd. BenchmarkFastAndShapes has a singletons-10000 row, 10,000 inputs with an empty first pair, and TestFastAndEmptyAllocatesNoContainer asserts a single allocation for 1,000 of them.

#566, the in-place benchmark. The in-place cases now intersect a fresh copy each iteration, cloned in batches outside the timer, and new clone+ cases show the end-to-end cost. The table is updated.

Also changed:

• Walking every input for every key made FastAnd up to 1.9x slower than the plain pairwise And it replaces when keys survive many inputs (1,024 inputs of 256 keys). Walking only the first four removes that.
• A key with several many-interval runs no longer intersects the runs into a new container; each run clears its gaps in the scratch bitmap.
• The random FastAnd test now uses inputs that share keys, up to 12 of them, so about a third of its cases reach the intersection; before, most stopped at a missing key. The shape benchmark puts a real input first and uses real run containers.
• ParAnd's worker ran the same loop as the new andKChain and now calls it.
• In runcontainer: count or test before allocating in run-bitmap intersections #566, the scratch path is used only where it is faster: runs covering more than 4096 values, or CPUs with the AVX-512 vector fill. For runs of many short intervals it was up to 2.9x slower. An in-place And with a full range no longer copies a bitmap or intersects a run, and Intersects between an array and a run is up to 1.9x faster.
• Both descriptions are rewritten, and a few claims that did not hold are corrected.

[guymolinari]

Thanks @tamirms. I reviewed #566 at 1c0052a and #572 at fccdc63 and independently reran the relevant checks with Go 1.24.4 on Linux/amd64, AMD EPYC 9V74.

Both original concerns are resolved:

• fastaggregation: count before allocating in FastAnd #572: Fixed-size cursor storage removes the large early-empty workspace allocation. dimension_003 now measures 80 B/op and one allocation, with a three-run median of 16.5 µs versus 32.6 µs on current master. The 10,000-singleton benchmark also measures 80 B/op and one allocation.
• runcontainer: count or test before allocating in run-bitmap intersections #566: The in-place benchmarks consume fresh clones outside timing, and the clone+ cases expose end-to-end costs. Fresh-input medians improved from 1.76 to 0.317 µs for four long ranges and 6.98 to 4.62 µs for 1,024 intervals, with zero timed allocations in both revised cases.

Both full test suites passed, along with the focused appengine-build checks on #572. Additional late-driver, late-empty, copy-on-write, and 1,024-input/256-surviving-key checks passed. Longer runs of the surviving-key benchmark showed no reproducible regression.

The scratch masking, container selection, and result ownership paths look sound. Please consider retaining the larger surviving-key workload as a benchmark.

My previous objections are resolved; I’m comfortable approving both, with #566 merged first.

[tamirms]

Thanks @guymolinari. Done: BenchmarkFastAndShapes now has a surviving-1024x256 row, 1,024 inputs of 256 keys that all survive, and it's in the table in the description. On the Xeon it takes 13.3 ms against 13.1 ms for pairwise And, with fewer allocations.