Skip to content

feat(cpc): add union result serialization - #335

Merged
tisonkun merged 2 commits into
apache:mainfrom
AhmedDusuki:perf/cpc-union-serialize
Nov 7, 2026
Merged

tisonkun merged 2 commits into
apache:mainfrom
AhmedDusuki:perf/cpc-union-serialize

Conversation

@AhmedDusuki

Copy link
Copy Markdown
Contributor

Addresses the empty and sparse union serialization paths in #319.

Summary

  • add CpcUnion::serialize() with output identical to to_sketch().serialize()
  • serialize empty and sparse union results without materializing an intermediate sketch
  • preserve the effective lg_k, seed hash, and merged-result metadata
  • retain the existing result-sketch conversion for bit-matrix unions
  • add serialization regression coverage and paired benchmarks

Details

For sparse unions, to_sketch() clones the accumulator solely to mark the result as merged before encoding it. CpcUnion::serialize() now reuses the sketch encoder with merged-result metadata while borrowing the accumulator. Empty unions use the same direct path.

Bit-matrix unions continue through to_sketch().serialize(). Encoding their window and surprise pairs directly can remain a separate follow-up as described in #319.

The new method produces byte-for-byte identical serialized images to the existing path across empty, sparse, hybrid, pinned, and sliding states. The tests also cover custom seeds, reduced precision, deserialization, and subsequent union updates.

Performance

Release benchmarks compared both implementations from the same PR build on an AMD Ryzen 5 2600X with Rust 1.98.1 on Windows. Each comparison used 100 repetitions; times are medians in milliseconds, with the full observed range shown in parentheses.

lg_k=18 union state Rounds Via result sketch Direct Ratio
Empty 4,000,000 286 ms (278–327) 218 ms (212–263) 0.76x
Sparse, 20,000 values 500 265 ms (248–308) 266 ms (259–290) 1.00x
Bit matrix, 80,000 values 200 315 ms (308–377) 315 ms (307–344) 1.00x

Direct serialization reduced the empty-union median by 24%. Sparse-union and bit-matrix timings remained effectively unchanged. The bit-matrix case intentionally retains the existing result-sketch conversion path.

A temporary counting allocator measurement produced:

lg_k=18 union state Via result sketch Direct
Empty 1 allocation, 40 bytes 1 allocation, 40 bytes
Sparse, 20,000 values 11 allocations, 293,376 bytes 10 allocations, 162,304 bytes
Bit matrix, 80,000 values 14 allocations, 815,792 bytes 14 allocations, 815,792 bytes

For the sparse case, the direct path removes the accumulator clone, saving one allocation and 131,072 allocated bytes per serialization. Output bytes were identical in every case.

The benchmark cases are included in this PR. The measurements were run with:

cargo +1.98.1 x bench -- 'cpc::serde::serialize_union/0' --exact --repeat 100 --raw
cargo +1.98.1 x bench -- 'cpc::serde::serialize_union/20000' --exact --repeat 100 --raw
cargo +1.98.1 x bench -- 'cpc::serde::serialize_union/80000' --exact --repeat 100 --raw

Validation

  • full workspace test suite passed, including 145 cross-language serialization tests and 147 doctests
  • cargo x check passed
  • benchmark smoke tests passed
  • nightly Clippy passed with warnings denied
  • nightly formatting check passed
  • documentation build passed
  • typos and hawkeye check passed
  • git diff --check passed

AhmedDusuki and others added 2 commits November 7, 2026 03:03
Serializing a CPC union currently requires converting it to a sketch. For sparse unions, that clones the accumulator solely to mark the result as merged.

Add CpcUnion::serialize and reuse the sketch encoder with merged-result metadata. Preserve the existing serialized bytes, effective lg_k, and seed hash. Keep bit-matrix unions on the existing conversion path.

Add coverage for all CPC flavors, custom seeds, reduced precision, deserialization, subsequent merges, and serialization benchmarks.
Keep nonempty accumulators ready for result extraction and serialization, removing the encoder override for merged metadata. Document the state invariants beside the union representation and keep public documentation focused on the serialization contract.

Focus serialization tests on result equivalence and continued merging across precision changes, and retain paired benchmarks against serialization through a result sketch.

@tisonkun tisonkun left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your contribution! LGTM.

I push a follow-up and it should be now OK to merge :D

@tisonkun
tisonkun merged commit 4618600 into apache:main Nov 7, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants