Skip to content

feat(theta,tuple): serialize union results directly - #336

Draft
tisonkun wants to merge 1 commit into
apache:mainfrom
tisonkun:codex/theta-tuple-union-serialization
Draft

tisonkun wants to merge 1 commit into
apache:mainfrom
tisonkun:codex/theta-tuple-union-serialization

Conversation

@tisonkun

@tisonkun tisonkun commented Nov 7, 2026

Copy link
Copy Markdown
Member

Adds ThetaUnion::serialize(ordered) and TupleUnion::serialize(ordered) so callers can serialize union results directly. Tuple serialization borrows retained summaries, avoiding the clones and associated allocations required by to_sketch(ordered).serialize().

The shared union code uses explicit owned and borrowed entry collection, with shared theta filtering, nominal-capacity trimming, and ordering. Each family's compact encoder lives in its serialization module and serves both compact sketches and unions. Theta output is byte-identical to to_sketch(ordered).serialize(); Tuple output is byte-identical when ordered and semantically equivalent when unordered, where entry order may differ.

Combines #322 and #333 by @Aschenbath, retaining contributor attribution in the commit.

Closes #320.
Closes #321.

Validation

  • cargo x prepare-testdata with the pinned C++, Go, and Java fixtures.
  • cargo x check across the feature matrix.
  • cargo x test: 812 tests, including doctests, passed.
  • cargo x lint and cargo +1.86.0 check -p datasketches --all-features passed.
  • cargo x bench -- serde --test passed, including all 40 new Theta/Tuple implementation cases.
  • Serialization coverage includes fresh/reset unions, exact and estimation modes, trimming, sampled inputs, non-empty results retaining no entries, custom seeds, summary clone counts, and subsequent merges of restored Tuple results.

Performance

Measured both paths at 0a287e2d2d296cdfb4a45d9e08430942452e154d on Apple M4 Max, macOS 26.6.2, Rust 1.101.0-nightly (8006091eb). Commands: cargo x bench -- theta::serde --raw and cargo x bench -- tuple::serde --raw. Each scenario reuses its union for the registered number of serializations; timing includes fixture construction and output destruction. Results below are median milliseconds and min–max over five interleaved repetitions.

Workload Rounds Via compact sketch Direct
Theta, unordered, 40 inputs 1,000,000 237 (233–249) 234 (229–248)
Theta, ordered, 8,000 inputs 10,000 323 (321–323) 321 (321–324)
Tuple labels, unordered, 100 20,000 244 (243–249) 44 (43–46)
Tuple labels, ordered, 80,000 500 379 (375–385) 73 (72–73)
Tuple u64, unordered, 80,000 5,000 124 (123–129) 115 (115–117)
Tuple u64, ordered, 80,000 5,000 251 (249–251) 272 (270–277)

The labels summary holds four strings per key. Its non-empty scenarios take roughly 17–29% of the compact-sketch path's time. Scalar Tuple results are mixed, including a roughly 9% slowdown in the ordered 80,000-input case. Theta ranges overlap. These measurements support the heap-backed-summary use case, not a general serialization speedup.

Add compact serialization to ThetaUnion and TupleUnion with shared result
filtering, trimming, and ordering. Reuse each family's compact encoder
and borrow Tuple entries to avoid cloning summaries during serialization.

Cover canonical results, custom seeds, summary clone avoidance, and
subsequent merges. Compare direct serialization with compact-sketch
serialization for scalar and heap-backed summaries.

Co-authored-by: Nash Aschenbath <nashaschenbath@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(tuple): serialize union results without cloning summaries feat(theta): serialize union results directly

1 participant