Repository navigation
Conversation
Add compact serialization to ThetaUnion and TupleUnion with shared result filtering, trimming, and ordering. Reuse each family's compact encoder and borrow Tuple entries to avoid cloning summaries during serialization. Cover canonical results, custom seeds, summary clone avoidance, and subsequent merges. Compare direct serialization with compact-sketch serialization for scalar and heap-backed summaries. Co-authored-by: Nash Aschenbath <nashaschenbath@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
ThetaUnion::serialize(ordered)andTupleUnion::serialize(ordered)so callers can serialize union results directly. Tuple serialization borrows retained summaries, avoiding the clones and associated allocations required byto_sketch(ordered).serialize().The shared union code uses explicit owned and borrowed entry collection, with shared theta filtering, nominal-capacity trimming, and ordering. Each family's compact encoder lives in its serialization module and serves both compact sketches and unions. Theta output is byte-identical to
to_sketch(ordered).serialize(); Tuple output is byte-identical when ordered and semantically equivalent when unordered, where entry order may differ.Combines #322 and #333 by @Aschenbath, retaining contributor attribution in the commit.
Closes #320.
Closes #321.
Validation
cargo x prepare-testdatawith the pinned C++, Go, and Java fixtures.cargo x checkacross the feature matrix.cargo x test: 812 tests, including doctests, passed.cargo x lintandcargo +1.86.0 check -p datasketches --all-featurespassed.cargo x bench -- serde --testpassed, including all 40 new Theta/Tuple implementation cases.Performance
Measured both paths at
0a287e2d2d296cdfb4a45d9e08430942452e154don Apple M4 Max, macOS 26.6.2, Rust 1.101.0-nightly (8006091eb). Commands:cargo x bench -- theta::serde --rawandcargo x bench -- tuple::serde --raw. Each scenario reuses its union for the registered number of serializations; timing includes fixture construction and output destruction. Results below are median milliseconds and min–max over five interleaved repetitions.The labels summary holds four strings per key. Its non-empty scenarios take roughly 17–29% of the compact-sketch path's time. Scalar Tuple results are mixed, including a roughly 9% slowdown in the ordered 80,000-input case. Theta ranges overlap. These measurements support the heap-backed-summary use case, not a general serialization speedup.