[tisonkun]

Count-Min derives row hashes differently across implementations, so serialized sketches can return incorrect estimates when read in another language. New sketches use serial version 2 hashing, while readers retain the legacy algorithm for version 1 images. The row seeds, bucket mapping, and version policy follow the proposed format in apache/datasketches-tck#18 (reply in thread).

• CountMinSketch::new and with_seed create version 2 sketches. Row seeds use XXH64(LE64(row_index), base_seed), items use XXH64, and buckets use (hash >> 1) % num_buckets.
• Deserialization selects the algorithm from the image. Version 1 sketches retain their legacy queries and updates and serialize back as version 1. Merging requires matching serial versions, dimensions, and seeds.
• Input methods retain the existing Hash contract. For portable inputs, use i64/u64, natural_extend wrappers or explicit 64-bit widening for narrower integers, and raw_bytes for strings and byte collections. The crate rejects big-endian targets at compile time.
• Empty inputs are not special-cased by the sketch. To match the proposed format's empty-input semantics, callers must skip empty string/byte updates and return zero for empty queries and bounds. Passing an empty raw_bytes value directly still hashes it and can update counters or return a nonzero estimate.

Compatibility

Version selection is internal to CountMinSketch; construction uses the existing API. Version 1 data remains readable and updatable with its original Rust input encoding. A new sketch cannot be merged with version 1 data, and version-1-only libraries cannot read new images. Migrating an existing sketch to version 2 requires rebuilding it from source items because the two algorithms map items to different counters. New sketches support 1–127 rows; readers still accept legacy version 1 images with more rows.

Cross-language version 2 interchange requires i64 counters, compatible item encodings, and caller-side empty-input handling. Bare strings, slices, narrow integers, and arbitrary custom Hash implementations are not automatically portable. Version 1 cross-language queries and updates remain unreliable.

Validation

The final PR head passed CI checks, benchmark smoke workloads, and tests on Linux, macOS, and Windows with both the MSRV and stable Rust.

Tests cover the published version 2 image and independent bucket vectors, including non-default seeds and value wrappers. Rust v0.5.0/v0.6.0 fixtures verify legacy behavior and serialization; existing C++ version 1 snapshots cover binary round-trips. Mixed-version merge rejection and the distinct constructor/legacy-reader row limits are also covered. Multi-implementation version 2 TCK snapshots remain a follow-up once available.

Refs: #297

[pmcgleenon]

1. Empty strings and byte arrays are counted.
Section 4 says an empty string or byte array is ignored: the update does nothing and a query returns 0. Here update_with_weight(raw_bytes::from_str(""), 5) adds 5 to the total weight and increments one counter per row, so the image differs from what Java and Go write for the same update stream, and estimate returns 5 where every other implementation returns 0.

The doc comment asks callers to skip these, but the spec indicates it should be handled by the implementation.

This fails today on assert!(sketch.is_empty()):

#[test]
fn v2_ignores_empty_strings_and_byte_arrays() {
    let mut sketch = CountMinSketch::<i64>::new(3, 1000).unwrap();
    sketch.update_with_weight(raw_bytes::from_str(""), 5);
    sketch.update_with_weight(raw_bytes::from_slice(&[]), 7);
    assert!(sketch.is_empty());

    sketch.update_with_weight(raw_bytes::from_str("apple"), 5);
    assert_eq!(sketch.estimate(raw_bytes::from_str("")), 0);
    assert_eq!(sketch.estimate(raw_bytes::from_slice(&[])), 0);
}

2. Bare narrow integers are not widened.
Section 4 says a narrower integer is widened to 64 bits first, sign-extended if signed. V2Hasher overrides only write_i64 and write_u64, so update(9i8) hashes a single byte and lands in buckets 406, 263, 103 instead of 490, 217, 440 (the 7.3 vector for integer 9).

#[test]
fn v2_widens_narrow_integers_to_64_bits() {
    let mut sketch = CountMinSketch::<i64>::new(3, 1000).unwrap();
    sketch.update_with_weight(9i64, 10);
    sketch.update_with_weight(-2i64, 20);

    assert_eq!(sketch.estimate(9i8), 10);
    assert_eq!(sketch.estimate(9u16), 10);
    assert_eq!(sketch.estimate(9usize), 10);
    assert_eq!(sketch.estimate(-2i32), 20);
    assert_eq!(sketch.estimate(-2isize), 20);
}

We could add something like this?

fn write_u8(&mut self, v: u8) { self.write_u64(u64::from(v)) }
fn write_u16(&mut self, v: u16) { self.write_u64(u64::from(v)) }
fn write_u32(&mut self, v: u32) { self.write_u64(u64::from(v)) }
fn write_usize(&mut self, v: usize) { self.write_u64(v as u64) }
fn write_i8(&mut self, v: i8) { self.write_i64(i64::from(v)) }
fn write_i16(&mut self, v: i16) { self.write_i64(i64::from(v)) }
fn write_i32(&mut self, v: i32) { self.write_i64(i64::from(v)) }
fn write_isize(&mut self, v: isize) { self.write_i64(v as i64) }

[pmcgleenon]

Maybe the doctests update can be a follow up PR?

• Doctests. Every example on update, update_with_weight, estimate, merge, serialize, deserialize, deserialize_with_seed, and the module docs still uses bare "apple", which the struct docs say is not portable. Switching them to raw_bytes::from_str or i64, as new already does, would stop the examples teaching the one pattern the docs warn against.

[tisonkun]

Updated to describe the merged implementation in 6856e45:

1. Empty inputs (review): the sketch does not automatically ignore empty encodings. Empty raw_bytes values are hashed like other inputs. To obtain the proposed format's semantics, callers must skip empty string/byte updates and return zero for empty queries and bounds. Filtering updates alone is insufficient because an empty query can collide with populated counters. The review's request for library-side empty-input handling is therefore not implemented in the merged code.
2. Integer widening (review): the generic Hash path does not automatically widen narrow integers. Portable callers use the existing natural_extend wrappers or explicitly widen to i64/u64 before updates and queries. Big-endian targets are rejected by the crate, so the 64-bit integer encoding is little-endian on supported targets.
3. Examples (review): examples use i64 items or raw_bytes strings, and the documentation identifies the wrappers needed for cross-language hashing.

The merged implementation uses XxHash64 directly and retains the generic Hash API without adding public methods. The prior claims in this reply about automatic empty-input handling and automatic integer widening do not describe the merged behavior.

The final PR head passed CI checks, benchmark smoke workloads, and tests on Linux, macOS, and Windows with the MSRV and stable Rust. Coverage includes published version 2 vectors, value wrappers, legacy fixtures, version preservation, mixed-version merge rejection, and version-specific row limits.