Add XXHash64, XXHash3, and XXHash128 to smithy-checksums by jonathan343 · Pull Request #870 · smithy-lang/smithy-python

jonathan343 · 2026-10-25T04:39:05Z

Note

Stacked on #868, targeting feat/smithy-checksums-crc64nvme so this PR contains only the XXHash algorithms.

Summary

S3 also supports XXHash64, XXHash3 (64-bit), and XXHash128 checksums. This PR adds them to smithy-checksums as XXHash64, XXHash3, and XXHash128, also available through new(). The native side uses the twox-hash crate (MIT, added to LICENSE-THIRD-PARTY). The pure-Python side is a port of the xxHash reference implementation, limited to the seed of zero and default secret that checksums use. It runs at about 35 MB/s, faster than the pure-Python CRC32C.

Example

import base64

import smithy_checksums

for name in ("XXHASH64", "XXHASH3", "XXHASH128"):
    checksum = smithy_checksums.new(name)
    checksum.update(b"Hello world")
    print(checksum.name, checksum.digest_size, base64.b64encode(checksum.digest()))

The native and pure-Python builds print the same thing. These are also the x-amz-checksum-xxhash* header values botocore's tests expect for the same input:

xxhash64 8 b'xQCwyRKzdtg='
xxhash3 8 b'tqy52Eo4/3Q='
xxhash128 16 b'c1H4mBL5c4K5HQWzHgTdfw=='

Testing

The known answers include 15 of xxHash's own sanity-test vectors per algorithm, from tests/sanity_test_vectors.h. They cover every XXH3 size class and block boundary. Both implementations are also compared against the CRT and against each other:

uv run pytest packages/smithy-checksums -k "xxhash and (known_answers or matches_crt or native_matches_pure)" --no-cov -q

18 passed, 183 deselected in 0.42s

I also ran all 12,483 seed-zero vectors in that file once, covering every input length from 0 to 4,160 bytes. Both implementations matched every one.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

kuhe · 2026-10-27T00:49:29Z

please also review aws/aws-sdk-js-v3#8383 while this is fresh 😄

Adds the smithy-checksums package, which provides the checksum algorithms Smithy clients need for flexible checksums. CRC32 and CRC32C are backed by a native Rust extension (PyO3, crc-fast) with a pure-Python fallback, and SHA-1, SHA-256, SHA-512, and MD5 come from hashlib through new(). Wheels are built by an in-tree PEP 517 backend: maturin when a Rust toolchain is available, hatchling (pure Python) otherwise. Source distributions are always built with hatchling. SMITHY_PYTHON_NATIVE_BUILD (auto, always, never) overrides the choice. If Rust is present but too old, or the native build fails, the build stops with instructions instead of silently falling back. Native wheels target abi3 for 3.12+, cp314t, and abi3t for 3.15t+. Native and pure-Python checksum objects are safe to share between threads. Repository setup: - CI builds the native extension in the main matrix, checks Rust formatting, lints, dependencies (cargo-deny), and third-party license attribution (cargo-about), builds and tests native wheels on Linux, macOS, and Windows, and tests the pure-Python wheel and an sdist install without Rust. - Adds make targets check-rust, lint-rust, and third-party-licenses. - License header checks cover .rs and .pyi files. - Dependabot updates Cargo dependencies.

Pins the toolchain used for development and CI in rust-toolchain.toml, so a new stable release can't introduce clippy lints that fail CI without a change here. Builds from a source distribution are unaffected and use the installed toolchain. CI now caches Rust builds with Swatinem/rust-cache, saving caches only on develop, and adds a job that checks each native extension builds with the minimum Rust version its Cargo.toml declares. The new jobs also get names and version comments on their pinned actions.

The byte-at-a-time table loop runs about a dozen bytecodes per byte, which caps it at roughly 15 MB/s however it's arranged; slicing-by-8 and 16-bit tables only reach 20-35 MB/s. A CRC without its initial value and final XOR is linear over GF(2), so each bit of the CRC of a fixed-size chunk is the parity of the chunk's bits under a fixed mask. Input is now buffered into 16 KiB chunks, each read as one integer, and every CRC bit is computed with an AND and bit_count(), which do the per-byte work in C. The masks are built lazily by doubling, in a few milliseconds. Remainders under 1 KiB still use the table. CRC32C goes from about 16 MB/s to about 210 MB/s, whatever the update size.

maturin packages everything under python-source, so a native wheel built from a source tree that's been imported from picked up its __pycache__ directories, including bytecode for modules that no longer exist. The hatchling targets already exclude build artifacts; this gives maturin the same treatment.

The native extension read read-only buffers in place through an unsafe slice, on the grounds that nothing could change them. The read-only flag only stops this reader from writing, though: a read-only memoryview of a bytearray still changes when another thread writes to the bytearray, which can happen while the interpreter is detached. Only bytes is immutable, so it's the only input that's still hashed in place; every other buffer is copied. Copying runs at about 40 GB/s, against about 100 GB/s in place. Also: - State::digest returns a fixed-size array, given by an associated Digest type, instead of a Vec. - Import pyo3 items explicitly instead of through the prelude.

Adds CRC64NVME, the algorithm S3 uses for full-object checksums, backed by the native extension with a pure-Python fallback, and available through new("crc64nvme").

CRC64NVME shares the chunked big-integer path with CRC32C, which takes it from about 15 MB/s to about 135 MB/s. Add it to the test that checks that path against a bit-at-a-time CRC around the chunk and tail boundaries.

Adds the XXHash checksum algorithms S3 supports, backed by the native extension (twox-hash) with a pure-Python fallback ported from the reference implementation, and available through new(). The pure-Python implementations are checked against the native ones and against the CRT.

XXH3 and XXH128 key each block of input with one big-integer XOR, read the keyed words as 32-bit halves, and accumulate with an unrolled loop that masks the accumulators once per block instead of after every operation. That takes them from about 39 MB/s to about 74 MB/s. XXH64's rounds are a serial chain of 64-bit operations per lane, so instead its four lanes are packed into one integer, 128 bits apart, and each step of a round runs on all four at once. Every lane is multiplied by the first prime in one operation per 64 KiB. Inputs under 512 bytes, where spreading the input costs more than it saves, use inlined per-lane rounds. That takes XXH64 from about 35 MB/s to about 57 MB/s. The native-versus-pure test now covers inputs past 64 KiB.

jonathan343 added this pull request to stack #869 October 25, 2026 04:49

jonathan343 force-pushed the feat/smithy-checksums-xxhash branch from bfa1af9 to e1b2e71 Compare October 25, 2026 04:59

jonathan343 force-pushed the feat/smithy-checksums-xxhash branch from e1b2e71 to 6dc8617 Compare October 27, 2026 15:47

jonathan343 force-pushed the feat/smithy-checksums-xxhash branch from 6dc8617 to 9c97635 Compare October 27, 2026 19:11

jonathan343 added 5 commits October 27, 2026 15:13

Describe what smithy-checksums adds in its changelog entry

f585c4f

Make the Checksum protocol runtime-checkable

4a29f61

jonathan343 force-pushed the feat/smithy-checksums-xxhash branch from 9c97635 to 6f993da Compare October 27, 2026 19:15

jonathan343 added 6 commits October 27, 2026 16:45

Add CRC64NVME to smithy-checksums

c410416

Adds CRC64NVME, the algorithm S3 uses for full-object checksums, backed by the native extension with a pure-Python fallback, and available through new("crc64nvme").

Check the chunked pure-Python CRC64NVME against a bitwise CRC

eef1834

CRC64NVME shares the chunked big-integer path with CRC32C, which takes it from about 15 MB/s to about 135 MB/s. Add it to the test that checks that path against a bit-at-a-time CRC around the chunk and tail boundaries.

jonathan343 force-pushed the feat/smithy-checksums-xxhash branch from 6f993da to ac36924 Compare October 27, 2026 21:04

Navigation Menu

Repository navigation

Add XXHash64, XXHash3, and XXHash128 to smithy-checksums - #870

Add XXHash64, XXHash3, and XXHash128 to smithy-checksums#870
jonathan343 wants to merge 11 commits into
feat/smithy-checksums-crc64nvmefrom
feat/smithy-checksums-xxhash

jonathan343 commented Oct 25, 2026 •

edited

Loading

Uh oh!

kuhe commented Oct 27, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

jonathan343 commented Oct 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Example

Testing

Uh oh!

kuhe commented Oct 27, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jonathan343 commented Oct 25, 2026 •

edited

Loading