Repository navigation
Add CRC64NVME to smithy-checksums - #868
Draft
jonathan343 wants to merge 9 commits into
Draft
jonathan343 wants to merge 9 commits into
jonathan343 wants to merge 9 commits into
Conversation
jonathan343
force-pushed
the
feat/smithy-checksums-crc64nvme
branch
from
October 25, 2026 04:59
aeded0f to
cfcc9f2
Compare
jonathan343
force-pushed
the
feat/smithy-checksums-crc64nvme
branch
from
October 27, 2026 15:47
cfcc9f2 to
82838e3
Compare
Adds the smithy-checksums package, which provides the checksum algorithms Smithy clients need for flexible checksums. CRC32 and CRC32C are backed by a native Rust extension (PyO3, crc-fast) with a pure-Python fallback, and SHA-1, SHA-256, SHA-512, and MD5 come from hashlib through new(). Wheels are built by an in-tree PEP 517 backend: maturin when a Rust toolchain is available, hatchling (pure Python) otherwise. Source distributions are always built with hatchling. SMITHY_PYTHON_NATIVE_BUILD (auto, always, never) overrides the choice. If Rust is present but too old, or the native build fails, the build stops with instructions instead of silently falling back. Native wheels target abi3 for 3.12+, cp314t, and abi3t for 3.15t+. Native and pure-Python checksum objects are safe to share between threads. Repository setup: - CI builds the native extension in the main matrix, checks Rust formatting, lints, dependencies (cargo-deny), and third-party license attribution (cargo-about), builds and tests native wheels on Linux, macOS, and Windows, and tests the pure-Python wheel and an sdist install without Rust. - Adds make targets check-rust, lint-rust, and third-party-licenses. - License header checks cover .rs and .pyi files. - Dependabot updates Cargo dependencies.
Pins the toolchain used for development and CI in rust-toolchain.toml, so a new stable release can't introduce clippy lints that fail CI without a change here. Builds from a source distribution are unaffected and use the installed toolchain. CI now caches Rust builds with Swatinem/rust-cache, saving caches only on develop, and adds a job that checks each native extension builds with the minimum Rust version its Cargo.toml declares. The new jobs also get names and version comments on their pinned actions.
The byte-at-a-time table loop runs about a dozen bytecodes per byte, which caps it at roughly 15 MB/s however it's arranged; slicing-by-8 and 16-bit tables only reach 20-35 MB/s. A CRC without its initial value and final XOR is linear over GF(2), so each bit of the CRC of a fixed-size chunk is the parity of the chunk's bits under a fixed mask. Input is now buffered into 16 KiB chunks, each read as one integer, and every CRC bit is computed with an AND and bit_count(), which do the per-byte work in C. The masks are built lazily by doubling, in a few milliseconds. Remainders under 1 KiB still use the table. CRC32C goes from about 16 MB/s to about 210 MB/s, whatever the update size.
jonathan343
force-pushed
the
feat/smithy-checksums-crc64nvme
branch
from
October 27, 2026 19:15
82838e3 to
fec00fb
Compare
maturin packages everything under python-source, so a native wheel built from a source tree that's been imported from picked up its __pycache__ directories, including bytecode for modules that no longer exist. The hatchling targets already exclude build artifacts; this gives maturin the same treatment.
The native extension read read-only buffers in place through an unsafe slice, on the grounds that nothing could change them. The read-only flag only stops this reader from writing, though: a read-only memoryview of a bytearray still changes when another thread writes to the bytearray, which can happen while the interpreter is detached. Only bytes is immutable, so it's the only input that's still hashed in place; every other buffer is copied. Copying runs at about 40 GB/s, against about 100 GB/s in place. Also: - State::digest returns a fixed-size array, given by an associated Digest type, instead of a Vec. - Import pyo3 items explicitly instead of through the prelude.
Adds CRC64NVME, the algorithm S3 uses for full-object checksums, backed by
the native extension with a pure-Python fallback, and available through
new("crc64nvme").
CRC64NVME shares the chunked big-integer path with CRC32C, which takes it from about 15 MB/s to about 135 MB/s. Add it to the test that checks that path against a bit-at-a-time CRC around the chunk and tail boundaries.
jonathan343
force-pushed
the
feat/smithy-checksums-crc64nvme
branch
from
October 27, 2026 21:04
fec00fb to
eef1834
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
Stacked on #867, targeting
feat/smithy-checksumsso this PR contains only CRC64NVME.Summary
S3 uses CRC64NVME for full-object checksums, and it's the algorithm S3 computes when an upload doesn't specify one. This PR adds it to
smithy-checksumsasCRC64NVME, also available throughnew("crc64nvme"). It reuses the native extension and the pure-Python CRC implementation from #867. The native side usescrc-fast'sCrc64Nvme. The pure-Python side reuses the table-driven CRC that CRC32C uses, with the CRC-64/NVME polynomial.Example
The native and pure-Python builds print the same thing:
Testing
The known answers are the CRC-64/NVME check value for
"123456789"and the four 64-bit CRC test cases from the NVMe NVM Command Set specification. Both implementations are also compared against the CRT and against each other:uv run pytest packages/smithy-checksums -k "crc64nvme and (known_answers or matches_crt or native_matches_pure)" --no-cov -qBy submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.