Fix multipart integrity and reduce encrypted read overhead - #156
Open
ServerSideHannes wants to merge 12 commits into
Open
ServerSideHannes wants to merge 12 commits into
ServerSideHannes wants to merge 12 commits into
Conversation
Bumps the security group with 1 update in the / directory: [h2](https://github.com/python-hyper/h2). Updates `h2` from 4.3.0 to 4.4.1 - [Changelog](https://github.com/python-hyper/h2/blob/master/CHANGELOG.rst) - [Commits](python-hyper/h2@v4.3.0...v4.4.1) --- updated-dependencies: - dependency-name: h2 dependency-version: 4.4.1 dependency-type: indirect dependency-group: security ... Signed-off-by: dependabot[bot] <support@github.com>
…4 updates Updates the requirements on [uvicorn[standard]](https://github.com/Kludex/uvicorn), [ruff](https://github.com/astral-sh/ruff), [boto3-stubs[s3]](https://github.com/youtype/mypy_boto3_builder) and [fakeredis](https://github.com/cunla/fakeredis-py) to permit the latest version. Updates `uvicorn[standard]` to 0.53.0 - [Release notes](https://github.com/Kludex/uvicorn/releases) - [Changelog](https://github.com/Kludex/uvicorn/blob/main/docs/release-notes.md) - [Commits](Kludex/uvicorn@0.52.4...0.53.0) Updates `ruff` from 0.16.5 to 0.16.8 - [Release notes](https://github.com/astral-sh/ruff/releases) - [Changelog](https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md) - [Commits](astral-sh/ruff@0.16.5...0.16.8) Updates `boto3-stubs[s3]` to 1.43.98 - [Release notes](https://github.com/youtype/mypy_boto3_builder/releases) - [Commits](https://github.com/youtype/mypy_boto3_builder/commits) Updates `fakeredis` from 2.37.1 to 2.38.0 - [Release notes](https://github.com/cunla/fakeredis-py/releases) - [Commits](cunla/fakeredis-py@v2.37.1...v2.38.0) --- updated-dependencies: - dependency-name: uvicorn[standard] dependency-version: 0.53.0 dependency-type: direct:production dependency-group: all-dependencies - dependency-name: ruff dependency-version: 0.16.8 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: all-dependencies - dependency-name: boto3-stubs[s3] dependency-version: 1.43.98 dependency-type: direct:production dependency-group: all-dependencies - dependency-name: fakeredis dependency-version: 2.38.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: all-dependencies ... Signed-off-by: dependabot[bot] <support@github.com>
ServerSideHannes
force-pushed
the
fix/integrity-and-streaming-performance
branch
from
September 28, 2026 12:27
2c68c39 to
58488e1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Multipart replacements could reuse GCM nonces or overwrite accepted bytes before signature validation. Completion could report success for the wrong upload, and stale or unavailable sidecars could select the wrong read path. This change binds publication and reads to an object generation and keeps each verified part attempt immutable.
Validation: the CI unit selection passed 779 tests locally (including 110 mock integration tests); the separate full unit-directory run passed 670 tests including its slow case. 27 real MinIO/HTTP compatibility tests passed (including the optional Redis scenario), 11 additional copy/concurrency tests passed, and all nine native-copy tests passed, including 1,280 MiB objects and concurrent copies. Ruff lint and format checks pass. Tests cover hash tampering below/at/above 8 MiB, rejected replacement, uncertain state publication, retry after failed completion, key selection, out-of-order assembly, resource cleanup, range recovery, and real backend preconditions.
A local ten-sample 32 MiB GET comparison against d45732d measured median latency of 317.12 → 289.26 ms and sampled RSS of 179.23 → 175.06 MiB. A separate 50-GET run stayed below 176 MiB sampled RSS. These are local measurements, not production forecasts; staging adds storage/copy work to ordinary multipart writes.
Deployment: this is a new write format. Drain legacy active uploads and switch the fleet together; old readers cannot read v3 generations. Persistent Redis is required to resume active uploads across restarts. Configure orphan-attempt lifecycle expiry longer than the supported upload/retry window. Generation manifests are retained because copies and versions may reference them. Unsupported checksum-trailer formats fail explicitly. No deployment or bucket-policy changes are included.
See docs/GENERATION_FORMAT.md for the commit protocol, migration requirements, cleanup policy, limitations and benchmark results; docs/CODE_REVIEW.md retains the original findings.
Also included (rebased onto main at f5e279c):
minio/minioandquay.io/minio/miniono longer allow anonymous pulls, which broke every integration shard and the daily Helm Install Test. Test fixtures and the Helm install workflow now use thepgsty/miniofork.e2e/suite is no longer tracked..s3proxy-internal/metadata and retries, but only when no client-visible object and no in-progress upload remains. Previously an emptied bucket still returnedBucketNotEmpty.STREAMING-UNSIGNED-PAYLOAD-TRAILERis accepted, and its CRC32, SHA1 or SHA256 trailer is checked against the decoded body before anything is published. This is what boto3 sends by default over HTTPS, so without it PutObject and UploadPart failed; verified with stock boto3 over TLS. CRC32C/CRC64NVME and the signed trailer mode are still rejected explicitly.Final CI status: all 11 pull-request checks passed for commit
58488e1; see the latest run for56b4143, and a manually dispatched Helm Install Test also passed on this branch (run).