Repository navigation
Store the evidence dataset as a release asset - #19
Merged
Merged
Conversation
…here bench/results/current/ held the whole measurement dataset in git: 1.3 MB in 17 files, of which traces.json and tuning.json are byte-exact pools of the raw batches. Every change to the measurement code made it stale and a re-measurement rewrote some 20,000 lines in a pull request. The owner decided that after v0.4.0 the dataset lives in a release asset and the repository keeps only what the documents link to. scripts/evidence_archive.py: - --package DATASET --release vX.Y.Z --archive FILE validates a recorded dataset and packs the archived artifacts - raw batches, the three evidence logs, reference and byte results and the measurement manifest - into a deterministic tar.gz (sorted names, no timestamps or owners, fixed gzip header). It writes bench/results/current/release-asset.json (release, asset name, download URL, SHA-256, measured commit) and the generated README. - --verify RESULTS downloads the asset, or reads AS_CACHE_EVIDENCE_ARCHIVE, requires the pinned SHA-256, extracts only the expected regular files by name, rebuilds the pooled results from the raw batches, validates every hash, command and observation count through record_evidence.verify_inputs, renders the README with the current generator and requires it to equal the committed copy. It refuses any other file in the results directory. - --render RESULTS regenerates the committed README from the asset. The archive carries neither the README nor the pooled files. The README is rendered at check time, so a wording change in the report generator needs only `make evidence-render`, never a new upload; verify_inputs gains check_report to skip the README hash for that path. make evidence-check now runs the asset verifier; make evidence-render is new. The renderer's provenance lines describe the new flow; the four changed lines are the only difference in the regenerated README, every table is unchanged. The 15 measurement files are removed from git. They were packed from the v0.4.0 tag's tree, so the asset is exactly the published dataset (measured at 1cb65fe): as-cache-evidence-v0.4.0.tar.gz, 65,793 bytes, SHA-256 d003b2886e635c3cb0df1b54d9b47bb1d292f6d69ee1ba8e81d15b476ec92aca, attached to the v0.4.0 release. The make-gate test moved from record_evidence_test.py to evidence_archive_test.py, rewritten for the new mechanism; it still asserts that the gate is read-only and rejects an edited README. Verified: - 10 new tests: deterministic packaging, round trip, changed archive bytes, extra / traversal / missing members, edited README, data left in the repository, a stale README hash inside the archive (generator change), a tampered raw batch with its own hash updated, mismatched commit, and an unreachable asset without a local copy. - Mutations, one at a time: skipping the hash check, the README comparison, the repository inventory or the commit match each fails its test. Skipping the member check at first passed, and the escaping member was written to the parent of the extraction directory; the test now requires the specific refusal and that nothing escapes, and fails under that mutation. - Packaging the v0.4.0 dataset twice gives identical bytes. - After uploading the asset, `make evidence-check` with no local copy downloads it and verifies: "README regenerated byte for byte". - make all: exit 0 in 98 s; golangci-lint 0 issues in 12 modules; 43 script and 12 release tests OK; 8 candidate consumer builds.
Links that pointed at measurement files in bench/results/current/ - the manifest, the three evidence logs, traces.json and tuning.json - name the release asset instead, since those files no longer exist in the repository. - docs/benchmarking.md: the recording procedure ends with packaging and `gh release upload`; a new "Where the data lives" section says what the repository and the asset each hold, what `make evidence-check` verifies, how to verify offline, that packaging is deterministic, that a report-generator change needs `make evidence-render` and no upload, and that an attached asset is never replaced. - docs/releasing.md: an "Evidence asset" section, and the proxy lesson from publishing v0.4.0 - probe for an unused version well before pushing tags, because a probe in the minute before the push gets "not found" cached for about thirty minutes. It also states an unsolved ordering problem plainly: for a release that changes measurements, the tagged commit should contain the new pointer, but the asset can only be attached once the release exists; until that is decided, a release with no new measurement keeps the previous pointer. - docs/evidence.md, docs/configuration.md, README.md: wording and links. - CHANGELOG [Unreleased]: the change, with the asset's size and hash.
The independent review of this branch found checks that no test held: - Duplicate members, symlinks and hard links are refused by the member check, but removing the duplicate check or isreg() left every test passing. - package() validates the dataset before packing, but skipping that left every test passing. - The pointer's release field is never checked: a URL under another release or host passes as long as it ends with the asset name. These tests fail now. - The results inventory counts ignored dotfiles, so a .DS_Store left by Finder fails `make evidence-check` on macOS. This test fails now. - Published assets carry the hashes of the pooled traces.json and tuning.json that verification rebuilds with combine(), so combine()'s byte output is now frozen for every attached asset: changing only its indentation made the real v0.4.0 asset fail with "artifact hash mismatch: traces.json". A golden-hash test pins it and says why. Red now: the pointer test (three cases) and the dotfile test. The other new tests pass and pin existing behaviour.
…n files The pointer check accepted any URL that ended with the asset name, so a release-asset.json naming v0.4.0 could fetch from another release, host or repository; only the pinned SHA-256 stood in the way. It now requires a version tag in "release", the asset name derived from it, and the exact download URL package() writes - both built by one function, so they cannot drift apart. The results inventory counted hidden files, so a .DS_Store left by Finder failed `make evidence-check` on macOS with "measurement data belongs in the release asset". Hidden files are ignored by Git and are not data; the inventory and package() now skip them. Verified: the pointer and dotfile tests from cdad4d1 pass; 23 tests OK. Re-running the review's surviving mutations one at a time - dropping the duplicate check, dropping isreg(), dropping the pointer check, letting package() skip validation, changing combine() to indent=1, counting dotfiles - each now fails its test. `make evidence-check` against the real v0.4.0 asset, downloaded, passes, and passes again with a .DS_Store present.
B1 (blocking in review): docs/benchmarking.md still told readers to run `python3 scripts/record_evidence.py --render bench/results/current` and said "the manifest records the new report-generator commit". That directory no longer has a manifest, so the command fails with FileNotFoundError. The paragraph now names `make evidence-render`, says the asset and pointer stay unchanged and CI checks the committed README against the committed generator, and keeps the --render instruction only for a full recorded dataset directory before packaging. N1: "a report-generator change needs only a regenerated README" holds for render_evidence.py, not for the pooling code. Published assets carry the hashes of the pooled files that verification rebuilds, so their byte output is frozen while any attached asset must stay verifiable; the docs now say so and name the pinning test. N5: deterministic packaging is stated for a given zlib build, and that verification never repacks. N6: `--package` rewrites release-asset.json and README.md relative to the working directory unless --results is given, and both must be committed; `make evidence-check` can only download a published release's asset, so a draft's archive is checked with AS_CACHE_EVIDENCE_ARCHIVE. docs/releasing.md words the ordering caveat precisely: a draft can hold the asset, but CI cannot download draft assets.
The second review round approved the branch with three non-blocking notes. R1: test_packaging_refuses_an_invalid_dataset deleted bytes.log, so it failed because archive_bytes could not read the file, not because package() validates the dataset; with validation removed, a complete dataset whose manifest records a failed command was packed without complaint. The test now uses a complete dataset with a failed recorded command and requires "measurement command failed" and no archive written. With validation removed, it fails on that assertion. R2: the comment claimed hidden files are ignored by Git; only .DS_Store is. It now says what is actually true: no dataset artifact is dot-named, so skipping hidden files cannot hide measurement data. R3: docs/releasing.md states that only plain vX.Y.Z releases can carry an evidence pointer, not pre-release tags. Verified: 48 script tests OK; `make evidence-check` against the downloaded v0.4.0 asset passes; ruff clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The measurement dataset behind the published tables moves from
bench/results/current/(1.3 MB in 17 files) to a release asset. The repository keeps the generatedREADME.md- the page the documents link to - andrelease-asset.json, which pins the asset's URL, SHA-256 and measured commit.Why. Every change to the measurement code made the committed dataset stale, and a re-measurement rewrote some 20,000 lines in a pull request. After v0.4.0 the owner decided the dataset ships with the release instead, keeping it verifiable without traces or a rerun.
What changed
scripts/evidence_archive.py:--packagevalidates a recorded dataset and writes a deterministic tar.gz of the raw batches, evidence logs, reference and byte results and the measurement manifest, plus the pointer and the README.--verifydownloads the asset (or readsAS_CACHE_EVIDENCE_ARCHIVE), requires the pinned SHA-256, extracts only the expected files, rebuilds the pooled results from the raw batches, validates every hash, command and observation count, renders the README with the current generator and requires it to equal the committed copy. It refuses any other file inbench/results/current/.--renderregenerates the README from the asset.make evidence-checkuses it;make evidence-renderis new.1cb65fe). It isas-cache-evidence-v0.4.0.tar.gz, 65,793 bytes, SHA-256d003b2886e635c3cb0df1b54d9b47bb1d292f6d69ee1ba8e81d15b476ec92aca, already attached to the v0.4.0 release.docs/benchmarking.mddescribes where the data lives and what the check verifies.docs/releasing.mdadds the evidence-asset step and the proxy lesson from publishing v0.4.0.Verified
make evidence-checkdownloads the attached asset and verifies it.make all: exit 0 in 98 s. golangci-lint reports 0 issues in 12 modules; 43 script tests and 12 release tests pass; 8 candidate consumer builds pass.Not solved here. The release ordering for the next release that changes measurements is undecided. The tagged commit should carry the new pointer, but the asset can only be attached once that release exists, while CI downloads the asset.
docs/releasing.mdstates this and the interim rule: a release that changes no measurement keeps the previous pointer.