[anth-volk]

Fixes #682

Base: main. main was merged into this branch twice, with merge commits (no rebase): 36c3d8d3 (with #615's final code) and ae059ebd (84 commits of main, on 2026-11-04).

Why

Microcosm is moving each dataset repository to a layout where:

• each build is published once in its own folder, releases/<release>/<build>/, and never changed;
• a root releases.json (microcosm-emitter#24, microcosm_emitter_interface.releases) records every published build of each release, which one is current, and which are withdrawn;
• no new Hugging Face tags are made.

Until a repository switches, it keeps publishing tagged builds with latest*.json pointer files. policyengine.py has to certify both, and never mix them in one repository.

What changes

One layout per repository (certification)

• The read: certification reads the repository's releases.json at main (_releases_record_at_main). Only Hugging Face's 404 for the file means the repository has none, unless its error code (X-Error-Code) is RepoNotFound or RevisionNotFound. Any other answer refuses certification, so a failed read is never taken for a tagged repository.
• A repository with releases.json: only builds in their own folders are certified. A tagged manifest, passed as --manifest-uri or --regional-manifest-uri, is refused (require_tagged_layout), and the error says to certify releases/<release>/<build>/release_manifest.json instead.
• A repository without it: tagged builds are certified exactly as before, and a build-folder manifest is refused because no record lists it.
• Where the rule applies: in certify_data_release. The input-surface measuring tool (regional_input_surface), which only reads a manifest, is unchanged.

Reading a build-folder manifest (provenance/manifest.py)

• Identity: DataReleaseManifest.release is new. A manifest with string release and build is read with build as DataBuildInfo(build_id=<build>). Both names must match the publisher's rule, ^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$. A string build without release, or release with an object build, is refused.
• Build details: the build's details are at the manifest's top level, where microcosm#1505's release writer puts them (decided 2026-10-29): built_at, built_with_model_package (version, git SHA, data build fingerprint) and built_with_core_package. They are read into build, so the built_with_model_package basis, the data build fingerprint and the TRO's start time apply as for tagged builds. A dump writes build back as the string and the details back at the top level, so merges round-trip it.
• release_build_id is unchanged. Such a build has no cut_tag, so it returns the build name, and two builds of one release never match.
• build_folder returns releases/<release>/<build>, or None for older manifests.
• DataReleaseArtifact.revision may be absent only in a build-folder manifest.

Certifying a build in its own folder (provenance/certification.py, through the shared read_build_folder)

• Where it is read from: the manifest URI must be the build's own folder, releases/<release>/<build>/release_manifest.json.
• The record at main: releases.json at main must list the build and must not mark it withdrawn, and the whole record must be one the release writer's parser accepts. policyengine.py repeats ReleasesRecord.parse and check_record without depending on the interface package: exact keys, schema_version 1, safe names and roles, published_at and withdrawn types, default and current, and the 4 MiB limit. The same applies to a local-area release passed with --regional-manifest-uri.
• The build's own files (its artifacts in the manifest's repository) are pinned at releases/<release>/<build>/<file> and at the URI's revision. Each path must be a bare file name, alone or after the folder; a nested path, . or .., or a backslash is refused. Artifacts from other repositories keep their own paths and pins.
• Names: the build's name becomes data_releases.uk.version, the certificate's data_build_id and the changelog fragment name.
• Withdrawn builds: unchanged by this round. Certification refuses a build the record marks withdrawn. Whether a loader or an installed bundle should refuse such a build, or load it with a warning, waits on a separate decision.

At import

• get_data_release_manifest checks a build-folder manifest against the bundle's pinned release_manifest_path. A manifest naming another release or build raises ReleaseBuildFolderError, a ValueError that is deliberately not a DataReleaseManifestUnavailableError, so the offline fallback to the bundled certificate never covers it.
• It then pins the build's files the same way certification does.
• Since the merge of main, this covers both the fetched manifest and main's bundle-pinned local manifest file (POLICYENGINE_<COUNTRY>_DATA_MANIFEST_FILE). Both go through _bind_to_bundle and then main's _validate_bundle_artifacts.
• Import doesn't read releases.json.

Docs and changelog: docs/engineering/skills/data-certification.md, section "Builds in their own release folders"; changelog.d/682.added.md.

The merge of main (ae059ebd)

There was one conflict, in provenance/manifest.py. main had moved the remote fetch into _fetch_data_release_manifest, added the local manifest file, and checked every loaded manifest against the bundle (_apply_release_revision, then _validate_bundle_artifacts). The fetched manifest now goes through _bind_to_bundle: read_build_folder for a build folder, _apply_release_revision otherwise, then the bundle check. The local manifest file got the same treatment in a red-green pair (ba7bd592 / 87bfd56f).

Environment variables

No variable is added, and no workflow or deployment file changes. The new releases.json read for tagged manifests uses the same token as the manifest read: certify_data_release's token argument, which scripts/bundle.py certify-data leaves unset, as before. CI's HUGGING_FACE_TOKEN secret for bundle verification is unchanged. The scheduled UK certification workflow reads policyengine/policyengine-uk-data-private through uk_qualification and doesn't call certify_data_release, so it is unaffected.

Tests

Only the tests this PR adds or changes were run locally, each by pytest node ID: an explicit list, and the runner refuses an empty one. That covers the existing certification tests whose shared HTTP fakes this round changes. CI runs the rest. All fixtures are synthetic; nothing was fetched.

Commit	Red (tests first)	Green
9a5286c2, 158b5aee, 0320e5cc	read, certify, and refuse at import a manifest naming another build (27 tests; see their commits)	27 passed
edc5356b / ea01e115	certify only while releases.json lists the build and hasn't withdrawn it: 5 of 6 failed	33 passed
ef297cb6 / 62419805	bare file names directly inside the folder: 7 of 7 failed (DID NOT RAISE)	40 passed
58d43c6f / 90301909	only records the writer accepts: 22 of 28 failed	68 passed
89c2cff3	the rules most likely to drift from the writer's	73 passed
ae059ebd merge of main	—	73 passed
ba7bd592 / 87bfd56f	TestImportCheckBuildFolder::test__given_local_manifest_of_its_folder__then_artifacts_resolve_in_it (pytest.fail: the bundle check refused the local copy) and …::test__given_local_manifest_naming_another_build__then_refuses (AssertionError: a ValueError, not ReleaseBuildFolderError) failed	both passed; 75 in the file
edbb041c / 79670733	7 new tests, 6 failed (details below)	80 in tests/test_release_build_folders.py passed; 49 existing node IDs in tests/test_certify_data_release.py and tests/test_regional_input_evidence.py (70 tests with their parameters) passed

The 6 failures at edbb041c:

• TestBuildFolderManifest::test__given_folder_layout_manifest__then_reads_top_level_build_details (assert None == '2026-10-19T08:33:23+00:00');
• …::test__given_folder_layout_manifest__then_dump_keeps_build_details_on_top (the same, for the dump);
• TestCertifyBuildFolder::test__given_build_built_with_the_model__then_certified_on_its_build (pytest.fail: "matches neither the build-time model (unrecorded)…");
• TestCertifyOnlyListedBuilds::test__given_tagged_release_without_releases_record__then_certifies (releases.json was never requested);
• …::test__given_tagged_release_where_releases_record_exists__then_refuses (DID NOT RAISE CertificationError);
• …::test__given_tagged_release_and_unreadable_releases_record__then_refuses (DID NOT RAISE CertificationError).

…::test__given_folder_layout_manifest_without_build_details__then_has_none passed: it guards the old behaviour.

One test change after red: the first build-details test compared the model package's model_dump(), which includes its unset sha256 and wheel_url. The green commit compares model_dump(exclude_none=True). The red failure was the missing built_at, before that comparison.

The existing tests' HTTP fakes: the tagged-layout certification tests in test_certify_data_release.py (and test_regional_input_evidence.py, which uses its _certify_uk) now answer the repository's releases.json with Hugging Face's "entry not found" 404 (_without_releases_record). Those repositories publish tags. They passed at red (70) and at green (70).

ruff format --check and ruff check pass on the changed files. mypy on the two changed modules reports only no-any-return at release_manifest_dir, which predates this PR.

Review rounds

• 2026-10-22: compared with microcosm-emitter#24 at 7a47f9a. No bugs; 89c2cff3 added the drift tests, and 72347a46 corrected a docstring.
• 2026-11-04: merged main and fixed the local manifest file. Added the top-level build details (H2c's decision) and the one-layout rule (decided 2026-11-04).

🤖 Generated with Claude Code