[anth-volk]

Fixes #1498

What this changes

No UK program is registered with the Modal runner yet. A UK build run there would record its bundle as local_only: the runner sets HF_HUB_OFFLINE=1 for every tool, so the build would hand its emitter nothing (#1460). This adds the workstation side for that case. After modal volume get has copied such a run back:

uv run python tools/stage_uk_rowwise_candidate.py \
  --run-dir ./modal-runs/<run_id>/state/<output directory> \
  --modal-receipt ./modal-runs/<run_id>/receipts/<stage>-<utc>.json

• Checks before any output is read. The command refuses a file that is not a microcosm-modal-us-stage-receipt/1 receipt, a receipt of a stage that did not finish (receipt_status_problems: not COMPLETED, or stopped_at_budget), and a run directory whose real location (after links and ..) is outside the copied-back state directory. That directory is state/ beside receipts/, or --modal-state-root, and both are resolved the same way. The receipt must also list the build manifest and every output it registers. An output in the receipt's outputs_not_mirrored that the copy-back lacks is named as such.
• The build manifest is compared unless its record is local_and_remote. Unless the build manifest's staged_dataset record has the mode local_and_remote, the manifest must have the receipt's SHA-256 and size. So a manifest edited to register another file the stage wrote, rename a role or change any other field is refused. An attempt to hand the directory over from the workstation that records its outcome in the manifest writes a local_and_remote record, whether the bundle landed, failed or was refused, so after such an attempt the manifest is not compared, and a second run on a landed bundle still works. A bundle the emitter queued without copying the manifest leaves the manifest as handed over, with the receipt's bytes and the build's record, and it is then still compared.
• What that check does not catch. The command reads the mode from the manifest itself, so it cannot tell a record a hand-over wrote from one edited to say local_and_remote. Such an edit also turns the manifest's comparison off, and the edits above are then not caught. No condition read from the manifest can close this; see "Review round 2" below.
• Then the outputs, whatever the record says. Every output must have the receipt's SHA-256 and size, so an output whose bytes differ from the receipt's is always refused. It lists every problem it finds. staged_manifest.json and sha256sums.txt are not compared, because staging writes them again.
• The same hand-over as a local directory. A directory that passes goes through the re-stage command's existing code unchanged: the write-access check, the resume of a recorded hand-over, a new hand-over through stage_bundle, the staged_dataset, staged_dataset_handover and staged_dataset_at_build records, the rule that a landed record is never replaced, and the note about the build's other records. The staged run id is the build's own, from its local_only record.
• HF_HUB_OFFLINE on the workstation is refused as for any re-stage, before an emitter starts.
• --local-only with --modal-receipt checks the directory against the receipt and writes staged_manifest.json and sha256sums.txt, but leaves the build manifest as it is, so a manifest that still holds the build's record is still compared by later runs with the receipt. It prints the local_only record the manifest does not get. A directory whose manifest records a landed bundle is left unchanged.
• --local-only without --modal-receipt still records the bundle as local_only in the build manifest, as before. On a copied-back directory that record can make every later run with the receipt refuse the directory, so the guide and the --local-only help warn against it and say to pass the receipt; if it has happened, fetch the run again with modal volume get.
• Error messages instead of tracebacks. A build manifest that is not valid JSON or not a JSON object, and an output that does not match its manifest digest, are reported as error: ..., with or without --modal-receipt.
• The receipt format and status check come from tools/modal_us_stage_plan.py (standard library only), imported the way tools/modal_us_stage.py imports it.
• Docs: the UK staging guide's "Staged datasets" section describes the command and which edits its checks catch; the Modal runbook points to it after step 5 and lists US dataset staging under "What this does not cover yet".

Review round 2

• --local-only could leave a directory that every later run with the receipt refused. It wrote a new local_only record into the build manifest. That record differs from the build's own when the build disabled its staged dataset (--no-staged-dataset or --no-staging), when --run-id names another run id, when a national build changed its outputs after recording its staged dataset (it rewrites its build record, and registers its score receipt when it has one), and after a hand-over. The manifest then still counted as the build's but no longer had the receipt's bytes. With --modal-receipt, --local-only now leaves the build manifest as it is (_leave_the_copied_back_manifest).
• An edited staged_dataset.mode turns the manifest's comparison off. The claims are narrowed: the description, the guide, the command's docstring and help text, the runbook's pointer and the changelog fragment now say the comparison applies unless the record is local_and_remote, and that an edit setting that mode is not caught, while the outputs still are.
• No stricter check is added. Any condition read from the manifest, such as requiring a record that validate_staged_dataset_delivery accepts as remote or a staged_dataset_handover block, can be written by the same edit, so it would let the manifest vouch for itself. A sound check would rebuild the manifest the stage wrote and compare its hash with the receipt's. That works only with the build's original staged_dataset record, and after a hand-over the manifest does not keep it: staged_dataset_at_build holds only the mode, status and error code. The record cannot be recomputed from the outputs either: a disabled record's opt-out reason is gone, another --run-id changes it, and a national build's record predates its rewritten build record. A rebuild from what is left would refuse legitimate second runs in those cases, or would have to skip the check, which an edit could then choose. Storing the build's record in the manifest at the first hand-over would make the check sound, because the rebuilt bytes must hash to the receipt's digest, so only the records the command writes could differ. It adds a key to the build manifest that is uploaded with the bundle, which is a decision for later, once a UK program is registered with the Modal runner.
• Tests. The refusal case that edits an output now does so under a record edited to say it was handed over, so it reaches the comparison with the output's receipt entry instead of the manifest's. A new test edits an output after a real failed hand-over.

Review round 3

No bugs. Documentation edits for its code-quality points 1 and 2, and a test for point 4:

• The --modal-receipt help and the Modal runbook's pointer now also say that the record is read from the manifest itself, so an edit that says a hand-over happened is not caught. Round 2's statement above about those two texts is now true.
• "Later runs with the receipt still compare it" now says it holds for a manifest that still has the build's record.
• Any attempt that records its outcome in the manifest writes a local_and_remote record, refused attempts included. Before, the texts said only "when the emitter copied the manifest".
• The command's module docstring is wrapped again, and its refusal sentence ("Every refusal comes before the command starts an emitter") no longer reads as the reverse of the one before it.
• The changelog fragment describes the hand-over before --local-only, which hands nothing over.
• The warning about --local-only without the receipt on a copied-back directory, above.
• test_a_build_manifest_the_emitter_did_not_copy_is_still_compared: a hand-over whose emitter queues the bundle without copying the build manifest (never_copy) leaves the manifest as handed over, and the next run with the receipt refuses an edit made after it. It guards existing behaviour.

Round 3's other points wait until a UK program is registered with the Modal runner: the stricter check it describes (comparing the build manifest with the receipt just before a new hand-over, whatever its record says), and the HF_HUB_OFFLINE refusal that still comes after every output is hashed.

What this does not do

• No submission to the collector. The command's emitter starts its Hugging Face module only (start_bundle_emitter()), and no build submits artifacts to the collector's artifact intake yet (Accept artifact submissions into private storage and run worker jobs microcosm-collector#12). That part waits on turning the intake on and then on builds submitting to it. Telemetry goes to the collector from wherever the build runs; for a Modal build, the container sends it.
• No US bundle. All of the runner's registered programs are US builds, and none writes a staged-dataset bundle. The package stage's release directory has its own release_manifest.json (an artifacts mapping, not outputs) and its own sha256sums.txt, the file name the bundle's checksum file uses. Staging a US build's datasets needs a private repository, a manifest that registers the outputs, and a command that reads it.

Tests

• packages/microcosm-build/tests/engine_free/uk/test_uk_modal_copy_back_staging.py (32 cases): a synthetic dense build run with HF_HUB_OFFLINE=1, moved into a copied-back Modal layout with a receipt (test_support/microcosm_build/modal_copy_back.py). It covers:
  • the hand-over and its records through the command's own stage_bundle;
  • refusals: an output edited under a record edited to say it was handed over, an output edited after a failed hand-over, a build manifest edited after a hand-over whose emitter did not copy it, an output whose receipt entry has other bytes, an output never copied to the runs volume, an output that no longer matches the manifest, an output missing from the receipt, a stage that did not finish, a stage stopped at its budget, a file that is not a receipt;
  • an edited build manifest: a field changed, another file the stage wrote registered as an output, a role renamed;
  • the receipt's status and the directory checked before any output is hashed;
  • a directory outside the state directory, --modal-state-root, a state directory that is a symbolic link, a run directory named through a link or through .., and a link in the state directory that leads outside it;
  • a second run on a landed bundle, with and without the receipt, and with --local-only; a retry after a first attempt without a credential, which records the queued hand-over as already_staged;
  • --local-only with the receipt leaving the build manifest for a later staging run: with the build's own local_only record, a build that disabled its staged dataset, another --run-id, and after a failed hand-over; --local-only without the receipt still recording the bundle in the build manifest;
  • HF_HUB_OFFLINE left set on the workstation; a malformed build manifest.
• test_a_local_only_check_of_a_copied_back_national_build_leaves_its_manifest in test_uk_rowwise_national_role.py: the same --local-only check on a national build run with HF_HUB_OFFLINE=1.
• test_a_modal_runs_copied_back_outputs_are_staged_by_the_re_stage_command in the installed-emitter tests runs the command against the installed emitter and the stand-in Hub, taking the run id from the build's local_only record.
• _remote_hub, _bundle_emitter_for and _edit_an_output move from test_uk_full_build_cli.py to test_support/microcosm_build/uk_full_build_cli.py, which both UK modules import. They serve 11 test cases there.

Run locally

Each by node ID from an explicit list that refuses to run when empty, with HF_HUB_OFFLINE=1, Python 3.13 and uv sync --all-packages --locked. CI runs everything else.

• Round 2's tests at a5ea5db76 (failing tests, before the fix): 4 failed and 5 passed. Each failure is an AssertionError on the assertion that the build manifest's bytes are unchanged after --local-only with the receipt:

  • test_a_local_only_check_with_the_receipt_leaves_the_build_manifest[a build that disabled its staged dataset], [another --run-id] and [after a failed hand-over];
  • test_a_local_only_check_of_a_copied_back_national_build_leaves_its_manifest.

  The 5 that passed guard behaviour the fix keeps: [the build's own local_only record], test_a_local_only_run_without_the_receipt_records_the_bundle_as_before, test_a_copied_back_bundle_that_landed_is_not_handed_over_again (now also with --local-only), test_an_output_edited_after_a_failed_hand_over_is_refused_by_its_receipt_entry, and test_a_copied_back_directory_that_does_not_verify_is_refused[an output edited under a record edited to say it was handed over]. The last two pin the narrowed claim, which needed no code change.

• The same 9 at 9e508a4fc (the fix), at 2af2d1c0f (after merging main, which brought Stop a build that must send telemetry when its emitter cannot start #1465) and at 6adad928f (wording only): all passed.

• Round 3, at dce6e1e5d (after merging main again at 0f0e777b2, which brought Calibrate the QRF sign gates out of fold, for every gated UK imputation #1391 and Give a stage launched on Modal a telemetry-only collector token #1473): the 9 and test_a_build_manifest_the_emitter_did_not_copy_is_still_compared, 10 passed.

• test_a_modal_runs_copied_back_outputs_are_staged_by_the_re_stage_command, with --run-integration, with microcosm-emitter 0.11.9 and 0.12.0 each installed into its own folder with the runner script's uv tool install command and constraints file: passed with both.

• In review round 1, 11 of the first 26 cases failed on an assertion before the change.

• ruff check and ruff format --check on the changed Python files, and tools/ci_test_plan.py verify.

Environment variables

No variable is added, and no workflow, Docker, Terraform or Modal configuration changes.

Variable	Read by	Provider	Wiring
HF_TOKEN, then HUGGINGFACE_TOKEN, then the token hf auth login saved	The emitter the command starts	The operator's own Hugging Face credential on the workstation	The emitter is a child process started without its own environment, so it inherits the command's. In the installed-emitter test, a placeholder HF_TOKEN the installed_emitter fixture sets, which also removes HUGGINGFACE_TOKEN and HUGGING_FACE_HUB_TOKEN; never a credential.
HF_HUB_OFFLINE, TRANSFORMERS_OFFLINE	hub_offline() in the command	The workstation shell; inside Modal, the runner's tool_environment sets HF_HUB_OFFLINE=1	Read from the command's own environment; when set, the command refuses to stage.
POPULACE_UK_STAGED_DATASET_REPO_ID	The --repo-id default	The workstation shell, optional (the default repository is in uk_runtime/staging.py)	Read when the arguments are parsed.
MICROCOSM_EMITTER_COMMAND	The emitter interface's launcher	The workstation shell, optional (otherwise PATH); in CI, tools/run_installed_emitter_tests.sh	The runner script sets it for the pytest step of each installed release (#1467, unchanged).
INSTALLED_EMITTER_VERSION	The installed_emitter fixture	tools/run_installed_emitter_tests.sh	As above.
XDG_CACHE_HOME	The emitter (its hand-over database)	The workstation shell, optional (default ~/.cache); a temporary directory in the tests	Inherited by the emitter process.
HF_ENDPOINT, HF_HUB_DISABLE_XET, HF_HUB_DISABLE_TELEMETRY, HF_HOME, proxy variables	The emitter's huggingface_hub, in the installed-emitter test only	The installed_emitter fixture	Set with monkeypatch and inherited by the emitter process.

MICROCOSM_TELEMETRY_REQUIRED, which #1465 added, is read only when a build starts its telemetry emitter; start_bundle_emitter(), which this command uses, does not read it. The MICROCOSM_MODAL_TELEMETRY_* variables #1473 added are read inside the Modal runner's own functions in tools/modal_us_stage_plan.py; the command imports that module only for RECEIPT_SCHEMA and receipt_status_problems, so it does not read them.

🤖 Generated with Claude Code