[anth-volk]

Fixes #1022

Same-Repository Draft PR Check

• This PR is a draft.
• This PR is from a branch in PolicyEngine/policyengine-sim-api, not a personal fork.

Changelog

• This PR adds a changelog fragment, changelog.d/<name>.<type>.md, whose type is breaking, added, changed, fixed or removed. It adds two: retry-dataset-downloads.fixed.md and move-latest-after-checks.changed.md.

Summary

Retry run-time dataset downloads.

• dataset_download_retry (main path) and its own Stage 12 copy, stage12_runtime.dataset_download_retry, wrap the run-time dataset load. Both main-path loads in simulation_runtime and the Stage 12 load in calculate_simulation_frames are wrapped.
• The load is tried at most three times, with a jittered wait that doubles (1–2 s, then 2–4 s).
• Retried: connection errors (including a read timeout while the body streams, as in release run 42040665530), timeouts, truncated response bodies and 5xx responses.
• Never retried: policyengine.py raises DatasetMaterializationError for 401, 403, 404 and a SHA-256 mismatch. Other 4xx responses (including 429) are not retried either.
• policyengine.py writes each download to a temporary file and moves it into place only after its SHA-256 matches, so a retry never reuses a partial file. No policyengine.py change is needed.
• requests is now a direct dependency, including in modal-simulation-image, the only group the Modal images install.

Publish a release's routes before its checks, and move latest after them.

• "Publish routing after all services pass" (update_routing) now adds only the release's routes and bundle manifest (--plan-out plans routes only), then completes its own journal entry. The routes are its policyengine.py version and its US and UK model versions, mapped to the deployed app. latest is left alone, except on the first publication into a routing state with no latest version.

• A new job, "Move latest to the release after every check" (publish_latest), runs after the stack verification, the authenticated tests and the Cloud Run promotion. It:

  • plans with --plan-latest-out, so latest moves only to a newer release, or with force_latest;
  • records and keeps its plan;
  • applies it;
  • checks /versions with the entrypoint smoke at the URL callers use, and at the custom hostname when one is configured. This is the EXPECTED_LATEST check that previously ran in the stack verification and the promotion;
  • completes its journal entry.

  If a step after recording fails, it restores the state its plan replaced. It journals a new state even when latest stays, so a rerun's attempt replaces an earlier attempt's recorded outcome.

• No job that runs after the routes job is upstream of it, so "Re-run failed jobs" after a failed check reruns the checks, the latest job and the finalizer, and does not republish the routes. A failed check leaves the routes in place.

• finalize_routing and simulation-deployment-ready.sh now also wait for, and require, the latest job.

• A Stage 12-only deploy (stage12-deploy.yml, deploy_existing_stack: false) skips both routing jobs, as before.

• Rollback with deploy_run_id (rollback-restore-routing.sh):

  • It downloads each attempt's routes plan and latest plan, and restores them oldest first.
  • An attempt's plan is required only from a job that started in that attempt. A rerun's job list repeats carried-over jobs with their original start time.
  • GitHub stops listing an attempt's artifact once its job reruns. A missing plan is therefore accepted only when restoring without it is still exact: either no downloaded plan came after it, or the downloaded plans either side of it show its change undone. Otherwise the rollback refuses.

• New rule sim.deploy.latest-moves-after-checks, enforced by test_deployment_assets_latest.py. Rule sim.deploy.routing-publication-journaled's statement is updated, and one node of sim.deploy.release-identity-smoke points at the moved test. These touch the rule files, which need approval from a core developer other than the author (sim.process.rule-machinery-code-owned).

Decided: removing a failed release's routes. Nothing removes them automatically, because a rerun of the failed jobs needs them. When a failed release will not be rerun, an operator runs simulation-rollback.yml with deploy_run_id, which restores the run's plans only while the routing state is still what that run published.

Effects to be aware of

• Requests that pin one of the release's US or UK model versions without a policyengine.py version reach the release from the routes step onward. This includes a model version it shares with the previous release, which the integration tests rely on, since they pin model versions. After a failed check they keep reaching it until a rerun succeeds or a deploy_run_id rollback removes its routes.
• The Cloud Run promotion now runs before latest moves. If the latest job fails, latest is put back but stable traffic stays on the new entrypoint revision.

Not changed here

• The release smoke apps' own download of the UK local-area dataset (uk_area_hdf_dataset_path) still has no retry.
• A deploy_run_id rollback of a run whose routes job reran after uploading its plan (for example after "Re-run all jobs") still refuses, because GitHub no longer lists the earlier routes plan. Roll back with policyengine_version instead.
• If both the step that puts latest back and the finalizer fail in one attempt, the first rerun's record step refuses and a second rerun completes the deploy. main behaves the same way for the routes job.
• The combined publication (routes and latest in one write: plan_routing_publication, build_next_routing_state and the direct publish mode of the CLI) remains available; no workflow uses it.

Testing

Written test-first. Each new or changed test was run on its own by pytest node ID; full suites were not run locally.

• projects/policyengine-simulation-executor: 85 test cases. These are the new test_dataset_download_retry.py, test_stage12_dataset_download_retry.py and version_registry/test_routing_split_publication.py, plus the updated CLI, writer and rollback tests in version_registry/.
• projects/policyengine-simulation-entry: 152 test cases. These are the new test_deployment_assets_latest.py and the updated routing, readiness, release-check, identity-smoke, promotion, script, invariant and rollback tests.
• CI on the first push: one failure, test_deployment_uses_gcloud_workflow_without_terraform, which counts the jobs that declare the release environment (16 with the new job, not 15). Fixed and run by node ID.
• Ruff format and check on every changed Python file, and Pyright for the executor and entry packages.
• shellcheck on the changed scripts, python3 scripts/run_quality_guards.py, .github/scripts/check-uv-locks.sh and .github/scripts/check-changelog-fragment.sh.
• The rollback script's job query was run against run 42040665530, and the artifact listings of runs 41703573393 and 42081132198 were read. All were read-only API calls.
• Three rounds of adversarial review; each substantiated finding was fixed test-first.
• Not run: any deploy, workflow dispatch or rerun, or change in Modal, Google Cloud or Hugging Face.

🤖 Generated with Claude Code