[anth-volk]

Fixes #993

Same-Repository Draft PR Check

• This PR is a draft.
• This PR is from a branch in PolicyEngine/policyengine-sim-api, not a personal fork.

Summary

A scheduled Modal function now marks the Public API's report runs and simulation runs failed when nothing will finish them: the run is pending or running but the Modal call recorded for it has ended, for example because its container ran out of memory before it could record the outcome, or the run is still pending with no Modal call recorded ten minutes after it was created, because the simulation service stopped before starting its call. It runs every ten minutes, never fails a run whose call is still running or queued, or whose state Modal cannot report, and two sweeps at once fail each run once. The deployment workflow deploys it with the executor.

This builds on #992, which records each run's Modal call ID when the coordinator or a worker claims the run; review only the commits after it. The simulation service also records, on a pending report run, the Modal call it starts.

How it tells that a call has ended

Every ten minutes the sweep lists the report runs, and then the simulation runs, that have been pending or running for at least ten minutes, counted from their first claim (started_at) or, while pending, from their creation (created_at). A pending run with no call recorded is failed without asking Modal: run_not_started: The run stayed pending with no Modal call recorded for it. Usually the simulation service stopped before starting its call; if it started one but failed to record it, that call's claim is refused once the run has failed. A running run with no call recorded is left to the 24-hour limit. For each recorded call, on a pending or a running run, it asks Modal once for the call graph the call is part of (modal.FunctionCall.from_id(call_id).get_call_graph()) and reads the status of the inputs whose call ID is the recorded one, wherever they are in the graph: a worker spawned inside a coordinator's input is nested under it.

• any PENDING input: the call is still running, queued, or waiting to run again; Modal restarts a pre-empted call under the same call ID and reports it pending meanwhile;
• every input SUCCESS, FAILURE, TERMINATED, TIMEOUT or INIT_FAILURE: the call has ended, and the run is marked failed with modal_call_ended: The Modal call recorded for this run ended without recording the run's outcome.;
• no input with that call ID, or any error asking (an unknown call, Modal unreachable): unknown, and the run is left as it is.

A simulation run is listed only while no pending or running report run names it as its baseline or reform (report_runs.baseline_simulation_run_id or reform_simulation_run_id). That report run's coordinator starts its children's calls once Modal runs it, and records their outcomes itself; after a pre-emption it cancels them and starts them again under new calls, and a sweep while it is queued or in that gap would otherwise fail a run that is about to start. Report runs are swept first, so a dead coordinator's simulation runs are reached in the same sweep.

A run still unfinished 24 hours after its first claim, or after its creation if it was never claimed, is failed whatever Modal reports, with or without a recorded call (run_exceeded_maximum_age). No attempt of these functions can run nearly that long (the coordinator's own time limit is 90 minutes), however often Modal pre-empts and restarts it; the limit stops a run staying running for good when Modal keeps reporting a call it won't retry as pending, no longer describes a call, or no call was recorded.

The failure is one conditional UPDATE … WHERE id = … AND status = <the listed status> AND modal_call_id = <the listed call> (or IS NULL), setting only the status, error, completion time and updated_at, so a run that was claimed, finished, moved to a new call, or had its call recorded by the simulation service since it was listed is left alone, and a second sweep running at the same time changes nothing the first changed.

Where it runs

A new Modal application, policyengine-simulation-run-sweep (src/modal/run_sweep_app.py), with one function, sweep_stuck_runs, on the cron schedule */10 * * * *, at most one container at a time, with the Modal worker role's database secret (policyengine-modal-worker-database) and the locked package group the v2 images install, without the bundle's data. It is not versioned with the .py bundle, so one copy runs per Modal environment rather than one per deployed v2 application. The deploy_executor job of simulation-deploy.reusable.yml deploys it after the versioned executor, with uv run --frozen modal deploy --env="${{ inputs.modal_environment }}" src/modal/run_sweep_app.py. Modal refuses that deployment until the Modal secret exists in the environment, so the job fails until then.

Not covered: household jobs, until the simulation service records their Modal calls. A run marked running with no recorded call is failed only by the 24-hour limit.

Tests

The store's listing and conditional failure run against a disposable Postgres migrated with the Public API's v2 chain, as the Modal worker role, with the worker's UPDATE limited to its columns, including two sweeps started together in threads, pending runs listed from their creation, a pending run whose call was recorded or claimed after the listing, a run with no recorded call, and simulation runs named, as a baseline or a reform, by pending, running, finished or several report runs, by a report run with no reform run, or by none; they skip when REPORT_RUN_TEST_DATABASE_URL and REPORT_RUN_TEST_SEED_DATABASE_URL are unset, as in CI. The sweep's decisions and the call-graph reading use stand-ins for Modal and the store. The Modal application's declaration is checked with the repository's fake Modal module, and the deployment step by reading the workflow.

Environment variables

• MODAL_WORKER_DATABASE_URL: read by sweep_stuck_runs. Provider: the Modal secret policyengine-modal-worker-database, created by an operator in the staging and main Modal environments (the step Record report runs and simulation runs from the v2 Modal coordinator and workers #992 waits for). Wiring: modal.Secret.from_name(..., required_keys=["MODAL_WORKER_DATABASE_URL"]) attached through secrets= to sweep_stuck_runs. Not verified: the secret does not exist yet.
• The Modal client in the container authenticates as the workspace the app is deployed in; no other variable is read.
• MODAL_TOKEN_ID, MODAL_TOKEN_SECRET: read by the new deployment step. Provider: GitHub Actions repository secrets (both exist; the beta and prod environments don't override them). Wiring: the beta and prod jobs of simulation-deploy.yml and of stage12-deploy.yml call the reusable workflow with secrets: inherit, and the step's env: reads secrets.MODAL_TOKEN_ID and secrets.MODAL_TOKEN_SECRET, as the executor's deploy step does.

What this waits for before merging

1. Record report runs and simulation runs from the v2 Modal coordinator and workers #992 merged (this PR is stacked on it), with everything it waits for: Record each run of a v2 simulation and report, and key results by run policyengine-api#4162 deployed, the Modal worker role and its grants, and the Modal secret policyengine-modal-worker-database in the staging and main Modal environments. Without the secret, the deployment's executor job fails.
2. An approving review.

After the first deployment, check that modal app list shows the app in each environment and that its first scheduled run returned a summary (checked, failed, running, unknown, left, failed_for_age, failed_without_call) without errors in modal app logs policyengine-simulation-run-sweep.

🤖 Generated with Claude Code