Record each run of a v2 simulation and report, and key results by run

.github/scripts/test_alembic_v2_lifecycle.sh

@@ -5,4 +5,5 @@ set -euo pipefail
 uv run --frozen --extra dev pytest -q \
   tests/unit/v2/test_alembic_v2.py \
   tests/integration/test_alembic_v2_lifecycle.py \
-  tests/integration/test_alembic_v2_catalog_tables.py
+  tests/integration/test_alembic_v2_catalog_tables.py \
+  tests/integration/test_alembic_v2_simulation_runs.py

.github/workflows/v2-integration-check.yml

@@ -79,8 +79,8 @@ jobs:
         run: uv run --frozen --extra dev coverage run -a --branch -m pytest -q tests/integration/test_v2_policy_persistence.py tests/integration/test_v1_policy_dual_write.py tests/integration/test_mysql_policy_dual_write.py tests/integration/test_v2_user_policy_mirroring.py tests/integration/test_v1_user_policy_dual_write.py
       - name: Test signed-in identity provisioning and user associations
         run: uv run --frozen --extra dev coverage run -a --branch -m pytest -q tests/integration/test_v2_user_identity_persistence.py
-      - name: Test comparison-run persistence and privilege reconciliation as the managed roles
-        run: uv run --frozen --extra dev coverage run -a --branch -m pytest -q tests/integration/test_v2_stage12_comparison_persistence.py tests/integration/test_v2_privilege_reconciliation.py
+      - name: Test comparison-run persistence, the simulation API's roles and privilege reconciliation as the managed roles
+        run: uv run --frozen --extra dev coverage run -a --branch -m pytest -q tests/integration/test_v2_stage12_comparison_persistence.py tests/integration/test_v2_simulation_role_privileges.py tests/integration/test_v2_privilege_reconciliation.py
       - name: Test v2 household persistence and immediate v1 copying
         run: uv run --frozen --extra dev coverage run -a --branch -m pytest -q tests/integration/test_v2_household_persistence.py tests/integration/test_v1_household_dual_write.py tests/integration/test_mysql_household_spm_dual_write.py
       - name: Test the row counts v1 ORM updates report on MySQL

changelog.d/v2-simulation-runs.changed.md

@@ -0,0 +1 @@
+API v2's database records each run of a simulation and of a report with the versions it used, keys every result to the run that produced it, and stores household calculation jobs for one household, one year and one policy. Result tables gain the economy output's fields that had no column.

docs/engineering/skills/alembic-migrations.md

@@ -255,9 +255,15 @@ and a revision runs once per database, so it can't repair drift.
 | `policyengine_v2_runtime` | The Public API on Cloud Run |
 | `policyengine_v2_catalog_writer` | `scripts/publish_v2_metadata_catalog.py` |
 | `policyengine_stage12_<environment>` | The simulation API and its Modal worker, on the temporary comparison-run tables |
+| `policyengine_v2_simulation_service` | The simulation API on Cloud Run: creates simulations, reports, their runs, budget-window runs and household jobs, and reads the policy, dynamic, household and catalog rows a run needs. It updates only the status, Modal call ID, error and completion columns of the runs and household jobs whose Modal calls it starts |
+| `policyengine_v2_modal_worker` | The simulation API's Modal workers: write their runs' and household jobs' status and results, and read the rows a run needs. It updates only a run's status, Modal call ID, error, time and result columns; the workers' code, not the database, keeps a worker to its own run |
 
 Every v2 table has an entry for every role, so "no access" is a decision, not
-an omission. The migration role owns every table and isn't listed. Three rules
+an omission. A privilege is declared on the whole table, or with
+`update_columns(...)` on only the columns it names; PostgreSQL then refuses a
+statement that sets any other column, so neither simulation API role can
+change what a run is. A role never holds both kinds of `UPDATE` on one table:
+the whole-table one would let it set every column. The migration role owns every table and isn't listed. Three rules
 check the declarations:
 
 - `api.v2.runtime-privileges-declared`: a unit test reads each connector
@@ -267,16 +273,29 @@ check the declarations:
   `policyengine_v2_runtime`, catalog publication as
   `policyengine_v2_catalog_writer`, and the comparison-run services as
   `policyengine_stage12_test`, holding only their declared privileges.
+  `tests/integration/test_v2_simulation_role_privileges.py` runs the
+  simulation API's writes as its two roles; their code lives in the simulation
+  API's repository.
 - `api.v2.deployed-privileges-match`: every deploy runs
   `scripts/reconcile_v2_privileges.py apply` right after the schema upgrade,
   then `verify`, before the catalog is seeded.
 
-`apply` grants each declared privilege a role lacks. It never revokes: it
-reports privileges the file doesn't declare, and the projects' default
-privileges, in the job summary. It grants nothing when a declared role or
-table is missing. The roles themselves are provisioned outside the
-repository; `--create-missing-roles` is accepted only for the disposable test
-database.
+`apply` grants each declared privilege a role lacks, as one `GRANT` per role
+and table and one per column privilege. It never revokes: it reports
+privileges the file doesn't declare, on whole tables or on columns, and the
+projects' default privileges, in the job summary. It grants nothing when a
+declared role or table is missing. `verify` checks each column of a column
+privilege with `has_column_privilege`, and fails when a role holds the same
+privilege on the whole table. The roles themselves are provisioned outside
+the repository; `--create-missing-roles` is accepted only for the disposable
+test database.
+
+A pull request may declare a role before an operator creates it by listing it
+in `ROLES_AWAITING_PROVISIONING` in `privileges.py`. While a listed role is
+missing, `apply` and `verify` skip it and name it in their output and the job
+summary instead of failing; once it exists, the next deploy grants and checks
+its privileges. Remove the role from the set once it exists in staging and
+production, so that its absence fails the deploy again.
 
 The declarations include write privileges for code paths production doesn't
 turn on: the dual-write copies into `legacy_policy_mappings` and
@@ -300,6 +319,7 @@ When a pull request adds or changes a v2 table, or a connector's use of one:
      tests/integration/test_v2_stage12_comparison_persistence.py \
      tests/integration/test_v2_metadata_routes.py \
      tests/integration/test_v2_catalog_publication.py \
+     tests/integration/test_v2_simulation_role_privileges.py \
      tests/integration/test_v2_privilege_reconciliation.py
    ```
 
@@ -483,3 +503,98 @@ schema.
 
 The dropped values are not recoverable. Adding a required column fails on a
 table that has rows, so the downgrade works only while both tables are empty.
+
+### API v2 simulation runs
+
+Revision `1356f658347d` (parent `c614b6bcccfc`) was autogenerated after these
+model changes:
+
+- `simulations` keeps only what to calculate. Its execution columns
+  (`status`, `error_message`, `started_at`, `completed_at`,
+  `household_result`) move to the new `simulation_runs`, one row per run with
+  the catalog version, country package version and PolicyEngine.py version it
+  used, and its Modal call ID; `output_dataset_id` is replaced by the run's
+  output file (`output_uri`, `output_sha256`).
+- A report run compares one pair of simulation runs: `report_runs` names its
+  `baseline_simulation_run_id` (required) and `reform_simulation_run_id`
+  (empty for a household report with only a baseline), each a foreign key to
+  `simulation_runs` with `RESTRICT`. The parts of an economy comparison that
+  had no result table are typed: `report_runs` gains the baseline's and
+  reform's cliff gap and share (four nullable numbers, checked to be set
+  together), the SPM receipt (`spm_provenance`, `jsonb`) and the data release
+  (`data_version`). The new `labor_supply_responses` holds one row per report
+  run (the earnings changes from the income and substitution effects and their
+  total, the revenue change, the two relative responses, and baseline, reform
+  and changed weekly hours split into the two effects), and the new
+  `labor_supply_decile_responses` one row per report run and income decile.
+- `poverty` and `inequality` belong to a simulation run (`simulation_run_id`);
+  the other result tables belong to a report run and name the two simulation
+  runs they compare (`baseline_simulation_run_id`, `reform_simulation_run_id`);
+  each of their rows is unique within its report run. `aggregates` and
+  `change_aggregates` are unchanged.
+- The result tables gain the economy output's fields that had no column: the
+  winner, loser and no-change shares of `congressional_district_impacts`, and
+  the UK disclosure control fields (`disclosure_tier`, checked to be
+  `reported` or `approximate`, and the four `_lower`/`_upper` interval bounds)
+  of `constituency_impacts` and `local_authority_impacts`. The detailed budget
+  names neither a program's entity nor whether it is a tax, so
+  `program_statistics.entity` and `is_tax` become nullable.
+  `decile_impacts.decile_variable` names the variable that groups households
+  when it is not the income variable's own deciles (for example
+  `household_wealth_decile`), and `poverty.filter_group` names the group a
+  filter selects (`child` for `age`); both join their table's unique key.
+- `report_runs` gains its Modal call ID.
+- Budget windows: a simulation run calculates one `year`, which joins the
+  index for a simulation's newest successful run, so a budget window compares
+  a report's same two simulations in every year. `reports.budget_window_size`
+  (1 to 75 years, an economy report with a year, ending by 2200) marks a
+  budget-window report. The new `budget_window_runs` holds one row per
+  calculation of one: its report, catalog version, trigger, whether its years
+  calculate their simulations again (`recalculate`), status, error and times.
+  `report_runs.budget_window_run_id` and `year` name the window run and year a
+  report run calculates, at most one report run per year of a window run; a
+  composite foreign key keeps it to a window run of the same report, and
+  deletes it with the window run (`CASCADE`), as deleting a report deletes its
+  runs.
+- `household_jobs` calculates one household (`household_id`) for one `year`
+  under one policy, and records its catalog and package versions and its
+  Modal call ID.
+- The new columns that name the Public API's rows are foreign keys with the
+  delete rules the same columns have on `simulations`:
+  `simulation_runs.tax_benefit_model_version_id` and `dataset_id` (`RESTRICT`)
+  and `region_id` (`SET NULL`), `household_jobs.household_id` and
+  `tax_benefit_model_version_id` (`RESTRICT`), and
+  `budget_window_runs.tax_benefit_model_version_id` (`RESTRICT`), 20 keys
+  between the two services' tables in all. `reports.user_id` gains an
+  index, which account deletion's `SET NULL` searches. Each service writes only
+  its own tables, and the foreign keys between them stay: PostgreSQL checks
+  each named row when a row is written, with the privileges of the named
+  table's owner, so a writing role needs no privilege on the named table.
+
+Missing values in the unique keys of `poverty`, `decile_impacts` and
+`program_statistics` count as equal (`NULLS NOT DISTINCT`, which PostgreSQL
+supports from version 15).
+
+It has three post-generation corrections of one kind, each commented in the
+revision: `simulation_runs.status` reuses the `v2_simulation_status` type the
+baseline created for `simulations.status`, and `budget_window_runs.status` and
+`trigger` reuse the baseline's `v2_report_run_status` and
+`v2_report_run_trigger`. The generated `create_table` would create each type
+again and fail, so each column uses `postgresql.ENUM(..., create_type=False)`.
+
+The upgrade adds required columns without server defaults to `household_jobs`,
+`report_runs` and the result tables, so it needs them empty, as they are while no managed
+role holds a write privilege on them; on a table with rows it fails, and the
+deploy stops before anything changes. The downgrade restores required columns
+of `simulations` and the result tables the same way, so it needs
+`simulations` empty, which leaves no run or result row either; it drops
+`household_jobs`' new columns with their values. The previous Public API revision keeps working at this revision:
+none of its roles can read or write a table the revision changes, except the
+catalog writer, which only counts the rows of `simulations`, `reports` and
+`report_runs`.
+`tests/integration/test_alembic_v2_simulation_runs.py` checks that the
+revision changes no other table, and downgrades and upgrades it again.
+
+The name `simulation_runs` is also a v1 Cloud SQL table. The v2 metadata
+validation (`policyengine_api/data/v2/metadata_validation.py`) now treats it as
+a name the two databases share, as it does `simulations`.

docs/generated/rules.md

@@ -285,9 +285,9 @@ flowchart LR
 | `api.migration.bundle-supported-by-simulation-api` | The installed PolicyEngine bundle is the latest one the simulation API has published, checked on every pull request and before each staging and production deploy. | mechanical | enforced | yes | CI job `ensure-policyengine-bundle-supported-by-simulation-api` | `pr.yml:ensure-policyengine-bundle-supported-by-simulation-api` | — |
 | `api.migration.contracts-consistent` | Migration route groups and workflow contracts are internally consistent, and their generated docs are current. | mechanical | enforced | yes | quality guard `migration-contracts` | `pr.yml:quality-guards` | **unproven** |
 | `api.migration.cross-database-integration` | Cross-database integration tests pass against real MySQL and Postgres. | mechanical | enforced | yes | CI job `v2-integration-check` | `pr.yml:v2-integration-check` | — |
-| `api.v2.deployed-privileges-match` | Every v2 deploy grants the managed roles each privilege policyengine_api/data/v2/privileges.py declares right after the schema upgrade, reports privileges it doesn't declare without revoking them, and stops before the catalog is seeded if a declared privilege is still missing. | mechanical | enforced | yes | CI job `seed`, `catalog-runtime-integration` | `seed-v2-database.yml:seed`, `v2-integration-check.yml:catalog-runtime-integration` | yes |
-| `api.v2.runtime-privileges-declared` | Every v2 table has an entry for each managed database role in policyengine_api/data/v2/privileges.py, and every privilege a v2 connector module needs (from its reads, creates, updates or deletes module name and its row locks) is declared for the role that runs it. | mechanical | enforced | yes | pytest `tests/unit/v2/test_privilege_manifest.py` | `pr.yml:test` | yes |
-| `api.v2.runtime-privileges-tested-as-runtime-role` | The cross-database integration tests run the v2 user services and the parameter tree read as policyengine_v2_runtime, catalog publication as policyengine_v2_catalog_writer, and the comparison-run services as policyengine_stage12_<environment>, holding only their declared privileges, so a statement those privileges don't allow fails CI. | mechanical | enforced | yes | CI job `v2-integration-check`, `catalog-runtime-integration` | `pr.yml:v2-integration-check`, `v2-integration-check.yml:catalog-runtime-integration` | — |
+| `api.v2.deployed-privileges-match` | Every v2 deploy grants the managed roles each privilege policyengine_api/data/v2/privileges.py declares, on the whole table or on the columns it names, right after the schema upgrade, reports privileges it doesn't declare without revoking them, and stops before the catalog is seeded if a declared privilege is still missing or a role holds on a whole table a privilege declared for only some of its columns. A declared role listed in ROLES_AWAITING_PROVISIONING that doesn't exist yet is named in the job summary and skipped instead. | mechanical | enforced | yes | CI job `seed`, `catalog-runtime-integration` | `seed-v2-database.yml:seed`, `v2-integration-check.yml:catalog-runtime-integration` | yes |
+| `api.v2.runtime-privileges-declared` | Every v2 table has an entry for each managed database role in policyengine_api/data/v2/privileges.py; a privilege declared on only some of a table's columns names columns of that table and is never also declared on the whole table; and every privilege a v2 connector module needs (from its reads, creates, updates or deletes module name and its row locks) is declared for the role that runs it, on the whole table. | mechanical | enforced | yes | pytest `tests/unit/v2/test_privilege_manifest.py` | `pr.yml:test` | yes |
+| `api.v2.runtime-privileges-tested-as-runtime-role` | The cross-database integration tests run the v2 user services and the parameter tree read as policyengine_v2_runtime, catalog publication as policyengine_v2_catalog_writer, the comparison-run services as policyengine_stage12_<environment>, and the simulation API's table writes as policyengine_v2_simulation_service and policyengine_v2_modal_worker, holding only their declared privileges, so a statement those privileges don't allow fails CI. | mechanical | enforced | yes | CI job `v2-integration-check`, `catalog-runtime-integration` | `pr.yml:v2-integration-check`, `v2-integration-check.yml:catalog-runtime-integration` | — |
 
 ```mermaid
 flowchart LR