[anth-volk]

Fixes #4224

Summary

Each stored v2 catalog version costs about 550 MB per environment, mostly its current-law parameter_values (about 1.2 million rows for both countries, about 391 MB) and its parameter_tree_children (about 108 MB). Nothing removed old versions, so every release added another 550 MB.

Catalog publication (scripts/publish_v2_metadata_catalog.py, run by the release workflow's seed jobs as policyengine_v2_catalog_writer) now records when it last published each version and, after its own transaction commits, removes those package-derived rows from older stored versions, at most two versions per run. A release that installs a pruned version again restores them, identically, from the installed packages. For a pruned version, the metadata routes that serve the removed rows answer 503 with Cache-Control: no-store; every other metadata route answers as before.

Migration scope: this changes the v2 catalog publication command and the /v2 metadata routes only. The Flask routes and /{country}/metadata are unchanged.

PolicyEngine.py version: this pull request includes the PolicyEngine bundle update to 6.16.0, the simulation API's latest, by merging #4223's branch, so the release check that the Public API's bundle matches the simulation API's passes. It is meant to merge on its own: it replaces #4223, and Fixes #4222, that update's issue. Once this merges, #4223's commit is in master.

Design

1. Recording publications (record_catalog_publication in policyengine_api/data/v2/catalog/publication_pruning.py). Every publication sets the new nullable tax_benefit_model_versions.catalog_last_published_at for each version it publishes, inside its transaction: a new version, a republish with no change, and a republish that restores a pruned version.
2. Which versions are kept (catalog_versions_to_prune). For each country's model, publication keeps the installed version and the two most recently published other versions (KEPT_OTHER_CATALOG_VERSIONS), ordered by catalog_last_published_at, or by created_at where that is null (rows stored before the column existed), with ties ordered by created_at and then version text. It chooses every other stored version not pruned already. Two are kept because the release workflow seeds before it deploys and promotes, and puts the previous revision back when promotion fails: after one failed promotion the serving version is the second most recently published other one, and after a rollback to an older version, the rollback's publication made that version the most recently published. This function is the only place the rule is decided.
3. How many per run (catalog_versions_to_prune_now). A run prunes at most two of the chosen versions (MAX_CATALOG_VERSIONS_PRUNED_PER_RUN), oldest created_at first. A release's US and UK versions are created in one transaction, so two is one release. The evidence's catalog_versions_left_to_prune counts the rest, which later publications prune.
4. When (_prune_older_versions in publication.py). After the publication transaction has committed, one transaction per version (_prune_if_still_chosen). Each takes the publication advisory lock, reads the model's stored versions again and applies catalog_versions_to_prune again; a version the rule no longer chooses (a publication that republished it committed meanwhile) is skipped and reported neither as pruned nor as failed. Otherwise prune_catalog_version sets catalog_values_pruned_at only if it is still null, then deletes, so the marker and the deletes commit or roll back together.
5. Failures don't fail the command (_returned_or_raised). Any exception while pruning a version, or while reading the stored versions, is recorded instead of raised, and that version's transaction rolls back; the others are still pruned. Each failure is listed in the evidence's catalog_pruning_failures (country, version or null, exception class, and a message naming only the database's error class and SQLSTATE, or for any other exception only its class, never the statement, its parameters or row data) and printed to stderr. The command exits 0, because the publication has committed, and the next publication retries.
6. Restoration (restore_pruned_catalog_versions). When publication runs for a stored version whose catalog_values_pruned_at is set, it inserts the version's current-law values from the staged catalog with the statement a new version's publication uses (so with the same catalog value IDs) and clears the column, inside the publication transaction and before the usual exact comparison of the stored version with the staged catalog. The existing writers then write the version's parameter tree and search file, as for any version that has none. The integration test checks that the restored rows equal what the version's first publication wrote.
7. Routes. require_current_law_values raises CatalogValuesPrunedError (a MetadataCatalogUnavailableError); run_metadata_read maps it to 503 with Cache-Control: no-store. Catalog selection moved from V2MetadataService to MetadataCatalogSelection in services/v2/metadata/catalog_version_selection/services.py (a move: services.py was 5 lines under its length limit). The seven service reads of pruned rows select with reads_pruned_data=True: a pruned selection answers before the rows are read, and a selection the process kept while the version was pruned is checked against the database again first (one more query, only then), so once a publication restores the version the process serves it again and keeps the new selection. A selection kept from before the pruning reads catalog_values_pruned_at again when a read finds no rows.

Renamed in the move, since another module now calls them: _select_catalog to MetadataCatalogSelection.select, _select_paginated_catalog to select_page, _require_unpruned to require_unpruned.

Exactly what is deleted

For each pruned version, in one transaction:

• its current-law parameter_values: rows whose parameter_id is one of the version's parameters and whose policy_id and dynamic_id are both null;
• its parameter_tree_children rows;
• its parameter_search_files rows;

and its tax_benefit_model_versions.catalog_values_pruned_at is set to the current time.

Nothing else is deleted or changed. In particular: no parameters row (parameter_values.parameter_id references it with ON DELETE CASCADE, so deleting one would delete saved reforms' values), no tax_benefit_model_versions row or other column of it, no parameter_nodes, variables, regions, datasets, policies or dynamics row, and no parameter_values row with a policy_id or dynamic_id. No foreign key references the three tables, so the deletes cascade nowhere (checked in the disposable database with pg_constraint).

What the first runs in each environment delete

Before this, no version has catalog_last_published_at, so the first publication orders the stored versions by created_at: it keeps the installed version and the two most recently created others, chooses the rest, and prunes the oldest release's two versions (about 1.2 million value rows, about 550 MB). Each later publication prunes the next oldest release's versions until none are left; the evidence's catalog_versions_left_to_prune shows how many remain. The stored versions in staging and production weren't counted for this pull request; an earlier estimate put production at about nine older versions.

503 for a pruned version

For a pruned version these routes answer 503, with Cache-Control: no-store and no ETag, and the message "PolicyEngine.py <version>'s current-law parameter values for <country> were removed from the catalog, with its parameter tree and search file; they return when PolicyEngine.py <version> is installed again":

• GET /v2/parameter-values
• GET /v2/parameter-values/{value_id}
• GET /v2/parameters/children
• GET /v2/parameters/search-file, also when revalidated with If-None-Match
• GET /v2/model-catalog/variables (embeds the current-law values of the parameters variables add or subtract)
• GET /v2/model-catalog/parameters (embeds each parameter's current-law values)

The OpenAPI document describes the 503 on each. The other metadata routes serve kept rows and answer 200. A process that built a version's /v2/model-catalog/variables body before the version was pruned keeps serving that body (200, and 304 to its ETag) for the process's life; the body equals what restoring the version writes. The process's whole-version response cache never keeps a 503, and requests waiting on the same read receive its Cache-Control too.

Privileges and migration

• policyengine_api/data/v2/privileges.py: policyengine_v2_catalog_writer gains DELETE on parameter_values, parameter_tree_children and parameter_search_files. It already held UPDATE on tax_benefit_model_versions, which both new columns need. No other role changes. The deploy's existing scripts/reconcile_v2_privileges.py apply step grants them before the catalog is seeded.
• Revision 334859ef4fec (parent c614b6bcccfc), autogenerated with uv run alembic -c alembic-v2.ini revision --autogenerate and formatted with ruff format as the other revisions are, adds the nullable catalog_last_published_at and catalog_values_pruned_at (timestamp with time zone) and drops them on downgrade. It replaces an earlier revision of this branch, ca12427ceb63, which only a disposable database had applied. V2_ALEMBIC_HEAD_REVISION is updated.
• A downgrade past 334859ef4fec is safe only while no version is pruned; the Alembic skill and the Stage 9 runbook's rollback section give the check and what to do otherwise.
• docs/engineering/skills/alembic-migrations.md said a package-derived catalog seeder "must not transform or delete retained domain data". Its new section "Catalog pruning: the seeder's one deletion" names pruning as the one exception, with the scope above, and says that the maintainers' approval of the exception is this pull request's review. It also records the accepted first-release window below as an exception to the rule that the serving revision works with the database changed on its own. The Stage 9 runbook, the v2 code organization skill and two docstrings said a stored catalog version never changes; they now say its current-law values may be pruned and restored identically from the same packages.

Environment variables

Seed job time limit. .github/workflows/seed-v2-database.yml's seed job now has timeout-minutes: 60 (was 30), because its release may publish a new version's catalog and prune in the same job. No step, variable or secret changes.

This change adds no environment variable and changes no secret or variable wiring. I searched the diff against origin/master for os.environ, getenv, environ, BaseSettings, settings fields, env: blocks, secrets. and vars.; the only match is the new integration tests reading V2_MIGRATION_DATABASE_URL.

Variable	Read by	Provider and wiring
V2_MIGRATION_DATABASE_URL	the new integration tests (existing variable)	literal job-level env: of catalog-runtime-integration in .github/workflows/v2-integration-check.yml, pointing at that job's postgres:17 service; the changed step is in that job
V2_ALEMBIC_DISPOSABLE_TEST, DEPLOYMENT_ENVIRONMENT	the integration fixtures (existing)	same job-level env:
V2_DATA_WRITE_DATABASE_URL	the publication command, unchanged; pruning uses the same engine	GitHub Actions environment secret, present in staging and production (gh secret list --env), passed in the publication step's env: in seed-v2-database.yml
HUGGING_FACE_TOKEN	the publication command's region check, unchanged	organization secret PE_UK_PRIVATE_HF_READ_TOKEN, readable by this repository (gh api repos/PolicyEngine/policyengine-api/actions/organization-secrets), passed in the same step's env:
V2_MIGRATION_DATABASE_URL (seed job)	the schema upgrade, qualification and privilege steps, unchanged	GitHub Actions environment secret in staging and production (gh secret list --env), set in each step's env:
V2_SUPABASE_PROJECT_REF, V2_SUPABASE_ENVIRONMENT	the seed job's database isolation check, unchanged	GitHub Actions environment variables in staging and production (gh variable list --env), set in the job's env:
PRODUCTION_V2_SUPABASE_PROJECT_REF	the same check, unchanged	GitHub Actions repository variable, set in the job's env:

Suppressions

• policyengine_api/data/v2/catalog/publication.py, _returned_or_raised: except Exception as error: # noqa: BLE001 # pruning failures are recorded, never raised. Pruning runs after the publication has committed, so no exception may fail the command, and the exception is returned to be recorded without its message. Ruff's BLE001 accepts a broad except only when it re-raises or logs the traceback (logging.exception or exc_info), and a traceback can print the statement's parameters and row data to the seed job's log. One helper holds the only suppression and serves both the read of the stored versions and each version's pruning.

Testing

All test runs use the disposable PostgreSQL 17 database policyengine_v2_alembic_test on 127.0.0.1:55473, with V2_MIGRATION_DATABASE_URL, V2_ALEMBIC_DISPOSABLE_TEST=1, DEPLOYMENT_ENVIRONMENT=test and UV_PYTHON=3.12 set, and each test selected by its node ID:

uv run --frozen --extra dev python -m pytest -q -p no:cacheprovider -rA --no-header <node IDs>

Red, before each change

1. 2026-11-06 14:52:24 UTC, the first new test files: collection failed with ModuleNotFoundError: No module named 'policyengine_api.data.v2.catalog.publication_pruning' and ImportError: cannot import name 'CatalogValuesPrunedError' / 'CatalogPruningFailure'.

2. 2026-11-06 14:52:46 UTC, the first changed tests in existing files: no attribute PrunedCatalogVersion / CatalogPruningFailure; the writer held DELETE on no table; the revision chain lacked the new revision; KeyError: 'catalog_values_pruned_at'; waiting requests' copies had no Cache-Control; the workflow didn't run the new test file; UndefinedColumn: column "catalog_values_pruned_at" does not exist.

3. 2026-11-06 16:44:45 UTC, a failed read of the stored versions: the message and stderr named version "None", and sqlalchemy.exc.DataError (DivisionByZero) was raised out of publish_catalog after the publication committed.

4. 2026-11-06 17:26:34 UTC, review round 1, batch 1 (10 failed, 2 passed):

   • test_a_failure_that_is_not_a_database_error_names_only_its_class: the message said "the database reported KeyError";
   • test_a_run_prunes_at_most_two_versions_oldest_first, test_a_run_with_no_more_than_the_limit_prunes_them_all: no attribute catalog_versions_to_prune_now;
   • test_completion_evidence_contains_only_reviewed_non_secret_fields: unexpected keyword catalog_versions_left_to_prune;
   • test_a_selection_kept_while_pruned_reads_only_the_marker_before_answering: the one statement was the parameter values read, not the marker read;
   • test_a_selection_kept_while_pruned_serves_the_version_once_restored: CatalogValuesPrunedError after the restoration;
   • test_a_process_that_selected_a_pruned_version_serves_it_once_restored: 503 after the restoration;
   • test_a_version_kept_by_the_time_its_transaction_runs_is_not_pruned: the US old version was pruned anyway;
   • test_a_non_database_error_rolls_back_that_version_and_publication_succeeds: KeyError raised out of publish_catalog;
   • test_a_run_prunes_at_most_two_versions_and_the_next_run_the_rest: the first run pruned all four;
   • passed without a change, as expected, because they pin behaviour that already existed: the variables-route step added to test_a_process_that_selected_a_version_before_its_pruning_answers_503, and test_a_version_without_current_law_rows_is_marked_pruned_with_zero_counts.

5. 2026-11-06 17:40:54 UTC, review round 1, batch 2 (22 failed, 2 passed): the rule tests failed with TypeError: ... unexpected keyword argument 'last_published_at' or with the old rule's choice (for example [('policyengine-us', '5.0.2')] == [('policyengine-us', '5.0.0')]); KeyError: 'catalog_last_published_at' in the model test; the revision test found one op.add_column, not two; UndefinedColumn: column "catalog_last_published_at" does not exist in the migration and publication-time integration tests; the integration tests with four stored versions found the old rule pruning a version the new rule keeps. The route test and the process restoration test passed, since both rules prune their oldest version.

6. Review round 2: three tests of behaviour the code already had. On unchanged code (18:16:20 UTC) all three passed; each was then run with the code it protects broken temporarily, and the code restored exactly afterwards (git checkout, empty git diff of policyengine_api/data/v2/catalog):

   • test_a_pruning_transaction_chooses_again_after_waiting_for_the_lock, with _acquire_publication_lock(connection) deleted from _prune_if_still_chosen (18:16:48 UTC): assert PrunedRows(parameter_values=2, parameter_tree_children=3, parameter_search_files=1) is None;
   • test_a_failed_republish_of_a_pruned_version_leaves_it_pruned, with restoration clearing catalog_values_pruned_at through a separate connection committed outside the publication transaction (18:17:06 UTC): assert None == datetime.datetime(2026, 11, 6, 18, 17, 9, 56630, tzinfo=datetime.timezone.utc), where None = _pruned_at(..., 'policyengine-us', '5.0.1');
   • test_restoring_a_version_stored_before_catalog_format_2_writes_values_only, with the parameter tree writer's skip of a version without a catalog format removed (18:17:19 UTC): {'tree_rows': [('', 'gov', 0, 'Gov', 'node', 1, ...), ...]} != {'tree_rows': []}.

   The lock test synchronizes on pg_locks: it waits until the pruning thread is blocked on the advisory lock, or has finished, and its sleep only paces the polling. It passed five runs in a row.

Green, on the branch tip

2026-11-06 18:20:04 UTC, commit 6d5ef4d64: 64 passed (53 node IDs, 64 cases with parametrization).

tests/unit/v2/test_catalog_publication_pruning.py::test_keeps_the_installed_and_the_two_most_recently_published_others
tests/unit/v2/test_catalog_publication_pruning.py::test_an_installed_version_older_than_the_others_is_kept_after_a_rollback
tests/unit/v2/test_catalog_publication_pruning.py::test_versions_already_pruned_are_not_pruned_again
tests/unit/v2/test_catalog_publication_pruning.py::test_each_country_model_is_chosen_from_separately
tests/unit/v2/test_catalog_publication_pruning.py::test_three_stored_versions_of_a_model_leave_nothing_to_prune
tests/unit/v2/test_catalog_publication_pruning.py::test_a_pruning_failure_names_the_database_error_but_no_statement_or_row
tests/unit/v2/test_catalog_publication.py::test_completion_evidence_contains_only_reviewed_non_secret_fields
tests/unit/v2/test_catalog_initialization.py::test_main_succeeds_and_prints_each_pruning_failure_to_stderr
tests/unit/v2/test_privilege_manifest.py::test_the_catalog_writer_deletes_only_the_rows_catalog_pruning_removes
tests/unit/v2/test_alembic_v2.py::test_v2_revision_chain_has_generated_policy_and_user_identity_changes
tests/unit/v2/test_alembic_v2.py::test_parameter_search_file_revision_creates_only_the_files_table
tests/unit/v2/test_alembic_v2.py::test_catalog_pruning_revision_adds_only_two_nullable_columns
tests/unit/v2/test_models.py::test_catalog_pruning_marker_is_a_nullable_timezone_aware_timestamp
tests/unit/v2/test_metadata_service_pruned_versions.py::test_reads_of_pruned_data_name_the_country_and_version
tests/unit/v2/test_metadata_service_pruned_versions.py::test_a_pruned_version_answers_so_even_where_a_read_needs_no_values
tests/unit/v2/test_metadata_service_pruned_versions.py::test_reads_of_kept_data_still_answer_for_a_pruned_version
tests/unit/v2/test_metadata_service_pruned_versions.py::test_a_selection_kept_from_before_pruning_reads_the_marker_again
tests/unit/v2/test_metadata_service_pruned_versions.py::test_an_unpruned_version_whose_read_finds_nothing_answers_as_before
tests/unit/v2/test_metadata_service_pruned_versions.py::test_a_read_that_finds_rows_does_not_read_the_marker
tests/unit/v2/test_metadata_pruned_version_routes.py::test_routes_serving_pruned_data_answer_503_that_no_cache_keeps
tests/unit/v2/test_metadata_pruned_version_routes.py::test_revalidating_a_pruned_version_search_file_answers_503_too
tests/unit/v2/test_metadata_pruned_version_routes.py::test_a_kept_whole_version_response_cache_does_not_keep_the_pruned_answer
tests/unit/v2/test_metadata_pruned_version_routes.py::test_openapi_describes_the_503_of_a_pruned_version
tests/unit/v2/test_metadata_response_cache.py::test_waiting_requests_receive_a_failed_read_s_cache_control
tests/unit/test_alembic_workflows.py::test_reusable_v2_integration_check_uses_disposable_databases_and_coverage
tests/integration/test_alembic_v2_catalog_tables.py::test_catalog_pruning_columns_are_added_null_and_dropped_without_loss
tests/integration/test_v2_catalog_publication_pruning.py::test_publication_prunes_versions_older_than_the_kept_ones_as_the_writer
tests/integration/test_v2_catalog_publication_pruning.py::test_republishing_a_pruned_version_restores_what_its_publication_wrote
tests/integration/test_v2_catalog_publication_pruning.py::test_a_pruning_failure_rolls_back_that_version_and_publication_succeeds
tests/integration/test_v2_catalog_publication_pruning.py::test_routes_answer_503_for_a_pruned_version_and_serve_its_kept_rows
tests/integration/test_v2_catalog_publication_pruning.py::test_a_process_that_selected_a_version_before_its_pruning_answers_503
tests/unit/v2/test_catalog_publication_pruning.py::test_a_failure_before_any_version_is_chosen_names_no_version
tests/unit/v2/test_catalog_initialization.py::test_main_prints_a_pruning_failure_that_chose_no_version_to_stderr
tests/integration/test_v2_catalog_publication_pruning.py::test_a_failure_to_read_the_stored_versions_is_reported_and_publication_succeeds
tests/unit/v2/test_catalog_publication_pruning.py::test_a_failure_that_is_not_a_database_error_names_only_its_class
tests/unit/v2/test_catalog_publication_pruning.py::test_a_run_prunes_at_most_two_versions_oldest_first
tests/unit/v2/test_catalog_publication_pruning.py::test_a_run_with_no_more_than_the_limit_prunes_them_all
tests/unit/v2/test_metadata_service_pruned_versions.py::test_a_selection_kept_while_pruned_reads_only_the_marker_before_answering
tests/unit/v2/test_metadata_service_pruned_versions.py::test_a_selection_kept_while_pruned_serves_the_version_once_restored
tests/integration/test_v2_catalog_publication_pruning.py::test_a_process_that_selected_a_pruned_version_serves_it_once_restored
tests/integration/test_v2_catalog_publication_pruning.py::test_a_version_kept_by_the_time_its_transaction_runs_is_not_pruned
tests/integration/test_v2_catalog_publication_pruning.py::test_a_non_database_error_rolls_back_that_version_and_publication_succeeds
tests/integration/test_v2_catalog_publication_pruning.py::test_a_run_prunes_at_most_two_versions_and_the_next_run_the_rest
tests/integration/test_v2_catalog_publication_pruning.py::test_a_version_without_current_law_rows_is_marked_pruned_with_zero_counts
tests/unit/v2/test_catalog_publication_pruning.py::test_a_seed_whose_promotion_failed_keeps_the_version_still_serving
tests/unit/v2/test_catalog_publication_pruning.py::test_a_version_republished_by_a_rollback_is_kept_by_the_next_release
tests/unit/v2/test_catalog_publication_pruning.py::test_a_version_without_a_publication_time_is_ordered_by_its_creation
tests/unit/v2/test_catalog_publication_pruning.py::test_versions_published_at_one_instant_are_ordered_by_creation_then_text
tests/unit/v2/test_models.py::test_catalog_publication_time_is_a_nullable_timezone_aware_timestamp
tests/integration/test_v2_catalog_publication_pruning.py::test_every_publication_records_when_it_published_the_version
tests/integration/test_v2_catalog_publication_pruning.py::test_a_pruning_transaction_chooses_again_after_waiting_for_the_lock
tests/integration/test_v2_catalog_publication_pruning.py::test_a_failed_republish_of_a_pruned_version_leaves_it_pruned
tests/integration/test_v2_catalog_publication_pruning.py::test_restoring_a_version_stored_before_catalog_format_2_writes_values_only

The deletion-scope and restoration integration tests publish as policyengine_v2_catalog_writer, and the route integration tests read as policyengine_v2_runtime, each holding only its declared privileges.

Fixes after the pull request was opened

• test_no_file_names_the_former_mixed_module refuses any dotted path ending in .catalog_selection (the removed policyengine_api.data.v2.catalog.catalog_selection module), and the new metadata.catalog_selection package matched it. Red: the test failed locally on policyengine_api/services/v2/metadata/services.py. The package is now catalog_version_selection; green: the test and the branch's 48 changed unit test cases pass.
• test_routes_serving_pruned_data_answer_503_that_no_cache_keeps built one parametrized path with uuid4() at import, so each release test shard collected a different test ID and the shard check failed. Red: two pytest --collect-only runs gave different IDs. The path now uses a fixed UUID; green: two collections give identical IDs and the tests pass.

Other checks

• On a freshly created empty database, after regenerating the revision: uv run alembic -c alembic-v2.ini upgrade head (to 334859ef4fec), alembic -c alembic-v2.ini check ("No new upgrade operations detected."), downgrade -1 (to c614b6bcccfc), upgrade head, and check again: all passed.
• uv run python scripts/reconcile_v2_privileges.py apply --create-missing-roles, then verify: both exit 0 ("Every declared v2 table privilege is held (test).").
• python scripts/export_migration_contracts.py: no generated file changed. python -m scripts.rules validate: passed.
• make quality on the branch tip: passed and changed no file.
• uv run mypy on the changed modules under policyengine_api/data/v2 (which the configured mypy run doesn't cover): 13 errors, 11 of them identical on the base commit. The other two are the new columns' Field(default=None, sa_type=sa.DateTime(timezone=True)), the declaration every timestamp column in the v2 models uses; mypy reports the same error for the existing ones (for example ParameterValue.end_date). The pruning module has none.
• BASE_SHA=origin/master bash .github/scripts/check_changelog_fragment.sh: one fragment added.

Risks

• First-release window (accepted by the maintainers). During the first release that includes pruning, the seed job prunes while the previous revision, whose code doesn't read catalog_values_pruned_at, still serves. From the seed job until promotion, and until the next successful release if promotion fails, that revision answers a request naming a pruned version in policyengine_version on /v2/parameter-values, /v2/model-catalog/parameters and /v2/model-catalog/variables with a 200 holding no current-law values, Cache-Control: public, max-age=86400 and an ETag, so a client or shared cache can keep that empty answer for a day; /v2/parameters/children and /v2/parameters/search-file answer 503 and /v2/parameter-values/{value_id} 404. Requests without policyengine_version read the running version, which is kept. The maintainers decided to ship pruning turned on in this pull request and accept the window, because nothing in production calls these routes with an older version until the v3 switch. The Alembic skill records it as an exception to the serving-revision compatibility rule.
• Two failed promotions in a row. Keeping two most recently published others covers one failed promotion or a rollback. Two consecutive releases whose promotions fail, followed by a third release, can still prune the version that is serving.
• The first release publishes 6.16.0 and prunes in one seed job. Because this pull request carries the bundle update, its release both publishes the new 6.16.0 catalog (a new version's publication took 17 to 20 minutes in production for 6.12.1) and runs the first pruning, inside the seed job's 60-minute limit. Pruning starts only after the publication has committed, so a timeout during pruning leaves the 6.16.0 catalog published, rolls back only the version being pruned, and a rerun of the seed job continues; the deploy jobs wait for the seed job, so the release stops until that rerun.
• Time per run. A run prunes at most two versions, about 1.2 million deleted rows, after a publication that already took 17 to 20 minutes in production, inside the seed job's 60-minute limit. If the job still runs out of time while pruning, it fails although the publication committed: the versions already pruned stay pruned, the interrupted one rolls back, and a rerun continues. A database statement timeout on the writer role, if one is set (not checked), would instead be reported as a pruning failure and retried by every later run.
• Disk space. Deleted rows free space inside the tables for later publications to reuse. PostgreSQL returns it to the operating system only after VACUUM FULL or an equivalent rewrite, which this doesn't run, so the database's reported size won't fall right away.
• Downgrade. Downgrading past 334859ef4fec drops the record of which versions are pruned; the parent revision's code would serve them with empty values and fail to republish them. It is safe only while no version is pruned.
• Traffic rollback to an older revision. A Cloud Run revision that runs a stored version other than the installed one and the two most recently published others may run a pruned one. A revision with this change answers 503 on the six routes above. A revision from before it doesn't read catalog_values_pruned_at: for its running version, so also for every request without policyengine_version, it answers /v2/parameter-values and the model catalog parameters and variables routes with empty 200s, Cache-Control: public, max-age=300 (a day for a named version) and an ETag. Either way only a publication by code with this change, with that PolicyEngine.py version installed, restores the rows.

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 98.00000% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.19%. Comparing base (f932adf) to head (bad0e7d).

Files with missing lines	Patch %	Lines
...yengine_api/data/v2/catalog/publication_pruning.py	96.47%	1 Missing and 2 partials ⚠️
policyengine_api/data/v2/catalog/publication.py	95.74%	0 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4225      +/-   ##
==========================================
+ Coverage   92.14%   92.19%   +0.05%     
==========================================
  Files         296      298       +2     
  Lines       16696    16895     +199     
  Branches     2722     2756      +34     
==========================================
+ Hits        15384    15577     +193     
- Misses        795      800       +5     
- Partials      517      518       +1     

Flag	Coverage Δ
integration	52.95% <96.00%> (+0.53%)	⬆️
unit	89.97% <77.20%> (-0.23%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.