Refuse dataset years the country model does not project to

changelog.d/729.added.md

@@ -0,0 +1 @@
+Added `policyengine.provenance.supported_years(country_id, dataset_path)`, which returns the years the installed country model supports a single-year dataset for: the years its `extend_single_year_dataset` projects the file to, built as the model's own `Simulation` builds them and without running any formula, from the data year onwards. The UK runs to policyengine-uk's default projection end (2024–2032 for a 2024–25 dataset on policyengine-uk 2.151.0). The US runs from the data year to the earlier of policyengine-us's default projection end and the last year of spm-calculator's SPM forecast (2024–2035 for `populace_us_2024` on policyengine-us 2.2.1 and spm-calculator 1.0.0), so society-wide runs no longer accept US years before the data year, such as 2022 and 2023. The UK qualification check and execution receipts now use this function rather than their own copies of the rule. Added `UnsupportedDatasetYearError`, a `ValueError` whose message and `country_id`, `dataset`, `years` and `supported_years` attributes name the refused years, the country and the supported years.

changelog.d/729.changed.md

@@ -0,0 +1 @@
+`create_datasets` and `ensure_datasets` (UK and US) now raise `UnsupportedDatasetYearError` for a requested year the source dataset is not supported for, after making the source available and before creating any year file; `allow_unsupported_years=True` turns the check off. The US used to create year files for any year, from inputs policyengine-core carried forward or uprated, and the UK failed with `No dataset found for year ...`. Year files that already exist are loaded without the check, so a US year file that an earlier release created for an unsupported year is still reused until it is deleted.

docs/microsim.md

@@ -152,6 +152,42 @@ List datasets already known to the country:
 pe.us.load_datasets()  # or pe.uk.load_datasets()
 ```
 
+### Supported years
+
+A dataset is supported only for the years the installed country model
+projects it to. `supported_years` builds that projection from a single-year
+dataset file as the model's own `Simulation` does, with the model's
+`extend_single_year_dataset` and without running any formula, and returns
+the years from the dataset's data year onwards:
+
+```python
+from policyengine.provenance import supported_years
+
+supported_years("us", "./data/populace_us_2024.h5")  # 2024 to 2035
+supported_years("uk", "./data/microcosm_uk_2024_25.h5")  # 2024 to 2032
+```
+
+- **UK:** from the data year to policyengine-uk's default projection end,
+  `PROJECTION_END_YEAR` (2032 in policyengine-uk 2.151.0), or the end of a
+  capital gains projection that ends earlier.
+- **US:** from the data year to the earlier of policyengine-us's default
+  projection end (the last year in its `gov.bls.cpi.cpi_u` parameter file,
+  2035 in policyengine-us 2.2.1) and the last year of spm-calculator's SPM
+  forecast (2035 in spm-calculator 1.0.0), after which the poverty figures
+  have no thresholds. Society-wide runs are not supported for years before
+  the data year, such as 2022 and 2023 for Populace US 2024, because the
+  certified dataset holds almost no data for them.
+
+A calculation for another year still returns results, because
+policyengine-core carries the latest projected inputs forward or uprates
+them, so a successful run is not evidence of support. `create_datasets` and
+`ensure_datasets` therefore raise `UnsupportedDatasetYearError` (from
+`policyengine.provenance`) for any other year before they create a year
+file. The error's `country_id`, `dataset`, `years` and `supported_years`
+attributes name the request and the years that are supported. Pass
+`allow_unsupported_years=True` to create the year files anyway. Year files
+that already exist in the data folder are loaded without the check.
+
 ### US local-area dataset
 
 Alongside the certified national default, the bundle registers a US dataset for

src/policyengine/provenance/__init__.py

@@ -7,6 +7,7 @@
 .. code-block:: python
 
     from policyengine.provenance import (
+        supported_years,
         get_release_manifest,
         get_data_release_manifest,
         build_trace_tro_from_release_bundle,
@@ -42,6 +43,12 @@
 from .dataset_materialization import (
     materialize_dataset as materialize_dataset,
 )
+from .dataset_years import (
+    UnsupportedDatasetYearError as UnsupportedDatasetYearError,
+)
+from .dataset_years import (
+    supported_years as supported_years,
+)
 from .manifest import (
     CertifiedDataArtifact as CertifiedDataArtifact,
 )

src/policyengine/provenance/dataset_years.py

@@ -0,0 +1,192 @@
+"""The years a country model supports a single-year dataset for.
+
+A supported year is one the installed country model projects its single-year
+dataset to. The model's own ``Simulation`` builds that projection from a
+single-year file with ``extend_single_year_dataset``; :func:`supported_years`
+builds it the same way, without running any formula, and keeps the years from
+the dataset's data year onwards.
+
+A calculation for another year still succeeds: policyengine-core fills a year
+outside the projection by carrying the latest projected inputs forward or
+uprating them. A result there is therefore not evidence that its year is
+supported, and ``create_datasets`` and ``ensure_datasets`` refuse such a year
+with :class:`UnsupportedDatasetYearError` unless asked not to.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Iterable
+from pathlib import Path
+from typing import Any, Union
+
+COUNTRY_IDS = ("uk", "us")
+
+
+class UnsupportedDatasetYearError(ValueError):
+    """Requested years are outside the years a dataset is supported for.
+
+    Attributes:
+        country_id: The country, ``"uk"`` or ``"us"``.
+        dataset: The dataset the years were requested for, as requested.
+        years: The requested years that are not supported, sorted.
+        supported_years: Every year the dataset is supported for, sorted.
+    """
+
+    def __init__(
+        self,
+        country_id: str,
+        dataset: str,
+        years: Iterable[int],
+        supported_years: Iterable[int],
+    ):
+        self.country_id = country_id
+        self.dataset = dataset
+        self.years = sorted({int(year) for year in years})
+        self.supported_years = sorted({int(year) for year in supported_years})
+        super().__init__(self._message())
+
+    def __reduce__(self) -> tuple[Any, ...]:
+        # The default reduction passes only the message back to __init__.
+        return (
+            type(self),
+            (self.country_id, self.dataset, self.years, self.supported_years),
+        )
+
+    def _message(self) -> str:
+        noun = "year" if len(self.years) == 1 else "years"
+        requested = ", ".join(str(year) for year in self.years)
+        return (
+            f"{self.country_id.upper()} dataset {self.dataset!r} is not supported "
+            f"for {noun} {requested}. Supported years: "
+            f"{_describe_years(self.supported_years)}. The installed model "
+            "projects the dataset only to its supported years; policyengine-core "
+            "would fill any other year by carrying the latest projected inputs "
+            "forward or uprating them. Pass allow_unsupported_years=True to "
+            "create year files for it anyway."
+        )
+
+
+def supported_years(country_id: str, dataset_path: Union[str, Path]) -> list[int]:
+    """Return the years the installed country model supports a dataset for.
+
+    ``dataset_path`` is a local single-year dataset file, such as the
+    certified source that ``create_datasets`` creates year files from. The
+    multi-year dataset is built as the country model's own ``Simulation``
+    builds it for a single-year file, with the model's
+    ``extend_single_year_dataset`` and its default projection end, and
+    without running any formula. The years it holds from the dataset's data
+    year onwards are supported, with these country rules:
+
+    - **UK:** policyengine-uk projects to ``PROJECTION_END_YEAR``, or to the
+      last year of a capital gains projection that ends earlier.
+    - **US:** policyengine-us projects to the last year in its
+      ``gov.bls.cpi.cpi_u`` parameter file. Years after the last year of
+      spm-calculator's SPM forecast are not supported, since the poverty
+      figures have no thresholds there. Years before the data year are not
+      supported either, although the SPM forecast covers 2022 and 2023: the
+      certified ``populace_us_2024`` holds almost no data for them (its
+      household weights sum to 57,240 in each of 2022 and 2023, against
+      122,537,088 in 2024, and employment income is zero).
+
+    Args:
+        country_id: ``"uk"`` or ``"us"``.
+        dataset_path: Path of the single-year dataset file.
+
+    Returns:
+        The supported years, sorted ascending without duplicates.
+    """
+
+    if country_id == "uk":
+        return _uk_supported_years(Path(dataset_path))
+    if country_id == "us":
+        return _us_supported_years(Path(dataset_path))
+    raise ValueError(
+        f"Unknown country {country_id!r}; expected one of {list(COUNTRY_IDS)}."
+    )
+
+
+def refuse_unsupported_years(
+    country_id: str,
+    dataset: str,
+    dataset_path: Union[str, Path],
+    years: Iterable[int],
+) -> None:
+    """Raise :class:`UnsupportedDatasetYearError` if a year is not supported.
+
+    The supported years are computed from ``dataset_path`` only when ``years``
+    holds at least one year.
+
+    Args:
+        country_id: ``"uk"`` or ``"us"``.
+        dataset: The dataset as requested, named in the error.
+        dataset_path: Path of the single-year dataset file.
+        years: The requested years.
+    """
+
+    requested = sorted({int(year) for year in years})
+    if not requested:
+        return
+    supported = supported_years(country_id, dataset_path)
+    unsupported = [year for year in requested if year not in supported]
+    if unsupported:
+        raise UnsupportedDatasetYearError(country_id, dataset, unsupported, supported)
+
+
+def _uk_supported_years(dataset_path: Path) -> list[int]:
+    from policyengine_uk.data.dataset_schema import UKSingleYearDataset
+    from policyengine_uk.data.economic_assumptions import extend_single_year_dataset
+    from policyengine_uk.system import system
+
+    dataset = UKSingleYearDataset(str(dataset_path))
+    data_year = _data_year(dataset)
+    multi_year_dataset = extend_single_year_dataset(dataset, system.parameters)
+    return sorted(
+        year for year in _projected_years(multi_year_dataset) if year >= data_year
+    )
+
+
+def _us_supported_years(dataset_path: Path) -> list[int]:
+    from policyengine_us.data.dataset_schema import USSingleYearDataset
+    from policyengine_us.data.economic_assumptions import extend_single_year_dataset
+    from spm_calculator.rolling_forecast import load_forecast
+
+    dataset = USSingleYearDataset(file_path=str(dataset_path))
+    data_year = _data_year(dataset)
+    multi_year_dataset = extend_single_year_dataset(dataset)
+    last_spm_forecast_year = max(int(year) for year in load_forecast().years)
+    return sorted(
+        year
+        for year in _projected_years(multi_year_dataset)
+        if data_year <= year <= last_spm_forecast_year
+    )
+
+
+def _data_year(dataset: Any) -> int:
+    """The year of a single-year dataset's observed data."""
+
+    return int(str(dataset.time_period)[:4])
+
+
+def _projected_years(multi_year_dataset: Any) -> set[int]:
+    """The years a country model's multi-year dataset holds."""
+
+    years = getattr(multi_year_dataset, "years", None)
+    if years is None:
+        years = list(multi_year_dataset.datasets)
+    return {int(year) for year in years}
+
+
+def _describe_years(years: list[int]) -> str:
+    """Sorted years as runs of consecutive years, such as ``2024–2035``."""
+
+    if not years:
+        return "none"
+    runs: list[list[int]] = [[years[0], years[0]]]
+    for year in years[1:]:
+        if year == runs[-1][1] + 1:
+            runs[-1][1] = year
+        else:
+            runs.append([year, year])
+    return ", ".join(
+        str(first) if first == last else f"{first}–{last}" for first, last in runs
+    )

src/policyengine/provenance/receipt.py

@@ -127,6 +127,7 @@
     model_validator,
 )
 
+from policyengine.provenance import dataset_years
 from policyengine.provenance.bytecode import code_shape as _code_shape
 from policyengine.provenance.bytecode import compile_source
 from policyengine.provenance.canonical import (
@@ -3818,25 +3819,14 @@ def _receipt_years(years: Any) -> list[int]:
 def _projected_years(
     dataset_path: str, dataset_sha256: str, model_version: str
 ) -> tuple[int, ...]:
-    """Years the installed policyengine-uk projects a single-year dataset to.
+    """Years the installed policyengine-uk supports a single-year dataset for.
 
-    The same rule as the qualification ``supported_years`` gate: build the
-    multi-year dataset as ``policyengine_uk.Simulation`` does for a
-    single-year file, without running a formula, and keep the years from the
-    dataset's own year onwards. Cached by file bytes and model version.
+    :func:`policyengine.provenance.dataset_years.supported_years` for the UK,
+    the rule the qualification ``supported_years`` check also uses, cached by
+    file bytes and model version.
     """
 
-    from policyengine_uk.data.dataset_schema import UKSingleYearDataset
-    from policyengine_uk.data.economic_assumptions import extend_single_year_dataset
-    from policyengine_uk.system import system
-
-    dataset = UKSingleYearDataset(dataset_path)
-    first = int(str(dataset.time_period)[:4])
-    multi_year = extend_single_year_dataset(dataset, system.parameters)
-    years = getattr(multi_year, "years", None)
-    if years is None:
-        years = list(multi_year.datasets)
-    return tuple(sorted(int(year) for year in years if int(year) >= first))
+    return tuple(dataset_years.supported_years("uk", dataset_path))
 
 
 def _certificate_fields(

src/policyengine/provenance/uk_candidate.py

@@ -33,6 +33,7 @@
 from packaging.specifiers import SpecifierSet
 from packaging.version import Version
 
+from policyengine.provenance import dataset_years
 from policyengine.provenance.manifest import DataReleaseManifest, _specifier_matches
 from policyengine.provenance.qualification import (
     CandidatePair,
@@ -422,24 +423,15 @@ def data_year(dataset_path: Path) -> int:
 
 
 def projected_years_from_model(dataset_path: Path) -> list[int]:
-    """The years policyengine-uk projects a single-year dataset to.
+    """The years the installed policyengine-uk supports a dataset for.
 
-    Builds the multi-year dataset exactly as ``policyengine_uk.Simulation``
-    does for a single-year file (``extend_single_year_dataset`` with the
-    system's parameters), without running any formula.
+    :func:`policyengine.provenance.dataset_years.supported_years` for the UK:
+    the years ``extend_single_year_dataset`` projects the single-year file
+    to, built as ``policyengine_uk.Simulation`` builds them, from the data
+    year onwards.
     """
 
-    from policyengine_uk.data.dataset_schema import UKSingleYearDataset
-    from policyengine_uk.data.economic_assumptions import extend_single_year_dataset
-    from policyengine_uk.system import system
-
-    multi_year = extend_single_year_dataset(
-        UKSingleYearDataset(str(dataset_path)), system.parameters
-    )
-    years = getattr(multi_year, "years", None)
-    if years is None:
-        years = list(multi_year.datasets)
-    return sorted(int(year) for year in years)
+    return dataset_years.supported_years("uk", dataset_path)
 
 
 def supported_years(