Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,9 @@ jobs:
- name: Validate limit-name registry
run: uv run tools/validate_limit_registry.py

- name: Validate warning-reason registry
run: uv run tools/validate_warning_reasons.py

- name: Test the registry validator
run: uv run --with jsonschema --with pyarrow --with pytest pytest tools/tests -q

Expand Down
4 changes: 3 additions & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,8 @@ The validators under `tools/` check the registries and vector suites without imp
```bash
uv run tools/validate_registry.py
uv run tools/validate_unit_registry.py
uv run tools/validate_limit_registry.py
uv run tools/validate_warning_reasons.py
uv run tools/validate_vectors.py
uv run tools/validate_frame_vectors.py
uv run tools/validate_unit_vectors.py
Expand All @@ -46,7 +48,7 @@ uv run --with jsonschema --with pyarrow --with pytest pytest tools/tests -q

The last command tests the validators themselves. [tools/README.md](tools/README.md) says what each validator checks. Each package's tests drive its implementation against the same vectors or fixtures, and `uv run pytest` runs them.

A change to `registries/codes.json` or `registries/units.json` also needs the packaged mirror under `python/spec-kit/src/dfo_spec_kit/` to match byte for byte. The validators fail if the two differ. Edit the file under `registries/` and copy it to the mirror. [registries/README.md](registries/README.md) has the format.
A change to `registries/codes.json`, `registries/units.json`, `registries/limit-names.json` or `registries/warning-reasons.json` also needs the packaged mirror under `python/spec-kit/src/dfo_spec_kit/` to match byte for byte. The validators fail if the two differ. Edit the file under `registries/` and copy it to the mirror. [registries/README.md](registries/README.md) has the format.

## Branches and commits

Expand Down
2 changes: 2 additions & 0 deletions python/spec-kit/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@ and this package adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Added

- Warning code `bilateral_reading_unavailable` in `registries/codes.json` and its bundled mirror: a `search.resolve` warning on the raw_text, parsed and dataset_resubmit doors. It marks a DAC2a total ODA answer or reading whose bilateral reading is unavailable or undecided, so its figures include donors' core contributions to multilateral agencies. It carries the reading's spec_id, and `params.why` names the reason, one of the reasons the warning-reason registry lists for this code, which may add params of its own.
- The warning-reason registry: `registries/warning-reasons.json`, its schema `schemas/registry/warning-reason-registry.schema.json`, a bundled mirror, and `dfo_spec_kit.registry.WARNING_REASONS` (a mapping of `WarningReason` entries keyed by code, then reason). It lists the values a warning carries in `params.why`, starting with the five reasons of `bilateral_reading_unavailable`. Where a reason adds params beside `why`, `WarningReason.params` holds them as (name, definition) pairs sorted by name. `load_warning_reasons` raises `ValueError` on an unknown `format_version` or a reason listed twice for a code. `tools/validate_warning_reasons.py` checks the file, that each code is an active warning, and its mirror.
- The limit-name registry: `registries/limit-names.json`, its schema
`schemas/registry/limit-name-registry.schema.json`, a bundled mirror, and
`dfo_spec_kit.registry.LIMIT_NAMES` (a mapping of `LimitName` entries keyed
Expand Down
22 changes: 22 additions & 0 deletions python/spec-kit/EXPORTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -243,6 +243,28 @@ kind = "function"
stability = "provisional"
reason = "bundled-data"

[[export]]
name = "WARNING_REASONS"
module = "dfo_spec_kit.registry"
kind = "constant"
stability = "provisional"
policy = "contents track registries/warning-reasons.json; a reason is added, never redefined"
reason = "bundled-data"

[[export]]
name = "WarningReason"
module = "dfo_spec_kit.registry"
kind = "model"
stability = "provisional"
reason = "bundled-data"

[[export]]
name = "load_warning_reasons"
module = "dfo_spec_kit.registry"
kind = "function"
stability = "provisional"
reason = "bundled-data"

# --- dfo_spec_kit.currency ---------------------------------------------
# The currency-code shape test, in a module that reads no data file.
[[export]]
Expand Down
1 change: 1 addition & 0 deletions python/spec-kit/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ build-backend = "uv_build"
source-include = [
"src/dfo_spec_kit/registry/codes.json",
"src/dfo_spec_kit/registry/limit-names.json",
"src/dfo_spec_kit/registry/warning-reasons.json",
"src/dfo_spec_kit/units/units.json",
]

Expand Down
72 changes: 72 additions & 0 deletions python/spec-kit/src/dfo_spec_kit/registry/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,11 @@
and checked by `tools/validate_limit_registry.py`) lists the values a
`resource_exhausted` problem body carries in `limit_name`. `LIMIT_NAMES` is
that registry, keyed by name.

The warning-reason registry (`registries/warning-reasons.json`, mirrored the
same way and checked by `tools/validate_warning_reasons.py`) lists the values
a warning carries in `params.why`. `WARNING_REASONS` is that registry, keyed
by code and then by reason.
"""

from __future__ import annotations
Expand Down Expand Up @@ -169,15 +174,82 @@ def load_limit_names() -> Mapping[str, LimitName]:

LIMIT_NAMES: Mapping[str, LimitName] = load_limit_names()


@dataclass(frozen=True, slots=True)
class WarningReason:
"""One reason a warning of `code` can name in `params.why`.

`params` lists each param the warning carries beside `why` for this
reason as a (name, definition) pair, sorted by name, and is empty when
the reason adds none.
"""

code: str
reason: str
definition: str
params: tuple[tuple[str, str], ...] = ()


_WARNING_REASONS_FORMAT_VERSION = 1


def load_warning_reasons() -> Mapping[str, Mapping[str, WarningReason]]:
"""Load and parse the packaged warning-reason mirror
(`registry/warning-reasons.json`), keyed by code and then by reason.

Raises:
ValueError: The file's `format_version` is not the one this loader
reads, or it lists one reason twice for a code.
"""
raw_text = (
resources.files("dfo_spec_kit.registry")
.joinpath("warning-reasons.json")
.read_text(encoding="utf-8")
)
return _warning_reasons_from_json(json.loads(raw_text))


def _warning_reasons_from_json(
data: dict[str, Any],
) -> Mapping[str, Mapping[str, WarningReason]]:
if data["format_version"] != _WARNING_REASONS_FORMAT_VERSION:
raise ValueError(
f"warning-reasons.json has format_version {data['format_version']!r}; "
f"this loader reads {_WARNING_REASONS_FORMAT_VERSION}"
)
by_code: dict[str, dict[str, WarningReason]] = {}
for raw in data["entries"]:
entry = WarningReason(
code=raw["code"],
reason=raw["reason"],
definition=raw["definition"],
params=tuple(sorted(raw.get("params", {}).items())),
)
reasons = by_code.setdefault(entry.code, {})
if entry.reason in reasons:
raise ValueError(
f"warning-reasons.json lists {entry.code}/{entry.reason} twice"
)
reasons[entry.reason] = entry
return MappingProxyType(
{code: MappingProxyType(reasons) for code, reasons in by_code.items()}
)


WARNING_REASONS: Mapping[str, Mapping[str, WarningReason]] = load_warning_reasons()

__all__ = [
"LIMIT_NAMES",
"REGISTRY",
"WARNING_REASONS",
"Classification",
"CodeEntry",
"CodeMapping",
"CodeRegistry",
"Lifecycle",
"LimitName",
"WarningReason",
"load_limit_names",
"load_registry",
"load_warning_reasons",
]
20 changes: 20 additions & 0 deletions python/spec-kit/src/dfo_spec_kit/registry/codes.json
Original file line number Diff line number Diff line change
Expand Up @@ -194,6 +194,26 @@
],
"lifecycle": "retired"
},
{
"code": "bilateral_reading_unavailable",
"classification": "warning",
"definition": "An answer or reading of a published total ODA table, on a part with no provider axis and no provider entry asking for total ODA, reads total ODA, which includes donors' core contributions to multilateral agencies, because its bilateral reading is unavailable or undecided for it. The warning carries the spec_id of the reading it concerns, and params.why names the reason, one of the reasons spec-kit's warning-reason registry lists for this code, which may add params of its own.",
"owning_contract": "search",
"owner": "platform",
"introduced_in": "v1",
"mappings": [
{
"endpoint": "search.resolve",
"doors": [
"raw_text",
"parsed",
"dataset_resubmit"
],
"channel": "diagnostics_warning"
}
],
"lifecycle": "active"
},
{
"code": "binding_folded",
"classification": "warning",
Expand Down
33 changes: 33 additions & 0 deletions python/spec-kit/src/dfo_spec_kit/registry/warning-reasons.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
{
"format_version": 1,
"entries": [
{
"code": "bilateral_reading_unavailable",
"reason": "destination_unclassified",
"definition": "The data holds a destination that no side declaration places, so the bilateral destinations cannot be told apart from the rest."
},
{
"code": "bilateral_reading_unavailable",
"reason": "option_not_covered",
"definition": "The bilateral reading lacks a year the answer shows."
},
{
"code": "bilateral_reading_unavailable",
"reason": "read_window_not_covered",
"definition": "The bilateral reading holds every year the answer shows but lacks a year the request reads beyond them."
},
{
"code": "bilateral_reading_unavailable",
"reason": "scope_undecided",
"definition": "Search could not decide which reading the answer uses, so the answer reads total ODA."
},
{
"code": "bilateral_reading_unavailable",
"reason": "slot_not_served",
"definition": "The bilateral reading does not serve a slot the answer binds.",
"params": {
"slot": "The property of the slot the bilateral reading does not serve."
}
}
]
}
48 changes: 46 additions & 2 deletions python/spec-kit/tests/test_registry.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,15 +9,26 @@
from __future__ import annotations

from pathlib import Path
from typing import Any

import pytest

from dfo_spec_kit.registry import (
LIMIT_NAMES,
REGISTRY,
WARNING_REASONS,
CodeEntry,
CodeMapping,
_warning_reasons_from_json,
load_registry,
)

_REASON = {
"code": "bilateral_reading_unavailable",
"reason": "scope_undecided",
"definition": "A reason.",
}

# Repo root is four levels up from this file (tests/ -> spec-kit -> python ->
# repo root); the canonical registry is always present in-repo.
_REPO_ROOT = Path(__file__).resolve().parents[3]
Expand Down Expand Up @@ -124,8 +135,8 @@ def test_entries_are_sorted_by_code() -> None:
assert codes == sorted(codes)


def test_limit_names_mirror_is_byte_identical_to_canonical_registry() -> None:
name = "limit-names.json"
@pytest.mark.parametrize("name", ["limit-names.json", "warning-reasons.json"])
def test_mirror_is_byte_identical_to_canonical_registry_file(name: str) -> None:
assert (_MIRROR_PATH.parent / name).read_bytes() == (
_REPO_ROOT / "registries" / name
).read_bytes()
Expand All @@ -136,3 +147,36 @@ def test_limit_names_are_keyed_by_name() -> None:
assert entry.name == "MAX_TABLE_ROWS"
assert entry.owning_contract == "presentation"
assert all(key == value.name for key, value in LIMIT_NAMES.items())


def test_warning_reasons_are_keyed_by_code_then_reason() -> None:
entry = WARNING_REASONS["bilateral_reading_unavailable"]["slot_not_served"]
assert (entry.code, entry.reason) == (
"bilateral_reading_unavailable",
"slot_not_served",
)
assert [name for name, _ in entry.params] == ["slot"]
assert all(
code == value.code and reason == value.reason
for code, reasons in WARNING_REASONS.items()
for reason, value in reasons.items()
)


def test_warning_reasons_are_hashable() -> None:
entry = WARNING_REASONS["bilateral_reading_unavailable"]["slot_not_served"]
assert hash(entry) == hash(entry)


@pytest.mark.parametrize(
"data",
[
pytest.param({"format_version": 2, "entries": []}, id="format-version"),
pytest.param(
{"format_version": 1, "entries": [_REASON, _REASON]}, id="duplicate"
),
],
)
def test_warning_reasons_loader_refuses(data: dict[str, Any]) -> None:
with pytest.raises(ValueError):
_warning_reasons_from_json(data)
2 changes: 2 additions & 0 deletions registries/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,4 +6,6 @@ This file is the canonical copy. `python/spec-kit/src/dfo_spec_kit/registry/code

`limit-names.json` lists the values a `resource_exhausted` problem body carries in `limit_name`, each the name of the limit the request crossed, sorted by name. An entry holds `name`, a one-sentence `definition` and `owning_contract`. `schemas/registry/limit-name-registry.schema.json` validates the file, and `uv run tools/validate_limit_registry.py` also checks the sort order, uniqueness and the package mirror `python/spec-kit/src/dfo_spec_kit/registry/limit-names.json`, read as `dfo_spec_kit.registry.LIMIT_NAMES`; edit this file and copy it there. A name keeps its meaning once published, and a new meaning takes a new name.

`warning-reasons.json` lists the values a warning carries in `params.why`, each the reason the warning of one code fired, sorted by `code` then `reason`. An entry holds `code` (an active warning in `codes.json`), `reason` (lower snake case, unique within its code), a one-sentence `definition` and, when the reason makes the warning carry more params beside `why`, `params`, each param name mapped to a one-sentence definition. `schemas/registry/warning-reason-registry.schema.json` validates the file, and `uv run tools/validate_warning_reasons.py` also checks the sort order, uniqueness, that each code is an active warning and the package mirror `python/spec-kit/src/dfo_spec_kit/registry/warning-reasons.json`, read as `dfo_spec_kit.registry.WARNING_REASONS`; edit this file and copy it there. A reason keeps its meaning once published, and a new meaning takes a new reason.

`units.json` declares how unit refs read as typed units, for `dfo_spec_kit.units`. `currency_prefixes` lists the prefixes (`""` and `Currency`) that, followed by a currency code, make a ref money in that currency with prices unstated. After `Currency` a code is any three upper-case ASCII letters. After the empty prefix the code must be one of `currency_codes`, the ISO 4217 active codes without `XTS` (testing) and `XXX` (no currency), plus the withdrawn `ANG`, `BGN` and `ZWL`, which Data Commons still types as `Currency` for older series, so a bare `TEU` or `PPM` reads as other. `units` lists every other typed ref, sorted by `ref`. Money rows carry `currency`, `prices` (`current`, `constant`, or `null` when the ref leaves them unstated) and `evidence`; a row with stated prices also carries `base_year` (`null` when unstated, always `null` for current prices) and `chained` (`true` for a chain-linked series, which only constant prices can be). Percent and count rows carry `ref`, `kind` and `evidence` only. Rate rows state a value per a count of something: `count_base` is that count (`1000` for "per 1,000 people"), and `counted` is what it counts as a plural noun phrase (`people`, `live births`), `null` when the ref states none. The graph holds neither as a field, so each rate row declares them from the ref and its label. `evidence` says where the reading comes from. A ref that scales its values (`SDG_CU_USD_M`, `SDG_NUM_TH`, `PeopleInThousands`) has no row and reads as other, as does any ref that matches no prefix and no row. No row may also match a prefix rule. `schemas/registry/unit-registry.schema.json` validates the file, and `uv run tools/validate_unit_registry.py` also checks the sort order, uniqueness, the prefix rule and the package mirror `python/spec-kit/src/dfo_spec_kit/units/units.json`; edit this file and copy it there.
20 changes: 20 additions & 0 deletions registries/codes.json
Original file line number Diff line number Diff line change
Expand Up @@ -194,6 +194,26 @@
],
"lifecycle": "retired"
},
{
"code": "bilateral_reading_unavailable",
"classification": "warning",
"definition": "An answer or reading of a published total ODA table, on a part with no provider axis and no provider entry asking for total ODA, reads total ODA, which includes donors' core contributions to multilateral agencies, because its bilateral reading is unavailable or undecided for it. The warning carries the spec_id of the reading it concerns, and params.why names the reason, one of the reasons spec-kit's warning-reason registry lists for this code, which may add params of its own.",
"owning_contract": "search",
"owner": "platform",
"introduced_in": "v1",
"mappings": [
{
"endpoint": "search.resolve",
"doors": [
"raw_text",
"parsed",
"dataset_resubmit"
],
"channel": "diagnostics_warning"
}
],
"lifecycle": "active"
},
{
"code": "binding_folded",
"classification": "warning",
Expand Down
33 changes: 33 additions & 0 deletions registries/warning-reasons.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
{
"format_version": 1,
"entries": [
{
"code": "bilateral_reading_unavailable",
"reason": "destination_unclassified",
"definition": "The data holds a destination that no side declaration places, so the bilateral destinations cannot be told apart from the rest."
},
{
"code": "bilateral_reading_unavailable",
"reason": "option_not_covered",
"definition": "The bilateral reading lacks a year the answer shows."
},
{
"code": "bilateral_reading_unavailable",
"reason": "read_window_not_covered",
"definition": "The bilateral reading holds every year the answer shows but lacks a year the request reads beyond them."
},
{
"code": "bilateral_reading_unavailable",
"reason": "scope_undecided",
"definition": "Search could not decide which reading the answer uses, so the answer reads total ODA."
},
{
"code": "bilateral_reading_unavailable",
"reason": "slot_not_served",
"definition": "The bilateral reading does not serve a slot the answer binds.",
"params": {
"slot": "The property of the slot the bilateral reading does not serve."
}
}
]
}
4 changes: 4 additions & 0 deletions schemas/registry/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,3 +112,7 @@ The registry has no severity or message template. Each occurrence supplies them
## Limit-name registry

`limit-name-registry.schema.json` (draft 2020-12) validates `registries/limit-names.json`, the names a `resource_exhausted` problem body carries in `limit_name`. Clients match on these values. The file is `{"format_version": 1, "entries": [...]}` with entries sorted by `name`, and each entry is closed to `name` (upper snake case, unique), `definition` (one sentence) and `owning_contract` (spelled as in the code registry). `uv run tools/validate_limit_registry.py` checks the schema, the sort order, uniqueness and the package mirror.

## Warning-reason registry

`warning-reason-registry.schema.json` (draft 2020-12) validates `registries/warning-reasons.json`, the values a warning carries in `params.why`. Clients match on these values. The file is `{"format_version": 1, "entries": [...]}` with entries sorted by `code` then `reason`, and each entry is closed to `code` (an active warning registered in `registries/codes.json`), `reason` (lower snake case, unique within its code), `definition` (one sentence) and an optional `params` object mapping each param the warning carries beside `why` for this reason to a one-sentence definition. `params` names no `why` and holds at least one param when present. `uv run tools/validate_warning_reasons.py` checks the schema, the sort order, uniqueness, that each code is an active warning and the package mirror.
Loading
Loading