Skip to content

release: apis 0a051f1 to production - #318

Merged
jm-rivera merged 4 commits into
mainfrom
release/2026-11-16
Nov 16, 2026
Merged

jm-rivera merged 4 commits into
mainfrom
release/2026-11-16

Conversation

@jm-rivera

Copy link
Copy Markdown
Contributor

Releases apis at 0a051f1 to production, cut on release/2026-11-16: four pull requests after the release at c9f3a0c.

  • fix(search,datasets): read DAC2a bilateral composites within the bilateral destination population #310 reads the DAC2a bilateral and inclusive composites within the bilateral destination population: countries, the organisations the DAC counts as bilateral recipients and the regional and bilateral unallocated lines, with no multilateral agency and no total of both sides. Search elects the provider reading before it reads, reads a population once and mints the Spec as the composite. France 2023 ODA by recipient reads 9,317.24m bilateral over 149 recipients and 15,412.03m inclusive over 163. A bilateral answer's provenance label reads "OECD DAC2a (bilateral flows)", and an answer that reads total ODA where the bilateral reading is unavailable warns bilateral_reading_unavailable with params.why naming one of five reasons. It pins spec-kit at 67a0fed, which registers that warning and its reasons.
  • fix(code-index): place DAC2a destinations by graph type #317 places each DAC2a destination on a side from its graph types, kept on every code index party row, in place of the country/ prefix and the hand-kept lists. Africa Finance Corporation reads as a multilateral donor. The release format moves to 3.
  • feat(graph): serve the graph's schema, its use by each dataset and a StatVar definition check #301 adds the graph schema routes: GET /api/graph/schema, /schema/terms, /schema/usage and POST /api/graph/schema/check.
  • refactor(api): serve the JSON Schema documents at /json-schema #307 moves the JSON Schema documents to GET /api/{graph,search,datasets}/json-schema. The old /api/search/schema and /api/datasets/schema paths answer 404. The SDK at main names them only in generated types and comments and calls neither.

Release order: release this before the website's graph explorer reaches main, since it reads GET /api/graph/schema. No migration, setting or secret change; GRAPH_SCHEMA_CHECK_TIMEOUT_SECONDS is new with a default of 30.

Format 3: the deploy's graph release verify ends no_pointer on the first production deploy, and the build publishes the first format-3 release before traffic moves. On dev that build took 17 minutes (verify at 09:12, release published 09:29). A hold on the production format-2 pointer does not apply to format 3. Data caches miss once, since the code index part changes.

Testing: CI, the eval gate and Deploy dev passed on 0a051f1. On dev, France 2023 ODA by recipient answered 9,317.24m bilateral over 149 recipients and 15,412.03m inclusive over 163 with no mint warning, and no destination_population_unavailable, graph_release_refused or IndexFormatError line was logged after the format-3 release opened.

GET /api/graph/json-schema, /api/search/json-schema and
/api/datasets/json-schema serve the JSON Schema of every request and
response body. The graph capabilities document lists the door as
json_schema, and the eval gate's readiness probe reads
/api/search/capabilities, which both trees serve.
…StatVar definition check (#301)

* feat(graph): serve the instance's ontology, its use by each provenance, and a StatVar definition check

Four routes on a new `ontology` door of the Graph surface.

`GET /api/graph/ontology` serves every Class and Property the instance
declares, the base Data Commons terms they and the code index's StatVar
rows name, their ancestors, and the members of each enumeration. A term
is the instance's own when a provenance outside `dc/base/` declares its
`typeOf` arc. `GET /api/graph/ontology/terms` lists every Class and
Property on the graph by dcid, name and declaring provenance. One build
of about 28 graph requests yields both, held in a single-flight cache
per graph generation and code index.

`GET /api/graph/ontology/provenances` serves, per provenance, the
patterns its StatVars follow, the constraint values it uses and its
facet facts, from the code and coverage indexes with no graph read.

`POST /api/graph/ontology/check` takes up to 2,000 candidate StatVar
definitions and returns one finding per fault: a node the graph does
not hold, a provisional or untyped value, a missing value, a wrong
kind, a value outside an instance property's range, a property outside
its domain, or a served StatVar redefined. The value test is the one
`code_index.export` refuses a release on, shared through
`code_index.constraint_values`.

The graph client returns the provenances that declared each arc:
`ArcTarget.provenances`, and `node_arc_subjects` and
`node_arc_subjects_many` for reverse arcs, which `OriginGraphClient`
routes as it routes `node_subjects`.

* fix(graph): refuse an ontology build whose provenances cannot be read

A node's `declared_by` comes from the provenances that declared its
`typeOf` arc. A node whose provenances read empty is read alone once,
and the build refuses with a retryable 503 when they stay empty, as it
does for a listing subject with no provenance.

Enumeration members read after the closure bring in their other types
and those types' ancestors. The build drops the unresolved-node
sentinel from every member and does not follow it. A term's name is
`display_name` over every name its listing entries carry
(`ArcSubject.names`), so both routes name a node the same way on every
build.

`generation` is null on the wire when the stamp has no token. The check
compares served definitions with the code index its document was built
from. `node_subjects` and `node_subjects_many` hold only dcids across
their walk; the arc-subject reads fold names, types and provenances as
entries arrive.

* fix(graph): bound the graph reads of one ontology check

A check request reads at most 20 distinct ids from the graph and
answers 400 `request_invalid`, before any read, when it needs more; an
absent id costs about 0.28 s on the dev graph, so 20 fit the 10 s
deadline. A row carries at most 32 constraint properties and 32
observation properties. The capabilities document publishes the three
limits.

Two kinds of value need no read. A value outside the ontology document,
on an instance property whose every range class is an instance
enumeration the document lists in full, is `value_outside_range`. A
number, a boolean, or a string that is not dcid syntax is
`node_not_held`.

An id the check does read is read with all its arcs, so a node the
graph holds with no `typeOf` is `node_untyped` whether or not the
document holds it. The lookup the check reads the document through is
built once per cached document.

* fix(graph): serve an untyped node as a fact and confirm empty provenance reads in the client

A node the graph holds with no `typeOf` arc has no declaring
provenance: the ontology document serves it with empty `provenances`
and a null `declared_by`. Only a typed node whose entries carry no
provenance refuses the build, with a non-retryable 503 that logs the
dcid.

`node_arc_targets_many` takes `confirm_empty`: an empty answer is read
again as a group and then alone, by the rule `node_arcs_many` already
applies, through one shared helper. It is off by default, and the
ontology build is its one caller.

A node's provenances are the union of every listing that names it and
its own `typeOf` read, so `/ontology` and `/ontology/terms` agree on
`declared_by`. With its backend off, an arc-subject read answers as
truncated. The provenances body is built in a worker thread. The
StatVar shape fields come from `FiveTuple`.

* fix(graph): read each unknown id of an ontology check once, within a cap one row always fits

The check reads no ancestry. `value_outside_range` fires when every
type of a value is a node of the ontology document and none reaches the
range; `property_outside_domain` fires when the population type is a
document node. The document is closed under `subClassOf`, so both rules
judge from it alone.

A request reads at most 41 ids, the most one row can name: five shape
fields, 16 constraint properties with their values, and four
observation properties. A value is read only when its property is a
held Property. Each read round is one `node_arcs_batch` call with
`confirm_empty`, an opt-in that reads an empty answer again in a group
and then alone, through the helper `node_arcs_many` uses. 41 absent ids
answer in 5 to 9 s on the dev graph, inside the route's own 30 s
deadline, `GRAPH_ONTOLOGY_CHECK_TIMEOUT_SECONDS`.

A value outside the document is `value_outside_range` without a read
only when every range class, and every enumeration below it in the
document, is an instance enumeration listed in full. `ref` takes the
dcid length ceiling, and a non-finite number is refused. Capabilities
publishes the row cap.

* fix(graph): read each ontology node once and say which member lists are complete

build_ontology reads a node's values and provenances in one
node_arc_targets_many call per level with confirm_empty, so the two
cannot disagree. The second provenance read and its refusal are gone.

OntologyEnumeration.complete replaces truncated. It is true only when
the list is every member the graph holds, and false for a base
enumeration, whose list is the members the code index names.
OntologyLookup.of reads it to decide which ranges need no graph read.

A build refusal that cannot change within a generation is kept as
RefusedOntology until the graph changes, so each later request answers
without a rebuild. The cache entry holds no code index: its key carries
the index hash, and the check uses the request's own index.

The door serialises in the models' field order.

* fix(graph): bound an ontology check before any work and run checks on their own workers

The check reads each id outside the document once through
node_arcs_many, asking only typeOf and name, with every empty answer
confirmed. A node with neither reads as absent. node_arcs_batch is as
on dev, and the GraphNodeReader Protocol no longer lists it.

plan_reads decides the request on the event loop as soon as the
document is in hand: more than CHECK_MAX_READ_IDS ids to read, or an
id of the UN origin while the catalogue does not serve it, is 400
request_invalid before anything is submitted. A check stops at
CHECK_MAX_FINDINGS findings with the same 400. A numeric constraint
value lies within plus or minus 2**53.

Checks run on a two-worker DcpReadExecutor of their own, which also
encodes the response. A slot frees when its job returns, a third check
answers 503 with Retry-After, and a check never takes one of the
shared DCP-read workers.

The handler is post_ontology_check.

* fix(graph): read each check id once, confirm empty member lists and serve a node with no declaring provenance

_Nodes.read skips ids already read, so a check asks each id once
across its two rounds. stat_type, measurement_qualifier and
measurement_denominator get the held-node test the constraint values
get.

build_ontology lists again, alone, each instance enumeration whose
batched listing came back empty. A node or term whose typeOf carries
no provenance is served with declared_by null and named in one warning
per build, and OntologyProvenanceUnreadError is removed. Base
enumerations find their members through one grouping of the
index-named nodes.

The provenances document builds once per code index and coverage build
through a single-flight cache.

The docs state the POST bound per backend, the largest response and
one measured time for a check of 41 absent ids.

* docs(graph): state the ontology door's docstrings and docs in plain sentences

Docstrings and comments of the ontology door and its client reads
drop joined sentences and contrast phrasing. The README, guide,
reference and deploy notes split sixteen sentences the same way.

* fix(graph): read an ontology check in one call and refuse a document with no instance declaration

check_rows reads every planned id in one node_arcs_many call, and
plan_reads returns that set. A value under a property the graph does
not hold is read and gets no finding.

A node's provenances are the ones its typeOf targets carry, and
declared_by is null only when none carries one. A document that holds
no node the instance declares is refused with
OntologyNoInstanceDeclarationError, a 503 kept for the generation.

The provenances document waits through aget_or_fetch_many, so a caller
whose wait ends answers 503 with Retry-After. The ontology cache holds
two entries, so a request under an unknown generation leaves the known
entry in place. parse_arc_targets interns provenance ids.

GraphDeps requires its ontology caches and check executor, and
SHAPE_FIELDS is a literal tuple.

* docs(graph): name the routes that answer the kept refusal and when the provenances body expires

The README names the ontology, terms and check routes for the kept 503
and says the provenances body also expires at the generation stamp's
maximum age. docs/reference.md keeps its table aligned.

* fix(graph): keep an ontology build refusal only under a known generation

Under an unknown generation nothing can clear a stored refusal before
the stamp's age runs out, so _build_entry raises it without storing it
and the door answers it retryable. Under a known generation the
refusal is kept and answers non-retryable, as before.

classify_ontology_exception maps a failed read through
graph_read_problem, as the members door does. ArcSubject no longer
carries types, which no caller read.

* docs(graph): say when an ontology build refusal is retryable

The README error table states both cases: kept and not retryable under
a known generation, rebuilt and retryable under an unknown one.

* fix(graph): read every value outside the document and bound the check by its deadline

- The check reads a constraint value the document does not hold
  instead of judging it outside a fully listed range, so a member of a
  subclass the document lacks is no longer reported as
  value_outside_range.
- post_ontology_check answers 504 when its deadline passes, while the
  worker stays held until its job returns.
- constraint_value_fault drops unresolved typeOf entries itself, so
  the export and the check give one verdict.
- The provenances route refreshes a stale generation before keying its
  cache, and one factory builds both caches.
- Term listings judge progress by what a page advanced, the build
  bypasses the shared arc-targets cache, and one probe policy,
  individual_probe_budget, serves node_arc_targets_many and
  node_arcs_many with existing callers' requests unchanged.
- node_arc_subjects with the UN origin off reports a complete listing,
  and the test fake answers a held node as the served client does.

* feat(graph): answer the check rows that fit and report row faults as findings

- POST /api/graph/ontology/check checks rows from the first while the
  ids to read stay within its read budget and the findings within
  their cap, and answers 200 with row_count and next_row (null when
  every row was checked).
- A fault in a row's content is a finding: a cell outside dcid syntax
  gets node_not_held, a missing measured property value_missing, and
  an id of an origin the deployment does not serve origin_not_served.
  A 400 means the envelope or a size bound, with a pointer naming the
  row.
- A finding carries row, ref, field, property, subject, value, served,
  code and severity.
- Capabilities publish the check caps a caller can count before
  sending.
- GET /api/graph/ontology/usage serves how each Provenance's StatVars
  use the model.

* test(graph): pin null finding members and next_row in the encoded check answer

* refactor(api): serve the JSON Schema documents at /json-schema

GET /api/graph/json-schema, /api/search/json-schema and
/api/datasets/json-schema serve the JSON Schema of every request and
response body. The graph capabilities document lists the door as
json_schema, and the eval gate's readiness probe reads
/api/search/capabilities, which both trees serve.

* refactor(graph): serve the instance's Data Commons schema at /api/graph/schema

GET /api/graph/schema, /schema/terms and /schema/usage and
POST /api/graph/schema/check serve the type layer the graph holds:
classes, properties, their arcs and enumeration members. Modules,
models, operation ids, cache names and the GRAPH_SCHEMA_* and
SCHEMA_FLIGHT_TIMEOUT_SECONDS settings take the same name.

* feat(graph): serve every property a schema node carries and bound the check answer

- Each node of GET /api/graph/schema carries other_properties: every
  outgoing property beyond its ten named fields, as node references or
  string literals. Literals over 2,000 characters and values past 100
  per property are left out and logged. The build reads all arcs
  through node_all_arc_targets_many, which confirms empty answers like
  the named-property reads.
- POST /api/graph/schema/check also stops before the row that would
  take the encoded answer past 4,000,000 bytes, and always checks the
  first row.

* feat(graph): name declaring origins, say why a refusal holds, and trim the check answer

- declared_by is dfo, base or un_system, from the origin catalogue's
  provenance table where it lists the provenance, else from the
  dc/base/ and undata/ prefixes. Range and domain rules apply to
  dfo-declared properties.
- A term listing past its cap and a build with no dfo-declared node
  answer 500. A 503 carries cause.code schema_building or check_busy.
- The check answers {generation, rows_checked, findings}. Rows and
  findings carry no ref, and not_a_node marks a literal or a string
  outside dcid syntax where a node belongs, so node_not_held means a
  well-formed dcid the graph does not hold.
- Schema nodes, terms and usage entries carry ref and label, as the
  node route does, and other_properties names a node by ref.

* perf(graph): pipeline the schema build, probe empties concurrently and warm at start-up

- build_schema starts each read once the reads it depends on land:
  index-named nodes during the term listings, each chunk expanded as
  it arrives, every DFO Class listed on its own, enumeration members
  once a chain reaches Enumeration. At most 24 reads are in flight,
  the bytes are those of the sequential build, and each phase's time
  is logged with the encode step.
- An empty answer under individual_probe_budget=None is read again
  alone on one process-wide pool of 16 probes, with no group re-read.
  An id counts as absent only after its own read answers empty, and a
  read error propagates.
- The check's read budget is 200 ids, a busy check answers
  Retry-After: 1, and the schema routes wait up to 3 s for the
  generation read.
- A live Graph lifespan builds the schema document before taking
  traffic, bounded at 20 s; a failed or slow warm build is logged and
  the first request builds.

* feat(graph): list instances for DFO value classes and judge ranges by read ancestry

- A Class node carries instances {refs, complete}. DFO classes that
  descend from Enumeration or that a DFO property's rangeIncludes
  names list every instance of themselves and their DFO subclasses,
  up to INSTANCE_LISTING_CAP. A base enumeration lists the instances
  StatVar rows use, never complete. Other base classes carry null.
- The check reads the subClassOf chain of a value's type when the
  type lies outside the document, inside the read budget, and judges
  the range against it. A chain it cannot read in full gives no
  verdict.

* fix(graph): count a checked row only with every read it needs, and stop reads at their deadline

- A row counts in rows_checked only when its named ids and its types'
  ancestry were all read, so each checked row gets the findings it
  gets alone. Row 0 may read 41 named ids and 40 ancestors; later
  rows yield to earlier ones, and 40 ids are held back once a row may
  need ancestry.
- An unresolved subClassOf target leaves a chain incomplete, which
  gives no range verdict.
- A failed schema build raises at once and abandons the reads still
  in flight.
- Single-id probes honour the caller's deadline in the queue and in
  flight, and raise the deadline error on expiry.

* fix(graph): name each dataset in schema usage from its provenance node

build_usage takes a provenance's label and description from its graph
node, read once per usage build with the same deadline and cache key,
so a dataset is named whether or not the code index declares it. A
failed read answers the schema door's read-failure problem and is not
kept.

* fix(graph): judge ranges by the build's closed subclass sets and mark cells the check cannot judge

The schema check reads each id a row names once, for typeOf and name, and makes no ancestry reads. A constraint value is in range when one of its types reaches a rangeIncludes class through the document; value_outside_range needs every type known and every range class closed, meaning the build listed its whole <-subClassOf tree and the document holds it. Every other cell gets the new warning range_not_judged, so a check never reports a clean result over a cell it could not decide. property_outside_domain needs a complete ancestry over the document. A row's findings no longer depend on the other rows in the request.

The build lists the descendant tree of every range class in batched node_arc_subjects_many requests. A base range within RANGE_CLOSURE_CAP (500) classes joins the served document; on dev that adds Organization's subclasses and leaves Thing out, for 1,482 nodes and a cold build of about 3.8 s. A DFO value list is complete only once every subclass under it was listed. The build records unresolved typeOf, subClassOf, rangeIncludes and domainIncludes targets and confirmed-absent ids for the check, and leaves the served bytes unchanged.

parse_arc_targets keeps a node reference and a literal with the same text as two targets, and marks unresolved entries only for node_all_arc_targets_many. A kept schema refusal expires after 60 s through a new value_ttl on SingleFlightCache.

* fix(graph): treat a held UN-origin type as unknown and warn on a wholly unresolved range

The build's subclass listings run on our mixer, so they cannot place a UN-origin class outside a range. A UN-origin type is unknown to the range rule wherever the document holds it, after a type whose ancestry meets the range has cleared the value. A DFO Property whose rangeIncludes holds only unresolved targets counts as ranged, so its cells get range_not_judged.
…teral destination population (#310)

DAC2a total ODA across recipients reads within the bilateral destination population, so a donor's recipients exclude multilateral agencies and the inclusive composite no longer counts core contributions twice. Search elects the provider reading before the read, reads a population part once, and mints the Spec as the composite on every door. bilateral_reading_unavailable carries the reasons spec-kit's warning-reason registry lists.

Closes #311
* feat(code-index): keep each party's graph types on its row

The code index export writes the sorted graph types of every party, and the loader refuses a party row without them, so a release built before this change fails verification and is rebuilt before a deploy moves traffic.

* feat(code-index): place DAC2a destinations by graph type

destination_side reads each party's graph types: Country is bilateral, an official Organization multilateral, DACUnspecifiedParty residuals multilateral unless a bilateral total lists them, and DACRecipientUnallocatedByCountry residuals bilateral. Every destination total other than all recipients, all multilateral recipients and Organization-typed totals is bilateral. party_facts reads the Organization type instead of the entity resolver's organisation records. The hand-kept group totals, multilateral residuals and bilateral unallocated lists go.

* fix(code-index): bump the release format and memoise residual sides

A code index whose party rows carry no types is unreadable by this image, so the release format moves to 3 and each image reads its own pointer. The residuals bilateral totals list are memoised per index, the loader refuses a malformed types value, and prose that named the removed lists now names destination_side.

* docs(code-index): tighten the side rule's docstrings and README
@jm-rivera
jm-rivera merged commit e302dc9 into main Nov 16, 2026
33 of 34 checks passed
@jm-rivera
jm-rivera deleted the release/2026-11-16 branch November 16, 2026 11:22

This branch was successfully deployed

1 active deployment
dev — 0a051f15 Deployed Nov 16, 2026 by jm-rivera via deploy / deploy #192
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant