Repository navigation
release: apis 0a051f1 to production - #318
Merged
Merged
Conversation
GET /api/graph/json-schema, /api/search/json-schema and /api/datasets/json-schema serve the JSON Schema of every request and response body. The graph capabilities document lists the door as json_schema, and the eval gate's readiness probe reads /api/search/capabilities, which both trees serve.
…StatVar definition check (#301) * feat(graph): serve the instance's ontology, its use by each provenance, and a StatVar definition check Four routes on a new `ontology` door of the Graph surface. `GET /api/graph/ontology` serves every Class and Property the instance declares, the base Data Commons terms they and the code index's StatVar rows name, their ancestors, and the members of each enumeration. A term is the instance's own when a provenance outside `dc/base/` declares its `typeOf` arc. `GET /api/graph/ontology/terms` lists every Class and Property on the graph by dcid, name and declaring provenance. One build of about 28 graph requests yields both, held in a single-flight cache per graph generation and code index. `GET /api/graph/ontology/provenances` serves, per provenance, the patterns its StatVars follow, the constraint values it uses and its facet facts, from the code and coverage indexes with no graph read. `POST /api/graph/ontology/check` takes up to 2,000 candidate StatVar definitions and returns one finding per fault: a node the graph does not hold, a provisional or untyped value, a missing value, a wrong kind, a value outside an instance property's range, a property outside its domain, or a served StatVar redefined. The value test is the one `code_index.export` refuses a release on, shared through `code_index.constraint_values`. The graph client returns the provenances that declared each arc: `ArcTarget.provenances`, and `node_arc_subjects` and `node_arc_subjects_many` for reverse arcs, which `OriginGraphClient` routes as it routes `node_subjects`. * fix(graph): refuse an ontology build whose provenances cannot be read A node's `declared_by` comes from the provenances that declared its `typeOf` arc. A node whose provenances read empty is read alone once, and the build refuses with a retryable 503 when they stay empty, as it does for a listing subject with no provenance. Enumeration members read after the closure bring in their other types and those types' ancestors. The build drops the unresolved-node sentinel from every member and does not follow it. A term's name is `display_name` over every name its listing entries carry (`ArcSubject.names`), so both routes name a node the same way on every build. `generation` is null on the wire when the stamp has no token. The check compares served definitions with the code index its document was built from. `node_subjects` and `node_subjects_many` hold only dcids across their walk; the arc-subject reads fold names, types and provenances as entries arrive. * fix(graph): bound the graph reads of one ontology check A check request reads at most 20 distinct ids from the graph and answers 400 `request_invalid`, before any read, when it needs more; an absent id costs about 0.28 s on the dev graph, so 20 fit the 10 s deadline. A row carries at most 32 constraint properties and 32 observation properties. The capabilities document publishes the three limits. Two kinds of value need no read. A value outside the ontology document, on an instance property whose every range class is an instance enumeration the document lists in full, is `value_outside_range`. A number, a boolean, or a string that is not dcid syntax is `node_not_held`. An id the check does read is read with all its arcs, so a node the graph holds with no `typeOf` is `node_untyped` whether or not the document holds it. The lookup the check reads the document through is built once per cached document. * fix(graph): serve an untyped node as a fact and confirm empty provenance reads in the client A node the graph holds with no `typeOf` arc has no declaring provenance: the ontology document serves it with empty `provenances` and a null `declared_by`. Only a typed node whose entries carry no provenance refuses the build, with a non-retryable 503 that logs the dcid. `node_arc_targets_many` takes `confirm_empty`: an empty answer is read again as a group and then alone, by the rule `node_arcs_many` already applies, through one shared helper. It is off by default, and the ontology build is its one caller. A node's provenances are the union of every listing that names it and its own `typeOf` read, so `/ontology` and `/ontology/terms` agree on `declared_by`. With its backend off, an arc-subject read answers as truncated. The provenances body is built in a worker thread. The StatVar shape fields come from `FiveTuple`. * fix(graph): read each unknown id of an ontology check once, within a cap one row always fits The check reads no ancestry. `value_outside_range` fires when every type of a value is a node of the ontology document and none reaches the range; `property_outside_domain` fires when the population type is a document node. The document is closed under `subClassOf`, so both rules judge from it alone. A request reads at most 41 ids, the most one row can name: five shape fields, 16 constraint properties with their values, and four observation properties. A value is read only when its property is a held Property. Each read round is one `node_arcs_batch` call with `confirm_empty`, an opt-in that reads an empty answer again in a group and then alone, through the helper `node_arcs_many` uses. 41 absent ids answer in 5 to 9 s on the dev graph, inside the route's own 30 s deadline, `GRAPH_ONTOLOGY_CHECK_TIMEOUT_SECONDS`. A value outside the document is `value_outside_range` without a read only when every range class, and every enumeration below it in the document, is an instance enumeration listed in full. `ref` takes the dcid length ceiling, and a non-finite number is refused. Capabilities publishes the row cap. * fix(graph): read each ontology node once and say which member lists are complete build_ontology reads a node's values and provenances in one node_arc_targets_many call per level with confirm_empty, so the two cannot disagree. The second provenance read and its refusal are gone. OntologyEnumeration.complete replaces truncated. It is true only when the list is every member the graph holds, and false for a base enumeration, whose list is the members the code index names. OntologyLookup.of reads it to decide which ranges need no graph read. A build refusal that cannot change within a generation is kept as RefusedOntology until the graph changes, so each later request answers without a rebuild. The cache entry holds no code index: its key carries the index hash, and the check uses the request's own index. The door serialises in the models' field order. * fix(graph): bound an ontology check before any work and run checks on their own workers The check reads each id outside the document once through node_arcs_many, asking only typeOf and name, with every empty answer confirmed. A node with neither reads as absent. node_arcs_batch is as on dev, and the GraphNodeReader Protocol no longer lists it. plan_reads decides the request on the event loop as soon as the document is in hand: more than CHECK_MAX_READ_IDS ids to read, or an id of the UN origin while the catalogue does not serve it, is 400 request_invalid before anything is submitted. A check stops at CHECK_MAX_FINDINGS findings with the same 400. A numeric constraint value lies within plus or minus 2**53. Checks run on a two-worker DcpReadExecutor of their own, which also encodes the response. A slot frees when its job returns, a third check answers 503 with Retry-After, and a check never takes one of the shared DCP-read workers. The handler is post_ontology_check. * fix(graph): read each check id once, confirm empty member lists and serve a node with no declaring provenance _Nodes.read skips ids already read, so a check asks each id once across its two rounds. stat_type, measurement_qualifier and measurement_denominator get the held-node test the constraint values get. build_ontology lists again, alone, each instance enumeration whose batched listing came back empty. A node or term whose typeOf carries no provenance is served with declared_by null and named in one warning per build, and OntologyProvenanceUnreadError is removed. Base enumerations find their members through one grouping of the index-named nodes. The provenances document builds once per code index and coverage build through a single-flight cache. The docs state the POST bound per backend, the largest response and one measured time for a check of 41 absent ids. * docs(graph): state the ontology door's docstrings and docs in plain sentences Docstrings and comments of the ontology door and its client reads drop joined sentences and contrast phrasing. The README, guide, reference and deploy notes split sixteen sentences the same way. * fix(graph): read an ontology check in one call and refuse a document with no instance declaration check_rows reads every planned id in one node_arcs_many call, and plan_reads returns that set. A value under a property the graph does not hold is read and gets no finding. A node's provenances are the ones its typeOf targets carry, and declared_by is null only when none carries one. A document that holds no node the instance declares is refused with OntologyNoInstanceDeclarationError, a 503 kept for the generation. The provenances document waits through aget_or_fetch_many, so a caller whose wait ends answers 503 with Retry-After. The ontology cache holds two entries, so a request under an unknown generation leaves the known entry in place. parse_arc_targets interns provenance ids. GraphDeps requires its ontology caches and check executor, and SHAPE_FIELDS is a literal tuple. * docs(graph): name the routes that answer the kept refusal and when the provenances body expires The README names the ontology, terms and check routes for the kept 503 and says the provenances body also expires at the generation stamp's maximum age. docs/reference.md keeps its table aligned. * fix(graph): keep an ontology build refusal only under a known generation Under an unknown generation nothing can clear a stored refusal before the stamp's age runs out, so _build_entry raises it without storing it and the door answers it retryable. Under a known generation the refusal is kept and answers non-retryable, as before. classify_ontology_exception maps a failed read through graph_read_problem, as the members door does. ArcSubject no longer carries types, which no caller read. * docs(graph): say when an ontology build refusal is retryable The README error table states both cases: kept and not retryable under a known generation, rebuilt and retryable under an unknown one. * fix(graph): read every value outside the document and bound the check by its deadline - The check reads a constraint value the document does not hold instead of judging it outside a fully listed range, so a member of a subclass the document lacks is no longer reported as value_outside_range. - post_ontology_check answers 504 when its deadline passes, while the worker stays held until its job returns. - constraint_value_fault drops unresolved typeOf entries itself, so the export and the check give one verdict. - The provenances route refreshes a stale generation before keying its cache, and one factory builds both caches. - Term listings judge progress by what a page advanced, the build bypasses the shared arc-targets cache, and one probe policy, individual_probe_budget, serves node_arc_targets_many and node_arcs_many with existing callers' requests unchanged. - node_arc_subjects with the UN origin off reports a complete listing, and the test fake answers a held node as the served client does. * feat(graph): answer the check rows that fit and report row faults as findings - POST /api/graph/ontology/check checks rows from the first while the ids to read stay within its read budget and the findings within their cap, and answers 200 with row_count and next_row (null when every row was checked). - A fault in a row's content is a finding: a cell outside dcid syntax gets node_not_held, a missing measured property value_missing, and an id of an origin the deployment does not serve origin_not_served. A 400 means the envelope or a size bound, with a pointer naming the row. - A finding carries row, ref, field, property, subject, value, served, code and severity. - Capabilities publish the check caps a caller can count before sending. - GET /api/graph/ontology/usage serves how each Provenance's StatVars use the model. * test(graph): pin null finding members and next_row in the encoded check answer * refactor(api): serve the JSON Schema documents at /json-schema GET /api/graph/json-schema, /api/search/json-schema and /api/datasets/json-schema serve the JSON Schema of every request and response body. The graph capabilities document lists the door as json_schema, and the eval gate's readiness probe reads /api/search/capabilities, which both trees serve. * refactor(graph): serve the instance's Data Commons schema at /api/graph/schema GET /api/graph/schema, /schema/terms and /schema/usage and POST /api/graph/schema/check serve the type layer the graph holds: classes, properties, their arcs and enumeration members. Modules, models, operation ids, cache names and the GRAPH_SCHEMA_* and SCHEMA_FLIGHT_TIMEOUT_SECONDS settings take the same name. * feat(graph): serve every property a schema node carries and bound the check answer - Each node of GET /api/graph/schema carries other_properties: every outgoing property beyond its ten named fields, as node references or string literals. Literals over 2,000 characters and values past 100 per property are left out and logged. The build reads all arcs through node_all_arc_targets_many, which confirms empty answers like the named-property reads. - POST /api/graph/schema/check also stops before the row that would take the encoded answer past 4,000,000 bytes, and always checks the first row. * feat(graph): name declaring origins, say why a refusal holds, and trim the check answer - declared_by is dfo, base or un_system, from the origin catalogue's provenance table where it lists the provenance, else from the dc/base/ and undata/ prefixes. Range and domain rules apply to dfo-declared properties. - A term listing past its cap and a build with no dfo-declared node answer 500. A 503 carries cause.code schema_building or check_busy. - The check answers {generation, rows_checked, findings}. Rows and findings carry no ref, and not_a_node marks a literal or a string outside dcid syntax where a node belongs, so node_not_held means a well-formed dcid the graph does not hold. - Schema nodes, terms and usage entries carry ref and label, as the node route does, and other_properties names a node by ref. * perf(graph): pipeline the schema build, probe empties concurrently and warm at start-up - build_schema starts each read once the reads it depends on land: index-named nodes during the term listings, each chunk expanded as it arrives, every DFO Class listed on its own, enumeration members once a chain reaches Enumeration. At most 24 reads are in flight, the bytes are those of the sequential build, and each phase's time is logged with the encode step. - An empty answer under individual_probe_budget=None is read again alone on one process-wide pool of 16 probes, with no group re-read. An id counts as absent only after its own read answers empty, and a read error propagates. - The check's read budget is 200 ids, a busy check answers Retry-After: 1, and the schema routes wait up to 3 s for the generation read. - A live Graph lifespan builds the schema document before taking traffic, bounded at 20 s; a failed or slow warm build is logged and the first request builds. * feat(graph): list instances for DFO value classes and judge ranges by read ancestry - A Class node carries instances {refs, complete}. DFO classes that descend from Enumeration or that a DFO property's rangeIncludes names list every instance of themselves and their DFO subclasses, up to INSTANCE_LISTING_CAP. A base enumeration lists the instances StatVar rows use, never complete. Other base classes carry null. - The check reads the subClassOf chain of a value's type when the type lies outside the document, inside the read budget, and judges the range against it. A chain it cannot read in full gives no verdict. * fix(graph): count a checked row only with every read it needs, and stop reads at their deadline - A row counts in rows_checked only when its named ids and its types' ancestry were all read, so each checked row gets the findings it gets alone. Row 0 may read 41 named ids and 40 ancestors; later rows yield to earlier ones, and 40 ids are held back once a row may need ancestry. - An unresolved subClassOf target leaves a chain incomplete, which gives no range verdict. - A failed schema build raises at once and abandons the reads still in flight. - Single-id probes honour the caller's deadline in the queue and in flight, and raise the deadline error on expiry. * fix(graph): name each dataset in schema usage from its provenance node build_usage takes a provenance's label and description from its graph node, read once per usage build with the same deadline and cache key, so a dataset is named whether or not the code index declares it. A failed read answers the schema door's read-failure problem and is not kept. * fix(graph): judge ranges by the build's closed subclass sets and mark cells the check cannot judge The schema check reads each id a row names once, for typeOf and name, and makes no ancestry reads. A constraint value is in range when one of its types reaches a rangeIncludes class through the document; value_outside_range needs every type known and every range class closed, meaning the build listed its whole <-subClassOf tree and the document holds it. Every other cell gets the new warning range_not_judged, so a check never reports a clean result over a cell it could not decide. property_outside_domain needs a complete ancestry over the document. A row's findings no longer depend on the other rows in the request. The build lists the descendant tree of every range class in batched node_arc_subjects_many requests. A base range within RANGE_CLOSURE_CAP (500) classes joins the served document; on dev that adds Organization's subclasses and leaves Thing out, for 1,482 nodes and a cold build of about 3.8 s. A DFO value list is complete only once every subclass under it was listed. The build records unresolved typeOf, subClassOf, rangeIncludes and domainIncludes targets and confirmed-absent ids for the check, and leaves the served bytes unchanged. parse_arc_targets keeps a node reference and a literal with the same text as two targets, and marks unresolved entries only for node_all_arc_targets_many. A kept schema refusal expires after 60 s through a new value_ttl on SingleFlightCache. * fix(graph): treat a held UN-origin type as unknown and warn on a wholly unresolved range The build's subclass listings run on our mixer, so they cannot place a UN-origin class outside a range. A UN-origin type is unknown to the range rule wherever the document holds it, after a type whose ancestry meets the range has cleared the value. A DFO Property whose rangeIncludes holds only unresolved targets counts as ranged, so its cells get range_not_judged.
…teral destination population (#310) DAC2a total ODA across recipients reads within the bilateral destination population, so a donor's recipients exclude multilateral agencies and the inclusive composite no longer counts core contributions twice. Search elects the provider reading before the read, reads a population part once, and mints the Spec as the composite on every door. bilateral_reading_unavailable carries the reasons spec-kit's warning-reason registry lists. Closes #311
* feat(code-index): keep each party's graph types on its row The code index export writes the sorted graph types of every party, and the loader refuses a party row without them, so a release built before this change fails verification and is rebuilt before a deploy moves traffic. * feat(code-index): place DAC2a destinations by graph type destination_side reads each party's graph types: Country is bilateral, an official Organization multilateral, DACUnspecifiedParty residuals multilateral unless a bilateral total lists them, and DACRecipientUnallocatedByCountry residuals bilateral. Every destination total other than all recipients, all multilateral recipients and Organization-typed totals is bilateral. party_facts reads the Organization type instead of the entity resolver's organisation records. The hand-kept group totals, multilateral residuals and bilateral unallocated lists go. * fix(code-index): bump the release format and memoise residual sides A code index whose party rows carry no types is unreadable by this image, so the release format moves to 3 and each image reads its own pointer. The residuals bilateral totals list are memoised per index, the loader refuses a malformed types value, and prose that named the removed lists now names destination_side. * docs(code-index): tighten the side rule's docstrings and README
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Releases apis at
0a051f1to production, cut onrelease/2026-11-16: four pull requests after the release atc9f3a0c.bilateral_reading_unavailablewithparams.whynaming one of five reasons. It pins spec-kit at67a0fed, which registers that warning and its reasons.country/prefix and the hand-kept lists. Africa Finance Corporation reads as a multilateral donor. The release format moves to 3.GET /api/graph/schema,/schema/terms,/schema/usageandPOST /api/graph/schema/check.GET /api/{graph,search,datasets}/json-schema. The old/api/search/schemaand/api/datasets/schemapaths answer 404. The SDK at main names them only in generated types and comments and calls neither.Release order: release this before the website's graph explorer reaches main, since it reads
GET /api/graph/schema. No migration, setting or secret change;GRAPH_SCHEMA_CHECK_TIMEOUT_SECONDSis new with a default of 30.Format 3: the deploy's graph release verify ends
no_pointeron the first production deploy, and the build publishes the first format-3 release before traffic moves. On dev that build took 17 minutes (verify at 09:12, release published 09:29). A hold on the production format-2 pointer does not apply to format 3. Data caches miss once, since the code index part changes.Testing: CI, the eval gate and Deploy dev passed on
0a051f1. On dev, France 2023 ODA by recipient answered 9,317.24m bilateral over 149 recipients and 15,412.03m inclusive over 163 with no mint warning, and nodestination_population_unavailable,graph_release_refusedorIndexFormatErrorline was logged after the format-3 release opened.