Skip to content

fix(search,datasets): read DAC2a bilateral composites within the bilateral destination population - #310

Merged
jm-rivera merged 13 commits into
devfrom
fix/dac2a-bilateral-populations
Nov 15, 2026
Merged

jm-rivera merged 13 commits into
devfrom
fix/dac2a-bilateral-populations

Conversation

@jm-rivera

@jm-rivera jm-rivera commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

DAC2a aid type 206 is total ODA and stores multilateral agencies and DACAllRecipients as recipients. An open destination under ONEDAC2aBilateral therefore read agency rows, and ONEDAC2aBilateralAndImputedMultilateral counted core contributions twice. This change reads both composites within the bilateral destination population. For France 2023 (net, current USD) Search now answers 9,317.24m bilateral over 149 recipients and 15,412.03m inclusive over 163, both equal to the OECD totals, with no agency in frames, options or scope.

Closes #311

Changes

  • code_index.RolePopulation and destination_population declare the bilateral side: 207 members, the named destinations and the DACDevelopingCountries total. code_index/multilaterals.py classifies destinations and fails closed on unknown ones.
  • code_index.composites._derived_identity binds the population into each component identity; ComponentIdentity.populations tags it and table_identity returns TableIdentity. An absent destination reads DACDevelopingCountries.
  • dc_sdmx.Among carries the population to the mint. Row readers send its members; name readers skip it, so coverage_below_scope no longer names population members.
  • Datasets mint and definition doors refuse a composite bound to an agency or DACAllRecipients with 422 facet_not_available before the registry write.
  • Search: discover_role_scope(within=...), identity_edit.composite_table_identity returning TableRead, the options door bound, whole_totals reading population.total, and the composite headline reading each component's own bound. The resubmit door refuses such identities with 422 requested_option_unavailable, cause multilateral_destination.
  • Fresh path: region/reading.py elects a DAC2a part's provider reading from pre-read state over the request's read window, reads once within the bilateral destination population (ReadBinding), and build_spec(composite=CompositeMint) mints the Spec as the composite on every door. decide_scope splits into choose_provider and build_part_scope, and within is required on materialise, run_materialise and graph_confirm_resolve. A reading whose composite cannot be minted fails closed with facet_not_available.
  • New warning bilateral_reading_unavailable when the bilateral default cannot apply. params.why is one of the five reasons spec-kit's warning-reason registry lists (provider.WithheldWhy), with params.slot for slot_not_served; tests/platform/test_registry_binding.py checks them against the installed WARNING_REASONS. role_scope_defaulted gains params.axis.
  • Scoped composite mints apply the scope's entities and window to every component. On dev they read the unscoped component.
  • observations.composite_facet_ref resolves a composite's provenance facet from its component tables: data_commons and source stay when every component names the same one. Composite answers, the bilateral default included, carry data_commons.id dfo.
  • The eval's materialisation check counts each composite component within its population (dq-crs-053 counted 11,263 rows against the answer's 7,842). The 72 DAC2a cases that now answer the bilateral composite expect provenance/ONEDAC2aBilateral with origin dfo; dq-crs-054 and dataset-aid-kenya-not-imputed record measure failures dev shares.
  • Bundles, citations and labels name "OECD DAC2a (bilateral flows)". Conformance C66d to C66i, C149 and C201, and the Search README cover populations.

Notes for reviewers

Testing: targeted tests for the changed modules, ruff, codespell and ty through pre-commit; the full suite on CI. A live check against dev reproduced the France 2023 figures above, one run_materialise per population part, equal SSE streams and door refusals, and fresh answers equal to resubmits on spec_id, scope, entities, headline, coverage and download.

…teral destination population

DAC2a aid type 206 is total ODA and stores multilateral agencies and DACAllRecipients as recipients, so an open destination under ONEDAC2aBilateral read agency rows and ONEDAC2aBilateralAndImputedMultilateral counted core contributions twice.

A destination role can now be open within a declared population. code_index.RolePopulation declares the bilateral side (207 members, the named destinations and the DACDevelopingCountries total); composites._derived_identity binds it into each component identity and tags it on ComponentIdentity.populations. The mint reads it as dc_sdmx.Among, Search discovers role scope within it, the options door offers only its members, and an absent destination reads DACDevelopingCountries. Unclassified destinations in coverage refuse the composite. An identity binding these composites to an agency or DACAllRecipients refuses with 422.

A fresh Search part that settles on a population composite is read again within the population through resolve_spec_resubmit, so fresh answers equal resubmits. The new bilateral_reading_unavailable warning reports when the bilateral default cannot apply. Scoped composite mints now apply the scope's entities and window to every component.

The dfo-spec-kit pin moves to 9092fec, which registers bilateral_reading_unavailable.
… share

A DAC2a total now defaults to the ONEDAC2aBilateral composite, and assemble_composite built its provenance facet with only basis, ref, label and account, so resolved.facets.provenance lost data_commons and source. observations.composite_facet_ref resolves a composite's facet from its component tables' catalogue rows: source and data_commons are kept when every component names the same one, and left out otherwise. Both the series facets and the resolved facet use it.
The materialisation check counted a composite with only the roles bound in resolved.entities, so an open destination under the bilateral composite counted agencies and DACAllRecipients (11,263 against the answer's 7,842 on dq-crs-053). _CountedRead.within carries each component's population into the series count and the latest-year election. The DAC2a cases that now answer the bilateral composite expect provenance/ONEDAC2aBilateral with origin dfo.
…parture gate

core_contribution_read describes the data (a multilateral agency named as a recipient reads total ODA to it), so it joins DATA_DESCRIBING_DEPARTURES. Test doubles of ResolveState gain from_request.
dq-crs-054 and dataset-aid-kenya-not-imputed answer the net netting basis where they expect gross, and the second also adds DACOfficialDonors as an origin. Both fail the same way on dev; editing their provenance made the eval treat them as new cases, so their known failures now name these checks.
A DAC2a part now elects its provider reading from pre-read state over the request's read window (region/reading.py), reads once within the bilateral destination population, and mints the Spec as the composite through build_spec(composite=CompositeMint). The restatement layer that resolved the population a second time and reconciled it onto the table answer is deleted: with_population_read, restated_reading, with_composite_answer, with_restated_keys, recounted_scope, assemble_composite, with_bilateral_readings, as_merged_composite and ResolveState.request_warnings.

decide_scope splits into choose_provider before the read and build_part_scope after it, and within is required on materialise, run_materialise and graph_confirm_resolve. The bilateral_reading_unavailable warning carries only the params.why reasons spec-kit's warning-reason registry lists.

Closes #311
A flagged match whose composite cannot be minted, and a candidate set whose elected reading cannot, now answer facet_not_available as the definite path does. The definite carve-out refusals state the decision's bilateral_reading_unavailable warning. A provider option refused at an entity, provenance or shape member marks only that option composite_not_servable and the scope still builds. Duplicate candidates are skipped before their composite is minted, and _suggest_nearest_real takes the part's scope selection, operations and catalogue as required arguments.
A part naming no destination reads the population total through the read (reading.ReadIn), so state.place, the bindings and nearest-real relaxations keep the places the request named, and resolved.entities lists the total without the identity binding it. The election decides reach by provider.counted_source, the rule the scope after the read uses, over a canonical pre-read identity, and one ScopeFacts per elected dataset serves every election of a part.
…nts it

CompositeMint.role_scope_warnings takes the count from the Spec's own role axis facts, so the warning and the facts agree when a component's parties cannot be listed, and a composite candidate whose scope lists no member drops the warning as any other candidate does.
The registry binding test reads the installed WARNING_REASONS, so it checks Search's bilateral_reading_unavailable reasons against the registry on every run.
…ad disagree

A composite elected without its population read answers as the table with bilateral_reading_unavailable scope_undecided, an election that raises answers undecided over the answer identity, and a window mismatch the election window explains answers the table with no degraded note. A flagged refusal that read no rows keeps its own refusal, nearest-real suggestions read and list the population total as answers do, and roles_read keeps every role key it is given.
Candidates warn only for readings that ground, a refused composite is noted once per region, a composite candidate probes its slot options as a definite answer does, and the role-scope restatement rewrites only the answer's own warning. A discarded flagged match leaves no degraded-read note, the spec_id kind points only its off-side composite refusal at /input/spec_id, and reading.named_entity_data keeps read totals out of identity entities at every door.
…lect-reading-first

refactor(search): elect the provider reading before the read
@github-actions github-actions Bot removed the run-eval label Nov 15, 2026
@jm-rivera
jm-rivera merged commit 7af2d8d into dev Nov 15, 2026
31 of 32 checks passed
@jm-rivera
jm-rivera deleted the fix/dac2a-bilateral-populations branch November 15, 2026 21:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Elect the provider reading before Search reads the table

1 participant