Repository navigation
fix(search,datasets): read DAC2a bilateral composites within the bilateral destination population - #310
Merged
Conversation
…teral destination population DAC2a aid type 206 is total ODA and stores multilateral agencies and DACAllRecipients as recipients, so an open destination under ONEDAC2aBilateral read agency rows and ONEDAC2aBilateralAndImputedMultilateral counted core contributions twice. A destination role can now be open within a declared population. code_index.RolePopulation declares the bilateral side (207 members, the named destinations and the DACDevelopingCountries total); composites._derived_identity binds it into each component identity and tags it on ComponentIdentity.populations. The mint reads it as dc_sdmx.Among, Search discovers role scope within it, the options door offers only its members, and an absent destination reads DACDevelopingCountries. Unclassified destinations in coverage refuse the composite. An identity binding these composites to an agency or DACAllRecipients refuses with 422. A fresh Search part that settles on a population composite is read again within the population through resolve_spec_resubmit, so fresh answers equal resubmits. The new bilateral_reading_unavailable warning reports when the bilateral default cannot apply. Scoped composite mints now apply the scope's entities and window to every component. The dfo-spec-kit pin moves to 9092fec, which registers bilateral_reading_unavailable.
5 tasks
… share A DAC2a total now defaults to the ONEDAC2aBilateral composite, and assemble_composite built its provenance facet with only basis, ref, label and account, so resolved.facets.provenance lost data_commons and source. observations.composite_facet_ref resolves a composite's facet from its component tables' catalogue rows: source and data_commons are kept when every component names the same one, and left out otherwise. Both the series facets and the resolved facet use it.
The materialisation check counted a composite with only the roles bound in resolved.entities, so an open destination under the bilateral composite counted agencies and DACAllRecipients (11,263 against the answer's 7,842 on dq-crs-053). _CountedRead.within carries each component's population into the series count and the latest-year election. The DAC2a cases that now answer the bilateral composite expect provenance/ONEDAC2aBilateral with origin dfo.
…parture gate core_contribution_read describes the data (a multilateral agency named as a recipient reads total ODA to it), so it joins DATA_DESCRIBING_DEPARTURES. Test doubles of ResolveState gain from_request.
dq-crs-054 and dataset-aid-kenya-not-imputed answer the net netting basis where they expect gross, and the second also adds DACOfficialDonors as an origin. Both fail the same way on dev; editing their provenance made the eval treat them as new cases, so their known failures now name these checks.
A DAC2a part now elects its provider reading from pre-read state over the request's read window (region/reading.py), reads once within the bilateral destination population, and mints the Spec as the composite through build_spec(composite=CompositeMint). The restatement layer that resolved the population a second time and reconciled it onto the table answer is deleted: with_population_read, restated_reading, with_composite_answer, with_restated_keys, recounted_scope, assemble_composite, with_bilateral_readings, as_merged_composite and ResolveState.request_warnings. decide_scope splits into choose_provider before the read and build_part_scope after it, and within is required on materialise, run_materialise and graph_confirm_resolve. The bilateral_reading_unavailable warning carries only the params.why reasons spec-kit's warning-reason registry lists. Closes #311
A flagged match whose composite cannot be minted, and a candidate set whose elected reading cannot, now answer facet_not_available as the definite path does. The definite carve-out refusals state the decision's bilateral_reading_unavailable warning. A provider option refused at an entity, provenance or shape member marks only that option composite_not_servable and the scope still builds. Duplicate candidates are skipped before their composite is minted, and _suggest_nearest_real takes the part's scope selection, operations and catalogue as required arguments.
A part naming no destination reads the population total through the read (reading.ReadIn), so state.place, the bindings and nearest-real relaxations keep the places the request named, and resolved.entities lists the total without the identity binding it. The election decides reach by provider.counted_source, the rule the scope after the read uses, over a canonical pre-read identity, and one ScopeFacts per elected dataset serves every election of a part.
…nts it CompositeMint.role_scope_warnings takes the count from the Spec's own role axis facts, so the warning and the facts agree when a component's parties cannot be listed, and a composite candidate whose scope lists no member drops the warning as any other candidate does.
The registry binding test reads the installed WARNING_REASONS, so it checks Search's bilateral_reading_unavailable reasons against the registry on every run.
…ad disagree A composite elected without its population read answers as the table with bilateral_reading_unavailable scope_undecided, an election that raises answers undecided over the answer identity, and a window mismatch the election window explains answers the table with no degraded note. A flagged refusal that read no rows keeps its own refusal, nearest-real suggestions read and list the population total as answers do, and roles_read keeps every role key it is given.
Candidates warn only for readings that ground, a refused composite is noted once per region, a composite candidate probes its slot options as a definite answer does, and the role-scope restatement rewrites only the answer's own warning. A discarded flagged match leaves no degraded-read note, the spec_id kind points only its off-side composite refusal at /input/spec_id, and reading.named_entity_data keeps read totals out of identity entities at every door.
…lect-reading-first refactor(search): elect the provider reading before the read
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DAC2a aid type 206 is total ODA and stores multilateral agencies and
DACAllRecipientsas recipients. An open destination underONEDAC2aBilateraltherefore read agency rows, andONEDAC2aBilateralAndImputedMultilateralcounted core contributions twice. This change reads both composites within the bilateral destination population. For France 2023 (net, current USD) Search now answers 9,317.24m bilateral over 149 recipients and 15,412.03m inclusive over 163, both equal to the OECD totals, with no agency in frames, options or scope.Closes #311
Changes
code_index.RolePopulationanddestination_populationdeclare the bilateral side: 207 members, the named destinations and theDACDevelopingCountriestotal.code_index/multilaterals.pyclassifies destinations and fails closed on unknown ones.code_index.composites._derived_identitybinds the population into each component identity;ComponentIdentity.populationstags it andtable_identityreturnsTableIdentity. An absent destination readsDACDevelopingCountries.dc_sdmx.Amongcarries the population to the mint. Row readers send its members; name readers skip it, socoverage_below_scopeno longer names population members.DACAllRecipientswith 422facet_not_availablebefore the registry write.discover_role_scope(within=...),identity_edit.composite_table_identityreturningTableRead, the options door bound,whole_totalsreadingpopulation.total, and the composite headline reading each component's own bound. The resubmit door refuses such identities with 422requested_option_unavailable, causemultilateral_destination.region/reading.pyelects a DAC2a part's provider reading from pre-read state over the request's read window, reads once within the bilateral destination population (ReadBinding), andbuild_spec(composite=CompositeMint)mints the Spec as the composite on every door.decide_scopesplits intochoose_providerandbuild_part_scope, andwithinis required onmaterialise,run_materialiseandgraph_confirm_resolve. A reading whose composite cannot be minted fails closed withfacet_not_available.bilateral_reading_unavailablewhen the bilateral default cannot apply.params.whyis one of the five reasons spec-kit's warning-reason registry lists (provider.WithheldWhy), withparams.slotforslot_not_served;tests/platform/test_registry_binding.pychecks them against the installedWARNING_REASONS.role_scope_defaultedgainsparams.axis.observations.composite_facet_refresolves a composite's provenance facet from its component tables:data_commonsandsourcestay when every component names the same one. Composite answers, the bilateral default included, carrydata_commons.iddfo.provenance/ONEDAC2aBilateralwith origindfo; dq-crs-054 and dataset-aid-kenya-not-imputed record measure failures dev shares.Notes for reviewers
67a0fed, the merge of feat(registry): register the bilateral_reading_unavailable Search warning spec-kit#54.DACAllRecipientsnow refuse with 422.withinset, where a first version read the whole table and then the population. A parsed "total ODA" request with a rank gradesconfidencemedium, as the same request without a rank does: its status menu holds a second admissible reading with data.Testing: targeted tests for the changed modules, ruff, codespell and ty through pre-commit; the full suite on CI. A live check against dev reproduced the France 2023 figures above, one
run_materialiseper population part, equal SSE streams and door refusals, and fresh answers equal to resubmits on spec_id, scope, entities, headline, coverage and download.