Repository navigation
feat!: adopt the client contract at spec-kit 2f7462a - #36
Merged
Merged
Conversation
- typescript: trimEnd in core/text.ts trims trailing characters in one scan and replaces every trailing-trim regex, which ran in quadratic time on long runs (CodeQL js/polynomial-redos). Two tests escape their input in full (js/incomplete-sanitization). - python: the transport builds its own httpcore2 pools with the attempt-deadline backend through public names only, reads proxies from urllib, and drops the <2.16 bounds on httpx2 and httpcore2. - Both clients set answer.timed_out when any part's own reason is timed_out, and expose the party totals a rendering left out as Dataset.party_totals_left_out (partyTotalsLeftOut), saved under dfo.party_totals_left_out. - typescript: Answer.facts and partFacts give the checked facts of each part. - The dashboard journeys check the local-stack seed once and stop with one failure that names the missing readings. - CI installs the lowest permitted versions on Python 3.12 and the newest on 3.14, weekly as well, and tests the apache-arrow peer at 21.0.0 and the newest 21.x. Struct columns that differ between members now stack on pyarrow 15 to 20, and float16 columns compare on pyarrow 15 to 25. polars needs 1.3. Closes #11 Closes #24
Both packages move to 0.2.0. Every breaking change is listed under Unreleased in python/CHANGELOG.md and typescript/CHANGELOG.md. - Errors: every error a public call raises is a DfoError, FormatError and BundleError included, and a damaged file given to dfo.load raises one. Calls that raised ValueError or TypeError still do. Messages end in [<code> trace=<id>]. The old-form shims in Answer.choose and the TypeScript options map are gone. - Timeouts: the client's timeout bounds a whole call unless the call sets its own, and attempt_timeout (attemptTimeout) bounds one attempt. docs/settings.md in each package lists every setting. - entities= takes dcids or what lookup.resolve returns. - fetch is overloaded per target in both languages, and type tests pin the arguments each target refuses. - Datasets: Python gains to_bytes() and load(bytes); TypeScript renames bytes() to toBytes(), gains save(path) on Node, and spells licence as Python does. A cache hit says so in the repr, and saved files record dfo.question and dfo.fetched_at. - TypeScript: Answer.parts types only the members the client checks, isKnownPart checks them, QueryOption is Option, and the root exports the nine names that match Python's top level, with the rest under ./errors and ./types. - Docs: the recipes run without warnings a reader should ignore, and a nightly workflow runs every documented example against dev. Closes #6 Refs #19
tests/test_typing.py runs ty on a forwarded fetch target, so the python-lowest job pins ty from uv.lock beside the other test tools.
pyarrow raises a plain OSError, outside ArrowException, when a page header fails to decode, so dfo.load let it escape DfoError. _open_saved opens a path before parsing, so a missing file still raises FileNotFoundError, and turns any OSError while decoding into BundleError not_a_bundle.
Regenerates the Python error map and wire types and the TypeScript actions, client codes and OpenAPI types from it.
Both clients post the party totals cut as `party_totals: false`, read it in either spelling in refs and served renderings, and keep writing refs with `without_party_totals: true` so 0.1.0 can read them. The release workflow runs scripts/release_preflight.py first and refuses to publish until production accepts the key (dfo-apis 133262c5e on). DFO_TIMEOUT and DFO_MAX_RETRIES set the client defaults, with a strict grammar and the precedence call > client > variable > default. Python defaults `timeout` and `max_retries` to `dfo.types.FROM_ENVIRONMENT`, so `timeout=None` means no client bound and leaves the variable unread. The Python cache lives under the `dfo` platform directory. With no explicit root it reads the legacy `readerkit` directory, counts an artifact only when it matches its digest and a manifest only when it passes the cached-manifest checks, and copies a hit in by rename. FormatError and BundleError carry their reason as `code`. A ref asked of a folder bundle, an unsupported proxy scheme from the environment and an unresolved lookup Match raise `ref_unavailable`, `proxy_invalid` and `match_unresolved`. Citations read `licence*` before `license*`, `party_totals_left_out` reads leniently, and `dfo.load` checks a saved file's format version and party-totals members against its ref.
The client contract moves to spec-kit e5efe70. A party-totals entry carries `kept_in`, the series and time periods in which its totals stayed (`dfo.types.KeptSeries`, TypeScript `KeptSeries`), and a `role_key` (`roleKey`) read from the contract's column table, so `financingProvider` reads as `origin`. Readers drop an entry that breaks the served order or holds an empty string. A derived dataset reports what each root's read left out, from the derived manifest's roots, in the order of the ref member's roots. Its saved entries name those roots, and `dfo.load` refuses the member's own id. The text line reads "Party totals left out of the readings" for a derived dataset.
Spec-kit #52 now carries spec-kit dev, so SOURCE names its merge with dev. The contract files are unchanged.
jm-rivera
marked this pull request as ready for review
November 14, 2026 14:35
_root_party_totals keyed a dict on each served root's spec_id and content_version, so a list there raised TypeError and failed the fetch. Such a root now names no totals, as the docstring says. dfo.search and dfo.fetch name DFO_TIMEOUT in their timeout fallback, and an orphaned comment above _merge_artifacts goes.
…Error A deadline taken from the client's timeout now reads "the client's timeout passed", for pin, cards, history, lookups and raw requests as well as fetch. dfo.load takes any bytes-like object, and a read that fails at the disk raises its own OSError where it was reported as a damaged file. A TypeScript fetch of a ref or share refuses entities before it resolves a Match, as Python does.
On pyarrow 15 to 20 the client stacked members whose struct columns differ by round-tripping cells through Python, which cut nanoseconds from time64[ns] fields, dropped field nullability and metadata, and missed structs inside maps. pyarrow 21 promotes structs in concat_tables, so the fallback and its version gate go. A float16 column is cast to float64 before its unique values are taken, which needs no version check.
# Conflicts: # python/tests/datasets/test_dataset.py
# Conflicts: # python/src/dfo/_client/default.py # python/src/dfo/_core/timeouts.py # python/src/dfo/_http/transport.py # python/src/dfo/_sync/_http/transport.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Both clients adopt the client contract at spec-kit 2f7462a: the party totals cut posts as
party_totals: false, the timeout and retry count readDFO_TIMEOUTandDFO_MAX_RETRIES, the Python cache moves to adfodirectory that still reads the oldreaderkitone, the new client codes, licence spelling and saved-file checks follow the contract, and the party totals report names each entry's role key, the series and periods where totals stayed, and a derived dataset's roots. With #34 this settles the remaining 0.2.0 surface.Closes #19
Changes
contract/SOURCEmoves to spec-kit 2f7462a, with the generated error maps, wire types and TypeScript OpenAPI types regenerated from it.Rendering.to_optionsand TypeScriptmintOptionspostparty_totals: falsefor the cut.party_totals_cut(Python) andpartyTotalsCutandservedCut(TypeScript) read the cut in either spelling in the ref loaders and every manifest and frame reader. Refs the clients write keepwithout_party_totals: true, and recipe roots keep the old key..github/workflows/release.ymlgains apreflightjob that both publish jobs need. It runsscripts/release_preflight.pyagainst production with no key: a control fetch that posts no cut key, then a default fetch that postsparty_totals: false.resolve_timeoutandresolve_max_retries(Python_http/transport.py) andresolveTimeoutandresolveMaxRetries(TypeScripthttp/transport.ts) implement the timeout-and-retries rule. Python defaults both arguments todfo.types.FROM_ENVIRONMENT, and TypeScript reads the variables in Node._config/paths.pyand_datasets/cache.py: the platform root isdfo, and with no explicit root the cache reads the legacyreaderkitdirectory, verifies each hit, copies it in by rename, and never writes, evicts or counts there.FormatErrorandBundleErrorcarry their reason ascode.ref_unavailable(Dataset.refon a folder bundle),proxy_invalid(a proxy scheme from the environment) andmatch_unresolved(an unresolvedMatchas an entity) replaceinvalid_argumentthere.licence*with alicense*fallback,party_totals_left_outreads leniently, and Python'sdfo.loadchecks a saved file's format version and party-totals members against its ref, tested against every example incontract/saved-files/.PartyTotalsLeftOutgainskept_in(dfo.types.KeptSeries; TypeScriptkeptInandKeptSeries), the series and time periods in which its totals stayed, androle_key(roleKey) from the contract'scolumnstable, sofinancingProviderreads asorigin. Readers drop an entry that breaks the served order or holds an empty string, anddfo.loadrefuses one.roots[].party_totals_left_out, paired with the ref member's roots (a served root whose spec id or content version is no string names none) and in their order. Saved entries name those roots,dfo.loadrefuses the member's own id, and the text line reads "Party totals left out of the readings".party_totals_left_out, andtypescript/test/data/answer-choose-candidate-two-parts.jsonis regenerated from the re-recorded scenario.Notes for reviewers
check_contract_pin.pyfails here until #299 is on dfo-apis dev. fix(datasets): keep a year's whole totals and report each root's cut apis#303, which serveskept_inand the derived roots' reports, is on dev.party_totalstoday: the preflight's default fetch returns 400request_invalidat/options/party_totals, and its control fetch passes. The release job stops there by design.timeout=Noneon a Python client now means no client bound, distinct from leaving the argument out, which readsDFO_TIMEOUT.max_retries=Noneraises, as the contract's vector requires.ref_unavailableandproxy_invalidare Python only.dfo cache clearempties thedfodirectory alone, so legacy entries still hit after a clear. The docs say so.Testing: Python ruff, ty, lint-imports, the three generators'
--checkandpytest(4,551 passed); TypeScript lint, typecheck,generate:check, test (2,195 passed) and build; actionlint;check_contract_pin.pyagainst #299's pins at 2f7462a; the live journeys against dev at apis aacbe6a4 (Python 9 passed, TypeScript 9 passed, cross-language ref included); the local-stack journeys against the same commit (Python 14 passed, TypeScript 1 passed); the docs examples against the dev website (Python 68 passed, TypeScript 30 passed); a differential check of 2,407kept_inlists in which both readers, the spec-kit validator and the dfo-apis writer agree; a fetch of "OOF to India by donor" on dev returning 1968'sDACOfficialDonorsrow withrole_keyoriginand itskept_inseries; and the preflight against production, which stops at the default fetch as described above.