Skip to content

feat!: adopt the client contract at spec-kit 2f7462a - #36

Merged
jm-rivera merged 14 commits into
mainfrom
feat/contract-sdk-0.2
Nov 14, 2026
Merged

jm-rivera merged 14 commits into
mainfrom
feat/contract-sdk-0.2

Conversation

@jm-rivera

@jm-rivera jm-rivera commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Both clients adopt the client contract at spec-kit 2f7462a: the party totals cut posts as party_totals: false, the timeout and retry count read DFO_TIMEOUT and DFO_MAX_RETRIES, the Python cache moves to a dfo directory that still reads the old readerkit one, the new client codes, licence spelling and saved-file checks follow the contract, and the party totals report names each entry's role key, the series and periods where totals stayed, and a derived dataset's roots. With #34 this settles the remaining 0.2.0 surface.

Closes #19

Changes

  • contract/SOURCE moves to spec-kit 2f7462a, with the generated error maps, wire types and TypeScript OpenAPI types regenerated from it.
  • Python Rendering.to_options and TypeScript mintOptions post party_totals: false for the cut. party_totals_cut (Python) and partyTotalsCut and servedCut (TypeScript) read the cut in either spelling in the ref loaders and every manifest and frame reader. Refs the clients write keep without_party_totals: true, and recipe roots keep the old key.
  • .github/workflows/release.yml gains a preflight job that both publish jobs need. It runs scripts/release_preflight.py against production with no key: a control fetch that posts no cut key, then a default fetch that posts party_totals: false.
  • resolve_timeout and resolve_max_retries (Python _http/transport.py) and resolveTimeout and resolveMaxRetries (TypeScript http/transport.ts) implement the timeout-and-retries rule. Python defaults both arguments to dfo.types.FROM_ENVIRONMENT, and TypeScript reads the variables in Node.
  • _config/paths.py and _datasets/cache.py: the platform root is dfo, and with no explicit root the cache reads the legacy readerkit directory, verifies each hit, copies it in by rename, and never writes, evicts or counts there.
  • FormatError and BundleError carry their reason as code. ref_unavailable (Dataset.ref on a folder bundle), proxy_invalid (a proxy scheme from the environment) and match_unresolved (an unresolved Match as an entity) replace invalid_argument there.
  • Citations read licence* with a license* fallback, party_totals_left_out reads leniently, and Python's dfo.load checks a saved file's format version and party-totals members against its ref, tested against every example in contract/saved-files/.
  • PartyTotalsLeftOut gains kept_in (dfo.types.KeptSeries; TypeScript keptIn and KeptSeries), the series and time periods in which its totals stayed, and role_key (roleKey) from the contract's columns table, so financingProvider reads as origin. Readers drop an entry that breaks the served order or holds an empty string, and dfo.load refuses one.
  • A derived dataset reports what each root's read left out, from the derived manifest's roots[].party_totals_left_out, paired with the ref member's roots (a served root whose spec id or content version is no string names none) and in their order. Saved entries name those roots, dfo.load refuses the member's own id, and the text line reads "Party totals left out of the readings".
  • The contract replays compare party_totals_left_out, and typescript/test/data/answer-choose-candidate-two-parts.json is regenerated from the re-recorded scenario.
  • Settings, cache and errors pages and both CHANGELOGs describe the changes.

Notes for reviewers

Testing: Python ruff, ty, lint-imports, the three generators' --check and pytest (4,551 passed); TypeScript lint, typecheck, generate:check, test (2,195 passed) and build; actionlint; check_contract_pin.py against #299's pins at 2f7462a; the live journeys against dev at apis aacbe6a4 (Python 9 passed, TypeScript 9 passed, cross-language ref included); the local-stack journeys against the same commit (Python 14 passed, TypeScript 1 passed); the docs examples against the dev website (Python 68 passed, TypeScript 30 passed); a differential check of 2,407 kept_in lists in which both readers, the spec-kit validator and the dfo-apis writer agree; a fetch of "OOF to India by donor" on dev returning 1968's DACOfficialDonors row with role_key origin and its kept_in series; and the preflight against production, which stops at the default fetch as described above.

- typescript: trimEnd in core/text.ts trims trailing characters in one
  scan and replaces every trailing-trim regex, which ran in quadratic
  time on long runs (CodeQL js/polynomial-redos). Two tests escape
  their input in full (js/incomplete-sanitization).
- python: the transport builds its own httpcore2 pools with the
  attempt-deadline backend through public names only, reads proxies
  from urllib, and drops the <2.16 bounds on httpx2 and httpcore2.
- Both clients set answer.timed_out when any part's own reason is
  timed_out, and expose the party totals a rendering left out as
  Dataset.party_totals_left_out (partyTotalsLeftOut), saved under
  dfo.party_totals_left_out.
- typescript: Answer.facts and partFacts give the checked facts of
  each part.
- The dashboard journeys check the local-stack seed once and stop with
  one failure that names the missing readings.
- CI installs the lowest permitted versions on Python 3.12 and the
  newest on 3.14, weekly as well, and tests the apache-arrow peer at
  21.0.0 and the newest 21.x. Struct columns that differ between
  members now stack on pyarrow 15 to 20, and float16 columns compare
  on pyarrow 15 to 25. polars needs 1.3.

Closes #11
Closes #24
Both packages move to 0.2.0. Every breaking change is listed under
Unreleased in python/CHANGELOG.md and typescript/CHANGELOG.md.

- Errors: every error a public call raises is a DfoError, FormatError
  and BundleError included, and a damaged file given to dfo.load
  raises one. Calls that raised ValueError or TypeError still do.
  Messages end in [<code> trace=<id>]. The old-form shims in
  Answer.choose and the TypeScript options map are gone.
- Timeouts: the client's timeout bounds a whole call unless the call
  sets its own, and attempt_timeout (attemptTimeout) bounds one
  attempt. docs/settings.md in each package lists every setting.
- entities= takes dcids or what lookup.resolve returns.
- fetch is overloaded per target in both languages, and type tests
  pin the arguments each target refuses.
- Datasets: Python gains to_bytes() and load(bytes); TypeScript
  renames bytes() to toBytes(), gains save(path) on Node, and spells
  licence as Python does. A cache hit says so in the repr, and saved
  files record dfo.question and dfo.fetched_at.
- TypeScript: Answer.parts types only the members the client checks,
  isKnownPart checks them, QueryOption is Option, and the root
  exports the nine names that match Python's top level, with the rest
  under ./errors and ./types.
- Docs: the recipes run without warnings a reader should ignore, and
  a nightly workflow runs every documented example against dev.

Closes #6
Refs #19
tests/test_typing.py runs ty on a forwarded fetch target, so the
python-lowest job pins ty from uv.lock beside the other test tools.
pyarrow raises a plain OSError, outside ArrowException, when a page
header fails to decode, so dfo.load let it escape DfoError. _open_saved
opens a path before parsing, so a missing file still raises
FileNotFoundError, and turns any OSError while decoding into
BundleError not_a_bundle.
Regenerates the Python error map and wire types and the TypeScript
actions, client codes and OpenAPI types from it.
Both clients post the party totals cut as `party_totals: false`, read
it in either spelling in refs and served renderings, and keep writing
refs with `without_party_totals: true` so 0.1.0 can read them. The
release workflow runs scripts/release_preflight.py first and refuses
to publish until production accepts the key (dfo-apis 133262c5e on).

DFO_TIMEOUT and DFO_MAX_RETRIES set the client defaults, with a strict
grammar and the precedence call > client > variable > default. Python
defaults `timeout` and `max_retries` to `dfo.types.FROM_ENVIRONMENT`,
so `timeout=None` means no client bound and leaves the variable unread.

The Python cache lives under the `dfo` platform directory. With no
explicit root it reads the legacy `readerkit` directory, counts an
artifact only when it matches its digest and a manifest only when it
passes the cached-manifest checks, and copies a hit in by rename.

FormatError and BundleError carry their reason as `code`. A ref asked
of a folder bundle, an unsupported proxy scheme from the environment
and an unresolved lookup Match raise `ref_unavailable`, `proxy_invalid`
and `match_unresolved`. Citations read `licence*` before `license*`,
`party_totals_left_out` reads leniently, and `dfo.load` checks a saved
file's format version and party-totals members against its ref.
The client contract moves to spec-kit e5efe70. A party-totals entry
carries `kept_in`, the series and time periods in which its totals
stayed (`dfo.types.KeptSeries`, TypeScript `KeptSeries`), and a
`role_key` (`roleKey`) read from the contract's column table, so
`financingProvider` reads as `origin`. Readers drop an entry that
breaks the served order or holds an empty string.

A derived dataset reports what each root's read left out, from the
derived manifest's roots, in the order of the ref member's roots. Its
saved entries name those roots, and `dfo.load` refuses the member's
own id. The text line reads "Party totals left out of the readings"
for a derived dataset.
@jm-rivera jm-rivera changed the title feat!: adopt the client contract at spec-kit 7ef7f71 feat!: adopt the client contract at spec-kit e5efe70 Nov 14, 2026
Spec-kit #52 now carries spec-kit dev, so SOURCE names its merge
with dev. The contract files are unchanged.
@jm-rivera jm-rivera changed the title feat!: adopt the client contract at spec-kit e5efe70 feat!: adopt the client contract at spec-kit 2f7462a Nov 14, 2026
@jm-rivera
jm-rivera marked this pull request as ready for review November 14, 2026 14:35
_root_party_totals keyed a dict on each served root's spec_id and
content_version, so a list there raised TypeError and failed the fetch.
Such a root now names no totals, as the docstring says. dfo.search and
dfo.fetch name DFO_TIMEOUT in their timeout fallback, and an orphaned
comment above _merge_artifacts goes.
…Error

A deadline taken from the client's timeout now reads "the client's
timeout passed", for pin, cards, history, lookups and raw requests as
well as fetch. dfo.load takes any bytes-like object, and a read that
fails at the disk raises its own OSError where it was reported as a
damaged file. A TypeScript fetch of a ref or share refuses entities
before it resolves a Match, as Python does.
On pyarrow 15 to 20 the client stacked members whose struct columns
differ by round-tripping cells through Python, which cut nanoseconds
from time64[ns] fields, dropped field nullability and metadata, and
missed structs inside maps. pyarrow 21 promotes structs in
concat_tables, so the fallback and its version gate go. A float16
column is cast to float64 before its unique values are taken, which
needs no version check.
# Conflicts:
#	python/tests/datasets/test_dataset.py
# Conflicts:
#	python/src/dfo/_client/default.py
#	python/src/dfo/_core/timeouts.py
#	python/src/dfo/_http/transport.py
#	python/src/dfo/_sync/_http/transport.py
Base automatically changed from feat/public-surface-0.2 to main November 14, 2026 18:07
@jm-rivera
jm-rivera merged commit 4b0c393 into main Nov 14, 2026
15 checks passed
@jm-rivera
jm-rivera deleted the feat/contract-sdk-0.2 branch November 14, 2026 18:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Settle the public surface of both clients before releasing 0.1.0

1 participant