Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,27 @@ permissions:
contents: read

jobs:
# Both clients post `party_totals: false` for the default cut. Production must
# accept that key before either client publishes.
preflight:
runs-on: ubuntu-latest
defaults:
run:
working-directory: python
steps:
- uses: actions/checkout@v7

- name: Install uv
uses: astral-sh/setup-uv@v10.0.0

- name: Set up Python 3.12
run: uv python install '3.12'

- name: Check that production accepts party_totals
run: uv run --python 3.12 python ../scripts/release_preflight.py

python:
needs: preflight
if: startsWith(github.ref_name, 'python-v')
runs-on: ubuntu-latest
environment:
Expand Down Expand Up @@ -41,6 +61,7 @@ jobs:
run: uv publish

typescript:
needs: preflight
if: startsWith(github.ref_name, 'typescript-v')
runs-on: ubuntu-latest
environment:
Expand Down
2 changes: 1 addition & 1 deletion contract/SOURCE
Original file line number Diff line number Diff line change
@@ -1 +1 @@
4ad77392b3ca9ae61d4c0522981878682d7d2b90
2f7462a91408a18ff2bf97872fed735a8c052382
16 changes: 12 additions & 4 deletions python/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,25 +8,33 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),

### Added

- `data.party_totals_left_out` names the group totals a fetch left out of each member, as `dfo.types.PartyTotalsLeftOut` entries of spec id, party role, dcids and provenance, and the repr counts them. `data.save` writes them under a new `dfo.party_totals_left_out` key, which 0.1.0 ignores, and `dfo.load` reads a file without it.
- `entities=` on `fetch` and `answer.fetch` takes the `Match` and `Entity` objects `dfo.lookup.resolve` returns beside dcids, as in `entities={"destination": dfo.lookup.resolve("Nigeria")}`. A match that is not `resolved` raises `InvalidRequestError` (`invalid_argument`) before any request, printing the match and a candidate to pass instead. An awaitable in its place, such as `client.lookup.resolve(...)` on an `AsyncClient` without `await`, raises the same error and names the missing `await`.
- `data.party_totals_left_out` names the group totals a fetch left out of each member, as `dfo.types.PartyTotalsLeftOut` entries of spec id, party role, dcids, provenance and `kept_in`, the series whose rows kept those totals, each a `dfo.types.KeptSeries` of the series' cells and the periods in which it kept them. `role_key` names the `entities=` key of the entry's column, `origin` or `destination`, or None for a column the client does not know. The repr counts the totals on each role with its role key, as `15 on financingProvider (origin)`. Entries keep the manifest's order, and two can share a role and a provenance. `data.save` writes them under a new `dfo.party_totals_left_out` key, with `kept_in` where an entry holds it, which 0.1.0 ignores, and `dfo.load` reads a file without it.
- A derived Dataset reports the party totals each reading's read left out, from the derived manifest's `roots[].party_totals_left_out`, each entry named by the reading's spec id in the order of the ref's roots, and its repr's line reads `Party totals left out of the readings`. The report is empty against a server older than it and when DFO could not read a reading.
- `docs/datasets.md` says when the default keeps a total. A role the spec binds keeps what it binds. On an open role the cut judges each series, the frame's columns other than the role, `time_period` and provenance, in each period: a period with single parties drops its totals, a period of only totals keeps its whole ones, and with none whole it drops them when the series has single parties in another period and keeps them otherwise.
- `entities=` on `fetch` and `answer.fetch` takes the `Match` and `Entity` objects `dfo.lookup.resolve` returns beside dcids, as in `entities={"destination": dfo.lookup.resolve("Nigeria")}`. A match that is not `resolved` raises `InvalidRequestError` (`match_unresolved`) before any request, printing the match and a candidate to pass instead. An awaitable in its place, such as `client.lookup.resolve(...)` on an `AsyncClient` without `await`, raises the same error and names the missing `await`.
- `docs/settings.md` lists every client option and environment variable with its default and which source wins.
- `data.to_bytes()` returns the Parquet file `data.save(path)` writes, and `dfo.load` reads those bytes, or any bytes-like object such as a `bytearray` or `memoryview`, as it reads a path.
- `data.from_cache` says whether a fetch read any member's file from the cache, and `data.fetched_at` when the data left the server, in UTC. The repr adds `Cached: read from the cache, fetched <time>` on a cache hit online, and the `Stale:` line under `DFO_OFFLINE=1` ends with the same time. The cache records each download's time under `fetched/sha256/<hex>`. A file 0.1.0 stored has none, and the repr says `fetch time unknown`. Eviction counts each time's bytes against the cache's size limit, removes it with its file, and removes a time whose file is gone.
- `data.question` holds the question of the answer a dataset was fetched from. `data.save` writes it under `dfo.question` and the fetch time under `dfo.fetched_at`, each a JSON string, and leaves out either key when it has no value. 0.1.0 ignores both keys, and `dfo.load` reads a file without them.
- `DFO_TIMEOUT`, in seconds, and `DFO_MAX_RETRIES` set `timeout` and `max_retries` for a `Client`, `AsyncClient` or `dfo.configure` built without them. An argument wins over the variable, and `timeout=None` leaves `DFO_TIMEOUT` unread so each call keeps its own default. A `DFO_TIMEOUT` other than digits with an optional fraction above 0, such as `300` or `2.5`, or a `DFO_MAX_RETRIES` other than digits from 0 to 10, raises `InvalidRequestError` (`invalid_argument`) when the client is built, naming the variable; the client behind `dfo.search` and `dfo.fetch` raises it at the first call, and `import dfo` reads neither. `dfo.types.FROM_ENVIRONMENT` is the default of both arguments, typed by `ClientTimeoutArgument` and `MaxRetriesArgument`.
- `dfo.types.CLIENT_TIMEOUT` is the default of every call's `timeout` and stands for the client's `timeout`, and `dfo.types.CallTimeout` types the argument, so a function that wraps a call can pass the default on.

### Changed

- A fetch posts the party-totals cut, its default, as `party_totals: false` in the build request's `options`, the spelling DFO's API takes from dfo-apis 133262c5e on, in place of `without_party_totals: true`. Release 0.2.0 needs a server that accepts `options.party_totals`, and the release workflow checks that production does before it publishes. A ref and a served rendering read in either spelling, and `data.ref` still writes `without_party_totals: true`, so 0.1.0 reads a ref this release writes. The client follows the API contract at spec-kit 7ef7f71.
- A citation source's licence reads from the served `licence` and `licence_url`, falling back to `license` and `license_url` when the response lacks them, as a response from a server before the rename does.
- A fetch reads `trace.party_totals_left_out` leniently. A trace that is no object or a member that is no list names no totals, and a malformed entry, such as one with an empty `role`, entity or period or a `kept_in` out of the contract's order, is dropped while the entries beside it still read. A manifest with a broken trace no longer raises `ContractError`.
- `dfo.load` refuses a file whose `dfo.format_version` disagrees with its `dfo.ref` (`2` exactly when the ref holds a derived member, `1` otherwise), a `dfo.party_totals_left_out` entry that names neither a dataset member of `dfo.ref` nor a root of a derived member, such as the derived member's own id, or lists them out of member then root order, and an entry with an empty `role`, entity or period or whose `kept_in` breaks the contract's shape or order, each as `FormatError` (`format_invalid`).
- `development-finance` installs beside any `httpx2` and `httpcore2` from 2.15 on. The transport builds its connection pools through their public interfaces, so a new minor release of either needs no new client release.
- `development-finance` needs `pyarrow>=21`, the first release whose `concat_tables` stacks members whose struct columns differ by field name, order or type.
- The `polars` extra needs `polars>=1.3`, the first release whose `DataFrame` reads a `Dataset` through the Arrow stream. A CI job installs every floor and runs the suite.
- **Breaking:** `timeout` on `Client`, `AsyncClient`, `dfo.configure` and `with_options` bounds each whole call, retries, waits and downloads included, for every call that passes no `timeout` of its own, and defaults to None, which leaves `search` at 60 seconds and `fetch` and `dfo.analysis` waiting as long as the server takes. The bound on one request attempt, which `timeout` set in 0.1.0, is the new `attempt_timeout`, 30 seconds by default. `Client(timeout=60)` from 0.1.0 code now stops each call after 60 seconds with `DfoTimeoutError` (`deadline_exceeded`). Pass `attempt_timeout=60` for the old meaning. A call's `timeout=None` still waits without a bound, and a timeout's message names the option that ran out.
- **Breaking:** `with_options(timeout=None)` returns a copy with no client bound on its calls, so each call keeps its own default. In 0.1.0 it kept the client's `timeout`. Leave `timeout` out to keep it now.
- **Breaking:** `Client.fetch`, `AsyncClient.fetch` and `dfo.fetch` carry one overload per target, so a type checker rejects an option the target ignores: `window`, `entities`, `facets`, `layout` and `party_totals` go with a `Spec` only, `latest` with a `DatasetRef` or `Recipe`, `progress` with anything but a `Recipe`, and `as_of` and `slot` with a share. A last overload takes a target typed as the union of all four with the options every target takes, `strict` and `timeout`. The run-time checks stay.
- **Breaking:** the platform default cache root is `dfo` in place of `readerkit`: `~/.cache/dfo` on Linux (or `$XDG_CACHE_HOME/dfo`), `~/Library/Caches/dfo` on macOS, and `%LOCALAPPDATA%\dfo\Cache` on Windows. With no `cache_dir`, `DFO_CACHE_DIR` or `BBLOCKS_CACHE_DIR`, a file or manifest missing from the cache is read from the same path under the 0.1 `readerkit` root when it passes the checks a cached entry passes, and copied in without a fetch time, so an offline fetch of something 0.1 downloaded still loads. The client never writes, evicts or clears the `readerkit` directory, and `dfo cache clear` empties the `dfo` one only.
- **Breaking:** error messages end with `[<code> trace=<id>]`, or `[<code>]` with no trace id, in place of `[<code> · trace <id>]`. The bracket text can change in any release. `error.code` and `error.trace_id` are the stable fields.
- **Breaking:** `dfo.errors.FormatError` and `dfo.errors.BundleError` derive from `DfoError` as well as `ValueError`, so `except DfoError` around `dfo.load(path)` catches them. Their `code` is None and `reason` names the fault. `dfo.load` of a saved file from a later format version raises `FormatError` at `/dfo.format_version`.
- **Breaking:** calls that raised a bare `ValueError` or `TypeError` for bad input raise a `DfoError` subclass with a client code, which is still a `ValueError` or `TypeError` where it was one, so `except ValueError` and `except TypeError` keep catching it. `DatasetRef(...)` with a member of the wrong kind (also a `TypeError`) or of mixed kinds, `BundleLimits` with a limit that is not a positive integer, `data.ref` on a bundle opened from a folder, and a proxy URL whose scheme is not http, https, socks5 or socks5h raise `InvalidRequestError` (`invalid_argument`); `ApiKey("")` raises `AuthenticationError` (`key_empty`), and `ApiKey` of a non-string (also a `TypeError`) or a key the credentials file cannot store raises it with `key_invalid`; `dfo.auth.login()` with no key raises `AuthenticationError` (`key_empty`), and over a broken credentials file with `credentials_file_invalid`; `RawResponse.json()` on a body that is not JSON raises `ContractError` (`unexpected_response`).
- **Breaking:** `dfo.errors.FormatError` and `dfo.errors.BundleError` derive from `DfoError` as well as `ValueError`, so `except DfoError` around `dfo.load(path)` catches them. Their `code` equals `reason`, which names the fault. `dfo.load` of a saved file from a later format version raises `FormatError` at `/dfo.format_version`.
- **Breaking:** calls that raised a bare `ValueError` or `TypeError` for bad input raise a `DfoError` subclass with a client code, which is still a `ValueError` or `TypeError` where it was one, so `except ValueError` and `except TypeError` keep catching it. `DatasetRef(...)` with a member of the wrong kind (also a `TypeError`) or of mixed kinds, and `BundleLimits` with a limit that is not a positive integer, raise `InvalidRequestError` (`invalid_argument`); `data.ref` on a bundle opened from a folder raises it with `ref_unavailable`, and a proxy URL whose scheme is not http, https, socks5 or socks5h with `proxy_invalid`; `ApiKey("")` raises `AuthenticationError` (`key_empty`), and `ApiKey` of a non-string (also a `TypeError`) or a key the credentials file cannot store raises it with `key_invalid`; `dfo.auth.login()` with no key raises `AuthenticationError` (`key_empty`), and over a broken credentials file with `credentials_file_invalid`; `RawResponse.json()` on a body that is not JSON raises `ContractError` (`unexpected_response`).

### Removed

Expand Down
2 changes: 1 addition & 1 deletion python/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -232,7 +232,7 @@ with dfo.Client(timeout=300.0, max_retries=3) as client:

Run `dfo auth login` to store a key, and `dfo auth status` to see which key the client sends. [Keys and credentials](docs/keys.md) covers the lookup order.

Every error derives from `dfo.errors.DfoError` and carries a `code`, None for `FormatError` and `BundleError`, which carry a `reason`. The message ends with `[code]`, or `[code trace=id]` when a response carried one. The bracket text can change between releases. `error.code` and `error.trace_id` are the stable fields. A refusal names the fix, such as the call that settles an incomplete answer. [Errors](docs/errors.md) lists every class by what you do next. The types answers and datasets contain live in `dfo.types`.
Every error derives from `dfo.errors.DfoError` and carries a `code`. On `FormatError` and `BundleError` the `code` equals their `reason`. The message ends with `[code]`, or `[code trace=id]` when a response carried one. The bracket text can change between releases. `error.code` and `error.trace_id` are the stable fields. A refusal names the fix, such as the call that settles an incomplete answer. [Errors](docs/errors.md) lists every class by what you do next. The types answers and datasets contain live in `dfo.types`.

## Limits the client enforces

Expand Down
2 changes: 1 addition & 1 deletion python/docs/cache.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ The cache records the time beside each file it downloads. A file stored by relea

## Where the cache lives

The cache root comes from `cache_dir=`, then `DFO_CACHE_DIR`, then `BBLOCKS_CACHE_DIR`, then the platform's cache directory, such as `~/.cache/readerkit` on Linux. [Settings](settings.md#cache-directory) gives each platform's directory.
The cache root comes from `cache_dir=`, then `DFO_CACHE_DIR`, then `BBLOCKS_CACHE_DIR`, then the platform's cache directory, such as `~/.cache/dfo` on Linux. [Settings](settings.md#cache-directory) gives each platform's directory. Under the platform default, the client also reads the files and manifests release 0.1 cached under `readerkit`, such as `~/.cache/readerkit` on Linux, so an offline fetch of something 0.1 downloaded still loads. It copies each one it reads into its own cache and leaves the `readerkit` directory as it found it.

The entries sit under `<root>/v1/dfo/cache1`. After the fetch above, the cache holds one file and its manifest:

Expand Down
2 changes: 2 additions & 0 deletions python/docs/compute.md
Original file line number Diff line number Diff line change
Expand Up @@ -267,6 +267,8 @@ A `Dataset` an operation returns has one `DerivedMember` in `members`, and its r

`data.left_out` is a tuple of `dfo.types.LeftOutSeries`, empty when no step dropped a series.

`data.party_totals_left_out` holds the group totals each reading's read left out, as [Sum a column](datasets.md#sum-a-column) describes, each entry named by the reading's spec id, in the order of the recipe's readings. The repr counts them on a `Party totals left out of the readings` line. The tuple is empty when the readings left nothing out, against a server older than this report, and when DFO could not read a reading.

`dfo.fetch(ref)` reads a derived ref's exact version, and `dfo.fetch(ref, latest=True)` computes its recipe again on the newest data. A ref of several derived members raises `InvalidRequestError` before any request.

### `dfo.pin`
Expand Down
Loading
Loading