Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -165,9 +165,9 @@ jobs:
uv venv --python 3.12 "$VIRTUAL_ENV"
uv export --frozen --only-group dev --no-emit-project --no-hashes \
--output-file "$RUNNER_TEMP/dev-lock.txt" > /dev/null
grep -iE '^(pytest|pytest-cov|pyright|duckdb)==' "$RUNNER_TEMP/dev-lock.txt" \
grep -iE '^(pytest|pytest-cov|pyright|ty|duckdb)==' "$RUNNER_TEMP/dev-lock.txt" \
| sed 's/ .*//; s/;.*//' > "$RUNNER_TEMP/test-tools.txt"
test "$(wc -l < "$RUNNER_TEMP/test-tools.txt")" -eq 4
test "$(wc -l < "$RUNNER_TEMP/test-tools.txt")" -eq 5
uv pip install --resolution lowest-direct \
-r "$RUNNER_TEMP/test-tools.txt" -e ".[pandas,polars,xlsx]"

Expand Down
88 changes: 88 additions & 0 deletions .github/workflows/docs-examples.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
name: Docs examples

# scripts/check_docs_examples.py runs every python block in README.md,
# python/README.md and python/docs/*.md, and every ts block in
# typescript/README.md and typescript/docs/*.md, against the dev API through the
# dev website's /api/v1 proxy, which needs no ID token. One job per language. The
# Markdown report lists each block's outcome, the warnings each passed Python
# block showed, and each failure; the job writes it to the run summary and
# uploads it.

on:
schedule:
- cron: "47 5 * * *"
workflow_dispatch:

permissions:
contents: read

concurrency:
group: docs-examples
cancel-in-progress: false

jobs:
docs-examples:
name: docs examples (${{ matrix.language }})
runs-on: ubuntu-latest
# The script bounds each Python file and each TypeScript program at 1,800 s.
timeout-minutes: 60
strategy:
fail-fast: false
matrix:
language: [python, typescript]
env:
DFO_BASE_URL: https://dfo-dev-website-staging-zms5lj4zhq-nn.a.run.app/api/v1
steps:
- uses: actions/checkout@v7

# The script runs under the python project's environment for both languages.
- name: Install uv
uses: astral-sh/setup-uv@v10.0.0

- name: Set up Python
run: uv python install 3.12

- name: Install Python dependencies
working-directory: python
run: uv sync --frozen --python 3.12

# Reads the pnpm version from typescript/package.json's packageManager.
- name: Install pnpm
if: matrix.language == 'typescript'
uses: pnpm/action-setup@v6
with:
package_json_file: typescript/package.json

- name: Set up Node
if: matrix.language == 'typescript'
uses: actions/setup-node@v7
with:
node-version: "24"
cache: pnpm
cache-dependency-path: typescript/pnpm-lock.yaml

- name: Install TypeScript dependencies
if: matrix.language == 'typescript'
working-directory: typescript
run: pnpm install --frozen-lockfile

- name: Run the docs examples
working-directory: python
run: >-
uv run --no-sync python ../scripts/check_docs_examples.py
--language ${{ matrix.language }}
--report "$RUNNER_TEMP/docs-check-${{ matrix.language }}.md"

- name: Write the report to the run summary
if: ${{ !cancelled() }}
run: |
report="$RUNNER_TEMP/docs-check-${{ matrix.language }}.md"
if [ -f "$report" ]; then cat "$report" >> "$GITHUB_STEP_SUMMARY"; fi

- name: Upload the report
if: ${{ !cancelled() }}
uses: actions/upload-artifact@v7
with:
name: docs-check-${{ matrix.language }}
path: ${{ runner.temp }}/docs-check-${{ matrix.language }}.md
if-no-files-found: warn
14 changes: 11 additions & 3 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ Both packages group their modules into the same seven layers. A module imports a
| core | `dfo/_core/` | `src/core/` | errors and warnings, refusal mapping, the public data classes, wire parsing and the DatasetRef format |
| generated | `dfo/_generated/` | `src/generated/` | literals and types the generators write from the contract |

The public names sit above every layer. In Python they are `dfo/__init__.py` and the task modules beside it, `dfo/errors.py`, `dfo/types.py`, `dfo/lookup.py` and `dfo/auth.py`. Each layer's `__init__.py` holds only a docstring. In TypeScript they are the entry points at the `src/` root (`index.ts`, `index.node.ts`, `wire.ts` and `arrow.ts`), which re-export from the layers. `_version.py` and `src/version.ts` hold the version constant and sit below every layer. The sync twin of each Python async module, under `dfo/_sync/`, sits in the layer of its source.
The public names sit above every layer. In Python they are `dfo/__init__.py` and the task modules beside it, `dfo/errors.py`, `dfo/types.py`, `dfo/lookup.py` and `dfo/auth.py`. Each layer's `__init__.py` holds only a docstring. In TypeScript they are the entry points at the `src/` root (`index.ts`, `index.node.ts`, `types.ts`, `errors.ts`, `wire.ts` and `arrow.ts`), which re-export from the layers. `_version.py` and `src/version.ts` hold the version constant and sit below every layer. The sync twin of each Python async module, under `dfo/_sync/`, sits in the layer of its source.

Two checks enforce the rule, and each fails on a runtime import from a higher layer:

Expand Down Expand Up @@ -131,7 +131,15 @@ The async modules are the source of both clients, and `scripts/gen_sync.py` deri

## Docs examples

`scripts/check_docs_examples.py` runs every `python` block in `README.md`, `python/README.md`, and `python/docs/*.md`, and every `ts` or `typescript` block in `typescript/README.md` and `typescript/docs/*.md`, against the API `DFO_BASE_URL` names. It needs the network and spends Search calls, so CI leaves it out. Run it against dev after a docs change. The dev API is private, so start a local proxy that signs each request as your gcloud account, which needs `run.routes.invoke` on the service:
`scripts/check_docs_examples.py` runs every `python` block in `README.md`, `python/README.md`, and `python/docs/*.md`, and every `ts` or `typescript` block in `typescript/README.md` and `typescript/docs/*.md`, against the API `DFO_BASE_URL` names. It needs the network and spends Search calls, so pull requests do not run it. `.github/workflows/docs-examples.yml` runs it every night, and on demand from the Actions tab, against the dev API through the dev website's `/api/v1` proxy, one job per language, and attaches the Markdown report to the run. Run it yourself against dev after a docs change. The website proxy needs no credentials:

```bash
cd python
DFO_BASE_URL=https://dfo-dev-website-staging-zms5lj4zhq-nn.a.run.app/api/v1 \
uv run python ../scripts/check_docs_examples.py --report docs-check.md
```

To reach the dev API directly, which is private, start a local proxy that signs each request as your gcloud account, which needs `run.routes.invoke` on the service:

```bash
gcloud run services proxy dfo-dev-apis --region northamerica-northeast1 --project dev-fin-observatory --port 8083
Expand All @@ -145,7 +153,7 @@ DFO_BASE_URL=http://localhost:8083/api \
uv run python ../scripts/check_docs_examples.py --report docs-check.md
```

It prints a line per block, with `file:line`, `pass`, `fail` or `skip`, and the error or the reason, then the counts. A failure names the docs line the error came from. The script exits 1 when a block fails, a process fails after its last block, or none ran. Name files, such as `python/docs/errors.md`, to check only those; a name that matches no file stops the script with exit 2, and pass `--language python` or `--language typescript` to check one client. `--list` prints each block and how it will run, offline. `--timeout` bounds each Python file and each TypeScript program, 1,800 seconds by default. `python/tests/test_check_docs_examples.py` tests the parsing and planning offline.
It prints a line per block, with `file:line`, `pass`, `fail` or `skip`, and the error or the reason, then the counts. A failure names the docs line the error came from. A Python block that passes but shows a warning prints the warning with the docs line that caused it, and the report lists each such block under Warnings, so a recipe that warns shows up without failing the run. Read each one. A warning a reader would learn to ignore means the example should change, either to a question Search answers as asked or to `dfo.fetch(spec)` after the answer is read. The script exits 1 when a block fails, a process fails after its last block, or none ran. Name files, such as `python/docs/errors.md`, to check only those. A name that matches no file stops the script with exit 2, and pass `--language python` or `--language typescript` to check one client. `--list` prints each block and how it will run, offline. `--timeout` bounds each Python file and each TypeScript program, 1,800 seconds by default. `python/tests/test_check_docs_examples.py` tests the parsing and planning offline.

The Python blocks of a file run in order in one process and one namespace, as a notebook runs them, so a block can use the names an earlier block set. The TypeScript blocks of a file form programs. A block that holds an `import` statement starts a program, and each block after it without one continues it. The script joins a program's blocks into one module, so a later block sees what the earlier ones declared. Imports of `@developmentfinance/client` load the TypeScript source, so no build is needed, but Node 22.18 or later and `pnpm --dir typescript install` are.

Expand Down
17 changes: 17 additions & 0 deletions python/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,15 +9,32 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
### Added

- `data.party_totals_left_out` names the group totals a fetch left out of each member, as `dfo.types.PartyTotalsLeftOut` entries of spec id, party role, dcids and provenance, and the repr counts them. `data.save` writes them under a new `dfo.party_totals_left_out` key, which 0.1.0 ignores, and `dfo.load` reads a file without it.
- `entities=` on `fetch` and `answer.fetch` takes the `Match` and `Entity` objects `dfo.lookup.resolve` returns beside dcids, as in `entities={"destination": dfo.lookup.resolve("Nigeria")}`. A match that is not `resolved` raises `InvalidRequestError` (`invalid_argument`) before any request, printing the match and a candidate to pass instead. An awaitable in its place, such as `client.lookup.resolve(...)` on an `AsyncClient` without `await`, raises the same error and names the missing `await`.
- `docs/settings.md` lists every client option and environment variable with its default and which source wins.
- `data.to_bytes()` returns the Parquet file `data.save(path)` writes, and `dfo.load` reads those bytes, or any bytes-like object such as a `bytearray` or `memoryview`, as it reads a path.
- `data.from_cache` says whether a fetch read any member's file from the cache, and `data.fetched_at` when the data left the server, in UTC. The repr adds `Cached: read from the cache, fetched <time>` on a cache hit online, and the `Stale:` line under `DFO_OFFLINE=1` ends with the same time. The cache records each download's time under `fetched/sha256/<hex>`. A file 0.1.0 stored has none, and the repr says `fetch time unknown`. Eviction counts each time's bytes against the cache's size limit, removes it with its file, and removes a time whose file is gone.
- `data.question` holds the question of the answer a dataset was fetched from. `data.save` writes it under `dfo.question` and the fetch time under `dfo.fetched_at`, each a JSON string, and leaves out either key when it has no value. 0.1.0 ignores both keys, and `dfo.load` reads a file without them.
- `dfo.types.CLIENT_TIMEOUT` is the default of every call's `timeout` and stands for the client's `timeout`, and `dfo.types.CallTimeout` types the argument, so a function that wraps a call can pass the default on.

### Changed

- `development-finance` installs beside any `httpx2` and `httpcore2` from 2.15 on. The transport builds its connection pools through their public interfaces, so a new minor release of either needs no new client release.
- `development-finance` needs `pyarrow>=21`, the first release whose `concat_tables` stacks members whose struct columns differ by field name, order or type.
- The `polars` extra needs `polars>=1.3`, the first release whose `DataFrame` reads a `Dataset` through the Arrow stream. A CI job installs every floor and runs the suite.
- **Breaking:** `timeout` on `Client`, `AsyncClient`, `dfo.configure` and `with_options` bounds each whole call, retries, waits and downloads included, for every call that passes no `timeout` of its own, and defaults to None, which leaves `search` at 60 seconds and `fetch` and `dfo.analysis` waiting as long as the server takes. The bound on one request attempt, which `timeout` set in 0.1.0, is the new `attempt_timeout`, 30 seconds by default. `Client(timeout=60)` from 0.1.0 code now stops each call after 60 seconds with `DfoTimeoutError` (`deadline_exceeded`). Pass `attempt_timeout=60` for the old meaning. A call's `timeout=None` still waits without a bound, and a timeout's message names the option that ran out.
- **Breaking:** `with_options(timeout=None)` returns a copy with no client bound on its calls, so each call keeps its own default. In 0.1.0 it kept the client's `timeout`. Leave `timeout` out to keep it now.
- **Breaking:** `Client.fetch`, `AsyncClient.fetch` and `dfo.fetch` carry one overload per target, so a type checker rejects an option the target ignores: `window`, `entities`, `facets`, `layout` and `party_totals` go with a `Spec` only, `latest` with a `DatasetRef` or `Recipe`, `progress` with anything but a `Recipe`, and `as_of` and `slot` with a share. A last overload takes a target typed as the union of all four with the options every target takes, `strict` and `timeout`. The run-time checks stay.
- **Breaking:** error messages end with `[<code> trace=<id>]`, or `[<code>]` with no trace id, in place of `[<code> · trace <id>]`. The bracket text can change in any release. `error.code` and `error.trace_id` are the stable fields.
- **Breaking:** `dfo.errors.FormatError` and `dfo.errors.BundleError` derive from `DfoError` as well as `ValueError`, so `except DfoError` around `dfo.load(path)` catches them. Their `code` is None and `reason` names the fault. `dfo.load` of a saved file from a later format version raises `FormatError` at `/dfo.format_version`.
- **Breaking:** calls that raised a bare `ValueError` or `TypeError` for bad input raise a `DfoError` subclass with a client code, which is still a `ValueError` or `TypeError` where it was one, so `except ValueError` and `except TypeError` keep catching it. `DatasetRef(...)` with a member of the wrong kind (also a `TypeError`) or of mixed kinds, `BundleLimits` with a limit that is not a positive integer, `data.ref` on a bundle opened from a folder, and a proxy URL whose scheme is not http, https, socks5 or socks5h raise `InvalidRequestError` (`invalid_argument`); `ApiKey("")` raises `AuthenticationError` (`key_empty`), and `ApiKey` of a non-string (also a `TypeError`) or a key the credentials file cannot store raises it with `key_invalid`; `dfo.auth.login()` with no key raises `AuthenticationError` (`key_empty`), and over a broken credentials file with `credentials_file_invalid`; `RawResponse.json()` on a body that is not JSON raises `ContractError` (`unexpected_response`).

### Removed

- **Breaking:** `Answer.choose` takes no second positional argument. The earlier `choose(part, option)` form now fails as any call with too many arguments does. Pass `choose(option, part=part)`.

### Fixed

- `dfo.load` of a damaged Parquet file or ZIP raises `BundleError` (`not_a_bundle`), and of a bundle whose XLSX is damaged `FormatError`, each with pyarrow's or `zipfile`'s error as its cause. Both used to escape as pyarrow and `zipfile` errors. A read that fails at the disk, such as `EIO`, still raises its own `OSError`.
- `answer.timed_out` is true when a part's own no-data reason is `timed_out`, as well as when Search's diagnostics say so, and `Answer.from_json` works it out again.

## [0.1.0] - 2026-11-13
Expand Down
Loading
Loading