diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 3a4490a..50c8d67 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -165,9 +165,9 @@ jobs: uv venv --python 3.12 "$VIRTUAL_ENV" uv export --frozen --only-group dev --no-emit-project --no-hashes \ --output-file "$RUNNER_TEMP/dev-lock.txt" > /dev/null - grep -iE '^(pytest|pytest-cov|pyright|duckdb)==' "$RUNNER_TEMP/dev-lock.txt" \ + grep -iE '^(pytest|pytest-cov|pyright|ty|duckdb)==' "$RUNNER_TEMP/dev-lock.txt" \ | sed 's/ .*//; s/;.*//' > "$RUNNER_TEMP/test-tools.txt" - test "$(wc -l < "$RUNNER_TEMP/test-tools.txt")" -eq 4 + test "$(wc -l < "$RUNNER_TEMP/test-tools.txt")" -eq 5 uv pip install --resolution lowest-direct \ -r "$RUNNER_TEMP/test-tools.txt" -e ".[pandas,polars,xlsx]" diff --git a/.github/workflows/docs-examples.yml b/.github/workflows/docs-examples.yml new file mode 100644 index 0000000..25842b5 --- /dev/null +++ b/.github/workflows/docs-examples.yml @@ -0,0 +1,88 @@ +name: Docs examples + +# scripts/check_docs_examples.py runs every python block in README.md, +# python/README.md and python/docs/*.md, and every ts block in +# typescript/README.md and typescript/docs/*.md, against the dev API through the +# dev website's /api/v1 proxy, which needs no ID token. One job per language. The +# Markdown report lists each block's outcome, the warnings each passed Python +# block showed, and each failure; the job writes it to the run summary and +# uploads it. + +on: + schedule: + - cron: "47 5 * * *" + workflow_dispatch: + +permissions: + contents: read + +concurrency: + group: docs-examples + cancel-in-progress: false + +jobs: + docs-examples: + name: docs examples (${{ matrix.language }}) + runs-on: ubuntu-latest + # The script bounds each Python file and each TypeScript program at 1,800 s. + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + language: [python, typescript] + env: + DFO_BASE_URL: https://dfo-dev-website-staging-zms5lj4zhq-nn.a.run.app/api/v1 + steps: + - uses: actions/checkout@v7 + + # The script runs under the python project's environment for both languages. + - name: Install uv + uses: astral-sh/setup-uv@v10.0.0 + + - name: Set up Python + run: uv python install 3.12 + + - name: Install Python dependencies + working-directory: python + run: uv sync --frozen --python 3.12 + + # Reads the pnpm version from typescript/package.json's packageManager. + - name: Install pnpm + if: matrix.language == 'typescript' + uses: pnpm/action-setup@v6 + with: + package_json_file: typescript/package.json + + - name: Set up Node + if: matrix.language == 'typescript' + uses: actions/setup-node@v7 + with: + node-version: "24" + cache: pnpm + cache-dependency-path: typescript/pnpm-lock.yaml + + - name: Install TypeScript dependencies + if: matrix.language == 'typescript' + working-directory: typescript + run: pnpm install --frozen-lockfile + + - name: Run the docs examples + working-directory: python + run: >- + uv run --no-sync python ../scripts/check_docs_examples.py + --language ${{ matrix.language }} + --report "$RUNNER_TEMP/docs-check-${{ matrix.language }}.md" + + - name: Write the report to the run summary + if: ${{ !cancelled() }} + run: | + report="$RUNNER_TEMP/docs-check-${{ matrix.language }}.md" + if [ -f "$report" ]; then cat "$report" >> "$GITHUB_STEP_SUMMARY"; fi + + - name: Upload the report + if: ${{ !cancelled() }} + uses: actions/upload-artifact@v7 + with: + name: docs-check-${{ matrix.language }} + path: ${{ runner.temp }}/docs-check-${{ matrix.language }}.md + if-no-files-found: warn diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index ea07c40..2b90beb 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -23,7 +23,7 @@ Both packages group their modules into the same seven layers. A module imports a | core | `dfo/_core/` | `src/core/` | errors and warnings, refusal mapping, the public data classes, wire parsing and the DatasetRef format | | generated | `dfo/_generated/` | `src/generated/` | literals and types the generators write from the contract | -The public names sit above every layer. In Python they are `dfo/__init__.py` and the task modules beside it, `dfo/errors.py`, `dfo/types.py`, `dfo/lookup.py` and `dfo/auth.py`. Each layer's `__init__.py` holds only a docstring. In TypeScript they are the entry points at the `src/` root (`index.ts`, `index.node.ts`, `wire.ts` and `arrow.ts`), which re-export from the layers. `_version.py` and `src/version.ts` hold the version constant and sit below every layer. The sync twin of each Python async module, under `dfo/_sync/`, sits in the layer of its source. +The public names sit above every layer. In Python they are `dfo/__init__.py` and the task modules beside it, `dfo/errors.py`, `dfo/types.py`, `dfo/lookup.py` and `dfo/auth.py`. Each layer's `__init__.py` holds only a docstring. In TypeScript they are the entry points at the `src/` root (`index.ts`, `index.node.ts`, `types.ts`, `errors.ts`, `wire.ts` and `arrow.ts`), which re-export from the layers. `_version.py` and `src/version.ts` hold the version constant and sit below every layer. The sync twin of each Python async module, under `dfo/_sync/`, sits in the layer of its source. Two checks enforce the rule, and each fails on a runtime import from a higher layer: @@ -131,7 +131,15 @@ The async modules are the source of both clients, and `scripts/gen_sync.py` deri ## Docs examples -`scripts/check_docs_examples.py` runs every `python` block in `README.md`, `python/README.md`, and `python/docs/*.md`, and every `ts` or `typescript` block in `typescript/README.md` and `typescript/docs/*.md`, against the API `DFO_BASE_URL` names. It needs the network and spends Search calls, so CI leaves it out. Run it against dev after a docs change. The dev API is private, so start a local proxy that signs each request as your gcloud account, which needs `run.routes.invoke` on the service: +`scripts/check_docs_examples.py` runs every `python` block in `README.md`, `python/README.md`, and `python/docs/*.md`, and every `ts` or `typescript` block in `typescript/README.md` and `typescript/docs/*.md`, against the API `DFO_BASE_URL` names. It needs the network and spends Search calls, so pull requests do not run it. `.github/workflows/docs-examples.yml` runs it every night, and on demand from the Actions tab, against the dev API through the dev website's `/api/v1` proxy, one job per language, and attaches the Markdown report to the run. Run it yourself against dev after a docs change. The website proxy needs no credentials: + +```bash +cd python +DFO_BASE_URL=https://dfo-dev-website-staging-zms5lj4zhq-nn.a.run.app/api/v1 \ + uv run python ../scripts/check_docs_examples.py --report docs-check.md +``` + +To reach the dev API directly, which is private, start a local proxy that signs each request as your gcloud account, which needs `run.routes.invoke` on the service: ```bash gcloud run services proxy dfo-dev-apis --region northamerica-northeast1 --project dev-fin-observatory --port 8083 @@ -145,7 +153,7 @@ DFO_BASE_URL=http://localhost:8083/api \ uv run python ../scripts/check_docs_examples.py --report docs-check.md ``` -It prints a line per block, with `file:line`, `pass`, `fail` or `skip`, and the error or the reason, then the counts. A failure names the docs line the error came from. The script exits 1 when a block fails, a process fails after its last block, or none ran. Name files, such as `python/docs/errors.md`, to check only those; a name that matches no file stops the script with exit 2, and pass `--language python` or `--language typescript` to check one client. `--list` prints each block and how it will run, offline. `--timeout` bounds each Python file and each TypeScript program, 1,800 seconds by default. `python/tests/test_check_docs_examples.py` tests the parsing and planning offline. +It prints a line per block, with `file:line`, `pass`, `fail` or `skip`, and the error or the reason, then the counts. A failure names the docs line the error came from. A Python block that passes but shows a warning prints the warning with the docs line that caused it, and the report lists each such block under Warnings, so a recipe that warns shows up without failing the run. Read each one. A warning a reader would learn to ignore means the example should change, either to a question Search answers as asked or to `dfo.fetch(spec)` after the answer is read. The script exits 1 when a block fails, a process fails after its last block, or none ran. Name files, such as `python/docs/errors.md`, to check only those. A name that matches no file stops the script with exit 2, and pass `--language python` or `--language typescript` to check one client. `--list` prints each block and how it will run, offline. `--timeout` bounds each Python file and each TypeScript program, 1,800 seconds by default. `python/tests/test_check_docs_examples.py` tests the parsing and planning offline. The Python blocks of a file run in order in one process and one namespace, as a notebook runs them, so a block can use the names an earlier block set. The TypeScript blocks of a file form programs. A block that holds an `import` statement starts a program, and each block after it without one continues it. The script joins a program's blocks into one module, so a later block sees what the earlier ones declared. Imports of `@developmentfinance/client` load the TypeScript source, so no build is needed, but Node 22.18 or later and `pnpm --dir typescript install` are. diff --git a/python/CHANGELOG.md b/python/CHANGELOG.md index 5717740..4668897 100644 --- a/python/CHANGELOG.md +++ b/python/CHANGELOG.md @@ -9,15 +9,32 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), ### Added - `data.party_totals_left_out` names the group totals a fetch left out of each member, as `dfo.types.PartyTotalsLeftOut` entries of spec id, party role, dcids and provenance, and the repr counts them. `data.save` writes them under a new `dfo.party_totals_left_out` key, which 0.1.0 ignores, and `dfo.load` reads a file without it. +- `entities=` on `fetch` and `answer.fetch` takes the `Match` and `Entity` objects `dfo.lookup.resolve` returns beside dcids, as in `entities={"destination": dfo.lookup.resolve("Nigeria")}`. A match that is not `resolved` raises `InvalidRequestError` (`invalid_argument`) before any request, printing the match and a candidate to pass instead. An awaitable in its place, such as `client.lookup.resolve(...)` on an `AsyncClient` without `await`, raises the same error and names the missing `await`. +- `docs/settings.md` lists every client option and environment variable with its default and which source wins. +- `data.to_bytes()` returns the Parquet file `data.save(path)` writes, and `dfo.load` reads those bytes, or any bytes-like object such as a `bytearray` or `memoryview`, as it reads a path. +- `data.from_cache` says whether a fetch read any member's file from the cache, and `data.fetched_at` when the data left the server, in UTC. The repr adds `Cached: read from the cache, fetched