[Dev-iL]

The problem: a contributor, person or AI agent, can't tell from the repository which of CI's commands check a given change: each plugin's extras and installs are spread across .github/workflows/hamilton-main.yml, so contributors guess, run the wrong subset, or push and wait for CI.

What changes: scripts/verify, one command with a recipe per area under verification/, most running their CI step's commands or a slice of the main test step: affected names the recipes a change touches, and drive <recipe> runs one. It is optional, needs only Node 22.18 or newer, and adds no CI job. Nothing under hamilton/, tests/ or .github/ changes; pyproject.toml gains three source-release excludes, so CI runs here. Since the review, the macOS test failure and the sdist packaging are fixed.

Where this sits: split at the review's request into three stacked pull requests, one commit each:

1. Add a glossary, and fix the contributing and developer setup guides #1808, the glossary and documentation fixes;
2. Record eight foundational decisions, reconstructed from the codebase #1809, the decision records;
3. this pull request, commit d58764ab.

They merge bottom-up, each rebased onto main after the one below lands; until then GitHub shows the lower commits here too.

Shown working: check, doctor, the lint and verify-tool pass at d58764a; every recipe was driven and passed, except two left to CI; the checks for both fixes fail when a fix is undone; CI passed (Unit Tests, Contrib Test Workflow). Details under How I tested this.

Decisions (no decision record is reopened; reasons are folded below):

• Author's call: three stacked pull requests, one commit each, as the review asked.
• The recipes follow main's workflow, not the lockfile change in Commit uv.lock and install CI's test dependencies from it #1807.
• recipes-match-ci checks the recipes against the workflow.
• The recipes point to where versions are declared.
• ci-canary ships, not marked ask-first.
• The sdist excludes the three harness files by name.

Awaiting the author, raised in the review: whether to add a CI job for the harness.

Debt left: two recipes didn't run here, polars ran on a stand-in, the recipes copy CI's commands, two parsers read the recipes, the redirect packages and contrib/ have no area recipe, and two workflow issues sit outside this change; all six are at the end.

Before and after (an illustrative example). Task: check a change to the polars plugin before pushing it.

• Before: you search hamilton-main.yml for the steps that touch polars, copy each one's uv sync extras and uv pip install lines into a shell, and hope you found them all, or you push and wait for CI.
• After: scripts/verify affected names h_polars, plugin-extensions and suite; scripts/verify drive h_polars runs the same commands as CI's polars steps, says which scenario failed, and keeps its output.

Decisions, with their reasons

• Author's call: split the change into three stacked pull requests, one commit each, as the review asked, with this one keeping the harness and its review thread. Its branch was force-pushed once, to carry the stack.
• The recipes follow main's workflow as it is, with its uv pip install steps and no lockfile, rather than the lockfile change in Commit uv.lock and install CI's test dependencies from it #1807. Whichever of the two lands second updates the recipes.
• A script checks that the recipes still match the workflow (recipes-match-ci), including the Pythons each step's if: runs it on, rather than relying on someone remembering to copy each change.
• The recipes' prose points to where the workflow and tests/conftest.py declare versions (a step's if:, the test job's matrix, collect_ignore) rather than restating them. verify doctor reads the matrix through the drift check's own parser.
• The ci-canary rehearsal ships with the rest. It needs Docker and act, but it posts only to a local stub and reads one public run without credentials, so it is not marked ask-first.
• The source release excludes the three harness files by name. A scripts/verify* glob would also drop verify_apache_artifacts.py and the other release files that share the prefix.

What changes, in full

• scripts/verify is a launcher that checks the Node version and starts scripts/verify.mts, a dependency-free TypeScript file Node runs directly. It is adapted from manifest-dev 14.20.1's default harness, which ships no tests. Its header lists what was changed for Hamilton: the license comment before each recipe's front matter, upstream/main before origin as the default base, front-matter errors that name their file, a refusal of front-matter keys it does not read, and the exit-code help.
• verification/ holds:
  • harness.md: the settings and what every recipe assumes.
  • 25 recipes:
    • suite, with one scenario per workflow step.
    • Area recipes for the core modules and each plugin, each running its step's commands or a scoped part of the main test step.
    • ci-canary, which rehearses the workflow's daily schedule, fork guard and Slack report under act.
    • verify-tool, which checks the harness itself: its settings and recipes, doctor, its Node tests, the drift check and its tests, and the source release.
  • scripts/ci_canary_rehearsal.py: the act rehearsal. It stubs every job but the Slack notifier, compares the jobs that ran with the expected set exactly, and treats a missing Docker daemon, network or jq as unreachable rather than failed.
  • scripts/check_recipes_match_ci.py and its 51 tests.
  • scripts/doctor.sh, which verify doctor runs.
  • no evidence files: verify creates verification/evidence/ with its own .gitignore on first use, so drive output stays on the machine that ran it.
• scripts/verify.test.mts: 22 node --test tests of check, affected and drive on temporary fixture projects, each reached through a symbolic link.
• pyproject.toml: [tool.flit.sdist] excludes scripts/verify, scripts/verify.mts and scripts/verify.test.mts.
• CONTRIBUTING.md gains the scripts/verify bullet under "Finding your way around", and writeups/developer_setup.md one paragraph mentioning it as optional.

Harness issues also fixed while answering the review

• h_polars didn't cover the polars 2 readers main added, so verify affected named nothing for a change to them, and its list of loader kinds left out ndjson.
• verify ignored front-matter keys it doesn't read, so a recipe that misspelled ask-first drove without asking.
• The drift check compared a step's runner.os case-sensitively, accepted a Python version check as the check for a step's system packages, and crashed on a step whose run holds only apt lines, and its test of the committed tree accepted a count of zero scenarios.
• verify doctor passed a .venv on a Python CI doesn't test. It now fails it.
• The harness notes said a guarded scenario would pass having run nothing (not true since main removed the exit-5 hook), called every check adopted from the workflow without excepting verify-tool's own, and said nothing about the redirect packages' test or hamilton-contrib.yml, which now runs on pull requests.

How I tested this

All runs used Python 3.14 on Linux. The recipe drives ran at c91d9c5, this branch's head before the split; between it and d58764a only Markdown under writeups/adr/ differs, which no recipe reads apart from the lint, so the drives at c91d9c5 still apply; check, doctor, the lint and verify-tool were also run again at d58764a. openlineage (no PostgreSQL server here) and vaex (CI runs it on 3.11 and 3.12 only) stop as unreachable here; CI ran both on d58764a. This machine's CPU lacks AVX2, so polars ran on its compat runtime: a shim outside the repository installed polars-runtime-compat of the same polars version after each uv sync and uv pip install. CI runs the standard build.

check, doctor, the repository lint and verify-tool, at d58764a

$ scripts/verify check
check: ok
$ scripts/verify doctor
uv 0.13.5 (x86_64-unknown-linux-gnu)
no .venv yet: the first scenario creates it with the newest Python uv finds, which CI may not test (3.11 3.12 3.13 3.14 3.15); uv sync --python <one of those> picks one it does
doctor: ok
$ uv run prek run --all-files --hook-stage pre-merge-commit
# every hook passed
$ scripts/verify drive verify-tool
# all 6 scenarios passed: check, doctor, node-tests (22), recipes-match-ci, recipes-match-ci-tests (51), sdist

Recipes driven and their results, at c91d9c5

Each is scripts/verify drive <recipe> [scenario]; the drive keeps its full output on this machine under verification/evidence/. Polars ran on polars-runtime-compat of the same version, because this CPU lacks AVX2; dependencies were resolved fresh, as CI resolves them. At c91d9c5, .venv ran Python 3.14, and doctor reported "one CI tests (3.11 3.12 3.13 3.14 3.15)".

Drive	Result
suite static-checks	passed (prek, all hooks)
suite main	1679 passed, 8 skipped
suite integrations	13 passed, 1 xpassed
suite mcp	41 passed
suite pandas	13 passed
suite polars	11 passed
suite polars-1x	80 passed, 9 skipped, 39 xfailed
suite narwhals	2 passed
suite dask	38 passed
suite ray	3 passed
suite pyspark	99 passed
suite openlineage, openlineage	unreachable: no PostgreSQL server
suite vaex, h_vaex	unreachable: CI runs vaex on Python 3.11 and 3.12 only
caching	145 passed
cli	7 passed
data-quality	61 passed
driver	246 passed, 1 skipped
execution	59 passed, 1 skipped
experimental	8 passed
function-modifiers	435 passed
graph	306 passed
h_dask	38 passed
h_narwhals	2 passed
h_pandas	13 passed
h_polars	polars 11 passed; extensions 128 passed; polars-1x 80 passed, 9 skipped, 39 xfailed
h_ray	3 passed
h_spark	99 passed
io	58 passed
lifecycle	67 passed
mcp	41 passed
pandera	13 passed, 1 xpassed
plugin-extensions	273 passed, 6 skipped
registry	3 passed
verify-tool	all 6 scenarios passed: check; doctor; 22 Node tests; recipes-match-ci; its 51 tests; sdist
ci-canary	all 11 scenarios passed under act

The two fixes, at d58764a

The macOS test failure. Every fixture is reached through a symbolic link, as /var → /private/var is on macOS:

$ TMPDIR=<a directory reached through a symbolic link> node --test scripts/verify.test.mts
ℹ tests 22
ℹ pass 22
ℹ fail 0
# with the drive test's expectation put back to the path the test created:
ℹ pass 21
ℹ fail 1

The source release. uv build --sdist applies the same [tool.flit.sdist] rules as the release's flit build --no-use-vcs:

# file lists of the sdist built with the old exclude list, then with this branch's
$ comm -3 old.txt new.txt
scripts/verify
scripts/verify.mts
scripts/verify.test.mts
# 731 files before, 728 after
$ scripts/verify drive verify-tool sdist
# passed
# with the three exclude entries removed:
the sdist ships the files above without verification/ and .github/, which they read
# with a scripts/verify* glob in their place:
the sdist lost scripts/verify-sub-packages.md, which shares the scripts/verify prefix but is not the harness

The drift check fails when the recipes fall out of step

Each of the drift check's 51 tests plants one mismatch in a copy of the recipes or the workflow and asserts every line the check prints. Two planted changes in a scratch copy:

# the Test dask step gains -x, and Test ray's if: becomes "Matrix.Python-Version != '3.14' && matrix.python-version != '3.15'"
$ uv run --script verification/scripts/check_recipes_match_ci.py --root <copy>
suite.md: scenario dask: runs ['uv sync --group test --extra dask', 'uv run pytest plugin_tests/h_dask'] but step 'Test dask' runs ['uv sync --group test --extra dask', 'uv run pytest -x plugin_tests/h_dask'] (apart from prerequisite checks and NAME=value prefixes)
suite.md: scenario ray: runs on Python 3.11, 3.12, 3.13, 3.14, but step 'Test ray' runs on 3.11, 3.12, 3.13
recipes-match-ci: 2 mismatch(es)
# Test vaex's if: reads the matrix by index, a form the check does not read
suite.md: scenario vaex: step 'Test vaex' has an if: condition this check cannot read: "${{ matrix['python-version'] == '3.11' }}"
recipes-match-ci: 1 mismatch(es)

CI on this pull request

The pyproject.toml change puts this pull request inside hamilton-main.yml's pull_request filter, so the full matrix runs on it, and hamilton-contrib.yml runs too.

• On d58764a, the current head, both passed. Unit Tests ran the static checks and the tests on Python 3.11 to 3.15, including every polars step, Test openlineage on every version and Test vaex on 3.11 and 3.12, the two recipes that didn't run here; Contrib Test Workflow ran contrib/tests on Python 3.11 to 3.15. Notify Slack is skipped, as it is on every pull request.
• On c91d9c5, the head before the split, whose content differs only in writeups/adr/, both passed too: Unit Tests, Contrib Test Workflow.

Debt left

• Recipes that didn't run here, left to CI: openlineage (no PostgreSQL server here) and vaex (CI runs it on Python 3.11 and 3.12 only; the drives used 3.14). Both stop as unreachable here; CI ran both on d58764a, and they passed.
• Polars ran on its compat runtime here, through the shim described under How I tested this, because this CPU lacks AVX2. The polars version and API are the same, but the binary CI runs was not exercised on this machine; CI's polars steps passed with it on d58764a.
• The recipes copy their CI steps' commands rather than reading them from the workflow, so each workflow change needs a matching recipe edit. recipes-match-ci names every copy that drifts, but someone still makes the edit.
• Two parsers read the recipe files: scripts/verify.mts and the drift check each parse the front matter and scenarios their own way.
• The redirect packages and contrib/ have no area recipe. Only suite covers them, and no recipe adopts hamilton-contrib.yml; the harness notes say which test checks each.
• The workflow installs uv by piping an unpinned installer script to sh, and every job carries a hand-copied fork guard. Both are in .github/, which this pull request doesn't change. The canary rehearsal fails if a job is added without the guard, which limits the second.

[skrawcz]

AI-assisted preliminary review — findings must be fixed or technically refuted before human review.

The glossary and developer-guide cleanup are useful, and historical decision records could help contributors. The combined PR currently needs significant restructuring. It introduces 2,600+ lines of custom executable verification tooling plus 25 duplicated CI recipes, eight reconstructed ADRs marked Accepted, and unrelated documentation fixes in one 47-file change.

Concrete blockers reproduced during the AI-assisted review:

• No PR workflow runs for these changed paths.
• node --test scripts/verify.test.mts fails on macOS path canonicalization (19 passed, 1 failed: /tmp vs /private/tmp, likewise /var/folders).
• The Apache sdist ships scripts/verify* without verification/, so the released command has no settings/recipes and is incomplete.
• The reconstructed ADRs are marked Accepted even where rationale/alternatives were not recovered and no current public acceptance is cited.

Please split docs/glossary, proposed ADRs, and verification tooling. ADRs should remain proposed until publicly accepted with stable source links. If maintainers choose to own the harness, wire its tests into CI, make them cross-platform, and ship a complete runnable system or exclude it from the sdist. If keeping this combined approach, each point needs a technical/governance refutation.

[Dev-iL]

This was split into three stacked pull requests, one commit each:

1. Add a glossary, and fix the contributing and developer setup guides #1808, the glossary and documentation fixes;
2. Record eight foundational decisions, reconstructed from the codebase #1809, the decision records;
3. Add scripts/verify, a command that runs the checks CI runs, one recipe per area #1806 (this)