Skip to content

Print a per-component, per-target pass-rate summary at the end of each pytest run - #167

Draft
gaurav wants to merge 2 commits into
mainfrom
end-of-run-summary
Draft

gaurav wants to merge 2 commits into
mainfrom
end-of-run-summary

Conversation

@gaurav

@gaurav gaurav commented Nov 13, 2026

Copy link
Copy Markdown
Collaborator

A full run prints thousands of lines, so the header saying which targets and categories were tested scrolls away, and pytest's closing totals don't say how each service did on each target. This PR adds a "Babel Validation summary" block after pytest's own summary. It lists the targets and category filters, then a pass rate per component (tests/<dir>) and target, with the URL that was tested. When more than one --target is given, it also lists the tests whose outcome differs between targets.

=========================== Babel Validation summary ===========================
Targets: dev, ci-nameres-es; included categories: all; excluded categories: none
NodeNorm [dev] https://nodenormalization-sri.renci.org/: passed 3 of 4 run (75%) — 1 failed
NodeNorm [ci-nameres-es] https://nodenorm-es.ci.transltr.io/: passed 2 of 4 run (50%) — 2 failed
NameRes [ci-nameres-es] https://namelookup-es.ci.transltr.io/: passed 2 of 3 run (67%) — 1 xfailed
test_environment: passed 19 of 19 run (100%)
Outcomes differing across targets: 3
  nameres/test_nameres_api.py::test_openapi_json: dev=failed, ci-nameres-es=passed
  ...

Calls worth overruling

  • Pass rate = passed ÷ (everything except skipped). Rows filtered out by --category are skips, so they don't drag the percentage down; they're shown as a separate "N skipped". xfailed counts as run but not passed.
  • Outcomes are counted the way the dashboard counts them. summarize_run reuses classify_record, parse_nodeid and OUTCOME_SEVERITY from generate_report.py, applied to the same per-phase records that --report-jsonl writes. So a test counts once, as its worst phase, and the target is parsed out of the node ID. pytest_runtest_logreport now keeps those records in memory even when --report-jsonl isn't set.
  • The differing-outcomes list only compares targets a test actually ran on, and stops after 50 entries so it doesn't bring back the scrolling problem.

What's here

  • summarize_run in tests/_pytest_helpers.py, with unit tests in tests/unit/test_run_summary.py.
  • A pytest_terminal_summary hook in tests/conftest.py. It reads targets.ini directly rather than through get_target(): an unknown --target has already failed collection, and the summary shouldn't add a second traceback.
  • A paragraph in AGENTS.md describing the summary.

Verification

The new unit tests and the rest of the unit suite (about 280 tests) pass on 30a53b9. Live runs printed the expected summary for --target dev --target prod, for -n 4 --target dev --target ci-nameres-es (so xdist workers' reports reach it), and for a --category "Unit Tests" Google Sheet run, where the filtered rows showed up as skips.

Nothing is blocking, and nothing was deferred.

🤖 Generated with Claude Code

gaurav and others added 2 commits November 13, 2026 17:21
Takes the raw per-phase records conftest already builds for --report-jsonl and
returns summary lines: targets and category filters, a pass rate per component
(tests/<dir>) and target with the service URL, and, with several targets, the
tests whose outcome differs between them.

It reuses classify_record, parse_nodeid and OUTCOME_SEVERITY from
generate_report.py so the terminal and the dashboard count outcomes the same
way. The first design tagged items with a cross-target key built from
item.callspec.indices at collection time; parse_nodeid already strips the
target (including dashed names like ci-nameres-es) from the node ID, so that
was dropped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The header listing targets and category filters scrolls away in long runs, and
pytest's own totals don't say how each service did on each target. A
pytest_terminal_summary hook now prints summarize_run's lines after pytest's
summary. pytest_runtest_logreport keeps its records in memory for this whether
or not --report-jsonl is set.

The URL lookup reads targets.ini directly rather than through get_target(),
which raises on an unknown --target; that run has already failed at
collection, and the summary shouldn't add a second traceback.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant