[testomut]

Stacked on #15 (feat/coverage-depth, still open): the base of this PR is that branch, so the diff shows only this work. Merge #15 first; GitHub then retargets this PR to main.

Audit (before the changes)

Already in place, reused:

• param-cases.ts already gives one-rule-at-a-time constraint cases for query / header / cookie / path through the body-case engine:
  • required, type, enum, const, bounds, multipleOf, lengths, pattern, format;
  • values immediately outside each bound: minimum − step, maximum + step, the exclusive bound itself, the nearest non-multiple;
  • required parameters keep their documented values;
  • stable IDs and TS / Python parity.
• boundaryValues / queryVariants: the valid limits and enum values, as accepted-value.
• The Collection Testing Engine from PR feat(collections): deep collection testing, approved fixtures and test maintenance #14 (collection.ts, collection-tests.ts, collection-render.ts): strategies A / B / C, membership, filter, sort, pagination, uniqueness, lifecycle. The collection-cases.ts file named in the task does not exist; these three files are its role.

Gaps found:

1. Constraint cases exist only when a 400 / 422 is documented. Otherwise nothing at all is sent, and unspecified behavior had no non-failing category.
2. There were no generic probes (empty, non-numeric, decimal, zero, negative, huge, special characters).
3. A decimal for an integer was never sent, although the spec's type rejects it.
4. No case checked that the intended value actually went over HTTP.
5. Accepted values over the 5 / 15 limits were dropped silently, and accepted-value tests checked the status only.
6. Collections: no total / metadata check, no ascending ↔ descending consistency, no search, one mapping per resource.
7. The demo had no faults for parameter validation, total, persistence or search.

What changed

Area	Reused	New
Confirmed parameter cases	the engine, one rule at a time, stable IDs	an integer decimal inside the bounds (request.<in>#/<name>/decimal, rule type; multipleOf > 1 or an enum → CANNOT_ISOLATE_RULE); expectSentQuery / expect_sent_query on every query case; accepted values also check the response contract; dropped values → CASE_LIMIT_REACHED
Exploratory probes (src/generate/probes.ts)	requiredRequest, IDs, the contract validator	GET / HEAD scalar query parameters: missing, empty, text, decimal, zero, negative, large, special, numeric, case. They are deduplicated against confirmed rules and against values the documented range makes valid. A probe asserts only that its value was sent; the API's answer (status, documented, 5xx, contract) is recorded and never asserted. --run collects the answers into exploratory.jsonl and a report section
Collections	strategies A / B / C, controlled datasets, markers, polling	"match": "search" (case-insensitive: query in upper case, data in lower case); total (the same on every page, equal to the number listed); sort-reverse (descending = ascending reversed); "collections": [...] (several list operations per resource; collection still works)
Wire parity	—	Python query booleans are sent as true / false, like TypeScript (requests sent True)
Case IDs	registry, validation	vocabulary v5: sort-reverse, total, probe, the decimal type case
CI gates	baseline, parity, cleanup (#15)	a Python test that reassigns its own fixture fails the gate; fault injection now covers items, items/search, me, 17 faults

Found and fixed during verification:

• Fixture shadowing. The first Python total test wrote items, totals, _ = … inside a test whose resource fixture is items. Its finally then called list.remove, so the cleanup would not have run. Generated names were fixed, and the new gate rejects that output, now verified on the real generated code.
• Missing import. A TS contract file whose parameter cases have no query value referenced expectSentQuery without importing it. The check is now emitted only when needed.

Metrics (demo API, same spec for both columns)

	PR #15 generator	This PR
Active case IDs (demo / kitchen sink)	103 / 118	125 / 128
Documented statuses checked (demo)	—	19 / 24
Fault injection, TS and Python (17 faults)	14 / 17	17 / 17
Missed by the base	wrong-total, broken-search, truncate-decimal	—
Clean run without a fault	97 / 97	119 / 119 (probes never fail)

Which test catches each fault (mutation-check.ts --show, TS; Python identical):

Fault	Caught by
ignore-max / loose-types / ignore-enum	the confirmed maximum / type / enum cases
ignore-required	missing required query "q"
truncate-decimal	new decimal case limit=20.5
wrong-total	new /total equals the number of resources…
broken-search	new search filter test, and total
drop-update	the lifecycle and the update → read scenario
ignore-filter, wrong-sort, hide-created, duplicate, wrong-field, bad-pagination	collection tests (#14), plus total for filter / duplicates
require-optional, accept-expired, ignore-suspension	body variants and reviewed auth (#15)

sort-reverse catches no fault on its own here: wherever it fails, the individual sort tests fail too. It stays because the task asks for asc / desc consistency, and it is the read-only check of that on APIs where the individual order cannot be verified.

Gaps

• New: CANNOT_ISOLATE_RULE GET /items/search lifecycle. The update renames the item, and the search needs a required q, so the updated item cannot be found reliably. It is recorded in the baseline.
• Closed: none.
• New codes: none.
• Probes add CASE_LIMIT_REACHED / UNSAFE_LIVE_CHECK / MISSING_RESOURCE_SOURCE with rule probe where they apply; none on the demo.

Commands run (local, CI-equivalent)

• npx tsc --noEmit, npm run build: ok.
• npx vitest run: 233 passed, 2 skipped (oasdiff-only).
• Demo generate --run (TS and Python):
  • 125 tests, each with its own registered case ID;
  • only the deliberate stats/get/contract defect fails;
  • 10 probe observations per language, none with a 5xx or an undocumented status;
  • registry and manifest valid.
• scripts/coverage-summary.ts against the previous baseline: cases +35 / lost 0, gaps +1 (the one above). The baseline was then updated. Kitchen sink: cases +10 / lost 0.
• Kitchen sink: tsc and py_compile pass, registry valid, parity holds.
• scripts/mutation-check.ts, TS and Python, items + items/search + me: 17 / 17, clean without a fault.
• maintain on a changed spec: regenerated, validated, the stale mapping reported.

Limitations

• Probes cover scalar query parameters of GET / HEAD only; header / cookie probes and probes of write operations are not generated. Write operations get UNSAFE_LIVE_CHECK.
• A probe's outcome is evidence for a human. A 5xx is highlighted in the report but never fails the build.
• Read-only total is skipped when the collection has more than maxPages pages. Read-only sort-reverse and total re-read once on a mismatch, but heavy concurrent writes can still make them flaky.
• sort-reverse compares values, not ids, in read-only mode (ties have no defined order).
• Search is a case-insensitive substring on one string field; other search semantics (tokens, several fields, ranking) need their own mapping.
• Range filters (between=price,10,30) and default-scoped boolean flags (is_rental) are still not mapped.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

• X
• Mastodon
• Reddit
• LinkedIn