[testomut]

Why

An audit of main found checks that can catch real defects but were missing:

• request bodies were only ever sent as the documented example;
• oneOf / anyOf branches other than the example's were never sent;
• 403 and expired tokens were never tested;
• some gap codes named the wrong reason.

CI could not notice coverage going down: expected gaps were compared by count per code, with no operation or location, and nothing tracked active case IDs or write-test cleanup.

Baseline (main) → this PR

	Demo API	Kitchen sink
Active case IDs	86 → 90	112 → 118
Tests: functional / contract / scenarios	32 / 47 / 7 → 36 / 47 / 7	16 / 91 / 5 → 22 / 91 / 5
Documented statuses checked	16/21 → 17/22 (403 added to the spec and checked)	17/21 → 18/21
TS / Python parity	same → same	same → same
Write tests with guaranteed cleanup	all → all (now a CI gate)	all → all (now a CI gate)

Fault injection on the demo API, same spec, items + me:

Fault (DEMO_FAULT)	Generator from main	This PR (TS = Python)
filter ignored, sort reversed, record hidden, duplicates, wrong field, overlapping pages	6/6	6/6
require-optional (an optional field becomes required)	caught by chance, through a reviewed business scenario	caught by the body-variant tests
accept-expired (an expired token is accepted)	missed	caught
ignore-suspension (a refused account gets 200)	missed	caught
Total	7/9	9/9

What

Positive request-body variants (src/generate/variants.ts, variant-render.ts). These apply to create / update operations of a confirmed chain.

• Only the required fields (optional-omitted). Optional fields are removed outer first, and each removal is kept only while the payload stays valid. Nothing is removed inside oneOf / anyOf / conditional subtrees.
• Every other oneOf / anyOf branch (variant, request.body#<pointer>/@<n>). The branch the example sends is found by schema match, not by comparing values.
• Each variant is validated with ajv against the request schema before a test is written. A variant that cannot be built stays a gap with its reason.
• Each test expects the documented success status and a response that matches the contract, and removes what it created in finally. TypeScript and Python have identical case IDs.

Reviewed auth cases. A deps auth section holds environment variable names only, never the values:

"auth": {
  "forbidden": { "credentialEnv": "API_FORBIDDEN_TOKEN", "operations": ["getMe"] },
  "expired":   { "credentialEnv": "API_EXPIRED_TOKEN" }
}

• 403 is checked only on the operations named, and only on GET / HEAD with documented path values. A write gets UNSAFE_LIVE_CHECK; unknown path values get MISSING_RESOURCE_SOURCE.
• 401 with an expired credential uses the same safe request as the existing invalid-credential case.
• A test is skipped, with the reason, when its variable is unset.
• A section that does not match the spec is not used. Its problems become INVALID_REVIEWED_CONFIG gaps.

Accurate gap codes.

• The statuses of an operation no request is sent to carry the success call's reason. For example, POST /admin/reset 401 is now MISSING_CLEANUP, not NO_DOCUMENTED_REJECTION.
• An unchecked success status carries the reason the success call is blocked. For example, GET /reports/{id} 200 is now MISSING_RESOURCE_SOURCE.

Case ID vocabulary v4: optional-omitted, variant, expired-credentials. Registry validation checks that a variant location points at an existing branch.

CI quality gates (scripts/coverage-summary.ts, src/report/quality-gates.ts):

• ci/coverage-baseline.{demo,kitchen-sink}.json list every active case ID and every gap (code + operation + location). A lost case, or a gap the baseline does not list, fails the build until the PR updates the baseline with --write-baseline.
• TypeScript / Python parity: the same active case IDs.
• Cleanup: every test that creates data removes it in finally, through a fixture teardown, or with removeIfCreated right after the request and before any assertion. Unit tests include negative cases on which the gate must fail.
• The per-operation coverage summary (Markdown + JSON) is a coverage-summary artifact and also appears in the job summary.
• Fault injection covers items + me with all 9 fault modes.

Gap codes

• Closed:
  • UNCOVERED_VARIANT ×2 (kitchen sink POST / PUT /orders, payment);
  • UNCONFIRMED_BUSINESS_RULE for 403 (kitchen sink GET /orders);
  • NO_DOCUMENTED_REJECTION ×2 (demo: login / reset 401).
• Recoded to the actual reason: UNCONFIRMED_BUSINESS_RULE → MISSING_CLEANUP / MISSING_RESOURCE_SOURCE (demo login, reset, reports; kitchen sink cancel 202).
• New codes: none. INVALID_REVIEWED_CONFIG is reused for a bad auth section.

Verification (local, CI-equivalent)

• tsc, build, unit tests: 220 passed, 2 skipped (the oasdiff-only tests). 10 tests are new.
• Demo, generated and run once in TypeScript and Python:
  • 90 tests, each with its own registered case ID;
  • only the deliberate stats/get/contract defect fails;
  • registry and manifest are valid;
  • gaps match the expected ones;
  • all gates pass.
• Kitchen sink: compiles in TS and Python; registry valid; gates pass against the baseline.
• Fault injection: no fault → 84 pass, 0 fail; each of the 9 faults is caught in TS and in Python.
• The gate's own check: the first kitchen-sink baseline was written before the gap-code fix, and the gate flagged the recoded cancel 202 as an unexplained gap until the baseline was updated.

Limitations

• Search (?q=) is not part of the collection mapping yet.
• 403 for writes is not generated: a refused account's write could change data. It stays UNSAFE_LIVE_CHECK.
• Optional-field variants skip oneOf / anyOf / conditional subtrees, and arrays are varied at element 0 only.
• The cleanup gate inspects the generated code (finally / fixture / immediate removeIfCreated); it cannot prove the API actually deleted the record. The run-time cleanup log covers that.
• The demo's expired / suspended sessions are fixed test data of the demo server.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

• X
• Mastodon
• Reddit
• LinkedIn