Repository navigation
feat(coverage): complete, deterministic coverage from OpenAPI (staged) - #12
Merged
Merged
Conversation
Stage 1 + 2 of the coverage plan.
- docs/SUPPORT_MATRIX.md: OpenAPI keyword / check → supported, partial,
missing, with the stage that changes it.
- Query enum variants: each value is its own test with its own case ID
(vocabulary v2, functional/positive accepted-value,
request.query#/<name>/<value>); Python renders them too (it counted them
but rendered none). Values equal as strings are sent once.
- Cases the generator no longer produces become tombstones in cases.json.
- generate --run checks the physical tests: Playwright JSON annotations,
pytest collection via API_AGENT_CASES_OUT (conftest hook); report
generated.caseIdCheck {tests, missing, duplicates, unregistered}.
- CI: case-ID check, registry validate --spec, manifest validate for
TypeScript and Python, same active cases in both.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars.
 |
- src/generate/body-cases.ts: from a valid payload, one broken rule per
case at any depth (request.body#/customer/address/city,
request.body#/items/*/price): required, type, nullable, enum (string,
number, integer, boolean), const, min/maxLength, pattern, format,
minimum/maximum, exclusive min/max (3.1 numeric and 3.0 boolean),
multipleOf, min/maxItems, uniqueItems, items, min/maxProperties,
additionalProperties: false; allOf merged.
- Rules that cannot be broken alone, oneOf / anyOf / not / discriminator,
and missing samples become coverage gaps with a code and the exact
location (CANNOT_ISOLATE_RULE, UNSUPPORTED_SCHEMA_KEYWORD,
MISSING_VALID_SAMPLE). Fixes minProperties sending {} (it also broke
required).
- Limit: --max-cases (default 60); potential / generated / dropped per
operation in coverage-gaps.json, dropped ones as CASE_LIMIT_REACHED.
- tests/fixtures/coverage/kitchen-sink.yaml: synthetic spec with a wide
keyword set. Every generated case is checked with ajv: exactly one
error, the expected keyword, the expected instance path (68 cases).
Demo API: 58 tests in TypeScript and in Python, only stats/get/contract
fails; every test has its own registered case ID.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- src/generate/param-cases.ts: a valid request (every required query / header / cookie parameter with its documented value) in which exactly one parameter breaks exactly one rule; rules from the body engine (required, type for number / integer / boolean, enum, const, pattern, format, bounds, multipleOf, lengths, min/maxItems, uniqueItems). - Arrays serialized as documented: form explode true / false, spaceDelimited, pipeDelimited; header simple; cookies in the Cookie header. - Path parameters: invalid type / format / pattern / enum / length; numeric bounds on write operations are UNSAFE_LIVE_CHECK gaps (an out-of-range id could address a real record). - Required header / cookie parameters are sent in every generated call (TypeScript optionsWith, Python pyRequired); one without a documented value blocks the operation. - One data-driven loop over whole requests in TypeScript and Python; catalog keeps style / explode. The old query-only generator is removed. - Tests: every case leaves the other parameters valid and breaks its own rule; serialization; path safety; gaps; TS/Python parity on the fixture. Demo API: 58 tests in TypeScript and Python, only stats/get/contract fails. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…on (stage 5) validator.ts and contract.py: - response headers: required ones present, documented ones match their schema (numbers / booleans coerced, arrays comma-separated); - no body expected for 204 / 205 / 304, HEAD and responses without content: a body that arrives is "unexpected-body"; - an empty body where a schema is documented is "empty-body" (an undocumented Content-Type is reported first; JSON null is a body); - text media types with a schema are validated as strings; - writeOnly properties must not appear in a response and are never required in one; - what is not checked (no schema, unsupported media type) is a note: Playwright annotation contract-note, pytest warning. coverage-gaps.json lists MISSING_RESPONSE_SCHEMA and UNSUPPORTED_MEDIA_TYPE per response. tests/unit/validator.test.ts runs one table of 24 cases through both validators and requires identical results. Demo API: no new failures; only stats/get/contract fails. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
src/generate/error-cases.ts plans, per operation, the documented errors that can be triggered safely; the rest are gaps with codes. - 401 invalid credentials per security scheme (bearer / OAuth2 / OIDC, basic, API key in header / query / cookie), sent without real ones. - 405: a GET to a path that defines no GET, one case per path; a path with GET is UNSAFE_LIVE_CHECK. - 409: create twice with the same payload, only with a reviewed deps "conflictOn"; business scenario, both records removed in finally; otherwise UNCONFIRMED_BUSINESS_RULE. - 412: non-matching If-Match / If-Unmodified-Since on the test's own record. - 415: the documented body as text/plain, with confirmed cleanup. - 403 / 429 / other 4xx: gaps (UNCONFIRMED_BUSINESS_RULE, UNSAFE_LIVE_CHECK, UNREACHABLE_RESPONSE_STATUS). - Vocabulary v2: invalid-credentials, method-not-allowed, precondition-failed. TypeScript and Python render the same cases. - CI: the synthetic fixture (now with security, 405, 412, 415, 409, 403, 429, 418) is generated in both languages and must compile (tsc, py_compile) with a valid registry. Demo API: 59 tests per language (401 with an invalid token on GET /me passes live); only stats/get/contract fails. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s (stage 7) - Boundary-valid query values (minimum / maximum inside exclusive bounds and on multipleOf, shortest / longest string without pattern / format): one test and one accepted-value case ID each, TypeScript and Python; registry validation accepts these values (shared boundaries.ts). - deps "echoes": the create test asserts that the named fields come back as sent (only with this reviewed mapping; the spec does not state it). - Sampler: smallest multipleOf inside the bounds; 3.0 boolean and 3.1 numeric exclusive bounds combined with minimum; distinct elements for uniqueItems (or blocked with the reason). - oneOf / anyOf in a sent payload: the branches not exercised are UNCOVERED_VARIANT gaps at their location. Demo API: 61 tests per language (limit=1 and limit=100 pass live); only stats/get/contract fails. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e 8)
- src/scenarios/dsl.ts + schemas/scenarios.schema.json: a reviewed
scenarios file (--scenarios): steps with phases, path / query params and
bodies with {var} values saved by earlier steps (JSON pointers), the
documented status each step must get, cleanup steps.
- Strict validation (also `scenarios validate --spec`): operations and
parameters exist, statuses documented, variables saved before use, a
writing scenario has cleanup, the primary is an action, unique IDs.
- Generated in the primary operation's scenarios file, TypeScript and
Python: steps in order, cleanup in finally with failures reported. The
scenarioUid comes from the external id (never the title); registry
definition with steps and externalId.
- CRUD: deleting the same resource twice gets the documented 404.
- demo-api/scenarios.json (positive and negative) runs in CI in both
languages.
Demo API: 64 tests per language, both scenarios pass live, nothing left
behind; only stats/get/contract fails.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ecks in CI (stages 9-10) - src/generate/coverage.ts: stable gap codes; positive-blocked reasons as codes; per operation the generated count and missing code of each suite, unchecked statuses (with the precise reason when an error gap names the status), unchecked request media types and security alternatives. - coverage-gaps.json gets a summary (operations, generated tests per suite, gaps by code); generate logs the totals. - Expected gaps: the synthetic fixture (unit test) and the demo API (CI, from the real run) must match exactly: a new gap or a silently vanished one fails. - CI: TypeScript / Python parity on the synthetic fixture too. - README: "What is covered, and what is not" (spec-derived, needs confirmation, never generated); docs/SUPPORT_MATRIX.md completed; CHANGELOG. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
testomut
marked this pull request as ready for review
November 9, 2026 20:14
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Coverage completeness, in stages.
docs/SUPPORT_MATRIX.mdlists every OpenAPI keyword / check with its state (supported, partial, missing, not generated on purpose) and the stage that changed it. One commit per stage; each verified on the demo API in TypeScript and Python before the next.accepted-valueID each (vocabulary v2), Python renders them too; tombstones for cases no longer generated; case IDs checked on the real run (Playwright report, pytest collection)request.body#/items/*/price), all listed keywords, exactly one broken rule each (ajv-verified);--max-caseswith dropped cases reportedUNSAFE_LIVE_CHECK)conflictOn), 412, 415; 403 / 429 / others as gapsechoes, sampler fixes (multipleOf, exclusive bounds, uniqueItems), uncovered oneOf / anyOf branches reported--scenarios, schema, strictscenarios validate), repeated deleteVerification
npm run typecheckclean;npm test192 pass, 2 skipped (real-oasdiff tests).demo-api/scenarios.json): 64 physical tests per language, each with its own registered case ID; onlystats/get/contract(the deliberate defect) fails; nothing left behind; registry and manifest valid; the same active cases in both languages; coverage gaps exactly as expected.tests/fixtures/coverage/kitchen-sink.yaml(CI): generated in both languages, compiles (tsc,py_compile), valid registry, 112 identical cases in TypeScript and Python.Honest limits (reported as gaps, listed in the matrix)
oneOf / anyOf / not / discriminator negatives; array-item constraints in parameters; 403 without a reviewed role setup; 429; expired tokens; 405 where only unsafe methods are undocumented; list / filter / pagination without a reviewed scenario; eventual consistency polling (no polling step in the scenario DSL yet); non-JSON / non-text bodies; optional-field variants of create payloads; multiple
examplesas variants.