Repository navigation
feat(collections): deep collection testing, approved fixtures and test maintenance - #14
Merged
Merged
Conversation
…t maintenance Collection Testing Engine for list operations, from a reviewed deps "collection" mapping checked strictly against the spec (mismatch = INVALID_REVIEWED_CONFIG): - strategy A (create & verify): controlled datasets with a unique marker in an isolating filter, cleanup in finally; filter (match + non-match), sort (3 distinct values created out of order), pagination (page size + 1 records), uniqueness, persisted fields by policy, bounded polling; - strategy B (approved fixtures): generate --fixtures compares a list with a fixture a person approved; fixtures capture writes candidates only; fixtures refresh --draft-pr proposes a changed candidate (one PR per operation); - strategy C (read-only) with explicit skips; strategy recorded per operation; - create -> list -> update -> list -> delete -> list scenario; - case ID vocabulary v3; same cases in TypeScript and Python. maintain: spec diff -> deterministic regeneration -> registry and manifest validation -> one Draft PR per suite (updated, never duplicated, never merged) with a summary of changed operations, tests, expectations, fixtures and gaps. Fault injection: the demo API simulates six list defects (DEMO_FAULT); scripts/mutation-check.ts requires each to be caught. The previous generator's 65 demo tests catch 0/6; the collection tests catch 6/6 in both languages. Fixes: a single resource with an array field is not an unconfigured collection; hand-written _support/api/*Api.ts stay in the generation manifest. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars.
 |
- "strategy": "read-only" in a collection mapping: no records are created for the checks even when a create is confirmed; - page-number paging with a fixed page size (no size parameter); - read-only checks of a public list go through publicApi, never a login; - read-only filter tests take up to 5 values from the list and require the filtered result to be a subset of the unfiltered list (re-read once); - a sort case ID may carry a reviewed value the spec does not enumerate. The subset check catches the defect of Toolshop's deliberately defective build (by_brand=1 returns rentals 27-29 the list never shows) and passes on the regular build, in TypeScript and Python. Demo fault injection stays 6/6. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…en the .env one is missing or refused The .env account is used while it works. When it is missing or its login is refused (401/403/422, e.g. the test environment reset its data), the agent registers one account before --run and the generated tests reuse it; a worker of a standalone run registers at most once. A working account never causes a registration; .env is never changed. Verified on the demo API: wrong password -> 1 registration in an agent run, 1 per worker standalone; right password -> none. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…wright globalSetup Playwright starts a fresh worker after a failed test, so per-worker registration in auto mode registered an account per failure (4 for 3 failures on Toolshop). _support/global-setup.ts now settles the account once in the main process and hands the credentials to every worker through the environment (memory only). init adds it to playwright.config.ts when present. Verified on Toolshop: one registration for a run with failures and restarted workers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…'s globalSetup when it has no register settings Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The generated conftest registers at most one account per pytest session: new always, auto only when the .env account is missing or refused (401/403/422). Credentials stay in memory; .env is never changed. The CLI no longer refuses new accounts for Python. Verified on the demo API: auto with a wrong password -> 1 registration, with the right one -> none, new -> 1; an agent-driven Python run in auto mode registers once and the suite reuses it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The goal is catching API defects, not adding more tests. On the demo API the current generator's 65 tests miss all six injected list defects. This PR adds collection tests that catch all six, with the same results in TypeScript and Python.
DEMO_FAULT)What
Collection Testing Engine. A list operation gets deep tests only from a reviewed deps
collectionmapping. Every pointer, parameter, sort value and paging parameter in it is checked against the spec. A mismatch becomes anINVALID_REVIEWED_CONFIGgap, never a guess.finally.generate --fixtures <dir>compares the list with a fixture a person approved. Fields are compared by policy:exact,tolerance:<n>,schema,ignore, plus anexhaustiveoption.fixtures capturewrites candidates only.fixtures refresh --draft-prproposes a changed candidate in one Draft PR per operation. Approved files are never written by the agent.coverage-gaps.json.collection-contains,filter,sort,pagination,uniqueandfixture.maintain. Spec diff → deterministic regeneration → registry and manifest validation → one Draft PR per suite. An open PR is updated rather than duplicated, and nothing is merged. The summary lists:Fault injection. The demo API gets
offset/tag/sortonGET /items, sixDEMO_FAULTmodes, and twocatalogreference items, which the approved demo fixture describes.scripts/mutation-check.tsruns a generated suite once per fault, on its own port.CI.
--depsand--fixtures.Fixes found along the way:
Item.tags) was reported as an unconfigured collection._support/api/*Api.tsdropped out of the generation manifest on regeneration, somaintainwould have counted them as removed.Verification (local, CI-equivalent)
tsc, build, and unit tests: 206 passed, 2 skipped. That includes 12 new collection, fixture and strategy tests.stats/get/contractdefect fails;demo-api/expected-gaps.json.maintain, checked twice:limitmaximum and a removed sort value: 3 files really updated, 20 header-only, the stale-pricemapping reported, validation passed;fixtures capture/refreshagainst the local demo: the candidate is written, and with no difference under the policies it reports "nothing to refresh".Limitations
collectionmapping drops all its collection tests for that operation. They come back once the mapping is fixed, and the summary says why they're missing.maintain --draft-prandfixtures refresh --draft-pr(gh pr list/edit, force-with-lease on the bot branch) was not run against GitHub, to avoid opening throwaway PRs.maintainruns without oasdiff and regenerates the whole suite (stated in its summary). Locally it ran with oasdiff 1.34.1.--runonly finds Playwright suites undertests/api(the repository'splaywright.config.ts). This was already the case.