[testomut]

Why

The AI should take part in creating tests by default, and when no AI tool is available the agent should work without it.

What

On by default, never required.

• generate uses a console AI tool you are logged in to: API_AI / --ai-runner, else claude, else codex on PATH. No API keys.
• When no tool is found, the tool fails, or --no-ai is passed, the run says so and is generated exactly as before. Verified on the demo: same 125 cases, same gaps.

New AI task: read-only sources of path values. A safe operation such as GET /brands/{brandId} was blocked with MISSING_RESOURCE_SOURCE, because there was no real id to call it with.

• The AI picks the list operation and field that hold existing ids, e.g. brandId ← GET /brands #/0/id.

• Spec check before use. The source must be:

  • a safe list operation callable as is;
  • addressed by a pointer through a list element, not a summary field such as /total;
  • an identifier of a fitting type: an integer may fill a string parameter, a decimal never.

  Low confidence is rejected.

• The test only reads: list → record by id. It is skipped with the reason when there is no record, or when the list is refused (e.g. it needs another role).

• Marking: ai-proposed annotation (pytest marker ai_proposed), plus a report section "AI participation".

• Reproducible. Answers are cached in the suite (_support/ai-sources.json, part of the generation manifest). A regeneration, maintain (which never calls the AI) and a run without AI all give the same tests.

• Reviewed form: the deps file "sources": [{ "operationId", "param", "from", "pointer" }].

Unchanged. Unconfirmed write dependencies stay proposals (ai-deps.proposed.json): an AI answer never creates or deletes data. The --ai-max-calls default is now 25.

Results

Toolshop, real stand, through Claude Code:

• 11 answers in 140 s; on the next run 8 came from the cache and only 2 new calls were made (the dependency proposals, which are not cached).
• 8 sources passed the spec check: brands, category tree, related / specs of a product, invoices ×3, users.
• Tests: 310 total. 239 passed (+8: brands, category tree, related, specs). 63 failed, the same known spec mismatches as before. 8 skipped with the reason: a fresh account has no invoices, and GET /users answers 403 to a non-admin.
• MISSING_RESOURCE_SOURCE 22 → 15. Nothing left on the stand.

Unit tests: 242 passed, 2 skipped. Seven are new, using a fake console tool:

• the spec check rejects an unknown operation, a write operation, a missing pointer, a summary field and a type mismatch;
• a low-confidence answer is rejected;
• the cache replays without AI;
• a failing tool stops only the AI step;
• generated TS / Python tests are read-only, skipped without data, marked, and have the same case IDs.

Limitations

• An AI source is checked against the spec, not proven correct: a wrong but type-compatible field would read some other record. That is why these tests only read, are marked unreviewed, and are listed for review.
• Only the AI answers for sources are cached; dependency proposals are asked again on each run.
• run (analysis of an existing project) keeps its opt-in --ai.
• CI has no AI tool, so it exercises the "without AI" path.

[coderabbitai]

Important

• Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration

• Source mode: Auto
• Repository: Unavailable
• Branch: Unavailable
• File: No configuration file from this repository applied
• Commit: Unavailable
• Layers: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: 8a0b22d2-5759-486b-bb1c-1184cd6067e7

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

• X
• Mastodon
• Reddit
• LinkedIn