Skip to content

feat(skills): add the verify-interface verification skill - #16

Merged
Upd4ting merged 3 commits into
campaign/integrationfrom
feat/verify-interface-skill
Oct 31, 2026
Merged

Upd4ting merged 3 commits into
campaign/integrationfrom
feat/verify-interface-skill

Conversation

@Upd4ting

@Upd4ting Upd4ting commented Oct 31, 2026 •

Copy link
Copy Markdown
Member

Why

Antony wants every AntelopeJS verification skill (V5) on campaign/integration. interface-subscriptions has the shared verify-interface layer from the pack and nothing of its own. A backend author imports CreateSubscription, CancelSubscription, ValidateSubscriptionRequest, the error classes and the status and event vocabularies, calls them through core next to a billing provider, and a provider author runs the shipped conformance helpers. This PR adds the repo-owned layer that drives that path. No module source changes: outside .agents/ and .claude/ the diff is the one-line AGENTS.md pointer that antelopejs-sync --verify interface writes.

What changed

  • verify.config.ts plus helper modules: 14 journeys over 12 features. The interface owns no billing logic, so its executable surface is the validator, the error classes, the frozen vocabularies and the five InterfaceFunctions. Journeys drive each the way its consumer does:
    • every-function (pack) and calls-through-core load the packed interface in ajs project dev next to a stub provider and call the five functions with the arguments a backend passes (idempotency key positionally, optional trailing provider given and left out). The stub logs the arguments in order, the second view.
    • validate-fields, validate-order, validate-interval and validate-dates run the packed ValidateSubscriptionRequest against every rule in docs/4.implementing.md: messages, error class chain, first-rule order, strict future dates with a fixed now, epoch seconds refused, past dates accepted without now, NaN and infinities always refused.
    • status-vocabulary and wire-names pin the frozen status, end reason, interval, timing and event strings, antelopeSubscription and the 366-day cap. error-classes pins names, messages, providerCode, status and the declined payment.
    • signatures and field-types compile strict consumer probes against the packed .d.ts (argument order, optional fields, literal unions, what JsonObject accepts and refuses).
    • suite-helpers, suite-defaults and suite-files load the shipped tests/support module through the package's own exports with and without the README's environment variables.
  • features/: smoke, calls, signatures, field-types, validation-fields, validation-order, validation-interval, validation-dates, statuses, vocabularies, errors, conformance (plus the README map). seeded-breaks/: 12 patches, one per non-README feature, each editing only files its feature covers (smoke edits src/functions.ts, as the other interface skills do). seeded-doors/: contract, format, publish.
  • Doors: Format on src/types.ts (the status, interval, timing, event strings and the metadata key providers store and consumers branch on), contained by statuses/status-vocabulary and vocabularies/wire-names; Contract (src/index.ts, src/functions.ts, package.json); Publish (release files). No Data, Outbound or Exposure door applies: the package stores nothing, calls nothing and serves nothing.
  • The skill type-lints clean under 0.2.17 with no ratchet file and no suppression.
  • Limit: the stub provider echoes its arguments, so calls proves what reaches a provider, not what a provider answers. Provider behaviour belongs to the conformance suite, which needs a real provider and credentials.

Proofs

Head 7c96927 on origin/campaign/integration fa3d013, run by /home/antony/.cache/campaign/h8/work/v5-interfaces-tools/proof.sh (log proofs/interface-subscriptions.log). Journeys in the order the driver ran them:

PASS smoke/every-function
PASS calls/calls-through-core
PASS conformance/suite-helpers
PASS conformance/suite-defaults
PASS conformance/suite-files
PASS errors/error-classes
PASS field-types/field-types
PASS signatures/signatures
PASS statuses/status-vocabulary
PASS validation-dates/validate-dates
PASS validation-fields/validate-fields
PASS validation-interval/validate-interval
PASS validation-order/validate-order
PASS vocabularies/wire-names
## Verify: PASS

Doors, containments, evidence after clean, containers:

FLAGGED contract (path rules flag: contract)
FLAGGED format (path rules flag: format, contract)
FLAGGED publish (path rules flag: contract, publish)
CONTAINED format statuses/status-vocabulary: it passes unpatched and fails a check with its door
CONTAINED format vocabularies/wire-names: it passes unpatched and fails a check with its door
evidence files kept after clean: 179, next to run.json
containers left: none

Seeded breaks (antelopejs-verify seeded, each against the unpatched control):

CAUGHT smoke by every-function
CAUGHT calls by calls-through-core
CAUGHT conformance by suite-defaults
CAUGHT errors by error-classes
CAUGHT field-types by field-types
CAUGHT signatures by signatures
CAUGHT statuses by status-vocabulary
CAUGHT validation-dates by validate-dates
CAUGHT validation-fields by validate-fields
CAUGHT validation-interval by validate-interval
CAUGHT validation-order by validate-order
CAUGHT vocabularies by wire-names

Mutation check beyond the shipped patches: 32 further breaks of documented behaviour (validator rules and messages, error fields, vocabularies, frozen lists, optional fields, JsonObject, exports, the conformance helpers) were applied one at a time to the working tree, each run through every journey by mutate.py (log proofs/mut-subscriptions.log). 26 are CAUGHT by at least one journey and 6 did not build (the contract suite in src/tests fails to compile), so they cannot ship; none was MISSED.

Agent-skills V4 eval (main at 1fe3a82, --min-catch 1.0 --min-door-catch 1.0 --jobs 3, exit 0):

| Case | Outcome | Verdict | Wall | Cost | Turns | Note |
| --- | --- | --- | --- | --- | --- | --- |
| break-calls | caught | FAIL | 1.1 min | $0.08 | 10 |
| break-conformance | caught | FAIL | 0.6 min | $0.09 | 10 |
| break-errors | caught | FAIL | 0.5 min | $0.08 | 9 |
| break-field-types | caught | FAIL | 0.6 min | $0.07 | 7 |
| break-signatures | caught | FAIL | 0.6 min | $0.09 | 11 |
| break-smoke | caught | FAIL | 0.5 min | $0.08 | 10 |
| break-statuses | caught | FAIL | 0.5 min | $0.09 | 9 |
| break-validation-dates | caught | FAIL | 0.4 min | $0.06 | 6 |
| break-validation-fields | caught | FAIL | 0.5 min | $0.09 | 10 |
| break-validation-interval | caught | FAIL | 0.6 min | $0.08 | 9 |
| break-validation-order | caught | FAIL | 0.4 min | $0.06 | 7 |
| break-vocabularies | caught | FAIL | 0.5 min | $0.08 | 8 |
| control | no alarm | PASS+NOTES | 0.3 min | $0.08 | 7 |
| door-contract | caught | FAIL | 0.7 min | $0.11 | 11 |
| door-format | caught | FAIL | 0.7 min | $0.08 | 9 |
| door-publish | caught | PASS+NOTES | 0.6 min | $0.10 | 11 |

Catch rate (seeded breaks): 12/12 (100%)
Door catch rate (seeded doors): 3/3 (100%)
Unpatched head (no false alarm): 1/1 (100%)
Unpatched head with a feature touched (no false alarm): none ran
Catch rate where the head with the feature touched does not FAIL (the break is what fails): none ran
Cost per case: $0.08, 0.6 min, 116552 input tokens (cache included), 2158 output tokens. Total $1.32 over 16 cases.

Repo checks and api-diff on head 7c96927:

$ nice pnpm run checks
Checks: 8 passed, 0 failed.
$ campaign/tools/api-diff-slot.sh origin/campaign/integration
API: 1 packages, 3 entries unchanged against origin/campaign/integration (committed HEAD).
API: skipped .agents/skills/verify-interface, private.
API: skipped test-helper entry ./dist/tests/*.d.ts in .; test helpers need a test runner and are outside both proof halves.
Runtime API: 1 packages, 2 entries unchanged against origin/campaign/integration (committed HEAD).
Runtime API: skipped test-helper entry ./dist/tests/*.js in .; test helpers need a test runner and are outside both proof halves.
No behavioural probe ran.
Suppressions in authored code files (origin/campaign/integration and committed HEAD). A lint directive counts once per rule it names; `*` is a directive naming none.
| Suppressions | base | HEAD |
| --- | ---: | ---: |
| Lint suppressions, all rules | 0 | 0 |
| `@ts-expect-error` | 1 | 1 |
| `@ts-ignore` | 0 | 0 |
| `SAFETY:` comments | 0 | 0 |
| Lint config rule-offs, all rules | 0 | 0 |
<!-- antelopejs-proof base=fa3d01347839a81f7d14d4c0661da62c547473fc head=7c969271f70e88240f1af68fd32586c6272e3602 lines=14 digest=5b83c32657e4a676 command: antelopejs-api-diff origin/campaign/integration -->

Attention

  • SKILL.md (shared layer, from sync) says --base origin/main; on this campaign the base is origin/campaign/integration. That text belongs to tooling-configs.
  • Two proof drivers started by mistake ran at once for a few minutes and interleaved one log. The numbers above come from a clean rerun on the final head, after both had ended.
  • minor-bump is not used: it needs the npm registry and this skill avoids a journey that can end NOT VERIFIED.

@Upd4ting

Copy link
Copy Markdown
Member Author

Verdict: PASS+NOTES

Head verified: 7c969271f70e88240f1af68fd32586c6272e3602 (base campaign/integration fa3d013, which is also the merge-base). Independent verifier v5i on Opus (the owner ran on Sonnet). Fresh clones under /home/antony/.cache/campaign/h8/verify/v5i/, Node 24 (.nvmrc), pnpm install --frozen-lockfile.

Goal (Antony): every V5 verification skill lands on campaign/integration and catches breaks of documented behaviour. This PR adds the repo-owned verify-interface layer.

Tool version

The repo pins @antelopejs/tooling-configs 0.2.17, which predates tooling-configs#128 (merged 0d448f0). With 0.2.17 a failed journey outside plan.run still renders PASS, and on this diff plan.run is smoke only. So I built the tooling-configs campaign/integration tip de99b0e (contains 0d448f0) with campaign/tools/tgz.sh (sha256 4c8ea077...b71c24) and wired it into a second clone (subs-tc) with use-tgz.sh, uncommitted. All journey, seeded, containment and mutation runs below used it (verify has #128 printed by the driver). checks, api-diff and the V4 eval ran on the unmodified PR clone at 0.2.17.

Journeys at head (driver bin/proof.sh, log logs/subs.log)

  • map exit 0. plan: unmapped [], run [smoke].
  • new -> doctor (all ok) -> smoke and all 11 other features: 14/14 PASS.
  • verdict --pr 16 with #128: ## Verify: PASS. The 13 journeys outside the plan are listed as (outside the diff's features), all pass. My first render was FAIL only because my local package.json swap opened contract and publish doors; with those two rows dismissed as false positives it is PASS (logs/subs-verdict2.log). Not published.
  • clean: 179 evidence files kept. No containers left.

Seeded breaks and doors (rerun)

  • seeded smoke calls conformance errors field-types signatures statuses validation-dates validation-fields validation-interval validation-order vocabularies: 12/12 CAUGHT, each with a failing check against its passing control.
  • seeded-doors: contract, format, publish FLAGGED. containments: format CONTAINED by statuses/status-vocabulary and vocabularies/wire-names.

My own breaks (driver bin/mutate.py, specs bin/mut-subs.json, bin/mut-subs-pack.json, logs logs/mut-subs*.log)

17 breaks of documented behaviour, none equal to a shipped patch or to one of the owner's 32. Each was applied to the tree, built, run through every feature, then reverted. Tree clean afterwards.

Break Documented by Result
mutual-exclusivity rule moved after the date rules docs/4.implementing.md rule order CAUGHT validate-dates, validate-order
currency rule before amount rule same CAUGHT validate-fields, validate-order
currency regex unanchored (EURO passes) "exactly three uppercase letters" CAUGHT validate-fields
trialing terminal only canceled is terminal CAUGHT status-vocabulary
omitted intervalCount refused "an explicit intervalCount", defaults to 1 CAUGHT validate-interval, validate-fields, validate-dates
statuses not in lifecycle order TSDoc "in lifecycle order" CAUGHT status-vocabulary
SubscriptionPaymentError drops providerCode "Each carries providerCode" CAUGHT error-classes, every-function
SubscriptionPaymentError.name = base name error table CAUGHT error-classes, every-function
base SubscriptionError.name = Error error table CAUGHT error-classes, every-function
billingAnchor validated with trialEnd's value date rules CAUGHT validate-dates
every broken rule joined into one message "naming the first failed rule" CAUGHT validate-dates, validate-fields, validate-order
subscription.payment_succeeded renamed event table did not build (src/tests compile)
quarter interval added day, week, month, year CAUGHT field-types, validate-interval, validate-order, wire-names
CANCELLATION_TIMINGS unfrozen frozen vocabularies CAUGHT wire-names
inWholeSeconds off the whole second shipped helper TSDoc CAUGHT suite-helpers, suite-defaults
tarball drops dist/tests ("!dist/tests" in files) README conformance suite CAUGHT suite-files
tarball drops dist/errors.* package installs and loads CAUGHT every-function, calls-through-core, suite-files

16 built, 16 CAUGHT, 0 missed; 1 did not build, so it cannot ship.

V4 eval (agent-skills main 1fe3a82, --min-catch 1.0 --min-door-catch 1.0 --jobs 3)

Catch rate 12/12, door catch rate 3/3, unpatched head no false alarm 1/1 (control PASS+NOTES), $1.27 over 16 cases, exit 0 (eval/subs/eval.log). Reproduces the owner's 12/12, 3/3.

Repo gates

  • GITHUB_BASE_REF=campaign/integration pnpm run checks: Checks: 8 passed, 0 failed. Without that variable the suppressions check compares with main (where @ts-expect-error is 0) and fails. CI sets it, and CI is green.
  • api-diff (slot script) against origin/campaign/integration: API 3 entries unchanged, runtime 2 entries unchanged. Suppressions all 0 -> 0 except @ts-expect-error 1 -> 1. guardrails.suppressions.json is not in the diff.
  • Outside .agents/ and the .claude/skills/verify-interface link, the only changed file is AGENTS.md, with the one-line pointer to the skill.
  • CI on 7c96927: ci / Node .nvmrc, ci / checks, checks, GitGuardian pass. Conventional Commits, no attribution.

Notes (non-blocking)

  1. Most journeys run <repo>/dist/index.js (journey-kit.ts packed() passes context.repo), not the tarball new extracted. The title says "the packed interface" and the PR body says the validators run "the packed ValidateSubscriptionRequest". The code is the same build, so behaviour breaks are caught. Packaging breaks are caught here only because every-function and calls-through-core load the real tarball and suite-files reads the files field. A rename to "the built interface" would be accurate.
  2. The 0.2.17 pin cannot see #128. A re-proof that runs every feature on this repo's own toolchain will report PASS even with a failing journey outside the plan until the pack bump lands.
  3. SKILL.md (sync-written) says --base origin/main; on this campaign the base is origin/campaign/integration. That text belongs to tooling-configs.

Evidence: /home/antony/.cache/campaign/h8/verify/v5i/logs/ (subs.log, subs-verdict2.log, mut-subs.log, mut-subs-pack.log), /home/antony/.cache/campaign/h8/verify/v5i/eval/subs/, subs-checks.log, subs-apidiff.log.

@Upd4ting
Upd4ting merged commit b6645bd into campaign/integration Oct 31, 2026
6 checks passed
@Upd4ting
Upd4ting deleted the feat/verify-interface-skill branch October 31, 2026 23:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant