Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 40 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
name: CI

on:
pull_request:
push:
branches: [main]

permissions:
contents: read

concurrency:
group: ci-${{ github.ref }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}

jobs:
test:
name: Typecheck and test (Node ${{ matrix.node }})
runs-on: ubuntu-latest
timeout-minutes: 15
strategy:
fail-fast: false
matrix:
# Both majors package.json's engines field claims to support.
node: [20, 22]

steps:
- uses: actions/checkout@v4

- uses: actions/setup-node@v4
with:
node-version: ${{ matrix.node }}
cache: npm

# The optional audio deps (mic, speaker) only matter for `npm run call`;
# npm tolerates their native build failing on a headless runner.
- run: npm ci

- run: npm run build

- run: npm test
65 changes: 65 additions & 0 deletions improvements.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,7 @@ you which stack PR closes the row.**
| 29 | SO linking sent filtered `assistantIds` arrays | Silent unlink of live-but-untracked assistants | None | RESOLVED 2026-08-03 (#51) |
| 30 | Tool-linking pass could PATCH a raw assistant slug | Mid-push 400 naming the wrong resource | None | RESOLVED 2026-08-03 (#51) |
| 31 | Unresolved references handled 3 inconsistent ways, no dangling-ref check | Same authoring mistake, three different failure modes | None | Open |
| 32 | Test suite never ran in CI; 20 tests rotted after the hash store | Regression guards for #22/#23 silently stopped running | None | RESOLVED 2026-09-30 (#56) |

**Active backlog after cleanup:** `#2`, `#6`, `#8`, `#12`, `#20`, `#24–#26`, `#31`, and the open remainder of `#27` (wiring the listing-completeness verdict into push/delete/audit, and moving `cleanup.ts` onto the shared pager). Resolved entries stay in this file as historical incident notes per the maintenance directive; stale superseded backlog rows are not duplicated.

Expand Down Expand Up @@ -1646,6 +1647,70 @@ three behaviors gets a chance to run.

---

## 32. The test suite never ran in CI, so 20 tests rotted unnoticed after the hash-store migration

**[RESOLVED 2026-09-30] (#56)**

### Problem

`npm test` was not wired into any workflow β€” `.github/workflows/` held only
`promotion.yml` β€” so nothing stopped a merge that broke tests. The
hash-store migration (#41) moved drift baselines out of the state file into
`.vapi-state-hash/<org>/<uuid>` and made the sync commands refuse legacy
state, and 20 tests failed from that merge onward without anyone noticing.

### Current behavior (Verified, before the fix)

- All 20 failures were stale tests, not engine bugs: fixtures still wrote
`lastPulledHash` / `lastPushedHash` into state (now stripped by
`asResourceState` / `upsertState`, or refused by the legacy-state gate),
the `checkDriftForUpdate` tests omitted the new required `env`, and one
test pinned `both-diverged` for the converged edge that
`src/drift.ts:50` now deliberately classifies as `clean`.
- Three of the dead tests were the regression guards #41 itself added for
#22 (dashboard rename keeps the local filename, same-name clobber) and #23
(stale baseline must not block a push). They had never passed.
- `push-stale-baseline-noop` would not have exercised its case even with a
migrated fixture: empty `credentials` makes `maybeBootstrapState`
(`src/push.ts:509-525`) treat state as uninitialized, and the bootstrap pull
rewrites the stale baseline before the drift check runs.
- The hash store resolves beside `src/` (`src/hash-store.ts:26-31`), not
under a temp dir, so in-process tests that push write baselines into the
developer's real store β€” `tool-assistant-cycle.test.ts` left one under
`.vapi-state-hash/test-fixture-org/` on every run.

### Risk

Any engine regression could merge green. The rename, clobber and
phantom-drift fixes had no working coverage.

### Current mitigation

None needed once the fix below lands; CI now fails the PR.

### Possible fix (landed)

- `.github/workflows/ci.yml` runs `npm run build` and `npm test` on every PR
and on pushes to `main`, on Node 20 and 22 (the `engines` range).
- Fixtures moved to the hash store: spawn-based tests seed
`<tmp>/.vapi-state-hash/<env>/<uuid>` (the engine runs from the copied
`src/`), `drift.test.ts` seeds through `writeBaseline` under a throwaway
org slug and removes it, and `audit.ts` gained a `baselineReader` DI seam
(`src/audit.ts:86`) beside `stateLoader` / `listLocalIds`.
- `push-stale-baseline-noop` seeds a credential so no bootstrap pull runs, and
asserts that. Removing the agree-gate now fails it, as it does the
converged-edge `classifyDrift` test.
- `tool-assistant-cycle.test.ts` deletes the baseline it writes.

### Status

**RESOLVED 2026-09-30.** Tests are still outside `tsconfig.json`'s `include`,
so `npm run build` never typechecks them and the compiler could not have
caught these stale fixture shapes. Including them surfaces 37 existing type
errors today; widening `include` is a separate change.

---

## Out of scope (intentionally not improvements)

- **State file is identity-only and not git-ignored.** It's intentionally
Expand Down
19 changes: 15 additions & 4 deletions src/audit.ts
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,11 @@ export interface AuditOptions {
// frontmatter parsing of the assistant file on disk. Sync OR async return
// is accepted so tests can keep their fixtures plain-object.
readAssistantTools?: (resourceId: string) => unknown[] | Promise<unknown[]>;
// DI seam: swap the drift-baseline lookup. Defaults to the per-developer
// hash store (.vapi-state-hash/<env>/<uuid>), which resolves beside src/
// rather than under any temp dir β€” so in-process tests must inject this
// instead of seeding baseline files.
baselineReader?: (uuid: string) => string | undefined;
}

// ─────────────────────────────────────────────────────────────────────────────
Expand Down Expand Up @@ -243,10 +248,11 @@ function checkStateUuidCollisions(
function checkContentIdentical(
type: ResourceType,
state: StateFile,
baselineReader: (uuid: string) => string | undefined,
): { findings: AuditFinding[]; identicalSlugs: Set<string> } {
const byHash = new Map<string, string[]>();
for (const [resourceId, entry] of Object.entries(state[type])) {
const hash = readBaseline(VAPI_ENV, entry.uuid);
const hash = baselineReader(entry.uuid);
if (!hash) continue;
const slugs = byHash.get(hash) ?? [];
slugs.push(resourceId);
Expand Down Expand Up @@ -368,6 +374,7 @@ function checkContentDrift(
state: StateFile,
remote: VapiResource[],
localIds: string[],
baselineReader: (uuid: string) => string | undefined,
): AuditFinding[] {
const remoteByUuid = new Map(remote.map((r) => [r.id, r]));
const credReverse = credentialReverseMap(state);
Expand All @@ -390,7 +397,7 @@ function checkContentDrift(
);
const direction = classifyDrift({
localHash,
lastPulledHash: readBaseline(VAPI_ENV, entry.uuid),
lastPulledHash: baselineReader(entry.uuid),
platformHash,
});
if (direction === "clean") continue;
Expand Down Expand Up @@ -466,6 +473,8 @@ export async function runAudit(
const remoteFetcher = opts.remoteFetcher ?? fetchAllResources;
const readAssistantTools =
opts.readAssistantTools ?? defaultReadAssistantTools;
const baselineReader =
opts.baselineReader ?? ((uuid: string) => readBaseline(VAPI_ENV, uuid));

const state = stateLoader();

Expand Down Expand Up @@ -519,13 +528,15 @@ export async function runAudit(
const remoteUuids = new Set(remote.map((r) => r.id));
findings.push(...checkStateGhosts(type, state, remoteUuids));
findings.push(...checkDashboardOrphans(type, state, remote));
findings.push(...checkContentDrift(type, state, remote, localIds));
findings.push(
...checkContentDrift(type, state, remote, localIds, baselineReader),
);
}

findings.push(...checkStateUuidCollisions(type, state));

const { findings: identicalFindings, identicalSlugs } =
checkContentIdentical(type, state);
checkContentIdentical(type, state, baselineReader);
findings.push(...identicalFindings);

findings.push(...checkSiblingBaseSlug(type, state, identicalSlugs));
Expand Down
18 changes: 17 additions & 1 deletion tests/audit.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -24,8 +24,20 @@ import type { ResourceState, ResourceType, StateFile } from "../src/types.ts";
// Helpers β€” keep fixtures DI-friendly and avoid filesystem / network.
// ─────────────────────────────────────────────────────────────────────────────

// Drift baselines live in the per-developer hash store, not the state file, so
// fixtures record them here and `baseOpts` serves them via `baselineReader`.
// A hash-less entry clears its uuid so a baseline can't leak between tests
// that reuse the same fixture uuid.
const baselines = new Map<string, string>();

function makeStateEntry(uuid: string, hash?: string): ResourceState {
return hash ? { uuid, lastPulledHash: hash } : { uuid };
if (hash) baselines.set(uuid, hash);
else baselines.delete(uuid);
return { uuid };
}

function readFixtureBaseline(uuid: string): string | undefined {
return baselines.get(uuid);
}

// All sections start empty so callers only populate the type(s) under test.
Expand Down Expand Up @@ -54,6 +66,7 @@ function baseOpts(state: StateFile) {
stateLoader: () => state,
listLocalIds: (_t: ResourceType) => [] as string[],
readAssistantTools: (_id: string) => [] as unknown[],
baselineReader: readFixtureBaseline,
};
}

Expand Down Expand Up @@ -440,6 +453,7 @@ test("inline-tools: assistant with empty model.tools array β†’ 0 findings", asyn
const findings = await runAudit({
...baseOpts(state),
readAssistantTools: () => [],
baselineReader: readFixtureBaseline,
});
const inline = findings.filter((f) => f.rule === "inline-tools");
assert.equal(inline.length, 0);
Expand All @@ -455,6 +469,7 @@ test("inline-tools: readAssistantTools returns non-array (treated as no inline t
const findings = await runAudit({
...baseOpts(state),
readAssistantTools: () => [],
baselineReader: readFixtureBaseline,
});
const inline = findings.filter((f) => f.rule === "inline-tools");
assert.equal(inline.length, 0);
Expand Down Expand Up @@ -505,6 +520,7 @@ test("integration: orphan-yaml + collision + content-identical(4) + sibling-base
// 1 orphan-yaml: a local file with no state entry.
listLocalIds: (t) => (t === "assistants" ? ["stray-local"] : []),
readAssistantTools: () => [],
baselineReader: readFixtureBaseline,
});

// Total: 1 orphan-yaml + 1 collision + 2 content-identical + 1 sibling
Expand Down
Loading
Loading