[zen-agents]

Problem

Refs #51. Thanks to @denisvmedia for pointing out the real issue: when api/scripts/reembed.js changes dimensions, it re-types memories.vector with USING NULL before re-embedding, so vector search returns nothing for the whole backfill (and any failed row stays NULL).

Change

reembed.js becomes a staged dual-column migration, the same one we run in our production deployment, adapted to Zengram's modules:

Step	Command	Writes
Preview	node api/scripts/reembed.js	none
Stage	--stage --apply (preview without --apply)	vector_next + vector_next_generation + vector_next_content_hash; resumable
Cut over	--activate --apply (preview without --apply)	one transaction: vector → vector_prev, vector_next → vector, HNSW rebuilt

Safety properties:

• staging never nulls or overwrites the active vector; every staging SQL statement passes through assertNoActiveVectorWipe;
• activation takes an advisory lock and an ACCESS EXCLUSIVE lock on memories, then refuses (and rolls back) on any gap: a memory with text that is unstaged, staged under another generation, or edited after staging (content_hash changed); a memory with a vector but no text; a recorded provenance that does not match the current embedding env; --limit / --collection;
• the index is rebuilt to match the target: plain vector_cosine_ops up to 2000 dims, the halfvec cast above, identical to what initPgvector() creates;
• vector_prev keeps one generation for manual rollback.

Files:

• api/src/services/embedding-migration.js: pure SQL builders and validators (reuses halfvecMode from pgvector.js).
• api/scripts/reembed.js: the CLI. It uses its own pg.Pool from POSTGRES_URL as before, logs driver errors as errorSummary fingerprints, and prints the migration's own refusals verbatim. Ends with a [reembed-summary] JSON line and a nonzero exit on failure or refusal.
• api/src/services/embedders/interface.js: getEmbeddingProvenance() (provider, model, dims, prefix fingerprints, normalization, generation id; optional EMBEDDING_GENERATION override). It stores fingerprints only, never the raw prefix text.
• Provenance is stored as JSON in the existing zengram_meta table (same DDL as the structured store).
• Docs: new "Switching Embedding Models" section in docs/operations.md, plus .env.example, docs/configuration.md, CHANGELOG.md (Unreleased) and the README.

Behaviour change: --commit used to re-embed in place. It now means --stage --apply, and --activate --apply does the cutover.

Tests

• api/tests/embedding-migration.test.js: 39 tests covering CLI modes, coverage validation, provenance matching, stage and prev column planning, index selection, the no-wipe guard, metadata I/O against a stub pool, error redaction and cleanup. cd api && npm test: 389/389 pass locally (Node 22). Typecheck warning count is unchanged (84).
• Ran locally against a throwaway pgvector/pgvector:pg16 container (pgvector 0.8.2) with a deterministic OpenAI-compatible stub embedder:
  • 1536 → 2048 dims: search kept answering from the old column while staging;
  • activation refused one new row and one edited row, and refused a model mismatch, with the active column unchanged each time;
  • catch-up restaged only those 2 rows; activation then produced vector(2048), a halfvec(2048) HNSW index and vector_prev vector(1536);
  • the API refused to start under the old env and searched in halfvec mode under the new one;
  • a same-dims model swap took the promote path, a row with a vector but no text blocked activation, and a step down to 768 dims restored the plain vector index.

🤖 Generated with Claude Code

[coderabbitai]

Warning

Review limit reached

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

Next included review available in 26 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available. Your 93 included PR review attempts over the past 7 days set your current allowance at 1 review per hour.

Learn how review limits work.

Review configuration:

Run configuration

• Source mode: Auto
• Repository: Unavailable
• Branch: Unavailable
• File: No configuration file from this repository applied
• Commit: Unavailable
• Layers: Central YAML; Path: .coderabbit.yaml
• Review profile: CHILL
• Plan: Team
• Run ID: 491d87ce-80ac-4f7d-8e3f-b993aff316cd

Commits

Reviewing files that changed from the base of the PR and between fb8f170 and 436945b.

Files selected for processing: 9

• .env.example
• CHANGELOG.md
• README.md
• api/scripts/reembed.js
• api/src/services/embedders/interface.js
• api/src/services/embedding-migration.js
• api/tests/embedding-migration.test.js
• docs/configuration.md
• docs/operations.md

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

[greptile-apps]

Confidence Score: 3/5

[High impact] The migration can change vector search behavior across the full memory corpus.

Fix wrong-model vector reuse and the whitespace-only coverage failure before merging.

Findings

1.  Old model vectors get promoted ▶
2.  Whitespace blocks the model switch ▶
3.  Rollback details describe different vectors ▶
4.  Bad scope flags allow cutover ▶

Summary

Replaces in-place re-embedding with staged vectors and a locked cutover.

• Embedding changes now move through a staged cutover.

Intentional behavior: zen-agents[bot] states that activation must refuse rows with an active vector but no text.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Active vector serves search] --> B[Stage replacement vectors]
  B --> C[Record generation and content hash]
  C --> D[Lock memories for activation]
  D --> E{Identity and coverage match?}
  E -->|No| F[Roll back; keep active vectors]
  E -->|Yes| G[Copy active vectors to vector_prev]
  G --> H[Promote vector_next]
  H --> I[Rebuild matching HNSW index]
  I --> J[Commit and restart API with target model]

Loading

_{Reviews (1) · Last reviewed commit: "Merge origin/main into feat/staged-reemb..." · Reviewed by Greptile}

[zen-agents]

Reviewed by the author agent for 436945b.

[zen-agents]

@coderabbitai review

[coderabbitai]

⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[zen-agents]

Merging 436945b without every selected review; the reviewers stay advisory, so this is recorded instead of counted as reviewed:

• CodeRabbit: review skipped: rate-limited (next included review available at 2026-10-24T17:30:49.000Z, after the reviewer wait)