Repository navigation
feat: build embedding replacements beside active vectors with Ptah - #51
denisvmedia wants to merge 1 commit into
Conversation
|
PR author is not in the allowed authors list. |
|
Important Review skippedAuto reviews are limited based on label configuration. 🏷️ Required labels (at least one) (2)
Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository: ZenSystemAI/coderabbit/.coderabbit.yaml Review profile: CHILL Plan: Advanced Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
There was a problem hiding this comment.
Copilot review overview
🔵 Needs a closer look
The implementation is well tested, but its external migration tooling, table rewrite, and coordinated production cutover warrant final human operational review.
Review effort: Balanced
Findings: None
What changed in this PR
Adds an opt-in Ptah workflow for building and validating replacement embeddings without disrupting active vector search.
Changes:
- Makes pgvector reads and writes select a validated embedding column.
- Adds Ptah migration configuration, documentation, and integration testing.
- Prevents legacy re-embedding against candidate columns.
| File | Description |
|---|---|
README.md |
Links the migration guide. |
docs/embedding-generations.md |
Documents preparation, cutover, and retirement. |
api/src/services/pgvector.js |
Routes vector operations through the selected column. |
api/scripts/reembed.js |
Rejects candidate-column configurations. |
api/tests/storage-pgvector-helpers.test.js |
Tests column validation and SQL generation. |
api/scripts/ptah/verify.js |
Exercises the end-to-end migration workflow. |
api/scripts/ptah/source.sql |
Adds Ptah’s generated text source. |
api/scripts/ptah/embedding-v2.json |
Provides an example Ptah specification. |
.github/workflows/ci.yml |
Adds PostgreSQL/Ptah integration CI. |
.env.example |
Documents PGVECTOR_COLUMN. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
* feat(reembed): stage replacement vectors beside the active column reembed.js used to clear memories.vector before re-embedding when the dimensionality changed, so vector search returned nothing until the backfill finished (raised in #51). Replace it with a staged dual-column migration: - dry-run by default; --stage --apply embeds into vector_next with the embedding generation and each row's content_hash, resumably; - --activate --apply checks full coverage and the recorded provenance under an advisory lock and an ACCESS EXCLUSIVE table lock, copies the current vectors to vector_prev, promotes vector_next in one transaction and rebuilds the HNSW index (halfvec above 2000 dims); - activation refuses partial coverage, rows edited after staging, rows with a vector but no text, a provenance mismatch and --limit or --collection; the active column is never nulled while staging. Pure SQL builders and validators live in services/embedding-migration.js (reusing pgvector.js's halfvecMode); getEmbeddingProvenance() joins the embedder interface; provenance is kept in zengram_meta. Driver errors are logged as fingerprints, the migration's own refusals verbatim. Behaviour change: --commit now means --stage --apply. Refs #51 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(reembed): address review findings on the staged migration - Refuse to stage when EMBEDDING_GENERATION is pinned but the embedding identity changed: vectors staged under the old model would otherwise be reused and promoted under the new one (checked before any write, and reported by the --stage preview). - Use one embeddable-text rule in the CLI and the coverage SQL (first non-empty text field; whitespace-only text counts, as the API embeds it), so a whitespace-only memory no longer blocks activation forever. - Keep embedding_migration_previous paired with vector_prev: staging records the active identity under embedding_migration_pending_previous and only activation moves it, in the cutover transaction. - Reject malformed --limit/--collection values, and refuse --activate when either flag is supplied at all. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: zen-agents[bot] <302351347+zen-agents[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
Thank you, @denisvmedia, for this PR and for the care that went into it. The write-up, the integration test against real pgvector, and the handover procedure made the problem very clear. You were right about the core issue: when
We chose to keep the migration inside Zengram's own script instead of adding Ptah to operations. That is only because we want to keep the operator toolchain to what the repository already ships, not a judgement on Ptah, which looks like a thoughtful tool. So I'm closing this PR in favour of #64. Thanks again, and feedback on the staged approach is very welcome. |
|
Thank you for your considerations. If you decide to revise it in the future, please take a look to the article we wrote on this case: https://blog.ptah.run/posts/zengram-embedding-generations/ |
When
reembed.jschanges dimensions, it clears the active vector column before rebuilding the corpus. This adds an optional Ptah workflow that builds a replacement beside it, so the current encoder and search can keep serving during backfill.Hi, I'm Denis, a maintainer of Ptah. We want to make embedding model changes easier to verify on real application workloads. Ptah is open source; the site provides documentation and a browser playground for trying its schema commands without installation. Related proposal: #50.
The application change is confined to the pgvector storage layer:
PGVECTOR_COLUMNselects the column for search, ordinary upserts, and atomic fact/status supersession. It defaults to the existingvectorcolumn. Startup validates the selected column and dimension.reembed.jsrefuses a selected candidate column, preventing it from accidentally operating on the legacy column with the new encoder configuration.The guide includes the diagram, specification, commands, and handover procedure. Pause writers for final catch-up and deploy the encoder and column together. Zengram does not automatically follow Ptah's pointer; old vectors alone do not provide rollback after writes resume.
Validation passed on the submitted commit in fork CI: existing tests and API security audit on Node 20/22, plus the new PostgreSQL/pgvector integration job with released Ptah 0.11.4. The test checks a 1536-to-384 dimension change, old search while candidate embedding is in flight, provider failure/retry, preservation of old vectors/payloads/index, insert/update/delete catch-up, explicit approval, both new-column write paths, and tenant/collection filters. Its deterministic local provider tests migration mechanics, not model quality. The existing typecheck step remains non-blocking as configured upstream.