Skip to content

feat: add indexed pgvector file search storage - #278

Open
franciscojavierarceo wants to merge 2 commits into
mainfrom
codex/file-search-pgvector
Open

franciscojavierarceo wants to merge 2 commits into
mainfrom
codex/file-search-pgvector

Conversation

@franciscojavierarceo

@franciscojavierarceo franciscojavierarceo commented Sep 11, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Port the pgvector layer onto current main as a standalone PR, using the provider contracts shipped with the earlier file-search PRs. This replaces the obsolete foundation/executor changes in the old stacked diff.

  • Add PgvectorRepository, implementing the existing FileSearchRepository contract and composing with any FileBlobStore. Metadata publication, deletion, pagination, and durable blob cleanup reuse the SQL repository.
  • Add typed [file_search.pgvector] configuration for HNSW/IVFFlat, dimensions, candidate limits, and recall controls. SQL exact search and sqlite-vec retain their existing defaults.
  • Project unit-normalized float32 vectors transactionally from stored chunk JSON; use native distance/FTS candidates, typed SQL filters, bounded reads, and the existing final cosine/BM25/fusion ranker. IVFFlat waits for lists * 1000 training rows and uses database distance selection until ready.
  • Share metadata and cleanup forwarding through FileSearchRepository::metadata() defaults, retaining provider-specific publication, search, and sqlite-vec deletion.
  • Put optional PostgreSQL DDL and index maintenance under storage/schema/pgvector.rs. Supervisor mode verifies schema and index readiness without DDL, including deferred IVFFlat training.
  • Document startup DDL privileges, backfill/index locks, construction consistency across replicas, resource limits, approximate retrieval behavior, and store recreation when changing providers or dimensions.
  • Pin the PostgreSQL CI service to pgvector 0.8.2/PG17 and exercise the provider and HTTP/Responses paths with the official OpenAI client.

The older follow-up drafts (#280–#284) are preserved; their remaining features need separate ports onto the current architecture. This PR contains only the pgvector adapter, wiring, documentation, and validation.

Test Plan

Local validation uses PostgreSQL 16.15, pgvector 0.8.2, and a fresh database with English ICU collation. External inference and embeddings are deterministic fixtures; the gateway, HTTP APIs, SQL transactions, and vector indexes are real.

  • cargo test --workspace: passed (3140 tests; 35 ignored tests, with relevant database suites exercised separately).
  • cargo clippy --workspace --all-targets -- -D warnings: passed.
  • pre-commit run --all-files: passed, including Rust file-size checks and rustfmt.
  • pgvector_repository: 2 configuration tests and 6 real-database scenarios passed. Covers native HNSW/IVFFlat query plans, training fallback/rebuild, backfill/restart, all typed filter operators, SQL parameter binding, isolation, deletion, rollback, candidate limits, extreme float32 magnitudes, and concurrent replica initialization/retrieval.
  • Review follow-up (813f81a): all four comments addressed; local workspace tests, all-target Clippy, and pre-commit passed. All 6 pgvector integration tests and 4 sqlite-vec integration tests passed. Three new PostgreSQL tests passed, covering read-only supervisor startup, operator-owned IVFFlat training, and 418 filters across the same 37 chunks through SQL pushdown and SearchFilter::matches. CI now runs these new tests. Provider exclusivity is checked once in service setup.
  • Baseline validation at 3666ac5 — existing PostgreSQL suites: 4 schema tests, 5 persistence tests, and 1 portable file-search lifecycle test passed without enabling the vector extension in that database.
  • Baseline validation at 3666ac5 — official OpenAI 3.8.0 client: Files/vector stores/search passed against pgvector semantic/hybrid and SQLite keyword. All 9 citation fixtures passed over JSON/SSE, with and without search results, persisted retrieval, and continuation on both backends.

Baseline GitHub CI passed on 3666ac5 (new checks are running for review-fix commit 813f81a): Rust, PostgreSQL, sqlite-vec, embedded interpreter, and MSRV, coding CLIs, container smoke test, Python validation, all three Linux/macOS wheel builds, and pre-commit passed.

ANN candidate selection and candidate-bounded keyword scores can differ from full-corpus exact ranking; no benchmark or proprietary-model parity claim is made here.

franciscojavierarceo added a commit that referenced this pull request Nov 3, 2026
## Summary

Add typed file, vector-store, and search contracts; pluggable
`FileSearchRepository` and `FileBlobStore` interfaces; portable SQL
metadata and bounded exact retrieval; and local blob storage. Store
listing batches current attachment counts and usage. Durable,
token-fenced cleanup intents cover uploads and deletions; expired leases
cannot let a publisher expose missing content. Recovery retains
abandoned-upload tombstones for later sweeps when a delayed write
finishes after an initial cleanup pass. Local staging is addressable by
file ID, and cancellation drains publication before cleanup. This is the
independently compilable foundation of the file-search stack; HTTP
routes and periodic cleanup are in #485. CI uses the macOS 15 Apple
Silicon runner because GitHub is retiring macOS 14 and its scheduled
brownout canceled wheel validation.

Review order: #483 → #484 → #485 → #486 → #34. PostgreSQL vector
indexing remains in follow-up #278.

## Test Plan

- `cargo test --offline -p agentic-server-core --test
file_search_service`: 31 passed, 1 PostgreSQL test ignored locally.
Includes two-service upload/recovery interleavings, restart recovery
after metadata rejection plus failed blob removal, delayed-staging
cancellation, and a late-write/failed-rollback regression.
- `cargo test --offline -p agentic-server-core --lib
storage::local_files::tests`: 5 passed, including absent-directory and
deterministic staging cleanup.
- `cargo test --offline -p agentic-server-core --lib
storage::schema::tests`: 6 passed, 4 PostgreSQL tests ignored locally;
PostgreSQL CI exercises the final schema.
- `cargo clippy --offline --all-targets -- -D warnings`, `cargo fmt --
--check`, and repository pre-commit hooks: passed.
- The official OpenAI Python SDK HTTP integration test is in #485 and
passed locally against the rebuilt final stack.

---------

Signed-off-by: Francisco Javier Arceo <4163062+franciscojavierarceo@users.noreply.github.com>
@franciscojavierarceo
franciscojavierarceo force-pushed the feat/axum-ogx-integration-v2 branch 11 times, most recently from 5f11379 to d24e3b0 Compare November 6, 2026 03:45
franciscojavierarceo added a commit that referenced this pull request Nov 6, 2026
## Summary

Add a `FileSearchRepository` backed by sqlite-vec for semantic cosine
retrieval on a file-backed SQLite database. It keeps portable SQL
metadata and published-chunk visibility, normalizes embeddings before
indexing, and records a provider identity so stores cannot silently
switch indexes. Keyword and hybrid search continue through the bounded
exact ranker. SQLite metadata and vector rows now publish in one
transaction, so concurrent replicas cannot remove a winning attachment.
Orphan cleanup rechecks rows under a writer lock and immediately
reclaims stale rows and per-store tables; search skips empty stores and
unpublished rows, and uses bounded exact ranking when sqlite-vec's
4,096-candidate limit is exceeded. The loadable extension path is
deployment-controlled; #485 wires its configuration.

BEIR-sized corpus testing exposed the old 64 MiB serialized-corpus limit
and top-k edge cases with duplicate chunks and near ties. This branch
keeps the default exact-search limit at 64 MiB and gives sqlite-vec a
256 MiB limit, validates vector-index coverage, and adaptively fetches
candidates to preserve the requested document count.

Review order: #483 → #484 → #485 → #486 → #34. PostgreSQL vector
indexing remains in #278.

## Test Plan

- Review follow-up: the portable vector map is migration-owned;
extension-specific index SQL lives in
`storage/file_search/sqlite_vec_index.rs`, publication has one
implementation, and writes use `BEGIN IMMEDIATE` without a process-wide
mutex. Focused storage and schema tests passed; the original 3
sqlite-vec integration tests passed, including immediate orphan-row and
orphan-table recovery.
- `SQLITE_VEC_EXTENSION_PATH=/tmp/vec0.so cargo test -p
agentic-server-core --test sqlite_vec_repository -- --ignored`: 4 passed
in CI with sqlite-vec 0.1.9, covering publication, extreme finite
embeddings, restart, filtered and duplicate-chunk search, deletion,
concurrent attachment conflicts, unpublished rows, empty stores, and
searches beyond the 4,096-candidate limit.
- `cargo test -p agentic-server-core --lib`: 1,370 passed, 6 ignored.
`cargo clippy -p agentic-server-core --all-targets -- -D warnings`,
`cargo fmt -- --check`, and pre-commit hooks passed.
- Maral's capacity regression: a new sqlite-vec integration test
publishes a serialized store above 64 MiB, then exercises keyword,
hybrid, and filtered semantic searches against it. The Rust CI run
passed this test and the full ignored sqlite-vec integration suite.
- CI downloads the pinned Linux sqlite-vec release asset, verifies its
SHA-256, and runs the ignored integration tests.

### BEIR retrieval evidence (2026-11-03, benchmark commit `fd6b574`)

Compared the provider's semantic search with [OGX's published
vector-mode
results](https://ogx-ai.github.io/docs/building_applications/rag_benchmarks).
Used the official BEIR **test** corpora, queries, and relevance
judgments, title + body text, `cl100k_base` 512-token chunks with
128-token overlap, `nomic-ai/nomic-embed-text-v1.5` embeddings, and
top-10 document scoring with `pytrec_eval`. The 512/128 setting comes
from [OGX's documented benchmark
config](https://ogx-ai.github.io/docs/building_applications/rag_benchmarks),
but this was a local preparation, not [OGX's full API ingestion
path](https://github.com/ogx-ai/ogx/tree/12a51077cdd4f89b16df5eb6de12a88b0bfcf7df/benchmarking/rag).
A local harness loaded the prepared chunks into the actual
`SqliteVecRepository` and called its `search` method for every judged
query. sqlite-vec 0.1.9 ran on CPU.

| Dataset | Documents / chunks / queries | nDCG@10 OGX vector → ours |
Recall@10 OGX vector → ours | MAP@10 OGX vector → ours |
| --- | ---: | ---: | ---: | ---: |
| NFCorpus | 3,633 / 4,921 / 323 | 0.3106 → **0.3101** | 0.1482 → 0.1462
| 0.1153 → 0.1137 |
| SciFact | 5,183 / 6,604 / 300 | 0.6943 → **0.6968** | 0.8369 → 0.8296
| 0.6442 → 0.6496 |
| ArguAna | 8,674 / 9,278 / 1,406 | 0.3765 → **0.3928** | 0.7610 →
0.8037 | 0.2542 → 0.2624 |

All 2,029 searches succeeded. An independent exhaustive NumPy cosine
calculation on the same embeddings gave identical aggregate nDCG@10 for
SciFact and ArguAna; NFCorpus differed by 0.000084. This validates
sqlite-vec against an exact-cosine oracle on our prepared vectors. The
proximity to OGX's published vector scores is indicative, not exact
parity: OGX's current batch ingestion path applies whitespace cleaning
and appears to use 800/400-token file-processor defaults for automatic
chunking, despite the benchmark config documenting 512/128. The
published run does not include materialized chunk/embedding or per-query
result fixtures, so its exact input and rankings cannot yet be compared.

**Scope and limit:** This is a repository-level semantic retrieval test
with prepared embeddings; it does not exercise the Files/Vector Stores
HTTP APIs, automatic ingestion, hybrid reranking, Responses
`file_search`, or end-to-end answer generation. OGX's fourth BEIR
dataset, FiQA, produces 63,129 chunks and exceeds this branch's
10,000-chunk corpus bound, so full four-dataset parity remains
unvalidated.

---------

Signed-off-by: Francisco Javier Arceo <4163062+franciscojavierarceo@users.noreply.github.com>
@franciscojavierarceo
franciscojavierarceo force-pushed the feat/axum-ogx-integration-v2 branch 5 times, most recently from 2f85dcc to 487b73e Compare November 6, 2026 18:51
@franciscojavierarceo
franciscojavierarceo force-pushed the feat/axum-ogx-integration-v2 branch from 487b73e to f6a8f45 Compare November 6, 2026 19:43
franciscojavierarceo added a commit that referenced this pull request Nov 7, 2026
## Summary

Expose the Files and vector-store upload, lifecycle, attachment, and search routes over the pluggable core service. Add server configuration for local file bytes, embeddings, and an optional trusted sqlite-vec extension path. Wire the routes into authentication and OpenAPI, with deployment documentation. Reconcile abandoned blob cleanup intents at startup and periodically while the gateway runs. Local routes enforce the configured API key and request-body ceiling; deletion tolerates a cleanup marker already removed by reconciliation, and startup logs provider setup diagnostics. This layer does not register the Responses `file_search` built-in tool.

Review order: #485 → #486 → #34. #483 and #484 are merged. PostgreSQL vector indexing remains in #278.

## Test Plan

- `cargo test -p agentic-server --test file_search_http_test`: 6 passed, including missing, incorrect, and valid API-key cases and configured multipart/JSON body limits.
- `cargo test -p agentic-server-core --test file_search_service`: 32 passed, 1 ignored, including the deletion cleanup-marker race regression.
- Full workspace `cargo test --offline` passed on the restacked #34 head containing these fixes.
- `cargo test -p agentic-server --lib openapi`: 19 passed.
- `cargo test -p agentic-server --bin agentic-server config_file::tests`: 37 passed.
- `cargo build --offline -p agentic-server --bin agentic-server` and `cargo clippy --offline -p agentic-server -p agentic-server-core --all-targets -- -D warnings`: passed.
- `openai==3.8.0` client E2E: upload, list/retrieve/content, vector-store attachment, filtered search, detach, and deletion passed against the rebuilt local gateway. The Rust CI job runs this script.
- Formatting and repository pre-commit hooks passed on signed-off commits. CI also runs PostgreSQL file-search persistence, sqlite-vec, and bounded PDF ingestion checks.
- OpenAI reference cassette parity for Files and vector-store routes remains unverified: this environment has no OpenAI API credential. The focused local gateway HTTP suite and OpenAI SDK client E2E cover the route lifecycle; no reference cassette was hand-authored.


Signed-off-by: Francisco Javier Arceo <4163062+franciscojavierarceo@users.noreply.github.com>
@franciscojavierarceo
franciscojavierarceo force-pushed the feat/axum-ogx-integration-v2 branch 2 times, most recently from 8e187f2 to 7568050 Compare November 7, 2026 00:40
franciscojavierarceo added a commit that referenced this pull request Nov 8, 2026
## Summary

Register and execute the Responses `file_search` built-in tool through the existing typed gateway tool loop. The handler searches the configured file-search service, keeps retrieved passages private for model continuation, exposes `file_search_call.results` only when requested, and restores typed file citations in completed and streamed output. This is the top layer of a smaller stack; the Files and vector-store APIs, provider contracts, SQLite-vec adapter, and protocol types are reviewed below it.

Review order: #483 (storage contracts and portable SQL/local providers) → #484 (SQLite-vec) → #485 (Files and vector-store HTTP APIs and provider configuration) → #486 (Responses protocol and stream lifecycle) → this PR. The existing PostgreSQL/pgvector follow-up is #278; later retrieval and lifecycle work remains stacked above it.

The storage service accepts injected `FileSearchRepository` and `FileBlobStore` implementations. The lower layers configure SQLite-vec for semantic retrieval; Milvus and remote blob stores remain adapter extension points.

## Test Plan

- `cargo test -p agentic-server-core --test file_search_tool_test`: 6 passed, covering blocking and streaming calls, citations, opt-in results, mixed client tools, failures, and selector validation.
- `cargo test -p agentic-server-core --lib`: 1,385 passed, 6 ignored on the corrected, restacked file tree. Full workspace `cargo test --offline` also passed on that tree.
- Changed-crate all-target Clippy with warnings denied, formatting, and all-file pre-commit hooks passed on the corrected, restacked file tree.
- The failing Rust CI check at `33b20ba` came from an output-item test fixture missing the typed `logprobs: []` field. The corrected citation projection test passes locally; CI is running on the cleanly restacked head.
- SQLite-vec review fixes from #484 are carried through the stack: atomic publication, safe orphan cleanup, empty/unpublished row handling, and exact fallback above sqlite-vec's candidate limit.

Signed-off-by: Francisco Javier Arceo <arceofrancisco@gmail.com>
Signed-off-by: Francisco Javier Arceo <4163062+franciscojavierarceo@users.noreply.github.com>
@franciscojavierarceo
franciscojavierarceo changed the base branch from feat/axum-ogx-integration-v2 to main November 15, 2026 23:43
@franciscojavierarceo
franciscojavierarceo marked this pull request as ready for review November 15, 2026 23:55

@maralbahari maralbahari left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for porting this onto the current provider contract. Reusing FileSearchStorage for metadata and blob cleanup, the existing ranker, and the typed [file_search.pgvector] config all fit the architecture. Two things need to change before this can merge.

1. The provider repeats the metadata pass-throughs instead of reusing them. PgvectorRepository implements all 21 FileSearchRepository methods, and 17 of them only forward to self.metadata (FileSearchStorage). SqliteVecRepository already carries 16 of the same forwards. The only methods that differ per provider are index_identity, create_store, attach, search, and delete for sqlite-vec. Please let the trait provide the metadata methods once, for example as default methods over a fn metadata(&self) -> &FileSearchStorage, so each provider implements only its index operations. Then drop sqlite-vec's copies in the same PR, so adding pgvector shrinks the provider code instead of doubling it.

2. Startup alters a table the migrations own. PgvectorStorage::initialize_schema adds a generated embedding column and a GIN index to file_search_chunks, and runs CREATE EXTENSION, from storage/file_search/pgvector.rs. Schema changes go through migrations/ and backend-specific steps through storage/schema.rs (ARCHITECTURE.md, "Schema, migrations, and where data goes"). The sqlite-vec provider creates only its own table at startup and leaves file_search_chunks alone. Please do the same for pgvector: keep the projection in its own table, written in the same transaction as the chunks and deleted with them, or move the PostgreSQL-only DDL into schema.rs. Also state what happens in supervisor mode (AGENTIC_API_SCHEMA_READY), where operators migrate the schema and the gateway only verifies it. The provider currently runs its DDL either way.

Two smaller points are inline: the SQL filter pushdown needs a test that keeps it equivalent to SearchFilter::matches, and the sqlite-vec/pgvector exclusivity check is done twice.

Share repository metadata defaults across SQL, sqlite-vec, and pgvector. Move PostgreSQL DDL and index maintenance into the schema owner with read-only supervisor verification. Add differential SQL/Rust filter coverage and keep provider exclusivity validation in service setup.

Signed-off-by: Francisco Javier Arceo <4163062+franciscojavierarceo@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants