You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
fix(rag): accept non-uuid string doc_id in the qdrant client - #9597
With qdrant-client's in-memory client, QdrantClient.add_documents and aadd_documents raise ValueError: Point id doc-1 is not a valid UUID for a doc_id such as "doc-1", though BaseRecord.doc_id is a str. _create_point_from_document passes it unchanged as the point id (lib/crewai/src/crewai/rag/qdrant/utils.py:275-278 on main at 374c053920), and qdrant-client 1.14.3's in-memory client requires a string point id to parse as a UUID.
A string that is not a UUID is mapped to a uuid5 point id in its own namespace; the original is kept in the payload under crewai_doc_id.
_process_search_results returns the original as id and keeps the key out of metadata. The key is read only when it maps back to the point's id.
Integer ids, UUID strings and documents without a doc_id are stored as before.
Case
main374c053920
this PR
add_documents, "doc-1" (A)
ValueError
added; search returns 'doc-1'
aadd_documents, "doc-1" (D)
ValueError
added; asearch returns 'doc-1'
batch "7", "doc-2" (E)
ValueError, 0 stored
2 stored
"doc-1" twice (K)
ValueError
1 point, content new
UUID, no doc_id, int (F, G, H)
added
unchanged
Limit (M): an int 7 and a string "7" added in one call are two points, and both search back as '7'. Metadata that already holds crewai_doc_id together with a non-UUID doc_id raises a ValueError naming the key (T), as main already raised for that doc_id.
Verification
Tests added or updated for the changed behavior
Relevant tests and quality checks pass locally
New tests TestStringDocIds in lib/crewai/tests/rag/qdrant/test_client.py: 6 of 8 fail on main's source (the ValueError from the add), all 8 pass here, also in random order with xdist and on Python 3.10. In-memory Qdrant, no network. macOS.
ruff check and ruff format --check on the three changed files, and mypy on lib/crewai/src/crewai/rag/qdrant/utils.py: clean.
Not run: the full suite, a Qdrant server, Linux and Windows.
Additional context
Three existing tests asserted the raw id with a non-UUID string. They now use a UUID string, which still stores as the point id, and TestStringDocIds covers the other strings.
Written with an AI coding agent (Claude Code). CONTRIBUTING asks for the llm-generated label on AI-generated PRs; I can't add labels, so could a maintainer add it?
PR summary: Fixes Qdrant document ingestion failing on non-UUID string `doc_id`s by mapping them to derived `uuid5` point ids, stashing the original id in a reserved payload key, and restoring it when search results are built.
Review summary
This PR makes the Qdrant RAG client accept a non-UUID string doc_id: it is stored at a derived uuid5 point id with the original kept in a reserved payload key, and restored as the result id on search. I traced the mapping from every place a Qdrant point id is built or read (the add/upsert helpers in lib/crewai/src/crewai/rag/qdrant/utils.py, the client's collection-level delete/reset, the knowledge storage and tools adapter ingestion paths), checked who can actually supply a doc_id or metadata, and checked every consumer of a search result's id and metadata. The remaining observations are id-hygiene behaviour in the mapping rather than security defects: the fallback mapping is not injective (a non-UUID doc_id and the UUID equal to its hash reach the same point), an integer 7 and a string "7" are stored as two points that both search back as "7", and a document with a UUID-shaped doc_id can carry a caller-supplied crewai_doc_id that is returned as its id. Each requires the party supplying documents to a collection it already has write access to, and neither crosses a privilege boundary nor reaches a consumer that acts on the identifier, so no vulnerability is reported.
Ruled out (3)
Can a document overwrite another document's Qdrant point through the new doc_id mapping, since `_point_id("doc-1")` equals a UUID another caller can pass verba… The only trigger is the party that supplies the record to `add_documents`, which can already name the victim's point id directly (the id was used verbatim before this change) and can call `delete_collection`/`reset`, so no access is gained that does not already permit the same harm.
Can a caller-supplied `crewai_doc_id` metadata value, or a UUID-shaped doc_id paired with it, make a search result report another document's id? The restore path only trusts the payload value when `_point_id(stored)` hashes back to the point's own id, and no code in the repository reads `SearchResult["id"]` or `metadata["crewai_doc_id"]`, so a relabelled id reaches no decision.
Does an untrusted end user reach the new mapping, i.e. can a user or tenant supply `doc_id`/`metadata` on any in-repo ingestion path? Both write paths generate their own ids - `knowledge_storage.py` passes `{"content": doc}` records with no `doc_id`, and the tools adapter uses `str(uuid.UUID(sha256(...)[:32]))` - and the client exposes no per-id retrieve, update or delete that a spoofed id could be aimed at.
Review 1 on this PR · d898f32 · requested Nov 20 07:32 UTC when the PR was opened
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related issue
Fixes #9596
Summary
With qdrant-client's in-memory client,
QdrantClient.add_documentsandaadd_documentsraiseValueError: Point id doc-1 is not a valid UUIDfor adoc_idsuch as"doc-1", thoughBaseRecord.doc_idis astr._create_point_from_documentpasses it unchanged as the point id (lib/crewai/src/crewai/rag/qdrant/utils.py:275-278onmainat374c053920), and qdrant-client 1.14.3's in-memory client requires a string point id to parse as a UUID.uuid5point id in its own namespace; the original is kept in the payload undercrewai_doc_id._process_search_resultsreturns the original asidand keeps the key out ofmetadata. The key is read only when it maps back to the point's id.doc_idare stored as before.main374c053920add_documents,"doc-1"(A)ValueError'doc-1'aadd_documents,"doc-1"(D)ValueErrorasearchreturns'doc-1'"7","doc-2"(E)ValueError, 0 stored"doc-1"twice (K)ValueErrornewdoc_id, int (F, G, H)Limit (M): an int
7and a string"7"added in one call are two points, and both search back as'7'. Metadata that already holdscrewai_doc_idtogether with a non-UUIDdoc_idraises aValueErrornaming the key (T), asmainalready raised for thatdoc_id.Verification
Tests added or updated for the changed behavior
Relevant tests and quality checks pass locally
New tests
TestStringDocIdsinlib/crewai/tests/rag/qdrant/test_client.py: 6 of 8 fail onmain's source (theValueErrorfrom the add), all 8 pass here, also in random order with xdist and on Python 3.10. In-memory Qdrant, no network. macOS.pytest lib/crewai/tests/rag/qdrant lib/crewai/tests/knowledge lib/crewai/tests/rag/test_error_handling.py: 177 passed.ruff checkandruff format --checkon the three changed files, andmypyonlib/crewai/src/crewai/rag/qdrant/utils.py: clean.Not run: the full suite, a Qdrant server, Linux and Windows.
Additional context
Three existing tests asserted the raw id with a non-UUID string. They now use a UUID string, which still stores as the point id, and
TestStringDocIdscovers the other strings.Written with an AI coding agent (Claude Code). CONTRIBUTING asks for the
llm-generatedlabel on AI-generated PRs; I can't add labels, so could a maintainer add it?