[python] Add a table vector-search CLI command - #10163
jackylee-ch wants to merge 1 commit into
Conversation
| df = read.to_pandas(splits) | ||
|
|
||
| output_format = getattr(args, 'format', 'table') | ||
| if output_format == 'json': |
There was a problem hiding this comment.
--format json is not usable with the default vector-search projection. Arrow list/vector columns become numpy.ndarray values in pandas, and json.dumps(df.to_dict(...)) raises TypeError: Object of type ndarray is not JSON serializable . Since the searched vector column is included by default, this is the normal JSON path rather than an edge case. Please serialize from Arrow/Python-native values (for example to_pylist() ) or normalize NumPy arrays/scalars before calling json.dumps, and add a successful JSON-output test that includes the embedding column.
Validation: Direct parser probes passed. I reproduced both the ranking loss ( bitmap_order=[2,10] , score_order=[10,2] ) and the vector JSON ndarray failure. The catalog-backed CLI tests cannot initialize locally because of the repository’s Windows URI issue ( Unrecognized filesystem type in URI: c ). GitHub’s failing Python jobs are unrelated and all report the existing interval_partition_test.py SimpleNamespace.path_factory error.
61fd23f to
6d07fb7
Compare
The pypaimon CLI exposes `table full-text-search` but has no equivalent for vector search, even though the vector search API (`new_vector_search_builder`) is already available. Users who want to run a nearest-neighbor query from the command line have to fall back to writing Python. Add a `table vector-search` command that mirrors `full-text-search`: it takes the vector column, a query vector (comma-separated floats or a JSON-style array), an optional result limit, column selection and output format, runs the search locally and prints the matching rows.
6d07fb7 to
9ebd735
Compare
|
Thanks @Akash3121. JSON: fixed — added Ranking: |
2 participants
Purpose
The pypaimon CLI already exposes
table full-text-search, but there is no equivalent for vector search even though the vector search API (Table.new_vector_search_builder()) is fully implemented. To run a nearest-neighbor query from the command line users currently have to fall back to writing Python.This adds a
table vector-searchcommand that mirrorsfull-text-search: it takes the vector column (--column), a query vector (--query, given as comma-separated floats or a JSON-style array), an optional result limit (--limit, default 10), column selection (--select) and output format (--format table|json), runs the search locally and prints the matching rows.Tests
pypaimon/tests/cli_table_test.py: added unit tests for the query-vector parser (comma / JSON array / whitespace / empty / non-numeric) and CLI tests for the error paths (invalid vector, unknown column, unknown table).API and Format
No public API or format change; new CLI subcommand only, no default changed.
Documentation
Consistent with the existing
full-text-searchCLI command;--helplists the new command and its options.