Skip to content

Fix Arrow filter pushdown on dictionary columns and on zoned timestamps in seconds or milliseconds - #677

Merged
evertlammerts merged 2 commits into
v2.0-cyanopterafrom
fix-arrow-dictionary-pushdown
Oct 17, 2026
Merged

evertlammerts merged 2 commits into
v2.0-cyanopterafrom
fix-arrow-dictionary-pushdown

Conversation

@evertlammerts

Copy link
Copy Markdown
Member

Fixes #664, plus a wrong-result bug found on the way.

A pushed comparison on a dictionary-encoded Arrow column with binary, decimal or zoned timestamp values dereferenced null type info, raised an InternalException and invalidated the database. The filter walker received the column's index-side ArrowType; the engine stores the value type next to it. The walker now resolves a dictionary column, and a dictionary struct child, to its value ArrowType and Arrow schema node before building the scalar, as the engine's own scan does.

Separately, a pushed TIMESTAMPTZ comparison on a column stored in seconds or milliseconds truncated the microsecond constant into the column's unit, so a constant between two grid points returned wrong rows (plain columns too, on 1.5.6 as well). The scalar is now built in microseconds and labelled with the column's own zone, read from the Arrow schema the scan exports; pyarrow compares across units exactly, and ordinary constants still prune Parquet row groups.

Also:

  • NaN handling on float and double dictionaries compares the column with itself instead of calling is_nan, which pyarrow has no dictionary kernel for. This removes 24 expected failures in the existing NaN pushdown tests.
  • Run-end encoded columns are refused with a clear error instead of invalidating the database; pyarrow has no kernels for them.
  • A comparison on a field of a dictionary of struct raises pyarrow's error; pyarrow cannot reference a field through a dictionary.

Known limitations, pinned as expected failures: a constant finer than the column's unit loses Parquet row group pruning (correct rows, full scan); IN with an infinite zoned constant fails because the Python conversion turns infinity into a zoneless datetime. The polars backend has the same truncation since before this change and is out of scope.

@evertlammerts
evertlammerts force-pushed the fix-arrow-dictionary-pushdown branch 2 times, most recently from 3e2b05f to 4add9ba Compare October 16, 2026 13:41
@evertlammerts
evertlammerts force-pushed the fix-arrow-dictionary-pushdown branch from 4add9ba to 08120cd Compare October 16, 2026 13:43
@evertlammerts
evertlammerts added this pull request to the merge queue Oct 16, 2026
Merged via the queue into v2.0-cyanoptera with commit 581c32d Oct 17, 2026
32 checks passed
@evertlammerts
evertlammerts deleted the fix-arrow-dictionary-pushdown branch October 17, 2026 04:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Arrow filter pushdown: comparing a dictionary-encoded binary, decimal or TIMESTAMPTZ column raises an InternalException

1 participant