Repository navigation
perf(reader): read a TopK's best files first - #65
Conversation
DataFusion's PushdownSort marks the parquet source under a TopK so its shared work queue hands out files in sort order (newest first for ORDER BY ts DESC LIMIT n), and the TopK's dynamic filter then skips the rest before they are opened. LocalModeOptimizer runs after that rule and rebuilds the scan with LiquidParquetSource::from_parquet_source, which cannot copy the mark (it is private to ParquetSource), and LiquidParquetSource implemented neither try_pushdown_sort nor reorder_files. Liquid scans therefore handed files out in listing order. On a 30-day span table, "newest timestamp in a day" opened 22 to 28 of the day's 62 files, about 18 s on a first read from S3. - LiquidParquetSource: try_pushdown_sort answers Inexact when the leading sort key is a plain file column and keeps the order; reorder_files sorts the queue by each file's min statistic in the key's direction (a copy of DataFusion's crate-private helper). The plan shows sort_order_for_reorder. - LocalModeOptimizer: after the rewrite, offer each TopK's sort to its input again and swap in the Inexact input with replace_children. Rerunning PushdownSort instead would rebuild the SortExec with a new dynamic filter while the scan keeps the old one, so nothing would be pruned; two datafusion-local snapshot tests caught that. Row groups inside a file are still read in file order.
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
argparse on Python 3.14 refuses a type that is not callable, and `Optional[Path]` is not, so the TPC-DS job failed while generating its data (main too, since the runners moved to 3.14). `type=Path` with the existing `default=None` keeps the option optional.
|
The TPC-DS job failed before any Rust ran: |
📊 Benchmark ComparisonCurrent:
Compared Liquid vs DataFusionDefault on the same runner |
Summary
For
ORDER BY col DESC LIMIT n(a TopK), a liquid scan now hands its files out in sort order, as DataFusion's own parquet reader does. The TopK's dynamic filter then skips the remaining files before they are opened.Problem
DataFusion's
PushdownSortmarks theParquetSourceunder a TopK, so its shared work queue (reorder_files) hands out the most promising files first.LocalModeOptimizerruns after that rule. It rebuilds the scan withLiquidParquetSource::from_parquet_source, which cannot copy the mark, because the mark is private toParquetSource. AndLiquidParquetSourceimplemented neithertry_pushdown_sortnorreorder_files. So liquid scans handed files out in listing order: block by block, with the newest file last.On a 30-day span table (runtimedb sandbox, about 62 files per day), "newest timestamp in a day" (
ORDER BY "Timestamp" DESC LIMIT 1) opened 22 to 28 of the day's files. The first read from S3 took about 18 s. The data read was about 120 MB: the time went to opening files.Change
LiquidParquetSourcetry_pushdown_sortanswersInexactwhen the request's leading key is a plain column of the file schema, and keeps the order. In every other case it answersUnsupported, which is today's behaviour: a request that the declared ordering already satisfies, a request whose prefix it satisfies, or a sort on an expression.reorder_filessorts the work queue by each file'sminstatistic in the key's direction. It is a copy of DataFusion's crate-privatereorder_files_by_min_statistics.EXPLAINshowssort_order_for_reorder=[...].LocalModeOptimizerpush_topk_sorts_into_scansoffers each TopK's sort to its input again. OnInexact, it swaps in the new input withreplace_childrenand keeps theSortExec.PushdownSort. That rule rebuilds theSortExecon anInexactanswer, and the rebuilt TopK gets a new dynamic filter, while the scan keeps the old one that filter pushdown attached earlier. The scan would then prune nothing. Twodatafusion-localsnapshot tests caught this: their cachegetcounts went up.optimizer.enable_sort_pushdownis on.The
SortExecstays in every case, so results cannot change.Tests
liquid_topk_skips_the_older_files: 4 time-sliced files in 2 file groups, listed oldest first, withORDER BY ts DESC LIMIT 1. At most 2 of the 4 files are opened, and the answer is right. Without this change it opens all 4, and with a secondPushdownSortpass it also opens all 4.liquid_scan_hands_out_files_newest_first_under_a_topk: the scan keeps the sort, the queue order isd c b a, andEXPLAINshows the order.cargo fmt --check,cargo clippy --workspace --exclude dev-tools --all-targets --all-features -D warnings, and the tests ofliquid-cache-datafusionandliquid-cache-datafusion-localall pass (on Rust 1.96).Not in this PR