Skip to content

perf(reader): read a TopK's best files first - #65

Merged
shefeek-jinnah merged 2 commits into
mainfrom
perf/liquid-sort-pushdown
Nov 16, 2026
Merged

shefeek-jinnah merged 2 commits into
mainfrom
perf/liquid-sort-pushdown

Conversation

@shefeek-jinnah

Copy link
Copy Markdown
Collaborator

Summary

For ORDER BY col DESC LIMIT n (a TopK), a liquid scan now hands its files out in sort order, as DataFusion's own parquet reader does. The TopK's dynamic filter then skips the remaining files before they are opened.

Problem

DataFusion's PushdownSort marks the ParquetSource under a TopK, so its shared work queue (reorder_files) hands out the most promising files first. LocalModeOptimizer runs after that rule. It rebuilds the scan with LiquidParquetSource::from_parquet_source, which cannot copy the mark, because the mark is private to ParquetSource. And LiquidParquetSource implemented neither try_pushdown_sort nor reorder_files. So liquid scans handed files out in listing order: block by block, with the newest file last.

On a 30-day span table (runtimedb sandbox, about 62 files per day), "newest timestamp in a day" (ORDER BY "Timestamp" DESC LIMIT 1) opened 22 to 28 of the day's files. The first read from S3 took about 18 s. The data read was about 120 MB: the time went to opening files.

Change

  • LiquidParquetSource
    • try_pushdown_sort answers Inexact when the request's leading key is a plain column of the file schema, and keeps the order. In every other case it answers Unsupported, which is today's behaviour: a request that the declared ordering already satisfies, a request whose prefix it satisfies, or a sort on an expression.
    • reorder_files sorts the work queue by each file's min statistic in the key's direction. It is a copy of DataFusion's crate-private reorder_files_by_min_statistics.
    • EXPLAIN shows sort_order_for_reorder=[...].
  • LocalModeOptimizer
    • After the rewrite, push_topk_sorts_into_scans offers each TopK's sort to its input again. On Inexact, it swaps in the new input with replace_children and keeps the SortExec.
    • It deliberately does not rerun PushdownSort. That rule rebuilds the SortExec on an Inexact answer, and the rebuilt TopK gets a new dynamic filter, while the scan keeps the old one that filter pushdown attached earlier. The scan would then prune nothing. Two datafusion-local snapshot tests caught this: their cache get counts went up.
    • It runs only when optimizer.enable_sort_pushdown is on.

The SortExec stays in every case, so results cannot change.

Tests

  • liquid_topk_skips_the_older_files: 4 time-sliced files in 2 file groups, listed oldest first, with ORDER BY ts DESC LIMIT 1. At most 2 of the 4 files are opened, and the answer is right. Without this change it opens all 4, and with a second PushdownSort pass it also opens all 4.
  • liquid_scan_hands_out_files_newest_first_under_a_topk: the scan keeps the sort, the queue order is d c b a, and EXPLAIN shows the order.
  • The ascending order, sort pushdown turned off, no sort, and a sort on an expression are covered too, plus unit tests of the reorder helper.
  • cargo fmt --check, cargo clippy --workspace --exclude dev-tools --all-targets --all-features -D warnings, and the tests of liquid-cache-datafusion and liquid-cache-datafusion-local all pass (on Rust 1.96).

Not in this PR

  • Row groups inside a file are still read in file order, with no early stop. A follow-up can read them newest first and re-check the TopK's threshold at each row group.
  • Upstream (datafusion-contrib/liquid-cache) has the same gap, so this change can be offered there too.

DataFusion's PushdownSort marks the parquet source under a TopK so its
shared work queue hands out files in sort order (newest first for
ORDER BY ts DESC LIMIT n), and the TopK's dynamic filter then skips the
rest before they are opened. LocalModeOptimizer runs after that rule
and rebuilds the scan with LiquidParquetSource::from_parquet_source,
which cannot copy the mark (it is private to ParquetSource), and
LiquidParquetSource implemented neither try_pushdown_sort nor
reorder_files. Liquid scans therefore handed files out in listing
order. On a 30-day span table, "newest timestamp in a day" opened 22 to
28 of the day's 62 files, about 18 s on a first read from S3.

- LiquidParquetSource: try_pushdown_sort answers Inexact when the
  leading sort key is a plain file column and keeps the order;
  reorder_files sorts the queue by each file's min statistic in the
  key's direction (a copy of DataFusion's crate-private helper). The
  plan shows sort_order_for_reorder.
- LocalModeOptimizer: after the rewrite, offer each TopK's sort to its
  input again and swap in the Inexact input with replace_children.
  Rerunning PushdownSort instead would rebuild the SortExec with a new
  dynamic filter while the scan keeps the old one, so nothing would be
  pruned; two datafusion-local snapshot tests caught that.

Row groups inside a file are still read in file order.
@shefeek-jinnah
shefeek-jinnah requested a review from a team as a code owner November 16, 2026 05:32
@shefeek-jinnah
shefeek-jinnah requested review from zfarrell and removed request for a team November 16, 2026 05:32
claude[bot]
claude Bot previously approved these changes Nov 16, 2026
@codecov

codecov Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.07447% with 11 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/datafusion/src/reader/plantime/source.rs 95.03% 4 Missing and 4 partials ⚠️
src/datafusion/src/optimizers/mod.rs 98.60% 1 Missing and 2 partials ⚠️

📢 Thoughts on this report? Let us know!

argparse on Python 3.14 refuses a type that is not callable, and
`Optional[Path]` is not, so the TPC-DS job failed while generating its
data (main too, since the runners moved to 3.14). `type=Path` with the
existing `default=None` keeps the option optional.
@shefeek-jinnah

Copy link
Copy Markdown
Collaborator Author

The TPC-DS job failed before any Rust ran: benchmark/tpcds/tpcds_gen.py passes type=Optional[Path] to argparse, which Python 3.14 (now on the runners) rejects as not callable. Main's CI fails the same way since 13 Nov. I added a separate commit with type=Path (the default stays None), so the job can generate its data again. It can move to its own PR if you prefer.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

📊 Benchmark Comparison

Current: 6e413e60 (Liquid) vs Baseline: 6e413e60 (DataFusionDefault)

Query Cold Time Δ Warm Time Δ CPU Time Δ
Q1 2.0ms (2.0ms) +0.0% 0.000ms (0.000ms) +0.0% 0.000ms (0.000ms) +0.0%
Q2 7.0ms (5.0ms) +40.0% 3.5ms (5.0ms) -30.0% 7.0ms (7.0ms) +0.0%
Q3 13.0ms (12.0ms) +8.3% 4.5ms (11.0ms) -59.1% 3.0ms (23.0ms) -87.0%
Q4 13.0ms (10.0ms) +30.0% 3.0ms (9.0ms) -66.7% 3.0ms (18.0ms) -83.3%
Q5 56.0ms (48.0ms) +16.7% 43.0ms (51.5ms) -16.5% 4.0ms (20.5ms) -80.5%
Q6 223.0ms (108.0ms) +106.5% 70.5ms (102.5ms) -31.2% 22.0ms (78.0ms) -71.8%
Q7 1.0ms (1.0ms) +0.0% 1.0ms (0.000ms) +inf% 0.000ms (0.000ms) +0.0%
Q8 9.0ms (6.0ms) +50.0% 5.5ms (6.5ms) -15.4% 9.0ms (7.0ms) +28.6%
Q9 85.0ms (86.0ms) -1.2% 70.0ms (83.5ms) -16.2% 5.5ms (38.5ms) -85.7%
Q10 87.0ms (92.0ms) -5.4% 71.5ms (87.5ms) -18.3% 7.0ms (59.5ms) -88.2%
Q11 34.0ms (22.0ms) +54.5% 18.0ms (24.0ms) -25.0% 34.5ms (32.0ms) +7.8%
Q12 38.0ms (29.0ms) +31.0% 20.0ms (27.0ms) -25.9% 37.0ms (38.5ms) -3.9%
Q13 253.0ms (113.0ms) +123.9% 112.5ms (106.5ms) +5.6% 64.0ms (80.0ms) -20.0%
Q14 542.0ms (134.0ms) +304.5% 141.5ms (137.0ms) +3.3% 82.5ms (101.5ms) -18.7%
Q15 469.0ms (103.0ms) +355.3% 125.0ms (106.0ms) +17.9% 71.5ms (96.5ms) -25.9%
Q16 136.0ms (113.0ms) +20.4% 136.0ms (145.0ms) -6.2% 6.0ms (29.5ms) -79.7%
Q17 622.0ms (280.0ms) +122.1% 304.0ms (267.0ms) +13.9% 78.5ms (122.5ms) -35.9%
Q18 624.0ms (224.0ms) +178.6% 306.5ms (226.5ms) +35.3% 96.5ms (99.5ms) -3.0%
Q19 1.13s (515.0ms) +119.8% 392.0ms (480.5ms) -18.4% 92.0ms (141.5ms) -35.0%
Q20 12.0ms (10.0ms) +20.0% 3.0ms (10.5ms) -71.4% 8.0ms (20.5ms) -61.0%
Q21 563.0ms (176.0ms) +219.9% 464.5ms (175.5ms) +164.7% 281.5ms (281.0ms) +0.2%
Q22 993.0ms (171.0ms) +480.7% 620.5ms (166.0ms) +273.8% 164.0ms (343.0ms) -52.2%
Q23 1.93s (467.0ms) +312.8% 1.45s (465.0ms) +210.8% 530.5ms (752.5ms) -29.5%
Q24 24.73s (918.0ms) +2593.5% 977.5ms (942.0ms) +3.8% 592.5ms (2.56s) -76.8%
Q25 314.0ms (67.0ms) +368.7% 17.0ms (57.0ms) -70.2% 45.5ms (115.0ms) -60.4%
Q26 136.0ms (43.0ms) +216.3% 18.0ms (43.5ms) -58.6% 50.0ms (81.0ms) -38.3%
Q27 321.0ms (62.0ms) +417.7% 25.0ms (57.5ms) -56.5% 72.0ms (118.5ms) -39.2%
Q28 1.09s (212.0ms) +414.6% 937.0ms (230.5ms) +306.5% 210.5ms (309.5ms) -32.0%
Q29 1.49s (992.0ms) +50.2% 971.5ms (973.5ms) -0.2% 377.5ms (340.5ms) +10.9%
Q30 32.0ms (26.0ms) +23.1% 24.5ms (25.5ms) -3.9% 6.0ms (21.0ms) -71.4%
Q31 697.0ms (106.0ms) +557.5% 83.0ms (101.0ms) -17.8% 73.5ms (140.0ms) -47.5%
Q32 984.0ms (106.0ms) +828.3% 156.5ms (101.0ms) +55.0% 110.0ms (145.0ms) -24.1%
Q33 380.0ms (349.0ms) +8.9% 352.5ms (342.5ms) +2.9% 8.0ms (68.0ms) -88.2%
Q34 877.0ms (445.0ms) +97.1% 460.5ms (429.0ms) +7.3% 125.5ms (271.5ms) -53.8%
Q35 909.0ms (428.0ms) +112.4% 466.0ms (425.5ms) +9.5% 136.5ms (286.0ms) -52.3%
Q36 114.0ms (113.0ms) +0.9% 92.5ms (112.0ms) -17.4% 5.0ms (22.0ms) -77.3%
Q37 315.0ms (96.0ms) +228.1% 76.0ms (97.5ms) -22.1% 27.0ms (69.5ms) -61.2%
Q38 76.0ms (47.0ms) +61.7% 32.0ms (47.5ms) -32.6% 22.0ms (27.0ms) -18.5%
Q39 334.0ms (49.0ms) +581.6% 30.5ms (46.0ms) -33.7% 18.0ms (69.0ms) -73.9%
Q40 705.0ms (184.0ms) +283.2% 237.0ms (178.0ms) +33.1% 59.5ms (126.0ms) -52.8%
Q41 26.0ms (18.0ms) +44.4% 11.5ms (19.5ms) -41.0% 7.0ms (18.5ms) -62.2%
Q42 20.0ms (20.0ms) +0.0% 9.0ms (18.5ms) -51.4% 6.0ms (16.5ms) -63.6%
Q43 17.0ms (18.0ms) -5.6% 13.0ms (16.5ms) -21.2% 9.0ms (10.0ms) -10.0%

⚠️ LiquidCache is slower on 9 queries (warm)

  • Q7: warm +inf% (1.0ms vs 0.000ms)
  • Q28: warm +306.5% (937.0ms vs 230.5ms)
  • Q22: warm +273.8% (620.5ms vs 166.0ms)
  • Q23: warm +210.8% (1.45s vs 465.0ms)
  • Q21: warm +164.7% (464.5ms vs 175.5ms)
  • Q32: warm +55.0% (156.5ms vs 101.0ms)
  • Q18: warm +35.3% (306.5ms vs 226.5ms)
  • Q40: warm +33.1% (237.0ms vs 178.0ms)
  • Q15: warm +17.9% (125.0ms vs 106.0ms)

Compared Liquid vs DataFusionDefault on the same runner
Regressions: warm-time increases of at least 15%. Cold Time: first iteration; Warm Time: median of remaining iterations.

@shefeek-jinnah
shefeek-jinnah merged commit 8b3c813 into main Nov 16, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant