You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Spark window over a doubted input order gets a sort by its own keys only, and operators above it return wrong rows #7553
Found in #6785. CometExecRule.sortDoubtedWindowInputs (from #7256) puts a SortExec under a window that stays in Spark when Comet may report a false order for its input. The sort uses only the window's own required ordering. Spark planned the operators above that window around the longer order the input reported, so they lose it:
A second Spark window ordered by (v, w) above a window ordered by v is not sorted again, because mayReportFalseInputOrder now stops at the new sort.
A sortWithinPartitions(v, w) above was already removed as redundant.
Two ways this returns wrong rows on main (891008e):
The input really is in order. With one build row the join keeps it, but the inserted sort by v spills, and merging the spill files reorders rows that tie on v. Without the spill the same query matches Spark.
Steps to reproduce
AQE off, spark.sql.autoBroadcastJoinThreshold=-1, one input partition:
valcached= spark.range(0, 4000, 1, 1)
.selectExpr("id % 4 AS v", "id AS w")
.sortWithinPartitions("v", "w")
.cache()
valbuild= spark.range(2).selectExpr("id AS z")
cached.join(broadcast(build), expr("w % 2 <= z"))
.select("v", "w")
.selectExpr("v", "w",
"sum(w) OVER (ORDER BY v) AS s",
"row_number() OVER (ORDER BY v, w) AS r")
Both windows stay in Spark. The plan has Sort [v ASC NULLS FIRST] under the lower window and no sort under the upper one, and r differs from Spark: within v = 0, w = 0 comes after w = 3996. For the second case, use spark.range(1) as the build side, 200000 rows, and spark.shuffle.spill.numElementsForceSpillThreshold=10000.
Expected behavior
The rows match Spark. The inserted sort should restore the whole order the input reported (the part that covers the window's required ordering), not only the window's keys, so the operators Spark planned above it still see that order.
Additional context
#6785 sorts a native window's input the same way and has the same gap; its fix restores the full reported order natively. This issue covers the rule for windows that run in Spark.
Activity
dwsmith1983 commentedon Nov 7, 2026
take