Skip to content

Spark window over a doubted input order gets a sort by its own keys only, and operators above it return wrong rows #7553

Open

Description

@dwsmith1983

Describe the bug

Found in #6785. CometExecRule.sortDoubtedWindowInputs (from #7256) puts a SortExec under a window that stays in Spark when Comet may report a false order for its input. The sort uses only the window's own required ordering. Spark planned the operators above that window around the longer order the input reported, so they lose it:

  • A second Spark window ordered by (v, w) above a window ordered by v is not sorted again, because mayReportFalseInputOrder now stops at the new sort.
  • A sortWithinPartitions(v, w) above was already removed as redundant.

Two ways this returns wrong rows on main (891008e):

  1. The input really is out of order past the window's keys. A native broadcast nested loop join with two build rows emits its rows one build row at a time (Native broadcast nested loop joins report the streamed side's order without keeping it #6898). Sorting by v alone leaves w out of order, and row_number() OVER (ORDER BY v, w) above it is wrong.
  2. The input really is in order. With one build row the join keeps it, but the inserted sort by v spills, and merging the spill files reorders rows that tie on v. Without the spill the same query matches Spark.

Steps to reproduce

AQE off, spark.sql.autoBroadcastJoinThreshold=-1, one input partition:

val cached = spark.range(0, 4000, 1, 1)
  .selectExpr("id % 4 AS v", "id AS w")
  .sortWithinPartitions("v", "w")
  .cache()
val build = spark.range(2).selectExpr("id AS z")
cached.join(broadcast(build), expr("w % 2 <= z"))
  .select("v", "w")
  .selectExpr("v", "w",
    "sum(w) OVER (ORDER BY v) AS s",
    "row_number() OVER (ORDER BY v, w) AS r")

Both windows stay in Spark. The plan has Sort [v ASC NULLS FIRST] under the lower window and no sort under the upper one, and r differs from Spark: within v = 0, w = 0 comes after w = 3996. For the second case, use spark.range(1) as the build side, 200000 rows, and spark.shuffle.spill.numElementsForceSpillThreshold=10000.

Expected behavior

The rows match Spark. The inserted sort should restore the whole order the input reported (the part that covers the window's required ordering), not only the window's keys, so the operators Spark planned above it still see that order.

Additional context

#6785 sorts a native window's input the same way and has the same gap; its fix restores the full reported order natively. This issue covers the rule for windows that run in Spark.

Activity

  1. dwsmith1983 commented on Nov 7, 2026

    @dwsmith1983
    ContributorAuthor

    take

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions