fix: compare every row when arrays_overlap gets a scalar beside a column

[grorge123]

Which issue does this PR close?

Closes #6742.

Rationale for this change

When one argument of arrays_overlap is a scalar and the other is an array column, the native kernel turns the scalar into a one-row list while the column keeps its full length, and returns row 0's result as a scalar. The first row's answer is then used for the whole batch, or the kernel panics when the constant is the second argument. Split out of #5526.

What changes are included in this PR?

SparkArraysOverlap keeps a scalar argument as a one-row list that every output row reads, and returns an array result when the other argument is an array. It does not repeat the scalar to the batch length, because repeating a long list could overflow the list's 32-bit offsets. The existing NULL-scalar short-circuit is unchanged, and two scalar arguments still produce a scalar result.

How are these changes tested?

• A query at the end of array/arrays_overlap.sql compares a column-built array against a constant array in both argument orders, over four rows in one batch. On main the query panics, because of the constant-second order; the constant-first order on its own returns a wrong answer. It passes on the Spark 4.1, 3.5 and 3.4 profiles.
• Rust unit tests call invoke_with_args with a scalar on either side, for flat and struct elements. They also cover two scalars, and a 262,144-element constant beside 8,192 rows, which panics with an offset overflow if the scalar is repeated instead.

Assisted-by: Claude Code (claude-opus-5-5)

[sunchao]

Reviewed 0b69838 against base c8e6553 across five independent review scopes. No actionable findings.

The fix correctly compares every column row with the scalar list in both argument orders, preserves null and nested-element semantics, and avoids repeating a large scalar list to the batch length.

Local validation:

• All 30 arrays_overlap unit tests passed, including the four added here.
• Five additional review probes passed, covering empty batches, sliced scalar/column lists, LargeList, nested floats, and outer-null/empty/null-element combinations in both argument orders.
• Independently reproduced both failures in the pre-change implementation: constant-first returned an incorrect scalar result, and constant-second panicked on an out-of-bounds offset.
• Built the native library with HDFS disabled, then ran the focused arrays_overlap.sql fixture through CometSqlFileTestSuite on Spark 4.1.3. One test ran and passed, comparing Spark and Comet results.

The same broadcasting defect is present on branch-1.1, so this is a backport candidate.

CI is still running. Preflight, native build, lint, and Celeborn compatibility checks have passed. Rust tests, Spark 4.1 test shards, and TPC-H/TPC-DS verification remain pending, with no failures reported at this check. Other Spark profiles and performance benchmarks were not run locally.

[comphead]

Thanks @grorge123. I compared SparkArraysOverlap with Spark's ArraysOverlap in 3.4.3, 3.5.8, 4.0.1 and 4.1.1. The class is the same in all four apart from how NullIntolerant is declared, and from reading the code I did not find a gap in the new scalar path. One inline comment on how the new tests could reuse the existing SQL fixture.

[comphead]

From reading the code, the constant-second queries at lines 29 and 280 already have the #6742 shape. They pass on main because the VALUES insert at line 22 writes one file per row under local[5] (see #6628), so every batch has one row. Could we write test_arrays_overlap as one file instead of adding test_overlap_batch? INSERT INTO test_arrays_overlap SELECT /*+ COALESCE(1) */ * FROM VALUES ... AS t(a, b) is the pattern in min_max_floating_point.sql. A constant-first query whose answer changes by row, such as arrays_overlap(array(3, 4, 5), a), would complete it. With a one-file table I expect lines 29 and 280 to panic without the fix, and the table already has a null array, an empty array and null elements. The flat scalar_beside_column_* Rust test would then repeat this, and the struct one could be a query over a one-file test_overlap_struct. I have not run any of it.

[grorge123]

Thanks. fbe5f5a writes test_arrays_overlap and test_overlap_struct as one file each with COALESCE(1) and drops test_overlap_batch. It adds arrays_overlap(array(3, 4, 5), a) and a struct query with the constant on each side. It also removes the flat and struct scalar_beside_column_* Rust tests.