[rich7420]

Which issue does this PR close?

Closes #5994.

Rationale for this change

Native wide-decimal hashing used different bytes from Spark, so mixed native/JVM shuffle could lose matching join rows. Make precision 19–38 hashing Spark-compatible and enable native SQL hash/xxhash64.

What changes are included in this PR?

• Encode unscaled values as minimal signed big-endian bytes and optimize decimal-list hashing. Precision at most 18 is unchanged; retain Comet's wide-decimal xxhash64 kernel because the pinned upstream kernel differs.
• Remove wide-decimal guards from typed Dataset and shuffle-input conversion, enabling native shuffle for eligible inputs while retaining metadata-sensitive restrictions.
• Add scalar/nested NULL, signed-byte, partition-parity, mixed-shuffle and AQE coverage, benchmarks and support documentation.

How are these changes tested?

At 33682c0ee, upstream CI passed, including Rust, native build, lint, Spark 4.1 suites and TPC-H/TPC-DS. The merge includes #6740's Spark 4.2 timestamp fix; hashing code is unchanged from 8b1d26a32. Supplemental Spark SQL and cross-profile CI: requested via fork #35, but fork Actions is blocked and no current-head run started. Please apply run-spark-4.1-tests and run-all-spark-profiles upstream before queueing.

Historical fork CI covered Spark 4.1 SQL and mixed shuffle on Spark 4.1/4.2; its Spark 4.2 expression failure predates the main update and is not a current-head verdict.

Historical benchmarks at 27faa50ed measured 5.0–14.2x Murmur3 and 2.1–15.8x xxhash64 improvements over the pre-optimization native decimal-array path. Whole-query native array hashing averaged 83–89 ms versus 159–176 ms with Spark hash fallback. These local measurements were not rerun for this merge. Benchmark source.