perf: vectorize int, decimal, float and boolean casts to timestamp

[Ersa-tech]

Which issue does this PR close?

Closes #4942.

Rationale for this change

cast_int_to_timestamp, cast_decimal_to_timestamp, cast_float_to_timestamp, and cast_boolean_to_timestamp still built the output one row at a time with TimestampMicrosecondBuilder. These are the remaining per-element casts to timestamp in #4942.

What changes are included in this PR?

• Int casts use unary and saturating_mul. The multiply is infallible on a null slot.
• Boolean casts use from_unary. true is still 1 microsecond and false is 0.
• Float casts use try_unary in ANSI mode and unary_opt in Legacy and Try. The error string is built only when ANSI fails.
• Decimal casts use one unary_opt pass. The 256-bit division does not vectorize, and evaluating null slots was slower on every measured shape, including a batch with no nulls. Spark still truncates toward zero and does not raise.
• A batch with no nulls still has nulls() == None.

unary and from_unary transform the value stored under a null. The builder wrote 0 there. Arrow equality ignores those slots. A negative decimal scale still panics in 10_i128.pow, same as before.

This does not edit the float-to-int or decimal-to-int macros, or the test before test_spark_cast_float_min_value_to_string (open PR #6953).

Prepared with Cursor Grok 4.7. The semantics, tests, and benchmarks were reviewed.

How are these changes tested?

Rust tests pin saturation, decimal truncation and the low 64 bits, the ANSI error variant and the first offending value, Legacy and Try agreeing, and a NaN hidden under a null. cargo test -p datafusion-comet-spark-expr --lib conversion_funcs:: (124 tests). cargo fmt. cargo clippy --all-targets --workspace -- -D warnings from native/.

CometNativeCastSuite tests whose names contain to TimestampType (20 tests, Spark 4.1, release libcomet.so) all passed, including boolean, byte, short, int, long, float, double, and decimal to timestamp.

Criterion, 8192 rows, baseline main on the same machine (4 vCPU Intel Xeon, family 6 model 207, AVX-512, 15 GiB RAM, rustc 1.99.0). Bench profile: opt-level=3, lto=thin, codegen-units=1, overflow-checks=false. No shape regressed. Decimal with no nulls is under 10%.

cd native
export CARGO_PROFILE_BENCH_OPT_LEVEL=3 CARGO_PROFILE_BENCH_LTO=thin \
  CARGO_PROFILE_BENCH_CODEGEN_UNITS=1 CARGO_PROFILE_BENCH_OVERFLOW_CHECKS=false
cargo bench -p datafusion-comet-spark-expr \
  --bench cast_int_to_timestamp --bench cast_non_int_numeric_timestamp -- --save-baseline main
cargo bench -p datafusion-comet-spark-expr \
  --bench cast_int_to_timestamp --bench cast_non_int_numeric_timestamp -- --baseline main

The main column is these bench files run against main's kernels. main's own benches use different ids and shapes, so that column is not a comparison against main's bench output. The files cover i64 and f64 across null shapes, plus one i32 case and one f32 case.

Cast	No nulls	Sparse nulls	Dense nulls
i64	20.2 → 2.3 µs (−89%)	27.6 → 2.4 µs (−91%)	21.8 → 2.3 µs (−89%)
f64 legacy	42.3 → 34.4 µs (−19%)	50.2 → 33.0 µs (−34%)	23.2 → 5.5 µs (−76%)
f64 ANSI	42.2 → 35.5 µs (−14%)	50.3 → 32.8 µs (−35%)	23.4 → 5.6 µs (−76%)
decimal(38, 6)	92.8 → 84.2 µs (−9%)	94.7 → 76.4 µs (−19%)	31.2 → 11.6 µs (−63%)
bool	20.5 → 3.8 µs (−81%)	27.6 → 4.0 µs (−86%)	20.9 → 3.8 µs (−81%)

All-null batches: i64 −89%, f64 legacy −93%, f64 ANSI −94%, decimal −94%, bool −80%. A legacy f64 batch of NaN and infinities is −34%. f32 and the narrower integers use the same kernels. Decimal scales 0 and 18 with no nulls, measured on this same unary_opt kernel, were −9.6% and −9.8%.

[sunchao]

Reviewed all six changed files and four commits against base 7d8d9b1544b734c85a93cca2cc9d5380ad70accd, at full head bb661af85dc8d66706e19d7d122e22ee5b49658e. The PR is not a draft. Routed skills: review-comet-pr and review-comet-expression-pr, including their timestamp and optimization guidance. The snapshot and live discussion contain no reviews, conversation comments, inline comments, or review threads.

Summary

• Prior state and problem: Integer, decimal, floating-point, and boolean timestamp casts built output through per-row TimestampMicrosecondBuilder calls, adding bookkeeping to existing conversion arithmetic.
• Design approach: Replace those loops with Arrow unary, from_unary, try_unary, and unary_opt, selected according to whether conversion can fail or produce nulls.
• Correctness: Checked Spark’s interpreted and generated casts, DateTimeUtils, and DoubleExactNumeric across supported versions. Integer saturation, boolean 1/0 microseconds, decimal truncation and low-bit conversion, and ANSI floating-point checks retain their existing arithmetic. Isolated differential tests found no changed valid results, null placement, error variants, first-error selection, or timestamp types.
• Compatibility analysis: Compared Spark 3.4.3, 3.5.9, 4.0.4, 4.1.3, and 4.2.0. No new version-dependent behavior, support-level change, or fallback change appears. The affected routines on branch-1.1 at 8ac6fccde53d96fec22184be287747eeae481cac match the base implementations. No unintended observable change against that release was identified.
• Key design decisions: Infallible integer and boolean operations may evaluate masked values. Floating-point and decimal kernels skip null slots. ANSI processing preserves the first offending value in array order.
• Implementation sketch: Generic typed helpers replace the integer and float macros. Decimal conversion retains its i256 arithmetic. timestamp_micros_with_tz applies the requested label and removes an unnecessary all-valid null buffer. Tests, benchmarks, and the performance audit accompany the changes.
• Performance: The benchmarks cover 8,192-row batches with absent, sparse, dense, and all-null inputs, plus non-finite floats. Author-reported results show approximately 89% lower integer time, 81% lower boolean time, and 9% lower decimal time without nulls. I did not independently reproduce those timings. Source inspection found no introduced overhead meeting the P1/P2 bar.
• Design: The refactor stays within existing cast kernels and uses Arrow’s established array operations. It preserves the surrounding routing and timestamp representation, keeping the change straightforward to reason about.
• Abstraction & complexity: The typed helpers consolidate width-specific behavior. The shared output helper centralizes timezone and null-buffer normalization. Floating-point error formatting remains outside the successful conversion path. No unnecessary abstraction meeting the reporting bar was identified.
• Behavioral changes worth calling out: Integer and boolean payloads underneath null bits can differ from the builder’s zero payloads, while logical nulls remain unchanged. Sliced arrays and explicit all-valid masks were validated. Production TimestampType output retains its existing UTC label.
• Suggested improvements: No additional change meeting the P1/P2 reporting bar is requested.

Exact-head CI: The label workflow succeeded. Comet CI, CodeQL, and Check PR Title remain action_required. There is no completed build/test verdict for this SHA.

Validation: An isolated harness using the exact base/head kernel bodies, lockfile Arrow 59.3.0, and minimal enum/error scaffolding passed eight extracted timestamp tests and 7,650 differential cases covering all integer widths, both float widths, decimal scales, boundaries, null patterns, slices, errors, and timezone labels. Full cargo test --locked -p datafusion-comet-spark-expr --lib conversion_funcs:: could not resolve DataFusion 55.2.0 from the configured registry, both online and offline. JVM suites and Criterion benchmarks were not run. Project files remain unchanged.

No introduced P1/P2 issues found within this review.

[dwsmith1983]

Thanks for this. I checked the numbers and the semantics against 8dd5c1e, and the change holds up: the new kernels give the same values, null placement, error variants and first failing row as main, and they do not touch anything under a null in a way that can panic.

Benchmarks. Since the PR rewrites both bench files, I ran the PR's bench files against main's kernels as the baseline and against the PR head, alternating (main, PR, PR, main, main, PR) on an 8-core aarch64 machine without AVX-512, with the profile settings from the description.

• Every shape is faster, and the main and PR ranges never overlap, so "No shape regressed" holds. The size of the gain differs on this hardware because main's builder loop is two to five times faster here: i64 and bool with no nulls came out at -55% and -61%, the f64 no-null and sparse shapes at -43% to -71%, and decimal(38, 6) with no nulls at -9%.
• Decimal scales 0 and 18 are not in the bench files, so the -9.6% and -9.8% can't be reproduced from the PR. Adding them as extra shapes, I got -10% and -14%.
• With unary in place of unary_opt for decimal, a batch with no nulls was slightly faster here (54.4 vs 57.5 µs at scale 6), so "slower on every measured shape, including a batch with no nulls" didn't reproduce. unary_opt is still the right choice, since unary is far slower once there are nulls.
• It would help to say in the description that the main column comes from these bench files run against main's kernels, since main's own benches use different IDs and shapes. The new files also drop i8, i16, i32 and f32. The kernels are generic, but keeping one narrow int and f32 would keep those paths covered.

cargo test -p datafusion-comet-spark-expr --lib conversion_funcs:: runs 121 tests at this head, not 119.

Existing differences from Spark that the new tests lock in. These are all on main as well, so they are not regressions, but the PR rewrites this code and two of its new tests assert the current behavior. I reproduced each on Spark 4.1 with the cast running natively over a Parquet column:

1. With ANSI off, a finite double whose microseconds overflow becomes NULL, while Spark saturates. CAST(CAST(d AS TIMESTAMP) AS LONG) gives 9223372036854 in Spark and NULL in Comet for d = 1e16, 1e22 and 9.3e12, and -9223372036855 vs NULL for -1e16. Spark only nulls NaN and infinities there, and Rust's f64 as i64 already saturates the same way, so the legacy branch could return the cast value for finite inputs while TRY keeps returning NULL. The mixed case in the new test expects NULL for 1e16 under Legacy.
2. With ANSI on, NaN and infinities raise a different exception. Spark throws SparkDateTimeException with The value Infinity of the type "DOUBLE" cannot be cast to "TIMESTAMP". Comet throws SparkNumberFormatException with The value 'inf' of the type "STRING", so the source type, the spelling and the quoting all differ (the error class and SQLSTATE match). InvalidInputInCastToDatetime already maps to Spark's exception, and the new test asserts "inf".
3. A decimal with a negative scale panics. With spark.sql.legacy.allowNegativeScaleOfDecimal=true, CAST(CAST(c * 1E2 AS TIMESTAMP) AS LONG) returns -45600, 0, 12300 in Spark and fails in Comet with native panic: attempt to exponentiate with overflow from 10_i128.pow(scale as u32). CometCast.canCastFromDecimal reports the cast as compatible for any scale, unlike the decimal-to-string path, which checks that setting.

Fixing the first two here would be small since the code is being rewritten anyway, or they could go to follow-up issues. Either way, the tests would be better not asserting the current results for those cases.

Test gaps. I ran 30 hand-made mutants over the changed functions. 22 were caught, mostly by the new tests. Five real gaps survived:

• turning a valid decimal 0 into NULL (test_cast_decimal_to_timestamp checks value(0) but not validity);
• the -Infinity overflow text (f64::MIN would cover it);
• the from_type of the invalid-value error;
• the <=/>= bounds at 2^63 microseconds for floats;
• truncation toward zero for floats.

The last two were already untested on main.

A small one: the audit entry in conversion_funcs.md doesn't link the PR, unlike its neighbors.