Skip to content

feat(sql): quoted arguments and implicit comparisons read as DuckDB does (SCHEMA-5b) - #460

Merged
fupelaqu merged 4 commits into
mainfrom
feature/SCHEMA-5b
Nov 10, 2026
Merged

fupelaqu merged 4 commits into
mainfrom
feature/SCHEMA-5b

Conversation

@fupelaqu

@fupelaqu fupelaqu commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Quoted arguments and implicit comparisons read as DuckDB does (SCHEMA-5b, after #455 and #456). A quoted argument of COALESCE, GREATEST, LEAST or a CASE result, beside a number, a date, a timestamp or a boolean, is read as that type: GREATEST(n, '1.5') over an INT answers 2 (main refused it), and COALESCE(n, '3') is an INT (main answered the text '3' on every row). A query also compares a text with a number or a boolean, and a boolean with a number, the way DuckDB casts them. That covers 17 of the 25 scalar cells DuckDB answers and core refused. Along the way, four COALESCE / string-function rendering defects present on main are fixed: they produced wrong values or scripts that did not compile.

Closes no issue.

What changes

  • One arm-reading function decides how every conditional reads a quoted argument; its type, its rendering, the comparison rules and every consumer read the same function. Beside a whole type, a decimal text rounds half away from zero; past the column type's range it makes a BIGINT. A text that spells no value of that type leaves the expression as it was (COALESCE(n, 'abc') answers text, GREATEST(n, 'abc') is refused).
  • Implicit comparisons in a query. Text against a whole type compares as BIGINT, against DOUBLE / REAL / NUMERIC as DOUBLE, against BOOLEAN as BOOLEAN; a BOOLEAN against a whole type as 1 / 0. These apply to =, <>, !=, IN, NULLIF and CASE x WHEN, and the boolean-against-whole row also to ordering and BETWEEN. A stored text that spells no value fails the query, as in DuckDB. Ordering a text against a number stays refused. Outside a query (computed columns, pipelines, transforms) a text side keeps 0.24.0's verdict, so a pipeline never silently drops a document; the boolean rows apply everywhere.
  • A BOOLEAN beside whole numbers answers 1 / 0 in COALESCE, and in GREATEST / LEAST, in a query and in a computed column.
  • Nested beside a text, in a query, a conditional reading a quoted argument answers text (COALESCE(COALESCE(n, '1.5'), k) answers '2'), which also answers statements that were refused because a nested expression mixed text with a number or a date.
  • A literal-first COALESCE('2024-01-15', d) compared for equality with a text compares the literal's text (=, <>, !=, NULLIF, CASE … WHEN), and COALESCE(NULLIF(F, k), k) answers COALESCE(F, k), both in a query only. Main answered these with the same values (the literal-first forms) or the same dates in another text (COALESCE(NULLIF(F, k), k) over a CASE); refusing them would have broken working statements.
  • COALESCE fixes, in every scope (query, CREATE TABLE / ALTER TABLE, UPDATE … SET, transforms):
    • a CASE argument before the last one is tested whole for NULL: main tested the ELSE alone, so COALESCE(CASE WHEN c THEN a ELSE b END, k) answered NULL where c held and a was NULL, and a NULL ELSE did not compile;
    • the fold ends at a GREATEST / LEAST argument holding a literal, which is never NULL: COALESCE(GREATEST(n, 3), l) and its twins failed to compile on main (Cannot cast from [long] to [java.lang.Object]).
  • String functions over a number, a boolean or a conditional read the value's text, in a query and in a computed column: UPPER(n) stored no value, and LOWER(CASE … END) lowered only the ELSE.
  • CAST(b AS DATE | TIME | TIMESTAMP) over a BOOLEAN is refused in every scope, as DuckDB refuses it. Main answered the boolean itself in a date column.
  • A computed column comparing a BOOLEAN with a number compares it as 1 / 0, as a query does. Main compared the boolean itself and stored a wrong value or none.
  • Docs: functions_conditional.md, operators.md ("Comparing values of different types") and type_conversion.md list every moved statement by name.

Release notes

  • Quoted arguments take the type beside them: GREATEST(n, '1.5') = 2 (was refused); COALESCE(n, '3') is an INT (was text); COALESCE(d, '2024-01-15') is a DATE (was text). DESCRIBE, JDBC / Arrow types and CTAS / MV mappings move with it (keyword → integer / date), and so does every consumer: ORDER BY and GROUP BY keys follow the new type, a comparison or CAST(… AS VARCHAR) reads the number ('1.5' → '2'), and arithmetic answers (COALESCE(n, '1.5') + 1).
  • Text against number / boolean, boolean against number: these answer in a query for the operators DuckDB binds. A non-converting stored text fails the query. A stored 'on' / 'off' / ' true ' beside a boolean answers, where DuckDB fails.
  • BOOLEAN beside whole numbers answers 1 / 0: in COALESCE in a query (was true / false in an integer column), and in GREATEST / LEAST in a query (was refused) and in a computed column (accepted on main, but no value was stored).
  • Pipelines that ALTER TABLE re-creates (new bytes, values corrected): computed columns holding a quoted argument read as a value (COALESCE(n, '3') stores 3; '1.5' beside an integer stores 2, where Elasticsearch truncated it to 1); GREATEST / LEAST over a BOOLEAN beside whole numbers; CASE comparing a BOOLEAN with a number (the shapes are listed in functions_conditional.md); string functions over a number, a boolean or a conditional; COALESCE with a CASE argument before its last; COALESCE over a GREATEST / LEAST holding a literal.
  • Answered (wrong or failing), refused now: see the table below.
  • Refused before, answered now: COALESCE(GREATEST(n, '2'), k), COALESCE(GREATEST(d, '2024-01-15'), k) and the other nested shapes listed in functions_conditional.md; NULLIF(b, v) / NULLIF(v, b) over a BOOLEAN and a number in a computed column; NULLIF(COALESCE(d, '2024-01-15'), k) - d2 and COALESCE(NULLIF(COALESCE(d, '2024-01-15'), k), d2) - d2 in a computed column.
  • ISNULL / ISNOTNULL over a conditional in a WHERE now run: an operand with no column (NULLIF, COALESCE, CASE, ...) is evaluated by a script and tested as one value. Elasticsearch rejected these queries before (field name is null or empty).
  • Same value, other text: COALESCE(NULLIF(F, k), k) over a date CASE answers '2024-01-15' where main answered '2024-01-15T00:00Z'; over a timestamp CASE the literal answers '2024-01-15T10:30Z' where main answered '2024-01-15 10:30:00'.
  • No downstream rebuild: no case-class arity change.

Answered before (wrong or failing), refused now

Statement (k KEYWORD, d DATE, ts TIMESTAMP, n INT, b BOOLEAN) Main This PR
NULLIF(F, k), NULLIF(k, F), CASE F WHEN k, F = k and expressions over them, for F a conditional holding a quoted date or timestamp (COALESCE(d, '2024-01-15'), CASE WHEN … THEN d ELSE '2024-01-15' END, …; full list in functions_conditional.md) wrong values over the three CASE operands and the literal-first forms with a time part or a timestamp; a runtime failure over the others (measured on ES 8.18.3) refused by name
COALESCE(d, '2024-01-15') = k, COALESCE(n, '3') > k, COALESCE(n, '3') LIKE '1%', CAST(COALESCE(n, '3') AS DATE) failed on every row or answered wrong rows (Painless compared a number with a text) refused, as their unquoted twins
COALESCE('2024-01-15', d) < k / >= k failed on every document (measured) refused
Computed column NULLIF(COALESCE(n, '3'), k), NULLIF(k, COALESCE(n, '3')), NULLIF(COALESCE(n, '3'), '2024-01-15'), COALESCE(NULLIF(COALESCE(n, '3'), k), k | g) accepted; the pipeline stored no value on every document holding both columns refused by CREATE TABLE and ALTER TABLE, including any ALTER TABLE of a table that already holds one
CAST(b AS DATE | TIME | TIMESTAMP) answered the boolean in a date column refused in every scope

Acceptance evidence

Check Evidence
Quoted arguments, implicit comparisons, the 17 cells (DuckDB 1.5.5 as the oracle) QuotedArgumentConditionalSpec (testkit trait, wired for es6 jest / rest, es7 rest, es8 java, es9 java) and JavaClientSchemaConsumerSpec; QuotedArmReadingSpec, ImplicitComparisonCastSpec and the updated sql specs
main's values for every class this PR answers or refuses RUN on main on ES 8.18.3, with rows that tell the answers apart (the equality forms right on every document, ordering failing on every document, the CASE operands wrong)
No row main answers correctly is refused census on main, the previous round and this branch over QUERY, CREATE TABLE, ALTER TABLE, UPDATE, transform; every moved verdict or byte is listed in the docs
ES 8.18.3 branch leg trait + consumer spec + the class rows: 45 passed, 0 failed, every row matching its oracle (value and Java class), on both commits
ES 8.18.3 area leg (DateArithmeticSpec, DateFunctionExecutionSpec, ScrollCompletenessSpec, GroupByCompletenessSpec, ComputedProjectionSpec, PredicateFunctionResultSpec, JavaClientSpec) 278 passed, 0 failed, on the first commit (the second moves no query byte)
Unit suites on the final bytes: sql 2345/0, es8 bridge 353/0, sbt "+ compile" OK (2.12.20 and 2.13.16); on the first commit: bridge 353/0, es6 bridge 360/0, core 1654/0
Last fix: a literal-first COALESCE comparison entered directly in a transform or pipeline context renders as before (the rule is query-only) its pin fails on the first commit and passes on the second; sql 2345/0; ES 8.18.3 branch leg 45/0 with every result row identical to the first commit's; es8 bridge 353/0
ISNULL / ISNOTNULL over an operand with no column failing before the fix on ES 8.18.3 and 6.8.23 (exists on an empty field; a CASE operand measured answering the wrong rows until it was enclosed). On the fix: QuotedArgumentConditionalSpec 28/0 on both, the #382 test passing on both, and on ES 8.18.3 GatewayApiIntegrationSpec, PredicateFunctionResultSpec, DateFunctionExecutionSpec, GroupByCompletenessSpec, GroupedAggregateSemanticsSpec all passing; sql 2345/0, es8 bridge 353/0, es6 bridge 360/0
Last commit: the implicit-cast target and NULLIF's literal-in-place test computed once per scope generated Painless byte-identical to the previous head on 116 statements, verdicts unchanged; two new pins (one node asked in every scope order answers as a fresh node) pass on both heads and fail on one-slot mutants of each memo; sql suites pass (2346, the one failure being an earlier version of the new spec, whose corrected version passes 20/0), es8 bridge 353/0, es6 bridge 360/0, core 1654/0, sbt "+ compile" OK; ES 8.18.3 leg 434/0 (DateArithmeticSpec, DateFunctionExecutionSpec, ScrollCompletenessSpec, GroupByCompletenessSpec, ComputedProjectionSpec, PredicateFunctionResultSpec, JavaClientSpec, QuotedArgumentConditionalSpec, JavaClientSchemaConsumerSpec, GatewayApiIntegrationSpec, TypesPredicatesFunctionsSpec, GreatestLeastSpec)
CI lint line (headerCheck scalafmtSbtCheck scalafmtCheck test:scalafmtCheck) green on the committed bytes

Apart from the ES 6.8 runs of the ISNULL / ISNOTNULL row, not run locally: ES 6.8, 7.17 and 9.0, which this PR's CI runs, including the trait rows. Transforms were compared at the rendering level only, with no run. The new rules decide at render time, memoised per scope, and add no per-row work.

Cost

Main (bc88e45c) against this branch, interleaved, warm JIT, JVMs started directly, 5 rounds with the order rotated, machine load below 8. Planning is what a request pays before Elasticsearch sees it: parse, schema attach, type validation and Painless emission, over 116 statements (110 with the same verdict on both trees). Medians of the per-round sums:

Stage main previous head (96b5b858) this branch
parse 30,342 µs +0.7% +0.4%
attach 535 µs +2.2% +1.9%
validate 464 µs +13.7% +8.3% (+0.35 µs per statement)
emit 1,586 µs +11.3% +1.2%, within noise
total 32,990 µs +1.6% +0.8%, within noise
  • The last commit removes the repeated derivations the earlier ones added (the implicit-cast target of each comparison, NULLIF's literal-in-place test). Its Painless is byte-identical to the previous head's on all 116 statements.
  • What remains in validate is the comparison table's own type rule (a NULLIF: 6.5 µs on main, 9.3 µs here).
  • WHERE id IN (1, 2, 3) over a KEYWORD id emits more because it now emits the conversion; main compared the text with the numbers and matched no row.
  • Per row, on Elasticsearch 8.18.3 (9 statements both trees accept): −1.4% to +3.4%, within main's own variation between runs (up to 18%); the Painless is identical.
  • Nested conditionals: main's validation re-derives the arm types on every call, ×20 to ×50 per nesting level. A CREATE TABLE with a NULLIF shape at depth 6 takes 3.7 s on main and 2.7 ms here; a date CASE at depth 6 takes over 20 s on main and 52 ms here.

Recorded, not fixed

Measured the same on main and on this branch:

  • CASE WHEN COALESCE('2024-01-15', d) BETWEEN k AND g THEN … (two text bounds) is accepted and fails when the query runs.
  • A text that spells the same date in another form (k = '2024-01-15T00:00:00Z'), or no date ('3'), compares as text with a literal-first COALESCE('2024-01-15', d): not equal. Comparing it as a date needs a runtime date parse of k.
  • A nested COALESCE(COALESCE(GREATEST(n, 3), l), x) still null-tests the inner number and fails to compile.
  • A per-group COALESCE(GREATEST(MAX(n), MAX(l)), 0) renders Math.max(...) != null; it is refused before it runs.
  • In an ingest pipeline, CASE WHEN n IS NULL THEN … renders a condition that is always false, so a SCRIPT AS column takes the ELSE where n is missing.
  • COALESCE(<arm>, 7) declares BIGINT, but a row that takes the literal holds an Integer until the typed result path converts it.
  • GREATEST(d, '…') compared in a pipeline fails to compile; a GREATEST holding a NULL literal in a pipeline renders Math.max(null, ...).
  • NULLIF inside a CASE WHEN condition is not judged by the NULLIF type rule.
  • HAVING MAX(keyword | text) > '2024-01-01' keeps main's refusal, and comparisons over a text column are left to Elasticsearch: 8 of the 25 cells, on the 0.25.0 gate list.

Measured on this branch (the rendering is unchanged from main, not run on main):

  • A text function over a NULLIF that answers NULL (UPPER(NULLIF(k, '1'))) calls the method on NULL and fails the query.

Introduced by this branch, value-safe, recorded:

  • After COALESCE stops at a never-NULL argument, the script may declare a parameter it never reads (def param1 = ctx.l). Values are unaffected.

Release preconditions (0.25.0)

🤖 Generated with Claude Code

fupelaqu and others added 4 commits November 9, 2026 06:42
…oes (SCHEMA-5b)

Story SCHEMA-5b. Reviewed: r0:must-fix, r1:must-fix, r2:must-fix, r3:must-fix, r4:clean.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…cript entry (SCHEMA-5b)

Story SCHEMA-5b. Reviewed: r5:clean.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…script (SCHEMA-5b)

A conditional operand (NULLIF, COALESCE, CASE) has no column for `exists`
to test, so Elasticsearch rejected the query. Such an operand now takes the
script path, in both bridges, and is tested as one value.

The #382 integration test now asserts what a query does with a text against
a number: accepted, and failing when a text that spells no number is read.
The computed-column refusal is unchanged.

Story SCHEMA-5b. Reviewed: r6:clean, r7:clean.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… once per scope (SCHEMA-5b)

The implicit-cast table made every comparison re-derive its target type at
each of its renderers, and NULLIF re-ask, on every read of an argument's
type, whether a literal-first COALESCE stands in for that argument.

- Expression.implicitComparisonTarget is memoised per scope (PerScope), as
  literalReading is; InExpr and BetweenExpr override the computation.
- ComparisonRule.literalInPlaceOf asks whether the operand is a COALESCE
  before it walks both chains for aggregates.
- NullIf.first / second are memoised per scope.

No verdict and no generated script changes. Two specs pin that one node
answers each scope (query, transform, processor, computed-column
validation) as a fresh node does, in any order.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@fupelaqu

Copy link
Copy Markdown
Contributor Author

Cost of this PR, measured on the pushed head (also added to the description).

Main (bc88e45c) against this branch, interleaved, warm JIT, JVMs started directly, 5 rounds with the order rotated, machine load below 8. Planning is what a request pays before Elasticsearch sees it: parse, schema attach, type validation and Painless emission, over 116 statements (110 with the same verdict on both trees). Medians of the per-round sums:

Stage main previous head (96b5b858) this branch
parse 30,342 µs +0.7% +0.4%
attach 535 µs +2.2% +1.9%
validate 464 µs +13.7% +8.3% (+0.35 µs per statement)
emit 1,586 µs +11.3% +1.2%, within noise
total 32,990 µs +1.6% +0.8%, within noise
  • The last commit removes the repeated derivations the earlier ones added (the implicit-cast target of each comparison, NULLIF's literal-in-place test). Its Painless is byte-identical to the previous head's on all 116 statements.
  • What remains in validate is the comparison table's own type rule (a NULLIF: 6.5 µs on main, 9.3 µs here).
  • WHERE id IN (1, 2, 3) over a KEYWORD id emits more because it now emits the conversion; main compared the text with the numbers and matched no row.
  • Per row, on Elasticsearch 8.18.3 (9 statements both trees accept): −1.4% to +3.4%, within main's own variation between runs (up to 18%); the Painless is identical.
  • Nested conditionals: main's validation re-derives the arm types on every call, ×20 to ×50 per nesting level. A CREATE TABLE with a NULLIF shape at depth 6 takes 3.7 s on main and 2.7 ms here; a date CASE at depth 6 takes over 20 s on main and 52 ms here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant