Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -1085,16 +1085,32 @@ package object bridge {
existsQuery(identifier.name)
}

/** Does this `ISNULL` / `ISNOTNULL` operand have no column to test? `exists` needs a field name
* and a conditional (`NULLIF(a, b)`, `COALESCE(a, b)`) has none: Elasticsearch rejects an
* `exists` on an empty field, so the script path, which renders the operand, answers it.
*/
private[bridge] def hasNoField(identifier: Identifier): Boolean = identifier.name.trim.isEmpty

implicit def isNullCriteriaToQuery(
isNull: IsNullCriteria
)(implicit
timestamp: Long,
contextType: PainlessContextType = PainlessContextType.Query,
target: PainlessTarget = PainlessTarget.Default
): Query = {
if (hasNoField(isNull.identifier)) return scriptQueryOf(isNull)
import isNull._
not(existsQuery(identifier.name))
}

implicit def isNotNullCriteriaToQuery(
isNotNull: IsNotNullCriteria
)(implicit
timestamp: Long,
contextType: PainlessContextType = PainlessContextType.Query,
target: PainlessTarget = PainlessTarget.Default
): Query = {
if (hasNoField(isNotNull.identifier)) return scriptQueryOf(isNotNull)
import isNotNull._
existsQuery(identifier.name)
}
Expand Down
189 changes: 188 additions & 1 deletion documentation/sql/functions_conditional.md

Large diffs are not rendered by default.

56 changes: 52 additions & 4 deletions documentation/sql/operators.md
Original file line number Diff line number Diff line change
Expand Up @@ -634,13 +634,17 @@ SELECT * FROM emp WHERE department = 'IT';
SELECT * FROM orders
WHERE customer_id = shipping_customer_id;

-- Columns of incompatible types (id KEYWORD, n INT) are refused by name once the column types
-- are known, rather than answering no rows
-- Since 0.25.0, a text column against a number column (id KEYWORD, n INT) is compared as DuckDB
-- casts it: the text is read as a number (see "Comparing values of different types" below)
SELECT * FROM t WHERE id = n;
-- Error: Type mismatch: 'KEYWORD' is not compatible with 'INT' in expression: id = n

-- Columns whose types no comparison reads (a KEYWORD against a DATE) are refused by name once the
-- column types are known, rather than answering no rows
SELECT * FROM t WHERE id = d;
-- Error: Type mismatch: 'KEYWORD' is not compatible with 'DATE' in expression: id = d

-- Cast one side to the other's type
SELECT * FROM t WHERE id = CAST(n AS VARCHAR);
SELECT * FROM t WHERE id = CAST(d AS VARCHAR);
```

**String Comparison:**
Expand Down Expand Up @@ -681,6 +685,50 @@ SELECT * FROM emp WHERE manager = NULL; -- Returns no rows!
SELECT * FROM emp WHERE manager IS NULL;
```

**Comparing values of different types** *(since `0.25.0`)*:

A comparison between a text and a number or a boolean, and between a boolean and a number, answers
for the operators DuckDB binds, as DuckDB casts it:

| Left / right | Operators | Compared as |
|--------------------------------------|---------------------------------------------|--------------------|
| text and `TINYINT` … `BIGINT` | `=`, `<>`, `!=`, `IN`, `NULLIF`, `CASE x WHEN` | `BIGINT` |
| text and `DOUBLE`, `REAL`, `NUMERIC` | `=`, `<>`, `!=`, `IN`, `NULLIF`, `CASE x WHEN` | `DOUBLE` |
| text and `BOOLEAN` | `=`, `<>`, `!=`, `IN`, `NULLIF`, `CASE x WHEN` | `BOOLEAN` |
| `BOOLEAN` and `TINYINT` … `BIGINT` | every comparison, `BETWEEN`, `IN` | the whole type (`1` / `0`) |
| `BOOLEAN` and `DOUBLE`, `REAL` | `=`, `<>`, `!=`, `IN`, `NULLIF`, `CASE x WHEN` | `DOUBLE` (`1.0` / `0.0`) |

```sql
-- k KEYWORD holds '1', ' 1 ', '1.5'; n INT holds 1, 1, 2
SELECT * FROM t WHERE k = n; -- the three rows: '1.5' is read as 2
SELECT CASE WHEN k = 1 THEN 'one' ELSE 'other' END AS v FROM t;
SELECT NULLIF(k, 0) AS v FROM t; -- k itself, NULL where k reads as 0
SELECT * FROM t WHERE b = n; -- b BOOLEAN read as 1 / 0
```

- A text is read as a whole number by rounding half away from zero (`'1.5'` is `2`, `'-2.5'` is
`-3`), as a `DOUBLE` by the number it spells, as a `BOOLEAN` by the spellings a boolean literal
reads (`t`, `true`, `y`, `yes`, `on`, `1` / `f`, `false`, `n`, `no`, `off`, `0`, any case,
surrounding spaces ignored). A stored text that spells no such value (`'abc'`, `''`, `'1,5'`,
`'NaN'`, `'2'` beside a boolean) **fails the query**, as DuckDB fails it. `'on'`, `'off'` and
`' true '` beside a boolean answer, where DuckDB fails.
- Ordering a text against a number (`k < n`, `k BETWEEN 1 AND 2`) stays refused, as DuckDB refuses
it; so does a text against a `DATE` or a `TIMESTAMP` column.
- Over an `ARRAY<X>` column, a query compares the element it reads.
- A `HAVING` group filter reads a boolean metric as `1.0` / `0.0`: `HAVING MAX(b) > 0`.
- A bare column compared with a literal in a `WHERE` clause (`WHERE k = 1`) is still a query on the
column's mapping, which Elasticsearch coerces as it always has; the rule above applies to a
scripted comparison (`CASE WHEN k = 1`, a function of the column, two columns).
- Outside a query — a computed column (`SCRIPT AS`, including the ones a `CREATE TABLE … AS
SELECT` copies), an ingest pipeline, a materialized view's transform — a text side keeps
`0.24.0`'s verdict: `NULLIF(k, 0)` is refused there, because a pipeline would silently drop a
document whose text converts to nothing. The boolean rows apply everywhere. A conditional that
reads a quoted argument as a number is that number there too (see
[Conditional functions](functions_conditional.md)), so `NULLIF(COALESCE(n, '3'), k)`,
`NULLIF(k, COALESCE(n, '3'))` and `NULLIF(COALESCE(n, '3'), '2024-01-15')` are refused in a
computed column, where `0.24.0` accepted them over a pipeline that stored no value on every
document holding `n` and `k`.

---

### Operator: `<>`, `!=`
Expand Down
18 changes: 18 additions & 0 deletions documentation/sql/type_conversion.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,24 @@ SELECT '2025-09-11'::DATE AS d, '125'::BIGINT AS b FROM logs;
- `TRY_CAST` (`SAFE_CAST`) returns `NULL` instead of failing.
- `::` is syntactic sugar, easier to read in queries.
- Type inference relies on `baseType`, and explicit `CAST`/`TRY_CAST`/`::` updates the type context for following functions.
- *Since `0.25.0`.* A `BOOLEAN` cannot be cast to a `DATE`, a `TIME` or a `TIMESTAMP` — in a query,
a computed column and a pipeline alike, with `CAST`, `TRY_CAST` or `::` — as DuckDB refuses it:
`CAST(b AS DATE)` used to answer the boolean itself in a date column.
- *Since `0.25.0`.* A comparison between values of different types reads them as DuckDB casts them
(see [Comparing values of different types](operators.md#operator-)): a text compared with a
number is read as one, rounding half away from zero for a whole type. `CAST(k AS BIGINT)` keeps
its strict parse (`'1.5'` fails the cast).
- *Since `0.25.0`.* A string function (`UPPER`, `LOWER`, `TRIM`, `LENGTH`, `SUBSTRING`, `LEFT`,
`RIGHT`, `REPLACE`, `REVERSE`) over a number or a boolean reads its text: `UPPER(n)` and
`LENGTH(n)` answer (they failed the query), and over a `CASE` they read the whole `CASE`'s value.
The same holds in a computed column (`SCRIPT AS`), whose pipeline `ALTER TABLE` re-creates (see
[Conditional functions](functions_conditional.md)).
- *Since `0.25.0`.* A text that spells a number past the `DOUBLE` range (`'1e400'`, `'-1e400'`) is
read in two ways, as `0.24.0`'s literal rule and DuckDB read it. As a quoted literal it reads as no
number, like `'NaN'` and `'Infinity'`: `COALESCE(x, '1e400')` answers the text and
`GREATEST(x, '1e400')` is refused (DuckDB answers `1.5` and `inf`). As a stored text compared with
a `DOUBLE` column it is `+Infinity` or `-Infinity`, DuckDB's value: `k = x` is false and `k <> x`
true. A stored `'NaN'` or `'Infinity'` fails the query.

---

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1072,16 +1072,32 @@ package object bridge {
existsQuery(identifier.name)
}

/** Does this `ISNULL` / `ISNOTNULL` operand have no column to test? `exists` needs a field name
* and a conditional (`NULLIF(a, b)`, `COALESCE(a, b)`) has none: Elasticsearch rejects an
* `exists` on an empty field, so the script path, which renders the operand, answers it.
*/
private[bridge] def hasNoField(identifier: Identifier): Boolean = identifier.name.trim.isEmpty

implicit def isNullCriteriaToQuery(
isNull: IsNullCriteria
)(implicit
timestamp: Long,
contextType: PainlessContextType = PainlessContextType.Query,
target: PainlessTarget = PainlessTarget.Default
): Query = {
if (hasNoField(isNull.identifier)) return scriptQueryOf(isNull)
import isNull._
not(existsQuery(identifier.name))
}

implicit def isNotNullCriteriaToQuery(
isNotNull: IsNotNullCriteria
)(implicit
timestamp: Long,
contextType: PainlessContextType = PainlessContextType.Query,
target: PainlessTarget = PainlessTarget.Default
): Query = {
if (hasNoField(isNotNull.identifier)) return scriptQueryOf(isNotNull)
import isNotNull._
existsQuery(identifier.name)
}
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
/*
* Copyright 2025 SOFTNETWORK
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package app.softnetwork.elastic.client

import app.softnetwork.elastic.client.spi.JestClientSpi
import app.softnetwork.elastic.scalatest.ElasticDockerTestKit

class JestClientQuotedArgumentConditionalSpec
extends QuotedArgumentConditionalSpec
with ElasticDockerTestKit {
override lazy val client: GatewayApi = new JestClientSpi().client(elasticConfig)
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
/*
* Copyright 2025 SOFTNETWORK
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package app.softnetwork.elastic.client

import app.softnetwork.elastic.client.spi.RestHighLevelClientSpi
import app.softnetwork.elastic.scalatest.ElasticDockerTestKit

class RestHighLevelClientQuotedArgumentConditionalSpec
extends QuotedArgumentConditionalSpec
with ElasticDockerTestKit {
override lazy val client: GatewayApi = new RestHighLevelClientSpi().client(elasticConfig)
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
/*
* Copyright 2025 SOFTNETWORK
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package app.softnetwork.elastic.client

import app.softnetwork.elastic.client.spi.RestHighLevelClientSpi
import app.softnetwork.elastic.scalatest.ElasticDockerTestKit

class RestHighLevelClientQuotedArgumentConditionalSpec
extends QuotedArgumentConditionalSpec
with ElasticDockerTestKit {
override lazy val client: GatewayApi = new RestHighLevelClientSpi().client(elasticConfig)
}
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,21 @@ class JavaClientSchemaConsumerSpec extends AnyFlatSpecLike with ElasticDockerTes
"s2" -> """{"id":"s2","k":"b","sf":2.5,"hf":1.5,"u":8,"dn":"2024-02-01T00:00:00Z","ip":"10.0.0.2"}"""
)

// SCHEMA-5b (D14): a quoted argument read as the type beside it, a BOOLEAN beside a whole number,
// a text compared with a number
private val s5 = "s5c_t"

private val s5Mapping: String =
"""{"properties": {
| "id": {"type": "keyword"}, "n": {"type": "integer"}, "b": {"type": "boolean"},
| "k": {"type": "keyword"}
|}}""".stripMargin

private val s5Docs: Seq[(String, String)] = Seq(
"q1" -> """{"id":"q1","n":1,"b":true,"k":"5"}""",
"q2" -> """{"id":"q2","b":false,"k":"0"}"""
)

private def load(index: String, mappingJson: String, documents: Seq[(String, String)]): Unit = {
Try(client.deleteIndex(index))
client
Expand All @@ -112,10 +127,11 @@ class JavaClientSchemaConsumerSpec extends AnyFlatSpecLike with ElasticDockerTes
load(s7, s7Mapping, s7Docs)
load("s7c_p1", """{"properties": {"k": {"type": "keyword"}}}""", Seq("p1" -> """{"k":"007"}"""))
load("s7c_p2", """{"properties": {"k": {"type": "long"}}}""", Seq("p2" -> """{"k":7}"""))
load(s5, s5Mapping, s5Docs)
}

override def afterAll(): Unit = {
Seq(table, s7, "s7c_p1", "s7c_p2").foreach(index => Try(client.deleteIndex(index)))
Seq(table, s7, "s7c_p1", "s7c_p2", s5).foreach(index => Try(client.deleteIndex(index)))
super.afterAll()
}

Expand Down Expand Up @@ -484,4 +500,30 @@ class JavaClientSchemaConsumerSpec extends AnyFlatSpecLike with ElasticDockerTes
rows.map(_("ck")).distinct shouldBe Seq("C")
rowsAndSchema(run(s"SELECT k FROM $s7Pattern"))._1.map(_("k")).toSet shouldBe Set("007", "7")
}

"Arrow" should "read SCHEMA-5b's conditionals by the types core reports (D14)" in {
val sql = s"SELECT id, COALESCE(n, '3') AS c, GREATEST(b, 1) AS g, NULLIF(k, 0) AS z FROM $s5"
val result = run(sql)
val (rows, schema) = rowsAndSchema(result)
extractRows(result).map(_.toMap) shouldBe rows.map(_.toMap)
// COALESCE(n, '3') was VARCHAR, GREATEST(b, 1) and NULLIF(k, 0) refused, before SCHEMA-5b
schema.map(_.columns.map(c => c.name -> c.sqlType.typeId)) shouldBe Some(
Seq("id" -> "VARCHAR", "c" -> "INT", "g" -> "BIGINT", "z" -> "VARCHAR")
)
rows
.map(r => String.valueOf(r("id")) -> ((r("c"), r("g"), r.getOrElse("z", null))))
.toMap shouldBe Map(
"q1" -> ((Integer.valueOf(1), java.lang.Long.valueOf(1L), "5")),
"q2" -> ((Integer.valueOf(3), java.lang.Long.valueOf(1L), null))
)
}

"The JDBC driver" should "read SCHEMA-5b's conditionals by its match (D14)" in {
val rows = extractRows(run(s"SELECT id, COALESCE(n, '3') AS c, GREATEST(b, 1) AS g FROM $s5"))
rows
.map(r => ValueCoercion.coerceToString(r("id")) -> ValueCoercion.coerceToString(r("c")))
.toMap shouldBe
Map("q1" -> "1", "q2" -> "3")
rows.map(r => ValueCoercion.coerceToString(r("g"))).toSet shouldBe Set("1")
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
/*
* Copyright 2025 SOFTNETWORK
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package app.softnetwork.elastic.client

import app.softnetwork.elastic.client.spi.JavaClientSpi
import app.softnetwork.elastic.scalatest.ElasticDockerTestKit

class JavaClientQuotedArgumentConditionalSpec
extends QuotedArgumentConditionalSpec
with ElasticDockerTestKit {
override lazy val client: GatewayApi = new JavaClientSpi().client(elasticConfig)
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
/*
* Copyright 2025 SOFTNETWORK
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package app.softnetwork.elastic.client

import app.softnetwork.elastic.client.spi.JavaClientSpi
import app.softnetwork.elastic.scalatest.ElasticDockerTestKit

class JavaClientQuotedArgumentConditionalSpec
extends QuotedArgumentConditionalSpec
with ElasticDockerTestKit {
override lazy val client: GatewayApi = new JavaClientSpi().client(elasticConfig)
}
Loading
Loading