Skip to content

feat(core): describe a JOIN view body per table without running it - #462

Merged
fupelaqu merged 1 commit into
mainfrom
feature/SCHEMA-15
Nov 13, 2026
Merged

fupelaqu merged 1 commit into
mainfrom
feature/SCHEMA-15

Conversation

@fupelaqu

@fupelaqu fupelaqu commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adds GatewayApi.describePerTable(statement), an internal (private[elastic]) API that gives the column types of a JOIN body without running it. Each column is typed from the mapping of its own table. A materialized view needs these types when it is created, because Elasticsearch materializes the view: the relational engine never runs its body. Today describe refuses such a body with the cross-index JOIN 400 when the relational engine is not there.

describe, run and every public answer are unchanged.

What changes

  • describePerTable:
    • Which statements it types itself: one FROM table, standard JOIN legs over indices, and none of the following: derived table, CTE, set operation, correlated subquery, WHERE subquery. This is the JOIN shape RelationalClosureGuard.shapeOf names, now held in RelationalClosureGuard.JoinShape. The message bytes are unchanged.
    • How it types them: by describe's own builder. It reads every leg's mapping, attaches it, then runs validateResolved and statementSchema. So per-major dates, _meta types, index patterns and the _field_caps fallback for a user who may not read the mapping all follow the single-table rules.
    • When it answers None:
      • a leg's schema is unknown (every leg is read first, then all legs or none);
      • a column's qualifier names no leg (one index under two qualifiers, an UNNEST alias);
      • an item is a star.
    • A type error is a 400 with operation = "describe", the same refusal as over the single-table twin.
    • Any other statement gets describe's answer, verbatim.
    • Nothing runs: no search and no subquery. Tests count the searches and assert 0.
  • SearchApi:
    • resolvePerTable and legSchema: one lookupSchema per leg, through the cache and the reload memo that run uses;
    • reloadedFor takes its paths by name, so the single-table path is unchanged;
    • explicitPaths takes an optional filter, so one leg's selected paths can be picked.
  • ElasticClientDelegator forwards describePerTable to its delegate. A recording delegate behind MetricsElasticClient sees exactly one call.
  • Bare column names follow what a materialized view stores today: a bare name takes the main table's column.

Verification

  • A test in package app.softnetwork.elastic.extensions, with the client statically typed ElasticClientApi, calls describePerTable(create.search) on a parsed CREATE MATERIALIZED VIEW. It is green on Elasticsearch 8.18.3. This proves the API is reachable from that package.

  • The new tests were RED on main first (7 cases, the member absent), then green: DescribeSpec 14/0.

  • core/test 1663/0, sql MultiSchemasSpec 7/0, and sbt "+ compile" (2.12.20 and 2.13.16) with no error.

  • Docker-free client-module specs, 0 failed: es6rest 32, es6jest 58, es7rest 31, es8java 40, es9java 34.

  • Elasticsearch 8.18.3:

    • the new runner, MappingTypeReadingSpec and JavaClientReadOnlyUserSpec: 27/0;
    • the schema-reading area (JavaClientSchemaReadingSpec, JavaClientSchemaConsumerSpec, JavaClientSpec, JavaClientWhereSubqueryCompletenessSpec, JavaClient8ReplGatewayIntegrationSpec): 168/0.

    The Elasticsearch 6.8, 7.17 and 9.0 runners of PerTableDescribeSpec run in CI.

  • An independent review found nothing. It checked, among other things:

    • a joined leg with custom date formats, _meta types and date_nanos is typed as single-table describe types it, under the 6.8 and 8.x date rules;
    • a validation pass on the same AST beforehand leaves the answer unchanged;
    • 18 view bodies with WHERE / HAVING literals: no spurious refusal and no difference from the single-table twins.
  • Lint: headerCheck scalafmtSbtCheck scalafmtCheck test:scalafmtCheck passes on the committed bytes.

Cost

Median µs, measured by PerTableDescribeCostProbe (300 warm-up runs, then 1000 runs). The probe ran three times on each side, alternating main and this branch, each run in a fresh forked JVM. A run started only when the 1-minute load average was under 8. The machine had background load (6.5 to 9.8 during the runs), and main's own three runs differ by up to 2×.

statement main (median of 3) branch (median of 3) difference
single table 107.1 80.1 −25 %
joined table alone 40.2 22.3 −45 %
computed items 88.7 45.6 −49 %
GROUP BY 86.8 54.3 −37 %
* 66.7 50.6 −24 %
UNION ALL 60.2 78.2 +30 %
JOIN refused by describe (400) 24.6 12.3 −50 %

The JOIN refusal returns before any line this PR changes, yet it moves −50 % here, and moved +22 % in an earlier run that was not interleaved. Differences of that size are this machine's noise, in both directions. UNION ALL is the one shape slower on the branch in every warm run (+23 % to +44 %, about 15 µs). It crosses the changed code through the same line as the single-table statement, once per side of the UNION ALL, and the single-table statement is faster on the branch. No regression can be attributed to the change.

describePerTable on the JOIN body takes 113.8 to 187.1 µs (median 149.6), 1.07× to 1.95× the two single-table describe calls it replaces. It runs once, when a materialized view is created.

Release note

  • Downstream rebuild: GatewayApi, SearchApi, ElasticClientDelegator and RelationalClosureGuard gain members (internal only), so classes that mix these traits must be rebuilt. No case-class arity changes, and no public type changes.
  • No documentation change: the API is internal.

Recorded, not fixed

  • A bare column name in a JOIN body is attributed to the main table, as the materialized view generator already does. Run as a query, the same body goes to DuckDB, which binds a name only the joined table has to that table, and refuses as ambiguous a name both tables have. This divergence predates this PR, and this PR does not widen it. Aligning it is separate work.

🤖 Generated with Claude Code

…CHEMA-15)

Story SCHEMA-15. Reviewed: r0:clean.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@fupelaqu
fupelaqu marked this pull request as ready for review November 13, 2026 08:47
@fupelaqu
fupelaqu merged commit ec9b065 into main Nov 13, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant