Repository navigation
feat(core): describe a JOIN view body per table without running it - #462
Merged
Merged
Conversation
…CHEMA-15) Story SCHEMA-15. Reviewed: r0:clean. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
GatewayApi.describePerTable(statement), an internal (private[elastic]) API that gives the column types of a JOIN body without running it. Each column is typed from the mapping of its own table. A materialized view needs these types when it is created, because Elasticsearch materializes the view: the relational engine never runs its body. Todaydescriberefuses such a body with the cross-index JOIN 400 when the relational engine is not there.describe,runand every public answer are unchanged.What changes
describePerTable:FROMtable, standardJOINlegs over indices, and none of the following: derived table, CTE, set operation, correlated subquery,WHEREsubquery. This is the JOIN shapeRelationalClosureGuard.shapeOfnames, now held inRelationalClosureGuard.JoinShape. The message bytes are unchanged.describe's own builder. It reads every leg's mapping, attaches it, then runsvalidateResolvedandstatementSchema. So per-major dates,_metatypes, index patterns and the_field_capsfallback for a user who may not read the mapping all follow the single-table rules.None:UNNESTalias);operation = "describe", the same refusal as over the single-table twin.describe's answer, verbatim.SearchApi:resolvePerTableandlegSchema: onelookupSchemaper leg, through the cache and the reload memo thatrunuses;reloadedFortakes its paths by name, so the single-table path is unchanged;explicitPathstakes an optional filter, so one leg's selected paths can be picked.ElasticClientDelegatorforwardsdescribePerTableto its delegate. A recording delegate behindMetricsElasticClientsees exactly one call.Verification
A test in package
app.softnetwork.elastic.extensions, with the client statically typedElasticClientApi, callsdescribePerTable(create.search)on a parsedCREATE MATERIALIZED VIEW. It is green on Elasticsearch 8.18.3. This proves the API is reachable from that package.The new tests were RED on
mainfirst (7 cases, the member absent), then green:DescribeSpec14/0.core/test1663/0,sqlMultiSchemasSpec7/0, andsbt "+ compile"(2.12.20 and 2.13.16) with no error.Docker-free client-module specs, 0 failed: es6rest 32, es6jest 58, es7rest 31, es8java 40, es9java 34.
Elasticsearch 8.18.3:
MappingTypeReadingSpecandJavaClientReadOnlyUserSpec: 27/0;JavaClientSchemaReadingSpec,JavaClientSchemaConsumerSpec,JavaClientSpec,JavaClientWhereSubqueryCompletenessSpec,JavaClient8ReplGatewayIntegrationSpec): 168/0.The Elasticsearch 6.8, 7.17 and 9.0 runners of
PerTableDescribeSpecrun in CI.An independent review found nothing. It checked, among other things:
_metatypes anddate_nanosis typed as single-tabledescribetypes it, under the 6.8 and 8.x date rules;WHERE/HAVINGliterals: no spurious refusal and no difference from the single-table twins.Lint:
headerCheck scalafmtSbtCheck scalafmtCheck test:scalafmtCheckpasses on the committed bytes.Cost
Median µs, measured by
PerTableDescribeCostProbe(300 warm-up runs, then 1000 runs). The probe ran three times on each side, alternatingmainand this branch, each run in a fresh forked JVM. A run started only when the 1-minute load average was under 8. The machine had background load (6.5 to 9.8 during the runs), andmain's own three runs differ by up to 2×.GROUP BY*UNION ALLdescribe(400)The JOIN refusal returns before any line this PR changes, yet it moves −50 % here, and moved +22 % in an earlier run that was not interleaved. Differences of that size are this machine's noise, in both directions.
UNION ALLis the one shape slower on the branch in every warm run (+23 % to +44 %, about 15 µs). It crosses the changed code through the same line as the single-table statement, once per side of theUNION ALL, and the single-table statement is faster on the branch. No regression can be attributed to the change.describePerTableon the JOIN body takes 113.8 to 187.1 µs (median 149.6), 1.07× to 1.95× the two single-tabledescribecalls it replaces. It runs once, when a materialized view is created.Release note
GatewayApi,SearchApi,ElasticClientDelegatorandRelationalClosureGuardgain members (internal only), so classes that mix these traits must be rebuilt. No case-class arity changes, and no public type changes.Recorded, not fixed
🤖 Generated with Claude Code