Repository navigation
feat: sync parser with server SQL (0.1.21 → current, OSS + Enterprise) - #42
Merged
Merged
Conversation
Queries copied from the QuestDB server test suites live in tests/fixtures/server-queries.json, each with its source file and line. tests/server-queries.test.ts checks every entry four ways: it parses as the expected number of statements, toSql matches the reviewed output and round-trips, autocomplete suggests each next token, and the formatter preserves it. The token-by-token autocomplete walkthrough moves from docs-autocomplete.test.ts to tests/helpers so both suites share it.
The server accepts four literal forms that sql-parser misread. Most of them were silent: the query parsed without errors but with a different meaning. 0x7ee6... long256 constant was `0 AS x7ee6...` or split in two 1.5f, 1e36d float/double suffix was `1.5 AS f` or split in two E'abc' PostgreSQL string was `E AS abc` or split in two timestamp '..' typed literal (cast) was a column with an alias, or split Lexer: HexLiteral, FloatLiteral and EscapeStringLiteral tokens. `d` is only a double suffix after an exponent, so `1d` stays a day duration. Parser: the typed literal hangs off identifierExpression as an optional string after a castable type name (the server's processPgTimestampCast list plus nameTypeMap aliases). A separate primaryExpression alternative would be simpler, but autocomplete ignores GATEs, so it broke ARRAY[ suggestions and offered every type keyword wherever a value can start. `timestamp [ns] with time zone '...'` is its own small rule; content assist drops its leading TIMESTAMP tokens so only WITH after `timestamp` is new. Tests: 26 server queries (questdb + questdb-enterprise test suites) in the parity fixture, checked for parse, round trip, autocomplete and formatter, plus tests/literal-forms.test.ts for lexer, AST and autocomplete edge cases. Against 53,975 SQL strings from the server tests and docs: 218 now parse, 25 round trips are corrected, and the only newly rejected input is `cancel query 12.01f`, which the server rejects too.
Fixes a regression on master and adds the grouping-sets syntax of questdb/questdb#6853. The server (SqlParser.parseGroupByColumns, parseExplicitGroupingSets, hasGroupingSyntaxLookahead and the SAMPLE BY branch) accepts: GROUP BY ROLLUP(a, ...) | CUBE(a, ...) GROUP BY GROUPING SETS ((a, b), (a), ()) every set parenthesised GROUP BY expr, ..., ROLLUP(...) | CUBE(...) ROLLUP/CUBE last SAMPLE BY <interval> ROLLUP(...) | CUBE(...) | GROUPING SETS (...) [FROM ... TO ...] [FILL(...)] [ALIGN TO ...] Nothing may follow ROLLUP/CUBE/GROUPING SETS, GROUPING SETS must be the only GROUP BY item, and keys must be column references (ensureColumnReferences). cube, rollup, grouping and sets are not reserved: they are grouping syntax only when `(` (ROLLUP/CUBE) or SETS (GROUPING) follows. Before: #11 added the four words as reserved keywords with no grammar, so `SELECT rollup, cube, sets, grouping FROM t`, `CREATE TABLE t (sets INT, cube INT)` and `SELECT grouping(a) FROM t GROUP BY a` failed (they worked in 0.1.21), and every ROLLUP/CUBE/GROUPING SETS query failed. Now: - tokens.ts: Cube, Grouping, Rollup, Sets are non-reserved (IDENTIFIER_KEYWORD_NAMES), which alone fixes the regression. - parser: groupByClause takes groupingSetsSpec when LA(1..2) is ROLLUP( / CUBE( / GROUPING SETS, otherwise plain expressions with an optional trailing `, ROLLUP(...)|CUBE(...)`. The LA gates keep `GROUP BY rollup`, `GROUP BY a, rollup` and `GROUP BY grouping(x)` as plain keys, and stop `GROUP BY a, grouping sets (...)` (rejected by the server) from being read as `grouping` plus a second statement. sampleByClause takes an optional groupingSetsSpec right after the interval. - AST: new GroupingSets node {kind: rollup|cube|groupingSets, columns | sets}; SelectStatement.groupingSets holds the GROUP BY part (plain keys stay in groupBy), SampleByClause.groupingSets the SAMPLE BY part. Existing fields are unchanged. - toSql prints ROLLUP(...), CUBE(...), GROUPING SETS ((...), ()). - Autocomplete offers ROLLUP/CUBE/GROUPING after GROUP BY and after the SAMPLE BY interval, ROLLUP/CUBE after a plain GROUP BY key, SETS after GROUPING, and columns inside the parentheses. Not done: the server also runs parseGroupByColumns for a GROUP BY after `FROM t PIVOT (...)`; sql-parser does not accept any GROUP BY after a PIVOT clause (clause order), so that is left for a PIVOT change. The GROUP BY inside PIVOT (...) only takes plain keys on the server too. Tests: 28 server queries (GroupingSetsTest, LateralJoinTest, SampleByFillTest) in the parity fixture, plus tests/grouping-sets.test.ts (identifiers, AST, round trip, rejected forms, autocomplete). Red before: 128 failing (84 fixture checks + 44 focused); after: all 303 pass, full suite 15491 passed. Corpus (53,975 server/docs SQL): 226 fixed, 0 round trips changed, 0 regressed, 0 throws, formatter meaning unchanged.
…ut ENABLE Adds the dedup syntax of questdb/questdb#7885 and fixes older dedup gaps (reports §1.2). The server accepts (SqlParser CREATE TABLE dedup block, SqlCompilerImpl.alterTableDedupEnable and the ALTER TABLE dedup branch): CREATE TABLE / CTAS ... (DEDUP | DEDUPLICATE) (UPSERT | SKIP) KEYS(c, ...) [FORMAT PARQUET | NATIVE] [OWNED BY ...] ALTER TABLE t (DEDUP | DEDUPLICATE) DISABLE ALTER TABLE t (DEDUP | DEDUPLICATE) [ENABLE] (UPSERT | SKIP) KEYS(c, ...) ENABLE has been optional since v7.3. SHOW CREATE TABLE prints `DEDUP SKIP KEYS(...)`. `skip` and `deduplicate` are not reserved (neither is in SqlKeywords.KEYWORDS; upsert is, and stays reserved). Before: every SKIP form failed ("Expecting Upsert"), ALTER ... DEDUP UPSERT KEYS without ENABLE failed, ALTER ... DEDUPLICATE failed, and `... WAL DEDUPLICATE UPSERT KEYS(ts)` was silently split into three statements. `skip` was a reserved token with no grammar, so `SELECT skip FROM t` failed. Now: - keywords.ts: new `deduplicate`; tokens.ts: Deduplicate and Skip are non-reserved (IDENTIFIER_KEYWORD_NAMES). - parser: new rules dedupKeyword (DEDUP | DEDUPLICATE) and dedupModeKeys ((UPSERT | SKIP) KEYS(...)), shared by the CREATE TABLE dedupClause and the ALTER TABLE dedup action, where ENABLE is now optional. - AST: new optional `dedupMode: "upsert" | "skip"` on CreateTableStatement (set whenever dedupKeys is) and on DedupEnableAction. Existing fields are unchanged. - toSql prints the mode instead of the hard-coded UPSERT (absent mode still prints UPSERT). DEDUPLICATE is printed as DEDUP and ALTER always prints ENABLE (canonical form; same meaning on the server). The existing output spacing (`KEYS (..)` for ALTER) is kept. - Formatter: DEDUPLICATE is a clause/action phrase like DEDUP. - Autocomplete: UPSERT and SKIP after DEDUP / DEDUPLICATE, plus ENABLE and DISABLE after ALTER TABLE t DEDUP; DEDUPLICATE is offered wherever DEDUP is. Tests: 16 server queries (DedupSkipTest, CreateTableDedupTest, ShowCreateTableTest text block, WalWriterTest, ImportIODispatcherTest, O3ParquetStaleReaderTest, AlterTableSetTypeRestartTest, Enterprise EntShowCreateTableTest) in the parity fixture, plus tests/dedup-skip.test.ts (AST mode, round trips incl. FORMAT PARQUET / OWNED BY / CTAS, rejected forms, skip/deduplicate as identifiers, autocomplete). Red before: 83 failing (48 fixture checks + 35 focused); after: all 346 pass, full suite 15604 passed. Corpus (53,975 server/docs SQL): 27 fixed, 0 round trips changed, 0 regressed, 0 throws, formatter meaning unchanged.
… VOLUME Server (OSS #7263, SqlParser.parseExpireRowsClause) parses the relative retention form as `KEEP [N] HIGHEST|LOWEST ON <col> [PARTITION BY <cols>]` (expectTok "on"), and SHOW CREATE MATERIALIZED VIEW prints it with ON (RowExpiryUtil.appendKeepByClause). RowExpiryWindowTest:966-1001 asserts "'on' expected" for the form without ON. In CREATE MATERIALIZED VIEW the WHEN predicate ends at the clause boundary (expireRowsClauseBoundary): CLEANUP EVERY, IN VOLUME (and WITH / DEDUP, which mat views then reject; Ent OWNED BY via isExpireRowsClauseBoundary). Before: `... EXPIRE ROWS KEEP 2 LOWEST ON v PARTITION BY k` failed with "Unexpected token 'ON'" in CREATE and ALTER MATERIALIZED VIEW ... SET EXPIRE ROWS, while `KEEP HIGHEST v ...` (server error) was accepted. `EXPIRE ROWS WHEN v < 2.0 IN VOLUME 'v1'` was silently misparsed: IN was read as the IN operator, round trip `... IN VOLUME; v1` (two statements). Change: - parser expireRowsClause: CONSUME(On) between HIGHEST|LOWEST and the column. The WHEN predicate is parsed with a parser flag (inExpireRowsPredicate, restored in finally); setExpression's IN/BETWEEN option is gated off when the flag is set and the next tokens are IN VOLUME, so the predicate ends there and the mat view IN VOLUME clause takes it. Outside EXPIRE ROWS `x IN volume` is unchanged. CLEANUP, WITH, DEDUP and OWNED are not operators, so the predicate already stopped there. - toSql prints `HIGHEST|LOWEST ON <col>`. AST unchanged (keepExtremum `column` already holds the ON column; visitor reads ctx.On only in the KEEP LATEST branch). - tests/fixtures/docs-queries.json: the docs example used the old form without ON; now `KEEP HIGHEST ON px ...` (upstream docs need the same fix). Tests: 17 server queries (MatViewExpireRowsTest, MatViewExpireRowsHardeningTest text block, RowExpiryKeepLatestTest, RowExpiryWindowTest, SqlCompilerImplTest incl. IN VOLUME + OWNED BY, Ent EntRowExpiryAclTest), tests/expire-rows-keep-on.test.ts (AST, toSql, negatives without ON, IN operator kept inside the predicate, `x IN volume` outside, autocomplete), parser.test.ts EXPIRE ROWS describes, a formatter fixture. Red 74 failing -> green; full suite 15702 passed. Corpus: fixed 26 / rtChanged 0 / regressed 6 / throws 0 / formatterChangedMeaning 0. All 6 regressions are the form without ON that the server rejects: 5 from RowExpiryWindowTest:973-993 (expects "'on' expected") and the old docs example. Suggestions: +ON after KEEP [N] HIGHEST|LOWEST (CREATE and ALTER); +PARTITION, CLEANUP, IN, OWNED after KEEP ... ON <col>. Left out: `CLEANUP EVERY h` without a number (server defaults to 1, low priority).
The server parses the query body of CREATE MATERIALIZED VIEW
(SqlParser.parseCreateMatView), CREATE TABLE AS (parseCreateTableAsSelect,
'(' required), CREATE VIEW (parseCreateView), ALTER VIEW (parseViewSql) and
COPY (...) TO (parseCopy) with parseDml, which reads a bare table, a table
with clauses, a UNION of tables, a table function or `t PIVOT (...)` as an
implicit SELECT * FROM .... OSS #7263 made passthrough mat views valid
(MatViewTest: `create materialized view v5 as (base_price where price > 0)`,
`... v4 as base_price`).
Before: all of these failed with "Expecting Select" / "Unexpected token",
except the parenthesised CREATE VIEW form (no set operations there).
`CREATE TABLE y AS (x UNION ALL append)`, `COPY (t WHERE x > 0) TO ...`,
`copy (generate_series(...)) TO ...`, `ALTER VIEW v AS t WHERE ...` and
`CREATE VIEW v AS t WHERE x > 0` all errored.
Change:
- createMaterializedViewBody, createViewBody, alterViewStatement: body is
`(SELECT...)` | `(implicitSelectStatement)` | `SELECT...` |
unparenthesised implicitSelectStatement (gated on LA(1) != '('), as
createViewBody already did for the parenthesised implicit form. The mat
view body previously had independent optional '(' and ')', which also
accepted unbalanced parentheses; they are now paired.
- createTableBody (CTAS) and copyTo: inside the required parentheses,
selectStatement (LA(1) SELECT/WITH/DECLARE) or implicitSelectStatement.
- implicitSelectBody gains the optional PIVOT (...) clause (same position
as in simpleSelect), so `(t PIVOT (...) ORDER BY ...)` works in COPY and
in FROM subqueries/CTEs, as on the server. The top-level
`t PIVOT (...)` still goes to pivotStatement (unchanged AST).
- createViewBody now uses implicitSelectStatement (with UNION/EXCEPT/
INTERSECT) instead of implicitSelectBody.
- Visitor: implicit bodies become SelectStatement { implicit: true } in the
existing fields (CreateMaterializedViewStatement.query, CreateTable
asSelect, CreateView/AlterView query, CopyTo source). No AST type
changes. Mat view asParens is now taken from the body parenthesis only:
before, `AS SELECT ... TIMESTAMP(ts)` set asParens because TIMESTAMP's
'(' is in the same CST node. toSql unchanged (implicit selects already
print without SELECT * FROM; ALTER VIEW always prints parentheses).
- grammar-drift test: Select/Declare/With listed as the inline mat view
query (as for live views), since the unparenthesised SELECT is now an
OR alternative.
Tests: 20 server queries (MatViewTest x5, ArrayTest, O3FailureTest x2,
O3Test x4, QueryRegistryTest, IPv4Test, SqlParserTest x2 incl. ALTER VIEW,
Ent EntSqlParserTest, CopyExportTest x4) and
tests/implicit-select-bodies.test.ts (AST shape, round trips, unbalanced
parentheses / clauses after a parenthesised body / CTAS without '('
rejected, bare `COPY t TO` unchanged, top-level PIVOT unchanged,
autocomplete). Red 95 failing (55 fixture + 40 focused) -> green; full
suite 15833 passed.
Corpus: fixed 31 / rtChanged 0 / regressed 8 / throws 0 /
formatterChangedMeaning 0. Fixed: the queries above plus more CTAS
bare-table forms and `select ... from (cities PIVOT (...))`. All 8
regressions are truncated concatenation fragments with an unclosed
parenthesis (`create materialized view test as (select ts,`, ...), which
the server rejects with "')' expected"; the old optional ')' accepted them.
Suggestions: after `CREATE TABLE y AS (`, `COPY (`,
`CREATE [MATERIALIZED] VIEW v AS [(]`, `ALTER VIEW v AS [(]` table names
and UNNEST are offered; after the table inside those bodies the usual
table-source follow-ups (alias, joins, WHERE, ...); PIVOT after
`SELECT * FROM (t `. The 41 default positions are unchanged.
The visitor unquoted single-quoted strings in statement positions with image.slice(1, -1) and never turned '' into ', while toSql re-escapes them with escapeString. Every doubled quote got doubled again: CREATE USER bob WITH PASSWORD 'it''s' -> PASSWORD 'it''''s' COPY t FROM 'f.csv' WITH DELIMITER '''' -> DELIMITER '''''' The server only strips the outer quotes in these positions (GenericLexer.unquote, std/GenericLexer.java:111: password EntSqlCompilerImpl:2202, DELIMITER SqlParser:3165). So the SQL text has to survive unchanged, and before this fix it did not. Change: a shared unquoteSingle() helper (strip quotes, '' -> ') is now used in every position whose value toSql prints back single-quoted: passwords (CREATE/ALTER USER, SERVICE ACCOUNT), EXTERNAL ALIAS (CREATE/ALTER GROUP), TTL, JWK public keys, DROP TOKEN, IN VOLUME, partition lists (DROP/DETACH/ ATTACH/FORCE DROP/CONVERT, REINDEX PARTITION), SUSPEND WAL code/message (table + live view), REBASE WAL INTO, COPY file/destination/options/CANCEL id, CANCEL QUERY, live view START FROM, ANCHOR DAILY time/zone, SAMPLE BY TIME ZONE/OFFSET, mat view REFRESH START/TIME ZONE/RANGE (extractMaybeString), and INSERT BATCH o3MaxLag. The places that already unescaped (expression literals, typed literals, identifier StringLiteral) now use the same helper. Deliberately unchanged: single-quoted NAMES that toSql prints as "..." identifiers (FROM 'x', stringOrQualifiedName, tableNameOrString, OWNED BY via stringOrIdentifier). The server keeps their inner text verbatim, so FROM 'it''s' is table it''s (SqlParser:3047/7305/8076 unquote; questdb-ent EntShowCreateSupportTest: CREATE USER 'o''brien' stores principal o''brien). The current output "it''s" names that same table. Unescaping would print "it's", which is a different table, so the report's "wrong table" claim for this case was incorrect. extractIdentifierName is left as it is. Tests: tests/single-quote-unescape.test.ts (30 round trips, AST values, pinned verbatim names, expression literals). 3 server queries with '' escapes (EscapeTest:52/62, LatestByTest:3797) were added to server-queries.json as guards. They already passed: no server test uses '' in a statement value position. Red -> green: 34 failing -> all pass; full suite 15884 passed. Corpus: fixed 0 / rtChanged 0 / regressed 0 / throws 0 / formatterChangedMeaning 0 (the corpus has no '' in these positions). Suggestions: unchanged (0/43 positions).
Server: the lexer reads a number glued to letters as one token, so `1hour`, `1days`, `1sec`, `111ml` are rejected (SampleByTest:250 "Invalid unit: 1hour", SampleByTest:262 "1min", AlterTableO3MaxLagTest:363/370 "interval qualifier", SqlParser.parseTtlHoursOrMonths "invalid time unit"). SqlUtil.expectMicros (o3MaxLag) accepts s, m, ms, h, d, us and ns (SqlUtilTest:829 "1000ns"). Before: DurationLiteral had no trailing boundary, so invalid units were silently split into a duration plus a second statement: SAMPLE BY 1hour -> `1h; our`, TTL 1days -> `TTL 1 DAY; ays`, o3MaxLag = 1sec -> `1s; ec`, and the valid o3MaxLag = 100ns -> `100n; s`. Now: - DurationLiteral gets the `ns` unit and a (?![\w\u0080-\uFFFF]) boundary. - New MalformedDurationLiteral token (after DurationLiteral, before NumberLiteral): digits + a duration-unit letter + more word characters (`1hour`, `1days`, `1.5hour`, `100nsx`). No grammar rule accepts it, so it is a parse error, like on the server. A boundary alone would have turned `TTL 1days` into a valid `TTL 1 DAYS` (NumberLiteral + DAYS) and `SELECT 1hour` into `1 AS hour`. A dedicated token was chosen over a NumberLiteral lookahead: NumberLiteral is untouched (GEOHASH(8c), MEMORY LIMIT 1G still split as before) and the formatter keeps a glued run like `10.5mFROM` as one word instead of backtracking into opaque fragments. Classified as a literal in the formatter and never suggested in autocomplete. - All valid units still lex as one token: 1h 30m 1M 1y 100T 1U 1n 1_000T 2W 12H 1.5d 1ms 1us 100ns 2Y 2D. Tests: 11 server queries (guards for n/U/T/m/ms/us units, TTL 1D, INSERT BATCH o3MaxLag, TOLERANCE w, GEOHASH(8c), MEMORY LIMIT 1G) + tests/duration-boundary.test.ts. Red before: 24 failing -> 548/548 pass; full suite 15975 passed. Corpus: fixed 0 / rtChanged 0 / regressed 3 / throws 0 / formatterChangedMeaning 0. All 3 regressions are server-rejected inputs that used to be silently split: `o3MaxLag = 111days`, `o3MaxLag = 111ml` (AlterTableO3MaxLagTest expects "interval qualifier") and `sample by 1hour` (SampleByTest expects "Invalid unit: 1hour").
Server: ExpressionNode.of types a literal token as BIND_VARIABLE when it is
`$` + digits or starts with `:` (FunctionParser.createBindVariable binds it).
`:name` is glued in ExpressionParser.processColon: a lone `:` that does not
follow a completed operand (literal, constant, `)`, `]`;
isCompletedOperand) and is not followed by whitespace takes the next token
as the name. So `arr[1:n]` / `arr[i:j]` stay slices, while `x = :sym`,
`IN (:a, :b)`, `LIMIT :lo, :hi`, `-:v` and even `arr[:1]` are bind
variables (ArrayTest:4428 "undefined bind variable: :1"). SAMPLE BY takes
an expression and, when it is not a full period, a one-letter unit
(SqlParser.expectSample): `SAMPLE BY $1 d`.
Before: `$` was a lexer error and `:name` "Unexpected token ':'" in every
position (WHERE, VALUES, LIMIT, SELECT items, function args, IN lists,
DECLARE values, UPDATE SET, SAMPLE BY FROM/TO, ...).
Now:
- New BindVariable token (custom lexer pattern, before Colon): `$\d+` with
a word boundary (`$1abc`, `$a`, `$` stay lexer errors), and
`:[A-Za-z0-9_\u0080-\uFFFF]+` only when the previous token does not end an
operand (Identifier, QuotedIdentifier, `)`, `]`, literals, bind/@ variables,
TRUE/FALSE/NULL/NaN). Keywords are operators/clause words (`LIMIT :n`,
`THEN :x`), except a non-reserved keyword glued to the colon, which is a
column name (`arr[offset:n]`). `::` and `:=` keep lexing first.
- primaryExpression accepts BindVariable -> new AST node
`{ type: "bindVariable", name: "$1" | ":sym" }` (name as written);
toSql prints it verbatim. Covers LIMIT, FROM/TO and every expression slot.
- sampleByClause: `BindVariable <unit letter>` (gated to the SAMPLE BY unit
letters); duration is "$1 d". `SAMPLE BY $1` without a unit is an error,
as on the server.
- Formatter: BindVariable is a literal; a slice Colon followed by a gap and
a word/`:` keeps the gap (`arr[: 5]`), since gluing would create `:5`.
Autocomplete never suggests the token; walkthrough skips BindVariable and
Colon.
- `arr[:5]` (no space) now reads as index by bind variable `:5` (server
semantics) instead of an open-start slice; `arr[: 5]` stays a slice.
parser.test.ts updated accordingly.
Tests: 38 server queries (OSS CoveringIndex*, PGJobContextTest, LimitTest,
QwpEgressBind*, MatViewExpireRowsHardeningTest, DeclareTest,
SqlCodeGeneratorTest, ArrayCreateFunctionFactoryTest, InLongFunction*,
MonotonicTimestampPruningTest, UpdateTest, SampleByNanoTimestampTest,
HashJoinGroupByRuntimeConstantTest, PercentileWindowFunctionTest,
SubsampleTest, CopyExportTest, CompiledFilterRegressionTest,
DeclareOperatorPrecedenceTest, GroupByTest, IntervalFilterTest,
OrderByEncodeSortTest, ParquetTest, RuntimeConstFunctionTest,
SubsampleWildcardProjectionTest, LatestByTest, ILikeSymbol*, ArrayTest slice
guards; Ent ViewAuditTest, TableColdPartitionsTest) +
tests/bind-variables.test.ts. Red before: 150 failing (110 fixture + 40
focused) -> 727/727 pass; full suite 16201 passed.
Corpus: fixed 528 (305 `$n`, 223 `:name`) / rtChanged 0 / regressed 0 /
throws 0 / formatterChangedMeaning 0.
Left out: ALIGN TO CALENDAR TIME ZONE $1 / WITH OFFSET $1 and FILL($1),
FILL(PREV($1)) (11 corpus strings) - those positions are not expressions in
sql-parser yet (AST stores strings) and belong to sample-by-fill-align.
TOLERANCE $1 and SWITCH ROLE ... TIMEOUT $1 stay errors (server rejects).
Server: GenericLexer treats '`' as a quote (Chars.isQuote: ', ", `). A
backtick token has no escapes - it ends at the next backtick, so `a``b` is
two tokens - and GenericLexer.unquote strips the quotes wherever an
identifier is expected (PivotTest `where \`BTC-USD_buy\` > 25780`,
CopyExportTest `copy \`test_table❤️\` to ...`, DeclareTest `\`@c\` + 1`).
SqlParser.validateIdentifier allows letters, digits, '_' and '$' after the
first character of an unquoted identifier (SqlParserTest
testGoodNonQuotedAlias: `select 'a' a1$_`, `select 'a' _a1$`); `$`+digits
at the start of a token stays a bind variable (ExpressionNode.of).
Before: '`' and '$' were lexer errors ("unexpected character"), so
`a$b` lexed as `a` + error + `b` and `select$x` as SELECT + ...
Now:
- QuotedIdentifier also matches `...` (no escapes, may be empty like "").
visitor unquoteIdentifier(): "..." still unescapes "", `...` content is
verbatim (`a""b` names a""b). toSql prints the canonical "..." form.
- Identifier allows '$' after the first character; the keyword negative
lookahead (tokens.ts createKeywordToken) includes '$', so `select$x`,
`from$` are identifiers. `$1` stays BindVariable, `$a` stays an error.
- Named bind variables allow '$' after the first name char (`:a$b`,
processColon glues the next token); `:$1` stays Colon + `$1` (formatter
lexeme oracle).
- toSql escapeIdentifier leaves names with a non-leading '$' unquoted
(`"price$1"` -> price$1, the same identifier on the server); a leading
'$' still needs quotes.
- Autocomplete: '`' is a word boundary like '"'. Backtick table names
already resolve columns (QuotedIdentifier images are sliced).
Tests: 7 server queries (PivotTest, CopyExportTest, DeclareTest x2,
SqlParserTest x2, RowExpiryWindowTest guard) + tests/lexer-quoting.test.ts.
Red before: 62 failing (17 fixture checks + 45 focused) -> all pass; full
suite 16282 passed.
Corpus: fixed 6 / rtChanged 1 / regressed 0 / throws 0 /
formatterChangedMeaning 0. Fixed: the 5 queries above plus
`select 'a' ``` which the server rejects (empty alias) - accepted and the
empty alias dropped by toSql, exactly as sql-parser already does for
`select 'a' ""` (pre-existing looseness, not changed here). rtChanged:
RowExpiryWindowTest `"price$1"` now prints unquoted price$1 (same name).
…ty statements Server: SqlUtil.fetchNext counts `/*` / `*/` (blockCount), so block comments nest (`SELECT 1 /* a /* nested */ still */ FROM t`, DeclareTest:5139-5168, SqlParserTest:6397-6427 hints with nested comments) and a comment that is never closed runs to the end of the input (SqlParserTest.testBlockCommentUnclosed). SqlCompilerImpl.compileBatch skips empty statements via getNextValidTokenPosition (PGMultiStatementMessageTest: `select 1;;`, `;;`, ` ; ;`, `; /* comment */`, `select 1;;/*c*/;--c\n;select 2;`). Before: BlockComment was a lazy regex, so the first `*/` closed a nested comment and the rest leaked out as `still * / ...` (error or silent extra statement); an unterminated `/*` lexed as Divide + Star; any `;` beyond one per statement gave "Redundant input". Changes: - lexer: BlockComment is a custom matcher (exported blockCommentEnd) that nests and runs to end of input when unclosed; line_breaks kept. - parser: statements = MANY(;) MANY2(statement MANY3(;)). Visitor/AST unchanged; empty statements produce no AST node. cst-types regenerated. - formatter lexer: its BlockComment uses the same nesting (closed comments only); an unterminated (nested) comment stays a tolerant token as before. Deliberately NOT replicated (server tokenizer artefacts): `--` inside a block comment starting a line comment, and quotes inside a block comment (GenericLexer still reads quoted spans there, so `/* '*/' */` stays open; the pairing quote is searched to end of input, so on the server `SELECT 1 /* don't */ FROM t WHERE s = 'x'` is just `SELECT 1`). Decision approved by the supervisor; DeclareTest:5142/5148/5157/5720 (`/* '*/' */`) stay unsupported. Here quotes and `--` inside /* */ are plain text. Tests: 15 server queries (DeclareTest, SqlParserTest, PGMultiStatementMessageTest) in server-queries.json + tests/comments-empty-statements.test.ts (41 tests: nesting, unterminated, quotes/-- inside comments, stray */, empty statements, formatter scan/format, autocomplete). recovery.test.ts: `;;;` and `SELECT 1;;;;;SELECT 2` no longer produce errors (server accepts both). Red before: 57 failing (26 focused + 31 fixture) -> all pass; full suite 16383 passed. SqlParserTest:6427 kept in the focused test, not the fixture: the formatter wraps that long hint comment differently on a second pass (pre-existing, also without nesting). Corpus: fixed 10 / rtChanged 0 / regressed 0 / throws 0 / formatterChangedMeaning 0. Fixed are the nested-comment DeclareTest / SqlParserTest queries plus three the server rejects for other reasons: MatViewExpireRowsHardeningTest:1221 (contextual "unterminated block comment in EXPIRE ROWS clause"), DeclareTest:5177 (nested lists, semantic), and DeclareTest:5168 whose `FROM @x` prints as "@x" (pre-existing toSql issue). Suggestions: statement starts after `;; ` / `; ` / `; /* c */ ; ` (were empty); normal follow-ups after a nested comment and inside an unterminated one (as for `--` comments). Default positions unchanged.
….5e3m
Server (FunctionParser.createConstant): a literal token that
Numbers.parseDouble accepts is a DOUBLE, and FastDouble skips an optional
d/D suffix, so `1d`, `0d`, `1.5D` are doubles. A token ending in m/M
(`(c | 32) == 'm'`) is a DECIMAL parsed by DecimalParser.parse, which also
accepts an exponent (`1.5e3m`, `2.5e-3M`; Decimal256Test:1839 `1.5e2M`).
Before: `99m` / `1d` lexed as DurationLiteral and the AST typed them as
literalType "duration" in expressions (`cast(99m as byte)`, `10d * 0d`,
`VALUES (..., 100m)`), and `1.5e3m` silently misparsed as `1.5e3 AS m`.
Changes:
- lexer: DecimalLiteral gains the exponent form
`\d[\d_]*(\.[\d_]*)?[eE][+-]?\d[\d_]*[mM]\b` (an exponent never forms a
duration, so either case of m is safe). `99m`, `99M`, `1.5M`, `1d`, `1.5d`
stay DurationLiteral tokens, so SAMPLE BY / TTL / REFRESH EVERY / window
frames / WINDOW JOIN / HORIZON LIST are unchanged.
- visitor literal(): a DurationLiteral ending in m/M/d/D is typed
`{ literalType: "number", value, raw }` (value parsed without the suffix
and underscores; raw kept, so toSql output is unchanged). Other units
(1h, 30s, 1ms, 100T, ...) stay durations: they are not numbers on the
server. DecimalLiteral value parsing strips M as well as m.
- Duration-valued expression positions keep a bare duration: table params
(`o3MaxLag = 10m` / `7d`, SqlUtil.expectMicros) and DECLARE values
(`DECLARE @Unit := 1m ... SAMPLE BY @unit` substitutes the token into
SAMPLE BY). A new helper bareDurationLiteral() detects an expression that
is a single DurationLiteral token; `DECLARE @x := 1m + 1` is typed as a
number expression.
- COPY options with an expression value keep a suffixed number verbatim
(`COMPRESSION_LEVEL 3d` still prints `3d`, it used to come from the
duration path; a decimal like `1.5m` there now also keeps its suffix).
AST: no new fields; numbers carry raw as before. No grammar change
(cst-types unchanged).
Tests: 19 server queries added to tests/fixtures/server-queries.json
(CastDecimalToByte/CastByteToDecimal, InsertTest, DecimalFullScaleTest,
LiveViewSmokeTest, Ent ViewAuditTest DECLARE, MulDouble, UnionAllCast,
GreatestNumeric, GroupByRewrite + guards TtlTest TTL 1M, AlterTableO3MaxLag,
HORIZON LIST, mat view SAMPLE BY 10m); no server SQL test uses the
exponent-m form (only Decimal256Test unit tests), so it is covered in the
focused tests/numeric-literal-typing.test.ts (51 tests: lexer, AST typing,
unchanged duration contexts, COPY, autocomplete, formatter).
Red -> green: 29 failing focused tests -> 51/51; the 76 fixture checks are
guards (toSql was already identical). Full suite 16510 passed.
Corpus: fixed 0 / rtChanged 0 / regressed 0 / throws 0 /
formatterChangedMeaning 0 (toSql keeps raw; the corpus has no exponent-m).
Suggestions: after `SELECT 1.5e3m ` and `WHERE price > 1.5e3m ` now the
normal expression follow-ups (m used to be read as an alias).
…RECISION
Server (ExpressionParser):
- `<expr> AT TIME ZONE '<tz>'`: AT after a completed operand, then TIME,
ZONE and a constant; rewritten to to_timezone(expr, tz). `at time 'x'`
and a dangling `at time` fail with "did you mean 'at time zone <tz>'?"
(TimestampAtTimeZoneTest). at/time/zone stay usable as aliases.
- `timestamp[_ns] with time zone` as a cast target: inside CAST(.. AS ..)
(Scope.CAST_AS) and after `::` (isColonColon on the op stack).
- `double precision`: PRECISION after `double` is ignored in expressions
(isPrecisionKeyword), SqlParser.toColumnType skips it for CREATE TABLE
columns and CTAS `cast(col as type)` (incl. `double precision[]`).
ALTER TABLE ADD COLUMN / ALTER COLUMN TYPE (addColumnWithType) do not.
Before: every form failed, and `x::timestamp at time zone 'UTC'` was
silently split into `x::TIMESTAMP AS at; time AS zone; UTC`.
Changes:
- keywords: `at` and `precision`, both non-reserved (server reserves
neither).
- parser: typeCastExpression takes an optional
`AT TIME ZONE (StringLiteral | EscapeStringLiteral)` after the `::` casts
(gated on LA(2) = TIME so `SELECT ts at FROM t` keeps the alias; not
repeated, the server's to_timezone rewrite does not chain). It binds
tighter than arithmetic and inside unary minus, like the server.
New castDataType rule (CAST and `::`): timestampWithTimeZoneType |
doublePrecisionType | dataType, as gated alternatives so autocomplete
offers WITH only after TIMESTAMP[_NS] and PRECISION only after DOUBLE.
columnDefinition gets an argument: DOUBLE PRECISION only when called from
CREATE TABLE (ALTER ADD COLUMN shares the rule); CTAS castDefinition also
takes it. dataType itself is unchanged.
- AST: new expression node `atTimeZone { expression, timeZone }`. Cast /
column dataType strings are "TIMESTAMP WITH TIME ZONE",
"TIMESTAMP_NS WITH TIME ZONE", "DOUBLE PRECISION[]..." (spelling kept,
same as the existing typed literal's typeName).
- toSql prints `expr AT TIME ZONE 'tz'` and the dataType strings.
- autocomplete: PRECISION is dropped under alterTableAction (rule args are
invisible to content assist).
- formatter: WITH after TIMESTAMP/TIMESTAMP_NS followed by TIME ZONE is not
a WITH clause (it used to break the line, also for the existing
`timestamp with time zone '...'` literal).
- cst-types regenerated.
Tests: 19 server queries in tests/fixtures/server-queries.json
(TimestampAtTimeZoneTest x8 incl. alias guards, InsertTest x2,
SimpleTableTest, SqlParserTest x2, TimestampQueryTest x4, PGArraysTest x2);
no server test parses DOUBLE PRECISION in CREATE TABLE (only expected
model strings), covered in tests/type-syntax.test.ts (51 tests: AST,
precedence, aliases, rejected forms, CREATE/CTAS/ALTER, formatter,
autocomplete). Red -> green: 78 failing (52 fixture checks + 26 focused)
-> all pass; full suite 16637 passed.
Corpus (53,975): fixed 21 / rtChanged 2 (the silent 3-statement splits of
`::timestamp[_ns] at time zone 'UTC'`) / regressed 3 / throws 0 /
formatterChangedMeaning 0. All 3 regressions are server-rejected
TimestampAtTimeZoneTest negatives (testFail2 `at time 'UTC'`,
testFailDangling2/3 `at time`) that used to split silently into
`AS at; time ...`.
Suggestions: AT after a completed operand (SELECT/WHERE/ORDER BY/GROUP BY/
OVER/DECLARE values), TIME after `AT`, WITH after `::timestamp` and
`CAST(x AS TIMESTAMP`, PRECISION after DOUBLE in casts, CREATE TABLE
columns and CTAS casts (not in ALTER ADD COLUMN). The `[` / `)` fallback
punctuation after `CAST(x AS TIMESTAMP|DOUBLE ` is no longer shown because
a keyword is now suggested.
The server accepts three predicate forms that sql-parser rejected:
- `a IS [NOT] TRUE|FALSE`: ExpressionParser.validateIsKeyword accepts
IS [NOT] NULL|TRUE|FALSE and rewrites it to `=` / `!=`
(ExpressionParserTest.testIsTrue/testIsNotTrue/testIsNotFalse).
- `a <> ALL (...)` / `a != ALL (...)`: ExpressionParser isAllKeyword turns the
preceding `<>`/`!=` into the `<>all` function (allFunctions); any other
operator is "unexpected operator", no operator is "missing operator"
(ExpressionParserTest.testAllNotEqual/testAllInvalidOperator/
testAllNoOperator, AllNotEqStrFunctionFactoryTest).
- `g NOT WITHIN (...)`: NOT before a SET operator becomes
OperatorExpression.UnarySetNegation (in / between / within);
ExpressionParserFuzzTest generates `x not within y`.
Before: all three were parse errors ("Unexpected token").
Changes:
- relationalExpression: `Is Not? (Null | True | False)`. New AST node
`isBoolean { expression, value, not }` (isNull is unchanged); toSql
`a IS [NOT] TRUE|FALSE`.
- equalityExpression: optional `All` after NotEquals. AST stays a binary node
with operator "<> ALL" / "!= ALL" (as typed; plain <>/!= still print `!=`);
the right operand is the usual relationalExpression, so `all(x)` is a paren
node and toSql gives `a <> ALL (x)`. Parens are not required (the server's
ALL branch does not require them either). toSql precedence table knows the
two operators (13, same as <>).
- setExpression: `Not?` before Within. WithinExpression gets an optional
`not` (set only when true, so plain WITHIN keeps its AST shape); toSql
`g NOT WITHIN (...)`.
- cst-types regenerated.
Tests: 6 server queries (AllNotEqStrFunctionFactoryTest) in
server-queries.json; tests/predicates.test.ts covers the expression-only
server tests wrapped in WHERE, AST shapes, negatives (IS 3, IS NOT $1, `||
all`, `= all`, `> all`, bare `all`), formatter and suggestions.
Red before: 45 failing (18 fixture + 27 focused) -> all pass; full suite
16708 passed.
Corpus (54k): fixed 6 (the AllNotEqStrFunctionFactoryTest queries) /
rtChanged 0 / regressed 0 / throws 0 / formatterChangedMeaning 0.
Suggestions: +TRUE/FALSE after `IS` and `IS NOT`, +ALL after `<>`/`!=`,
+WITHIN after an operand's `NOT`; nothing else changed.
The server accepts ROWS | RANGE | GROUPS frame modes and EXCLUDE CURRENT ROW | GROUP | TIES | NO OTHERS (ExpressionParser.parseWindowSpec / parseWindowFrameClause). sql-parser rejected GROUPS, EXCLUDE GROUP and EXCLUDE TIES. Adds the groups / ties keywords (non-reserved) and the grammar options, with server queries in the parity fixture and tests/window-frames.test.ts.
…omparisons
The server's ExpressionParser accepts these; sql-parser rejected them:
- member access after `)`: `(x).a`, `(pg_catalog.pg_class()).relnamespace`,
`pg_catalog.pg_class().y` (DOT dereference; DeclareTest:4542/4545,
ExpressionParserTest:672). typeCastExpression: MANY(Dot identifier) gated on
the previous token being `)`. New AST node memberAccess { expression, member }.
- stacked prefix operators `- -a`, `~-a`, `-~-~1` (ExpressionParserTest:1863):
unaryExpression OPTION -> MANY; one unary node per operator. toSql keeps a
space in `- -a` so it does not become a `--` comment.
- `NOT NOT x` (BooleanSubQueryPredicateTest:252) and NOT as a comparison
operand `x = NOT y` (CompiledFilterIRSerializerTest:301): notExpression
MANY(NOT); equality right operand may be `NOT <notExpression>`.
- chained comparisons `a = 1 = true`, `a < b < c`, `a > 1 IS NULL`:
equality/relational OPTION -> MANY, left-associative binary nodes
(visitor orders operators/operands by offset).
Tests: 3 server queries (fixture) + tests/expression-gaps.test.ts;
red 13 failing -> all green, full suite 16784 passed.
Corpus: fixed 13 (pg_class member access, NOT NOT) / rtChanged 0 /
regressed 0 / throws 0 / formatterChangedMeaning 0.
Suggestions: +IS after `price > 1 `, +NOT after `a = `, +LIKE/ILIKE after
`a = 1 ` (all valid chain continuations).
…select after DECLARE/WITH, WITH/DECLARE set-op branches Server (questdb#7602; SqlParser.parseDeclare, isMarkedDeclarationAhead, parseQuery) accepts, sql-parser rejected: - `DECLARE [OVERRIDABLE] [AUDITED] @x := ...` markers in any order per variable (ViewAuditTest, DeclareTest). New non-reserved keyword `audited`; declareAssignment MANY(OVERRIDABLE|AUDITED). AST: DeclareAssignment.audited. toSql prints OVERRIDABLE before AUDITED. Duplicate markers are accepted (server rejects; looser is fine). - trailing comma in a value list `('a',)` (any parenthesised list) and after the last declaration (`DECLARE @lim := 1, audited WHERE ...`), gated on a `[markers] @var` lookahead so audited/overridable stay table names. - implicit select after a DECLARE / WITH prefix: `DECLARE @x := 1 (SELECT @x)`, `DECLARE @lim := 1 t WHERE x > @lim`, `(WITH c AS (...) c)`, top-level `WITH c AS (...) c`. selectStatement = DECLARE [WITH] body | WITH body | selectBody; new rule prefixedQueryBody (selectBody | implicitSelectStatement). - set-operation branch starting with WITH or DECLARE (WithClauseTest:649/654, DeclareTest:5616): setOperation alt -> selectStatement (later set ops nest into that branch; toSql prints them flat, which re-parses the same). Tests: 7 server queries (fixture) + tests/declare.test.ts; red 27 -> green, full suite 16818 passed. Corpus: fixed 121 / rtChanged 0 / regressed 0 / throws 0 / formatterChangedMeaning 0. Suggestions: +AUDITED after `DECLARE ` (OVERRIDABLE already there); table names after `DECLARE @x := 1 ` and `WITH .. AS (..) `; markers/WITH/SELECT/ tables after `DECLARE @x := 1, `; +DECLARE,WITH after `UNION ALL `.
…ONE / WITH OFFSET <expr> The server parses all three with expr() (SqlParser.expectSample, FILL and ALIGN TO handling in parseSelectClauses): - SAMPLE BY <expr> [one-letter unit]: `@q h` (was a silent 2-statement split), `2 h`, `(1m)`, `(10+20)m`, `(1+2)*10L m`, `$1 d`. sampleByClause = bare unit letter | expression + optional unit. The unit is optional here (server wants it unless the expression is a full period) - looser, so `SAMPLE BY $1` / `SAMPLE BY foo` now parse (tests updated). AST unchanged: duration = expression SQL [+ " " + unit]. - FILL values are expressions: -1, @f, true, PREV(abs(a)), 1+2, 100L, NaN, '5'. AST fill stays string[]: plain names upper-cased (PREV/LINEAR/NONE), PREV(...) upper-cased, NULL; others printed as SQL - string values now keep their quotes (FILL('5') printed FILL(5) before). - ALIGN TO CALENDAR TIME ZONE <expr> [WITH OFFSET <expr>]: @tz, $1, rnd_str('foo','bar') (was a silent split). AST: timeZone/offset strings unchanged for string literals / names; new optional timeZoneExpression / offsetExpression for other values (toSql prints them). - Autocomplete: content assist filters the top level of those expressions (keeps literal / variable tokens) so SAMPLE BY / FILL( / TIME ZONE do not offer every column, function and operator; FILL( now offers PREV, LINEAR, NONE next to NULL. Tests: 8 server queries (fixture) + tests/sample-by-fill-align.test.ts; red 37 -> green; full suite 16872 passed. Corpus: fixed 61 / rtChanged 8 (2 former silent splits, 6 FILL('str') now keeping quotes) / regressed 0 / throws 0 / formatterChangedMeaning 0. Suggestions: +PREV,LINEAR,NONE after `FILL(` / `FILL(PREV, `; `(` fallback after `SAMPLE BY ` and `TIME ZONE `; clause keywords after `SAMPLE BY 2 ` (unit optional); `(` no longer offered after `FILL(PREV `.
…OT before SAMPLE/GROUP BY), LATEST ON @var, open-ended LIMIT Server (SqlParser.parseFromClause): deprecated LATEST BY right after the table, then joins -> WHERE -> LATEST ON -> PIVOT -> SAMPLE BY -> GROUP BY -> WINDOW -> SUBSAMPLE -> ORDER BY -> LIMIT. LATEST ON/BY operands go through expectLiteral with decls, so @variables are allowed. LIMIT accepts `5,` and `,5` (missing lower bound defaults to 0, SqlCodeGenerator.getLoFunction). Before: `... LATEST ON ts PARTITION BY s SAMPLE BY 1d`, `... PIVOT (...) GROUP BY|SAMPLE BY ...`, `FROM t LATEST BY s WHERE ...`, `PARTITION BY @c`, `LIMIT 5,` and `LIMIT ,5` were errors. Now: simpleSelect and implicitSelectBody share selectClauseChain(), a set of alternatives covering the server order AND every order accepted before (SAMPLE BY -> LATEST, GROUP BY -> PIVOT, WHERE -> LATEST BY); written as alternatives so autocomplete does not offer a clause twice. New rule latestOnColumn (columnRef | @var; AST keeps a one-part name "@var", printed raw). limitClause: `LIMIT ,5` -> AST lowerBound literal 0 (toSql `LIMIT 0, 5`), `LIMIT 5,` -> `LIMIT 5`. AST: optional SelectStatement.pivotBeforeGroupBy so toSql keeps PIVOT before SAMPLE/GROUP BY. toSql now prints LATEST BY before WHERE and LATEST ON before SAMPLE BY (server order; same AST meaning). Tests: 8 server queries (LatestByHoistEquivalenceTest:139, LatestByTest:1472, QueryExecutionTimeoutTest:331, SqlParserTest:8285, DeclareTest:2005/875, LimitTest:176/192) + tests/clause-order.test.ts. Red 26 -> green; full suite 16910 passed. Corpus: fixed 18 / rtChanged 1 (`(tab where x = 100 latest by x)` now prints LATEST BY before WHERE) / regressed 0 / throws 0 / fmtChangedMeaning 0. Suggestions: +WHERE,SAMPLE after `LATEST BY|ON ... ` (WHERE is only valid after LATEST BY; ruleStack can't tell them apart), +SAMPLE,GROUP after `PIVOT (...) `, +UNION/EXCEPT/INTERSECT after `LIMIT 5, ` (open upper bound).
Server (SqlParser.parsePivot): after the closing `)` an optional table alias (setModelAliasAndGetOptTok, `[AS] alias` unless a tableAliasStop word); the FOR expression stops at its top-level IN (setStopOnTopINOperator), so `FOR year(ts) IN`, `FOR year - 2000 IN`, `FOR (SELECT 'NL')::VARCHAR IN`; IN-list values take an alias with or without AS (`2010 D2`). Before: `PIVOT (...) p` silently split into a second statement `p`, `PIVOT (...) AS p` errored in SELECT form; FOR accepted only a column reference; `IN (2010 D2)` errored. Now: PIVOT clause (simpleSelect / implicitSelectBody) and pivotStatement take `[AS] alias`; implicit alias gated like table aliases (new implicitTableAliasAllowed(), the former tableRef gate) and additionally not when the next token is PIVOT/WHERE, so consecutive statements without `;` (`t PIVOT (...)\nu PIVOT (...)`) still split. AST: optional PivotClause.alias (toSql ` AS p`). FOR operand = bitOrExpression (the level right below IN/BETWEEN/comparisons; no GATE so autocomplete is not misled). pivotInValue: optional AS. PIVOT inside sub-query/CTE bodies already worked (implicit-select-bodies). Tests: 3 server queries (PivotTest:2914, 2934, 415) + tests/pivot.test.ts (alias forms, clause words after `)`, FOR expressions, IN aliases; no one-line server test uses a PIVOT alias). Red 13 -> green; full suite 16927 passed. Corpus: fixed 1 / rtChanged 0 / regressed 0 / throws 0 / fmtChangedMeaning 0 (most PivotTest queries are text blocks, not in the corpus). Suggestions: +AS after `PIVOT (...) `; clause keywords after `PIVOT (...) p `; expression starts after `FOR `.
… alias / sub-query / + offsets, UNNEST (cols) without alias Server (SqlParser.parseJoin, setModelAliasAndGetOptTok, tableAliasStop, expectIntervalLiteral, parseUnnest0): - `tolerance` and `range` stop an implicit table alias, so `ASOF|LT JOIN t2 TOLERANCE 2s` and `WINDOW JOIN p RANGE BETWEEN ...` need no alias/ON; EXCLUDE/INCLUDE PREVAILING likewise. - HORIZON JOIN takes a table or parenthesised sub-query (expectTableNameOrSubQuery) with an optional alias; offsets may be signed with `-` or `+`. - UNNEST(...) (v): column aliases without a table alias. Before: TOLERANCE/RANGE were taken as the table alias (error at `2s` / BETWEEN); HORIZON JOIN required `tableName alias`; `+1m` errored; `UNNEST(x)(v)` silently split into a 2nd statement `(v)`. Now: the tableRef alias gate is a method (implicitTableAliasAllowed, shared with the PIVOT alias) that also refuses TOLERANCE, RANGE and EXCLUDE/INCLUDE before PREVAILING (still fine as explicit `AS` aliases and identifiers). horizonJoin uses tableRef (RANGE refused by the gate, `LIST (` by the LParen check); AST unchanged (JoinClause.table is the tableRef). horizonOffset accepts `+` (AST string "+1m"). unnestClause: column list independent of the alias (alias labelled unnestAlias in the CST). Tests: 7 server queries (AsOfJoinTest:1987/5251, SqlParserTest:14994/14224, DeclareTest:3753, BooleanSubQueryRuntimeGateTest:321, HorizonJoinProjectionTest:1615) + tests/join-aliases.test.ts (no server test uses `+` offsets). Red 20 -> green; full suite 16958 passed. Corpus: fixed 10 / rtChanged 2 (former silent UNNEST(..)(val) splits) / regressed 0 / throws 0 / fmtChangedMeaning 0. Suggestions: after `HORIZON JOIN q ` ON/RANGE/LIST/joins/TIMESTAMP (was an error state: mandatory alias); after `HORIZON JOIN q ON (sym) ` operators + RANGE/LIST.
Server: FROM/JOIN sources may be schema-qualified function calls
(`pg_catalog.pg_class()`, `information_schema.tables()`, used throughout the
catalogue tests); parseDml accepts `UNION DISTINCT` as plain UNION.
Before: `FROM information_schema.tables()` errored at `(`;
`UNION DISTINCT` errored.
Now: tableFunctionName = identifier (Dot identifier)*; the AST name is the
dotted string ("pg_catalog.pg_class"). In tableRef the tableName alternative
now comes before tableFunctionCall, gated off only when `name[.name]* (`
follows AND the function call backtrack succeeds (so inputs that parsed
before as a table keep doing so). Order matters for content assist: it
ignores GATEs and prunes later alternatives once a path consumes the input,
so with the function alternative first `FROM sys.t ` only offered `(`
(docs autocomplete walkthrough). setOperation: UNION [ALL | DISTINCT],
EXCEPT/INTERSECT [ALL] (no DISTINCT suggestion after EXCEPT/INTERSECT); AST
unchanged (DISTINCT == UNION; toSql prints UNION).
Tests: 4 server queries (InformationSchemaTablesFunctionFactoryTest:42,
LiveViewTest:1193, InformationSchemaColumnsFunctionFactoryTest:78,
SqlParserTest:13928) + tests/from-sources.test.ts (pg_class/pg_namespace
JOIN, _pg_expandarray). Red 13 -> green; full suite 16976 passed.
Corpus: fixed 13 / rtChanged 0 / regressed 0 / throws 0 /
fmtChangedMeaning 0. Suggestions: +DISTINCT after `UNION `; alias/join
keywords after `FROM pg_catalog.pg_class() ` (was an error state).
…l options, CAST/INDEX in any order Server: parseCreateTableColumns skips `[PRECISION] [NOT] NULL` after a column type / PARQUET(...); SqlCompilerImpl.addColumnWithType skips `[NOT] NULL` (ALTER TABLE ADD COLUMN, ALTER COLUMN TYPE); parseCreateTableCastDef reads `SYMBOL [CAPACITY n] [CACHE|NOCACHE]`; parseCreateTable loops over `, CAST(..)` and `, INDEX(..)` in any order. Before: all of these errored. Now: new rule columnNullModifier (`[PRECISION] [NOT] NULL`, may match nothing; PRECISION gated to CREATE TABLE and filtered from autocomplete) used by columnDefinition and ALTER COLUMN TYPE; visitor ignores it (toSql drops the modifiers, like the server). castDefinition takes CAPACITY/CACHE/NOCACHE (AST: optional CastDefinition.symbolCapacity / cache, printed by toSql). CTAS `, CAST | , INDEX` in one loop (toSql still prints casts before indexes). Tests: 6 server queries (HashJoinTest:136, SqlParserTest:3045/2889/2839, AlterTableAddColumnTest:295, CreateTableTest:468) + tests/create-table-options.test.ts. Red 21 -> green; full suite 17003 passed. Corpus: fixed 18 / rtChanged 0 / regressed 0 / throws 0 / fmtChangedMeaning 0. Suggestions: +NOT, NULL after a column type (CREATE TABLE, ADD COLUMN, ALTER COLUMN TYPE); +CAPACITY/CACHE/NOCACHE inside CTAS CAST(.. AS type.
…D/INCREMENTAL REFRESH, one-letter TTL units Server (SqlParser.parseCreateMatView): REFRESH [IMMEDIATE | MANUAL | EVERY .. | INCREMENTAL] [DEFERRED] [PERIOD(..)] AS — the mode is optional and INCREMENTAL is the obsolete name of IMMEDIATE; TTL is parsed by SqlParserCallback.parseTtlSettings independently of PARTITION BY; parseTtlHoursOrMonths accepts `<n> <unit>` where unit goes through PartitionBy.ttlUnitFromString (HOUR(S)..YEAR(S) and h/d/w/m/y). Before: `.. SAMPLE BY 30s) TTL 12 HOURS`, `REFRESH AS`, `REFRESH DEFERRED`, `REFRESH INCREMENTAL`, `TTL 2 h` all errored. Now: new rule ttlClause (DurationLiteral | number + unit word | number + one-letter identifier, gated) shared by CREATE TABLE and CREATE MATERIALIZED VIEW; the mat-view TTL moved out of materializedViewPartition into its own optional clause after PARTITION BY (AST unchanged: stmt.ttl). REFRESH takes an optional materializedViewRefresh (PERIOD handled by the existing following option); materializedViewRefresh accepts INCREMENTAL (AST mode "immediate", not suggested) and bare DEFERRED (also reachable from ALTER MATERIALIZED VIEW SET REFRESH — looser than the server there). Tests: 3 server queries (StoragePolicyWriterTest:2510/2637, MatViewReloadOnRestartTest:970) + tests/mat-view-create-options.test.ts. Red 12 -> green; full suite 17018 passed. Corpus: fixed 4 / rtChanged 0 / regressed 0 / throws 0 / fmtChangedMeaning 0. Suggestions: +TTL after the mat-view body; +DEFERRED, AS after `REFRESH `; +DEFERRED after ALTER .. SET REFRESH; unit words after `TTL 2 ` in mat views.
…E COLUMN lists, TYPE SYMBOL .. INDEX
Server (SqlCompilerImpl): alterTableAddColumn ignores an optional COLUMN and
reads `[IF NOT EXISTS] name type` per comma-separated column;
alterTableRenameColumn loops over `old TO new` pairs; addColumnWithType (also
used by ALTER COLUMN .. TYPE) reads SYMBOL [CAPACITY n] [CACHE|NOCACHE]
[INDEX [TYPE ..] [CAPACITY n]].
Before: `ADD x INT` (~40 server tests) and rename lists errored; `TYPE
SYMBOL INDEX` silently split into a second statement `INDEX`.
Now: ADD -> new rule alterTableAddColumns ([COLUMN] addColumnItem {,
addColumnItem}); addColumnItem = [IF NOT EXISTS] columnDefinition. AST:
first-column IF NOT EXISTS stays AddColumnAction.ifNotExists; later columns
get ColumnDefinition.ifNotExists (toSql prints it before the column). RENAME
COLUMN takes `, old TO new` pairs (AST RenameColumnAction.additionalRenames).
ALTER COLUMN TYPE gets alterColumnTypeIndex (optional INDEX +
indexTypeOptions; AST AlterColumnAction.indexed/indexType/indexInclude/
indexCapacity) before the [NOT] NULL modifier.
Tests: 7 server queries (CompositeColumnMappingTest:87,
AlterTableChangeColumnTypeTest:933, AggregateTest:2198,
AlterTableAddColumnTest:82/324, TableWriterAsyncCmdTest:406, ent
HttpCallbackTest:33) + tests/alter-table-columns.test.ts. Red 22 -> green;
full suite 17050 passed.
Corpus: fixed 34 / rtChanged 3 (TYPE SYMBOL INDEX splits fixed) / regressed 0
/ throws 0 / fmtChangedMeaning 0. Suggestions: +IF after `ADD ` and
`ADD x INT, `; column names after `RENAME COLUMN a TO b, `; +INDEX after
`TYPE SYMBOL `.
Server (SqlCompilerImpl CONVERT branch + alterTableDropConvertDetachOrAttachPartition[ByList]): CONVERT PARTITION TO PARQUET|NATIVE then LIST '<p>' [, ...] | WHERE <expr>, then for PARQUET an optional WITH (bloom_filter_columns = .., fpp = ..). Before: the server form (~500 server tests) silently split into `CONVERT PARTITION TO PARQUET; LIST AS "2024-01-01"` (LIST was only accepted before TO); with WITH it errored. Now: convertPartitionTarget accepts LIST after the target (the old LIST-before-TO form still parses). AST: ConvertPartitionAction.listAfterTarget so toSql keeps the order. Unquoted partition names are not added (they lex as arithmetic; not trivial). Tests: 5 server queries (AdaptiveParquetColumnConversionRestartTest:75, ArrayTest:2399, ParquetWriteTest:2774, DataSyncFailureCleanupTest:62, CoveringIndexParquetNativeRoundTripTest:273) + tests/convert-partition.test.ts. Red 17 -> green; full suite 17073 passed. Corpus: fixed 11 / rtChanged 195 (all `TO .. LIST '..'` splits fixed) / regressed 2 (`.. TO PARQUET LIST` / `.. TO NATIVE LIST` with no partition name: the server rejects them, maybeExpectToken "partition name"; they only "parsed" before as a silent split) / throws 0 / fmtChangedMeaning 0. Suggestions: +LIST after `CONVERT PARTITION TO PARQUET `; +WHERE, WITH after the LIST.
Server: compileAlterMatView's SUSPEND branch calls parseSuspendWal, the same parser tables and live views use (optional `WITH <code|'tag'>, '<message>'`). Before: `ALTER MATERIALIZED VIEW mv SUSPEND WAL WITH 24, 'msg'` errored (the table and live view rules already accepted the WITH form). Now: new rule suspendWalWith used by the mat-view SUSPEND WAL alternative; AST AlterMaterializedViewSuspendWal gets optional code / message (printed by toSql). Tests: no server test uses the WITH form for mat views (only plain SUSPEND WAL, which already parsed); tests/alter-mat-view-wal.test.ts. Red 2 -> green; full suite 17075 passed (one run of the RSI per-position <100ms autocomplete timing test flaked under parallel load; a direct bench shows no slowdown vs HEAD and it passes on rerun). Corpus: fixed 0 / rtChanged 0 / regressed 0 / throws 0 / fmtChangedMeaning 0. Suggestions: +WITH after `ALTER MATERIALIZED VIEW mv SUSPEND WAL `.
…ISCARD/CLOSE/UNLISTEN/RESET
The server dispatches these statement-start keywords to PG-compatibility
executors (SqlCompilerImpl.registerKeywordBasedExecutors): compileBegin /
compileCommit / compileRollback (optional TRANSACTION), compileDeallocate
(statement name), compileSet (SET [SESSION|LOCAL] name {=|TO} v[, v]* or any
tokens up to `;`, e.g. SET TIME ZONE 'UTC'), compileNoOp for
DISCARD/CLOSE/UNLISTEN/RESET (one argument token).
Before: BEGIN/COMMIT/ROLLBACK [TRANSACTION] were silently parsed as implicit
selects (`COMMIT AS TRANSACTION`); SET <name> ..., DEALLOCATE, DISCARD, CLOSE,
UNLISTEN, RESET were parse errors.
Now: new rules transactionStatement, deallocateStatement, pgNoOpStatement and
setStatement (SET followed by a loose run of identifiers/literals/=/TO/,/...;
SET TYPE [BYPASS] WAL keeps its own rule, gated on TYPE BYPASS|WAL). AST
nodes transaction / deallocate / pgNoOp / set (raw args). New keywords are
non-reserved (SELECT begin FROM t still works) and highlight as keywords.
Autocomplete: inside SET only TO is offered after the name.
Tests: 9 server fixture entries + tests/pg-session-statements.test.ts.
Red 21 -> green; full suite 17116 passed.
Corpus: fixed 47 / rtChanged 26 (implicit-select misparses of BEGIN/COMMIT/
ROLLBACK now transaction statements) / regressed 1 (prose
"commit() must build ..." - server runs compileCommit and rejects `(`) /
throws 0 / formatterChangedMeaning 0.
…c/partition_by, VACUUM PARTITIONS
Server:
- DROP ALL[;] or legacy DROP ALL TABLES (SqlCompilerImpl.compileDrop).
- UPDATE tbl [AS] alias SET ... (SqlParser.parseUpdate; SqlParserUpdateTest).
- COPY .. TO .. WITH compression_codec <literal> / partition_by <literal>
use expectLiteral, so quoted values ('zstd', 'DAY') work (SqlParser.parseCopy).
- VACUUM PARTITIONS t is the legacy spelling of VACUUM TABLE t
(SqlCompilerImpl.compileVacuum).
Before: all four were parse errors. Now: TABLES optional after DROP ALL;
optional AS before the UPDATE alias; StringLiteral accepted for
compression_codec and partition_by in copyOption; VACUUM TABLE|PARTITIONS.
AST unchanged (DROP ALL prints as DROP ALL TABLES, VACUUM PARTITIONS as
VACUUM TABLE). Formatter grammar-drift: UPDATE alias AS is inline by design.
Tests: 6 server fixture entries + tests/misc-statements.test.ts.
Red 18 -> green; full suite 17144 passed (RSI autocomplete timing test
flaked under parallel load once; passes alone).
Corpus: fixed 16 / rtChanged 0 / regressed 0 / throws 0 /
formatterChangedMeaning 0. Suggestions: +PARTITIONS after VACUUM, +AS after
UPDATE t.
…ec, PG type aliases as data types Server: - Index type names go through IndexType.valueOf, which also takes the single words POSTING_DELTA / POSTING_EF (besides POSTING [DELTA|EF]). - ParquetEncoding names include byte_stream_split; ParquetCompression names include default (PARQUET(enc, default), COPY compression_codec default). - ColumnType.nameTypeMap aliases: text smallint bigint real bytea regclass regprocedure array. Before: TYPE POSTING_DELTA, PARQUET(BYTE_STREAM_SPLIT), PARQUET(plain, default) and compression_codec default were parse errors; the type aliases were plain identifiers (not highlighted / suggested as types). Now: new non-reserved tokens PostingDelta, PostingEf, ByteStreamSplit (constants) and Text, Smallint, Bigint, Real, Bytea, Regclass, Regprocedure, Array (dataTypes, exported as ArrayType to avoid shadowing the global). All are IdentifierKeyword, so they stay usable as names. dataType accepts the aliases explicitly; typed literals (text 'x') use the new tokens instead of an image check; ARRAY[...] literals consume the Array token; EXPLAIN (FORMAT TEXT) accepts the Text token. Autocomplete: ARRAY is not offered at every expression start (only as a type name). Tests: 4 server fixture entries (2 regression guards for ::real / ::text[] with a `text` column) + tests/option-names.test.ts. Red 5 -> green; full suite 17163 passed. Corpus: fixed 2 / rtChanged 0 / regressed 0 / throws 0 / formatterChangedMeaning 0. Suggestions: + alias types after :: / CAST AS / column types; +POSTING_DELTA,POSTING_EF after INDEX TYPE; +BYTE_STREAM_SPLIT, DEFAULT in PARQUET(...); +TEXT after EXPLAIN (FORMAT.
- REGCLASS, REGPROCEDURE and bare ARRAY were offered as column types in CREATE TABLE / ADD COLUMN / ALTER COLUMN TYPE. The server only accepts them as cast targets (SqlUtil.toPersistedType), so they are now offered after `::` / CAST only. - UNLIMITED was the only suggestion after cpu_weight / max_active_queries / max_queued_queries / queue_timeout =. The server accepts it for memory_limit only (EntSqlCompilerImpl resource group options). Parsing is unchanged. Found in the PR #42 audit.
…TITION BY These slots consumed a bare identifier, which autocomplete classifies as a new name, so nothing was offered after `DEDUP UPSERT|SKIP KEYS(`, `KEEP HIGHEST|LOWEST ON` or `KEEP LATEST [ON] / PARTITION BY`. They now go through a keyColumn rule (identifier wrapper, same AST) that autocomplete treats like columnRef: the columns of the table in scope are offered, with no tables or functions. Found in the PR #42 audit.
- Remove `SELECT a.from FROM t a`: the server rejects it too (KeywordAsTableNameTest), so it is not a gap. - Formatter: one unparsable statement disables capitalization for the whole input, not just that statement. - Update bind-variable and password notes for the fixes in this PR; NULL is no longer suggested as a password. - Add accepted gaps found by the audit: DOUBLE PRECISION typed literals, $ / leading digit in DECLARE names, grantees named parquet / native, GRANT phrase suggestion noise, and the unclosed backtick after accepting a completion in the web console.
…as slice bounds f367be1 treated a non-reserved keyword before `:` as an operand whenever any `[` was open, so `arr[row_number() OVER (PARTITION BY :k)]` (parsed before, valid on the server) lexed `BY:` as a slice and failed. The lexer now looks at the innermost open delimiter: inside `(...)` the ordinary rule applies. WHEN / THEN / ELSE only keep a following bind when a CASE is open in the current bracket, so `arr[when:n]` / `arr[else :n]` are slices and `arr[CASE WHEN x THEN :lo ELSE :hi END:n]` keeps its binds. Found in the PR #42 follow-up review.
…word eb8b2e5 glued adjacent pieces of an unquoted password, but also crossed quoted tokens: `CREATE SERVICE ACCOUNT sa WITH PASSWORD 'pw'OWNED BY g` became the password "'pw'OWNED" plus a statement `BY g`. A quoted value now ends on its own (string, double-quoted and E'' tokens are never glued). The server reads any token as the password (EntSqlCompilerImpl.parsePassword), so passwordValue also accepts every keyword, reserved or not, through a new AnyKeyword token category, and malformed durations: `PASSWORD select`, `1hour` and `12true` now parse. AnyKeyword is only consumed in this slot and is never suggested. Parser start-up time is unchanged. Found in the PR #42 follow-up review.
…being defined b104836 offered columns in these slots, but from the wrong scope: - CREATE MATERIALIZED VIEW ... EXPIRE ROWS KEEP ... ON / PARTITION BY offered the base table's columns, while the server resolves the names against the view query's output (SqlCompilerImpl: 'invalid EXPIRE ROWS KEEP column' for a renamed base column). The slot now offers the query's output names (aliases), plus the query's tables when it selects `*`. - CREATE TABLE ... DEDUP KEYS( fell back to every column in the schema. It now offers the columns declared in the statement only. ALTER TABLE / ALTER MATERIALIZED VIEW keep using the target. Found in the PR #42 follow-up review.
Per the PR #42 follow-up review: the remaining server-valid gaps are deferred follow-ups (the package owner's decision), not something that meets the never-stricter-than-the-server goal. The web console highlighting problems already show with the parser version the console uses (0.1.20), and the completion problems are functional, not cosmetic.
columnDefinition consumes these keywords under the keywordName label when they name a column, but the capitalizer only exempted the identifier rule, so `create table t (is int, ilike int, declare int)` came out as `IS INT, ILIKE INT, DECLARE INT`. Tokens under that label are now names. The IS / ILIKE operators are still capitalized. Found in the PR #42 review.
…/ GROUPING SETS The server parses each grouping key as an expression and then checks that it resolves to a column (SqlParser.parseParenthesizedColumnList, parseExplicitGroupingSets), so ROLLUP((x)), CUBE((x)), GROUPING SETS (((x)), ()) and ROLLUP(@k) with `DECLARE @k := x` are valid. A shared groupingKey rule now takes a column or a variable, optionally parenthesized. The AST drops the parentheses; a variable key is a VariableRef (GroupingSets.columns / sets are now GroupingKey = ColumnRef | VariableRef). Found in the PR #42 review.
The server reads one lexer token as the password, and `@x`, `$1` and `abc@def` are each one token there. Here `PASSWORD abc@def` silently became password 'abc' plus a statement `@def` (four statements with OWNED BY, and no completion after the value), and `@x` / `$1` failed. passwordValue now takes variable and bind tokens as a value or as glued pieces, so P@ssw0rd-style passwords stay one value and OWNED BY still follows. Found in the PR #42 review.
45ec94d scoped DEDUP KEYS / EXPIRE ROWS column suggestions, but only for a column list and a flat view query: - CREATE TABLE out AS (SELECT ts, k AS output_k FROM base) ... KEYS( offered the base columns (k, v), which the server rejects ("DEDUP column doesn't exist"); CREATE TABLE out AS (base) offered `base` itself as a column. - A view selecting `*` from a sub-query offered the base table's columns instead of the sub-query's output names. CREATE TABLE now tells a column list from AS (query), and one resolver gives the output of a query for both CTAS and materialized views: aliases and column names, with `*` / `t.*` resolved through sub-queries in FROM / JOIN down to physical tables. Found in the PR #42 review.
format(..., { capitalize: true }) collected syntax offsets from one parse of
the whole input, so a single unsupported statement left every keyword in the
script lowercase (`select 1; select .5m + 1; select 2`). When the whole input
does not parse, each `;`-separated statement is now parsed on its own and the
ones that parse are capitalized; the unsupported one stays as written.
Semicolons come from the lexer, so those inside strings, quoted names and
comments do not split. Found in the PR #42 review.
Adds a review status section (fixed findings and deferred follow-ups), removes ROLLUP(@k) and the whole-script capitalization limitation (both fixed), adds the deferred bind-name and `any` gaps and the newly reported UI items (comment guard inside nested comments, ARRAY contents highlighting), and updates the password and keyword-column notes.
4eb1baa covered @name, but an @ before a digit, a trailing @ or a lone @ lexes as AtSign, which the password reader did not take: `abc@123 OWNED BY g` failed and lost the owner, and `@123` / `abc@` failed. AtSign is now a password piece, first or glued. A lone `$` is still a lexer error (deferred). Found in the PR #42 follow-up review.
5205d84's resolver read join.table as the source, but a join's table is a tableRef wrapper, so joined tables were lost: CTAS `SELECT * FROM base CROSS JOIN other` stopped suggesting other's columns (a regression from f6d62d4). It also expanded every source for `t.*` and did not know CTE names. The resolver now collects FROM / JOIN / comma sources with their aliases, expands `*` from all of them and `t.*` only from the source aliased or named t, and resolves sub-queries and CTEs (this query's WITH and enclosing ones) to their output recursively. When a key slot's output cannot be resolved, the suggestion builder no longer falls back to every schema column there. Found in the PR #42 follow-up review.
…atus List exactly which password forms are covered (including @123 / abc@ / abc@123) and that a lone $ is deferred; describe the key-column scope as resolving declared columns or the query output through aliases, stars, joins, comma sources, sub-queries and CTEs.
One plain-language list of every known gap after PR #42, sorted by severity (High / Medium / Low / By design), with the end-user effect in the web console for each. It merges the previous known-limitations document, the PR #42 review findings that remain open, and a short list of what the reviews found and was fixed.
The gap list is kept outside the repository.
The permission list is parsed loosely (any 1-3 words), so content assist offered every word the grammar allows there: after `GRANT CONVERT ` it offered MATERIALIZED, SERVICE, TABLE, GROUP, ON and TO, where only PARTITION continues a permission, and after `GRANT ALTER COLUMN ` it offered ON / TO before the required TYPE / CACHE. In the permission list, explicit keywords are now kept only when they continue a server permission name (PERMISSION_NAMES), and ON / TO / FROM / `,` only once the typed words form a complete permission. The embedded TO of CONVERT PARTITION TO PARQUET|NATIVE still works; what follows ON is unaffected; parsing is unchanged. Two docs fixtures use `GRANT ALTER MATERIALIZED VIEW`, which is not a server permission (Permission.namePermissionMap has ALTER COLUMN TYPE / CACHE, ALTER SYMBOL CAPACITY and ALTER VIEW only), so they skip the autocomplete walkthrough; they still parse. The docs need fixing upstream.
…/ WITH The implicit-select support after DECLARE / WITH wrapped the query body in a new prefixedQueryBody rule, so `DECLARE @x := 1 SELECT ...` produced selectStatement > prefixedQueryBody > selectBody instead of selectStatement > selectBody. The web console walks that CST to merge notebook variables into a user DECLARE block (ui src/scenes/Editor/Notebook/declareUtils.ts checks selectStatement.children.selectBody); with the new shape it skipped the merge, and 5 of its declareUtils tests fail against this branch. The choice between SELECT and the implicit select is now inlined into selectStatement, so selectBody / implicitSelectStatement are direct children again and prefixedQueryBody is gone. Parsing, toSql, formatting and suggestions are unchanged (54k corpus and suggestion diff identical); the console's full unit suite passes against this build (3392 tests).
The server lexer does not split on `@`, so a variable is `@` followed by any identifier characters, including a leading digit or `$`. VariableReference required a letter or `_` first, so `DECLARE @1 := 1 SELECT @1` lexed as `@` + `1` and the formatter wrote `DECLARE @ 1 := 1 SELECT @ 1`, which the server rejects; `@foo$bar` split at the `$`. Passwords with `@` (abc@123, @123), $1 / :name binds and array slices are unchanged (tests added); corpus and suggestion diffs show no other change.
The server skips PRECISION after DOUBLE inside an expression (ExpressionParser: "ignore 'precision' keyword after 'double'"), so `SELECT double precision '1'` is the typed literal double '1'. It was read as a column `double` with alias `precision`, then `'1'` as a second statement; `+ 1` after it was a parse error, and suggestions stopped. A doublePrecisionLiteral alternative (like timestampWithTimeZoneLiteral) handles it. It is listed after identifierExpression, which yields to it by GATE: content assist prunes later alternatives once an earlier one completes, so `SELECT double precision ` still offers FROM for the column-plus-alias form. The only suggestion change is PRECISION after `SELECT double `; the corpus diff is unchanged.
javier
approved these changes
Nov 12, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Expands the editor parser's support for SQL accepted by QuestDB OSS and Enterprise, using server changes since 0.1.21 and server grammar/test cases.
The goals are highlighting, useful autocomplete, formatting, and acceptance of server-valid syntax. This is a tolerant editor parser with known gaps, not a complete server-parser clone or a server semantic validator. AST serialization may normalize source spelling; document formatting aims to preserve meaning.
Changes
tests/fixtures/server-queries.jsonholds queries copied from the server test suites, each withfile:line.tests/server-queries.test.tschecks entries in four ways (the autocomplete walkthrough has explicit exclusions):ROLLUP,CUBEandGROUPING SETS. This also fixes a regression on master:rollup,cube,setsandgroupingwere reserved, so they no longer worked as names. AlsoDEDUP SKIP/DEDUPLICATE,EXPIRE ROWS KEEP … ON, and implicit select bodies such asAS (t WHERE …).0x…,1.5f,E'…', typed literals (timestamp '…',ipv4 '…',double precision '…');$1and:name, backticks,$inside identifiers, and DECLARE variable names such as@1and@foo$bar;''in quoted values.AT TIME ZONE,WITH TIME ZONEcasts,DOUBLE PRECISION;IS [NOT] TRUE/FALSE,<> ALL (...),NOT WITHIN;GROUPSframes,EXCLUDE GROUP/EXCLUDE TIES;(expr).member, chained comparisons.AUDITED/OVERRIDABLE, FILL / SAMPLE BY / ALIGN TO with expressions, the server's clause orders;FOR <expr>, join alias stop words (TOLERANCE,RANGE, …), HORIZON JOIN options;pg_catalog.*()andinformation_schema.*()in FROM,UNION DISTINCT.[NOT] NULLcolumn modifiers, CTASCAST/INDEXoptions;TTL/REFRESHvariants;ADDwithoutCOLUMN,RENAME COLUMNlists,TYPE SYMBOL … INDEX;CONVERT PARTITION TO … LIST, mat viewSUSPEND WAL WITH.BEGIN/COMMIT/ROLLBACK,DEALLOCATE, PGSET,DISCARD/CLOSE/UNLISTEN/RESET,DROP ALL,UPDATE … AS, quoted COPY options,VACUUM PARTITIONS.files()is offered as a table-valued function rather than a scalar.storage_policies()remains in the function catalog; both callable and view-style forms parse.left,right,window,by,group,when,then,else,show,copy, … Alias gates mirror the server's stop lists.CREATE VIEW … WITH AUDIT, all 80 permission names in GRANT/REVOKE,WITH VERIFICATION, ASSUME lists,ADD COLUMN … OWNED BY, quoted memory limits, unquoted external aliases and passwords, REST token options.Feature commits record server source references and focused tests. Follow-up fixes cover completion scope, value preservation, and editor integration.
Verification
0c429ea): 61 files / 17,525 tests passed; lint, typecheck, and build passed.63a6badc: 138 UI unit files / 2,602 tests passed, plus real-editor highlighting, completion, and insertion checks.Worth a look
SELECT timestamp 'x'is now a typed literal, as on the server, not a column alias.BEGIN,COMMITand similar are statements, not implicit table selects.ARRAYis a non-reserved constructor keyword, exported asArrayType, and is not listed or suggested as a bare datatype. Concrete array types such asDOUBLE[], array constructors, and identifiers namedarrayremain supported. The parser retains lenient acceptance of the old bare type spelling.ROLLUP,CUBEandGROUPINGafter GROUP BY / SAMPLE BY;PREV,LINEARandNONEinFILL(;GROUPSin window frames;ATandISafter expressions; the session statements at statement start; full permission lists afterGRANT.pg_catalog.lists catalog functions valid in the current context: table-valued functions in FROM/JOIN, scalar functions in expressions. A table alias namedpg_catalogretains column completion.information_schema.*suggestions require a prefix/suffix match against the qualified or unqualified name. An explicitly typedinformation_schema.lists its functions; a following member fragment is matched by prefix/suffix. TypingSELECTno longer exposes loose fuzzy matches.@/$forms; preserve keyword-named columns and bind/slice boundaries; capitalize valid statements even when a neighboring statement has unsupported syntax.DEDUPLICATE→DEDUPVACUUM PARTITIONS→VACUUM TABLELIMIT ,5→LIMIT 0, 5;. That is the root cause of silent splits on unsupported syntax.Web console
Companion PR: questdb/ui#630 implements quote/comment handling, highlighting, and completion insertion in Monaco. Backtick identifiers use the theme's normal identifier text style; their contents no longer receive keyword styling. The UI also distinguishes bind variables from slice colons inside arrays.
The UI PR keeps its committed parser dependency at 0.1.20. The parser's new grammar, catalog namespace completion, information_schema filtering, and backtick lexer behavior require a release containing this PR. The local playground used a file dependency for combined verification; that link is not part of the UI PR.
The CST shape for ordinary
DECLARE ... SELECTbodies remains compatible with the console's existing notebook-global merging.Remaining boundaries
(, incomplete pasted strings, shorthand notebook-global injection, and execution statement splitting are outside these fixes.