[SangJunBak]

From the durable column information we recorded in #39818, we derive the column information mz_columns and convert it to a builtin mv.

See commit messages for more details

Motivation

Part of sql-494

Description

We report the column defaults from a create_sql statement, generate the mz_builtin_columns for our builtin collections since they don't exist in our catalog, then we convert mz_columns to a builtin mv using a combination of mz_builtin_columns and mz_catalog_raw.

Verification

• Expanded the existing SLT with a more thorough test suite that asserts against a column of each type in our system.
• Diffed this relation using the prod analytics workload replay against main and this branch.

[linear-code]

SQL-494

[def-]

QA LLM Review

1. HIGH -- A column named $serde_json::private::Number panics every catalog MV on mz_catalog_server

src/catalog-decode/src/create_sql.rs:264

item_details now uses user-chosen column names as JSON object keys in column_defaults, and Jsonb's packer treats a map whose first key is $serde_json::private::Number as an arbitrary-precision number. With that column name, Jsonb::from_serde_json returns Err and to_jsonb's .expect("valid JSONB") (src/expr/src/scalar/func/impls/catalog.rs:26) panics inside the mz_catalog_server dataflows. Any authenticated user can trigger this with a temporary table, which needs no grants, and it crash-loops the catalog server replicas.

Details

Trigger: CREATE TEMPORARY TABLE t ("$serde_json::private::Number" int);. item_details returns {"column_defaults": {"$serde_json::private::Number": null}, ...}. The Collector in src/repr/src/adt/jsonb.rs (KeyClassifier → KeyClass::Number → NumberParser) rejects it with invalid type: null, expected a valid number. I ran the verbatim Collector/KeyClassifier code against serde_json 1.0.151 (arbitrary_precision + preserve_order) and confirmed it. A DEFAULT 5 on that column, followed by a second column, fails with invalid length 2. With a single column, the whole map is silently replaced by the number 5.

Impact: parse_catalog_create_sql runs on every Item row in mz_columns, mz_tables, mz_views, mz_materialized_views, mz_history_retention_strategies and others, all IN CLUSTER mz_catalog_server. The clusterd panic hook aborts the process, so every replica of that cluster crash-loops and catalog introspection stalls (SHOW, pg_catalog, console, dbt). Dropping the table probably does not recover it. A restarted dataflow resumes at the MV's stuck write frontier, so it re-evaluates the row inserted at the creation time. Before this change, item_details emitted only fixed keys.

Fix: do not key JSON by user identifiers. Emit column_defaults as an array in column order (create_sql column order matches the latest desc, VERSION ADDED columns included) and read it with i.column_defaults->>((c.data->'key'->>'position')::int). Separately, have to_jsonb map the error to EvalError::InvalidCatalogJson rather than .expect, so a future packing failure fails one MV instead of aborting the replica. A regression test that creates the column above and then reads mz_columns would cover both.

[bosconi]

I've re-run three jobs from nightly 18550 on this head (889c34a), because DDL looked much slower there than at the previous head or on main:

• testdrive with SIZE 8 1 timed out at 68 min after 62 of 148 files. It passed in 38 min at the previous head (18541) and 34 min on main (18553). Over the files both runs reached, statements took about 6x longer: CREATE TABLE averaged 0.258 s against 0.056 s, and COMMIT 0.720 s against 0.080 s.
• SLT (2 replicas) 7 timed out at 67 min, against 27 min on main.
• Workload Replay (1% initial data) 2: object creation on workload_ddnet.yml regressed 29.1%, against a 20% threshold.

The code change between the two heads is small, so this may be the agents, and the re-run on fresh agents should tell us: testdrive, SLT, workload replay. I'll follow up with the results in my review.

[aljoscha]

Did a first pass, looks good, but would be good to have @ggevay take a look.

I did have some inline questions. Also, that change from a string "NULL" to SQL NULL, have to think about if we want to do that. It feels correct, but it's a change, so ... 🤷‍♂️

[aljoscha]

weird that we had this ...

[SangJunBak]

Yeah I thought so too! Re:

Also, that change from a string "NULL" to SQL NULL, have to think about if we want to do that. It feels correct, but it's a change, so ...

Gabor suggested changing it to a SQL NULL here https://linear.app/materializeinc/issue/SQL-494/convert-mz-columns-and-mz-index-columns-to-catalog-views#comment-839801c2, mainly to keep pg parity

[aljoscha]

did we change anything about this issue here?

[aljoscha]

ah okay, that resolves my earlier question ... 😅

[aljoscha]

now I'm out of my depth, might be good to have @ggevay take a look at the SQL at least, to make sure it's good, I'm still reviewing the rest of the PR

[aljoscha]

what's up with these changes?

[SangJunBak]

Ah sorry, should've been more clear in the commit message. This is a result of replacing the string 'NULL' with a SQL NULL. The motivation was brought up by Gabor https://linear.app/materializeinc/issue/SQL-494/convert-mz-columns-and-mz-index-columns-to-catalog-views#comment-839801c2

[aljoscha]

what's up with these change?

[SangJunBak]

Same motivation #40001 (comment). Here the "hasDefault" column turns to false which makes sense since if you look at the table definition, they have no column defaults.

[def-]

QA LLM Review

1. HIGH -- One wide temporary view OOM-loops mz_catalog_server, because mz_columns copies each relation's whole ItemColumns row onto every one of its columns

src/catalog/src/builtin/mz_catalog.rs:673

user_columns unnests c.data->'value'->'columns' while each output row still carries the full data JSON. The kind = 'ItemColumns' filter only runs after the unnest, so an N-column relation produces N rows that each hold an O(N) payload. Any user without privileges can run CREATE TEMPORARY VIEW with 2000 columns, and the mz_catalog_server replica is OOM-killed on every restart. Dropping the view does not stop it, and catalog introspection (SHOW, pg_catalog, console, dbt) stalls for the whole environment.

Details

In the plan in catalog_server_explain.slt, l0 is Table Function jsonb_array_elements(((#0{data} -> "value") -> "columns")) with Project: #0..=#5, so #0{data} is kept. The kind filter sits in the consumers of l0 because l1 (the custom-type join) reads l0 without it. l1's Arrange (#5{custom_type}) then keeps data resident for every custom-typed column. column_defaults is carried per column the same way (Project: #1..=#4 and both custom-type arrangements), so a table with long default expressions also costs O(N²).

Repro: local bin/environmentd at this head, where replicas are limited to 4 GiB. I connected as a new non-superuser role with no CREATE privilege and ran CREATE TEMPORARY VIEW tv AS SELECT NULL::int4 AS "c00000xxx…", … with 2000 columns and 206-character names (448 KB of SQL). Its ItemColumns row is 562 KB, so the unnest produces about 1.1 GB per copy, and l0 has three consumers. The mz_catalog_server replica was OOM-killed 3 s later and every ~10 s after that. SHOW TABLES hung while SELECT 1 worked. It kept crash-looping after DROP VIEW, and also after the temporary view's session ended, until I stopped the environment (41 OOM kills over 8 minutes in the temporary-view run). The likely reason is that each restart re-hydrates from the MV's stalled write frontier, which still includes the row's insertion. Control: the same steps on the base branch, which records the same 562 KB ItemColumns row but keeps mz_columns as a table, caused no OOM, and mz_columns returned 3923 rows. Memory grows with columns² × name length, so a larger replica only raises the number of columns needed. Legitimate wide relations pay the quadratic cost on every hydration and on every change to the item.

Fix: filter and project before unnesting, and unnest the defaults separately so no per-column row carries an O(N) value:

item_columns AS (
    SELECT
        mz_internal.parse_catalog_id(data->'key'->'id') AS id,
        data->'value'->'columns' AS columns
    FROM mz_internal.mz_catalog_raw
    WHERE data->>'kind' = 'ItemColumns'
),
defaults AS (
    SELECT i.id, d.ordinality, d.value AS "default"
    FROM items AS i,
        jsonb_array_elements_text(i.column_defaults) WITH ORDINALITY AS d(value, ordinality)
),

user_columns then reads FROM item_columns AS c JOIN items AS i ON i.id = c.id CROSS JOIN LATERAL jsonb_array_elements(c.columns) WITH ORDINALITY AS col(value, ordinality) … LEFT JOIN defaults AS d ON d.id = c.id AND d.ordinality = col.ordinality. With this change the plan drops columns right after the table function. On the same replica where the PR's user_columns SQL was OOM-killed at 4 GiB, the rewrite returned the 2000 rows in under a second. On a table, view and materialized view with list, map and record columns, it returned the same rows as the PR's SQL.

2. LOW -- mz_columns.default reports the first DEFAULT of a column, but the planner applies the last

src/catalog-decode/src/create_sql.rs:271

When a column has more than one DEFAULT, item_details takes the first non-NULL one, but plan_create_table overwrites the default with each option, so the last one wins (src/sql/src/plan/statement/ddl.rs:343). CREATE TABLE dd (a int DEFAULT 1 DEFAULT 2, b int DEFAULT 5 DEFAULT NULL) is accepted, and INSERT INTO dd DEFAULT VALUES stores a = 2 and b = NULL. mz_columns, and therefore pg_attrdef and information_schema.columns, report 1 and 5 instead. The in-memory path this PR removes reported the planner's value. Fix: take the last ColumnOption::Default and then apply is_null_literal to it.

[ggevay]

LGTM, minor things

[ggevay]

mz_columns is in mz_catalog, so misc/python/materialize/checks/all_checks/builtin_version_pin.py should record the conversion: "mz_columns": MzVersion.parse_mz("v26.49.0"), or the release this ends up in. It's None there today. The same goes for mz_index_columns in #39427.

[ggevay]

On a synthetic catalog with 14k items and 324k columns, the mz_columns dataflow holds 165 MB of arrangements on a 1-worker mz_catalog_server. The old table's index was 12 MB, and it stays as mz_columns_ind. Most of the 165 MB is per-column state: the JSON value of every column is carried through the delta join's arrangements, and the defaults join adds more.

Extracting the scalar fields right after the unnest halves it. I ran this as a user MV over mz_catalog_raw next to an exact copy of the user branch here. The rows are identical and the size is 166 → 86 MB:

    cols AS (
        SELECT
            c.id,
            col.ordinality,
            col.value->>'name' AS name,
            (col.value->>'nullable')::bool AS nullable,
            (col.value->>'type_oid')::oid AS type_oid,
            (col.value->>'type_mod')::int4 AS type_mod,
            CASE
                WHEN jsonb_typeof(col.value->'custom_type') = 'object'
                THEN mz_internal.parse_catalog_id(col.value->'custom_type')
            END AS custom_type
        FROM
            item_columns AS c
            CROSS JOIN LATERAL jsonb_array_elements(c.columns) WITH ORDINALITY AS col(value, ordinality)
    )

The rest of user_columns then joins cols to items, custom and defaults as it does today.