[SangJunBak]

From the durable index key information we recorded in #39818, we derive the index key information from it and convert mz_index_columns to a builtin mv.

See commit messages for more details

Motivation

Part of sql-494

Verification

• Diffed this relation using the prod analytics workload replay against main and this branch.
• Created an SLT for index columns that assert expected behavior as well as behavior for indexes where the keys are varying expressions

[linear-code]

SQL-494

[def-]

QA LLM Review

1. MEDIUM -- mz_index_columns arranges every key row with its index's whole IndexColumns record, so its memory grows with the square of the key count

src/catalog/src/builtin/mz_catalog.rs:1030

user_index_columns unnests k.data->'value'->'columns' before joining to the Item row, and it computes index_id from k.data after the join. The optimizer therefore arranges (data, value, ordinality) for every key, and each of those rows carries the full O(k) record. That is about 34·k² bytes held resident on mz_catalog_server for as long as the index exists, and rebuilt on every hydration. The old builtin table cost O(k).

Details

The tell in the new catalog_server_explain.slt golden is →Arrange (((#0{data} -> "key") -> "id")) directly above →Table Function jsonb_array_elements[with_ordinality].

Measured with CREATE INDEX i ON t (a, a, …):

• Arrangement size of the mz_index_columns dataflow: 8.6 MB at 500 keys, 34 MB at 1000, and 136 MB at 2000. At the base, a 2000-key index grows all catalog-server arrangements combined by 0.37 MB.
• 30 indexes of 2000 keys: mz_index_columns is the largest arrangement on the catalog server at 3.9 GB for 60k rows. For comparison, mz_show_indexes_ind is 4.4 MB for 188k rows. Replica RSS peaked at 4.1 GB and settled at 2.45 GB. At the base the same workload settles at 859 MB.
• Privileges: CREATE on a cluster and a schema are enough, so any role that can create indexes can force this. A default index on a wide relation pays the same cost without any intent.

Fix: project the id and the columns array before unnesting, and join to the item before the LATERAL, as mz_columns does. This shape plans with the join on one row per index and the FlatMap after it, with no arrangement downstream:

FROM
    (
        SELECT
            data->'key'->'id' AS id,
            mz_internal.parse_catalog_id(data->'key'->'id') AS index_id,
            data->'value'->'columns' AS columns
        FROM mz_internal.mz_catalog_raw
        WHERE data->>'kind' = 'IndexColumns'
    ) AS k
    JOIN (
        SELECT
            data->'key'->'gid' AS id,
            mz_internal.parse_catalog_create_sql(data->'value'->'definition'->'V1'->>'create_sql')->'key_parts' AS key_parts
        FROM mz_internal.mz_catalog_raw
        WHERE data->>'kind' = 'Item'
    ) AS i ON i.id = k.id
    CROSS JOIN LATERAL jsonb_array_elements(k.columns) WITH ORDINALITY AS ik(value, ordinality)

This also stops parse_catalog_create_sql from running once per expression key, and the Item arrangement keeps only key_parts instead of every item's full jsonb. The cost is one parse per item, the same as mz_columns already pays.

[ggevay]

LGTM, minor things.

[ggevay]

This join arranges the full raw Item row of every item, create_sql included, keyed by gid (the Arrange (((#0{data} -> "key") -> "gid")) over Project: #0 in catalog_server_explain.slt). On a synthetic catalog with 14k items, that's about one copy of all create_sql text on mz_catalog_server (the mz_index_columns dataflow holds 53 MB there).

Only index items can match. Prefiltering them arranges only their rows. On that catalog, run as a user MV next to an exact copy of user_index_columns, the rows are identical and the size is 55 → 2.2 MB:

Suggested change

	JOIN mz_internal.mz_catalog_raw AS i ON
	i.data->>'kind' = 'Item' AND
	i.data->'key'->'gid' = k.data->'key'->'id'
	JOIN mz_internal.mz_catalog_raw AS i ON
	i.data->>'kind' = 'Item' AND
	i.data->'value'->'definition'->'V1'->>'create_sql' LIKE 'CREATE INDEX %' AND
	i.data->'key'->'gid' = k.data->'key'->'id'

Every persisted index's create_sql starts with CREATE INDEX ", CREATE DEFAULT INDEX included, because planning fills in the name and key parts and drops IF NOT EXISTS.