Repository navigation
Conversation
Design for extracting, validating, and installing LIR plans as materialized views, building on pinned LIR. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The envelope bundles imports, plan objects, and the output id. Parsing disables serde_json's recursion limit and runs on a fresh stack segment, then rewrites readable `$row` objects into the default row serialization. This enables serde_json's unbounded_depth feature workspace-wide. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
parse_envelope pre-scans the text iteratively and rejects nesting above MAX_JSON_DEPTH before parsing, so parse and drop recursion is bounded by the cap instead of by the input. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Add a validator for LIR dataflows that checks scoping (Get resolution, Let/LetRec placement and ids, unique LirIds, exports) and arity and shape (column references, MFP predicate positions, join closures, reduce and top-k plans, parallel vector lengths) against what rendering trusts. The walk is stack-safe under a recursion guard and accumulates every error it finds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An object whose output is empty and that has no export, such as a view that folds to an empty constant under an index, now flows to the objects that read it instead of being rejected. Operators that state their output arity keep it over an empty input. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The validator simulates the forms (raw collection and arrangement keys) rendering gives every node's output, and checks every read against them, for empty inputs too. Stated (key, permutation, thinning) triples and join stream thinnings must be canonical. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Rendering reads only the keys of a Get's stated arrangements, and lowering states identity permutations there for arrangements whose permutation makes a residual MFP the identity. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ping work Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…icity Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Discarding a type pass restored its bookkeeping by cloning sets that grow with the plan validated so far, which the budget did not count. An undo log now restores in time proportional to the discarded pass, and the arity inference of recursive bindings charges each node it walks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…plans Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A visit was charged one unit whatever its operator held, so a large constant or a wide collection visited many times did far more work than the budget suggested. Each visit is now charged the serialized size of its operator, rows by their encoded length, plus the width of its output. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Charging content made a single pass over the builtin mz_views index, which holds a constant of the generated builtin views' SQL, cost 686,000 of the 1,000,000 units. The budget now adds eight units per unit of plan content, so large plans pass and only amplification is rejected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Column types are trees that the rules clone and unify per column, so output width now counts type nodes, with types over 256 nodes treated as unknown, and imported types count toward the budget. Local bindings are found through a map instead of a scan, which cost quadratic work for a LetRec of many bindings. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A column type of more than 256 nodes is tracked as unknown, and a typing rule treats an unknown argument as satisfying it. In user plans an allowlisted function such as `and` could therefore be applied to a record literal and panic at evaluation. With the allowlist enforced, every site that produces an unknown type now reports it (imports, literals, constant columns, function and table function output types), and function applications, bool, limit and temporal bound checks, and join equivalence classes reject an unknown type. Without the allowlist nothing changes. Also cap the fan-out of `type_size` before queuing a record's fields, and assert the result of `resolves_many_locals_quickly`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… imports Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CREATE MATERIALIZED VIEW ... FROM PLAN now builds its dataflow from the plan instead of optimizing. Import descriptions come from the catalog through the DataflowBuilder import functions, the validator checks the dataflow in user plan mode and chooses the sink's arrangement, and the output type must match the declared schema. Boot rebuilds the dataflow from the plan stored in create_sql, and EXPLAIN PHYSICAL PLAN FOR CREATE ... FROM PLAN validates and renders a plan without installing it. Readers of the optional MIR expressions handle plan views: timeline selection uses the view's dependencies, read-then-write checks temporal bounds in the installed plan and skips index dependencies, and queries reject replacements defined by plans. Boot ordering sees plan imports. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A replacement materialized view can be defined by an LIR plan and applied to a target without inferred keys. Targets with keys are rejected by the existing schema check, because a plan's declared schema has no keys. Plan views that import a replaced materialized view keep running, and their stored plans name the target by its new item id. Errors on the FROM PLAN path report user SQLSTATEs: invalid plans, imports and column definitions use the new PlanError::InvalidLirPlan (42P17), and EXPLAIN stages other than the physical plan, which a plan does not have, use PlanError::NoSqlDefinition (0A000), for both EXPLAIN ... CREATE MATERIALIZED VIEW ... FROM PLAN and installed plan views. Reading a replacement defined by a plan reports 0A000. Building a plan's dataflow at CREATE time catches panics and reports them as internal errors. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add a create-lir-dataflow command that builds a materialized view dataflow from an LIR plan envelope the way CREATE MATERIALIZED VIEW ... FROM PLAN does, validates it in user plan mode, and submits it. The lir_mutation test applies seeded single mutations to optimizer plans of a join, a reduce, a top-k, a recursive transitive closure, a temporal filter and a constant, validates every mutant, and, given a clusterd binary, renders every accepted mutant on a replica and checks that the replica survives. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ow panics Enable soft assertions and set ore_overflowing_behavior to panic on the replica the lir_mutation test renders mutants on, as CI does, so that both surface as crashes. Assert that a default run generates at least 1000 mutants. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Some accepted mutants make the replica grow without bound. Let the lir_mutation test run clusterd under a wrapper command such as memcap, in a process group of its own so that a restart ends both, and record the mutant being rendered in pending.json. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Driver::peek_count now sums multiplicities instead of collecting a row per copy, and the lir_mutation test counts a sink's output with it. A mutated constant with a diff of i64::MAX otherwise exhausted the test's memory when it read the output back. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tion replica Another clusterd on the default port 6878 otherwise keeps the replica from starting. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* lir_mutation says on stderr when it skips the replica phase, documents how to run it, and notes that it skips the transient-id checks of parse_user_envelope. * A Replica kills clusterd and its wrapper on drop. clusterd stays in the test's process group, so a Ctrl-C ends it too. * create-lir-dataflow takes a monotonic flag per relation, and a relation without a shard is typed by its schema or else an index. * A lir_plan.spec scenario runs create-lir-dataflow end to end. * Findings: constant rows with huge diffs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`misc/lir-datalog` compiles Datalog rules over imported relations to an LIR plan envelope for `CREATE MATERIALIZED VIEW ... FROM PLAN`. Plans are built from Python types that `gen_lir_types.py` generates from the serde registry snapshot, with a codec for serde_json's encoding, using only the standard library. Recursive relations are `LetRec` bindings with an iteration limit, and every relation is made distinct by a `Reduce`. `lir_plans_datalog.slt` installs compiled plans for transitive closure, mutual recursion, fixed-length paths, a filter and a cross join, and compares them with the same queries in SQL. The unit tests check that the generated types are fresh and round-trip every envelope in the LIR plan SLT files. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Datalog means the fixpoint, so recursive bindings now report an error when they reach the iteration limit, and truncation needs `--return-at-limit`. The SQL reference in the sqllogictest uses `ERROR AT RECURSION LIMIT`, and a new case checks the error of a plan compiled with `--limit 2`. The compiler also accepts integer arguments as equality filters, rejects `_`, imports only the relations the output depends on, and rejects a zero-arity output relation, which `FROM PLAN` cannot declare. The generated encoder range-checks integers and rejects non-finite floats. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
demo.sh walks through extracting a materialized view's plan as JSON, rewriting a cross join filtered by an OR of equalities into a union of equi-joins with improve_plan.py, rejecting a broken variant, installing and comparing the improved plan, swapping it under the SQL view through a replacement, and installing a Datalog transitive closure. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Consolidation and reductions add diffs and joins multiply them, so a user constant with a large diff overflows on the replica, which panics under `ore_overflowing_behavior=panic`. In user plan mode the validator now requires every constant row diff to be between -1 and 1. Rename `ValidateConfig::enforce_function_allowlist` to `user_plan`, since the flag enables every user-plan-only rule, not just the allowlist. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`finalize_dataflow` printed and parsed every optimizer dataflow under soft assertions, which CI pays on every peek. The round trip now runs only with `MZ_LIR_ROUND_TRIP_CHECK=1`. The validator keeps running under soft assertions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A plan names its objects with transient placeholders, which were installed unchanged. The optimizer allocates transient ids from the same space, and compute introspection keys `mz_lir_mapping` by object id, so two dataflows sharing an id corrupted each other's mapping when one was dropped. After validation, every object is renumbered to an id from the coordinator's transient id allocator, at creation and at boot, and every read of it is rewritten to match. Validation errors still name the plan's own placeholders. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fix the `expect` message on a materialized view's locally optimized expression, and make `EXPLAIN ... AS LIR JSON` return an internal error unless the dataflow exports exactly one sink. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`EXPLAIN` of a materialized view defined by a plan shows the fresh transient ids its objects are built under, which depend on how many ids the coordinator allocated before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Statements no longer ignore failures, except the rejection of step 4, which the demo now asserts by its error text. Step 6 compared two views that run the same plan after the swap, so it now compares the swapped view with the view's SQL query run directly over the tables, and both comparisons fail the demo if a side returns extra rows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Variants were nested classes that did not derive from the enum class, so pyright rejected every variant passed where the enum's type is declared. The generator now emits each variant as a module-level subclass of its enum, declared and bound as an attribute of the enum class, which keeps `Enum.Variant(...)` working. The remaining pyright errors are fixed at their sites. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Clippy disallows `std::panic::catch_unwind`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Skunkworks prototype that treats LIR as a second input language next to SQL. A plan can be extracted with
EXPLAIN PHYSICAL PLAN AS LIR JSON, validated upfront so that no accepted plan can panic or hang a replica or environmentd, and installed withCREATE [REPLACEMENT] MATERIALIZED VIEW ... FROM PLAN. Everything is behindenable_lir_plans, and the work builds on the pinned LIR format.A toy Datalog compiler in
misc/lir-datalogtargets the same format, andmisc/lir-datalog/demodrives the end-to-end demo. The design is indoc/developer/design/20261019_user_provided_lir_plans.md, and what the prototype showed and did not show is in..._findings.md. This is not meant to merge as is.Posted by Claude Code.
🤖 Generated with Claude Code