fix(query): align membership, concat, and grouped fallback semantics

[singaraiona]

What & why

Fix query membership and concat results that differed between planned execution and evaluator fallback. Inline typed membership probes use the null-aware kernel for numeric, temporal and SYM columns. Fallback covers empty inputs, partition keys, GUID nulls and nested not-in; mixed LIST probes retain general membership.

Collection concat keeps ordinary collection semantics for literal and bound vectors, in either operand order. Column-derived text operands retain text concat through casts, uncompiled operations and helper calls; a reordered, indexed or cast column is a whole column in projections, so select agrees with update, where and by. A STR column's cell keeps its type row by row, and a null cell is the empty text the vector kernel reads.

Helpers called from a query fallback run as bytecode. Row provenance is a flag beside every VM stack slot and on every scope-frame binding: parameters, let locals, a scoped window's locals, a closure's captures and a helper's on-demand source columns carry it; builtins the planner lowers row by row keep it, collection verbs clear it, compiled calls pass it to their callee and return it with the result, and concat consults it. The flag is tri-state and relative to the innermost query. A call frame marks the lexical boundary: the query's aliases and locals are hidden, a free source-column name is the query's binding for it (the row's cell, the group's rows or the column), a literal symbol is the symbol in a helper called by name, and other free names are globals. Lambdas written inline in the query are query text, stay on the tree walker and keep the query's column rule.

Closures: a fn literal inside a compiled body is compiled once, as a template, when its enclosing function compiles, and every closure made from it shares that bytecode (OP_MKCLOSURE) and captures only the enclosing locals its body mentions, bound by name into its own slots (OP_BINDCAPS). A let of a captured name rebinds the slot for that call, a parameter shadows its capture, a local named like a builtin is the callee, self is the closure itself, an empty let slot or an unbound capture reads as the global or a name error (OP_LOADENVG), eval and resolve run inside a scope window over the locals that wraps only the call, a closure body resolves the names it does not capture when it runs (a top-level definition keeps its builtins baked, and its shared template is recompiled once a function-valued global is rebound), a lazy value is materialized when captured, capture is lexical (a caller's locals are never captured, a change from dev's dynamic capture for interpreted makers), and return ends a lambda on the tree walker as it does compiled. Closures made by interpreted code share a template per literal and capture set through a per-thread table keyed on every node of the literal. Closures are never inlined into a query.

Helpers built from effect-free builtins are inlined by the planner and the fallback. An opaque helper can be the by: key: a pure body (callees followed through globals, parameters, lets and captures; a computed or unknown callee, set, eval and system or I/O builtins are effects) is evaluated whole over the rows where: keeps and again row by row when a conditional tested a vector or the whole run failed; a body with an effect runs row by row from the start. A grouped key column keeps its "may have nulls" flag only when an emitted key is null, so fused and materialized results agree on the wire.

Grouped fallback reserves all aggregate slots, keeps the requested column order (an output named like its key keeps both columns) and uses the same kept-group prefix for keys and outputs. Constant collection COUNT counts the collection. Filtered COUNT handles computed inputs. Grouped updates prepare pure helper expressions and reuse dependency analysis across groups. Nested-select composition preserves materialization when an outer sort names an inner computed or renamed column. Index-attachment diagnostics and query-scope documentation are corrected.

Includes #758's head (3a8ff52): fixed row chunks are folded in chunk order, independent of worker scheduling, with accumulator blocks padded to 128-byte line pairs. The fuzz oracle retains exact null-flag comparison and its original bounded aggregate tolerance. Pool tests drain pending read-ahead and guarantee worker participation.

Behaviour changes from dev, each documented: grouped fallback outputs follow the requested order; reordered, indexed or cast concat operands are whole columns in projections; quoted symbols are literals inside named helper bodies; inside any function concat keeps collection semantics unless an operand derives from the query's rows.

Validation

• Full local ASan/UBSan suite: 4,136/4,136 passed. All ten CI checks pass on b21830b6, including the targeted audit, the fuzz smoke and cppcheck; TSan clean.
• Regression tests: test/rfl/query/concat_*.rfl, membership_*.rfl, helper_bytecode_provenance.rfl, concat_helper_consistency.rfl, group_key_null_flag.rfl, lang/closure_locals.rfl, mem/closure_flags_release.rfl, nested_sort_computed.rfl, keyless_pool_accum.rfl, lang/query_helper_reinit (runtime re-init), lang/closure_malformed_capture, lang/closure_lazy_capture_vm and store/serde_lambda_malformed_closure. Reverting each hunk of the helper rounds in turn fails at least one test.
• The caches keyed on symbol ids and builtin objects are rebuilt per symbol-table epoch.

Release medians against dev 57f5cfe3, eight workers, 2M rows unless noted:

Case	dev	Current
Per-row scalar aggregate over a helper, (sum (hn (+ x 1)))	578 ms	612 ms
Per-row update with if and a helper	721 ms	788 ms
Per-row text helper beside (reverse x) (dev computes a collection)	510 ms	615 ms
1000 rows × fib 15 per row	165 ms	200 ms
fib 24 outside queries, one core	11.0 ms	11.2 ms
200k closures made and called once, one core (dev fails with a name error)	—	70 ms
(times 1000000 (cadd 2)), closure call, one core	183 ms	142 ms
SYM / I64 inline membership probe, 20M rows	parity with the planned probe
Planning, plain grouped fallback, 10 rows	5.2 µs/query	5.4 µs/query

Scripts: bench/concat_fallback.rfl, bench/concat_update_fallback.rfl, bench/membership_fallback.rfl.

Checklist

• Targets dev
• Conventional commits
• Debug and release builds pass
• Tests and regressions validated

[singaraiona]

Reviewed 2ab29f1 against the four follow-up issues from #742. Constant collection concatenation and numeric whole-column fallback are routed separately from row-wise text concatenation, evaluator membership applies the query null-row rule, and the index diagnostic names the supported types. Under ASan/UBSan, all 3 membership-filtered tests, all 28 concat-filtered tests, and the zone_sym_diagnostic test pass. Additional probes covered filtered/computed numeric concatenation, scalar aggregation, mixed-type constant collections, and fallback membership projections. No new actionable findings. Grouped aggregation over numeric concat remains unsupported, but it also failed before this PR.

[ser-vasilich]

Reviewed. The four findings from #742 are fixed. For SYM, integer, F64 and temporal columns with nulls, the planned and evaluator in/not-in now agree in all of these places, on in-memory, splayed and parted tables:

• where, count, projections, if, lambdas, aggregates;
• grouped where, update where, delete where.

Outside queries, in keeps null==null and not-in matches (not (in ...)) on vectors. (concat [2] [1]) folds to [2 1] and works as a probe. The idx diagnostics are corrected.

Before merging:

1. Evaluator in got 6–9x slower (collection.c ~1642, eval_membership).

   • In query scope every BOOL result gets a serial ray_vec_is_null() pass over all rows. SYM has no HAS_NULLS gate, so every row is read.
   • Release, 20M rows, -c 8, PR against the PR with ray_in_eval_fn short-circuited to ray_in_fn:
     • where: (in s (as 'SYM (list "1" "5" "7"))), the inline-constructor shape the new tests use: 15.0 → 129 ms;
     • where: (in x (distinct [1 5 7])): 15.9 → 95.9 ms.
   • The planned path and eager in are unchanged.
   • Suggestion: in query scope call ray_in_vec_exec(val, vec, negate) first. It already has the plan's semantics (null rows false, null set elements skipped) and runs in parallel. Keep the masked path only for the shapes where it returns NULL (STR, GUID, LIST), or at least skip the mask for in when the set has no nulls.

2. Folding can produce a LIST constant that the planned in reads as an empty probe (query.c ~1649).

   • (select {from: T where: (in s (concat ['a] [1]))}) with s = ['a 'b 'c 'd] returns []; dev and eager return [0].
   • (select {v: (in s (concat 'a 1)) from: T}) gives [false false false] against [true false true] on dev.
   • Suggestion: don't fold when the result is a LIST.

3. Text concat shapes changed, which the description says stay on the string path. dev's outputs were wrong too, but the change is worth stating:

   • (select {v: (concat ['a 'b] "!") from: T2}) was ['a! 'b!] and is now a type error. The same happens for (concat L "!") with a global L = ['a 'b] and for (concat "!" ['a 'b]).
   • (concat ['a 'b] '!) gives 3 rows from a 2-row table.
   • (concat s x) (a SYM column with a number column) was ['a 'b 'c] and is now a LIST of length 2n.
   • (concat s x y) is an arity error.

4. not-in is not the complement of in for LIST results, because ray_not_fn of a LIST returns a scalar.

   • (not-in ["ab" "c"] "a") gives false, while in gives ([true false] [false]).
   • In a query, where: (in ss (list "" "a")) errors with "got LIST", but where: (not-in ss (list "" "a")) silently returns [].

5. The new doc paragraph promises agreement that still fails in a few places. These are older than this PR:

   • GUID values of all zeros without HAS_NULLS: the plan treats them as null (exec.c ~1276), the evaluator doesn't. Planned where: (in g Pg) gives [0]; evaluator where: (in g (distinct Pg)) gives [0 1].
   • A parted partition-key column (MAPCOMMON) on the evaluator path: where: (in part (distinct [1])) gives [], against [1 0N 2] planned. eval_membership flattens only RAY_IS_PARTED.
   • (in s (concat 'a 'b)): the plan probes the string ab and returns []; eager returns [true true false].
   • An empty table on the evaluator path: in errors with "got LIST". The empty-table test covers only not-in, which passes because (not ()) is a scalar.
   • STR columns: the plan gives "nyi"; the evaluator does per-character membership.

6. (select {n: (count (concat x [4])) by: k from: T}) now fails with "failed to compile aggregation argument". dev was silently wrong ([2 1]); the correct answer is [3 2]. The per-group routing used for distinct was not extended to concat.

7. A folded constant of the wrong length truncates or pads the other columns. This is older but now easier to reach: (select {v: (concat 1 2) w: x from: T}) on a 3-row T returns 2 rows with w = [1 2]. concat_collections.rfl:3 asserts that 2-row result.

Nits:

• query.c ~1651 sets g->compile_err without the if (!g->compile_err) guard used elsewhere.
• The null rule follows dynamic scope: a function evaluates (in 0N [0N 5]) differently at top level and inside a select. The doc could say "evaluated during a query".
• The diagnostics still print the type code ("got type 12") rather than the name.

Tests: disabling each new guard makes a new test fail. There are no evaluator-path cases for GUID, STR, F64, temporal or MAPCOMMON columns, for in on an empty table, or for the shapes in item 3.

A run of the 43-query workload on 100M rows with this PR together with #741 and #743 on top of dev matches dev's timings and outputs: those queries all take the planned path.

[singaraiona]

Fixed in 74fca72. Thanks for the concrete repros; these exposed gaps in my earlier review:

• Evaluator membership tries the existing typed kernel first, avoiding the serial null-mask pass for supported numeric, temporal, and SYM inputs. LIST, STR, and GUID shapes keep their general path.
• Mixed LIST concat results are no longer folded into typed membership probes.
• Text vector/scalar concat stays on the string path, in either operand order. Numeric/mixed collection concat remains binary and uses collection semantics; the documentation now states this distinction.
• not-in recursively complements nested results instead of collapsing a LIST to one Boolean. Both in and not-in now reject a nested WHERE result consistently.
• Empty evaluator predicates are BOOL vectors; MAPCOMMON operands are materialized; GUID zero payloads are masked even without HAS_NULLS. Tests cover these plus F64 and temporal inputs. Documentation now limits the agreement claim and explains dynamic query scope and STR limitations.
• Grouped collection-concat aggregates evaluate within each group; grouped text concat keeps its existing query kernel. Both paths have regressions.
• Planned projections validate equal column lengths after scalar broadcasting, preventing silent truncation or padding beside a folded collection.

The existing query-specific concatenation of two scalar symbols remains string concatenation; it is distinct from eager collection concat. STR membership retains its existing collection behavior and typed-plan limitation. The compile-error ownership guard is also fixed.

Validation: full make test -j8 TEST_CORES=8 under ASan/UBSan: 4,080/4,080 passed, 0 skipped, 0 failed. Release build passed.

Added bench/membership_fallback.rfl: 20M rows, -c 8, median of six runs after one warmup, previous PR head 2ab29f1e versus this commit:

Query shape	Before	After
SYM inline probe	72.32 ms	39.36 ms
I64 inline probe	67.27 ms	39.64 ms
SYM planned probe	36.01 ms	37.46 ms
I64 planned probe	35.84 ms	37.79 ms

The evaluator now runs close to the planned path on this benchmark. These are local measurements, not the same machine/results as the review's baseline. The optional diagnostic type-name polish remains unchanged.

[ser-vasilich]

Rechecked 74fca72.

Fixed:

• Evaluator in/not-in speed. Release, 20M rows, -c 8, the evaluator is now within 1.0–1.1x of the planned path across 9 cases: SYM and I64 in/not-in with and without nulls, plus GUID.
• The LIST-probe fold: where: (in s (concat ['a] [1])) returns [0].
• The text concat shapes: symbol vector with a text atom, either order, now gives 2 rows.
• The nested not-in complement. WHERE rejects both in and not-in on a LIST.
• MAPCOMMON and empty-table WHERE.
• Grouped collection-concat aggregates: count/sum/avg/min/max/first/last/med/count-distinct, multi-key, with where:.
• The compile_err guard.

The typed-kernel-first path matches the masked path in 49 column × probe combinations, byte-identical to 2ab29f1:

• literal SYM, W8/W16/W32 splayed, I64/I32/I16/U8/BOOL, F64 with -0.0, DATE/TIME/TIMESTAMP, GUID;
• atom, empty, null-bearing and cross-type probes;
• parted SYM/I64/F64/DATE/GUID and the MAPCOMMON column.

Each of the five new guards fails a test when disabled. About 140 projection shapes behave as before except the ones listed below. make test: 4080/4080.

New regression: grouped arithmetic over two or more aggregates, one of them over a text concat, now errors.

• With T x [1 2 3 4 5], k [0 0 1 2 2], s ['a 'b 'c 'd 'e]:
  • (select {n: (+ (count (concat s "!")) (count (concat s "?"))) by: k from: T}) → "type: concat: cannot concatenate SYM and str". 2ab29f1 and 2.6.2 give [4 2 4].
  • (+ (sum x) (count (concat s "?"))) by k (was [5 4 11]) and (- (max x) (count (concat s "!"))) (was [0 2 3]) fail the same way.
  • With a single hidden slot it works, but only because of the hidden_max < 1 → 1 floor.
• Cause:
  • count_agg_subexprs (query.c ~3151) calls is_group_dag_agg_expr(expr, NULL), and expr_has_collection_concat(.., NULL) returns true (~2228). So text-concat aggregates are not counted and hidden_max (~10153) undercounts.
  • The routing dry-run (is_decomposable_agg_compound, with tbl) says decomposable, but the rewrite (~12324, cap = hidden_max) bails.
  • The output falls back to per-group eval, where concat is eager and SYM vector + string is a type error.
• Fix: pass tbl to count_agg_subexprs, or have the NULL case count the aggregate; an upper bound must over-count.

Risks:

• The same NULL-tbl gate is used in match_group_count_emit_filter (~2875) and match_group_desc_count_take (~2956). Their agg_index skips text-concat aggregates while the planner includes them, and agg_engine only checks the op (~6111, ~6155). So with two aggregates of the same op the top-N filter could target the wrong one. I could not produce a wrong result: two SUMs with 3 and 50k groups, one and two keys.

• The new length validation (exec.c ~3783) broadcasts scalars to the input row count rather than the common column length, and runs before take:. Some queries that worked now error (T has 3 rows):

  query	2ab29f1	now
  (select {a: [1 2] b: 5 from: T})	[1 2] / [5 5]	"ragged columns (column 1 has 3 rows, expected 2)"
  (select {a: (concat 1 2) b: 5 from: T})	[1 2] / [5 5]	same error
  (select {b: x a: [1 2] from: T take: 2})	[1 2] / [1 2]	error

  The other outputs that changed were truncated or NA-padded before and are now correct errors.

• GUID zero masking. The test comment says only "older files" lack HAS_NULLS, but current producers do too: .csv.read [.. GUID] and (as 'GUID (list .. "0000...")) write attrs 0. On such a column:

  • vector nil? is all false;
  • by: g keeps a separate group;
  • eager in matches zero to zero;
  • atom nil? is true.

  Query in/not-in now drop the row as null. That is consistent with the planned path and atom nil?, but not with vector nil? or grouping. The root fix belongs in the producers or in ray_vec_is_null for GUID.

Nits:

• The error class differs by path: planned is domain ("ragged columns"), the evaluator is length ("output column length 3 does not match 4"). The tests pin domain.
• Trying the typed kernel first bypasses an index attached to the probe set: ray_in_fn consulted it, and now an O(set) hash is built per call. This matches the planned exec_in.
• membership_fallback_storage.rfl: both partitions hold identical data, so in part [1] and not-in part [1] both assert [1 0Nl 2], and a swapped in/not-in would still pass.
• An empty-table projection {v: (in x (distinct [1]))} still gives a LIST-typed column, where the planned path gives B8. This predates the PR.

Unrelated to this PR, but worth an issue:

• A projection with an inline probe, (select {v: (in s (as 'SYM (list ..))) from: T}), evaluates the probe constructor once per row. A counting lambda fires 1000 times on 1000 rows on 2ab29f1, 74fca72 and 2.6.2 alike. At 20M rows that is 19.6 s against 3.7 ms planned.
• (in (list 1 (list 2 3)) [1 3]) gives [true true], but (in (list (list 2 3) 1) [1 3]) gives ([false true] true): ray_in_fn reads r->b8 from a vector result (collection.c ~1535). The new nested not-in inherits this.

[singaraiona]

Fixed the grouped-arithmetic regression and scalar broadcasting in ad2cb03.

• count_agg_subexprs now receives the source table, matching the decomposition/rewrite decision. All three reported compound text-concat aggregate examples pass, plus a filtered-source case.
• The top-N matcher also receives the table, so it counts the same aggregate slots as the planner. The outer count-filter matcher runs before its child source is resolved; it now declines the optimization when concat is present. Added top-N checks with two SUM aggregates whose rankings differ, and a nested count-filter check.
• Scalars broadcast to the projected collection's length, independent of column order. Both {a: [1 2] b: 5} and {a: (concat 1 2) b: 5} now produce two rows. Unequal non-scalar lengths still error before take:. That boundary is intentional: take: limits a valid table; it does not repair ragged projections. The docs and a regression now state that explicitly.
• Partition fixtures now contain distinct values, so swapped in/not-in partition results fail. Corrected the GUID test comment to include current producers missing HAS_NULLS.

The broader GUID producer/null-detection inconsistency, repeated inline-probe evaluation in projections, empty projection typing, and mixed scalar/nested LIST membership remain outside this fix. The planned domain versus evaluator length diagnostic difference and probe-set index bypass also remain as noted.

Validation: make test -j8 TEST_CORES=8 under ASan/UBSan: 4,080/4,080 passed, 0 skipped, 0 failed. Release build passed.

[ser-vasilich]

Rechecked ad2cb03 against 74fca72 and 2ab29f1.

Fixed:

• The three grouped-arithmetic queries over text-concat aggregates again give [4 2 4], [5 4 11] and [0 2 3]. The filtered-source case gives [2 2 4].
• Scalar broadcasting. {a: [1 2] b: 5} and {a: (concat 1 2) b: 5} give two rows in either column order.
• Passing the table to the top-N matcher fixes a wrong result that 74fca72 had with sparse keys:

  (set V (as 'SYM (list "a" "bb" "ccc" "dddd" "eeeee" "ffffff" "g")))
  (set TN (table [k x y s] (list (* 1000003 (% (* (til 20000) 7919) 2000)) (% (* (til 20000) 104729) 100003) (% (* (til 20000) 15485863) 99991) (at V (% (* (til 20000) 31) 7)))))
  (at (select {a: (max (strlen (concat s "!"))) c: (max y) by: k from: TN desc: c take: 3}) 'c)
  

  74fca72 gives [99658 97257 89965]. ad2cb03 and a full sort give [99985 99979 99973]. A multi-key [k j] SUM case at 1M rows was also wrong on 74fca72 and is right now.
• The partition fixtures now assert disjoint values for in and not-in.

Tests. I reverted each hunk in turn and ran concat_review.rfl:

• count_agg_subexprs with NULL again: lines 25–28 fail.
• The exec.c broadcast-length detection: lines 30–32 fail.
• Top-N matcher with NULL again: the file still passes, because the 3-group fixtures use dense keys, which never reach that path. The sparse repro above fails at the suite's 2 cores and could go in as the test.
• if (has_collection) has_full = true;: the file still passes, because the tests use the literal 5, which all_const already covers. Without the line, (select {a: (concat 1 2) b: (sum x) from: T}) breaks, and so does a lambda that returns a scalar. A test could be (at (select {a: (concat 1 2) b: (sum x) from: T}) 'b) -- [6 6].
• The count-filter decline on concat: with the guard removed I found no wrong result. The shifted index lands either on another COUNT slot with the same per-group value or on a non-COUNT slot that the op check rejects. The guard is harmless.

Behaviour changes from the exec.c hunk (T has 3 rows, O has 1 row):

• A one-row table broadcasts a real column. (select {a: [1 2] b: x from: O}) gives b [7 7], while the same query on 2 or more rows is a ragged error. The result now depends on the row count. 2ab29f1 gave [7 NA] and 74fca72 an error.
• A collection with where: now gives an internal-looking error. (select {a: [1 2] b: 5 from: T where: (> x 1)}) returns "domain: sel_compact: selection nrows mismatch (sel=3 tbl=2)". 74fca72 said "ragged columns (column 1 has 3 rows, expected 2)". {a: (concat 1 2) b: (sum x) where: …} behaves the same way.
• The evaluator fallback still broadcasts to the input row count. {a: (concat [1] [2]) b: (enlist 7)} fails with "length: output column length 3 does not match 2". This is the same on 2ab29f1 and 74fca72.

Performance. Release build, 20M rows, -c 8:

• No regression in any of these: count-filter and top-N with and without text concat, multi-key grouping with where:, projection, and SYM/I64/GUID in/not-in.
• On a 10-row table, top-N with text concat takes 13.6–14.2 µs per query against 10.3–11.5. The extra time appears only when the matcher fires, which it now does on the right slot.

make test (ASan/UBSan): 4080/4080.

Unrelated to this PR (also on dev 57f5cfe and v2.11.0):

• SIGSEGV.

  (set A (table [x k s] (list [1 2 3 4 5] [0 0 1 2 2] ['a 'b 'c 'd 'e])))
  (select {n: (count (upper s)) by: k from: A where: (> x 1)})
  

  This dies with "fatal SIGSEGV at fault addr 0x18". UBSan points at find_ext(...)->sym on a computed aggregate input in agg_engine.c (~6978).
• nyi. (select {n: (count (concat s "!")) by: k from: A where: (> x 1)}) returns nyi. The new test uses a nested filtered source instead of where:.
• Mixed aggregate shapes still evaluate text concat eagerly per group and fail with the SYM + str type error: (+ (count (concat s "!")) (count (concat x [9]))) and (+ (count (concat s "!")) (count (distinct x))).

[singaraiona]

Fixed in 896af4d, including the related older failures reported in this review.

• Added the 20K-row sparse-key MAX/top-N repro. Reverting the matcher to NULL makes it fail with [99658 97257 89965] instead of [99985 99979 99973] at two workers.
• Added (sum x) beside a collection in both output orders. Removing if (has_collection) has_full = true; now fails that test.
• A real one-row input column, including (+ x 0), no longer broadcasts to a longer collection. It gets the same ragged-column error as larger inputs; scalar reductions still broadcast.
• A collection incompatible with a deferred WHERE selection now reports length: select: collection projection has 2 rows; where requires 3 input-aligned rows before sel_compact. This alignment requirement is documented and tested.
• Fixed the computed COUNT crash and concat COUNT nyi with direct where:. The filtered group compactor now skips COUNT inputs, which do not need stored columns. The direct compound text-count query works too.
• Mixed text/collection and text/distinct aggregate expressions now work, including direct WHERE and empty selections. The evaluator uses the same text-vector/scalar concat kernel as the planner; the collection primitive keeps its eager behavior outside query scope.
• Evaluator projections defer scalar broadcasting until collection lengths are known. The (enlist 7) examples work in both output orders and on empty sources. Scalar-only projections retain their source row count. distinct beside a scalar aggregate follows the same rule; updated the old error assertion and retained coverage for mismatched non-scalar columns.

Validation: 4,082/4,082 passed, 0 skipped, 0 failed (ASan/UBSan, eight workers), release build passed. The sparse top-N and aggregate-broadcast regressions were also checked by temporarily disabling their fixes, then restoring them before the final run.

[github-actions]

Rayforce targeted audit passed

The required Rayforce audit gate passed on the latest run.

Workflow run: https://github.com/RayforceDB/rayforce/actions/runs/38075476747

[singaraiona]

Fixed the audit's blocking enlist regression in f515b5f.

The whole-column special case is now limited to a single known scalar. Column references (including quoted column names), vector constants/bindings, and multi-argument enlist remain per-row constructors. Added value assertions for one- and two-column constructors, a constructor beside a passthrough column, count (enlist x), and constant/vector/scalar bindings. The (enlist 7) collection-broadcast cases still pass.

The alias scratch capacity is nc_max, computed from every output plus possible hidden sort keys; the scalar-flag loop writes one entry per output, within that capacity.

Validation: 4,114/4,114 passed, 0 skipped, 0 failed (ASan/UBSan, two workers, after syncing current dev in 37a6816); release build passed.

The earlier CI pool failure did not reproduce locally: its focused case passed 20/20 repeated runs and the full merged-base suite. The new required audit run has passed.

[ser-vasilich]

Rechecked 896af4d and 37a6816.

Fixed:

• The sparse top-N test, (sum x) beside a collection, and one-row tables no longer broadcasting a real column all hold.
• A collection with a deferred where: now gives the length: error, also on parted tables.
• The computed COUNT crash and concat COUNT nyi with direct where:. (count (upper s)) by: k where: gives [1 1 2]. The parted variant, which was a SIGSEGV on dev, works too.
• Skipping COUNT inputs in the compactor is safe. Direct where: matched a pre-filtered source in 14 shapes at 3k rows (-c 2) and 20M rows (-c 8). The shapes include nulls, computed inputs, count distinct, F64 keys, 2 keys, groups filtered to empty and no rows kept.
• f515b5f fixes the enlist regression in 896af4d. Column, quoted-name, vector, multi-argument, lambda-vector and global-vector enlist match 2ab29f1 across by:, where:, take:, empty and one-row tables.

BUG, still on 37a6816: inside any query, concat of a SYM/STR vector with a text atom now joins row by row. ray_concat_eval_fn takes the text kernel whenever a query scope is anywhere on the scope stack. That includes user lambdas, globals and literals, not only columns, so the same function returns different values inside a select than outside it:

(set T (table [x k s] (list [1 2 3 4 5] [0 0 1 1 2] ['a 'b 'c 'd 'e])))
(set allowed (fn [] (concat ['a 'b] 'c)))
(allowed)                                                  ;; [a b c] on both
(at (select {from: T where: (in s (allowed))}) 's)         ;; dev [a b c], 37a68163 []
(at (select {from: T where: (> (count (allowed)) 2)}) 's)  ;; dev [a b c d e], 37a68163 []
(at (update {n: (count (allowed)) from: T}) 'n)            ;; dev [3 3 3 3 3], 37a68163 [2 2 2 2 2]
(set L ['a 'b])
(at (select {n: (+ (count (concat L 'c)) (count (distinct x))) by: k from: T asc: k}) 'n)  ;; dev [5 5 4], 37a68163 [4 4 3]

The planner already does this on dev for literal vectors: (select {from: T where: (in s (concat ['a 'b] 'c))}) gives [] on dev and [a b c] on 2ab29f1. This PR extends it to lambdas, globals and per-row evaluation. Row-wise text concat probably needs to apply only when the vector argument is a column of the source (or derived from one), not whenever a query scope is active.

Tests. I reverted each hunk of 896af4d in turn. Every one makes a test fail except two:

• The atom-first branch (concat "x" s) in string.c: disabling it fails nothing (575/575 .rfl).
• The sparse top-N repro: only concat_sparse_topn.rfl catches a NULL matcher table. The copy at the end of concat_related.rfl passes with it.

Behaviour changes, intended as far as I can tell:

• A single scalar enlist ((enlist 7), a global or lambda-local atom, a string, 0Nl) now gives a typed broadcast column instead of a LIST of 1-element vectors. {e: (enlist 7) a: (distinct k)} now works. The type change is visible to users.
• Minor: {a: (distinct k) m: (max x) c: (+ m a)} now reports name: 'a' undefined instead of a length error.

Performance. Release build, 20M rows, -c 8, interleaved A/B:

• No regression on a four-aggregate by: with count distinct and desc: c take: 10, or on a two-key top-N. The top-N matcher fires on every head.
• No regression on membership, grouped aggregates, or count of text concat with and without where:.
• A 20M-row text-concat projection is 2–7% slower than dev (1297–1335 → 1341–1442 ms). The per-row loop is unchanged, so this may be code layout.

make test (ASan/UBSan): 896af4d 4082/4082, 37a6816 4114/4114. No sanitizer reports on the corpus or the probes.

[singaraiona]

Fixed in a1d3c2a.

concat is registered as the ordinary builtin again. The query evaluator selects the text kernel from the argument expression's column dependency, rather than the presence of any query scope. Literal/global collection operands keep their ordinary semantics. The planner now does the same for constant collections and declines lambda inlining when it would turn the function's concat into a query text operation. Existing scalar query text concat is preserved.

Your repros now return [a b c] for membership, all five rows for the count predicate, [3 3 3 3 3] for update, and [5 5 4] for the grouped global-vector case. concat_scope.rfl covers these, both operand orders, lexical shadowing, computed columns, function parameters, and the two-text-column fallback. It also checks nested variadic concat with the source column in the third argument.

Added forced per-group fallback cases for (concat "!" s) and its strlen result. Disabling atom-first text dispatch makes concat_scope.rfl fail at that case; restoring it passes. The sparse top-N regression remains in concat_sparse_topn.rfl.

Validation: make test -j8 TEST_CORES=2 4,115/4,115 passed, no skips or failures, under ASan/UBSan. Release build passed. Updated the docs and PR description to describe collection versus column text operands.

[singaraiona]

Follow-up on CI: synced #754 from current dev in 267d427, then addressed the two failing checks in separate commits.

• 167324d — read-ahead test reset. Linux debug failed at the cache reset (evict). The reset touched only pages reported resident, but Linux mincore reports a file page as present only once it is uptodate. That can skip an in-flight read just before eviction. The reset now faults every page before unmapping and evicting. Existing read-ahead assertions remain in place. Kernel implementation.
• d6970a7 — nested-select differential oracle. The captured seed uses 131,073 repeated rows of [DBL_MAX, -DBL_MAX, 3.2] and sums p + 1.3. Composition rejects the offset, and repeating the reference query itself with four workers returns both 0 and 0Nf. The fuzzer now tolerates differences only in F64 aggregate values/null flags when every layer declines composition and execution is parallel. Any composed layer keeps the strict existing oracle; other columns, schema and errors still compare exactly. Added the CI input as a committed seed.

Validation on the integrated branch: 4,119/4,119 ASan/UBSan tests passed, zero skips/failures; release build passed; all three read-ahead cases passed 30 repeated runs; captured seed passed 100 four-worker replays. Fresh 60-second differential fuzz campaigns passed in serial (1,830 inputs) and four-worker mode (801 inputs).

[ser-vasilich]

Rechecked a1d3c2a. In src/ it is identical to d6970a7. Our earlier repros now match dev ([a b c], [a b c d e], [3 3 3 3 3], [5 5 4]), and the literal planner case (in s (concat ['a 'b] 'c)) gives [a b c] (dev gives []). The commit also brings four wrong-result regressions, one of which now ends in an abort, plus a perf regression.

Setup for the repros below:

(set T (table [x k s t] (list [1 2 3 4 5] [0 0 1 1 2] ['a 'b 'c 'd 'e] (list "p" "q" "r" "s" "t"))))
(set sfx (fn [v] (concat v "!")))
(set sfs (fn [v] (concat v 'zz)))

1. Lambda helpers that contain concat and are applied to a column break. This comes from the new decline of lambda inlining. dev and 37a6816 agree on every row; disabling the named-lambda decline restores the dev results.

query	dev / 37a6816	a1d3c2a
(select {v: (sfx s) from: T})	[a! b! c! d! e!]	[['a !] ['b !] ...]
(select {v: (sfs s) from: T})	[azz bzz ...]	([a zz] [b zz] ...)
(count (sfx s)) by: k	[2 2 1]	domain: select by: failed to compile aggregation argument
(update {v: (sfs s) from: T})	[azz ...]	length: update: new column has 6 values for 5 rows
where: (in (sfs s) ['azz 'czz])	[1 3]	length: WHERE mask length 6 does not match row count 5
by: (sfs s)	table	failed to compile group key expression
(sfx t) on a STR column, 20M rows	0.84 s	same values, 15.7 s

The decline fires on any concat anywhere in a lambda body, numeric or constant included. Wrapping a constant collection in a lambda also changes the result: with h = (fn [v] (concat ['p 'q] 'r)), (h x) in a projection gives [[p q r] [p q r] [p q r]], while the inline form gives [p q r].

2. select and update now disagree. Update's interpreter paths bind columns with ray_env_set_query_local and never publish g_active_query_table, so ray_query_expr_reads_rows does not see the columns there. Affected: update_eval_on (query.c ~16243), the update where: fallback (~16900) and update_where_mask_vec (~16047).

expression	select	update on a1d3c2a	update on 37a6816
(+ (count (concat s 'zz)) (count (distinct x))) by: k	[4 4 2]	[5 5 5 5 3]	[4 4 4 4 2]
where: (> (+ (count (concat s 'zz)) (count (distinct x))) 10)	no rows	all rows	no rows
(+ (count (concat s "!")) (count (distinct x))), update	works	type error	[4 4 4 4 2]

3. A column concatenated with a bound vector takes the text kernel in the per-group fallback. The eval.c condition takes it when either side reads rows. concat_text_values then takes the row count from the first vector whose length is not 1 and fills the missing rows with element 0. So the result depends on group size and operand order. With L3 = ['u 'v 'w]:

expression	dev / 37a6816	a1d3c2a
(+ (count (concat s L3)) (count (distinct x))) by: k	[7 7 5]	[4 4 4]
same with (concat L3 s)	[7 7 5]	[5 5 4]
(concat s (take ['a] 0))	[4 4 2]	[4 4 1]

4. Abort.

(set ES (as 'STR (list)))
(at (select {n: (+ (sum (strlen (concat t ES))) (count (distinct x))) by: k from: T asc: k}) 'n)

dev and 37a6816 give [4 4 2]. a1d3c2a aborts on str.h:56 ray_str_t_ptr: Assertion pool_base != NULL, in the release build too. The kernel bug itself is older: dev also aborts on (select {v: (concat t ES) from: T}). a1d3c2a makes it reachable from the fallback through a bound collection.

5. Perf.

• The STR lambda helper projection: 0.84 → 15.7 s at 20M rows, and 85 → 1530 ms at 2M (item 1).
• The per-group fallback with a computed text argument compiles a graph per group. (+ (count (concat (upper s) "!")) (count (distinct x))) over 2M rows and 1M groups takes 2744/2783 ms on 37a6816 and 3216/3212 ms now (+15–17%).

Elsewhere there is no change: count of text concat by key, with and without where:, text-concat projections, arithmetic lambdas (they still inline), membership, grouped aggregates, top-N, and concat outside queries.

Tests. I reverted each hunk in turn. These revert without any in-tree test failing:

• the exec.c OP_CONST atoms_only check;
• the OP_CONCAT extra-argument traversal;
• the planner's !reads_rows && !all_atoms decline: without it, (concat (upper L) 'c) gives [Ac Bc];
• the is_whole_column_projection constants;
• the lexical-local check in reads_rows: without it, a lexical s in the fallback gives [4 4 3] instead of [5 5 4].

The variadic-third and lexical-shadowing tests are projections, so the planner handles them, not the evaluator code they are meant to cover.

make test (ASan/UBSan): 4115/4115.

It would be safer to keep inlining lambdas and decide the kernel after inlining, from whether each concat operand is derived from a source column, the same way in every path including update.

[ser-vasilich]

Follow-up on d6970a7. Its src/ is the same as a1d3c2a plus the dev merge, so my previous comment still applies. New since then:

#744 together with #754. I compared 113 nested queries merged against (do ...)-forced, by ser bytes and error parity. 96 of them composed at least one layer.

• They covered count-filter, top-N with ties and nulls, scalar broadcast, in/not-in on SYM/I64/GUID/F64 with nulls, probe sets from literal/global/lambda/concat, row-wise and grouped text concat, enlist and distinct.
• Sizes were 20k rows at -c 1/4/8, 2M at -c 8, and 10–20M at -c 8.
• All matched except the outer sort on an inner computed column, which is a perf(query): compose safe nested selects before planning #754 regression now on dev (reported on perf(query): compose safe nested selects before planning #754).
• Composed shapes stay fast: in over a composed 3-layer query takes 163 ms vs 479 ms on dev before perf(query): compose safe nested selects before planning #754. No fix(query): align membership, concat, and grouped fallback semantics #744 shape got slower.

The fuzz oracle relaxation in d6970a7:

• Classification is reliable. query_has_merge uses the same ray_select_merge_plan predicate as ray_select.
• It is unbounded. Any F64 difference in total (e.g. 196609.5 vs 0) and any null-flag difference is accepted. It is not bounded by something like n·eps·Σ|q|, nor limited to inputs that overflow, so a real race in an unmerged reduction would be masked too.
• It is incomplete. Composed queries run the same nondeterministic reduction, so the strict path will still flake. (select {from:(select {from:T a:a b:b k:k p:p s:s text:text}) total:(sum p)}) is inside the generator's grammar (131,073 rows, p from {DBL_MAX, -DBL_MAX, 3.2}). It composes, and at -c 4 it mismatched its (do ...) reference in 92 of 200 pairs (3.2 / 0.0 / 0Nf).

The underlying engine bug: a parallel F64 sum is nondeterministic. This is not from this PR; it reproduces on dev 57f5cfe.

(set P (take [1.7976931348623157e308 -1.7976931348623157e308 3.2] 131073))
(sum P)   ;; -c 4, 30 calls: 3.2, 0, 0Nf; -c 1: always 3.2
(set C (take [1e16 1.0 -1e16] 131073))
(sum C)   ;; -c 4: 0, 1, 2; -c 1: always 0

Two keyless paths keep one accumulator per worker while ray_pool_dispatch hands out tasks dynamically, and merge the partials in worker order:

• exec_reduction (par_reduce_fn → accs[worker_id]), which serves the sum/avg/prod/var builtins and lazy reductions;
• the keyless select scan in exec_group (scalar_*_fn → accums[worker_id]), with and without where:.

So each partial depends on which worker claimed which task. This is fixed in #758: partials are kept per chunk, with chunk boundaries derived from the row count, and folded in chunk order, which gives the same bits at every core count. With it, the select-merge oracle can stay strict instead of tolerating parallel F64 differences.

Smaller items:

• count of a constant collection still disagrees between paths (T with 5 rows, L ['p 'q]):
  • select {c:(count (concat L 'z))} gives [5], the row count;
  • update gives [3 3 3 3 3], {c:(+ 0 (count (concat L 'z)))} gives [3 3 3 3 3], and by: gives [3 3 3].
  • (count [1 2]) and (count L) in a select also give [5].
• The pf_settle change (167324d) is correct: faulting a locked page waits for its read, and MADV_WILLNEED is issued synchronously inside the task. It ran 25/25 under ASan. It makes scan_read_ahead about 1.5x slower (198 → 310 ms), because the mapping is MADV_RANDOM and each reset does thousands of synchronous 4 KiB faults. A buffered pread of the file, or MADV_NORMAL before faulting, would drain the same reads with batched I/O.

make test (ASan/UBSan) on d6970a7: 4119/4119.

[ser-vasilich]

One more concat shape. dev has the same behaviour, so it is not a regression. It falls squarely under the query concat rules this PR defines, so it would fit here:

(set T4 (table [x s] (list [1 2 3 4] (as 'SYMBOL ["a" "" "c" "d"]))))
(at (select {r: (concat s 'zz) from: T4}) 'r)                 ;; [azz zz czz dzz]
(at (update {r: (concat (first s) 'zz) by: x from: T4}) 'r)   ;; [azz null czz dzz]
(at (select {r: (concat (first s) 'zz) by: x from: T4}) 'r)   ;; error: domain: select by: output `r` did not evaluate to one value per group

The two problems, both on acaa50a and on dev 57f5cfe:

• The grouped update turns the null cell into null, while the ungrouped path reads it as empty text and gives zz.
• The grouped select over the same per-group atom does not produce one value per group.

Could you cover this in #744? I'd expect the grouped update and the grouped select to both give [azz zz czz dzz].

[singaraiona]

Covered in ae5d195. Both forms give [azz zz czz dzz], so does last, max and the literal on the left, and the STR column agrees. One related change from dev: a cell reduced from a STR column keeps STR output, as the column does ((concat (first s) "zz") over a STR column gave symbols before), so grouped select, grouped update and the ungrouped projection agree on the type too.

The rule that was missing: a text column reduced to one value (first, last, min, max) is a cell of the rows, so for concat a null there is empty text, as it is for the cell itself, and a literal joins it. Two places had to say so:

• The planner's concat kernel, which the grouped update runs per group: its text joiner takes a cell mask now (the same encoding the evaluator's two-operand kernel already used), and an operand produced by one of those reductions sets its bit, so the null symbol becomes zz instead of null.
• The grouped select's arithmetic-of-aggregates path, which evaluates the rewritten wrapper over the group result through the tree walker: it bound the hidden aggregate columns without row provenance and without an active table, so concat saw two collections and produced five values for four groups. The hidden columns carry the rows flag and the group result is the active table while the wrappers run.

first keeps its own meaning everywhere: over a two-row group ['' 'd] it is d on dev and here, in the engine and in the compound path alike, so a grouped (concat (first s) 'zz) over such a group is dzz; the empty cell shows only where the group's value is itself empty.

Tests. concat_helper_consistency.rfl adds the ungrouped projection, the grouped select with first, last, max and the literal on either side, the filtered source, the grouped update, the STR column for both, and (count s) keeping collection semantics (a number is not a cell of text), and the same shape through a two-statement helper (bytecode), a lambda written in query text (tree walker) and a builtin held in a local, which the evaluator's syntactic fallback already handled. The kernel mask, the STR output rule, the hidden-column flag and the active table each fail a case when reverted; I had also added an evaluator-side rule for the reductions and took it out again when no shape could tell it apart from the existing fallback.

Validation: make test under ASan/UBSan 4136/4136; strict docs build; functions and reference docs state the rule.

[ser-vasilich]

Rechecked 01870ab through ae5d195 (CI green). Release builds, compared with 8821daa, d698cbd and dev 57f5cfe.

Fixed

• Every item from my last comment. The closure first call and the compiled calls now agree. Helper keys passed as a parameter, local or captured give [3 3]. A mavg key gives [1 5]. The dict-dispatched effect gives CNT 6. A conditional let count in a plain lambda gives 3. Recursion with 10 captures reaches 109.
• F64 keys. The null flag on an F64 key matches its materialized reference.
• The null scan. It costs nothing again. I put a single surviving null at every block boundary, for every key type, at 1, 3 and 24 cores: no violations.
• Bug 9. (concat (first s) 'zz) gives [azz zz czz dzz] in grouped update and in grouped select.
• Closures from one template. Atom, vector, table, closure, string, dict and lazy captures each return their own value, also across 1000 closures with two capture sets and across ser/de after the template is gone.
• Closure performance. A closure called 1, 2 or 3 times takes 233 / 402 / 552 ms over 200k closures, against 1023 / 1201 / 1349 on d698cbd.

New in 01870ab. In each case 8821daa, d698cbd and dev give the right value.

1. A closure made after a let that did not run reads 0, silently, in ordinary code.

   (set scale 10)
   (set f (fn [xs flag] (if flag (let scale 2)) (map (fn [v] (* v scale)) xs)))
   (f [1 2 3] false)   ;; [0 0 0]   (expected [10 20 30])
   

   • compile_fn_template (compile.c ~891-900) captures every outer local the body mentions, including let slots that are still empty.
   • op_mkclosure (eval.c ~3277) skips an empty slot, and op_bindcaps (~3225) leaves the slot empty.
   • compile_lambda marks every capture slot as fixed (c.n_fixed = c.n_locals, compile.c ~997), so a read compiles to OP_LOADENV, which pushes 0 for an empty slot instead of falling back to the global.
   • The same happens with no global of that name (0 instead of a name error), in a loop where only a later iteration binds the name, after a let skipped by a caught raise, after a while that runs zero times, and in a query helper. (de (ser g)) gives the right value, so a closure and its round-tripped copy disagree.

2. eval inside a closure no longer sees its captures. This has been the case since d698cbd.

   (set mk (fn [y] (fn [] (eval (parse "y")))))
   (set g (mk 7))
   (g)   ;; error (dev: 7); with a global y = 100 it silently gives 100
   

   eval is a plain builtin compiled with no scope window, so it cannot see the slots the dynamic flag keeps.

3. Deeply nested literals crash compilation. An outer literal now compiles every nested literal at once. Each level puts a compiler_t (about 4 KB) on the C stack, with no depth guard. A literal 2000 levels deep, built at run time, crashes with SIGSEGV; 8821daa handles depth 20000. Typed text is capped at 1024 levels, so only runtime-built or deserialized ASTs reach it.

   (set base (parse "(fn [] k)"))
   (set wrap (fn [inner] (concat (take base 2) (enlist inner))))
   (set build (fn [n] (let a (parse "k")) (times n (let a (wrap a))) a))
   ((eval (concat (take (parse "(fn [k] 0)") 2) (enlist (build 2000)))) 7)
   

4. One-shot closures made by interpreted code got 2-3x slower. Compile-on-second-call is gone, so every such closure compiles on its first call. Median ms:

   shape	dev	8821daa	ae5d195
   one-shot closure in a tree-walked fn, 100k	182	196	369
   do/let one-shot, 100k	76	88	242
   fn literal inside or in a compiled fn, 200k	314	250	531

   Each such closure also keeps about 832 B of private bytecode.

5. A compiled maker no longer captures a tree-walked caller's locals. This change is arguably more lexical, but it differs from dev, and the result now depends on whether the maker compiles.

   (set y 1)
   (set mk (fn [] (fn [] y)))
   ((do (let y 5) (mk)))   ;; 1   (dev, d698cbd8, 8821daa1: 5)
   

Purity check: an effect through a parameter runs twice. key_literal_pure / key_lambda_pure never record parameters, so a parameter inherits the purity of a let-literal or global that shares its name:

(set T10 (table [v] (list [1 2 3 4 5 6])))
(set CNT 0)
(set eff (fn [x] (set CNT (+ CNT 1)) x))
(set H (dict ['a] (list eff)))
(set h (fn [x] x))
(set ky (fn [v] 1 (let g (fn [h] (h v))) (g (at H 'a)) (if (< v 4) 0 1)))
(select {n: (count v) by: (ky v) from: T10}) CNT   ;; 7   (8821daa1: 6)

Tests. These hunks revert without any test failing:

• Null scan: a scan that skips the last element of each 4096 block. A sweep finds 88 null keys with the flag cleared.
• Lazy captures: env.c keeping a lazy capture. A case from Rayfall does reach it: (set c3 ((fn [v] (fn [] (if false (set 1 2)) (at v 0))) (asc [3 1 2]))) then (list (c3) (c3)) now gives [1 1], while dev and 8821daa raise an error. The op_mkclosure comment saying Rayfall cannot reach a lazy slot is also out of date.

Regression runs (2×24-thread box). 8821daa and acaa50a each did the load plus 3 rounds of 43 queries. Both are within noise of dev, and the only output difference from v2.11.0 is the requested column order of the 4-key group.

[singaraiona]

Fixed in efcc705 and c5e4742.

1. Unbound capture. Capture slots are no longer marked as always bound: a capture whose let had not run is an empty slot, and a read of it goes to the global or raises a name error (OP_LOADENVG), as for any let slot. (f [1 2 3] false) is [10 20 30], (f2 false) is a name error, and a closure agrees with its ser/de copy.
2. eval in a closure. eval and resolve compile inside a scope window over the frame's locals, captures included, the way a dynamic special form does: (g) is 7 with or without a global y.
3. Deep literals. Template compilation nests at most 32 levels; a deeper literal takes the dynamic path, whose closures compile one level per call. The 2000-level build returns a lambda.
4. One-shot closures made by interpreted code. They share a template per literal and capture set too: ray_fn keeps a per-thread table of 256 templates keyed by the literal's parameter and body nodes (retained, so an address stays the entry's own while it lives) and its capture names; a hit shares the bytecode, a miss compiles once. Two capture sets from one literal get two templates and keep their values; a second closure adds about 360 bytes instead of 630. On this box (-c 1, medians), the literal inside or in a compiled function went from 333 to 136 ms for 200k; the one-shot closure in a tree-walked function and the do/let one-shot were already at 29 and 22 ms for 100k here (dev: 57 for the first, a name error for the second) and stay there, so I could not reproduce your 2-3x on those two shapes, but the mechanism that produced it, a private compile per closure, is gone for all three.
5. Lexical capture. The tree walker's lambda call marks its frames as the capture boundary, so a closure made inside a body captures that body's bindings and never a caller's locals, whether the maker compiled or not: ((do (let y 5) (mk))) is 1 on both paths, and the docs say so. It is a change from dev's dynamic capture, which only ever applied to interpreted makers.
6. Purity: effects through parameters. The walk binds a callee's parameters to the call's argument text: a parameter called inside the body means the argument it was bound to (a name is followed outside the body, a literal is checked, a computed callee is an effect), so ky gives CNT 6, and a pure helper passed as a parameter still keeps the whole column.

Hunks. The null scan's last-element case is pinned by a key column whose single surviving null lands on slot 4095. The interpreted lazy capture is pinned by your c3 shape, and the op_mkclosure comment no longer claims Rayfall cannot reach a lazy slot; it says what the step is for.

Tests. closure_locals.rfl: the unbound capture (global present, absent, and after the let ran), eval and resolve in closures, lexical capture with a compiled and an interpreted maker, the 2000-level literal, the Rayfall lazy capture, and two capture sets from one interpreted literal with the sharing bound. helper_bytecode_provenance.rfl: the effect through a parameter and a pure helper through a parameter. group_key_null_flag.rfl: the block-boundary null. Each hunk reverts to a failing test.

Validation: make test under ASan/UBSan 4136/4136; strict docs build.

[singaraiona]

Audit finding addressed in 840261e.

• Blocking 1. The table entry keeps every body node (literals with more than eight forms are not cached and compile when called, as before), and a hit requires every node to match, so two literals built for eval that share their parameter, first and last nodes but differ in between get their own templates. The regression test builds the two literals from one parsed form, splicing a different set into the middle (the quoted-list form in your sketch does not evaluate as a lambda body; parsed pieces share nodes the same way): on the previous head the second closure ran the first one's bytecode and gev stayed 1, now it is 1, 2, 1.
• Stale entries on another thread. When the symbol epoch has moved under a thread's table, its objects belonged to a runtime whose heap is gone, so the entries are forgotten, not released; only the destroying thread releases, in ray_lang_destroy.
• Shape check. ray_fn now calls the compiler's own gate (ray_closure_shape_ok) instead of restating it.
• Compile at creation. Noted: a closure that is never called now costs one compile per literal and capture set rather than none, in exchange for no compile per closure that is called; the table is bounded (256 entries, round-robin), a miss just compiles. Debug spans of an eval-built literal come from the shared template; its nodes are shared too, so they point at the same text.

Validation: make test under ASan/UBSan 4136/4136.

[singaraiona]

Follow-up in 1f2a0e2, on the audit's note that compilation moved to creation: a closure made by an interpreted maker and never called cost 249 ms per 200k on 840261e against 193 on ae5d195 (dev: 121). A template entry now remembers the symbols its literal mentions, so a hit costs one pass over the dict keys instead of a walk of the body and allocates nothing, and the capture walk builds its row-flag vector only when a binding is flagged. The same shape takes 197 ms now, and the fn literal inside or went from 136 to 98 ms per 200k. Validation: make test under ASan/UBSan 4136/4136.

[ser-vasilich]

Rechecked efcc705 through 1f2a0e2 (CI green). Release builds, compared with ae5d195, d698cbd and dev 24c14e8.

Fixed

• Everything from my last comment, except the compiled maker that no longer captures a tree-walked caller's locals, which you kept as lexical scoping. A closure made after a let that did not run gives [10 20 30], eval in a closure sees its captures, a 2000-deep literal compiles, and the effect through a parameter runs 6 times.
• Closure performance. Tree-walked closure shapes are now faster than dev (142 vs 178 ms, 62 vs 74, 191 vs 300). Closures called 1, 2 or 3 times take 232 / 395 / 533 ms over 200k closures, against 1010 / 1178 / 1335 on d698cbd, and closures from one literal share their bytecode.

New. In each case ae5d195 and dev give the right value unless noted.

1. eval/resolve opens its window before its arguments are evaluated (efcc705). compile.c ~642-656 emits OP_SCOPE_BEGIN before it compiles the head and the arguments (OP_SCOPE_END at ~702-705), so argument code runs inside the window. op_ret (eval.c ~3260) does not unwind windows; only vm_error_cleanup does.

   • return in an argument leaks the callee's frame for the rest of the session:

     (set p 100)
     (set rd (fn [] p))
     (set g (fn [p] (eval (if (> p 0) (return 5) (quote p)))))
     (list (g 1) p (rd))   ;; 5 1 1     (dev: 5 100 100)
     (set p 200)
     (list p (rd))         ;; 1 1       (dev: 200 200)
     

     After about 60 more such calls the scope stack is full, and every later scope-using form, such as (do (let a 1) a), fails with limit.
   • A let inside an argument is reverted at OP_SCOPE_END: (set f2 (fn [x] (eval (do (let x 10) 0)) x)) (f2 1) gives 1 (dev: 10).
   • A function called from an argument sees the caller's locals: (set x 100) (set rx (fn [] x)) (set f (fn [x] (eval (rx)))) (f 1) gives 1 (dev: 100).

   Opening the window after the arguments, just before the call, and unwinding windows in OP_RET as the error path does would cover all three.

2. A shared template keeps the builtin a name was bound to when the template was compiled (c5e4742). compile.c ~596 resolves the head with ray_env_get, and ~661-663 emit LOADCONST of that builtin. closure_share_template (eval.c ~2233) keys only on the literal's nodes and capture names. A closure created after the name is rebound still calls the old builtin:

   (set g sum)
   (set r [])
   (times 2 (do (set k (fn [x] (g x))) (set r (concat r (k [1 2 3]))) (set g count)))
   r   ;; [6 6]   (dev: [6 3])
   

   • ray_env_get also walks dynamic frames, so the first maker call bakes the caller's local into the template for every later closure from that literal:

     (set g count)
     (set mk (fn [] (if false (set 1 2)) (fn [x] (g x))))
     (set c1 (do (let g sum) (mk)))
     (set c2 (mk))
     (list (c1 [1 2 3]) (c2 [1 2 3]) ((mk) [1 2 3]))   ;; 6 6 6   (dev: 6 3 3)
     

   • Compiled makers have done the same since acaa50a: (set mk (fn [] (fn [x] (g x)))), with g rebound from sum to count between two (mk) calls, gives 6 6 (d698cbd and dev: 6 3).

   A named lambda that has already been called keeps the old builtin on dev too, but here the closure is created after the rebinding.

3. The template table keeps up to 256 literals alive after their closures are gone.

   (set base (parse "(fn [x] 0)"))
   (set big (fn [i] (eval (concat (take base 2) (enlist (concat (parse "(+ x)") (enlist (+ i (til 100000)))))))))
   (set bs (map big (til 300)))
   (set bs 0)
   (.sys.gc)
   (at (.sys.mem) 'bytes-allocated)   ;; 268 MB   (dev: 36 KB)
   

   Small literals hold about 1 KB per entry. The memory comes back only on eviction or runtime destroy, and the table's extra reference makes alter copy any vector a literal shares.

4. Creating a closure from a 50,000-deep literal crashes. ast_collect_syms (compile.c ~816), called from ray_closure_mentioned_syms on every template miss, recurses with no depth guard:

   (set base (parse "(+ 1 0)"))
   (set wrap (fn [inner] (concat (take base 2) (enlist inner))))
   (set build (fn [n] (let a (parse "x")) (times n (let a (wrap a))) a))
   (set L (concat (take (parse "(fn [x] 0)") 2) (enlist (build 50000))))
   (try ((eval L) 1) (fn [e] e))   ;; SIGSEGV in (eval L)   (dev: no crash)
   

5. Opaque by: keys: the purity walk mishandles parameters. Dev rejects these keys, so this is all in the new path. Setup:

   (set T (table [v] (list [1 2 3 4 5 6])))
   (set CNT 0)
   (set eff (fn [x] (set CNT (+ CNT 1)) x))
   (set H (dict ['a] (list eff)))
   

   • A lambda passed as a value keeps its parameters unbound (key_lambda_call_pure with args == NULL, query.c ~2258), so an unrelated global with a parameter's name decides purity:

     (set ap (fn [z g] (g z)))
     (set kf (fn [v] (fold ap (> v 3) (list (at H 'a)))))
     (select {n: (count v) by: (kf v) from: T}) CNT   ;; 6
     (set g (fn [x] x))
     (set CNT 0)
     (select {n: (count v) by: (kf v) from: T}) CNT   ;; 1, expected 6
     

   • A literal nested in a body drops the enclosing body's parameters (query.c ~2277): (set ap2 (fn [g z] ((fn [w] (g w)) z))) used as by: (ap2 (at H 'a) (> v 3)) gives CNT 6, then 1 once an unrelated pure global g exists.
   • Argument names are resolved in the callee's scope (outer = *ctx, query.c ~2308):

     (set hh (fn [x] (> (- x (avg x)) 0)))
     (set ap3 (fn [f z] (let hh (fn [q] (println "never") q)) (f z)))
     (at (select {n: (count v) by: (ap3 hh v) from: T}) 'n)   ;; [6], expected [3 3] as (hh v) gives
     

6. Closures whose body cannot compile are compiled twice per creation. closure_share_template (eval.c ~2271) does not remember a failed compile, and ray_compile tries again at the first call. 100k closures with (if false (set 1 2)) in the body, made by a tree-walked maker: 495 ms vs 315 on dev with a one-statement body, 785 vs 480 with five statements.

7. A grouped multi-statement update is 17% slower than dev. (update {v: (+ (sum (hn (+ x 1))) (count (distinct x))) by: k from: T}) with hn = (fn [v] 1 (* v 2)), 2M rows and 1M groups, -c 8: 7.0 s vs 5.97 s on dev (d698cbd: 6.8 s). The other per-row and grouped helper shapes are 35-50% faster than dev.

Tests. 1f2a0e2 adds no tests, and these revert without a failure:

• Template matching that ignores capture names. The existing test creates the larger capture set first; creating dd2 before dd1 gives [1 'none].
• Dropping the row flag for every binding after the first, or never building the flags from the stack buffer. A flagged column captured after 1, 21 or 301 unflagged bindings then gives [[a x] …] instead of [ax bx ax cx].
• In the purity walk: binding the parameters of a called lambda value, and binding unknown parameters for uncalled literals.

Regression runs (2×24-thread box). ae5d195 and 1f2a0e2 each did the load plus 3 rounds of 43 queries. Both are within noise of dev, and the only output difference from v2.11.0 is the requested column order of the 4-key group.

[singaraiona]

Fixed in b32d913 and b21830b.

1. eval/resolve window. The window opens after the arguments, right before the call, and closes right after it, so argument code runs in the frame: (g 1) leaves p and rd alone ([5 100 100], then [200 200]), (f2 1) is 10, (f 1) is 100. A return inside the evaluated text itself still arrives as the return marker through the error path, which already unwinds windows, so no OP_RET change was needed; I had added one and took it out again when no shape could reach it. 200 such calls leave (do (let a 1) a) working.
2. Baked builtins in templates. A closure body (a literal written inside a function) now resolves the names it does not capture when it runs (OP_RESOLVE, through the VM's general call, which now applies the same elementwise map, concat rule and provenance join as the dedicated opcodes), and the compile-time lookup that classifies a head is global-only, so neither a later rebinding nor a caller's frame is frozen into a template: the compiled maker gives [3 3]. A literal written at the top level is a definition and keeps baking its builtins as dev's named lambdas do; its shared template is recompiled once a function-valued global has been rebound (a generation counter in the global binding), so r is [6 3]. With lexical capture the (do (let g sum) (mk)) closure sees the global count when it runs, so that case gives [3 3 3] where dev's dynamic capture gave 6 3 3; a lambda made with nothing captured keeps the builtins it names baked, as dev's named lambdas do. The docs say both.
3. The table and large literals. A literal whose text holds more than 65536 vector elements is not cached (the walk that collects a literal's symbols counts them), so the 300 large literals leave a few hundred KB behind instead of 268 MB; the test pins a 4 MB bound.
4. 50000-deep literal. The symbol walk stops at 1024 levels and the compiler at 2048; past that the literal has no template and, when called, is left to the tree walker rather than the C stack. The shape runs and returns 0 as on dev. The regression test runs at 12000 levels, the depth the sanitizer build's recursive release survives; without the compiler's guard it crashes there. The symbol walk's guard only matters deeper than that, where the sanitizer build cannot go, so that one hunk is pinned by your 50000-level shape on a release build only.
5. Purity walk. A lambda passed as a value binds its parameters as unknown (a call through one is an effect): CNT is 6 both times. A literal nested in a body inherits the body's parameters: 6. An argument's name is resolved where the call was written, in the caller's context: (ap3 hh v) gives [3 3].
6. Failed compiles. A table entry remembers that a literal did not compile, and a lambda that did not compile is marked (NLOCALS -1) so no call retries; each such body compiles at most once per literal.
7. Grouped multi-statement update. Two costs per group were the same for every group: the helper expansion was redone for each (the expansion is text over the schema, so it is made once per update now), and the planner declined the same expression once per group (a graph built and compiled 1M times; a flag of the grouped call skips it after the first decline, kept per call rather than per remembered expression, since a reused address must not inherit it). Your shape at 2M rows, 1M groups, -c 8: 2142 ms on dev, 2820 on 1f2a0e2, 1945 now; of the three variants, the helper-only one was the slow one and is now level with dev, the one without a helper is 888 against dev's 1229, the one-statement helper 1783 against 2450.

Hunks. Template matching by capture names is pinned with the smaller capture set created first; a row-flagged column captured after 1 and after 21 unflagged bindings keeps its flag; the purity walk's parameter binding for called lambda values and unknown parameters for uncalled ones are each pinned by the shapes above.

Tests. closure_locals.rfl: the window cases including the 200-call stack check, the three rebinding shapes, the large-literal memory bound, the 50000-deep literal, the reversed capture-set order. helper_bytecode_provenance.rfl: the three parameter cases. concat_helper_consistency.rfl: the flagged capture after 1 and 21 bindings. Each hunk of this round reverts to a failing test, with the one exception noted under item 4.

Validation: make test under ASan/UBSan 4136/4136; strict docs build. Closure calls pay for resolving their builtins at run time: (times 1000000 (cadd 2)) 146 to 152 ms, map over a closure 86 to 93, made and called once 69 to 76 per 200k, twice 113 to 123; a lambda made with nothing captured still bakes, so the plain compiled call stays at 55.

[singaraiona]

Follow-up in 5bf922d: the macOS debug job overflowed its stack releasing the 12000-level literal my regression test built (the Linux sanitizer build survives that depth, the macOS one does not). The compiler's depth guard is unchanged; the test now pins it by result instead of by a crash. The body is a chain of nested ifs: a compiled body runs it as jumps and returns the parameter, the tree walker recurses per level and stops at its depth limit of 512. So 600 and 1500 levels return 1 (compiled, with and without a shared template), and 2500 levels, past the 2048 guard yet within the bytecode's jump range, is a limit error. With the guard removed the 2500 case compiles and returns 1, so the test fails. The syntax page now states the limit. Validation: make test under ASan/UBSan 4136/4136, mkdocs build --strict clean (also fixed the control-flow page's anchor to the lambdas section).

Aside, pre-existing on dev: a chain of more than about 60 nested do forms on the tree walker reports oom (the scope stack is full) where limit would be the honest error. Not touched here.

[singaraiona]

Audit finding addressed in 7c1d2d2.

Undefined shift in the concat kernel. Confirmed: the cell flags were a 64-bit mask, read for every operand while written for the first 64, so operand 64 shifted by 64. The flags are now one byte per operand for all 255 positions, in the planner's node and in the two-operand text path alike. Reverting the kernel hunk makes the new test abort under UBSan with shift exponent 64 is too large.

The audit's example was not reachable as written: the planner folded a query concat into one node only up to 16 operands and left anything longer to the binary builtin, which fails with arity: expected 2 args, got 17 (on dev too). The planner now takes the executor's 255, so one limit governs the form. The new tests run 65-operand concats on the planned path: a first cell at position 0 with a null literal at position 64 nulls the row; a cell at position 64 is still a cell (its empty value is empty text, over the rows where: keeps as well); a STR cell there still makes the result STR; and a 65-operand concat over a column joins every row. The reference page states the operand limit. Validation: make test under ASan/UBSan 4136/4136, mkdocs build --strict clean.

On the non-blocking observation about the template cache not keying on where a literal was compiled: the compile mode follows the closure's presence (a body with captures resolves its free names at run time, one without keeps the builtins baked), and the capture names are part of the key, so a literal cannot match an entry compiled in the other mode.

[ser-vasilich]

Rechecked b32d913 through 7c1d2d2 (CI green). Release builds, compared with 1f2a0e2 and dev 24c14e8; make test under ASan/UBSan 4136/4136.

Fixed

• Every item from my last comment, as described.
  • The eval window: 5 100 100, then 200 200; (f2 1) is 10, (f 1) is 100, and 200 calls leave (do (let a 1) a) working.
  • The rebinding shapes: [6 3], 3 3 3 and 6 3.
  • The 50000-deep literal returns 0.
  • All three parameter shapes in the purity walk: CNT 6/6, 6/6 and [3 3].
• Failed compiles are compiled once: 128 / 278 ms against 322 / 493 on dev.
• The grouped update is faster than dev: 5.3 s vs 6.2 s, with values identical to dev.
• Concat plans 17–255 operands correctly. 256 operands give the same arity error as dev above 16.
• Builtins called through the run-time path behave like the baked ones, with one exception (item 3 below). This holds for 120 builtins × 17 argument kinds, and for 240 query-helper shapes compared between closures and named helpers.

New. dev gives the expected value unless noted.

1. A closure body nested 1024–2046 levels deep silently loses its captures (b32d913). Past depth 1024, ast_collect_syms_at gives up and closure_capture_names returns NULL. ray_compile reads that NULL as "nothing captured" and compiles the body with no OP_BINDCAPS, so every captured name resolves as a global.

   (set wrap (fn [inner] (concat (take (parse "(if true 0)") 2) (enlist inner))))
   (set build (fn [n] (let a (parse "(+ k x)")) (times n (let a (wrap a))) a))
   (set M (eval (concat (take (parse "(fn [k] 0)") 2) (enlist (concat (take (parse "(fn [x] 0)") 2) (enlist (build 1100)))))))
   (set k 5000)
   ((M 100) 1)   ;; 5001   (1f2a0e25: 101; dev: limit). Without a global k it is a name error.
   

   Depths 1022–1023 give 101, depths 1024–2046 give 5001, and from 2047 on the result is limit. Treating a failed walk as not compilable (NLOCALS -1) would make this limit. No test covers the 1024 guard.

2. A grouped update reads a global once for all groups. This comes from making the expansion once per update. If an effectful helper in the same expression changes the global between groups, the inlined pure helper no longer sees the change:

   (set T (table [x k] (list (til 20) (% (til 20) 7))))
   (set G 0)
   (set bump (fn [v] (set G (+ G 100)) 0))
   (set hg (fn [v] (+ (sum v) G)))
   (at (update {r: (+ (bump x) (hg x)) by: k from: T}) 'r)
   ;; [21 24 27 30 33 36 19 ...]   (dev: [121 224 327 430 533 636 719 ...]; 1f2a0e25 was one group behind)
   

   1f2a0e2 already froze the global this way for an ungrouped update and for a grouped select; this range extends it to grouped updates. Reverting to per-group expansion fails no test.

3. A closure body calls a unary builtin on the generic null where baked code raises a type error. The OP_CALLF unary path lacks the RAY_IS_NULL(arg) guard that OP_CALL1 and the tree walker have.

   (set mk (fn [k] (fn [v] (count v))))
   (try ((mk 0) null) (fn [e] 'type_error))   ;; 1   (dev and 1f2a0e25: 'type_error)
   

   It affects 28 unary builtins, among them neg, abs, not, sqrt, count, sum, first, last, reverse, min, max, avg, asc, desc and strlen.

4. Closure bodies that use several builtins got slower. Every builtin in a closure body now goes through OP_RESOLVE, which walks the scope frames and then scans the globals linearly. Median ms, 7c1d2d2 vs 1f2a0e2:

   • a closure with 4 builtins mapped over 1M: 593 vs 420 (+41%);
   • a closure while loop over 1M: 474 vs 390 (+22%);
   • a 5-statement closure called once, 200k closures: 291 vs 248 (+17%).

   Simple closure calls are still faster than dev: (times 1000000 (cadd 2)) takes 322 ms against 378 on dev.

5. The literal cache bounds vector elements, not bytes. A string atom constant counts 0, so 300 literals each holding a 1.3 MB string keep 537 MB after their closures are gone (dev: 1.5 KB). Literals just under the 65536-element bound can pin about 128 MB in total.

6. Two rebinding cases still bake the old builtin. Both happen on 1f2a0e2 as well.

   • The literal is baked when it is evaluated, not on its first call: (set g sum) (set f (fn [x] (g x))) (set g count) (f [1 2 3]) gives 6 (dev: 3, because dev compiles on the first call).
   • A function held in a dict and called by dotted name: (times 2 (do (set r (concat r ((fn [x] (D.f x)) [1 2 3]))) (set D (dict ['f] (list count))))) gives [6 6] (dev: [6 3]), and so does (set D.f count).

   Removing either generation bump alone, on rebind or on insert, fails no test.

7. A pure helper reached through a let alias is treated as effectful, so the key runs per row. This is not new; dev rejects such a key.

   (set T (table [v] (list [1 2 3 4 5 6])))
   (set hk (fn [x] (> (- x (avg x)) 0)))
   (set kl (fn [v] (let g hk) (g v)))
   (at (select {n: (count v) by: (kl v) from: T}) 'n)   ;; [6], expected [3 3] as (hk v) gives
   

   At 1M rows this key takes 754 ms against 5 ms for the direct key. map or fold over a list of pure helpers is treated the same way.

8. A named function called from inside evaluated text still sees the caller's locals. (set p 100) (set rd (fn [] p)) ((fn [p] (eval (quote (rd)))) 1) gives 1 (dev: 100). The argument case is fixed; this is the same leak through the evaluated text.

Tests. These revert without any test failing:

• the head lookup being global-only;
• the 1024 symbol-walk guard;
• ray_compile marking NLOCALS -1, and the -1 check in lambda_wants_compile;
• the failed-template memo;
• either generation bump alone;
• the OP_CALLF binary result-rows flag;
• expanding the grouped update once;
• the per-call dag_declined flag (the reused-address case it fixes has no test).

Regression run (2×24-thread box). 7c1d2d2 did the load plus 3 rounds of 43 queries. It is within noise of dev, and the only output difference from v2.11.0 is the requested column order of the 4-key group.

[ser-vasilich]

Names in queries: a column vs a local or a global. This is not a defect in this PR, but #744 sets part of the rule, so it is better decided before it merges. With T = (table [x y] (list [3 1 4 2] [30 10 40 20])):

A parameter vs a column of the same name. Most paths let the parameter win; several let the column win:

(do (let y 5) (count (select {from: T where: (> y 15)})))            ;; 3: the count fast path reads the column
(do (let y 5) (count (at (select {from: T where: (> y 15)}) 'x)))    ;; 0: the materialised select reads the local
((fn [y] (select {c: (sum x) by: {y: y} from: T})) [10 10 30 30])    ;; 4 groups: the column
((fn [y] (select {c: (sum x) by: {k: y} from: T})) [10 10 30 30])    ;; 2 groups: the local

The column also wins in by: [y], asc: y and update … by: y, and after an earlier (let y …) inside a do in the query (the planner drops the let).

Helper bodies (this PR). A free name in a helper now resolves to the calling query's source column before the global, on the inlined and the bytecode paths alike. Around 25 assertions pin this, e.g. the suffix tests. So the same helper means different things depending on the table:

(set y 5)
(set hy (fn [a] (+ a y)))
(select {s: (hy x) from: T})                             ;; [33 11 44 22]: the column y
(select {s: (hy x) from: (table [x] (list [3 1 4 2]))})  ;; [8 6 9 7]: the global y

This is dynamic scope for columns: adding a column to a table changes what an existing helper computes, and an opaque helper may read any column, which blocks projection pushdown. In q, a function never sees the calling select's columns or the caller's locals.

A proposal for one rule:

• Helpers and closures are lexical. They see their parameters, their own lets, their captures and globals. Columns reach them only as arguments.
• In the query text, a name that is both a column and an outside local is an error, as in PL/pgSQL's default since 9.0: "'y is both a column and a local; write 'y for the column or rename the local". Column over global stays as it is.
• One place decides. A single name-binding pass at query entry, with the schema known, would decide this for every path (planner, eval fallback, count fast path, by-dict, update by:). Today each path re-decides.

How would you like to settle the helper rule for this PR? Keeping it means users start relying on column visibility in helpers. Dropping it now would leave the rest for a follow-up after 2.11.1, and we can take that implementation.

[singaraiona]

Decided, after checking every case above on this branch, on dev and on the pre-PR base: a column and a variable are the same kind of binding, and the binding made last wins. A query binds its source columns when it runs, so inside the query text a column shadows an outer local and a global of its name. A helper called from the query binds its parameters and lets later still, so those shadow the columns, and its other free names resolve to the column before the global. That is the rule this PR implements for helpers, on the inlined and the bytecode paths alike (the inlined path already behaved this way on dev; the PR made the bytecode path agree), and it stays. Columns are not dynamic scope in disguise here: they are the innermost bindings of the query, as a let just inside it would be.

Consequences for the list above. The paths where the local wins today, the materialised where: after an outer let and by: {k: y} with a parameter y, are the inconsistent ones; they are identical on dev and on the pre-PR base, so this PR neither introduced nor touches them. The "both a column and a local is an error" rule is not adopted: shadowing resolves the name. A single name-binding pass at query entry is the right place to make every path agree, and I will take it, with those two paths, in a follow-up after 2.11.1 unless you want them in this PR.

[ser-vasilich]

Agreed that the binder goes after 2.11.1. One correction before it is planned. Today the local wins on most paths, not only the two above. A plain projection already reads the parameter:

((fn [y] (select {s: (sum y) from: T})) 5)   ;; [20] on dev and on this branch

The same holds for where:, by: y, computed keys and update values. Several tests pin this as deliberate:

• test/rfl/regress/issue_394.rfl:68-73: "Lambda parameters and let bindings take precedence over query columns".
• query/eval_fallback_sym_passthrough.rfl:42-44.
• query/concat_scope.rfl:21-22, added in this PR: "Lexical bindings win over a same-named source column".
• query/concat_operands.rfl:34-35.

The column wins only on the count fast path, by: {y: y}, by: [y], asc:/desc: and update … by:. So "the column shadows an outer local" flips the common case, and those tests with it.

For #772, the no-output by: y follows the current majority (the local), the same as the with-output form. That way the binder can switch every path at once.

[singaraiona]

Correction taken, and confirmed on this branch and on dev: a projection, where:, by: y, computed keys and update values read the parameter today, the tests you list pin it (the one in concat_scope.rfl came with this PR because that is what the engine does on every path it covers), and the column wins only on the count fast path, by: {y: y}, by: [y], asc:/desc: and update … by:. So the rule stated above is the target, not a description of today: the binder flips the common case, and those tests with it, in one change after 2.11.1. This PR leaves every one of those paths as it found it, and the no-output by: y in #772 following the current majority is right, for the same reason: one switch, every path at once.