feat(hotblocks): shadow queries with the new query engine

[mo4islona]

hotblocks: shadow queries with the new query engine

A sampled share of queries (--shadow-percent) is re-run chunk by chunk by both engines in the background, on the snapshot and block bounds of the response. Clients never wait for it and never see its result.

The new engine lives in query_engine, apart from the shadow comparison, so the main query path can take it over later. It reads its metadata catalogs (evm.yaml, solana.yaml, ...) from --metadata-dir; the files must come from the pinned engine revision, and strict parsing refuses keys that revision does not know. Without the directory the new engine is off and nothing is shadowed, whatever the share.

Both engines are re-run, in random order, so the timings are taken under the same conditions. Parse and compile stay outside the clock: a response compiles once and reads many chunks. Responses are compared as values, because field order, string escaping and float spelling differ by design. The engines weigh blocks differently, so a large answer can be cut at different blocks; the new engine then answers again for exactly the legacy blocks, without a weight budget, so that answers and timings cover the same work.

What it reports:

• hotblocks_shadow_runs_total{outcome}: match, mismatch, legacy_error, new_error, both_error, skipped.
• hotblocks_shadow_slow_runs_total, hotblocks_shadow_run_duration_seconds{engine}, hotblocks_shadow_percent, hotblocks_shadow_in_flight.
• An error log per value difference or one-sided failure, and a warning per slow run (--shadow-slow-ratio, --shadow-slow-margin-ms), with the first differing block, JSON path, both values and the query narrowed to the chunk's bounds. At most --shadow-log-limit per 5 minutes (default 20; 0 turns them off), one per query shape. The metrics count every run either way.

At most --shadow-max-in-flight runs at once (default 2), on as many threads of their own, so they never queue ahead of a client's query; the legacy engine's polars operations still use the shared pool. A chunk that finds no free slot is skipped, so shadow work never queues and pins few snapshots. Off by default.

tests/shadow_engine.rs runs the binary at 100% over EVM and Solana storage, with copies of the pinned engine revision's EVM and Solana catalogs in tests/metadata, waits until no run is in flight, and requires every run to match. To show that the shadow reports a disagreement, other tests edit one line of the EVM catalog so that the new engine answers differently, fails, or cuts every answer after one block; they require the mismatch and the error to be counted and logged, the cut answers to be compared on the same blocks even when the query selects no block numbers, and a log limit of 0 to silence the logs but not the counts. Another test checks that nothing is shadowed without catalogs.

The engine is pinned to subsquid/sqd-query-engine@9525538 on its master, which includes the reader for this storage (subsquid/sqd-query-engine#38).

🤖 Generated with Claude Code