Repository navigation
Scanner: backfill the deferred secondary history after fast catch-up - #2167
Conversation
|
🔍 OpenCodeReview found 7 issue(s) in this PR.
📄
|
| "INSERT INTO ml.misc_data (name, value) VALUES ('secondary_indexed_height', $1) | ||
| ON CONFLICT (name) DO UPDATE SET value = EXCLUDED.value;", |
There was a problem hiding this comment.
Both backends (this upsert and the in-memory setter) write the watermark unconditionally with no monotonic guard. If the value is ever written out of order (e.g. a re-processed/lower block during replay or a concurrent writer), the watermark regresses and the backfill pass will re-index blocks — or worse, if a stale value is written after a newer one within different transactions, secondary tables newer than the watermark could be treated as not yet indexed and skipped on the next catch-up. Consider guarding the update, e.g. ... DO UPDATE SET value = EXCLUDED.value WHERE ... with a comparison on the decoded height, or an application-level monotonic check.
Suggestion:
| "INSERT INTO ml.misc_data (name, value) VALUES ('secondary_indexed_height', $1) | |
| ON CONFLICT (name) DO UPDATE SET value = EXCLUDED.value;", | |
| "INSERT INTO ml.misc_data (name, value) VALUES ('secondary_indexed_height', $1) | |
| ON CONFLICT (name) DO UPDATE SET value = EXCLUDED.value | |
| WHERE decode_u64(ml.misc_data.value) < decode_u64(EXCLUDED.value);" |
| "SELECT t.transaction_id, t.tx_global_index | ||
| FROM ml.transactions t | ||
| JOIN ml.blocks b ON t.owning_block_id = b.block_id | ||
| WHERE b.block_height = $1 | ||
| ORDER BY t.tx_global_index ASC;", |
There was a problem hiding this comment.
This query joins ml.transactions on owning_block_id, but the schema only creates indexes on ml.blocks (block_height) and ml.transactions (tx_global_index DESC) — there is no index on transactions.owning_block_id. During backfill this is called per block, and the planner will likely fall back to scanning ml.transactions (which grows to cover the whole chain) for every invocation. Consider adding an index on ml.transactions (owning_block_id) alongside the new watermark logic.
Suggestion:
| "SELECT t.transaction_id, t.tx_global_index | |
| FROM ml.transactions t | |
| JOIN ml.blocks b ON t.owning_block_id = b.block_id | |
| WHERE b.block_height = $1 | |
| ORDER BY t.tx_global_index ASC;", | |
| self.just_execute( | |
| "CREATE INDEX transactions_owning_block_id ON ml.transactions (owning_block_id);", | |
| ) |
| let mut db_tx = self.storage.transaction_rw().await?; | ||
|
|
||
| let best_height = db_tx.get_best_block().await?.block_height(); |
There was a problem hiding this comment.
The backfill opens one read-write transaction and holds it across up to max_blocks blocks (the daemon passes SECONDARY_BACKFILL_CHUNK = 1_000). Each block performs many awaited reads/writes; on Postgres this is a very long-running write transaction holding locks and building a large WAL/undo footprint, which can stall concurrent API traffic or hit statement/idle timeouts. Since the watermark already makes the pass resumable chunk-by-chunk (as the restart tests demonstrate), consider using a smaller default chunk, or committing per block/per few blocks inside the loop.
| let delegation = db_tx | ||
| .get_delegation_at_or_below_height(*delegation_id, block_height) | ||
| .await? | ||
| .expect("Delegation should exist"); |
There was a problem hiding this comment.
Several failures in this new production backfill path abort via expect/panic! rather than the propagated Result the function already returns: .expect("Delegation should exist"), .expect("must be present"), .expect("Unable to encode destination"), panic!("Empty signature for htlc"), and .expect("proper signature") on witness decode. The live scan path (update_tables_from_transaction_inputs/outputs) uses the same expect style, so this is consistent with existing code, but the backfill runs over large historical ranges on stored data and a single inconsistent row or non-standard witness will crash the whole scanner process mid-pass, losing the entire chunk's work instead of failing the chunk cleanly and advancing the watermark only over the verified prefix. Consider converting these into ApiServerStorageError::LowLevelStorageError results, matching the documented fail-loudly-but-cleanly behavior used for missing block rows earlier in this function.
Suggestion:
| let delegation = db_tx | |
| .get_delegation_at_or_below_height(*delegation_id, block_height) | |
| .await? | |
| .expect("Delegation should exist"); | |
| let delegation = db_tx | |
| .get_delegation_at_or_below_height(*delegation_id, block_height) | |
| .await? | |
| .ok_or_else(|| { | |
| ApiServerStorageError::LowLevelStorageError(format!( | |
| "Backfill: missing delegation {delegation_id} at height {block_height}" | |
| )) | |
| })?; |
| let mut db_tx = self.storage.transaction_rw().await?; | ||
|
|
||
| let best_height = db_tx.get_best_block().await?.block_height(); | ||
| let Some(watermark) = db_tx.secondary_indexed_height().await? else { | ||
| // History was never deferred; the tables are complete. | ||
| return Ok(None); | ||
| }; | ||
| if watermark >= best_height { | ||
| return Ok(None); | ||
| } |
There was a problem hiding this comment.
The backfill loop holds a single read-write transaction open across the whole chunk (up to 1000 blocks, each with many reads and inserts). On Postgres this keeps row locks for the entire chunk and competes with the live sync cycle and API-server writers/readers for the duration, since the backfill runs inside superve_sync between sync cycles. Also, when watermark >= best_height the function still opens and commits an RW transaction just to discover there is nothing to do — a read-only probe (transaction_ro) first would avoid the needless write transaction on every successful sync cycle.
Suggestion:
| let mut db_tx = self.storage.transaction_rw().await?; | |
| let best_height = db_tx.get_best_block().await?.block_height(); | |
| let Some(watermark) = db_tx.secondary_indexed_height().await? else { | |
| // History was never deferred; the tables are complete. | |
| return Ok(None); | |
| }; | |
| if watermark >= best_height { | |
| return Ok(None); | |
| } | |
| // Probe with a read-only transaction first so a no-op backfill does | |
| // not open (and lock) a write transaction on every sync cycle. | |
| { | |
| let ro = self.storage.transaction_ro().await?; | |
| let best_height = ro.get_best_block().await?.block_height(); | |
| match ro.secondary_indexed_height().await? { | |
| None => return Ok(None), | |
| Some(watermark) if watermark >= best_height => return Ok(None), | |
| _ => {} | |
| } | |
| } | |
| let mut db_tx = self.storage.transaction_rw().await?; |
| if db_tx.secondary_indexed_height().await?.is_none() { | ||
| let best_height = db_tx.get_best_block().await?.block_height(); | ||
| if best_height > BlockHeight::zero() { | ||
| logging::log::info!( | ||
| "Seeding the secondary history watermark at genesis: an older build may \ | ||
| have deferred history rows during fast catch-up" | ||
| ); | ||
| db_tx.set_secondary_indexed_height(BlockHeight::zero()).await?; | ||
| } | ||
| } |
There was a problem hiding this comment.
Startup seeding unconditionally pins the watermark at genesis for any pre-watermark database with a non-genesis tip — including databases written by older builds that never used fast catch-up and whose history tables are already complete. On upgrade this silently schedules a full-history rebuild backfill (re-writing every address/token history row for the whole chain), a long, heavyweight workload the operator did not ask for. Consider gating the seed behind a marker (e.g. a flag recording that a catch-up round actually deferred rows) or at least logging a prominent warning with the estimated rebuild size so operators can anticipate the I/O cost.
Suggestion:
| if db_tx.secondary_indexed_height().await?.is_none() { | |
| let best_height = db_tx.get_best_block().await?.block_height(); | |
| if best_height > BlockHeight::zero() { | |
| logging::log::info!( | |
| "Seeding the secondary history watermark at genesis: an older build may \ | |
| have deferred history rows during fast catch-up" | |
| ); | |
| db_tx.set_secondary_indexed_height(BlockHeight::zero()).await?; | |
| } | |
| } | |
| logging::log::warn!( | |
| "Seeding the secondary history watermark at genesis: the next sync cycles \ | |
| will rebuild address/token history for the full chain ({best_height} blocks)" | |
| ); |
Summary
The scanner's fast catch-up mode (more than
CATCH_UP_THRESHOLDblocks behind the node) defers the secondary history tables (address and token transaction listings) to index faster — but the pass that was supposed to rebuild them afterwards was never written, so any scanner instance that catches up across a large range keeps permanent holes in its history APIs.This PR implements the backfill pass:
secondary_indexed_heightwatermark tracks how far the history tables are complete; it is initialized when catch-up first defers, advanced by full-mode batches only when no deferred gap sits below them, and clamped together with the disconnected rows on reorgsbackfill_secondary_indexesrebuilds the deferred history rows by re-reading the locally stored blocks through a read-only collector that mirrors the live scan's secondary extraction arm by arm; it writes exactly the two history tables (plus the watermark) and is structurally incapable of touching balances, UTXOs, orders, pools or statistics (the collector is bounded by the read-only storage trait), so re-scanned ranges cannot double-countNonewatermark with a non-genesis tip is treated as possibly-deferred and re-seeded to genesis, forcing an idempotent full history rebuild); released versions never shipped catch-up mode, so this only affects unreleased-build operatorsThe tests include the equivalence check: a chain scanned fully in full mode ends up with byte-identical history rows, balances and UTXOs compared to the same chain scanned in catch-up mode followed by the backfill, exercised with simulated restarts mid-backfill (fresh state wrappers over the same storage between chunks).