Skip to content

Scanner: backfill the deferred secondary history after fast catch-up - #2167

Merged
nullPointerEnjoyer merged 4 commits into
masterfrom
scanner-backfill
Nov 20, 2026
Merged

nullPointerEnjoyer merged 4 commits into
masterfrom
scanner-backfill

Conversation

@nullPointerEnjoyer

Copy link
Copy Markdown
Collaborator

Summary

The scanner's fast catch-up mode (more than CATCH_UP_THRESHOLD blocks behind the node) defers the secondary history tables (address and token transaction listings) to index faster — but the pass that was supposed to rebuild them afterwards was never written, so any scanner instance that catches up across a large range keeps permanent holes in its history APIs.

This PR implements the backfill pass:

  • a persisted secondary_indexed_height watermark tracks how far the history tables are complete; it is initialized when catch-up first defers, advanced by full-mode batches only when no deferred gap sits below them, and clamped together with the disconnected rows on reorgs
  • backfill_secondary_indexes rebuilds the deferred history rows by re-reading the locally stored blocks through a read-only collector that mirrors the live scan's secondary extraction arm by arm; it writes exactly the two history tables (plus the watermark) and is structurally incapable of touching balances, UTXOs, orders, pools or statistics (the collector is bounded by the read-only storage trait), so re-scanned ranges cannot double-count
  • the daemon runs a 1,000-block backfill chunk per successful sync cycle while in full mode; live indexing keeps priority
  • databases written by builds predating the watermark get seeded at startup (a None watermark with a non-genesis tip is treated as possibly-deferred and re-seeded to genesis, forcing an idempotent full history rebuild); released versions never shipped catch-up mode, so this only affects unreleased-build operators
  • inconsistent local databases (a missing block or transaction-index row) fail loudly, and the watermark only ever advances to the last fully processed height

The tests include the equivalence check: a chain scanned fully in full mode ends up with byte-identical history rows, balances and UTXOs compared to the same chain scanned in catch-up mode followed by the backfill, exercised with simulated restarts mid-backfill (fresh state wrappers over the same storage between chunks).

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

🔍 OpenCodeReview found 7 issue(s) in this PR.

  • ✅ Successfully posted inline: 2 comment(s)
  • 📋 Routed to summary by policy: 3 comment(s)
  • ⏭️ Skipped (overlap with history): 2 comment(s)

bug · low

📄 api-server/scanner-lib/src/blockchain_state/mod.rs (L197-L206)

⚠️ GitHub could not post this as an inline comment: Routed to summary (severity low · category bug)

Errors from set_address_transactions_at_height / set_token_transaction_at_height are mapped to a fixed string, discarding the original storage error and the address/token id and height that would pinpoint the failing row in a partial backfill. Preserve the source error and include the identity of the row being written.

💡 Suggested Change

Before:

                for (address, transactions) in address_transactions {
                    db_tx
                        .set_address_transactions_at_height(address.as_str(), transactions, height)
                        .await
                        .map_err(|_| {
                            ApiServerStorageError::LowLevelStorageError(
                                "Unable to set address transactions".to_owned(),
                            )
                        })?;
                }

After:

                for (address, transactions) in address_transactions {
                    db_tx
                        .set_address_transactions_at_height(address.as_str(), transactions, height)
                        .await
                        .map_err(|e| {
                            ApiServerStorageError::LowLevelStorageError(format!(
                                "Unable to set address transactions for {address} at \
                                 height {height}: {e}"
                            ))
                        })?;
                }

bug · low

📄 api-server/scanner-lib/src/blockchain_state/mod.rs (L1359-L1360)

⚠️ GitHub could not post this as an inline comment: Routed to summary (severity low · category bug)

expect("Unable to encode destination") in insert_address panics on encoding failure even though the surrounding function returns Result<_, ApiServerStorageError> and every other data-dependent failure in the backfill is propagated as an error. Since the closure is sync, either pre-validate outside the closure and propagate via ?, or make the closure return a Result so a malformed stored Destination fails the backfill gracefully instead of crashing the daemon.

💡 Suggested Change

Before:

        let address = Address::<Destination>::new(chain_config, destination.clone())
            .expect("Unable to encode destination");

After:

        let address = Address::<Destination>::new(chain_config, destination.clone())
            .map_err(|e| {
                ApiServerStorageError::LowLevelStorageError(format!(
                    "Backfill: unable to encode destination: {e}"
                ))
            })?;

other · low

📄 api-server/scanner-lib/src/blockchain_state/mod.rs (L457-L463)

⚠️ GitHub could not post this as an inline comment: Routed to summary (severity low · category other)

The comment claims a pre-existing watermark row is kept at the smaller, lagging value, but the code only writes the watermark when secondary_indexed_height() is None — an existing (possibly larger or smaller) row is never compared or clamped. If the intent really is "keep the smaller value", the guard should read the current watermark and write min(current, common_block_height); otherwise the comment should be corrected to match the None-only behavior. As written, a stale higher watermark from a previous catch-up round could let a Full batch later advance over the still-deferred gap (since complete_below_batch only checks watermark >= common_block_height).

💡 Suggested Change

Before:

if defer_secondary && db_tx.secondary_indexed_height().await?.is_none() {
    // First deferred batch: the history tables are complete up to
    // and including the common ancestor height. If a row is
    // already present (a previous catch-up round), keep the
    // smaller, lagging value.
    db_tx.set_secondary_indexed_height(common_block_height).await?;
}

After:

if defer_secondary {
    let watermark = db_tx.secondary_indexed_height().await?;
    if watermark.is_none_or(|w| w > common_block_height) {
        db_tx.set_secondary_indexed_height(common_block_height).await?;
    }
}

Comment on lines +1689 to +1690
"INSERT INTO ml.misc_data (name, value) VALUES ('secondary_indexed_height', $1)
ON CONFLICT (name) DO UPDATE SET value = EXCLUDED.value;",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug · medium
Both backends (this upsert and the in-memory setter) write the watermark unconditionally with no monotonic guard. If the value is ever written out of order (e.g. a re-processed/lower block during replay or a concurrent writer), the watermark regresses and the backfill pass will re-index blocks — or worse, if a stale value is written after a newer one within different transactions, secondary tables newer than the watermark could be treated as not yet indexed and skipped on the next catch-up. Consider guarding the update, e.g. ... DO UPDATE SET value = EXCLUDED.value WHERE ... with a comparison on the decoded height, or an application-level monotonic check.

Suggestion:

Suggested change
"INSERT INTO ml.misc_data (name, value) VALUES ('secondary_indexed_height', $1)
ON CONFLICT (name) DO UPDATE SET value = EXCLUDED.value;",
"INSERT INTO ml.misc_data (name, value) VALUES ('secondary_indexed_height', $1)
ON CONFLICT (name) DO UPDATE SET value = EXCLUDED.value
WHERE decode_u64(ml.misc_data.value) < decode_u64(EXCLUDED.value);"

Comment on lines +1708 to +1712
"SELECT t.transaction_id, t.tx_global_index
FROM ml.transactions t
JOIN ml.blocks b ON t.owning_block_id = b.block_id
WHERE b.block_height = $1
ORDER BY t.tx_global_index ASC;",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

performance · medium
This query joins ml.transactions on owning_block_id, but the schema only creates indexes on ml.blocks (block_height) and ml.transactions (tx_global_index DESC) — there is no index on transactions.owning_block_id. During backfill this is called per block, and the planner will likely fall back to scanning ml.transactions (which grows to cover the whole chain) for every invocation. Consider adding an index on ml.transactions (owning_block_id) alongside the new watermark logic.

Suggestion:

Suggested change
"SELECT t.transaction_id, t.tx_global_index
FROM ml.transactions t
JOIN ml.blocks b ON t.owning_block_id = b.block_id
WHERE b.block_height = $1
ORDER BY t.tx_global_index ASC;",
self.just_execute(
"CREATE INDEX transactions_owning_block_id ON ml.transactions (owning_block_id);",
)

Comment on lines +155 to +157
let mut db_tx = self.storage.transaction_rw().await?;

let best_height = db_tx.get_best_block().await?.block_height();

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

performance · medium
The backfill opens one read-write transaction and holds it across up to max_blocks blocks (the daemon passes SECONDARY_BACKFILL_CHUNK = 1_000). Each block performs many awaited reads/writes; on Postgres this is a very long-running write transaction holding locks and building a large WAL/undo footprint, which can stall concurrent API traffic or hit statement/idle timeouts. Since the watermark already makes the pass resumable chunk-by-chunk (as the restart tests demonstrate), consider using a smaller default chunk, or committing per block/per few blocks inside the loop.

Comment on lines +1392 to +1395
let delegation = db_tx
.get_delegation_at_or_below_height(*delegation_id, block_height)
.await?
.expect("Delegation should exist");

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug · medium
Several failures in this new production backfill path abort via expect/panic! rather than the propagated Result the function already returns: .expect("Delegation should exist"), .expect("must be present"), .expect("Unable to encode destination"), panic!("Empty signature for htlc"), and .expect("proper signature") on witness decode. The live scan path (update_tables_from_transaction_inputs/outputs) uses the same expect style, so this is consistent with existing code, but the backfill runs over large historical ranges on stored data and a single inconsistent row or non-standard witness will crash the whole scanner process mid-pass, losing the entire chunk's work instead of failing the chunk cleanly and advancing the watermark only over the verified prefix. Consider converting these into ApiServerStorageError::LowLevelStorageError results, matching the documented fail-loudly-but-cleanly behavior used for missing block rows earlier in this function.

Suggestion:

Suggested change
let delegation = db_tx
.get_delegation_at_or_below_height(*delegation_id, block_height)
.await?
.expect("Delegation should exist");
let delegation = db_tx
.get_delegation_at_or_below_height(*delegation_id, block_height)
.await?
.ok_or_else(|| {
ApiServerStorageError::LowLevelStorageError(format!(
"Backfill: missing delegation {delegation_id} at height {block_height}"
))
})?;

Comment on lines +155 to +164
let mut db_tx = self.storage.transaction_rw().await?;

let best_height = db_tx.get_best_block().await?.block_height();
let Some(watermark) = db_tx.secondary_indexed_height().await? else {
// History was never deferred; the tables are complete.
return Ok(None);
};
if watermark >= best_height {
return Ok(None);
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug · medium
The backfill loop holds a single read-write transaction open across the whole chunk (up to 1000 blocks, each with many reads and inserts). On Postgres this keeps row locks for the entire chunk and competes with the live sync cycle and API-server writers/readers for the duration, since the backfill runs inside superve_sync between sync cycles. Also, when watermark >= best_height the function still opens and commits an RW transaction just to discover there is nothing to do — a read-only probe (transaction_ro) first would avoid the needless write transaction on every successful sync cycle.

Suggestion:

Suggested change
let mut db_tx = self.storage.transaction_rw().await?;
let best_height = db_tx.get_best_block().await?.block_height();
let Some(watermark) = db_tx.secondary_indexed_height().await? else {
// History was never deferred; the tables are complete.
return Ok(None);
};
if watermark >= best_height {
return Ok(None);
}
// Probe with a read-only transaction first so a no-op backfill does
// not open (and lock) a write transaction on every sync cycle.
{
let ro = self.storage.transaction_ro().await?;
let best_height = ro.get_best_block().await?.block_height();
match ro.secondary_indexed_height().await? {
None => return Ok(None),
Some(watermark) if watermark >= best_height => return Ok(None),
_ => {}
}
}
let mut db_tx = self.storage.transaction_rw().await?;

Comment on lines +252 to +261
if db_tx.secondary_indexed_height().await?.is_none() {
let best_height = db_tx.get_best_block().await?.block_height();
if best_height > BlockHeight::zero() {
logging::log::info!(
"Seeding the secondary history watermark at genesis: an older build may \
have deferred history rows during fast catch-up"
);
db_tx.set_secondary_indexed_height(BlockHeight::zero()).await?;
}
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug · medium
Startup seeding unconditionally pins the watermark at genesis for any pre-watermark database with a non-genesis tip — including databases written by older builds that never used fast catch-up and whose history tables are already complete. On upgrade this silently schedules a full-history rebuild backfill (re-writing every address/token history row for the whole chain), a long, heavyweight workload the operator did not ask for. Consider gating the seed behind a marker (e.g. a flag recording that a catch-up round actually deferred rows) or at least logging a prominent warning with the estimated rebuild size so operators can anticipate the I/O cost.

Suggestion:

Suggested change
if db_tx.secondary_indexed_height().await?.is_none() {
let best_height = db_tx.get_best_block().await?.block_height();
if best_height > BlockHeight::zero() {
logging::log::info!(
"Seeding the secondary history watermark at genesis: an older build may \
have deferred history rows during fast catch-up"
);
db_tx.set_secondary_indexed_height(BlockHeight::zero()).await?;
}
}
logging::log::warn!(
"Seeding the secondary history watermark at genesis: the next sync cycles \
will rebuild address/token history for the full chain ({best_height} blocks)"
);

@nullPointerEnjoyer
nullPointerEnjoyer merged commit 5c5c381 into master Nov 20, 2026
21 checks passed
@nullPointerEnjoyer
nullPointerEnjoyer deleted the scanner-backfill branch November 20, 2026 17:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants