Repository navigation
Optimize BlockHeaderDB - #56
Open
edmundnoble wants to merge 6 commits into
Open
edmundnoble wants to merge 6 commits into
edmundnoble wants to merge 6 commits into
Conversation
The generic implementation (chainBranchEntries) resolves the branch bounds
and then traverses the branches by looking up the parent of each entry with
an individual point lookup. Each of those lookups seeks the ranked block
header table from its root and allocates a fresh set of read options, and
the upper bounds get resolved twice (once by chainBranchEntries and once
again by getBranch).
Since that table is keyed by (block height, block hash), all entries of a
block height are stored next to each other. This implements branchEntries
for BlockHeaderDb directly on top of a single table iterator:
* branchStream walks the branch level by level (the iterator equivalent
of getBranchResolved), yielding upperActive \ lowerActive at each rank
and descending to the parents.
* branchParents resolves the parents with a single iterSeek into that
shared iterator and validates the seek via the value's block hash,
which is cheaper than decoding the key with iterKey.
* The bounds are resolved once, and the lower bounds are skipped
entirely when there are no upper bounds.
* reduceUpperBounds keeps the chainBranchEntries optimization of
reducing the upper bounds to ancestors of the requested max rank,
which bounds the depth of the traversal.
TreeDB now exports applyRankDesc so that the rank filter isn't duplicated.
Benchmarks (bench/Chainweb/BlockHeaderDBBench.hs: a 1000 block trunk with a
50 block fork), mean of before -> after:
branchEntries/page (limit 250, maxRank=tip) 1.932 ms -> 1.351 ms -30%
branchEntries/page (limit 250, maxRank=tip, lower=fork) 453 us -> 297 us -34%
branchEntries/page (limit 250, maxRank=tip-500) 1.991 ms -> 1.373 ms -31%
The remaining benchmarks are unchanged, they don't use this code path:
ancestors/trunk-tip (1000 blocks) 10.08 ms -> 10.16 ms
seekAncestor/unique rank (tip - 500) 16.7 us -> 17.3 us
seekAncestor/fork region (tip - 25) 870 us -> 885 us
forkEntry (50 block fork) 823 us -> 851 us
branchDiff (50 block fork) 837 us -> 887 us
getBranchIncreasing (last 100 blocks) 974 us -> 983 us
Adds Chainweb.Test.BlockHeaderDBBranch, which compares the new
implementation against the generic one for identical queries (streamed
items, reported count and Eos) over random trees, over random chains with
forks of random length at random heights, and over a large deterministic
grid on a 200 block trunk with a 150 block fork.
edmundnoble
force-pushed
the
push-vwszunmkpuvq
branch
from
November 7, 2026 23:27
5ff5452 to
d70ba2a
Compare
| branchEntries db k l mir mar lower upper f | ||
| -- If the upper bounds are empty the result is empty, too. In that case | ||
| -- the lower bounds are not even resolved. | ||
| | HS.null upper = f (return (0, Eos True)) |
Member
There was a problem hiding this comment.
Not sure this test is necessary here. Uppers empty case will be caught later
| -- Resolve the upper bounds. Keys that aren't in the database are | ||
| -- ignored. | ||
| uppers <- resolveBounds db _getUpperBound upper | ||
| if HS.null uppers |
Member
There was a problem hiding this comment.
Probably not necessary here neither. Will be caught later.
| iterSeek it (RankedBlockHash (int r) h) | ||
| iterValue it >>= \case | ||
| Just v | ||
| | view blockHash (_getRankedBlockHeader v) == h -> |
Member
There was a problem hiding this comment.
What does this mean?
What are the cases where blockHash doesn't match with the DB key ?
| | otherwise = do | ||
| lowers <- liftIO $ getEntriesHs db _getLowerBound lowerBounds | ||
|
|
||
| let mar = getMax $ fromJuste $ |
Member
There was a problem hiding this comment.
Maybe I miss something ..
But why applying Just on every element; and then doing fromJuste ?
Member
There was a problem hiding this comment.
NVM, we need this... because it's needed by foldMap ..
But I think we should manage properly the "Nothing" case
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Net results are something like 1.67x speedup.
Old benchmarks
benchmarking block-header-db/getBranch/ancestors/trunk-tip (1000 blocks) time 10.94 ms (10.71 ms .. 11.18 ms) 0.998 R² (0.996 R² .. 0.999 R²) mean 10.50 ms (10.37 ms .. 10.64 ms) std dev 337.9 μs (272.8 μs .. 420.5 μs)benchmarking block-header-db/getBranch/branchEntries/page (limit 250, maxRank=tip)
time 2.091 ms (2.042 ms .. 2.127 ms)
0.997 R² (0.995 R² .. 0.998 R²)
mean 2.136 ms (2.104 ms .. 2.177 ms)
std dev 116.8 μs (93.00 μs .. 145.9 μs)
variance introduced by outliers: 38% (moderately inflated)
benchmarking block-header-db/getBranch/branchEntries/page (limit 250, maxRank=tip, lower=fork-point)
time 501.2 μs (493.7 μs .. 509.9 μs)
0.998 R² (0.996 R² .. 0.999 R²)
mean 484.8 μs (479.0 μs .. 490.6 μs)
std dev 20.85 μs (17.25 μs .. 27.56 μs)
variance introduced by outliers: 36% (moderately inflated)
benchmarking block-header-db/getBranch/branchEntries/page (limit 250, maxRank=tip-500)
time 2.200 ms (2.161 ms .. 2.244 ms)
0.997 R² (0.995 R² .. 0.999 R²)
mean 2.157 ms (2.134 ms .. 2.179 ms)
std dev 85.21 μs (65.99 μs .. 108.2 μs)
variance introduced by outliers: 24% (moderately inflated)
benchmarking block-header-db/seekAncestor/unique rank (tip - 500)
time 17.03 μs (16.95 μs .. 17.12 μs)
1.000 R² (1.000 R² .. 1.000 R²)
mean 16.85 μs (16.76 μs .. 16.93 μs)
std dev 293.3 ns (246.3 ns .. 363.4 ns)
variance introduced by outliers: 14% (moderately inflated)
benchmarking block-header-db/seekAncestor/fork region (tip - 25)
time 908.2 μs (887.7 μs .. 935.4 μs)
0.995 R² (0.992 R² .. 0.998 R²)
mean 917.8 μs (905.5 μs .. 930.8 μs)
std dev 42.08 μs (35.87 μs .. 53.31 μs)
variance introduced by outliers: 35% (moderately inflated)
benchmarking block-header-db/fork/forkEntry (50 block fork)
time 835.3 μs (829.7 μs .. 841.0 μs)
0.999 R² (0.998 R² .. 0.999 R²)
mean 837.6 μs (829.3 μs .. 846.0 μs)
std dev 28.12 μs (22.05 μs .. 36.38 μs)
variance introduced by outliers: 24% (moderately inflated)
benchmarking block-header-db/fork/branchDiff (50 block fork)
time 848.6 μs (842.0 μs .. 855.5 μs)
0.999 R² (0.999 R² .. 0.999 R²)
mean 837.3 μs (829.8 μs .. 843.8 μs)
std dev 24.26 μs (19.71 μs .. 33.03 μs)
variance introduced by outliers: 18% (moderately inflated)
benchmarking block-header-db/fork/getBranchIncreasing (last 100 blocks)
time 994.8 μs (991.2 μs .. 998.2 μs)
1.000 R² (1.000 R² .. 1.000 R²)
mean 984.3 μs (980.1 μs .. 988.0 μs)
std dev 13.33 μs (10.36 μs .. 16.68 μs)
New benchmarks
benchmarking block-header-db/getBranch/ancestors/trunk-tip (1000 blocks) time 10.31 ms (9.864 ms .. 10.72 ms) 0.993 R² (0.988 R² .. 0.997 R²) mean 10.11 ms (9.959 ms .. 10.29 ms) std dev 451.7 μs (344.5 μs .. 579.0 μs) variance introduced by outliers: 19% (moderately inflated)benchmarking block-header-db/getBranch/branchEntries/page (limit 250, maxRank=tip)
time 1.394 ms (1.371 ms .. 1.417 ms)
0.999 R² (0.998 R² .. 1.000 R²)
mean 1.359 ms (1.353 ms .. 1.369 ms)
std dev 29.14 μs (21.02 μs .. 41.62 μs)
variance introduced by outliers: 10% (moderately inflated)
benchmarking block-header-db/getBranch/branchEntries/page (limit 250, maxRank=tip, lower=fork-point)
time 300.2 μs (297.7 μs .. 302.6 μs)
0.999 R² (0.999 R² .. 1.000 R²)
mean 293.2 μs (291.4 μs .. 294.9 μs)
std dev 6.340 μs (5.258 μs .. 8.571 μs)
variance introduced by outliers: 14% (moderately inflated)
benchmarking block-header-db/getBranch/branchEntries/page (limit 250, maxRank=tip-500)
time 1.398 ms (1.393 ms .. 1.404 ms)
1.000 R² (1.000 R² .. 1.000 R²)
mean 1.393 ms (1.385 ms .. 1.398 ms)
std dev 18.81 μs (15.65 μs .. 25.87 μs)
benchmarking block-header-db/seekAncestor/unique rank (tip - 500)
time 16.61 μs (16.54 μs .. 16.69 μs)
1.000 R² (1.000 R² .. 1.000 R²)
mean 16.52 μs (16.45 μs .. 16.59 μs)
std dev 234.6 ns (180.5 ns .. 291.0 ns)
benchmarking block-header-db/seekAncestor/fork region (tip - 25)
time 867.7 μs (834.6 μs .. 886.4 μs)
0.996 R² (0.995 R² .. 0.998 R²)
mean 846.1 μs (832.7 μs .. 857.0 μs)
std dev 40.34 μs (34.30 μs .. 55.27 μs)
variance introduced by outliers: 38% (moderately inflated)
benchmarking block-header-db/fork/forkEntry (50 block fork)
time 815.6 μs (810.5 μs .. 820.4 μs)
0.999 R² (0.999 R² .. 1.000 R²)
mean 809.2 μs (804.1 μs .. 814.6 μs)
std dev 17.55 μs (14.63 μs .. 22.46 μs)
variance introduced by outliers: 11% (moderately inflated)
benchmarking block-header-db/fork/branchDiff (50 block fork)
time 826.2 μs (820.8 μs .. 830.4 μs)
1.000 R² (0.999 R² .. 1.000 R²)
mean 821.2 μs (816.5 μs .. 825.6 μs)
std dev 14.91 μs (12.44 μs .. 21.01 μs)
benchmarking block-header-db/fork/getBranchIncreasing (last 100 blocks)
time 963.3 μs (960.2 μs .. 966.5 μs)
1.000 R² (0.999 R² .. 1.000 R²)
mean 955.8 μs (950.5 μs .. 960.2 μs)
std dev 19.18 μs (13.47 μs .. 27.07 μs)
benchmarking cut-db/local: catch up one 399-block step from genesis
time 839.3 ms (790.9 ms .. 890.5 ms)
1.000 R² (0.998 R² .. 1.000 R²)
mean 980.0 ms (918.6 ms .. 1.075 s)
std dev 91.11 ms (18.69 ms .. 120.2 ms)
variance introduced by outliers: 22% (moderately inflated)
benchmarking cut-db/100ms latency: catch up one 399-block step from genesis
time 1.749 s (1.728 s .. 1.782 s)
1.000 R² (1.000 R² .. 1.000 R²)
mean 1.728 s (1.707 s .. 1.738 s)
std dev 20.08 ms (753.3 μs .. 24.14 ms)
variance introduced by outliers: 19% (moderately inflated)