Partition index and filter blocks of the entry location RocksDB

[merlimat]

Problem

DbLedgerStorage's entry location index (KeyValueStorageRocksDB, DbConfigType.EntryLocation) keeps index and
filter blocks in the block cache (cache_index_and_filter_blocks=true). It uses the default index type and one
full Bloom filter (10 bits/key) per SST, with a 64 MiB target file size. A full filter covers the whole file: about
8.7 MiB for a 64 MiB SST of (ledgerId, entryId) keys.

The block cache is new LRUCache(size). RocksDB splits any cache of 32 MiB or more into 64 shards, and a block
lives in one shard. The default size is maxDirectMemory / ledgerDirs / 10, so with the default 2 GiB of direct
memory and one ledger directory it is 205 MiB and each shard holds 3.2 MiB. An 8.7 MiB filter block needs a shard
of at least 8.7 MiB, which means a cache of about 560 MiB per ledger directory.

In RocksDB 10.10.1's LRUCacheShard, inserting a block bigger than its shard first evicts every unreferenced
entry in the shard, then inserts the block anyway. Release() frees it immediately because the shard is over
capacity. The filter is not pinned, so every Get() on every overlapping SST:

• reads the whole multi-MiB filter again,
• verifies its checksum and parses it,
• and empties that shard of the cache.

With buffered I/O, BookKeeper's default, these reads come from the page cache and lookups become CPU-bound.
With direct I/O, or when the page cache is tight, they go to disk. EntryLocationIndex.getLocation() runs on
every read-cache miss of a bookie, so this limits bookie read throughput.

Example

Current code (KeyValueStorageRocksDB.initializeRocksDBWithBookieConf):

this.cache = new LRUCache(blockCacheSize);                                  // 64 shards when >= 32 MiB
...
tableOptions.setFilterPolicy(new BloomFilter(bloomFilterBitsPerKey, false));  // one full filter per SST
tableOptions.setCacheIndexAndFilterBlocks(true);                            // read through the cache, not pinned

The benchmark drives EntryLocationIndex through the default bookie-conf path:

• 50M keys: 1,000 ledgers × 50,000 entries, written in sorted, flush-like batches.
• Natural leveled compaction left 10 SSTs: 2 in L0, 3 in L5, 5 in L6. Data is 459 MiB and the largest filter is 8.67 MiB.
• Random getLocation() from 4 threads: 5 s warm-up, then 20 s measured.
• Buffered I/O with all files in the page cache, on an Apple M5 Pro.
• Counters come from the RocksDB PerfContext.

block cache (shard)	gets/s before → after	filter block reads per get	bytes read per get	CPU per get
64 MiB (1 MiB)	1,746 → 176,144 (×101)	3.58 → 1.47	16.2 MB → 34 KB	2,275 µs → 23 µs
256 MiB (4 MiB)	2,182 → 216,277 (×99)	1.44 → 0.74	12.8 MB → 26 KB	1,814 µs → 19 µs
1 GiB (16 MiB)	834,446 → 757,691 (−9%)	0 → 0.06	2.4 KB → 1.8 KB	4.7 µs → 5.2 µs

• Before the change, at 64 and 256 MiB the reader threads copied 27 GB/s out of the page cache, almost all of it filter blocks.
• At 64 MiB the block cache held no filter block at the end; at 256 MiB it held one.
• With partitioning, filter reads are 4 KiB partitions, and 2,600 to 16,000 of them stay cached.
• A third run used the new code with partitioning disabled on the old DB: 2,226 gets/s, the same as before.

Floor lookups ran on the same DB, right after the get phase in the same process: getLastEntryInLedger()
(getFloor) on a random ledger, 4 threads, 5 s warm-up, then 20 s measured.

block cache (shard)	floor lookups/s before → after	blocks read per lookup	bytes read per lookup	CPU per lookup
64 MiB (1 MiB)	76,928 → 77,076 (+0.2%)	2.37 → 2.40	73 KB → 73 KB	51.7 µs → 51.7 µs
256 MiB (4 MiB)	271,134 → 258,448 (−4.7%)	0 → 0.006	0 → 184 B	14.6 µs → 15.4 µs
1 GiB (16 MiB)	274,781 → 260,038 (−5.4%)	0 → 0	0 → 0	14.4 µs → 15.3 µs

• Floor lookups use iterators, which don't consult Bloom filters, so the oversized filter blocks never slowed them down.
• Each lookup checks one index block per sorted run (4 here). At 64 MiB the data blocks it needs don't stay cached, so before and after both read about 2.4 blocks from the page cache.
• At 256 MiB and 1 GiB everything is cached. The second index level adds about 0.9 µs of CPU per lookup, about 5% slower.
• The new code with partitioning disabled ran 268,566 floor lookups/s at 256 MiB, the same as before.

Change

tableOptions.setCacheIndexAndFilterBlocks(true);
if (partitionedIndexAndFiltersEnabled) {
    tableOptions.setIndexType(IndexType.kTwoLevelIndexSearch);
    tableOptions.setPartitionFilters(true);
    tableOptions.setPinTopLevelIndexAndFilter(true);
}

This is the setup RocksDB recommends when index and filter blocks live in the block cache
(https://github.com/facebook/rocksdb/wiki/Partitioned-Index-Filters):

• Index and filter blocks are split into partitions of about 4 KiB (metadata_block_size).
• Only the top-level index and filter blocks are pinned: about 47 KiB per 64 MiB SST in the benchmark, charged to the block cache.
• The partitions are cached like data blocks.

Other changes:

• The new setting dbStorage_rocksDB_partitionedIndexAndFiltersEnabled defaults to true and turns this off. It is documented in bk_server.conf and reference/config.md.
• conf/entry_location_rocksdb.conf.default and its test copy set index_type=kTwoLevelIndexSearch, partition_filters=true and pin_top_level_index_and_filter=true.

Compatibility:

• SSTs keep format_version=5.
• Partitioned indexes and filters are readable by RocksDB 7.9.2 (BookKeeper 4.17 / Pulsar 4.0), so downgrades still work. 10.10's default decouple_partitioned_filters=true is readable by every version that reads partitioned filters.
• Tested by reading the SSTs from the benchmark with rocksdbjni 7.9.2: checksums of all 10 SSTs verify, 100K gets of present keys and 100K of absent keys return correct results with the partitioned Bloom filters in use, and a full scan returns all 50M keys.
• Only SSTs written after the upgrade use the new layout; old and new SSTs can be read side by side. Existing SSTs are rewritten by normal compaction, or right away through /api/v1/bookie/entry_location_compact or entryLocationCompactionInterval.

Trade-off: the second index level costs an extra lookup. When the whole index fits in the block cache with room
to spare (the 1 GiB row), point gets are about 9% slower. Floor lookups are about 5% slower whenever the blocks
they touch are cached (the 256 MiB and 1 GiB rows). Real location indexes are usually much larger than the block
cache.

Testing

• New test KeyValueStorageRocksDBTest#testEntryLocationPartitionedIndexAndFilters:
  • Writes and compacts an SST three ways: the default bookie-conf path, the same path with the setting off, and the shipped options file.
  • Checks the index type with SstFileReader (index partitions) and the filter type from the SST's metaindex key (partitionedfilter. vs fullfilter.).
  • Fails without the change, on both the bookie-conf and options-file paths.
• org.apache.bookkeeper.bookie.storage.ldb tests pass, except DbLedgerStorageWithDirectEntryLoggerTest. It fails locally because there is no native-io JNI library for macOS/aarch64; this change doesn't touch it.
• checkstyle, spotbugs and spotless are clean.
• The benchmark and the 7.9.2 read check above are scratch programs and are not committed.

[lhotari]

LGTM. Thanks for the careful analysis and benchmark behind this fix for the oversized filter blocks. The option combination (two-level index, partitioned filters, pinned top-level blocks) is valid for RocksDB 10.10.1 and stays on format_version 5, and the on-disk layout is readable by older releases. The only note is a non-blocking test suggestion inline.