Skip to content

perf: flat sets for the simulator's per-request block sets - #4

Merged
LauYeeYu merged 3 commits into
mainfrom
perf/flat-id-set
Nov 10, 2026
Merged

LauYeeYu merged 3 commits into
mainfrom
perf/flat-id-set

Conversation

@LauYeeYu

@LauYeeYu LauYeeYu commented Nov 9, 2026

Copy link
Copy Markdown
Collaborator

Fixes #3.

What this does

The first commit is @haochengxia's patch from #3, unchanged. It adds FlatIdSet, a flat open-addressing set that clears by bumping an epoch, for alpha_ and missing. It also builds alpha_resident_ only on the path that reads it. The second commit runs clang-format.

The third commit changes how FlatIdSet hashes, because on an AMD Zen 4 machine the patch made the pin-heavy runs slower:

LRU at 16k blocks, first 100k freeinference requests, 3 runs each wall time (s)
main (std::unordered_set) 303, 315, 309
#3's FlatIdSet (each id hashed on its own) 339, 351, 350
this PR (ids hashed in groups of 8) 288, 280, 285

The pin predicate is asked about the same pinned blocks again for every eviction, as cache_skip_pinned rescans them from the queue tail. lcp ids are dense and given out in first-access order, so a request's blocks mostly have consecutive ids. std::unordered_set's id % prime puts them in consecutive buckets, and its nodes are allocated in insertion order, so those lookups stay in a few cache lines. Hashing each id on its own scatters them over the table. On the Skylake-SP in #3, a 64-bit division costs more than those cache misses; on Zen 4 (7975WX) the division is cheap and the misses dominate. Hashing id >> 3 and keeping the low 3 bits puts 8 consecutive ids in one cache line of keys, and still scatters the groups. Two other variants were tried and rejected:

  • Storing each slot's key and stamp together, which made no difference.
  • Indexing by the id itself, which clustered and ran past 600 s.

How it was tested

  • testPrefixsim and testLcp pass, also under LeakSanitizer.
  • The RESULT lines are identical to main on every run below, and on all 43 algorithms of the default build on qwen traceA (_pos, --block-id raw) at 2k and 8k blocks. The one exception is lfuda at 2k, which takes over an hour and was not compared. The same holds for 8 algorithms on the raw (non-_pos) qwen traceA, where ids repeat within a request. cacheus at 2k fails the residency check on main too, at the same request and block, with the same message. That bug is on main and is not touched here.

Full traces, --cost-model qwen3coder30b_blksz_16, every run on its own core of a Threadripper PRO 7975WX, main and this PR at the same time:

run main (s) this PR (s) change
freeinference, lru 16k 1068.5 955.5 −11%
freeinference, random_compute 16k 1168.5 1177.5 +1%
freeinference, s3fifo 64k 725.2 684.1 −6%
freeinference, clock 64k 247.7 234.3 −5%
freeinference, fifo 64k 164.4 137.4 −16%
freeinference, lfu 64k 110.1 84.9 −23%
freeinference, arc 64k 118.8 98.9 −17%
freeinference, lru 256k 117.6 104.7 −11%
freeinference, lru 64k (median of 3) 131.8 110.6 −16%
chutes, lru 64k (median of 3) 91.1 80.0 −12%

Single runs vary by up to 20% on this machine (main's three freeinference lru 64k runs took 111, 132 and 132 s), so read the single-run rows as rough.

Not in this PR

Most of the time on pin-heavy runs is the rescan itself, not the set lookup. LRU at 16k on freeinference takes 1,081 s with pins and 79 s with --no-pin, with the same block hit ratio (0.610717) and compute saving ratio (0.533338); the pins only avoid 69.7M self-evictions. cache_skip_pinned starts from the queue tail on every evict call, so phase 2 walks a request's pinned blocks once per freed slot. An evict_n in libCacheSim's LRU, FIFO, Clock, Sieve and SLRU that walks the queue once per request would remove that. It belongs in libCacheSim, so it is left for a separate change there.

haochengxia and others added 3 commits November 9, 2026 16:27
The pin predicate, alpha_is_pinned, runs for every pinned object that
cache_skip_pinned walks past on an eviction: on freeinference, 6.0e9
calls with LRU at 64k blocks (9 per block accessed) and 2.2e11 at 16k.
Each was an std::unordered_set::count, a 64-bit division and a node
chase (about 6.5 ns), and the request's sets were rebuilt node by node
(about 114 ns per insert).

alpha_ and missing_ become a FlatIdSet: open addressing with a
multiplicative hash, cleared in O(1) by bumping an epoch, so one
instance serves every request without allocating (about 1 ns per
lookup, 9 ns per insert). alpha_resident_ is built only on the path
that reads it, for algorithms that do not report their victims.

Results are unchanged: the RESULT lines are identical. Run time drops
by 12% to 41% (7% to 29% when built with tcmalloc) across LRU, FIFO,
CLOCK, ARC, LFU, S3-FIFO and RandomCompute (Xeon Gold 6142).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@haochengxia

Copy link
Copy Markdown
Contributor

I also tested evict_n lol. It indeed works with clock, lru, and fifo. It would be nice if we can resume the pin scan rather than from the sractch every time.

@haochengxia

Copy link
Copy Markdown
Contributor

BTW, as a user with experience from libCacheSim, personally I would expect to have the output similar to interval miss ratio (probably interval compute saving ratio here) given by libCacheSim if possible.

@LauYeeYu

LauYeeYu commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator Author

@haochengxia In PrefixSim we use hit ratio and compute saving ratio because in LLM serving, people reports hit ratio. I agree this is different from the convention of libCacheSim but it think using hit ratio and compute saving ratio might make more sense in the context of LLM prefix cache.

@haochengxia

Copy link
Copy Markdown
Contributor

@haochengxia In PrefixSim we use hit ratio and compute saving ratio because in LLM serving, people reports hit ratio. I agree this is different from the convention of libCacheSim but it think using hit ratio and compute saving ratio might make more sense in the context of LLM prefix cache.

Agreed—hit ratio and compute saving ratio make sense for LLM prefix caching. I meant adding interval-level reporting, e.g., reporting both metrics every N requests or over fixed trace-time windows, so we can see how cache performance changes throughout the trace. The current output appears to report only the aggregate results at the end of each run.

@LauYeeYu

Copy link
Copy Markdown
Collaborator Author

@haochengxia In PrefixSim we use hit ratio and compute saving ratio because in LLM serving, people reports hit ratio. I agree this is different from the convention of libCacheSim but it think using hit ratio and compute saving ratio might make more sense in the context of LLM prefix cache.

Agreed—hit ratio and compute saving ratio make sense for LLM prefix caching. I meant adding interval-level reporting, e.g., reporting both metrics every N requests or over fixed trace-time windows, so we can see how cache performance changes throughout the trace. The current output appears to report only the aggregate results at the end of each run.

I see. That makes sense! We can add it (probably through another PR).

@LauYeeYu
LauYeeYu merged commit 085f35c into main Nov 10, 2026
4 checks passed
@LauYeeYu
LauYeeYu deleted the perf/flat-id-set branch November 10, 2026 04:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Perf] A flat set for the simulator's per-request block sets

2 participants