Repository navigation
perf: flat sets for the simulator's per-request block sets - #4
Conversation
The pin predicate, alpha_is_pinned, runs for every pinned object that cache_skip_pinned walks past on an eviction: on freeinference, 6.0e9 calls with LRU at 64k blocks (9 per block accessed) and 2.2e11 at 16k. Each was an std::unordered_set::count, a 64-bit division and a node chase (about 6.5 ns), and the request's sets were rebuilt node by node (about 114 ns per insert). alpha_ and missing_ become a FlatIdSet: open addressing with a multiplicative hash, cleared in O(1) by bumping an epoch, so one instance serves every request without allocating (about 1 ns per lookup, 9 ns per insert). alpha_resident_ is built only on the path that reads it, for algorithms that do not report their victims. Results are unchanged: the RESULT lines are identical. Run time drops by 12% to 41% (7% to 29% when built with tcmalloc) across LRU, FIFO, CLOCK, ARC, LFU, S3-FIFO and RandomCompute (Xeon Gold 6142). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
I also tested evict_n lol. It indeed works with clock, lru, and fifo. It would be nice if we can resume the pin scan rather than from the sractch every time. |
|
BTW, as a user with experience from libCacheSim, personally I would expect to have the output similar to interval miss ratio (probably interval compute saving ratio here) given by libCacheSim if possible. |
|
@haochengxia In PrefixSim we use hit ratio and compute saving ratio because in LLM serving, people reports hit ratio. I agree this is different from the convention of libCacheSim but it think using hit ratio and compute saving ratio might make more sense in the context of LLM prefix cache. |
Agreed—hit ratio and compute saving ratio make sense for LLM prefix caching. I meant adding interval-level reporting, e.g., reporting both metrics every N requests or over fixed trace-time windows, so we can see how cache performance changes throughout the trace. The current output appears to report only the aggregate results at the end of each run. |
I see. That makes sense! We can add it (probably through another PR). |
Fixes #3.
What this does
The first commit is @haochengxia's patch from #3, unchanged. It adds
FlatIdSet, a flat open-addressing set that clears by bumping an epoch, foralpha_andmissing. It also buildsalpha_resident_only on the path that reads it. The second commit runs clang-format.The third commit changes how
FlatIdSethashes, because on an AMD Zen 4 machine the patch made the pin-heavy runs slower:main(std::unordered_set)FlatIdSet(each id hashed on its own)The pin predicate is asked about the same pinned blocks again for every eviction, as
cache_skip_pinnedrescans them from the queue tail. lcp ids are dense and given out in first-access order, so a request's blocks mostly have consecutive ids.std::unordered_set'sid % primeputs them in consecutive buckets, and its nodes are allocated in insertion order, so those lookups stay in a few cache lines. Hashing each id on its own scatters them over the table. On the Skylake-SP in #3, a 64-bit division costs more than those cache misses; on Zen 4 (7975WX) the division is cheap and the misses dominate. Hashingid >> 3and keeping the low 3 bits puts 8 consecutive ids in one cache line of keys, and still scatters the groups. Two other variants were tried and rejected:How it was tested
testPrefixsimandtestLcppass, also under LeakSanitizer.mainon every run below, and on all 43 algorithms of the default build on qwen traceA (_pos,--block-id raw) at 2k and 8k blocks. The one exception islfudaat 2k, which takes over an hour and was not compared. The same holds for 8 algorithms on the raw (non-_pos) qwen traceA, where ids repeat within a request.cacheusat 2k fails the residency check onmaintoo, at the same request and block, with the same message. That bug is onmainand is not touched here.Full traces,
--cost-model qwen3coder30b_blksz_16, every run on its own core of a Threadripper PRO 7975WX,mainand this PR at the same time:main(s)Single runs vary by up to 20% on this machine (
main's three freeinference lru 64k runs took 111, 132 and 132 s), so read the single-run rows as rough.Not in this PR
Most of the time on pin-heavy runs is the rescan itself, not the set lookup. LRU at 16k on freeinference takes 1,081 s with pins and 79 s with
--no-pin, with the same block hit ratio (0.610717) and compute saving ratio (0.533338); the pins only avoid 69.7M self-evictions.cache_skip_pinnedstarts from the queue tail on everyevictcall, so phase 2 walks a request's pinned blocks once per freed slot. Anevict_nin libCacheSim's LRU, FIFO, Clock, Sieve and SLRU that walks the queue once per request would remove that. It belongs in libCacheSim, so it is left for a separate change there.