Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 45 additions & 0 deletions doc/design.md
Original file line number Diff line number Diff line change
Expand Up @@ -794,6 +794,51 @@ partial-node policy occasionally punches a gap mid-prefix instead. That is the
contiguity difference that the removed `prefix_hit_ratio` metric used to
summarise in one number, now visible in full rather than averaged away.

## Interval reports (`--report-interval`)

The `RESULT` line averages over the whole trace. To see how the ratios move
through it, `--report-interval <s>` prints a line every `s` seconds of trace
time, as cachesim's option of the same name does:

```bash
prefixsim --trace freeinference.lcp.zst --cache-size 64k --algo lru \
--cost-model qwen3coder30b_blksz_16 --report-interval 86400
```
```
freeinference.lcp.zst lru 24.00 hour: 15761 requests, block hit ratio 0.9439, interval block hit ratio 0.9439, compute saving ratio 0.9391, interval compute saving ratio 0.9391
freeinference.lcp.zst lru 48.00 hour: 33679 requests, block hit ratio 0.9434, interval block hit ratio 0.9428, compute saving ratio 0.9390, interval compute saving ratio 0.9389
freeinference.lcp.zst lru 72.00 hour: 73808 requests, block hit ratio 0.9428, interval block hit ratio 0.9417, compute saving ratio 0.9385, interval compute saving ratio 0.9376
freeinference.lcp.zst lru 96.00 hour: 179312 requests, block hit ratio 0.9293, interval block hit ratio 0.8981, compute saving ratio 0.9349, interval compute saving ratio 0.9260
freeinference.lcp.zst lru 120.00 hour: 244833 requests, block hit ratio 0.9252, interval block hit ratio 0.9127, compute saving ratio 0.9297, interval compute saving ratio 0.9141
freeinference.lcp.zst lru 144.00 hour: 281994 requests, block hit ratio 0.9239, interval block hit ratio 0.9201, compute saving ratio 0.9307, interval compute saving ratio 0.9335
freeinference.lcp.zst lru 168.00 hour: 327548 requests, block hit ratio 0.9175, interval block hit ratio 0.8967, compute saving ratio 0.9259, interval compute saving ratio 0.9118
lru block hit ratio 0.9175 compute saving 0.9259
RESULT trace=freeinference.lcp.zst algo=lru cache_size=65536 ...
```

Each line gives the trace time in hours, the requests served so far, and both
ratios over the run so far and over the interval just closed. The wording
follows cachesim's `interval miss ratio` lines, so a script that parses one
parses the other with a changed regex. The lines go to stdout as they happen,
ahead of each algorithm's summary and `RESULT` lines.

An interval is cut as cachesim cuts it:

- **Trace time** is a request's `timestamp` less the first request's: the
completion time in fi-new and chutes, the arrival time in the Qwen traces.
- **An interval closes** at the first served request at least `s` seconds after
the previous report, and includes that request. The times on the lines are
those requests' times, so they drift past multiples of `s`, and a quiet
stretch of the trace makes one long interval rather than several empty ones.
- **The end of the trace** closes one more interval if anything was served
after the last report, so the intervals always add up to the `RESULT` line.
cachesim stops at its last full interval.
- **Skipped requests** (larger than the cache) are left out of every count, as
everywhere else, and do not close an interval.

The reports are read off the phase-1 statistics and change nothing else: the
`RESULT` line is the same with or without the option. It is off by default.

## Adding an input format

Implement `TraceReader` in `src/trace.cpp` and add one
Expand Down
1 change: 1 addition & 0 deletions doc/quickstart.md
Original file line number Diff line number Diff line change
Expand Up @@ -148,6 +148,7 @@ The other diagnostics are described in
| `--no-pin` | let eviction take blocks the arriving request already holds |
| `--no-verify` | skip the check that every block is resident after each request |
| `--dump-holes <dir>` | write each request's runs of missed blocks, per algorithm |
| `--report-interval <s>` | every `s` seconds of trace time, print both ratios so far and over the last interval, like cachesim's option ([how](design.md#interval-reports---report-interval)) |
| `--block-id raw` | JSONL only: use the trace's hashes as block ids without positional encoding ([when](design.md#block-identity---block-id)) |

---
Expand Down
43 changes: 43 additions & 0 deletions src/main.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ struct Options {
std::string dump_holes_dir;
std::vector<std::string> algorithms{"lru"};
int64_t cache_size = 0;
int64_t report_interval = 0; ///< seconds of trace time; 0 = off
prefixsim::TraceFormat format = prefixsim::TraceFormat::kQwenJsonl;
bool format_given = false; ///< else guessed from the file name
prefixsim::CostModel cost_model; // uniform
Expand Down Expand Up @@ -106,6 +107,10 @@ void print_usage(const char *program) {
printf(
" its recompute holes (runs of missed "
"blocks).\n");
printf(
" --report-interval <s> Every <s> seconds of trace time, print the\n"
" ratios so far and over the last interval,\n"
" as cachesim does (default: 0, off).\n");
printf(
" --output <path> Also append the RESULT lines to this file.\n");
printf(" --list-algos Print supported algorithms and exit.\n");
Expand Down Expand Up @@ -214,6 +219,15 @@ bool parse_args(int argc, char **argv, Options &opts, bool &should_exit) {
opts.dump_holes_dir = value;
} else if (strcmp(arg, "--output") == 0) {
opts.output_path = value;
} else if (strcmp(arg, "--report-interval") == 0) {
char *end = nullptr;
errno = 0;
const long long seconds = strtoll(value, &end, 10);
if (end == value || *end != '\0' || seconds < 0 || errno != 0) {
fprintf(stderr, "error: bad --report-interval '%s' (seconds)\n", value);
return false;
}
opts.report_interval = static_cast<int64_t>(seconds);
} else if (strcmp(arg, "--algo") == 0) {
opts.algorithms = split_commas(value);
} else if (strcmp(arg, "--cache-size") == 0) {
Expand Down Expand Up @@ -288,6 +302,28 @@ std::string holes_path_for(const Options &opts, const std::string &algorithm) {
std::to_string(opts.cache_size) + "_" + algorithm + ".txt";
}

std::string base_name(const std::string &path) {
const size_t slash = path.find_last_of('/');
return slash == std::string::npos ? path : path.substr(slash + 1);
}

/// One interval report, worded like cachesim's "interval miss ratio" line so
/// that scripts reading one read the other with a changed regex.
void print_interval(const std::string &trace, const std::string &algorithm,
const prefixsim::IntervalReport &report) {
printf(
"%s %s %.2f hour: %lld requests, block hit ratio %.4f, interval block "
"hit ratio %.4f, compute saving ratio %.4f, interval compute saving "
"ratio %.4f\n",
trace.c_str(), algorithm.c_str(), report.time_sec / 3600,
static_cast<long long>(report.total.n_requests),
report.total.block_hit_ratio(), report.interval.block_hit_ratio(),
report.total.compute_saving_ratio(),
report.interval.compute_saving_ratio());
// Shown as the run goes, even when stdout is a pipe or a file.
fflush(stdout);
}

/// Fail before the trace is read rather than after. Loading a multi-GB trace
/// takes real time, and discovering a missing output directory on the far side
/// of it wastes the whole run.
Expand Down Expand Up @@ -446,6 +482,13 @@ int main(int argc, char **argv) {
}
config.replay_deepest_first = opts.replay_deepest_first;
config.dump_holes_path = holes_path;
if (opts.report_interval > 0) {
config.report_interval_sec = opts.report_interval;
config.on_report = [trace = base_name(opts.trace_path),
algorithm](const prefixsim::IntervalReport &report) {
print_interval(trace, algorithm, report);
};
}

// A stream is consumed by one algorithm; the next one reopens the file.
if (streaming && reader == nullptr) {
Expand Down
33 changes: 33 additions & 0 deletions src/simulator.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -228,6 +228,7 @@ bool Simulator::run(const std::vector<Request> &requests, std::string &error) {
for (const Request &request : requests) {
if (!serve(request, error)) return false;
}
report_rest();
return true;
}

Expand All @@ -251,12 +252,19 @@ bool Simulator::run(TraceReader &reader, std::string &error) {
while (reader.next(request)) {
if (!serve(request, error)) return false;
}
report_rest();
return true;
}

bool Simulator::serve(const Request &request, std::string &error) {
const size_t n_pos = request.blocks.size();

// Trace time starts at the first request, served or not, as in cachesim.
if (!clock_started_) {
first_timestamp_ = request.timestamp;
clock_started_ = true;
}

alpha_.clear();
for (const obj_id_t id : request.blocks) alpha_.insert(id);

Expand Down Expand Up @@ -375,6 +383,14 @@ bool Simulator::serve(const Request &request, std::string &error) {
}
}
}

if (config_.report_interval_sec > 0) {
last_served_time_ = request.timestamp - first_timestamp_;
if (last_served_time_ - last_report_time_ >=
static_cast<double>(config_.report_interval_sec)) {
report_interval(last_served_time_);
}
}
return true;
}

Expand Down Expand Up @@ -545,6 +561,23 @@ void Simulator::dump_holes(const Request &request) {
fprintf(holes_file_, "\n");
}

void Simulator::report_interval(double time_sec) {
IntervalReport report;
report.time_sec = time_sec;
report.total = stats_;
report.interval = stats_.since(last_report_stats_);
last_report_time_ = time_sec;
last_report_stats_ = stats_;
if (config_.on_report) config_.on_report(report);
}

void Simulator::report_rest() {
if (config_.report_interval_sec > 0 &&
stats_.n_requests > last_report_stats_.n_requests) {
report_interval(last_served_time_);
}
}

void Simulator::access(const Request &request) {
// Reverse order: the deepest (most private) block is touched first and the
// prefix root last, so recency-ordered algorithms leave the root at the MRU
Expand Down
49 changes: 49 additions & 0 deletions src/simulator.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@

#include <cstdint>
#include <cstdio>
#include <functional>
#include <string>
#include <unordered_set>
#include <vector>
Expand Down Expand Up @@ -66,6 +67,33 @@ struct Stats {
double compute_saving_ratio() const {
return total_cost > 0.0 ? saved_cost / total_cost : 0.0;
}

/// What the run did between `earlier`, a snapshot of the same run, and now.
Stats since(const Stats &earlier) const {
Stats d;
d.n_requests = n_requests - earlier.n_requests;
d.n_requests_skipped = n_requests_skipped - earlier.n_requests_skipped;
d.n_blocks = n_blocks - earlier.n_blocks;
d.n_block_hits = n_block_hits - earlier.n_block_hits;
d.total_cost = total_cost - earlier.total_cost;
d.saved_cost = saved_cost - earlier.saved_cost;
d.n_evictions = n_evictions - earlier.n_evictions;
d.n_self_evictions = n_self_evictions - earlier.n_self_evictions;
d.n_unexpected_evictions =
n_unexpected_evictions - earlier.n_unexpected_evictions;
d.n_unobserved_eviction_rounds =
n_unobserved_eviction_rounds - earlier.n_unobserved_eviction_rounds;
return d;
}
};

/// One interval report; see SimulatorConfig::report_interval_sec.
struct IntervalReport {
/// Trace time of the request that closed the interval, in seconds since the
/// trace's first request.
double time_sec = 0.0;
Stats total; ///< The run so far.
Stats interval; ///< Since the previous report.
};

struct SimulatorConfig {
Expand Down Expand Up @@ -99,6 +127,14 @@ struct SimulatorConfig {
/// holes -- the contiguous runs of blocks it had to recompute. Empty
/// disables it. See Simulator::dump_holes.
std::string dump_holes_path;
/// Trace time, in seconds, between interval reports to `on_report`; 0
/// disables them. The rule is cachesim's --report-interval: time is a
/// request's `timestamp` less the trace's first, and a served request that
/// lands at least this long after the previous report closes an interval
/// that includes it. What follows the last such request is reported when the
/// trace ends, which cachesim does not do. Skipped requests close nothing.
int64_t report_interval_sec = 0;
std::function<void(const IntervalReport &)> on_report;
};

/// Whether the phase-2 pin predicate (SimulatorConfig::pin_request_blocks) is
Expand Down Expand Up @@ -149,10 +185,23 @@ class Simulator {
/// Write this request's holes, one line per request (see dump_holes()).
void dump_holes(const Request &request);

/// Close an interval at trace time `time_sec` (see report_interval_sec).
void report_interval(double time_sec);

/// Report what the last interval left unreported, at the end of a run.
void report_rest();

cache_t *cache_ = nullptr;
SimulatorConfig config_;
Stats stats_;

// Interval reports, in seconds since the first request's timestamp.
bool clock_started_ = false;
double first_timestamp_ = 0.0;
double last_served_time_ = 0.0;
double last_report_time_ = 0.0;
Stats last_report_stats_;

request_t *req_buf_ = nullptr;
prefetcher_t *recorder_ = nullptr;
FILE *holes_file_ = nullptr;
Expand Down
Loading
Loading