Serves GGUFs through engined, which owns the pinned llama.cpp build and model lifecycle. Benchmarks quality, latency, and cost (energy) across tasks × prompt_variants × models. Judge modes: pairwise_all tournament or scored rubric. Emits JSON, Markdown, and a single-file HTML dashboard under results/.
./run.shPrerequisites, install, and configuration: HUMANS.md.
- Served through
engined's pinned llama.cpp Vulkan build; OpenAI-compatible client pairwise_alltournament orscored1–5 rubric; heuristic fallbacks- Quality, latency, and energy-based cost metrics
- JSON, Markdown, and single-file HTML dashboard output
- Any number of models; deterministic seeded synthetic dataset
./run.sh fetchpulls missing GGUFs from Hugging Face- Optional distributed worker (
python -m bench.worker) pulls jobs from a bakeoff-results queue
| Document | Audience | Contents |
|---|---|---|
| HUMANS.md | Operators & developers | Prerequisites, install, run, configure, troubleshoot, clean up |
| AGENTS.md | LLMs & contributors | Design invariants, hardware caveats, judge-mode selection, editing conventions |
| CONTRIBUTING.md | Contributors | PR checklist, commit style |
| config.yaml | Reference | Server, models, prompts, dataset, judge, cost, output (inline comments) |
Licensed under the terms in LICENSE.