A quant research benchmark whose leaderboard is a live market.
Give a coding agent a sandbox, $100 of credit and a day. Its task: build a model that beats professional quants who are being paid, right now, to beat it.
The task is real work — the kind that normally takes a skilled quant months of open-ended research. The opponents are ~250 live miners on Synth SN50, financially incentivised to win. The scorer is the subnet's own validator code. The metric is money.
Still unsolved. No frontier model, given one shot inside that envelope, has taken the top spot. Over a month of live scoring, the best champion earns, averaged over the three competitions, 38.5% of what the best other miner earns. It reached 1st in crypto-1h and crypto-24h, but averaged 134th and 101st there over the month, and 146th in com-equ-24h.
Every champion is scored the same way: registered as a new miner on its campaign's data cutoff, then run for the next 30 days on the prompts the validator actually sent, through the validator's own aggregation.
| Model | Campaign | Window | Score | crypto-1h | crypto-24h | com-equ-24h | Spend |
|---|---|---|---|---|---|---|---|
| gemini-4-argon | campaign-6 | 2026-10-11 → 11-10 | 38.5 | 134 | 101 | 146 | 15.9 h / $70.95 |
| claude-opus-5 | campaign-2 | 2026-07-27 → 08-26 | 33.5 | 106 | 91 | 169 | 3.8 h / $25.67 |
| claude-opus-5-5 | campaign-4 | 2026-09-14 → 10-14 | 29 | 146 | 199 | 105 | 2.0 h / $8.10 |
| gpt-6-astra | campaign-4 | 2026-09-14 → 10-14 | 21.4 | 149 | 156 | 200 | 0.8 h / $18.32 |
| claude-opus-5-5 | campaign-6 | 2026-10-11 → 11-10 | 14.3 | 190 | 171 | 180 | 3.1 h / $11.78 |
| claude-fable-5-1 | campaign-3 | 2026-08-29 → 09-28 | 13.5 | 144 | 180 | 182 | 2.9 h / $58.55 |
| gpt-6-astra | campaign-6 | 2026-10-11 → 11-10 | 13.4 | 180 | 213 | 192 | 4.8 h / $88.41 |
| gpt-6-astra | campaign-3 | 2026-08-29 → 09-28 | 11.6 | 164 | 149 | 211 | 3.2 h / $93.60 |
| gemini-3.1-pro-preview | campaign-2 | 2026-07-27 → 08-26 | 6.4 | 198 | 243 | 197 | — |
| muse-wonder-1.0 | campaign-6 | 2026-10-11 → 11-10 | 5.9 | 225 | 174 | 195 | 3.7 h / $7.16 |
| gpt-5 | campaign-2 | 2026-07-27 → 08-26 | 5.4 | 198 | 252 | 198 | — |
| claude-opus-5 | campaign-4 | 2026-09-14 → 10-14 | 4.9 | 230 | 220 | 231 | 12.2 h / $86.68 |
| gpt-6-sol | campaign-4 | 2026-09-14 → 10-14 | 3.5 | 236 | 214 | 240 | 15.4 h / $4.72 |
| gpt-5.6-sol | campaign-3 | 2026-08-29 → 09-28 | 0.5 | 240 | 249 | 245 | 4.3 h / $23.55 |
| gemini-3.8-flash | campaign-3 | 2026-08-29 → 09-28 | 0.1 | 251 | 250 | 246 | 0.3 h / $1.51 |
Score = 100 × mean over the three competitions of min(1, the champion's simulated emissions ÷ the best other miner's) — see docs/scoring.md. Each competition column is the champion's mean rank over the window's scoring rounds, in a field of up to 257 miners. One row per champion: a model that entered more than one campaign has a row for each. Spend is wall time and credits from the leg's own budget file, out of its campaign's envelope; a restarted leg reports only its final attempt.
Generated with synth-bench-leaderboard --tag cutoff-30d --all-legs --write README.md from each
leg's cutoff-30d verdict; every number is read from an archived verdict, never typed by hand.
- The opponents are paid to win. Not a held-out test set: a live field of professional quants competing for real emissions, who improve every week.
- It cannot be trained for. Scoring runs on market data that did not exist when the model was written, and the field moves. There is no answer key to memorise and no static set to overfit.
- It is ongoing. Every campaign is scored on a fresh window, so a result is a claim about a specific month that anyone can re-verify.
- The skill transfers. What it takes to win — calibrated uncertainty on real price paths under a deadline — is what any serious modelling job needs. A system that wins here is a system you can point at other well-defined modelling work.
- Anyone can enter. Bring your own model, agent framework or harness; the engine is public and a dry run costs nothing. A champion that reaches the top of the field earns subnet emissions — and the bragging rights of having beaten paid quants with an agent.
Beat it and you have a runnable, provably-scored AI quant, plus the receipts.
The task. One campaign runs one leg per model, sequentially on identical hardware. A leg gets a deliberately minimal starter model, a frozen snapshot of 1-minute prices, the subnet's official backtester, and a constitution: no lookahead, prices from the snapshot only, deliverables (journal, report, champion commit), and the scoring formula — pre-registered, so agents design for the metric instead of guessing it. docs/task.md.
The envelope. A fixed credit budget and wall clock, metered through one proxy ledger so every model is measured with the same instrument. At 85% of either the harness issues a non-negotiable landing order; nominating is final and forfeits the rest. Where an agent chooses to stop is part of what is measured.
The score. The champion is injected into the archived field and run through the validator's own aggregation. Because the subnet pays via a softmax, the Score is deliberately top-heavy: mid-pack earns ≈ 0, so ranks are published beside it — per competition and per asset — to show how close a zero was. docs/scoring.md.
Reproducibility. Every campaign publishes its config, the constitution each agent read, its journal and report, the champion's source at the nominated commit, and the verdict JSON. Rescore any champion yourself: docs/reproduce.md. Or deploy one as a live miner: docs/deploy-a-champion.md.
| Campaign | Panel | Envelope |
|---|---|---|
| campaign-1 | gemini-3.1-pro-preview, glm-5.2, kimi-k3 | $100 / 24 h per model |
| campaign-2 | claude-opus-5, gpt-5, revised constitution | $100 / 18 h per model |
| campaign-3 | claude-fable-5-1, gpt-5.6-sol, gpt-6-astra, gemini-3.8-flash | $100 / 18 h per model |
| campaign-4 | claude-opus-5, claude-opus-5-5, gpt-6-astra, gpt-6-sol | $100 / 18 h per model |
| campaign-6 | claude-opus-5-5, gpt-6-astra, gemini-4-argon, muse-wonder-1.0 | $100 / 18 h per model |
| campaign-7 | claude-opus-5-5, gpt-6-astra, gemini-4-argon, kimi-k3.1-pro, claude-fable-5-2, gemini-4-carbon | $100 / 18 h per model |
Each leg directory is the champion: the agent's whole workspace at the commit it nominated —
modeling.py and the calibration artifacts it loads, report.md (the agent's own write-up, the most
interesting artifact here), journal.md (one entry per iteration, failures included), its research
scripts — plus CHAMPION (the nomination record), the verdict (verdict.json in the archived legs,
verdict/verdict.json with its charts and score frames in newer ones) and any verdict-<tag>
diagnostic windows, and the serving shell miner.py / entrypoint.sh that runs it as a live miner.
From campaign-4 on, the workspace sits in the leg's champion_source/ and the serving shell is
generated from it at deploy time (synth_lib.serving.unpack_champion).
From campaign-7 on, a leg splits the workspace in three:
champion_source/holds only the files the champion reads while serving.SERVING.jsonlists each one with its hash and why it is there, and the calls that proved the set: served alone, it gives the same output as the whole tree at the champion's commit on every call.research_source/holds the research and training code behind the champion, withRESEARCH.json.workspace.bundleholds everything the agent committed:git clone workspace.bundle ws && git -C ws checkout <CHAMPION sha>.
campaign-7 is not on the leaderboard: its champions run as live miners rather than through a verdict.
campaign-2 also carries three older in-sample windows per leg. They are the overfitting check, not a second leaderboard: they showed the 1-hour edge decaying with distance from the data cutoff while the 24-hour model held up. Read them with docs/scoring.md.
The engine is synth_lib.benchmark; this repo is the
campaigns and results. uv sync, then docs/run-a-campaign.md. A full panel
needs a GPU-capable box, provider keys and a few hundred dollars; a cli: fake dry run costs nothing.
- Four windows. Scores are relative to whoever was mining that month: campaign-2 is scored on 2026-07-27 → 08-26, campaign-3 on 08-29 → 09-28, campaign-4 on 09-14 → 10-14 and campaign-6 on 10-11 → 11-10, so only rows from the same campaign are a head-to-head, even for the same model.
- Pseudo-out-of-sample. A window starts on its campaign's data cutoff, the last day of the agents' price snapshot, and the agents could query the live field's API while they worked.
- campaign-2 is scored from rebuilt workspaces. Its original workspaces were not kept, so each
champion was rebuilt from its leg directory as one commit (the
champion_shaits verdict records, not the one inCHAMPION), fed the minute closes it was written for. - One box, sequential legs. No interference between models, but no control over regime differences between them either.
- Constitutions, journals and reports are published verbatim, as the agents received and wrote them, with one exception: rule 5 of the constitution named a directory in the operator's own research repository, and that path is redacted here. Nothing else was edited.
MIT (this repo). Agent-authored reports and journals are published as written.