Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,9 @@ uv run synth_lib/preparation/market_data.py --days 7

`--days N` downloads today and the N days before it (N + 1 daily partitions).
Today's partition is marked `is_final=False` and is re-downloaded on every
subsequent run.
subsequent run. A past day that starts before the venue's retention (Hyperliquid
keeps its last 5000 minutes) comes back partial: it is stored once with
`is_final=False`, for an archive ingest to complete, and not downloaded again.

### Re-download existing partitions

Expand Down
33 changes: 21 additions & 12 deletions synth_lib/benchmark/constitution/CAMPAIGN.md.tmpl
Original file line number Diff line number Diff line change
Expand Up @@ -64,8 +64,9 @@ are deployed as miners on the subnet, and emission rewards two things a mean CRP

- **Every prompt answered.** A request that times out, raises, or returns a wrong path count is a
miss, and a miss is scored like the field's 95th percentile on that prompt.
- **Contexts rougher than the snapshot.** A live context can hold longer gaps, and a shorter real
history after the miner restarts.
- **Contexts rougher than the snapshot.** A live context can hold longer gaps, and fewer real days
than your `CONTEXT_DAYS`: some venues serve only their last 3 days of minutes, so on a new miner
the oldest days of a longer context can be NaN.

None of this makes position a target: emission is earned by a low CRPS on every prompt, with no
exceptions.
Expand Down Expand Up @@ -124,8 +125,8 @@ horizon before the window and yours is never treated as late.
**Read `DATA.md` at the workspace root FIRST**: it is generated from this exact snapshot at
setup and carries the measured truth — the partition schema, the minute semantics (a minute
without a trade is a flat candle with `trade_count` 0, NOT market hours; NaN close = no price
at all; missing file = before the market's first trade), and a
per-asset coverage table with density by era. Two caveats it quantifies that will bite if
at all; missing file = before the market's first trade, or before the snapshot's first day),
and a per-asset coverage table with density by era. Two caveats it quantifies that will bite if
ignored: the early months of the newer Hyperliquid listings are genuinely THIN (low realized
volatility there is real, not smoothing), and these are venue-specific perpetual prices with
their own basis (xyz:SP500 trades around 7,500 — not the cash index).
Expand Down Expand Up @@ -264,11 +265,19 @@ horizon before the window and yours is never treated as late.
`simulate(asset, start_time, time_increment, time_length, num_simulations, context_prices)`
-> tuple `(start_timestamp, time_increment, *paths)` — each path: a list of
`time_length//time_increment + 1` floats starting at the last price in the context.
**The `context_prices` contract, exactly**: a minute-indexed pd.DataFrame over the 7 days
preceding `start_time`, with columns `open, high, low, close, volume, trade_count`. `close`
is the series you are scored against; the rest are context. It MAY CONTAIN NaN in every
column — your `simulate()` must tolerate them (drop, never interpolate). Evaluation passes
the raw frame: handle gaps inside your model, not in a loader you hope evaluation shares.
**The `context_prices` contract, exactly**: a minute-indexed pd.DataFrame over the
`CONTEXT_DAYS` days preceding `start_time`, with columns `open, high, low, close, volume,
trade_count`. `close` is the series you are scored against; the rest are context. It MAY
CONTAIN NaN in every column, on whole days before an asset's first trade or before the
snapshot's first day too — your `simulate()` must tolerate them (drop, never interpolate).
Evaluation passes the raw frame: handle gaps inside your model, not in a loader you hope
evaluation shares.
**`CONTEXT_DAYS` is yours to choose**, and to measure like any other parameter: a plain
integer at module level in `agent/modeling.py`, from 3 to 30, not computed (a module without
it gets 7). `agent/predict.py`, the interface check, the scoring and the live miner all read
it, and any other value fails every one of them. One value serves every competition: slice a
shorter context inside `simulate()` where you want one. The interface check reports the wall
time of one call at your length.
Runnable OFFLINE (weights/artifacts committed or included in the workspace).
8. **If your model uses randomness at all, seed it from `SYNTH_BENCHMARK_SEED`.** Nothing
here asks you to introduce randomness — a deterministic champion ignores this rule. If you
Expand Down Expand Up @@ -321,10 +330,10 @@ by the snapshot. Lessons earlier agents paid for:
- **Test a feature by what it adds to a fast EWMA of realized volatility**, not by its own
predictive power. Range estimators, volume, trade counts and the leverage effect act on future
volatility through realized volatility, which the EWMA has already priced. What it cannot hold
is information outside the context: the calendar, and anything older than the 7 days
is information outside the context: the calendar, and anything older than the `CONTEXT_DAYS`
`simulate()` receives.$calendar_method
- **Check what `simulate()` can see before building on it**: one asset's 7-day context per
call, nothing else. A feature that needs another asset's current prices cannot be served.
- **Check what `simulate()` can see before building on it**: one asset's `CONTEXT_DAYS` context
per call, nothing else. A feature that needs another asset's current prices cannot be served.
- **Count the asset-window cells where a change helps**, across disjoint windows, not only its
average: a small gain signed the same way in most cells survives; a larger one that flips sign
between windows is noise.
Expand Down
84 changes: 69 additions & 15 deletions synth_lib/benchmark/generate_predictions.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,11 @@

Predictions land one file per day, under
`<out>/<asset>/<time_length>_<time_increment>/date=<YYYY-MM-DD>.npy` — float32, shaped
(prompts, simulations, steps) — beside a `.json` naming the prompt start_times it holds. A day
whose file already covers every requested start_time is skipped, so pointing --out-dir at a
directory that survives the run makes a re-score reuse the paths instead of regenerating them.
(prompts, simulations, steps) — beside a `.json` naming the prompt start_times it holds, the
champion's CONTEXT_DAYS and `context_head_nan_minutes`, the most minutes one of the day's contexts
runs before its first real close. A day whose file already covers every requested start_time is
skipped, so pointing --out-dir at a directory that survives the run makes a re-score reuse the paths
instead of regenerating them, with the contexts they were generated from.

Usage:
python generate_predictions.py --modeling agent/modeling.py --asset BTC \\
Expand All @@ -35,14 +37,18 @@
import importlib.util
import json
import sys
from datetime import timedelta
from datetime import date, timedelta
from pathlib import Path
from types import ModuleType
from typing import Callable

import numpy as np
import pandas as pd

CONTEXT_MINUTES = 7 * 24 * 60
# The champion's context length: `CONTEXT_DAYS` in its modeling.py, a plain int within these bounds,
# and this default when the module does not set it.
DEFAULT_CONTEXT_DAYS = 7
CONTEXT_DAYS_BOUNDS = (3, 30)
# Duplicated from synth_lib.preparation.config rather than imported: this file runs inside a
# champion's own venv, which does not have synth_lib.
OHLCV_COLUMNS = ["open", "high", "low", "close", "volume", "trade_count"]
Expand All @@ -65,6 +71,28 @@ def context_end(start: pd.Timestamp, time_length: int, issuance_lead: int | None
return start - pd.Timedelta(seconds=lead + BAR_SECONDS)


def context_slice(
frame: pd.DataFrame, start: pd.Timestamp, time_length: int, context_days: int, issuance_lead: int | None = None
) -> pd.DataFrame:
"""The context of the prompt at `start`: the bars from `context_days` before it to `context_end`."""
return frame.loc[start - pd.Timedelta(days=context_days) : context_end(start, time_length, issuance_lead)]


def nan_head_minutes(context: pd.DataFrame) -> int:
"""How many minutes the context runs before its first real close."""
real = context["close"].notna().to_numpy()
return int(real.argmax()) if real.any() else len(real)


def context_days_of(module: ModuleType) -> int:
"""The champion's `CONTEXT_DAYS`, or the default when its module does not set it."""
value = getattr(module, "CONTEXT_DAYS", DEFAULT_CONTEXT_DAYS)
low, high = CONTEXT_DAYS_BOUNDS
if type(value) is not int or not low <= value <= high:
raise ValueError(f"CONTEXT_DAYS must be a plain int from {low} to {high} in modeling.py, got {value!r}")
return value


# What simulate() receives as context_prices: the minute OHLCV frame, or the Series of closes (NaN
# kept) of the contract before OHLCV, for a champion written against that one.
CONTEXT_MODES = ("ohlcv", "close")
Expand Down Expand Up @@ -95,7 +123,7 @@ def covered_start_times(index_path: Path, paths_path: Path) -> set[str]:
return set() # truncated by an interrupted write; regenerate the day


def load_simulate(modeling_path: Path) -> Callable:
def load_module(modeling_path: Path) -> ModuleType:
# A champion imports the modules beside it by name, as when its folder is the script's own.
folder = str(modeling_path.resolve().parent)
if folder not in sys.path:
Expand All @@ -104,7 +132,11 @@ def load_simulate(modeling_path: Path) -> Callable:
assert spec is not None and spec.loader is not None # always true for a .py file path
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return module.simulate
return module


def load_simulate(modeling_path: Path) -> Callable:
return load_module(modeling_path).simulate


def store_root(data_root: Path, asset: str) -> Path:
Expand All @@ -115,11 +147,16 @@ def store_root(data_root: Path, asset: str) -> Path:


def load_minute_prices(data_root: Path, asset: str, start: pd.Timestamp, end: pd.Timestamp) -> pd.DataFrame:
"""Minute OHLCV over [start, end] from daily partitions. Missing partitions raise: a silent
"""Minute OHLCV over [start, end] from daily partitions.

The days before the asset's first partition, before the market's first trade or the store's first
day, are NaN on every minute, as in a live context. A missing partition after it raises: a silent
hole here would shrink every context that spans it without anyone noticing."""
root = store_root(data_root, asset)
stored = (date.fromisoformat(p.stem.removeprefix("date=")) for p in root.glob("date=*.parquet"))
first_day = min(stored, default=start.date())
frames = []
day = start.date()
day = max(start.date(), first_day)
while day <= end.date():
path = root / f"date={day.isoformat()}.parquet"
if not path.exists():
Expand All @@ -132,9 +169,18 @@ def load_minute_prices(data_root: Path, asset: str, start: pd.Timestamp, end: pd
f"skips settled partitions that already exist."
) from exc
day += timedelta(days=1)
if not frames:
raise FileNotFoundError(
f"no partition for {asset} from {start.date()} to {end.date()}: its first is {first_day}"
)
frame = pd.concat(frames, ignore_index=True)
frame["timestamp"] = pd.to_datetime(frame["timestamp"], utc=True)
frame = frame.set_index("timestamp").sort_index()
if start.date() < first_day:
head = pd.date_range(
pd.Timestamp(start.date(), tz="UTC"), frame.index[0], freq="1min", inclusive="left", name="timestamp"
)
frame = pd.concat([pd.DataFrame(np.nan, index=head, columns=OHLCV_COLUMNS), frame])
return frame.loc[start:end]


Expand Down Expand Up @@ -169,6 +215,7 @@ def generate_day(
num_simulations: int,
issuance_lead: int | None = None,
context_mode: str = "ohlcv",
context_days: int = DEFAULT_CONTEXT_DAYS,
) -> int:
"""Generate one day's prompts and write the pair. Returns how many were generated, 0 if the
day was already covered.
Expand All @@ -182,12 +229,12 @@ def generate_day(
return 0

runs = []
head = 0
for t in times:
# THE no-lookahead line: the context stops at the last bar the prompt's issuance allows,
# never at t — the bar labelled t closes after the prompt starts.
context = price_frame.loc[
t - pd.Timedelta(minutes=CONTEXT_MINUTES) : context_end(t, time_length, issuance_lead)
]
context = context_slice(price_frame, t, time_length, context_days, issuance_lead)
head = max(head, nan_head_minutes(context))
if context_mode == "close":
context = context["close"]
out = simulate_fn(
Expand All @@ -212,6 +259,8 @@ def generate_day(
"time_length": time_length,
"num_simulations": int(stacked.shape[1]),
"num_steps": int(stacked.shape[2]),
"context_days": context_days,
"context_head_nan_minutes": head,
}
)
)
Expand All @@ -232,6 +281,7 @@ def generate(
prompt_times: list[pd.Timestamp] | None = None,
issuance_lead: int | None = None,
context_mode: str = "ohlcv",
context_days: int = DEFAULT_CONTEXT_DAYS,
) -> tuple[int, int]:
"""Returns (generated, reused) prompt counts."""
by_day: dict[str, list[pd.Timestamp]] = {}
Expand All @@ -252,6 +302,7 @@ def generate(
num_simulations,
issuance_lead,
context_mode,
context_days,
)
generated += made
reused += 0 if made else len(times)
Expand Down Expand Up @@ -301,9 +352,11 @@ def main() -> None:
prompt_times = None
if args.prompt_times is not None:
prompt_times = [_utc(t) for t in json.loads(args.prompt_times.read_text())]
prices = load_minute_prices(args.data_root, args.asset, start - pd.Timedelta(minutes=CONTEXT_MINUTES), end)
module = load_module(args.modeling)
days = context_days_of(module)
prices = load_minute_prices(args.data_root, args.asset, start - pd.Timedelta(days=days), end)
generated, reused = generate(
load_simulate(args.modeling),
module.simulate,
args.asset,
start,
end,
Expand All @@ -316,8 +369,9 @@ def main() -> None:
prompt_times=prompt_times,
issuance_lead=args.issuance_lead_seconds,
context_mode=args.context,
context_days=days,
)
print(f"{args.asset} tl={args.time_length}: {generated} predictions ({reused} reused)")
print(f"{args.asset} tl={args.time_length}: {generated} predictions ({reused} reused), CONTEXT_DAYS {days}")


if __name__ == "__main__":
Expand Down
43 changes: 31 additions & 12 deletions synth_lib/benchmark/nomination.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,12 +7,16 @@
import os
import subprocess
import sys
import time
from dataclasses import dataclass
from pathlib import Path
from types import ModuleType

import pandas as pd
import yaml

from synth_lib.benchmark.generate_predictions import context_days_of, context_slice, load_module

REPO_ROOT = Path(__file__).resolve().parent.parent


Expand Down Expand Up @@ -84,37 +88,44 @@ def load_simulate(modeling_path: Path):


def probe_simulate(
modeling_path: Path,
module: ModuleType,
*,
asset: str = "BTC",
time_increment: int = 300,
time_length: int = 86_400,
num_simulations: int = 1000,
) -> tuple[bool, str]:
"""Checks the shape of the contract's return value. Synthetic context => no network needed here;
real network isolation is provided by the calling sandbox (--network none).
"""Checks the shape of the contract's return value, for the champion's loaded modeling module.
Synthetic context => no network needed here; real network isolation is provided by the calling
sandbox (--network none).

num_simulations defaults to the validator's real serving size
(synth.validator.prompt_config.PromptConfig.num_simulations = 1000) rather than a smaller
convenience value: a champion that breaks at 1000 paths (memory, a hardcoded shape) must
fail here, at nomination time, instead of later at verdict/live time where
synth.validator.response_validation_v2 rejects any response with a different path count."""
synth.validator.response_validation_v2 rejects any response with a different path count.

The context has the shape the verdict hands: the champion's CONTEXT_DAYS, cut by context_slice. On
success the message is the call's wall time."""
try:
simulate = load_simulate(modeling_path)
days = context_days_of(module)
start = pd.Timestamp("2026-01-05T00:00:00Z")
idx = pd.date_range(end=start, periods=7 * 24 * 60, freq="1min", tz="UTC")
context = pd.DataFrame(
idx = pd.date_range(start - pd.Timedelta(days=days), start, freq="1min")
frame = pd.DataFrame(
{c: 100.0 for c in ("open", "high", "low", "close")} | {"volume": 1.0, "trade_count": 1.0},
index=idx,
)
out = simulate(
context = context_slice(frame, start, time_length, days)
began = time.monotonic()
out = module.simulate(
asset=asset,
start_time=start.isoformat(),
time_increment=time_increment,
time_length=time_length,
num_simulations=num_simulations,
context_prices=context,
)
seconds = time.monotonic() - began
if not isinstance(out, tuple) or len(out) != 2 + num_simulations:
got = f"tuple of length {len(out)}" if isinstance(out, tuple) else type(out).__name__
return False, f"expected return: tuple(start, increment, *{num_simulations} paths) — got {got}"
Expand All @@ -124,7 +135,7 @@ def probe_simulate(
return False, f"path {i}: length {len(path)} != {expected_steps} or non-numeric values"
except Exception as exc: # any champion failure (including a malformed shape) is a probe failure
return False, f"simulate() raised: {type(exc).__name__}: {exc}"
return True, "ok"
return True, f"{seconds:.1f} s"


def probe_simulate_subprocess(
Expand Down Expand Up @@ -178,16 +189,24 @@ def probe_simulate_subprocess(


def probe_all_shapes(modeling_path: Path) -> tuple[bool, str]:
"""Probes BOTH competition formats: 24h (86400/300) and 1h (3600/60).
"""Probes BOTH competition formats: 24h (86400/300) and 1h (3600/60), and reports the champion's
CONTEXT_DAYS with the wall time of one call of each.

A champion that ignores time_increment/time_length passes the 24h format and breaks on
crypto-1h — the check must therefore cover both, otherwise the failure only shows up at
verdict time, too late for the agent to fix it."""
try:
module = load_module(modeling_path)
days = context_days_of(module)
except Exception as exc: # an import that fails, or a CONTEXT_DAYS out of bounds
return False, f"{type(exc).__name__}: {exc}"
seconds = {}
for label, ti, tl in (("24h", 300, 86_400), ("1h", 60, 3_600)):
ok, msg = probe_simulate(modeling_path, time_increment=ti, time_length=tl)
ok, msg = probe_simulate(module, time_increment=ti, time_length=tl)
if not ok:
return False, f"[{label}] {msg}"
return True, "ok (24h + 1h)"
seconds[label] = msg
return True, f"ok (24h + 1h), CONTEXT_DAYS {days}, one call {seconds['24h']} (24h) and {seconds['1h']} (1h)"


def _main() -> None:
Expand Down
3 changes: 3 additions & 0 deletions synth_lib/benchmark/scaffold/modeling_starter.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,9 @@

import numpy as np

# Days of minute history simulate() receives as context_prices: a plain int from 3 to 30.
CONTEXT_DAYS = 7


def simulate(
asset: str,
Expand Down
3 changes: 3 additions & 0 deletions synth_lib/benchmark/scaffold/workspace/agent/modeling.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,9 @@

import numpy as np

# Days of minute history simulate() receives as context_prices: a plain int from 3 to 30.
CONTEXT_DAYS = 7


def simulate(
asset: str,
Expand Down
Loading
Loading