Skip to content

docs: add a readme for running the dabstep benchmark - #9

Merged
smithclay merged 1 commit into
mainfrom
docs/benchmark-readme
Oct 13, 2026
Merged

smithclay merged 1 commit into
mainfrom
docs/benchmark-readme

Conversation

@smithclay

Copy link
Copy Markdown
Owner

Adds benchmarks/dabstep/README.md, a step-by-step guide to running the benchmark, linked from the root README's layout section.

What it covers

  1. What the two arms are (baseline, codemode-git-examples), the defaults (glm-5.3-flash, 20 steps), and the dabstep-full1 tag for the reported run.
  2. Data: scripts/fetch.sh gets the context, the task splits and Adyen's official scorer (CC BY 4.0, gitignored).
  3. D1: build_sql.py, cf d1 create, and loading the SQL. Notes that the repo's D1 id belongs to the original database.
  4. Artifacts namespace and Vectorize index, linking to the root README for the index commands.
  5. The harness Worker: local cf dev --port 8790, or deployed with HARNESS_TOKEN set through --secrets-file so it's never open. Plus the /selftest check.
  6. Running: a smoke test, then the exact full-run commands for both arms (rolling concurrency 8), about 8.5 h per arm and $26.76 in Workers AI. Resumability, retries, the token budget guard, and why task order matters.
  7. Scoring: score.py and submitting to the leaderboard.
  8. Analysis: compare.py and learning_curve.py.
  9. Smaller slices and the unit tests.

Verification

  • Commands, flags and defaults were checked against scripts/*, src/index.ts and cloudflare.config.ts. Wall time comes from the full1 results; cost from docs/experiments.md.
  • npm test --workspace benchmarks/dabstep passes (7 tests). The dabstep-full1 tag exists.
  • Not re-run: loading the 25 MB SQL with cf d1 migrations apply; only a dry run was checked. The original database was loaded before this README existed. The README says so and gives wrangler d1 execute --remote --file as the fallback.

Covers fetching the data and official scorer, loading the context into D1,
the Artifacts and Vectorize resources, running the harness Worker locally
or deployed (HARNESS_TOKEN), the self-test, both arms of the full run
(flags, time, cost, resumability), scoring and leaderboard submission,
the paired comparison and learning curve, and the smaller slices.
@smithclay
smithclay merged commit b368a67 into main Oct 13, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant