Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 25 additions & 15 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,20 +4,28 @@ This is the canonical repository guidance for coding agents and automated
contributors. Tool-specific instruction files should point here instead of
duplicating policy.

## Checkpoint review branch

This branch prepares a focused foundation PR. Read
[v1-sprints/foundation-checkpoint-pr.md](v1-sprints/foundation-checkpoint-pr.md)
and the [public checkpoint scope](docs/v1/checkpoint.md) before using older
execution pointers below. The complete development history and its raw archives
remain in the original development checkout; omitted tests and pending candidate
validation are listed in the checkpoint manifest. The user explicitly requests
merging PR 27 after its declared candidate checks pass. Read the
[merge validation record](v1-sprints/checkpoint-merge-validation.md) for the frozen
commands and result location. Heavy CPU tests, builds, numerical audits and CUDA
run on bounded Modal resources; GitHub only verifies committed result bindings.
Local work is source/metadata inspection only. This merge is not a release or
engineering-v1 completion.
## Current position

PR 27 merged the [foundation checkpoint](v1-sprints/foundation-checkpoint-pr.md)
into main as `5ac60ee`. Read its [public scope](docs/v1/checkpoint.md) and
[merge validation record](v1-sprints/checkpoint-merge-validation.md) before using
older execution pointers below. The complete development history and its raw
archives remain in the original development checkout; omitted tests are listed in
the checkpoint manifest. That merge is not a release or engineering-v1 completion.

The hosted gate binds the committed Modal receipts to every tracked file outside
`docs/v1/evidence/pr27-modal-validation/`. Any other tracked change fails that gate
until bounded Modal validation of the new candidate is committed. Gate evidence for
heavy CPU tests, builds, numerical audits and CUDA comes from bounded Modal
resources; GitHub only verifies committed result bindings. Local runs are
development checks, not gate evidence.

Active work is [Sprint 080](v1-sprints/080-cuda-required-recipes.md). All eight
required device recipes are implemented. The
[required-matrix audit](benchmarks/v1/evidence/required-device-080/README.md) is
open for every cell: 322 of 1,362 bound cases have real T4 execution at the merged
source and 1,040 are collected but unexecuted. The next step is one frozen
real-device run of those cases; it needs a concrete hardware and upload allowance.

## Mission

Expand Down Expand Up @@ -382,7 +390,9 @@ failures and captures the false-improvement mechanism; its two diagnostics pass.
Installed D2 and nineteen CPU replays pass; full Normal conformance remains open.
Binary/Poisson objectives, loss-change comparisons and shared scalar recipes now
pass the bounded 153-case T4 matrix at `fe12beb`, with 418 regressions and retained
CPU inference evidence. Required multiclass, AFT and vector topology remain open.
CPU inference evidence. Multiclass, AFT and independent/shared squared vector
recipes pass their recipe-level T4 cohorts at `312d821`; their operation-level
cases are unexecuted at that source, so every required cell remains open.
All A1–A13 real evaluations and formal author/quality/cost gates remain open.

The user approved B03–B06 construction overlapping unfinished F0.3; see Sprint 018
Expand Down
89 changes: 89 additions & 0 deletions benchmarks/v1/evidence/required-device-080/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
# Required CUDA matrix audit (Sprint 080)

Verdict: **open**. No required cell passes. This audit executes nothing; it judges
the real T4 evidence already retained for merged source
`312d8213900c0e4b35e7bdcac3d92d74ca5dbb81` against the
[required device matrix](../../../../v1-sprints/080-required-device-matrix.json).

Of 1,900 collected GPU-marked cases, the matrix binds 1,362 to the eight required
cells and their shared operations. The retained T4 run executed 441 cases, all
passing; 322 of them are bound. The other 1,040 bound cases are collected but
have no execution at this source. No bound case failed or skipped.

| Cell | objective | domains | comparison | rounds | recipe | inference | residency | ownership |
|---|---|---|---|---|---|---|---|---|
| R1.squared | 0/72 | 0/12 | 0/5 | 0/36 | 0/13 | 0/2 | 0/37 | 0/5 |
| R1.binary | 0/2 | 0/28 | 0/31 | 0/3 | 11/11 | 8/11 | 0/32 | 5/13 |
| R1.multiclass | 0/4 | 0/24 | 0/65 | 0/9 | 30/30 | 24/33 | 0/68 | 6/20 |
| R4.poisson | 0/2 | 0/28 | 0/30 | 0/3 | 11/11 | 8/11 | 0/31 | 5/13 |
| R5.aft | 0/8 | 0/44 | 0/63 | 0/12 | 39/39 | 2/14 | 0/71 | 5/19 |
| R6.normal | 0/11 | 0/19 | 0/112 | 90/90 | 90/100 | 1/1 | 0/116 | 0/14 |
| R8.independent | 0/7 | 0/6 | 2/29 | 12/12 | 14/14 | 1/1 | 0/27 | 3/6 |
| R8.shared | 0/7 | 0/6 | 2/29 | 24/24 | 14/14 | 1/1 | 0/27 | 3/6 |

Each entry is passed/bound cases. A case can serve several obligations and cells.

| Shared operations | passed/bound |
|---|---|
| storage | 0/12 |
| preparation (categorical rejection) | 0/3 |
| aggregation | 0/21 |
| validation | 0/39 |
| splits | 0/65 |
| scalar_tree | 0/42 |
| exact_policy | 70/191 |
| vector | 0/75 |

## What this shows

- The checkpoint validation selected recipe-level trajectories. Six of eight cells
have complete recipe evidence at the merged source; Normal lacks ten anchor cases.
- No cell has operation-level execution at this source: objective geometry, stored
domains and the residency checks that exclude a hidden host computation are all
unexecuted. The scalar squared cell has no bound execution at all.
- Every cell's CPU-host controls passed in the retained Python 3.12 CPU run except
the two categorical-rejection controls, which postdate that run.

Earlier public runs at `af026ef`, `dd84247` and `fe12beb` executed many of these
operation tests for squared, Normal, binary and Poisson. They describe those source
revisions. Production changed afterwards, so they are not counted here. Multiclass,
AFT and vector operation tests have no public execution record at any revision.

## Files

- `collection.json`: the collected GPU-marked population and the host-control
modules, with the pytest arguments. Collection is not execution.
- `outcomes-312d821.json`: case outcomes reduced from the original receipts under
`docs/v1/evidence/pr27-modal-validation/`, with the SHA-256 of each input file.
- `audit-312d821.json`: the judge's complete report and every reason.

`src/` and `tests/` at the audited commit are byte-identical to `312d821`; the
collection additionally contains the three new categorical-rejection GPU cases.

## Reproduce

```bash
uv run python -m benchmarks.v1.required_device_judge collect --matrix v1-sprints/080-required-device-matrix.json --output benchmarks/v1/evidence/required-device-080/collection.json
```

```bash
uv run python -m benchmarks.v1.required_device_judge extract --matrix v1-sprints/080-required-device-matrix.json --source-commit 312d8213900c0e4b35e7bdcac3d92d74ca5dbb81 --junit docs/v1/evidence/pr27-modal-validation/gpu/gpu.xml --hardware docs/v1/evidence/pr27-modal-validation/gpu/gpu-hardware.stdout.txt --host-junit docs/v1/evidence/pr27-modal-validation/cpu312/cpu.xml --output benchmarks/v1/evidence/required-device-080/outcomes-312d821.json
```

```bash
uv run python -m benchmarks.v1.required_device_judge judge --matrix v1-sprints/080-required-device-matrix.json --plan planning/agent-boosting-foundation-plan.md --collection benchmarks/v1/evidence/required-device-080/collection.json --outcomes benchmarks/v1/evidence/required-device-080/outcomes-312d821.json --output benchmarks/v1/evidence/required-device-080/audit-312d821.json
```

The last command exits 1 while the verdict is open.

## Limits

- Bindings are derived from test names, docstrings and assertions at function
granularity. A passing bound case shows that test passed, not that the obligation
is exhaustively verified.
- The judge trusts the supplied JUnit and hardware files. Their binding to a
source, an installed package and a real device comes from the original receipts.
- A hidden host computation is excluded only through transfer-accounting and
forbidden-callback tests, a retained device observation and the CPU-host refusal
control. The evidence has no independent per-case device activity record.
- Task-metric parity on real applications, cost and E4 are outside this matrix.
Loading
Loading