Skip to content

fix Harbor oracle reward=1 path (engines, Playwright, fonts, CI, prisma, rsbuild) - #17

Merged
jazelly merged 8 commits into
mainfrom
cursor/oracle-reward-diagnosis-a1ff
Nov 15, 2026
Merged

jazelly merged 8 commits into
mainfrom
cursor/oracle-reward-diagnosis-a1ff

Conversation

@jazelly

@jazelly jazelly commented Nov 15, 2026 •

Copy link
Copy Markdown
Owner

Summary

Fixes so Harbor oracle reaches reward 1 on all 32 control + 32 agentic tasks, plus CI runner pin and shared per-repo base images.

Harbor / grading

  • Per-repo harbor.nodeImage (Node 22 vs 24) from engines.node (jsoncrack / Next 16 need ≥24)
  • Playwright chromium in repo base images + local ensurePlaywrightBrowsers / assertHostNodeForRepo
  • Prefetch Google fonts / next typegen in repo bases
  • rr7-epic-stack: postInstall copies .env, prisma migrate deploy + generate --sql
  • rsbuild-rsdoctor: ignoreInstallScripts + client graph build; setup edits after base install
  • tanstack: preInstall pins unpublished @tanstack/react-start@^1.169.6 → npm 1.169.5
  • webpack-portainer: --no-frozen-lockfile (pnpmfileChecksum vs configDependencies)
  • Shared bases: agentic-react/base-<repo>:<cacheKey> (SHA + Dockerfile + lockfiles); thin task + agentic overlay layers
  • Local validate-tasks --oracle mirrors via ensureRepoReady

Repo CI

  • Pin .github/workflows runs-on to ubuntu-24.04
  • Biome ignore benchmark/**

Results

64/64 Harbor oracle reward=1 (control × 32 + agentic × 32). CSVs: experiments/oracle-rewards.csv, experiments/oracle-timings.csv.

Repo base cache (mean environment_setup)

Repo Mean env_setup before (s) Mean env_setup after (s)
astro-web-check 154 3
next-turbopack-boilerplate 48 2
next-webpack-jsoncrack 36 7
rr7-epic-stack 37 5
rsbuild-rsdoctor 76 2
tanstack-start-trellaux 19 5
vite-shadcn-admin 28 6
webpack-portainer 40 2

Control (32)

Task Reward env_setup before (s) env_setup after (s)
astro-web-check__island-results-not-hydrated 1 220 7
astro-web-check__runtime-server-status-timings 1 188 3
astro-web-check__ssr-home-background-window 1 181 3
astro-web-check__ui-server-status-labels 1 178 3
next-turbopack-boilerplate__hydration-locale-switcher 1 76 3
next-turbopack-boilerplate__ssr-current-count-crash 1 78 3
next-turbopack-boilerplate__ui-counter-button-color 1 17 3
next-turbopack-boilerplate__validation-counter-range 1 14 3
next-webpack-jsoncrack__build-webpack-wasm-config 1 13 2
next-webpack-jsoncrack__hydration-hero-cta 1 12 2
next-webpack-jsoncrack__runtime-editor-title-crash 1 13 2
next-webpack-jsoncrack__ui-bottom-bar-valid-label 1 12 2
rr7-epic-stack__hydration-heading-viewport-attr 1 16 2
rr7-epic-stack__ssr-loader-missing-user-404 1 16 2
rr7-epic-stack__state-double-check-button 1 13 2
rr7-epic-stack__ui-marketing-heading 1 13 2
rsbuild-rsdoctor__build-broken-import 1 80 2
rsbuild-rsdoctor__logic-windows-asset-paths 1 87 2
rsbuild-rsdoctor__state-bundle-size-sort-order 1 81 2
rsbuild-rsdoctor__ui-plugins-page-labels 1 81 2
tanstack-start-trellaux__hydration-home-updated-at 1 14 4
tanstack-start-trellaux__state-column-card-order 1 11 4
tanstack-start-trellaux__type-board-loader-param 1 9 4
tanstack-start-trellaux__ui-card-remove-order-badge 1 8 4
vite-shadcn-admin__runtime-tasks-status-crash 1 29 2
vite-shadcn-admin__state-signout-keeps-user 1 57 2
vite-shadcn-admin__type-refactor-column-header-label 1 26 2
vite-shadcn-admin__ui-dashboard-export-button 1 26 2
webpack-portainer__build-webpack-alias-resolution 1 18 2
webpack-portainer__state-expandable-message-toggle 1 — 2
webpack-portainer__type-loading-button-root-cause 1 12 2
webpack-portainer__ui-add-button-default-label 1 12 2

Agentic (32)

Task Reward env_setup before (s) env_setup after (s)
astro-web-check__island-results-not-hydrated 1 25 1
astro-web-check__runtime-server-status-timings 1 26 1
astro-web-check__ssr-home-background-window 1 26 1
astro-web-check__ui-server-status-labels 1 391 1
next-turbopack-boilerplate__hydration-locale-switcher 1 12 2
next-turbopack-boilerplate__ssr-current-count-crash 1 12 1
next-turbopack-boilerplate__ui-counter-button-color 1 91 1
next-turbopack-boilerplate__validation-counter-range 1 82 1
next-webpack-jsoncrack__build-webpack-wasm-config 1 60 12
next-webpack-jsoncrack__hydration-hero-cta 1 59 12
next-webpack-jsoncrack__runtime-editor-title-crash 1 61 11
next-webpack-jsoncrack__ui-bottom-bar-valid-label 1 59 11
rr7-epic-stack__hydration-heading-viewport-attr 1 55 8
rr7-epic-stack__ssr-loader-missing-user-404 1 54 6
rr7-epic-stack__state-double-check-button 1 56 6
rr7-epic-stack__ui-marketing-heading 1 76 6
rsbuild-rsdoctor__build-broken-import 1 28 1
rsbuild-rsdoctor__logic-windows-asset-paths 1 84 1
rsbuild-rsdoctor__state-bundle-size-sort-order 1 87 1
rsbuild-rsdoctor__ui-plugins-page-labels 1 81 1
tanstack-start-trellaux__hydration-home-updated-at 1 27 6
tanstack-start-trellaux__state-column-card-order 1 25 6
tanstack-start-trellaux__type-board-loader-param 1 27 6
tanstack-start-trellaux__ui-card-remove-order-badge 1 26 6
vite-shadcn-admin__runtime-tasks-status-crash 1 21 9
vite-shadcn-admin__state-signout-keeps-user 1 23 8
vite-shadcn-admin__type-refactor-column-header-label 1 18 8
vite-shadcn-admin__ui-dashboard-export-button 1 21 13
webpack-portainer__build-webpack-alias-resolution 1 56 1
webpack-portainer__state-expandable-message-toggle 1 72 2
webpack-portainer__type-loading-button-root-cause 1 56 2
webpack-portainer__ui-add-button-default-label 1 58 2

No model trials. No secrets. Nothing still failing.

Open in Web Open in Cursor 

cursoragent and others added 5 commits November 15, 2026 07:05
…s, CI)

- Pin CI/release runners to ubuntu-24.04 (Playwright unsupported on 26.04)
- Per-repo Harbor Node images from engines.node (22 for ^22 caret, 24 for >=24)
- Install Playwright Chromium in images for browser-mode hidden-tests
- Prefetch next/font Google Fonts into Next mock map (fixes Docker ETIMEDOUT)
- Ignore next-env.d.ts / tsbuildinfo in only-changed; local validate asserts Node
- Add oracle solution for rr7 hydration-heading-viewport-attr
- Batch runner for control×32 + agentic×32 Harbor oracle matrix

Assisted-by: Composer
Co-authored-by: Jason Zhang <xzha4350@gmail.com>
Root *.tsx includes were pulling in benchmark/xstack archive fixtures
and failing `pnpm exec biome lint .` on CI.

Assisted-by: Composer
Co-authored-by: Jason Zhang <xzha4350@gmail.com>
next-turbopack-boilerplate gitignores next-env.d.ts; without
`npx next typegen` after install, `@typecheck` fails on image imports.

Assisted-by: Composer
Co-authored-by: Jason Zhang <xzha4350@gmail.com>
Epic Stack images need .env + prisma migrate/generate --sql so typecheck
and vitest grade. Rsdoctor skips root prepare (full monorepo build) and
builds the client graph after install. Local validate-tasks --oracle
mirrors postInstall via ensureRepoReady.

Assisted-by: Cursor Agent
Co-authored-by: Jason Zhang <xzha4350@gmail.com>
Injected bugs (e.g. broken import) must not run during image build
steps; baseline still includes the post-setup tree for agents.

Assisted-by: Cursor Agent
Co-authored-by: Jason Zhang <xzha4350@gmail.com>
@cursor cursor Bot changed the title benchmark: fix Harbor oracle reward 1 + CI ubuntu-24.04 pin fix Harbor oracle reward=1 path (engines, Playwright, fonts, CI, prisma, rsbuild) Nov 15, 2026
cursoragent and others added 3 commits November 15, 2026 09:29
TanStack Start example depends on unpublished @tanstack/react-start@^1.169.6;
preInstall pins npm 1.169.5. Portainer frozen install hits pnpmfileChecksum
mismatch from configDependencies — use --no-frozen-lockfile. Batch runner
gains --skip-ok to reuse existing reward=1 jobs.

Assisted-by: Cursor Agent
Co-authored-by: Jason Zhang <xzha4350@gmail.com>
…yers

Clone/install/postInstall/fonts/typegen/Playwright live in
agentic-react/base-<repo>:<cacheKey> (keyed by Dockerfile + SHA + lockfiles).
Task Dockerfiles FROM that base and only apply setup edits + baseline;
agentic overlay stays a thin layer on the task image. prepare-variants and
run-oracle-batch build/cache bases before trials.

Assisted-by: Cursor Agent
Co-authored-by: Jason Zhang <xzha4350@gmail.com>
…he timings

All 64 Harbor oracle trials reward=1 with shared repo bases. environment_setup
drops from tens–hundreds of seconds to ~2–7s per task after the base is cached.

Assisted-by: Cursor Agent
Co-authored-by: Jason Zhang <xzha4350@gmail.com>
@jazelly
jazelly merged commit 1180478 into main Nov 15, 2026
2 checks passed
@jazelly
jazelly deleted the cursor/oracle-reward-diagnosis-a1ff branch November 15, 2026 12:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants