Skip to content

fix(ci): migrate nightly jobs to new dtolnay/rust-toolchain schema - #2850

Merged
chaliy merged 6 commits into
mainfrom
fix/ci-nightly-toolchain-action
Nov 7, 2026
Merged

chaliy merged 6 commits into
mainfrom
fix/ci-nightly-toolchain-action

Conversation

@chaliy

@chaliy chaliy commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

What changed

Nightly CI jobs (Pyodide wheel, Miri, ASAN, fuzz) install their pinned nightly explicitly via rustup toolchain install instead of the removed toolchain: action input, which unblocks the dtolnay/rust-toolchain bump. Includes the other two safe bumps from #2847 (pnpm/action-setup, taiki-e/install-action). Supersedes #2847.

Then one consequence of that migration, found while reviewing it: cargo install does not follow a per-command toolchain selection, so both cargo-fuzz installs silently moved onto the pinned stable.

The migrated jobs install nightly alongside a stable default and select it per command (cargo +nightly-2026-10-30 fuzz build). cargo install in those jobs gets stable 1.96.0, where an unpinned source build resolves to the newest release — so an upstream MSRV bump breaks a tree that has not changed. That is exactly the cargo-geiger 0.14.0 failure test_pinned_ci_tools.py was written to prevent (nightly run 259 went red the next morning with the unsafe-code audit skipped).

The test did not catch it because it read the nightly exemption per job: a job that installed a nightly anywhere exempted every source build in it. It now reads per step — a step is exempt only when its own cargo resolves to a nightly (RUSTUP_TOOLCHAIN, or cargo +nightly-…) — and additionally covers bare cargo install run steps, which carry the same hazard in a spelling the test never looked at.

Also re-indents the nightly pin comment the migration orphaned at with: depth in ci.yml, fuzz.yml and nightly.yml.

Why

The dtolnay/rust-toolchain bump in #2847 (5b84223 → d86e0fa) removes the toolchain: input — the action now always installs the rust-toolchain.toml toolchain (stable 1.96.0). Nightly jobs silently got stable: Pyodide wheel failed (can't find crate for core, target never installed), and Miri/ASAN/fuzz would follow. Verified against upstream action.yml: valid inputs are only targets/target/components.

Before / After

The pin test, before the fix was applied (the regression it now catches):

FAIL: test_tools_on_the_pinned_toolchain_pin_a_version (workflow='ci.yml', job='fuzz-check', tool='cargo-fuzz')
  ci.yml:fuzz-check builds 'cargo-fuzz' from source on the pinned toolchain with no
  version; pin it as `tool: name@version` ...

FAIL: test_cargo_install_on_the_pinned_toolchain_pins_a_version (workflow='fuzz.yml', job='fuzz', tool='cargo-fuzz')
  fuzz.yml:fuzz runs `cargo install cargo-fuzz` on the pinned toolchain with no version;
  pass `--version <v>` or run it on the nightly (`cargo +nightly-... install`) ...

After: 4 passed, and 247 repo-script tests green.

Because nightly.yml and fuzz.yml only run on schedule, PR checks cannot exercise them — so both were dispatched on this branch and report here:

  • Build wheel (Pyodide/Emscripten) — success (the original failure mode)
  • Fuzz Compile Check — success (validates the RUSTUP_TOOLCHAIN route in ci.yml)
  • Fuzz Test (×9), Miri, AddressSanitizer, Security Analysis, Python multi-tenant stress soak — dispatched from this branch

publish-python.yml and python.yml both set RUSTUP_TOOLCHAIN on the pyodide build step itself, so the stable default never reaches it.

Risk

  • Medium: touches 6 workflow files, but each change is a mechanical install-step swap preserving pins/targets/components.
  • What can break: a nightly job if a rustup component name is wrong (Miri adds miri; ASAN adds rust-src) — covered by the dispatched runs above rather than assumed.
  • cargo-fuzz declares no rust_version, so the stable install happened to work; the fix is about the next upstream release, not a current break.

Checklist

  • Supersedes chore(ci): bump the github-actions group with 3 updates #2847 (close it on merge)
  • Per-step nightly exemption covered by tests, including bare cargo install
  • Knowledge updated (knowledge/operations/dependencies.md)
  • Nightly + fuzz dispatched on this branch rather than assumed
  • CI green — required before merge

Generated by Claude Code

The dtolnay/rust-toolchain action removed the `toolchain:` input (it now installs the rust-toolchain.toml toolchain). Bumping the pin (as #2847 does) silently installs stable 1.96.0 in nightly jobs, breaking the Pyodide wheel (-Z flags), Miri, ASAN (-Z build-std) and fuzz jobs.

Install the pinned nightlies explicitly via `rustup toolchain install` (targets/components preserved) and drop the now-dead matrix toolchain input.

Supersedes #2847.
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Nov 6, 2026 •

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
bashkit 671d2d0 Commit Preview URL

Branch Preview URL
Nov 07 2026, 01:21 PM

@bashkit-pr-reviewer

bashkit-pr-reviewer Bot commented Nov 6, 2026 •

Copy link
Copy Markdown

Review summary (head 671d2d04)

Verdict: Good. The only delta since 8292ef96 is ~68 lines in scripts/tests/test_pinned_ci_tools.py, and it fixes the open P2 outright. No findings.

What changed (verified in checkout): env-aware nightly detection — new ENV_EXPR/env_scopes/resolve_env/_is_nightly helpers resolve ${{ env.X }} through step→job→workflow scopes before matching, so the Pyodide jobs' rustup toolchain install ${{ env.RUST_NIGHTLY }} and their RUSTUP_TOOLCHAIN: ${{ env.RUST_NIGHTLY }} opt-in are now recognized (python.yml/publish-python.yml set RUST_NIGHTLY at job level, which is in the resolution scopes). A new meta-test, test_the_detector_follows_an_env_indirection, pins this down: every job with a rustup toolchain install step must be discovered as nightly (with a checked >= 4 non-vacuity guard; 6 such jobs exist), and any install step carrying RUSTUP_TOOLCHAIN must count as on-nightly. The unresolved-reference fallback is safe: matching is case-sensitive, so a dangling ${{ env.RUST_NIGHTLY }} (uppercase) does not match lowercase nightly. Workflow YAML, the Rust stress-test fix, and the knowledge entry are untouched since the last review.

Findings: P0/P1/P2: none. Prior P2 (pinned-ci-tools-nightly-detector-env-spelling): resolved by this push — closing it.

Context read beyond the diff: full test_pinned_ci_tools.py (332 lines), job-level RUST_NIGHTLY definitions in python.yml/publish-python.yml, repo-wide cargo install / rustup-install-step enumeration.

Not reviewed: no production code changed; CI behavior reasoned from YAML only (no builds run per reviewer workflow).

The toolchain action dropped its toolchain: input, so nightly jobs now install via run: rustup toolchain install nightly-.... The exempt-nightly detector only knew the legacy # nightly action-ref spelling and failed the migrated jobs (plus misclassified cargo-fuzz). Detect both; the discovery guard accepts either.

@bashkit-pr-reviewer bashkit-pr-reviewer Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

New push re-reviewed (test-only addition). One latent detector gap, non-blocking; the workflow migration itself still looks correct.

Comment thread scripts/tests/test_pinned_ci_tools.py Resolved
…ain migration

The toolchain action no longer accepts `toolchain:`, so the nightly jobs
install the pinned nightly alongside a stable default and select it per
command. `cargo install` does not follow that selection: both cargo-fuzz
installs (the cache-cargo-install-action step in ci.yml's fuzz-check and the
bare `cargo install` in fuzz.yml) silently moved onto the pinned stable, where
an unpinned source build resolves to the newest release and an upstream MSRV
bump breaks an unchanged tree. That is the cargo-geiger 0.14.0 failure the pin
test exists to prevent.

test_pinned_ci_tools read the nightly exemption per job, so a job that merely
installed a nightly somewhere exempted every source build in it. It now reads
per step: a step is exempt only when its own cargo resolves to a nightly, via
`RUSTUP_TOOLCHAIN` or `cargo +nightly-...`. The check also covers bare
`cargo install` run steps, which carry the same hazard in a spelling the test
did not look at.

Also re-indents the nightly pin comment the migration orphaned at `with:`
depth in ci.yml, fuzz.yml and nightly.yml.

Claude-Session: https://claude.ai/code/session_01Wy7MByvAMwapenBYpmq9HN
The nightly soak job added in #2874 has never passed. It runs
python_tenant_stress_tests at BASHKIT_STRESS_ROUNDS=300, and
one_session_reuses_its_budget_across_many_calls makes 10 * rounds() calls in a
single session: 3,000 against a default max_exec_calls of 1,000. It died at
call 1,000 on "session exec() call limit exceeded", never reaching what it
asserts, which is that memory charged in one call is released at its end.

The session cap is not what these tests measure, so tenant() now sets its own
with room for the largest soak the file can ask for. MAX_ROUNDS is one constant
behind both the BASHKIT_STRESS_ROUNDS bound and that headroom, so the two
cannot drift apart again the way they just did.

Reproduced at BASHKIT_STRESS_ROUNDS=101 (1,010 calls), which fails identically
on the cap and passes with this change.

Claude-Session: https://claude.ai/code/session_01Wy7MByvAMwapenBYpmq9HN

chaliy commented Nov 7, 2026

Copy link
Copy Markdown
Contributor Author

Python multi-tenant stress soak went red here, and it is not this PR's failure — but it is a real one, so I fixed it in 8292ef9 rather than leave it.

It only shows up here because I dispatched nightly.yml on this branch to validate the migration; nightly.yml never runs on PRs. This PR changes no Rust at all, and the job it tripped was added by #2874 earlier today — after nightly run 298 — so it has never run on main, and tonight's nightly would have gone red regardless of this PR.

Not a flake. The cause is exact:

thread '…one_session_reuses_its_budget_across_many_calls' panicked at
  crates/bashkit/tests/integration/python_tenant_stress_tests.rs:180:
  called `Result::unwrap()` on an `Err` value:
  Execution("session exec() call limit exceeded (1000 calls)")

The test loops 10 * rounds() calls in a single session. At BASHKIT_STRESS_ROUNDS=300 that is 3,000 calls, and tenant() takes the default max_exec_calls of 1,000 — so it died on an unrelated session cap at call 1,000 and never reached what it asserts, which is that memory charged in one call is released at its end. PR CI runs 3 rounds (30 calls), which is why nothing caught it.

Reproduced locally at BASHKIT_STRESS_ROUNDS=101 (1,010 calls): fails identically on the cap, and passes with the fix — all 5 tests green.

The fix gives tenant() its own session limits, sized off a single MAX_ROUNDS constant that also bounds BASHKIT_STRESS_ROUNDS, so the soak multiplier and the cap cannot drift apart again the way they just did. nightly.yml is re-dispatched on this branch to show the job green.

Migration validation (the reason for dispatching)

job result
Fuzz Testing (all 9 targets) ✅ — confirms cargo +nightly-2026-10-30 install cargo-fuzz --locked
Miri ✅ — rustup toolchain install … --component miri
AddressSanitizer ✅ — --component rust-src + -Z build-std
Security Analysis, Advisory Audit ✅
Build wheel (Pyodide/Emscripten) ✅ — the original failure mode
Fuzz Compile Check ✅ — the RUSTUP_TOOLCHAIN route in ci.yml

Generated by Claude Code

The nightly detector required a literal `nightly` token, so a job naming its
nightly through a variable was read as not having one. Both Pyodide jobs do
exactly that — `rustup toolchain install ${{ env.RUST_NIGHTLY }}` with
`RUSTUP_TOOLCHAIN` set to the same expression — which is the opt-in route this
same change introduced, so the "either spelling" claim was not true of the
tree it runs against. Nothing is mis-exempted today (neither job builds a tool
from source), but a source build added there would be told to pin a tool that
is already on a nightly.

`${{ env.NAME }}` is now resolved against step, job and workflow `env` before
the nightly test. An unresolved reference is left as written so the variable's
own name cannot stand in for its value: `RUST_NIGHTLY` is not evidence that
what it holds is a nightly.

The new test asserts it over the real workflows rather than a fixture: every
job installing a toolchain through rustup must be discovered, and a step
pinning `RUSTUP_TOOLCHAIN` to its job's nightly must count as running on it.
Six jobs covered, including the two that were missed.

Claude-Session: https://claude.ai/code/session_01Wy7MByvAMwapenBYpmq9HN
@chaliy
chaliy merged commit c1a6205 into main Nov 7, 2026
59 checks passed
@chaliy
chaliy deleted the fix/ci-nightly-toolchain-action branch November 7, 2026 14:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant