Skip to content

test(python): differential fuzz Monty against CPython - #2875

Merged
chaliy merged 3 commits into
mainfrom
claude/monty-diff-fuzz
Nov 7, 2026
Merged

chaliy merged 3 commits into
mainfrom
claude/monty-diff-fuzz

Conversation

@chaliy

@chaliy chaliy commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

What changed

  • New python_diff_fuzz target (features python,cpython): builds small programs from a grammar of the Monty-supported subset, runs each through the Monty fork and the CPython wasm runtime, and fails when stdout differs. Each statement is wrapped in try/except printing the exception type, so error types are compared too. Runs that hit a limit are skipped (the runtimes enforce limits differently). Nightly in fuzz.yml, build-checked in fuzz-check.
  • Fix: python_fuzz never actually ran Monty. It didn't set BASHKIT_ALLOW_INPROCESS_PYTHON, so every input hit the opt-in refusal message. Both targets now opt in and assert python3 really runs, so this can't silently regress.

Why

Crash-only fuzzing can't see wrong results. We now own the Monty fork and re-sync it with upstream; a differential oracle catches semantic regressions from local edits and re-syncs.

Before / After

python_fuzz, before (debug print of each run):

DBG exit=1 err="python3: in-process Python disabled by default for security; set BASHKIT_ALLOW_INPROCESS_PYTHON=1 to enable"

After: 309k execs in 326 s locally, coverage 26,976 edges (vs ~9.8k before), 0 crashes.

python_diff_fuzz, ~9.2k programs locally. One divergence class found, documented and excluded from the grammar:

c.append(len(0))   # c = 0
monty:   TypeError        (evaluates the argument first)
cpython: AttributeError   (looks up c.append first)

No other divergences.

Risk

  • Low: fuzz targets and CI wiring only. The diff target runs ~7 exec/s (CPython startup per program), fine for the 5-minute nightly slot.

Checklist

  • Tests added or updated
  • Backward compatibility considered (n/a)

python_diff_fuzz runs grammar-generated programs through the Monty fork
and the CPython wasm runtime and fails on any stdout difference. First
runs found Monty evaluates method-call arguments before the method
lookup (excluded from the grammar, documented).

python_fuzz never set BASHKIT_ALLOW_INPROCESS_PYTHON, so it only fuzzed
the opt-in refusal. Both targets now opt in and assert python3 runs.
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Nov 7, 2026 •

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
bashkit b97f346 Commit Preview URL

Branch Preview URL
Nov 07 2026, 04:31 AM

@bashkit-pr-reviewer bashkit-pr-reviewer Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Differential fuzz design is sound (grammar avoids SyntaxError-hash noise, both runtimes metered, refusal fails loudly). One oracle gap below; otherwise no blockers.

@bashkit-pr-reviewer

bashkit-pr-reviewer Bot commented Nov 7, 2026 •

Copy link
Copy Markdown

Review verdict: approve — rebase only, no new findings

Head b97f346 is a rebase onto new main (449075a); the PR contents are unchanged from the previously reviewed head: python_diff_fuzz.rs still 379 lines with the OverflowError-compared fix and KNOWN GAPS docs, python_fuzz.rs canary intact, same 8 files +423/−1.

Re-verified against the new base: all APIs the targets depend on are present with unchanged semantics — BashBuilder::python_with_limits / cpython_with_limits (with max_duration/max_memory/max_output), ExecResult::err refusal path with the "disabled by default" message on stderr, testing::fuzz_init/fuzz_exec. Prior findings all resolved, nothing outstanding.

Not built/run (no cargo in this shell); compile covered by the ci.yml build line.

@chaliy
chaliy merged commit e8dd56d into main Nov 7, 2026
38 checks passed
@chaliy
chaliy deleted the claude/monty-diff-fuzz branch November 7, 2026 04:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant