Repository navigation
test(python): differential fuzz Monty against CPython - #2875
Conversation
python_diff_fuzz runs grammar-generated programs through the Monty fork and the CPython wasm runtime and fails on any stdout difference. First runs found Monty evaluates method-call arguments before the method lookup (excluded from the grammar, documented). python_fuzz never set BASHKIT_ALLOW_INPROCESS_PYTHON, so it only fuzzed the opt-in refusal. Both targets now opt in and assert python3 runs.
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
bashkit | b97f346 | Commit Preview URL Branch Preview URL |
Nov 07 2026, 04:31 AM |
Review verdict: approve — rebase only, no new findingsHead Re-verified against the new base: all APIs the targets depend on are present with unchanged semantics — Not built/run (no cargo in this shell); compile covered by the |
What changed
python_diff_fuzztarget (featurespython,cpython): builds small programs from a grammar of the Monty-supported subset, runs each through the Monty fork and the CPython wasm runtime, and fails when stdout differs. Each statement is wrapped intry/exceptprinting the exception type, so error types are compared too. Runs that hit a limit are skipped (the runtimes enforce limits differently). Nightly infuzz.yml, build-checked infuzz-check.python_fuzznever actually ran Monty. It didn't setBASHKIT_ALLOW_INPROCESS_PYTHON, so every input hit the opt-in refusal message. Both targets now opt in and assertpython3really runs, so this can't silently regress.Why
Crash-only fuzzing can't see wrong results. We now own the Monty fork and re-sync it with upstream; a differential oracle catches semantic regressions from local edits and re-syncs.
Before / After
python_fuzz, before (debug print of each run):After: 309k execs in 326 s locally, coverage 26,976 edges (vs ~9.8k before), 0 crashes.
python_diff_fuzz, ~9.2k programs locally. One divergence class found, documented and excluded from the grammar:No other divergences.
Risk
Checklist