Skip to content

fix(test): make the python regex DoS wall-clock bounds hang detectors, not budgets - #2883

Merged
chaliy merged 1 commit into
mainfrom
eve/clever-mccarthy-gwhehq
Nov 7, 2026
Merged

chaliy merged 1 commit into
mainfrom
eve/clever-mccarthy-gwhehq

Conversation

@chaliy

@chaliy chaliy commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

What changed

Three wall-clock assertions in blackbox_dos now derive their bound from the VM time limit the test configures, via hang_bound(limit) = limit * 3, instead of hard-coding a number below it.

Coverage on main is currently red and this fixes it.

Why

Generate Rust Coverage failed at 38d2c4c with:

lookbehind: took 4.021935831s
  at crates/bashkit/tests/integration/python_security_tests.rs:1039

The test configured max_duration(5s) and then asserted elapsed < 4s. Those contradict: a run the VM's own limit permits fails the wall-clock check anyway, while exit_code correctly reports no timeout. Anything landing between 4s and 5s was always going to fail, with nothing wrong.

regex_huge_pattern_is_rejected_or_bounded had the identical inversion (4s against a 5s limit) and would have failed the same way; it's fixed here too rather than left to surface later.

regex_loop_is_stopped_by_time_limit already had it right — 6s over a 2s limit — so this makes that existing shape the rule rather than inventing one.

Before / After

# Before, under tarpaulin
test python_security_tests::blackbox_dos::regex_redos_patterns_are_bounded ... FAILED
  lookbehind: took 4.021935831s

# After, locally
test python_security_tests::blackbox_dos::regex_huge_pattern_is_rejected_or_bounded ... ok
test python_security_tests::blackbox_dos::regex_redos_patterns_are_bounded ... ok
test python_security_tests::blackbox_dos::regex_loop_is_stopped_by_time_limit ... ok
test result: ok. 6 passed; 0 failed

This is not a performance regression. All seven ReDoS cases together run in ~0.8s in a normal build (~0.12s each). Tarpaulin instrumentation is the multiplier — the whole integration suite takes ~240s there against ~36s uninstrumented — and fancy-regex 0.20's reworked lookbehind path (from #2876) was enough to cross a bound that had no margin to begin with. I measured this before changing the assertion, precisely to avoid filing a timing fix over a real slowdown.

Risk

  • Low. Test-only, and the security assertions are untouched: every case must still not hit the VM time limit (exit_code != 124), and the loop case must still be stopped by it (== 124).
  • Worth being explicit, since loosening a number in a security test deserves scrutiny: the bound being relaxed never tested the security property. The VM limit does that, and this assertion exists to catch a run that escaped the limit entirely — which limit * 3 still catches. It is not a weaker version of the old check; the old check was measuring the wrong thing against an impossible target.

Checklist

  • Tests added or updated — three assertions corrected, bound derived so they cannot contradict again
  • Measured the uninstrumented cost before treating it as a test bug
  • Backward compatibility considered — test-only
  • CI green — required before merge


<a href="https://backend.blacksmith.sh/track/enable-autofix?expires=1796652952&installation_model_id=447267&pr_number=2883&ref=codesmith_pr_footer&repository=everruns%2Fbashkit&return_to=https%3A%2F%2Fgithub.com%2Feverruns%2Fbashkit%2Fpull%2F2883&signature=1a8538efb78589da2fe1621f7df440ce1e31445ca28c1c9b70de9ff897f7dea3"&gt;&lt;picture&gt;&lt;source media="(prefers-color-scheme: dark)" srcset="https://pr-comments-assets.blacksmith.sh/codesmith/autofix-with-codesmith-dark.svg"&gt;&lt;source media="(prefers-color-scheme: light)" srcset="https://pr-comments-assets.blacksmith.sh/codesmith/autofix-with-codesmith-light.svg"&gt;&lt;img alt="Autofix with [code]smith" src="https://pr-comments-assets.blacksmith.sh/codesmith/autofix-with-codesmith-dark-3x.png" width="190" height="28"&gt;
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.


Generated by Claude Code

…, not budgets

Coverage on main went red at 38d2c4c: regex_redos_patterns_are_bounded failed
with "lookbehind: took 4.021935831s" against an `elapsed < 4s` assertion, while
the same test configured `max_duration(5s)` and `exit_code` correctly showed no
timeout. The two bounds contradicted each other — any run finishing between 4s
and 5s failed a test whose own limit permitted it.

The VM's `max_duration` is what these tests exercise; the wall clock is only
there to catch a run that escaped it entirely, so it has to sit above that
limit. `hang_bound(limit) = limit * 3` now derives it, which is the shape
regex_loop_is_stopped_by_time_limit already used (6s over a 2s limit).
regex_huge_pattern_is_rejected_or_bounded had the same inversion (4s against a
5s limit) and is fixed with it, before it could fail the same way.

Not a performance regression: all seven ReDoS cases together take ~0.8s in a
normal build. Tarpaulin instrumentation is what multiplies it — the whole
integration suite takes ~240s there — and fancy-regex 0.20's reworked
lookbehind path was enough to cross a bound that already had no margin.

The security assertions are untouched: each case must still not hit the VM time
limit (124), and the loop case must still be stopped by it.

Claude-Session: https://claude.ai/code/session_01Wy7MByvAMwapenBYpmq9HN
@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
bashkit 6837de8 Commit Preview URL

Branch Preview URL
Nov 07 2026, 02:16 PM

@bashkit-pr-reviewer

Copy link
Copy Markdown

Review summary — #2883

Verdict: Sound test-only fix, safe to merge once CI is green. No blocking findings; no review submitted (nothing to inline).

What the change does: blackbox_dos had elapsed < 4s wall-clock assertions guarding runs under a max_duration(5s) VM limit — any permitted 4–5s run failed while exit_code correctly showed no timeout. Tarpaulin's slowdown pushed the lookbehind case to 4.02s and turned coverage red. The PR derives the bound as hang_bound(limit) = limit * 3 (15s over 5s; 6s over 2s, unchanged in effect for the loop test) and documents it as a hang detector, not a budget. Security assertions are untouched: bounded cases still require exit_code != 124, the loop case still requires == 124.

Findings: None by severity (no P0/P1/P2 on the diff). Verified the three hunks in crates/bashkit/tests/integration/python_security_tests.rs (lines ~1005–1094): bounds now sit strictly above their limits, the shared helper prevents future inversions, and the explanatory comment records the decision per AGENTS.md.

Context read: full diff (not truncated); checkout succeeded; test module blackbox_dos (~lines 996–1125); AGENTS.md project rules; grep over knowledge docs for wall-clock/tarpaulin guidance.

Could not review: nothing — single-file diff fully visible; no build/test run per reviewer sandbox (reasoned from code).

Non-blocking note: the PR body contains a claude.ai/code/session_… link, which AGENTS.md forbids in PR bodies ("NEVER add links to Claude sessions"). Consider removing it; not a merge-blocker from this review.

@chaliy
chaliy merged commit 3dd87ea into main Nov 7, 2026
38 checks passed
@chaliy
chaliy deleted the eve/clever-mccarthy-gwhehq branch November 7, 2026 14:47
chaliy pushed a commit that referenced this pull request Nov 7, 2026
site-npm group. The Generate Rust Coverage failure on this PR was not its own:
it was red on main too (regex_redos_patterns_are_bounded hitting a wall-clock
bound tighter than the limit it tested), fixed in #2883, and a site-only npm
bump cannot reach Rust coverage. Green after one re-run.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant