Make benchmark failures block merges and explain the fix - #240
Merged
Merged
Conversation
A PR with a failing benchmark could still be merged: main has no required checks. Requiring the benchmark jobs themselves would block docs-only PRs forever, because the paths filter skipped the whole workflow and a required check that never reports stays "Expected". The workflow now always runs. A changes job decides once what to benchmark (the same rules as before, moved out of each matrix job), the benchmark jobs run only when needed, and benchmarks-ok always reports: it passes when they passed or were skipped. That is the check to require on main.
A failed run listed the files but not what to do. Both the Rakefile and the run script now point to a new CONTRIBUTING section: skip one report or the whole file with `if RUBY_VERSION >= ...`, and write new syntax (like <<~) in a form older Rubies can parse, since an `if` cannot help when the file does not parse.
A failure was only visible by opening each of the 18 job logs. When a benchmark job fails, it now adds an annotation per failed file, shown on the PR's changed files, and a list on the run's summary page, both pointing to the CONTRIBUTING section on newer-Ruby benchmarks. The run step uses `shell: bash` so pipefail is on and tee cannot turn a failing run into a passing step.
The workflow had grown long inline scripts. Each one is now a file the workflow calls, so the YAML shows what runs and when, and the scripts can be shellchecked on their own: - pick-benchmarks.sh: what a run benchmarks (was the changes job step) - run-benchmarks.sh: runs one Ruby's job and keeps its log - report-failures.sh: annotations and summary for failed benchmarks - check-benchmarks.sh: the benchmarks-ok check A change under .github/scripts/ now runs every benchmark, like a change to the workflow. run-benchmarks.sh sets pipefail itself, so the run step no longer needs `shell: bash`.
The file grew one addition at a time. The opening stays as it was; in between there is now a contents list and the sections in the order a contributor needs them: writing an entry, running it on other Rubies, and entries that need a newer Ruby. The license closes the file under its own heading.
JuanVqz
marked this pull request as ready for review
September 23, 2026 21:00
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
mainhas no required checks, and the failure doesn't say how to fix it.benchmarks-okcheck that always reports, so it can be required, explains the fix in the failure and in CONTRIBUTING, and shows failures on the PR itself.Changes
Step 1: a check that can be required
pathsfilter skipped the whole workflow, and a required check that never reports stays "Expected".changesjob decides once what to benchmark (same rules as before, moved out of each matrix job), the benchmark jobs run only when needed, andbenchmarks-okpasses when they passed or were skipped.Step 2: say how to fix it
docker/run-benchmarks.shfailure messages point to a new CONTRIBUTING section, "Benchmarks that need a newer Ruby": skip one report or the whole file withif RUBY_VERSION >= ..., and write new syntax (like<<~) in a form older Rubies can parse.Step 3: show failures on the PR
Step 4: shell code out of the workflow
.github/scripts/(pick-benchmarks.sh,run-benchmarks.sh,report-failures.sh,check-benchmarks.sh); the workflow went from about 110 to 75 lines. A change under.github/scripts/runs every benchmark.run-benchmarks.shsetspipefailitself. GitHub's default shell doesn't, so piping the run throughteewould otherwise have turned a failing run into a passing step.Step 5: CONTRIBUTING layout
After merge
benchmarks-okonmain(repo settings). Until then this PR changes what shows on a PR, but nothing blocks a red merge.Test plan
pick-benchmarks.shon real commits. PR (merge commit): README only skips; one benchmark runs just that file;Gemfileor a.github/scripts/change runs everything; delete-only skips; delete + edit runs the edited file. Push: README skips; a benchmark change runs everything; an unknown base runs everything.check-benchmarks.shpasses only for changes=success and rake=success or skipped.report-failures.shon a real failure log: one annotation per file (commas and colons escaped), the summary list, and the hint line isn't mistaken for a file.run-benchmarks.shexits 1 throughteewhen a benchmark fails.Rakefile, so its own CI runs the full suite through the new jobs.