Fix crashing benchmarks and fail CI on crashes - #239
Merged
Merged
Conversation
On Ruby 4.0, ruby-head, jruby-head and truffleruby-head, both OpenStruct benchmarks crashed with "cannot load such file -- ostruct (LoadError)": ostruct is no longer a default gem, and bundle exec only loads gems that are in the Gemfile. The gem is conditional because an unconditional one breaks Ruby 2.1 (ostruct 0.1.0 uses &.) and would replace the ostruct older Rubies ship.
Crashes seen in CI, now fixed:
- dig-vs-[]-vs-fetch.rb on 2.1 and 2.2: skip only the Hash#dig report
(Ruby 2.3+), the other five still run.
- slice-native-vs-before-native.rb on 2.1 to 2.4 and JRuby 9.1: skip only
the native Hash#slice report (Ruby 2.5+), so the pre-native versions the
file is about still run there.
- remove-extra-spaces-or-other-chars.rb on 2.1 and 2.2: <<~ (Ruby 2.3+)
made the file fail to parse. A plain heredoc builds the same string
(checked byte for byte).
- bsearch-vs-find.rb on JRuby: the default Java heap cap (1/4 of memory)
is too small for its 100M element array. It failed on JRuby 9.1 in CI
(3554MB cap) and on JRuby 10 locally (2232MB). Both now get -J-Xmx6g.
- On main (SHARE=1), JRuby 9.1 crashed every benchmark when sharing:
its bundled jruby-openssl fails the TLS handshake with ips.fastruby.io
("Received fatal alert: handshake_failure"). The Gemfile gives JRuby
before 9.2 a newer jruby-openssl (0.14.6 resolves). JRuby 10 already
connects, so it keeps its own.
The Rakefile ignored each system(...) result, so CI stayed green with 7 benchmarks crashing across Rubies. It now runs every file, then exits non-zero listing the ones that failed, so one crash does not hide others. docker/run-benchmarks.sh with file arguments stopped at the first failing file (set -e). It now runs them all and reports the same way.
JuanVqz
force-pushed
the
feature/clean-benchmark-runs
branch
from
September 23, 2026 19:13
70a7f74 to
25bc9d3
Compare
Every PR ran all 68 benchmarks on 18 Rubies (about 22 min per job, 18 of the org's 20 concurrent jobs), even for README-only changes. - paths filter: only changes to benchmarks, the Gemfile, Rakefile, compose.yaml, docker/ or this workflow run it, on PRs and on main. - On a PR, only the changed benchmark files run, unless a shared file changed, then everything runs. A PR that only deletes benchmarks runs nothing. Pushes to main always run everything. - File names reach the container through an env var with globbing off, so names like dig-vs-[]-vs-fetch.rb pass through as is and a PR cannot inject shell through a file name.
JuanVqz
force-pushed
the
feature/clean-benchmark-runs
branch
from
September 23, 2026 19:17
25bc9d3 to
8468a2f
Compare
JuanVqz
marked this pull request as ready for review
September 23, 2026 19:47
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Changes
Step 1:
ostructon Ruby 4.0+cannot load such file -- ostruct (LoadError): it is no longer a default gem, andbundle execonly loads gems in the Gemfile.gem 'ostruct' if RUBY_VERSION >= '3.4'. Theifmatters: without it Ruby 2.1 installs ostruct 0.1.0, which does not parse there, and older Rubies would benchmark the gem instead of the ostruct they ship.Step 2: benchmarks that need a newer Ruby
dig-vs-[]-vs-fetch.rb(2.1, 2.2): only theHash#digreport is skipped, the other five still run.slice-native-vs-before-native.rb(2.1 to 2.4, JRuby 9.1): only the nativeHash#slicereport is skipped, so the pre-native versions the file is about still run there.remove-extra-spaces-or-other-chars.rb(2.1, 2.2):<<~made the file fail to parse. A plain heredoc builds the same string, checked byte for byte.bsearch-vs-find.rb(JRuby): the default Java heap cap is too small for its 100M element array (JRuby 9.1 failed in CI at 3554MB, JRuby 10 locally at 2232MB). Both JRubies getJRUBY_OPTS=-J-Xmx6g.main(SHARE=1), every JRuby 9.1 benchmark crashed after printing its results withOpenSSL::SSL::SSLError: Received fatal alert: handshake_failure: its bundled jruby-openssl cannot connect to ips.fastruby.io. With Step 3 that would turn everymainrun red, since PR runs never share. JRuby before 9.2 now gets a newer jruby-openssl (0.14.6 resolves); JRuby 10 already connects and keeps its own.Step 3: fail the run when a benchmark crashes
docker/run-benchmarks.shwith file arguments did stop at the first failing file; it now reports the same way.Step 4: PRs run only what they change
pathsfilter: only changes tocode/**/*.rb,Gemfile,Rakefile,compose.yaml,docker/**or the workflow run Benchmarks. Of the 15 merged PRs before Fix CI by running benchmarks in Docker #238, 6 changed only README/CONTRIBUTING.mainalways run everything.dig-vs-[]-vs-fetch.rbpass through as is and cannot inject shell.Test plan
bsearch-vs-find.rb(heap), which passes with the heap change.docker/,[]in names, README only. actionlint is clean.