Fix CI by running benchmarks in Docker - #238
Conversation
Old Rubies (2.1 to 2.3, JRuby 9.1, TruffleRuby 22) and the head builds
are hard to install on a modern machine. compose.yaml has one service per
Ruby in the CI matrix, so anyone can run the benchmarks without depending
on their host:
docker compose run --rm ruby_2.1
docker compose run --rm ruby_2.1 code/general/some-benchmark.rb
Released MRI and JRuby 9.1 use the official images. The head builds and
TruffleRuby 22 have no official image, so docker/Dockerfile installs the
same prebuilt tarballs setup-ruby uses (and TruffleRuby's own release).
docker/run-benchmarks.sh copies the Gemfile out of the mounted repo so
each Ruby resolves its own Gemfile.lock and the host's is left alone.
The rake_older jobs asked for ubuntu-20.04, which GitHub retired, so they waited 24h for a runner and were cancelled, leaving main red. Every matrix entry is now a compose.yaml service run on ubuntu-latest, so CI no longer depends on which Rubies setup-ruby can still build for a given runner image, and it runs exactly what contributors run locally. The three jobs become one. Also: - fail-fast: false, so one broken head build does not cancel the rest - timeout-minutes: 60, so a stuck job fails in an hour instead of 24h - actions/checkout v3 -> v4
The README results are already on Ruby 4.0, but CI stopped at 3.2.
|
Okay, I'm not finding a good reason to run all those when opening a PR, especially when the changes are at README.md or CHANGELOG.md, so, I will came with a solution some that I already have in mind:
But for now we are fine this way... |
Summary
mainhas been red because therake_olderjobs (Ruby 2.1, 2.2, 2.3) requestubuntu-20.04, which GitHub retired. They wait 24h for a runner and get cancelled ("The job has exceeded the maximum execution time while awaiting a runner for 24h0m0s").Changes
Step 1: Docker setup
compose.yaml: one service per Ruby (ruby_2.1...ruby_4.0,ruby_head,jruby_9.1,jruby_head,truffleruby_22,truffleruby_head).docker/Dockerfileinstalls the same prebuilt tarballssetup-rubyuses (and TruffleRuby's own 22.3.1 release), for amd64 and arm64.docker/run-benchmarks.shruns all benchmarks, or only the files passed as arguments. It copies the Gemfile out of the mounted repo so each Ruby resolves its ownGemfile.lockand the host's is left alone.CONTRIBUTING.mddocuments usage and how to refresh the head images.Step 2: CI
docker compose run --rm -T <service>onubuntu-latest. Replaces the three jobs (rake,rake_old,rake_older).fail-fast: false,timeout-minutes: 60(a stuck job fails in an hour instead of 24h),actions/checkout@v4.SHARE=1onmainis still passed through to benchmark-ips.Step 3: Ruby 3.3, 3.4 and 4.0 added to compose and the matrix.
Notes
bigdecimal(no compiler), and musl's allocator could shift allocation-heavy results compared to the glibc builds CI used before.Test plan
bundle installand one benchmark successfully.ruby_2.1andjruby_9.1run under amd64 emulation.ruby_3.2throughdocker/run-benchmarks.shexited 0, and noGemfile.lockwas written on the host.docker compose run --rm ruby_2.1 code/array/array-concat-vs-+.rb