Skip to content

Fix CI by running benchmarks in Docker - #238

Merged
JuanVqz merged 3 commits into
mainfrom
feature/docker-rubies-ci
Sep 23, 2026
Merged

JuanVqz merged 3 commits into
mainfrom
feature/docker-rubies-ci

Conversation

@JuanVqz

@JuanVqz JuanVqz commented Sep 23, 2026

Copy link
Copy Markdown
Member

Summary

  • CI on main has been red because the rake_older jobs (Ruby 2.1, 2.2, 2.3) request ubuntu-20.04, which GitHub retired. They wait 24h for a runner and get cancelled ("The job has exceeded the maximum execution time while awaiting a runner for 24h0m0s").
  • Every Ruby in the matrix now runs as a Docker Compose service, in CI and locally, so the benchmarks no longer depend on which Rubies a given runner image supports, and anyone can run old Rubies, JRuby, and TruffleRuby without installing them.
  • Adds Ruby 3.3, 3.4 and 4.0 to the matrix. The README results are already on 4.0, but CI stopped at 3.2.

Changes

Step 1: Docker setup

  • compose.yaml: one service per Ruby (ruby_2.1 ... ruby_4.0, ruby_head, jruby_9.1, jruby_head, truffleruby_22, truffleruby_head).
  • Released MRI and JRuby 9.1 use the official images. The head builds and TruffleRuby 22 have no official image, so docker/Dockerfile installs the same prebuilt tarballs setup-ruby uses (and TruffleRuby's own 22.3.1 release), for amd64 and arm64.
  • docker/run-benchmarks.sh runs all benchmarks, or only the files passed as arguments. It copies the Gemfile out of the mounted repo so each Ruby resolves its own Gemfile.lock and the host's is left alone.
  • CONTRIBUTING.md documents usage and how to refresh the head images.

Step 2: CI

  • One job, matrix of compose services, docker compose run --rm -T <service> on ubuntu-latest. Replaces the three jobs (rake, rake_old, rake_older).
  • fail-fast: false, timeout-minutes: 60 (a stuck job fails in an hour instead of 24h), actions/checkout@v4.
  • SHARE=1 on main is still passed through to benchmark-ips.

Step 3: Ruby 3.3, 3.4 and 4.0 added to compose and the matrix.

Notes

  • Official Debian images were chosen over Alpine: on Alpine, 2.7 to 3.2 failed to install bigdecimal (no compiler), and musl's allocator could shift allocation-heavy results compared to the glibc builds CI used before.
  • The Rakefile ignores each benchmark's exit status, so a crashing benchmark does not fail the job. This predates this PR and is left for a follow-up.

Test plan

  • Locally (arm64 Mac): all 18 services ran bundle install and one benchmark successfully. ruby_2.1 and jruby_9.1 run under amd64 emulation.
  • Full run of all 68 benchmarks on ruby_3.2 through docker/run-benchmarks.sh exited 0, and no Gemfile.lock was written on the host.
  • CI on this PR runs the full matrix on GitHub's runners.
  • Try it: docker compose run --rm ruby_2.1 code/array/array-concat-vs-+.rb

Old Rubies (2.1 to 2.3, JRuby 9.1, TruffleRuby 22) and the head builds
are hard to install on a modern machine. compose.yaml has one service per
Ruby in the CI matrix, so anyone can run the benchmarks without depending
on their host:

    docker compose run --rm ruby_2.1
    docker compose run --rm ruby_2.1 code/general/some-benchmark.rb

Released MRI and JRuby 9.1 use the official images. The head builds and
TruffleRuby 22 have no official image, so docker/Dockerfile installs the
same prebuilt tarballs setup-ruby uses (and TruffleRuby's own release).

docker/run-benchmarks.sh copies the Gemfile out of the mounted repo so
each Ruby resolves its own Gemfile.lock and the host's is left alone.
The rake_older jobs asked for ubuntu-20.04, which GitHub retired, so they
waited 24h for a runner and were cancelled, leaving main red.

Every matrix entry is now a compose.yaml service run on ubuntu-latest,
so CI no longer depends on which Rubies setup-ruby can still build for a
given runner image, and it runs exactly what contributors run locally.
The three jobs become one.

Also:
- fail-fast: false, so one broken head build does not cancel the rest
- timeout-minutes: 60, so a stuck job fails in an hour instead of 24h
- actions/checkout v3 -> v4
The README results are already on Ruby 4.0, but CI stopped at 3.2.
@JuanVqz
JuanVqz marked this pull request as ready for review September 23, 2026 05:01
@JuanVqz
JuanVqz requested a review from etagwerker September 23, 2026 05:01
@JuanVqz JuanVqz self-assigned this Sep 23, 2026
@JuanVqz

JuanVqz commented Sep 23, 2026

Copy link
Copy Markdown
Member Author

Okay, I'm not finding a good reason to run all those when opening a PR, especially when the changes are at README.md or CHANGELOG.md, so, I will came with a solution some that I already have in mind:

  1. Skip docs-only PRs: add a paths filter so PR runs only happen when code/, Gemfile, compose.yaml, docker/ or the workflow change. main has no branch protection or required checks, so skipped runs can't block a merge.

  2. Run only what the PR changed: a PR that adds or edits one benchmark runs just that file on all 18 Rubies, about 1 to 2 minutes per job instead of 22. The run script already accepts file arguments. When Gemfile, compose.yaml, docker/ or the workflow change, it runs the full suite, because those can break anything.

But for now we are fine this way...

@JuanVqz
JuanVqz merged commit ff4b0ae into main Sep 23, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant