Skip to content

Save results as JSON and run YJIT and ZJIT variants - #241

Merged
JuanVqz merged 2 commits into
mainfrom
feature/jit-variants-and-comparable-results
Sep 23, 2026
Merged

JuanVqz merged 2 commits into
mainfrom
feature/jit-variants-and-comparable-results

Conversation

@JuanVqz

@JuanVqz JuanVqz commented Sep 23, 2026

Copy link
Copy Markdown
Member

Summary

  • Benchmark results only existed as text in job logs and as ips.fastruby.io reports, which don't record the Ruby, OS or machine. Each result is now also saved as JSON, together with where it ran, so a future results site can show only comparable numbers side by side.
  • Every CRuby run was interpreter-only. YJIT is built into Ruby 3.2+ but off by default, while many production apps run it (Rails 7.2 turns it on for new apps on Ruby 3.3+). CI now also runs YJIT and ZJIT variants.

Changes

Step 1: results as JSON

  • docker/collect_results.rb, loaded through RUBYOPT when RESULTS_DIR is set, so benchmark files stay untouched. Each Benchmark.ips report goes to RESULTS_DIR/<label>/<benchmark>.json with the Ruby, the variant and flags, the PR's own commit and number, and the environment: arch, CPU, core count, OS, glibc, compiler and optflags. A field that can't be read is null, and the others are kept.
  • CI sets RESULTS_DIR on every benchmark job and uploads a results-<label> artifact.
  • Works on every Ruby in the matrix, back to 2.1.

Step 2: YJIT and ZJIT variants

  • 8 more CI jobs: +yjit on ruby_3.1 to ruby_head, and +zjit on ruby_4.0 and ruby_head. Each is its own job (variant: [''] plus include, so the entries don't merge into the plain jobs).
  • The flags go through RUBYOPT, JRUBY_OPTS or TRUFFLERUBYOPT, so every benchmark process the Rakefile starts inherits them. The run stops if the variant isn't on, instead of quietly benchmarking the interpreter.
  • Only plain Rubies share to ips.fastruby.io, because shared reports don't record the Ruby or its flags.
  • CONTRIBUTING shows how to run a variant and keep results locally.

Test plan

  • Every flag reaches child processes through its variable: YJIT and ZJIT (RUBY_DESCRIPTION), plus JRuby --dev and TruffleRuby with compilation off, which are supported but not in the matrix.
  • Results written on ruby_3.4+yjit, ruby_4.0+zjit, ruby_2.1, jruby_9.1, jruby_head, truffleruby_22 and truffleruby_head, through both file mode and the full Rakefile run. ruby_3.4+zjit stops with "invalid option --zjit" and writes nothing.
  • The amd64 ruby:3.1 image has YJIT (+YJIT [x86_64-linux]), so ruby_3.1+yjit passes its check on CI.
  • Review fixes: arm64 records CPU implementer and part codes instead of null; a missing /etc/os-release or ldd leaves only those fields null; an empty RESULTS_LABEL falls back to the engine and version, plus the variant (ruby-3.4.10+yjit), so local plain and YJIT runs don't overwrite each other; RUBY_VARIANT without flags stops with "Variant yjit is not on"; a PR run records the PR's head commit and number instead of GitHub's temporary merge commit.
  • Matrix expansion checked against GitHub's documented include rules: 26 jobs, the 18 plain ones unchanged. actionlint is clean.
  • This PR changes the workflow and docker/, so its CI runs the full suite on all 26 jobs. It is also the first real check of the artifacts and of how many CPU models one run gets.

Results only existed as text in job logs and as ips.fastruby.io reports,
which do not record the Ruby, OS or machine. When RESULTS_DIR is set,
docker/collect_results.rb (loaded through RUBYOPT, so benchmark files
stay untouched) writes each Benchmark.ips report to
RESULTS_DIR/<label>/<benchmark>.json, with the Ruby, its flags, the run
and the environment: arch, CPU, core count, OS, glibc, compiler and its
flags. It records the PR's own commit and number (on a pull request,
github.sha is a temporary merge commit). A field that cannot be read is null, the others are kept.

CI sets RESULTS_DIR on every benchmark job and uploads the results as
a results-<label> artifact. The results site will read these.

Brought over from the parked prototype (988a455), without the site.
Every CRuby run was interpreter-only (YJIT is built into 3.2+ but off
by default), while many production apps run YJIT. CI now also runs
ruby_3.1 to ruby_head with --yjit, and ruby_4.0 and ruby_head with
--zjit: 8 more jobs, labeled like ruby_3.4+yjit.

RUBY_VARIANT_FLAGS goes through the variable each engine reads (RUBYOPT,
JRUBY_OPTS, TRUFFLERUBYOPT), since the Rakefile starts one process per
benchmark. The run then stops if the variant is not on, instead of
quietly benchmarking the interpreter.

Only plain Rubies share to ips.fastruby.io: shared reports do not record
the Ruby or its flags, so variant reports would be indistinguishable.
@JuanVqz
JuanVqz force-pushed the feature/jit-variants-and-comparable-results branch from c80788f to f4b5a55 Compare September 23, 2026 22:12
@JuanVqz
JuanVqz marked this pull request as ready for review September 23, 2026 22:20
@JuanVqz
JuanVqz merged commit d68fb31 into main Sep 23, 2026
28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant