feat: add paired Ruby SDK performance benchmarks by dustinbyrne · Pull Request #323 · PostHog/posthog-ruby

dustinbyrne · 2026-10-30T15:05:52Z

💡 Motivation and Context

Add every-PR performance reporting for the Ruby SDK using the same sdk-benchmarks/v1 workloads as Node, Python and .NET. Compares separately built and installed merge-base/head gems using the same head-side benchmark code.

All 20 flag/capture workloads, six alternating paired repetitions, four independent workload shards, and fresh worker processes for each measurement pass.
Native Ruby elapsed/CPU time, cumulative allocated-object counts, RSS and retained ObjectSpace bytes. Unsupported allocation-byte and enqueue-only allocation measurements remain explicit.
Controlled loopback delivery through public SDK APIs, independent correctness checks, exact sequence validation, and bounded worker execution.
Read-only PR execution plus the separately trusted shared publisher from ci: add trusted SDK benchmark report workflow .github#84. Human-readable comments flag ≥5% changes only when all six pairs agree; percentages do not gate CI.

No SDK runtime, public API or default behavior changes. No release changeset is needed. Automatic comments activate after the trusted caller lands on the default branch. Shared pins match the existing integrations; mainline repinning follows the shared PR's squash merge.

Related integrations: PostHog/posthog-js#5454, PostHog/posthog-python#1135, PostHog/posthog-dotnet#358.

💚 How did you test it?

12 coordinator controls and native controls covering fixtures, counters, generation changes, incomplete delivery and actual worker timeouts.
Complete paired smoke: 80 successful workers. Full six-pair execution: 480 successful workers; exact workload/pass/count/checkpoint conformance checked independently.
Actual results accepted by the shared schema, shard collector and renderer; failed-source output suppresses numerical comparisons.
bundle exec rspec: 2019 examples, 0 failures, 40 pending (optional OpenTelemetry integration bundle).
bundle exec rubocop: 145 files, no offenses; bundle exec rake public_api:check unchanged.
Python Black/Ruff, Actionlint and whitespace checks passed. Fresh read-only review recorded separately before publication.

Local durations are correctness/execution evidence only; hosted CI establishes the final latency budget. Intentional native scope differences and unsupported metrics are documented in CONTRIBUTING.md.

📝 Checklist

All commits are signed.
I reviewed the submitted code.
I added tests to verify the changes.
I updated the docs.
No breaking change or changelog entry.

🤖 Agent context

Autonomy: Human-driven (agent-assisted).

Implemented with Pi using file/shell tools and a fresh read-only reviewer. The benchmark preserves public SDK lifecycle and transport behavior; native Ruby counters are used where meaningful instead of equating allocation volume with retained memory. Human review is required.

stamphog

Not approved yet — waiting on the conditions below.

Re-add the stamphog label to request another review once you have addressed this.

stamphog can't auto-review this pull request because three gates refused it. The deny-list gate matched infra_cicd, since it adds .github/workflows/benchmarks.yml and .github/workflows/benchmark-report.yml. The size gate refused it because it has 1523 substantive lines against an 800-line ceiling. The tier gate classified it as T2-never (1897 lines across 13 files, two areas, a feature).

The refusal is final, so please ask a human reviewer to look at it. To make review easier, you could split the CI workflow changes from the benchmark scripts and fixtures so each part is smaller and more focused.

Gate mechanics and policy version

Gate		Result
prerequisites	✓	all clear
deny-list	✗	matches: infra_cicd
size	✗	too large for auto-review (1523L substantive in global — ceiling is 800L; 1523L, 11F total, 1897L/13F incl. docs/generated/snapshots)
tier	✗	classified as T2-never: T2-never (1897L, 13F, two-areas, feat)
stamphog 2.6.1		`.stamphog/policy.yml` @ `unknown` · reviewed head `b28bf78`

github-actions · 2026-10-30T15:07:26Z

posthog-ruby-sync Compliance Report

Date: 2026-11-04T20:28:32.113137+00:00
Duration: 55968ms

⚠️ Some Tests Failed

39/47 tests passed, 8 failed

Capture Tests

⚠️ 23/30 tests passed, 7 failed

View Details

Test	Status	Duration
Format Validation.Event Has Required Fields	✅	9ms
Format Validation.Event Has Uuid	✅	8ms
Format Validation.Event Has Lib Properties	✅	5ms
Format Validation.Distinct Id Is String	✅	6ms
Format Validation.Token Is Present	✅	5ms
Format Validation.Custom Properties Preserved	✅	5ms
Format Validation.Event Has Timestamp	✅	8ms
Format Validation.Non Utc Event Timestamp Is Converted To Utc	✅	6ms
Retry Behavior.Retries On 503	✅	5236ms
Retry Behavior.Does Not Retry On 400	❌	8ms
Retry Behavior.Does Not Retry On 401	❌	5ms
Retry Behavior.Respects Retry After Header	✅	8012ms
Retry Behavior.Implements Backoff	❌	271ms
Retry Behavior.Retries On 500	✅	5140ms
Retry Behavior.Retries On 502	✅	5116ms
Retry Behavior.Retries On 504	✅	5115ms
Retry Behavior.Max Retries Respected	❌	640ms
Deduplication.Generates Unique Uuids	✅	17ms
Deduplication.Preserves Uuid On Retry	✅	5111ms
Deduplication.Preserves Uuid And Timestamp On Retry	✅	10346ms
Deduplication.Preserves Uuid And Timestamp On Batch Retry	✅	5120ms
Deduplication.No Duplicate Events In Batch	✅	18ms
Deduplication.Different Events Have Different Uuids	✅	8ms
Compression.Sends Gzip When Enabled	✅	6ms
Batch Format.Uses Proper Batch Structure	✅	5ms
Batch Format.Flush With No Events Sends Nothing	✅	3ms
Batch Format.Multiple Events Batched Together	❌	16ms
Error Handling.Does Not Retry On 403	❌	4ms
Error Handling.Does Not Retry On 413	❌	5ms
Error Handling.Retries On 408	✅	5109ms

Failures

retry_behavior.does_not_retry_on_400

500, message='Internal Server Error', url='http://sdk-adapter:8080/capture'

retry_behavior.does_not_retry_on_401

500, message='Internal Server Error', url='http://sdk-adapter:8080/capture'

retry_behavior.implements_backoff

500, message='Internal Server Error', url='http://sdk-adapter:8080/capture'

retry_behavior.max_retries_respected

500, message='Internal Server Error', url='http://sdk-adapter:8080/capture'

batch_format.multiple_events_batched_together

Expected 1 requests, got 5

error_handling.does_not_retry_on_403

500, message='Internal Server Error', url='http://sdk-adapter:8080/capture'

error_handling.does_not_retry_on_413

500, message='Internal Server Error', url='http://sdk-adapter:8080/capture'

Feature_Flags Tests

⚠️ 16/17 tests passed, 1 failed

View Details

Test	Status	Duration
Request Payload.Request With Person Properties Device Id	✅	9ms
Request Payload.Flags Request Uses V2 Query Param	✅	6ms
Request Payload.Flags Request Hits Flags Path Not Decide	✅	8ms
Request Payload.Flags Request Omits Authorization Header	✅	9ms
Request Payload.Token In Flags Body Matches Init	✅	7ms
Request Payload.Groups Round Trip	✅	7ms
Request Payload.Groups Default To Empty Object	✅	6ms
Request Payload.Disable Geoip False Propagates As Geoip Disable False	✅	6ms
Request Payload.Disable Geoip Omitted Defaults To False	❌	6ms
Request Payload.Flag Keys To Evaluate Contains Only Requested Key	✅	5ms
Request Lifecycle.No Flags Request On Init Alone	✅	2ms
Request Lifecycle.No Flags Request On Normal Capture	✅	6ms
Request Lifecycle.Two Flag Calls Produce Two Remote Requests	✅	8ms
Request Lifecycle.Mock Response Value Is Returned To Caller	✅	6ms
Retry Behavior.Retries Flags On 502	✅	146ms
Retry Behavior.Retries Flags On 504	✅	150ms
Side Effect Events.Get Feature Flag Captures Feature Flag Called Event	✅	7ms

Failures

request_payload.disable_geoip_omitted_defaults_to_false

Field 'geoip_disable' not found in /flags request body at path 'geoip_disable'. Available keys: ['distinct_id', 'groups', 'person_properties', 'group_properties', 'flag_keys_to_evaluate', 'token']

github-actions · 2026-10-30T15:08:06Z

posthog-ruby-async Compliance Report

Date: 2026-11-04T20:29:20.654802+00:00
Duration: 98493ms

⚠️ Some Tests Failed

46/47 tests passed, 1 failed

Capture Tests

✅ 30/30 tests passed

View Details

Test	Status	Duration
Format Validation.Event Has Required Fields	✅	13ms
Format Validation.Event Has Uuid	✅	107ms
Format Validation.Event Has Lib Properties	✅	106ms
Format Validation.Distinct Id Is String	✅	107ms
Format Validation.Token Is Present	✅	110ms
Format Validation.Custom Properties Preserved	✅	108ms
Format Validation.Event Has Timestamp	✅	108ms
Format Validation.Non Utc Event Timestamp Is Converted To Utc	✅	11ms
Retry Behavior.Retries On 503	✅	5414ms
Retry Behavior.Does Not Retry On 400	✅	2110ms
Retry Behavior.Does Not Retry On 401	✅	2110ms
Retry Behavior.Respects Retry After Header	✅	8017ms
Retry Behavior.Implements Backoff	✅	15619ms
Retry Behavior.Retries On 500	✅	5212ms
Retry Behavior.Retries On 502	✅	5212ms
Retry Behavior.Retries On 504	✅	5211ms
Retry Behavior.Max Retries Respected	✅	15723ms
Deduplication.Generates Unique Uuids	✅	111ms
Deduplication.Preserves Uuid On Retry	✅	5212ms
Deduplication.Preserves Uuid And Timestamp On Retry	✅	10416ms
Deduplication.Preserves Uuid And Timestamp On Batch Retry	✅	5215ms
Deduplication.No Duplicate Events In Batch	✅	111ms
Deduplication.Different Events Have Different Uuids	✅	107ms
Compression.Sends Gzip When Enabled	✅	105ms
Batch Format.Uses Proper Batch Structure	✅	105ms
Batch Format.Flush With No Events Sends Nothing	✅	4ms
Batch Format.Multiple Events Batched Together	✅	111ms
Error Handling.Does Not Retry On 403	✅	2107ms
Error Handling.Does Not Retry On 413	✅	2108ms
Error Handling.Retries On 408	✅	5211ms

Feature_Flags Tests

⚠️ 16/17 tests passed, 1 failed

View Details

Test	Status	Duration
Request Payload.Request With Person Properties Device Id	✅	107ms
Request Payload.Flags Request Uses V2 Query Param	✅	106ms
Request Payload.Flags Request Hits Flags Path Not Decide	✅	107ms
Request Payload.Flags Request Omits Authorization Header	✅	107ms
Request Payload.Token In Flags Body Matches Init	✅	108ms
Request Payload.Groups Round Trip	✅	105ms
Request Payload.Groups Default To Empty Object	✅	106ms
Request Payload.Disable Geoip False Propagates As Geoip Disable False	✅	106ms
Request Payload.Disable Geoip Omitted Defaults To False	❌	106ms
Request Payload.Flag Keys To Evaluate Contains Only Requested Key	✅	105ms
Request Lifecycle.No Flags Request On Init Alone	✅	3ms
Request Lifecycle.No Flags Request On Normal Capture	✅	105ms
Request Lifecycle.Two Flag Calls Produce Two Remote Requests	✅	109ms
Request Lifecycle.Mock Response Value Is Returned To Caller	✅	106ms
Retry Behavior.Retries Flags On 502	✅	207ms
Retry Behavior.Retries Flags On 504	✅	241ms
Side Effect Events.Get Feature Flag Captures Feature Flag Called Event	✅	106ms

Failures

request_payload.disable_geoip_omitted_defaults_to_false

Field 'geoip_disable' not found in /flags request body at path 'geoip_disable'. Available keys: ['distinct_id', 'groups', 'person_properties', 'group_properties', 'flag_keys_to_evaluate', 'token']

posthog

PostHog Review · Standard · reviewed b28bf78: no findings. This pull request is large, so the review ran in 4 parts with less depth than usual.

marandaneto

The code looks good; no qualifying findings. Verdict: correct.

feat: add paired Ruby SDK performance benchmarks

b28bf78

dustinbyrne added the stamphog label Oct 30, 2026

dustinbyrne requested a review from a team as a code owner October 30, 2026 15:05

dustinbyrne self-assigned this Oct 30, 2026

stamphog Bot reviewed Oct 30, 2026

View reviewed changes

stamphog Bot removed the stamphog label Oct 30, 2026

posthog Bot reviewed Oct 30, 2026

View reviewed changes

dustinbyrne mentioned this pull request Oct 30, 2026

ci: add trusted SDK benchmark report workflow PostHog/.github#84

Merged

marandaneto approved these changes Nov 2, 2026

View reviewed changes

dustinbyrne added 3 commits November 3, 2026 14:46

ci: pin SDK benchmark reporting to mainline

4933353

fix: preserve benchmark failure reports after timeouts

1b06f3c

Merge main into feat/ruby-performance-benchmarks

71d3d55

dustinbyrne merged commit cd92a0e into main Nov 4, 2026
30 checks passed

dustinbyrne deleted the feat/ruby-performance-benchmarks branch November 4, 2026 21:07

Navigation Menu

Repository navigation

feat: add paired Ruby SDK performance benchmarks - #323

feat: add paired Ruby SDK performance benchmarks#323
dustinbyrne merged 4 commits into
mainfrom
feat/ruby-performance-benchmarks

dustinbyrne commented Oct 30, 2026

Uh oh!

stamphog Bot left a comment

Uh oh!

github-actions Bot commented Oct 30, 2026 •

edited

Loading

Failures

Failures

Uh oh!

github-actions Bot commented Oct 30, 2026 •

edited

Loading

Failures

Uh oh!

posthog Bot left a comment

Uh oh!

marandaneto left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

dustinbyrne commented Oct 30, 2026

💡 Motivation and Context

💚 How did you test it?

📝 Checklist

🤖 Agent context

Uh oh!

stamphog Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Oct 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

posthog-ruby-sync Compliance Report

⚠️ Some Tests Failed

Capture Tests

Failures

Feature_Flags Tests

Failures

Uh oh!

github-actions Bot commented Oct 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

posthog-ruby-async Compliance Report

⚠️ Some Tests Failed

Capture Tests

Feature_Flags Tests

Failures

Uh oh!

posthog Bot left a comment

Choose a reason for hiding this comment

Uh oh!

marandaneto left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions Bot commented Oct 30, 2026 •

edited

Loading

github-actions Bot commented Oct 30, 2026 •

edited

Loading