Skip to content

Retry the remote ARAX call on gateway errors and dropped connections - #263

Merged
maximusunc merged 1 commit into
BioPack-team:develfrom
bazarkua:adil-catrax
Nov 16, 2026
Merged

maximusunc merged 1 commit into
BioPack-team:develfrom
bazarkua:adil-catrax

Conversation

@bazarkua

@bazarkua bazarkua commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Problem

In Max's CI load test, about 3% of queries on the legacy ARAX path (Shepherd calling arax.ci.transltr.io) failed with 502s, while ARAX's own query tracker showed no errors (RTXteam/RTX#2949). call_arax failed the query on the first error.

Change

workers/arax/worker.py, legacy path only. The in-process path is unchanged.

  • A 502, 503 or 504, or a dropped connection (httpx.NetworkError, httpx.RemoteProtocolError), gets up to 5 tries, waiting 5, 10, 20 then 40 s, each plus up to half again at random.
  • Timeouts and other errors still fail on the first try.
  • A retry is cut off at the query's deadline, since nobody is waiting for its answer after that.
  • After a retry, the error says how many attempts were made, and the span records arax.attempts.

The test fixture now sets the retry delay to 0, so existing tests that hit a failed call don't wait between tries.

Under Max's CI load test about 3% of queries on the legacy path failed with
502s from arax.ci.transltr.io, while ARAX's query tracker showed no errors
(RTXteam/RTX#2949). call_arax failed the query on the first error.

- A 502, 503 or 504, or a dropped connection, now gets up to five tries,
  waiting 5, 10, 20 then 40 seconds, each plus up to half again at random.
- Timeouts and other errors still fail on the first try.
- A retry is cut off at the query's deadline, since nobody is waiting for
  its answer after that.
- After a retry, the error says how many attempts were made, and the span
  records arax.attempts.
@maximusunc

Copy link
Copy Markdown
Collaborator

Thanks for the PR Adil! Quick question, what is the reasoning behind adding randomness to the retry? I could see exponential backoff up to a certain point, but I think the randomness on top of that is just adding more complexity for no gain.

@codecov

codecov Bot commented Nov 16, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 80.95238% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 57.72%. Comparing base (8346e35) to head (7a17335).
⚠️ Report is 2 commits behind head on devel.

Files with missing lines Patch % Lines
workers/arax/worker.py 80.95% 5 Missing and 3 partials ⚠️
Files with missing lines Coverage Δ
workers/arax/worker.py 85.39% <80.95%> (-0.69%) ⬇️

... and 1 file with indirect coverage changes


Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 20dddbb...7a17335. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@maximusunc

Copy link
Copy Markdown
Collaborator

Ok, I see now why the randomness would be a good idea under heavy load, so that many queries that might have failed at once don't all get sent again at the exact same time, which might cause more overload issues.

@maximusunc
maximusunc merged commit 4ffdf50 into BioPack-team:devel Nov 16, 2026
2 checks passed
@bazarkua

Copy link
Copy Markdown
Contributor Author

yep exactly, when arax gets overloaded a bunch of queries fail at the same time, so without the jitter they all come back after 5s together and hit it again at once. thanks for merging!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants