Repository navigation
Retry the remote ARAX call on gateway errors and dropped connections - #263
Conversation
Under Max's CI load test about 3% of queries on the legacy path failed with 502s from arax.ci.transltr.io, while ARAX's query tracker showed no errors (RTXteam/RTX#2949). call_arax failed the query on the first error. - A 502, 503 or 504, or a dropped connection, now gets up to five tries, waiting 5, 10, 20 then 40 seconds, each plus up to half again at random. - Timeouts and other errors still fail on the first try. - A retry is cut off at the query's deadline, since nobody is waiting for its answer after that. - After a retry, the error says how many attempts were made, and the span records arax.attempts.
945a0ea to
7a17335
Compare
|
Thanks for the PR Adil! Quick question, what is the reasoning behind adding randomness to the retry? I could see exponential backoff up to a certain point, but I think the randomness on top of that is just adding more complexity for no gain. |
Codecov Report❌ Patch coverage is
... and 1 file with indirect coverage changes Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
|
Ok, I see now why the randomness would be a good idea under heavy load, so that many queries that might have failed at once don't all get sent again at the exact same time, which might cause more overload issues. |
|
yep exactly, when arax gets overloaded a bunch of queries fail at the same time, so without the jitter they all come back after 5s together and hit it again at once. thanks for merging! |
Problem
In Max's CI load test, about 3% of queries on the legacy ARAX path (Shepherd calling
arax.ci.transltr.io) failed with 502s, while ARAX's own query tracker showed no errors (RTXteam/RTX#2949).call_araxfailed the query on the first error.Change
workers/arax/worker.py, legacy path only. The in-process path is unchanged.httpx.NetworkError,httpx.RemoteProtocolError), gets up to 5 tries, waiting 5, 10, 20 then 40 s, each plus up to half again at random.arax.attempts.The test fixture now sets the retry delay to 0, so existing tests that hit a failed call don't wait between tries.