[yuechao-qin]

What

Both Kubernetes launchers now read a container log within their own budget, and give up quietly
when it runs out instead of letting the failure propagate into the caller's retry.

Per log that cannot be read	Before	After
Requests to the API server	5	1
Waits between them	5 × 1s	0
Worst case held by one lost log	~55s	~10s

10s is the launcher's existing request_timeout, which already bounds a single read.

Why

Reading a container log goes through the API server's proxy to the node's kubelet. A read that
fails has almost always lost the log for good: the node is gone or going, and the kubelet that
held the log went with it. OrchestratorService_Sql calls upload_log inside its own
five-attempt _retry, on a single-threaded loop — so every lost log costs that loop five
requests and five waits for a log that will never arrive.

Measured on one deployment over 72 hours: 323 failing log uploads produced 1723 attempt log
lines — 1723 / 323 = 5.33, the five attempts. Every failure in the window was the same
unreachable kubelet; the only other errors on that read path were 9 deleted Pods and one Pod
still initialising. Queries and raw figures are in
Shopify/oasis-backend#827, an internal
repository.

Key decisions

The budget wraps the log read only. The write to storage stays outside it and still raises,
so the caller's five retries continue to cover a failed upload — unlike a read from a dead node,
a failed write is worth retrying.

Giving up returns rather than raises. The caller catches Exception around upload_log, so
a launcher that raised after exhausting its own budget would simply have it applied five times.
Nothing is written on that path, so a log stored by an earlier attempt survives.

Nothing outside this file changes. The budget is the default arguments of one private
helper, so there is no class or base-class attribute for another launcher to inherit by
accident, and both call sites read _read_log_within_budget(self.get_log).

This bounds how long a failing read may hold the caller. What makes an unreachable kubelet stop
being an error at all is the companion change, #418;
the two are independent and can land in either order.

Verification

pytest --ignore=tests/instrumentation -q → 445 passed, black clean.

The tests build both launched containers through their real constructors and raise a real
ApiException carrying the body an unreachable kubelet actually returns, so the exception the
budget catches is one the Kubernetes client raised. One case drives the caller's real _retry
and asserts 1 request and 0 sleeps, against 5 and 5 for the same failure raised. Removing the
give-up return turns 7 of them red.

[Ark-kun]

Let's try a slightly different approach and move the configuration and the retries themselves to the launcher (Kubernetes launcher).

[Ark-kun]

Note the these are essentially unconfigurable (this is a private function that is always called without these optional parameters).

[Ark-kun]

Note that this log reading timeout is currently same as other request timeouts and is not affected by any of the new log timeout configurations.