Skip to content

Retry GCS requests Go's HTTP/2 client ends with its connection - #67

Merged
milos-lk merged 1 commit into
mainfrom
milos/gcs-retry-conn-errors
Nov 4, 2026
Merged

milos-lk merged 1 commit into
mainfrom
milos/gcs-retry-conn-errors

Conversation

@milos-lk

@milos-lk milos-lk commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

One more fix.

Follow-up to #66, which keeps a dead HTTP/2 connection from sitting until the idle deadline. This PR is about what happens to the requests when a connection ends anyway: the GCS client now retries the two errors with which Go's HTTP/2 client fails every open request on a connection it is closing.

How this differs from #66

  • Health-check HTTP/2 connections with pings #66 changes detection: a silent peer is pinged and the connection is dropped after 45 s with client connection lost, which the SDK already retries. It prevents the stall from ever reaching the idle-deadline race, but it only helps when the peer has gone quiet.
  • This PR changes classification: whatever closes a shared connection, the requests still open on it fail with one of two errors the SDK does not retry. After this PR it does. It covers two cases Health-check HTTP/2 connections with pings #66 cannot:
    • the server's own GOAWAY followed by a close before the in-flight streams finish (Google frontends do this at the end of a session's lifetime, debug session_timed_out); the peer is alive and answering pings the whole time;
    • a hard reset racing the same way as the stall, where there is no silence for a ping to notice.
  • Without this PR, one such connection event loses every upload in flight on it; with it, each is re-sent on a fresh connection with the existing backoff.

What changes

  • gcsShouldRetry is storage.ShouldRetry plus http2: client connection force closed via ClientConn.Close and http2: server sent GOAWAY and closed the connection, set client-wide with SetRetry(WithErrorFunc). Per-object retryers (gcsRetryAlways, the conditional write) inherit it and keep their own policy and backoff; the policy still decides which operations retry at all, this only widens which errors count.
  • The README's Retries bullet states it.

Worth knowing

  • Go re-sends only the streams a GOAWAY says the server never accepted (above LastStreamID); the ones already in flight get GoAwayError, which has no Temporary() and is not in the SDK's string list. Both error types are unexported in net/http, so a string match is the only handle, as the SDK uses for its own transport errors.
  • Retrying either is as safe as the SDK's existing retries of "connection reset" or "broken pipe": the request may have been received, which the overwrite policy and the DoesNotExist precondition both tolerate.
  • GCS is the only backend here whose endpoint negotiates HTTP/2, so no other store needs this.

Tests

  • TestGCSRetriesConnectionClose: a TLS HTTP/2 server that takes the request's HEADERS, answers GOAWAY with that stream still open and closes, which yields the real GoAwayError from Go's client; asserts the SDK's predicate rejects it and ours accepts it. The force-closed case uses the error as net/http wraps it. A nil error, a 404 and a connection reset are classified as before.

🤖 Generated with Claude Code

When Go's HTTP/2 client closes a connection, on its own write error or
on the server's GOAWAY followed by a close, every open request on it
fails with an error the GCS SDK's retry predicate does not match, so
each of those uploads is lost. The client's predicate now adds both.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@milos-lk
milos-lk marked this pull request as ready for review November 4, 2026 14:11
@milos-lk
milos-lk requested review from a team and theomonnom November 4, 2026 14:11
@milos-lk
milos-lk merged commit 975a7bf into main Nov 4, 2026
13 checks passed
@milos-lk
milos-lk deleted the milos/gcs-retry-conn-errors branch November 4, 2026 14:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants