Skip to content

Report client-closed connections (and more) accordingly - #992

Open
phbnf wants to merge 3 commits into
transparency-dev:mainfrom
phbnf:fix-499-client-closed
Open

phbnf wants to merge 3 commits into
transparency-dev:mainfrom
phbnf:fix-499-client-closed

Conversation

@phbnf

@phbnf phbnf commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator

Report client-closed connections as 499, and record the status code actually sent to the client

Thanks to @pimvanpelt for reporting this and doing the troubleshooting that tracked it down.

When a client hung up during the publication awaiter wait, TesseraCT logged an ERROR and counted a 500. Nobody received that 500: the connection was already gone. These requests are now reported as 499 (client closed request) and logged as warnings.

While fixing this, I also made sure the request log and HTTP metrics always record the status code that was actually sent to the client.


Changes

Area Before After
Client closed the connection ERROR log, status_code=500 in the request log and metrics WARN log, status_code=499 everywhere
Wrong HTTP method (405) Missing from tesseract.http.response.count; tesseract.http.request.duration had no status_code Recorded in both, with status_code=405
ParseForm failure (400) Same gap as 405 Recorded in both, with status_code=400
Handler returns non-200 without an error Client got a 500, but the handler's code was recorded status_code=500 recorded; span marked as an error
Handler panic No status in the request log or response counter Recorded as 500
Where the status is recorded Several places in ServeHTTP, some paths skipped Once, in a single defer, for the request log, response.count and request.duration

When the publication awaiter is enabled, add-(pre-)chain requests wait for
a checkpoint covering the new entry. If the client hangs up during that
wait, the request context is cancelled. ServeHTTP is meant to turn this into
a 499, but storage errors were wrapped with %v rather than %w, hiding the
context.Canceled error. The requests were therefore logged as ERRORs and
counted as 500s. Even with the right wrapping, the status was recorded in
the request log and metrics before being rewritten to 499.

More generally, the status recorded in the request log and metrics could
differ from the one sent to the client:
- 405 and form parsing 400 responses were never counted in
  tesseract.http.response.count, and had no status code attribute in
  tesseract.http.request.duration.
- Handlers returning a non-200 status without an error were recorded with
  that status, while clients received a 500.

Changes:
- Wrap storage errors in addChainInternal with %w.
- Record the final status code once, in a deferred function, in the request
  log, the response counter and the request duration histogram.
- Add tests covering every ServeHTTP return path, and an end-to-end
  add-(pre-)chain test with a client which closed the connection.
The client-closed case in TestServeHTTPRecordsFinalStatus used a fake
handler that wrapped the cancellation error itself, so it could not catch
the actual bug (storage errors wrapped with %v). Check the metrics in
TestAddChainClientClosedConnection instead, which goes through the real
storage path, and drop the redundant table case.

Merge the status tests into a single table using the real endpoints

TestHandlersRecordStatus replaces TestServeHTTPRecordsFinalStatus and
TestAddChainClientClosedConnection. Every case goes through the real
add-chain or add-pre-chain endpoint, with the publication awaiter enabled,
except the misbehaving handler case, which no real handler can trigger.
Replaces the setupTestLogWithPubAwaiter helper.
@phbnf phbnf changed the title Fix 499 client closed Report client-closed connections (and more) accordingly Nov 17, 2026
@phbnf
phbnf requested a review from AlCutter November 17, 2026 15:16
@phbnf
phbnf marked this pull request as ready for review November 17, 2026 15:16
@phbnf
phbnf requested a review from a team as a code owner November 17, 2026 15:16
Comment thread internal/ct/handlers.go
Comment on lines +206 to +208
// statusCode is the HTTP status code sent to the client. Every return path
// below must set it. It is recorded once, here, in the request log and
// metrics, so that they always agree with what the client received.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A cleaner way of enforcing this "must set" invariant might be to extract the code below into a func which returns the status code.

Comment thread internal/ct/handlers.go
latency := time.Since(startTime).Seconds()
reqDuration.Record(r.Context(), latency, metric.WithAttributes(attrs...))
if statusCode == 0 {
// The handler panicked: net/http will abort the connection.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it make sense to recover() here if we want to be sure this is actually what happened?

Comment thread internal/ct/handlers.go

if err != nil {
// If the client went away while the request was being processed, report
// it as such rather than as an internal server error.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This if condition could probably do with a bit of an explanation, too - do we really need to check both these places or are we missing another %w somewhere in a.handler?

Actually, how does this differentiate between ctx timing out and r.Context() being cancelled because the socket rolled up? It looks like ctx is a child of logCtx, which itself is a child of r.Context()?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants