Repository navigation
Conversation
When the publication awaiter is enabled, add-(pre-)chain requests wait for a checkpoint covering the new entry. If the client hangs up during that wait, the request context is cancelled. ServeHTTP is meant to turn this into a 499, but storage errors were wrapped with %v rather than %w, hiding the context.Canceled error. The requests were therefore logged as ERRORs and counted as 500s. Even with the right wrapping, the status was recorded in the request log and metrics before being rewritten to 499. More generally, the status recorded in the request log and metrics could differ from the one sent to the client: - 405 and form parsing 400 responses were never counted in tesseract.http.response.count, and had no status code attribute in tesseract.http.request.duration. - Handlers returning a non-200 status without an error were recorded with that status, while clients received a 500. Changes: - Wrap storage errors in addChainInternal with %w. - Record the final status code once, in a deferred function, in the request log, the response counter and the request duration histogram. - Add tests covering every ServeHTTP return path, and an end-to-end add-(pre-)chain test with a client which closed the connection.
The client-closed case in TestServeHTTPRecordsFinalStatus used a fake handler that wrapped the cancellation error itself, so it could not catch the actual bug (storage errors wrapped with %v). Check the metrics in TestAddChainClientClosedConnection instead, which goes through the real storage path, and drop the redundant table case. Merge the status tests into a single table using the real endpoints TestHandlersRecordStatus replaces TestServeHTTPRecordsFinalStatus and TestAddChainClientClosedConnection. Every case goes through the real add-chain or add-pre-chain endpoint, with the publication awaiter enabled, except the misbehaving handler case, which no real handler can trigger.
Replaces the setupTestLogWithPubAwaiter helper.
phbnf
marked this pull request as ready for review
November 17, 2026 15:16
AlCutter
reviewed
Nov 17, 2026
Comment on lines
+206
to
+208
| // statusCode is the HTTP status code sent to the client. Every return path | ||
| // below must set it. It is recorded once, here, in the request log and | ||
| // metrics, so that they always agree with what the client received. |
Collaborator
There was a problem hiding this comment.
A cleaner way of enforcing this "must set" invariant might be to extract the code below into a func which returns the status code.
| latency := time.Since(startTime).Seconds() | ||
| reqDuration.Record(r.Context(), latency, metric.WithAttributes(attrs...)) | ||
| if statusCode == 0 { | ||
| // The handler panicked: net/http will abort the connection. |
Collaborator
There was a problem hiding this comment.
Would it make sense to recover() here if we want to be sure this is actually what happened?
|
|
||
| if err != nil { | ||
| // If the client went away while the request was being processed, report | ||
| // it as such rather than as an internal server error. |
Collaborator
There was a problem hiding this comment.
This if condition could probably do with a bit of an explanation, too - do we really need to check both these places or are we missing another %w somewhere in a.handler?
Actually, how does this differentiate between ctx timing out and r.Context() being cancelled because the socket rolled up? It looks like ctx is a child of logCtx, which itself is a child of r.Context()?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Report client-closed connections as 499, and record the status code actually sent to the client
Thanks to @pimvanpelt for reporting this and doing the troubleshooting that tracked it down.
When a client hung up during the publication awaiter wait, TesseraCT logged an
ERRORand counted a 500. Nobody received that 500: the connection was already gone. These requests are now reported as 499 (client closed request) and logged as warnings.While fixing this, I also made sure the request log and HTTP metrics always record the status code that was actually sent to the client.
Changes
ERRORlog,status_code=500in the request log and metricsWARNlog,status_code=499everywheretesseract.http.response.count;tesseract.http.request.durationhad nostatus_codestatus_code=405ParseFormfailure (400)status_code=400status_code=500recorded; span marked as an errorServeHTTP, some paths skippeddefer, for the request log,response.countandrequest.duration