Repository navigation
Drain the log and stop background tasks on shutdown - #991
Conversation
… on shutdown Thread the shutdown function returned by tessera.NewAppender out to main and run it after draining the HTTP server on SIGINT/SIGTERM, then cancel the runtime context passed to NewLogHandler. - Tessera's appender shutdown blocks new Add calls and waits for a published checkpoint covering all locally assigned indices and for followers (such as GCP antispam) to catch up. Cancelling runCtx afterwards stops background coordinator, follower, awaiter, and root-fetch goroutines so in-flight Spanner R/W transactions roll back while OTel and Cloud Logging flush before exit. - Replace the hard-coded 60s HTTP drain with --shutdown_timeout (default 8s). Both the GCE module's docker stop and Cloud Run send SIGKILL after 10s, so 60s was never honoured. Cap HTTP drain at half of --shutdown_timeout (falling back to srv.Close) so requests near --http_deadline (default 10s) cannot starve the appender drain. cmd/tesseract/posix and cmd/tesseract/aws can adopt the same pattern in a follow-up.
Address review feedback: the lifecycle of the storage is now owned where
the storage is created, rather than by a wrapper in the gcp main, and
all three binaries shut down the same way.
- storage: CTStorageOptions takes the shutdown function returned by
tessera.NewAppender, and CTStorage gets a Shutdown method.
- tesseract.NewLogHandler returns a shutdown function alongside the
handler. It drains the storage, then stops the background tasks
NewLogHandler started, so the drain-before-cancel ordering lives in
one place. This changes NewLogHandler's signature.
- internal/lifecycle: the bounded HTTP-then-log drain moves here from
cmd/tesseract/gcp, with its tests.
- cmd/tesseract/{gcp,aws,posix}: all three define --shutdown_timeout
(default 8s) and call lifecycle.Shutdown on SIGINT/SIGTERM. For aws
and posix this replaces the hard-coded 60s HTTP-only drain. The
timeout is also passed to Tessera, which otherwise caps its own drain
at 60s.
- Document graceful shutdown in cmd/tesseract/README.md.
Address review feedback: - NewLogHandler returns a *LogHandler, which embeds http.Handler and has a Shutdown method, instead of a handler and a shutdown closure. Existing two-value call sites keep compiling. - Move the HTTP-then-log drain from internal/lifecycle to an exported tesseract.GracefulShutdown, so binaries outside this repository can use it too. It now takes a parent context. - Stop passing WithShutdownTimeout to Tessera: the context already bounds the drain. - Shorten the shutdown_timeout flag help; the README has the details. - Drop storage_test.go: TestNewLogHandlerShutdown covers the same path, and gains a case for a storage without a shutdown function.
| return srv, "http://" + ln.Addr().String() | ||
| } | ||
|
|
||
| func TestGracefulShutdownOrder(t *testing.T) { |
There was a problem hiding this comment.
Nit: In my experience in this repo and Tessera, LLMs have performed poorly at writing readable, table-driven tests. I know other folks have run into this as well, so maybe we should add an AGENTS.md or something like that.
In the meantime, I came to the conclusion chatting with AI and looking at the code that TestGracefulShutdownHTTPTimeoutReservesLogBudget and TestGracefulShutdownOrder could probably be merged into a single table-driven test. It's also adamant that we can get rid of mu, steps, and record (since handlerDone already tells us whether the HTTP handler has exited when shutdown runs)...
There was a problem hiding this comment.
It looks like you were right about the redundant variables. updated. WRT the table-driven thing I'm gonna leave that up to you. I personally am not fond of that pattern in go, but I'm admittedly very green when it comes to maintaining golang tests.
Address review nits: - Use consistent "HTTP server shutdown: starting.../done" and "TesseraCT log shutdown: starting.../done" log messages in GracefulShutdown. - Drop mu, steps, and record in TestGracefulShutdownOrder and TestGracefulShutdownHTTPTimeoutReservesLogBudget, asserting ordering directly via handlerDone and logCalled.
On SIGINT/SIGTERM, drain in-flight HTTP requests, then the Tessera appender, then stop background tasks, so instances don't exit holding Spanner locks that stall their replacements during rollouts.
API changes: the NewLogHandler return type, and CTStorageOptions is no longer comparable.