Skip to content

Drain the log and stop background tasks on shutdown - #991

Merged
patflynn merged 6 commits into
transparency-dev:mainfrom
patflynn:gcp-graceful-shutdown
Nov 16, 2026
Merged

patflynn merged 6 commits into
transparency-dev:mainfrom
patflynn:gcp-graceful-shutdown

Conversation

@patflynn

@patflynn patflynn commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

On SIGINT/SIGTERM, drain in-flight HTTP requests, then the Tessera appender, then stop background tasks, so instances don't exit holding Spanner locks that stall their replacements during rollouts.

  • storage: CTStorageOptions takes the shutdown func returned by tessera.NewAppender; CTStorage gets Shutdown.
  • tesseract.NewLogHandler returns a *LogHandler, which embeds http.Handler and has Shutdown. Existing two-value call sites keep compiling.
  • New tesseract.GracefulShutdown does the bounded HTTP-then-log drain; the gcp, aws and posix binaries all call it.
  • New --shutdown_timeout flag (default 8s) on all three binaries. For aws and posix this replaces a hard-coded 60s HTTP-only drain.

API changes: the NewLogHandler return type, and CTStorageOptions is no longer comparable.

… on shutdown

Thread the shutdown function returned by tessera.NewAppender out to main
and run it after draining the HTTP server on SIGINT/SIGTERM, then cancel
the runtime context passed to NewLogHandler.

- Tessera's appender shutdown blocks new Add calls and waits for a
  published checkpoint covering all locally assigned indices and for
  followers (such as GCP antispam) to catch up. Cancelling runCtx
  afterwards stops background coordinator, follower, awaiter, and
  root-fetch goroutines so in-flight Spanner R/W transactions roll back
  while OTel and Cloud Logging flush before exit.
- Replace the hard-coded 60s HTTP drain with --shutdown_timeout
  (default 8s). Both the GCE module's docker stop and Cloud Run send
  SIGKILL after 10s, so 60s was never honoured. Cap HTTP drain at half
  of --shutdown_timeout (falling back to srv.Close) so requests near
  --http_deadline (default 10s) cannot starve the appender drain.

cmd/tesseract/posix and cmd/tesseract/aws can adopt the same pattern in
a follow-up.
@patflynn
patflynn requested review from AlCutter and phbnf November 11, 2026 20:17
@patflynn
patflynn requested a review from a team as a code owner November 11, 2026 20:17
Comment thread cmd/tesseract/gcp/main.go Resolved
Address review feedback: the lifecycle of the storage is now owned where
the storage is created, rather than by a wrapper in the gcp main, and
all three binaries shut down the same way.

- storage: CTStorageOptions takes the shutdown function returned by
  tessera.NewAppender, and CTStorage gets a Shutdown method.
- tesseract.NewLogHandler returns a shutdown function alongside the
  handler. It drains the storage, then stops the background tasks
  NewLogHandler started, so the drain-before-cancel ordering lives in
  one place. This changes NewLogHandler's signature.
- internal/lifecycle: the bounded HTTP-then-log drain moves here from
  cmd/tesseract/gcp, with its tests.
- cmd/tesseract/{gcp,aws,posix}: all three define --shutdown_timeout
  (default 8s) and call lifecycle.Shutdown on SIGINT/SIGTERM. For aws
  and posix this replaces the hard-coded 60s HTTP-only drain. The
  timeout is also passed to Tessera, which otherwise caps its own drain
  at 60s.
- Document graceful shutdown in cmd/tesseract/README.md.
Comment thread cmd/tesseract/aws/main.go Resolved
Comment thread internal/lifecycle/lifecycle.go Resolved
Comment thread storage/storage_test.go Resolved
Comment thread cmd/tesseract/aws/main.go Resolved
Comment thread cmd/tesseract/gcp/main.go Resolved
Address review feedback:

- NewLogHandler returns a *LogHandler, which embeds http.Handler and has
  a Shutdown method, instead of a handler and a shutdown closure.
  Existing two-value call sites keep compiling.
- Move the HTTP-then-log drain from internal/lifecycle to an exported
  tesseract.GracefulShutdown, so binaries outside this repository can
  use it too. It now takes a parent context.
- Stop passing WithShutdownTimeout to Tessera: the context already
  bounds the drain.
- Shorten the shutdown_timeout flag help; the README has the details.
- Drop storage_test.go: TestNewLogHandlerShutdown covers the same path,
  and gains a case for a storage without a shutdown function.
@patflynn patflynn changed the title cmd/tesseract/gcp: drain Tessera appender and cancel background tasks on shutdown Drain the log and stop background tasks on shutdown Nov 13, 2026
Comment thread ctlog.go Resolved
Comment thread ctlog_test.go
return srv, "http://" + ln.Addr().String()
}

func TestGracefulShutdownOrder(t *testing.T) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: In my experience in this repo and Tessera, LLMs have performed poorly at writing readable, table-driven tests. I know other folks have run into this as well, so maybe we should add an AGENTS.md or something like that.

In the meantime, I came to the conclusion chatting with AI and looking at the code that TestGracefulShutdownHTTPTimeoutReservesLogBudget and TestGracefulShutdownOrder could probably be merged into a single table-driven test. It's also adamant that we can get rid of mu, steps, and record (since handlerDone already tells us whether the HTTP handler has exited when shutdown runs)...

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like you were right about the redundant variables. updated. WRT the table-driven thing I'm gonna leave that up to you. I personally am not fond of that pattern in go, but I'm admittedly very green when it comes to maintaining golang tests.

Address review nits:

- Use consistent "HTTP server shutdown: starting.../done" and
  "TesseraCT log shutdown: starting.../done" log messages in
  GracefulShutdown.
- Drop mu, steps, and record in TestGracefulShutdownOrder and
  TestGracefulShutdownHTTPTimeoutReservesLogBudget, asserting ordering
  directly via handlerDone and logCalled.
@patflynn
patflynn merged commit 36e50a1 into transparency-dev:main Nov 16, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants