Skip to content

storage: keep garbage collecting while there is a backlog - #1277

Open
weiscadence wants to merge 5 commits into
transparency-dev:mainfrom
weiscadence:gc-continue-past-bundle-limit
Open

weiscadence wants to merge 5 commits into
transparency-dev:mainfrom
weiscadence:gc-continue-past-bundle-limit

Conversation

@weiscadence

@weiscadence weiscadence commented Nov 11, 2026 •

Copy link
Copy Markdown

Fixes #1239.

Each GC pass stopped after maxBundlesPerRun bundles in all three drivers, so a log growing faster than ~431 entries/s at the default interval accumulated obsolete partial tiles/entry bundles faster than they were removed.

This is direction 1 from the issue: keep collecting while there's a backlog, bounded by the existing defaultGCTimeout. No maintainer has picked a direction yet, and @grempe offered to send the PR, so I'm happy to change or close this in favour of whatever is preferred (see the note on #1239).

  • garbageCollect (posix, AWS, GCP) now also returns whether it stopped because it hit the bundle limit.
  • garbageCollectorJob (posix, AWS, GCP) loops while that's true, returning ctx.Err() if the timeout expires mid-backlog. Progress from completed calls is already persisted, so the next tick resumes from there.
  • maxBundlesPerRun still bounds each call (and, on AWS/GCP, each DB transaction). No new option, and no behaviour change for logs without a backlog.
  • The d > maxBundles off-by-one mentioned in the issue is changed to >= in all three drivers, so each call handles exactly maxBundlesPerRun (100) bundles rather than 101.

Open question for maintainers: if a backlog cannot be cleared within the 30s timeout, the tick returns ctx.Err(), which the job logs as GarbageCollect failed at warn level even though it made progress. I kept the existing error path; I can make that case return nil (or log at debug) if you prefer.

Tests

  • New TestGarbageCollectLimit in each driver: seeds partial bundles/tiles for more bundles than several passes at a small limit, asserts more is reported until the end, that gcState/partials are fully collected, and that a further call is a no-op.
  • TestGarbageCollectBacklogDrains (posix) calls the extracted per-tick body, garbageCollectBacklog, directly on a 400-bundle backlog. It has no timers, so it does not depend on wall-clock timing. It replaces an earlier ticker-based version of the test.
  • Existing GC tests updated for the new return value.
  • go test ./storage/posix/... and the GC tests in ./storage/gcp pass locally. ./storage/aws GC tests need MySQL, which I didn't have, so they were skipped here and only compile-checked (go vet); CI should exercise them.
  • With the repro test from the issue, GC now covers the whole 400-bundle backlog within one tick (gcState goes 25600 → 102400). The repro's own assertion counts distinct gcState values rather than ticks, so it still reports two.

Also adds an AUTHORS/CONTRIBUTORS entry, as CONTRIBUTING.md asks for a first PR.

AI use: drafted by an AI agent (Claude); checked with the go test and go vet runs listed above.

@weiscadence
weiscadence requested a review from a team as a code owner November 11, 2026 15:04
@weiscadence
weiscadence requested a review from roger2hk November 11, 2026 15:04
@google-cla

google-cla Bot commented Nov 11, 2026

Copy link
Copy Markdown

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

@weiscadence
weiscadence force-pushed the gc-continue-past-bundle-limit branch from 3872768 to 2b1857a Compare November 11, 2026 15:38
@weiscadence

Copy link
Copy Markdown
Author

Pushed e0de61b: the off-by-one fix (>= maxBundles, in posix/aws/gcp) and an end-to-end test of the job draining a 400-bundle backlog. On the timeout question I kept the existing DeadlineExceeded behaviour; happy to change it if maintainers prefer.

Each garbage collection pass stopped after maxBundlesPerRun bundles, so a
log growing faster than ~431 entries/s at the default interval accumulated
obsolete partial tiles and entry bundles.

garbageCollect now reports whether it stopped because it hit the limit, and
garbageCollectorJob keeps calling it until the backlog is cleared or the
existing GC timeout expires. The limit continues to bound the work (and, for
AWS and GCP, the transaction) of each individual call.

Fixes transparency-dev#1239
- Collect at most maxBundles per call (>= rather than >).
- Add end-to-end test of the posix GC job draining a 400-bundle backlog.
As CONTRIBUTING.md asks for a first pull request.
@weiscadence
weiscadence force-pushed the gc-continue-past-bundle-limit branch from cb50c7e to ceaa23a Compare November 18, 2026 04:53
@grempe

grempe commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

Thanks for taking this on, and no apology needed: this is the direction I'd have sent. I ran it against our setups, in case the results help review.

Tests. The AWS tests you couldn't run pass against MySQL 8.4 (go test -p=1 -race ./storage/aws/... -is_mysql_test_optional=false), and so do go test -race ./... and golangci-lint.

Stress. I built backlogs with GC off, then collected them. main takes a tick per 101 bundles. With this PR, 400 to 4,000 bundles cleared in one tick on POSIX, AWS (an S3-compatible store with MySQL) and GCP (spannertest and fake-gcs-server). No obsolete partials were left, and fsck passed on every log afterwards. With 10 ms added each way to object storage traffic and the default 1-minute interval:

  • a 4,000-bundle AWS backlog took 4 ticks, about 1,050 bundles each until the last. Each of the first three ended at the 30 s timeout, and the next resumed from there;
  • at a steady 1,500 entries/s on AWS, collection caught up with the published checkpoint every tick.

Two test gaps, found by breaking the change on purpose:

  • On AWS and GCP the backlog loop is inline in garbageCollectorJob, so reverting it to a single call fails no test. Pulling it out as POSIX's garbageCollectBacklog is, and testing it on a backlog of several calls, would cover it.
  • Nothing pins the off-by-one fix. With d > maxBundles the limit tests still pass, since 38 bundles at 11 a call is still 4 calls. Checking each call's step would catch it (the POSIX test already records them in passes).

On your timeout question: with a long backlog, every tick logs a warning, and on AWS and GCP its error joins every in-flight prefix's failure, up to about 20 lines. Returning nil, or logging at info with the progress, when at least one call completed seems reasonable to me, keeping the warning for a tick that collected nothing. That's the maintainers' call, though.

A GCP note, not for this PR: GCP deletes with a single worker, so at a 20 ms round trip a 30 s tick covered about 500 bundles, against about 1,050 on AWS. That's still about five times the old rate.

This conflicts with my #1276 on one line of storage/aws/aws.go, in the sequencer interface. Keeping both lines resolves it, and the merged tree passes the storage tests. Whichever lands second needs that rebase.

Tested with the assistance of an AI coding agent.

Move the backlog loop out of garbageCollectorJob into garbageCollectBacklog,
as in the POSIX driver, and test it on a backlog needing several calls.
Check the number of bundles collected by each call in TestGarbageCollectLimit
so the >= maxBundles fix is pinned.
@weiscadence

Copy link
Copy Markdown
Author

Thanks @grempe for running this and for finding the gaps. Pushed df146b4:

  • AWS and GCP: the backlog loop is now garbageCollectBacklog, as in POSIX, with a new TestGarbageCollectBacklog in each that needs four calls to clear.
  • TestGarbageCollectLimit (AWS, GCP) now counts the bundles collected by each call and requires exactly maxBundles unless it is the last.

I checked by breaking the change on GCP: d > maxBundles fails the limit test (11 bundles per call instead of 10), and a single call instead of the loop fails the backlog test (1 call, want 4). The AWS tests are the same shape but I had no MySQL here, so they compile and vet but were not run.

@grempe

grempe commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

Thanks, that covers both on AWS and GCP. I ran df146b4 here:

  • The AWS tests pass against MySQL 8.4, the new TestGarbageCollectBacklog included (go test -p=1 -race ./storage/aws/... -is_mysql_test_optional=false). So do the GCP and POSIX tests and golangci-lint.
  • Breaking it the same ways as before, the single-call loop and d > maxBundles now fail tests on AWS and GCP.

One is left, on POSIX: its TestGarbageCollectLimit checks only where the last pass ends, so d > maxBundles in files.go still passes it. Checking that every pass but the last ends at the limit catches it. This fails with d > maxBundles and passes without:

for i, from := range passes[:len(passes)-1] {
	if want := uint64(i+1) * maxBundles * layout.EntryBundleWidth; from != want {
		t.Errorf("pass %d collected up to %d, want %d (passes: %v)", i, from, want, passes)
	}
}

Tested with the assistance of an AI coding agent.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Garbage collection covers at most 101 bundles per pass (~431 entries/s at the default interval)

2 participants