Repository navigation
storage: keep garbage collecting while there is a backlog - #1277
weiscadence wants to merge 5 commits into
Conversation
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
3872768 to
2b1857a
Compare
|
Pushed e0de61b: the off-by-one fix (>= maxBundles, in posix/aws/gcp) and an end-to-end test of the job draining a 400-bundle backlog. On the timeout question I kept the existing DeadlineExceeded behaviour; happy to change it if maintainers prefer. |
Each garbage collection pass stopped after maxBundlesPerRun bundles, so a log growing faster than ~431 entries/s at the default interval accumulated obsolete partial tiles and entry bundles. garbageCollect now reports whether it stopped because it hit the limit, and garbageCollectorJob keeps calling it until the backlog is cleared or the existing GC timeout expires. The limit continues to bound the work (and, for AWS and GCP, the transaction) of each individual call. Fixes transparency-dev#1239
- Collect at most maxBundles per call (>= rather than >). - Add end-to-end test of the posix GC job draining a 400-bundle backlog.
As CONTRIBUTING.md asks for a first pull request.
cb50c7e to
ceaa23a
Compare
|
Thanks for taking this on, and no apology needed: this is the direction I'd have sent. I ran it against our setups, in case the results help review. Tests. The AWS tests you couldn't run pass against MySQL 8.4 ( Stress. I built backlogs with GC off, then collected them.
Two test gaps, found by breaking the change on purpose:
On your timeout question: with a long backlog, every tick logs a warning, and on AWS and GCP its error joins every in-flight prefix's failure, up to about 20 lines. Returning nil, or logging at info with the progress, when at least one call completed seems reasonable to me, keeping the warning for a tick that collected nothing. That's the maintainers' call, though. A GCP note, not for this PR: GCP deletes with a single worker, so at a 20 ms round trip a 30 s tick covered about 500 bundles, against about 1,050 on AWS. That's still about five times the old rate. This conflicts with my #1276 on one line of Tested with the assistance of an AI coding agent. |
Move the backlog loop out of garbageCollectorJob into garbageCollectBacklog, as in the POSIX driver, and test it on a backlog needing several calls. Check the number of bundles collected by each call in TestGarbageCollectLimit so the >= maxBundles fix is pinned.
|
Thanks @grempe for running this and for finding the gaps. Pushed df146b4:
I checked by breaking the change on GCP: |
|
Thanks, that covers both on AWS and GCP. I ran df146b4 here:
One is left, on POSIX: its for i, from := range passes[:len(passes)-1] {
if want := uint64(i+1) * maxBundles * layout.EntryBundleWidth; from != want {
t.Errorf("pass %d collected up to %d, want %d (passes: %v)", i, from, want, passes)
}
}Tested with the assistance of an AI coding agent. |
Fixes #1239.
Each GC pass stopped after
maxBundlesPerRunbundles in all three drivers, so a log growing faster than ~431 entries/s at the default interval accumulated obsolete partial tiles/entry bundles faster than they were removed.This is direction 1 from the issue: keep collecting while there's a backlog, bounded by the existing
defaultGCTimeout. No maintainer has picked a direction yet, and @grempe offered to send the PR, so I'm happy to change or close this in favour of whatever is preferred (see the note on #1239).garbageCollect(posix, AWS, GCP) now also returns whether it stopped because it hit the bundle limit.garbageCollectorJob(posix, AWS, GCP) loops while that's true, returningctx.Err()if the timeout expires mid-backlog. Progress from completed calls is already persisted, so the next tick resumes from there.maxBundlesPerRunstill bounds each call (and, on AWS/GCP, each DB transaction). No new option, and no behaviour change for logs without a backlog.d > maxBundlesoff-by-one mentioned in the issue is changed to>=in all three drivers, so each call handles exactlymaxBundlesPerRun(100) bundles rather than 101.Open question for maintainers: if a backlog cannot be cleared within the 30s timeout, the tick returns
ctx.Err(), which the job logs asGarbageCollect failedat warn level even though it made progress. I kept the existing error path; I can make that case return nil (or log at debug) if you prefer.Tests
TestGarbageCollectLimitin each driver: seeds partial bundles/tiles for more bundles than several passes at a small limit, assertsmoreis reported until the end, that gcState/partials are fully collected, and that a further call is a no-op.TestGarbageCollectBacklogDrains(posix) calls the extracted per-tick body,garbageCollectBacklog, directly on a 400-bundle backlog. It has no timers, so it does not depend on wall-clock timing. It replaces an earlier ticker-based version of the test.go test ./storage/posix/...and the GC tests in./storage/gcppass locally../storage/awsGC tests need MySQL, which I didn't have, so they were skipped here and only compile-checked (go vet); CI should exercise them.Also adds an
AUTHORS/CONTRIBUTORSentry, as CONTRIBUTING.md asks for a first PR.AI use: drafted by an AI agent (Claude); checked with the
go testandgo vetruns listed above.