Repository navigation
Track file_cache usage incrementally instead of running du - #1124
Merged
Merged
Conversation
file_cache measured its cache directory with common.GetUsage (du on Linux, a directory walk on Windows) on every StatFs, every eviction-policy pass and, with a hard limit set, every write that grows a file. Replace this with common/cache_size.Tracker, which keeps a running byte count that the component updates as files are downloaded, written, truncated, renamed and deleted. - The tracker reconciles against a full measurement at most every five minutes to correct drift, for example from concurrent extending writes. - exceedsHardLimit now takes the growth in bytes, so callers compute it from the cached file's current size. Appends use the file size as the offset. - The eviction policy reads the tracker instead of measuring, so the duPresent fallback is no longer needed. - On Windows, GetUsage now returns an error when the directory walk fails instead of silently reporting a partial size. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
foodprocessor
requested review from
Dabnsky,
Ferelith-maker,
brayan-trejo and
jfantinhardesty
as code owners
November 10, 2026 10:17
foodprocessor
temporarily deployed
to
testing
November 10, 2026 10:17 — with
GitHub Actions
Inactive
foodprocessor
temporarily deployed
to
testing
November 10, 2026 10:17 — with
GitHub Actions
Inactive
foodprocessor
temporarily deployed
to
testing
November 10, 2026 10:17 — with
GitHub Actions
Inactive
foodprocessor
temporarily deployed
to
testing
November 10, 2026 10:17 — with
GitHub Actions
Inactive
foodprocessor
temporarily deployed
to
testing
November 10, 2026 10:17 — with
GitHub Actions
Inactive
foodprocessor
temporarily deployed
to
testing
November 10, 2026 10:17 — with
GitHub Actions
Inactive
foodprocessor
temporarily deployed
to
testing
November 10, 2026 10:17 — with
GitHub Actions
Inactive
foodprocessor
temporarily deployed
to
testing
November 10, 2026 10:17 — with
GitHub Actions
Inactive
foodprocessor
temporarily deployed
to
testing
November 10, 2026 10:17 — with
GitHub Actions
Inactive
foodprocessor
temporarily deployed
to
testing
November 10, 2026 10:31 — with
GitHub Actions
Inactive
foodprocessor
temporarily deployed
to
testing
November 10, 2026 16:49 — with
GitHub Actions
Inactive
foodprocessor
temporarily deployed
to
testing
November 10, 2026 16:49 — with
GitHub Actions
Inactive
foodprocessor
temporarily deployed
to
testing
November 10, 2026 16:49 — with
GitHub Actions
Inactive
foodprocessor
temporarily deployed
to
testing
November 10, 2026 16:49 — with
GitHub Actions
Inactive
foodprocessor
temporarily deployed
to
testing
November 10, 2026 16:49 — with
GitHub Actions
Inactive
foodprocessor
temporarily deployed
to
testing
November 10, 2026 16:49 — with
GitHub Actions
Inactive
foodprocessor
temporarily deployed
to
testing
November 10, 2026 16:49 — with
GitHub Actions
Inactive
foodprocessor
temporarily deployed
to
testing
November 10, 2026 16:49 — with
GitHub Actions
Inactive
foodprocessor
temporarily deployed
to
testing
November 10, 2026 16:49 — with
GitHub Actions
Inactive
foodprocessor
temporarily deployed
to
testing
November 10, 2026 17:02 — with
GitHub Actions
Inactive
foodprocessor
temporarily deployed
to
testing
November 10, 2026 22:10 — with
GitHub Actions
Inactive
foodprocessor
temporarily deployed
to
testing
November 10, 2026 22:10 — with
GitHub Actions
Inactive
foodprocessor
temporarily deployed
to
testing
November 10, 2026 22:10 — with
GitHub Actions
Inactive
foodprocessor
temporarily deployed
to
testing
November 10, 2026 22:10 — with
GitHub Actions
Inactive
foodprocessor
temporarily deployed
to
testing
November 10, 2026 22:10 — with
GitHub Actions
Inactive
foodprocessor
temporarily deployed
to
testing
November 10, 2026 22:10 — with
GitHub Actions
Inactive
foodprocessor
temporarily deployed
to
testing
November 10, 2026 22:10 — with
GitHub Actions
Inactive
foodprocessor
temporarily deployed
to
testing
November 10, 2026 22:10 — with
GitHub Actions
Inactive
foodprocessor
temporarily deployed
to
testing
November 10, 2026 22:10 — with
GitHub Actions
Inactive
foodprocessor
temporarily deployed
to
testing
November 10, 2026 22:25 — with
GitHub Actions
Inactive
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What type of Pull Request is this? (check all applicable)
Describe your changes in brief
file_cache measured its cache directory with
common.GetUsage(duon Linux, a directory walk on Windows) on everyStatFs, every eviction-policy pass and, whenhard-limitis set, every write that grows a file. On large caches that is a full directory scan in the write path.This PR adds
common/cache_size.Tracker, which keeps a running byte count that file_cache updates as files are downloaded, written, truncated, renamed and deleted:exceedsHardLimitnow takes the growth in bytes. Callers compute it from the cached file's current size, and appends use the file size as the write offset.duPresentfallback is gone.GetUsagenow returns an error when the directory walk fails instead of silently reporting a partial size.Behavior to watch: between corrections, the tracked size can differ from the real disk usage. Changes made to the cache directory outside cloudfuse aren't seen until the next correction.
This doesn't depend on #1049 or #1123. #1049 adds the same
common/cache_sizepackage (tiered_storage uses it) and the sameutil_windows.gochange, so merging this first lets #1049 mergemainwithout conflicts.Checklist
Validation (Linux; Windows checked by cross-compiling):
go build -tags fuse3 ./...andGOOS=windows go build ./...: passedgo vet -tags fuse3,unittest ./common/cache_size/ ./component/file_cache/and theGOOS=windowsequivalent: passedgo test -count=1 ./common/cache_size/ ./component/file_cache/ --tags=unittest,fuse3: passedgolangci-lint run --build-tags fuse3 --new-from-rev=origin/main ./...: 0 issues (Linux andGOOS=windows)New tests cover the tracker (clamping at zero, summing several paths, a missing path, the reconcile interval), cache-policy deletes, file_cache usage accounting across create, write, truncate, rename, delete and download, and the hard limit across several files. Appends have no dedicated test.
Related Issues