Skip to content

Track file_cache usage incrementally instead of running du - #1124

Merged
foodprocessor merged 3 commits into
mainfrom
perf/file-cache-size-tracker
Nov 10, 2026
Merged

foodprocessor merged 3 commits into
mainfrom
perf/file-cache-size-tracker

Conversation

@foodprocessor

@foodprocessor foodprocessor commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

What type of Pull Request is this? (check all applicable)

  • Refactor
  • Feature
  • Bug Fix
  • Optimization
  • Documentation Update

Describe your changes in brief

file_cache measured its cache directory with common.GetUsage (du on Linux, a directory walk on Windows) on every StatFs, every eviction-policy pass and, when hard-limit is set, every write that grows a file. On large caches that is a full directory scan in the write path.

This PR adds common/cache_size.Tracker, which keeps a running byte count that file_cache updates as files are downloaded, written, truncated, renamed and deleted:

  • The tracker corrects drift by measuring the cache at most every five minutes. Drift can come from, for example, concurrent writes that extend the same file.
  • exceedsHardLimit now takes the growth in bytes. Callers compute it from the cached file's current size, and appends use the file size as the write offset.
  • The eviction policy reads the tracker instead of measuring the cache, so the duPresent fallback is gone.
  • On Windows, GetUsage now returns an error when the directory walk fails instead of silently reporting a partial size.

Behavior to watch: between corrections, the tracked size can differ from the real disk usage. Changes made to the cache directory outside cloudfuse aren't seen until the next correction.

This doesn't depend on #1049 or #1123. #1049 adds the same common/cache_size package (tiered_storage uses it) and the same util_windows.go change, so merging this first lets #1049 merge main without conflicts.

Checklist

  • Tested locally
  • Added new dependencies
  • Updated documentation
  • Added tests

Validation (Linux; Windows checked by cross-compiling):

  • go build -tags fuse3 ./... and GOOS=windows go build ./...: passed
  • go vet -tags fuse3,unittest ./common/cache_size/ ./component/file_cache/ and the GOOS=windows equivalent: passed
  • go test -count=1 ./common/cache_size/ ./component/file_cache/ --tags=unittest,fuse3: passed
  • golangci-lint run --build-tags fuse3 --new-from-rev=origin/main ./...: 0 issues (Linux and GOOS=windows)

New tests cover the tracker (clamping at zero, summing several paths, a missing path, the reconcile interval), cache-policy deletes, file_cache usage accounting across create, write, truncate, rename, delete and download, and the hard limit across several files. Appends have no dedicated test.

Related Issues

file_cache measured its cache directory with common.GetUsage (du on Linux,
a directory walk on Windows) on every StatFs, every eviction-policy pass and,
with a hard limit set, every write that grows a file. Replace this with
common/cache_size.Tracker, which keeps a running byte count that the
component updates as files are downloaded, written, truncated, renamed and
deleted.

- The tracker reconciles against a full measurement at most every five
  minutes to correct drift, for example from concurrent extending writes.
- exceedsHardLimit now takes the growth in bytes, so callers compute it from
  the cached file's current size. Appends use the file size as the offset.
- The eviction policy reads the tracker instead of measuring, so the
  duPresent fallback is no longer needed.
- On Windows, GetUsage now returns an error when the directory walk fails
  instead of silently reporting a partial size.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

@jfantinhardesty jfantinhardesty left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

This branch was previously deployed

1 inactive deployment
testing — 56f093b5 Deployed Nov 10, 2026 by foodprocessor via Cleanup Test Accounts (1.27, linux) #3954
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants