Skip to content

Clear a cached artifact and queue its file in one transaction - #506

Merged
andrew merged 4 commits into
git-pkgs:mainfrom
pinguinfuss:issue-467-atomic-queue
Oct 26, 2026
Merged

andrew merged 4 commits into
git-pkgs:mainfrom
pinguinfuss:issue-467-atomic-queue

Conversation

@pinguinfuss

Copy link
Copy Markdown
Contributor

Fixes #467.

ClearArtifactCache (and so DiscardArtifact) and UpsertArtifact now write the record and the pending_deletes entry in one transaction, the same way ClearExpiredArtifact does since #428. If queueing fails, the record keeps its old path, so the file is never left in storage with nothing pointing at it.

Eviction gets this from ClearArtifactCache: the path is queued before the file is deleted, and the entry is dropped again once the delete worked. A failed delete leaves it queued for reclaim, as before.

I ran the build against SQLite and Postgres under load: concurrent fetches, stored files deleted and corrupted underneath the proxy, eviction with a tiny max_size, and kill -9 mid-load. None of the three paths orphaned a file.

One trade-off: if only the queue keeps refusing writes while the rest of the database works, a record whose file is missing or corrupt can no longer be cleared or replaced. Every request then refetches from upstream and the new copy is orphaned. In my test with an insert-refusing trigger on pending_deletes, that left more orphans than main does. A real database failure would hit both writes, so I left it as it is.

Not covered here: a crash between writing a fetched file and committing its record in fetchAndCache still orphans that file. That happens on main as well; I can open a separate issue for it.

…e transaction, so a failed queue write keeps the record instead of leaving a file nothing points at. Eviction drops the queue entry again once it has deleted the file itself.

@andrew andrew left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please handle replacement-file cleanup when the transaction rolls back because the cleanup queue refuses writes, and cover it through the download entrypoint.

Comment thread internal/database/queries.go
pinguinfuss and others added 3 commits October 26, 2026 08:49
…, it is still served to everyone sharing the fetch and deleted once the last of them has closed it, so it no longer stays in storage with nothing pointing at it. A record that committed despite the reported error keeps its file.
…e transaction itself, so that part of this branch is gone; ClearArtifactCache, eviction and the unrecorded refetch remain, and an unrecorded refetch counts as failed for instances waiting on its lease.
@andrew
andrew merged commit fddef8e into git-pkgs:main Oct 26, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Clearing a cached artifact and queueing its file aren’t atomic

2 participants