diff --git a/README.md b/README.md index f684039..c4acaab 100644 --- a/README.md +++ b/README.md @@ -1,14 +1,33 @@ # πŸš€ GitHub Actions Cache Server -This is a fast, lightweight, self-hosted drop-in replacement for the official GitHub hosted cache server, written in Rust. It is compatible with the official `actions/cache` action, so there is no need to change your workflow files and it even works with packages that internally use `actions/cache`. +A self-hosted, drop-in replacement for GitHub's cache service, written in Rust. Point your self-hosted runners at it and every `actions/cache` step, `setup-*` cache and `docker buildx` `type=gha` cache stays on your own hardware. No workflow changes. + +| | GitHub Actions Cache Server | +| ------------------------------- | --------------------------- | +| Image size | 29 MB | +| Startup to healthy | 0.5 s | +| Idle memory | 3 MiB | +| Upload (8 Γ— 128 MiB, parallel) | 2.3 GiB/s | +| First download | 7.9 GiB/s | +| Lookups (64 concurrent) | 34,000/s, p99 7 ms | +| Small save + restore (256 KiB) | 1,143/s | + +SQLite + filesystem on a Ryzen 9 5950X with NVMe. Reproduce with `cargo run --release --example benchmark`. ## Features -- πŸ”₯ **Compatible with official `actions/cache` action** -- ⚑ Very fast and memory efficient: a single Rust binary in a tiny image that starts in under a second and idles at a few MiB of memory. -- πŸ“¦ Supports multiple storage solutions and is easily extendable. -- πŸ”’ Secure and self-hosted, giving you full control over your cache data. -- 😎 Easy setup +- πŸ”₯ **Works with everything that uses the cache**: `actions/cache`, `actions/setup-node` & friends, BuildKit `type=gha`, any tool built on `@actions/cache` +- πŸ—„οΈ **Your storage**: filesystem, S3 (RustFS, MinIO, R2, ...), Google Cloud Storage or Azure Blob Storage +- 🐘 **Your database**: SQLite, PostgreSQL or MySQL +- πŸ”’ **Secure by default**: verifies the runner's OIDC token, scopes every cache to its repository, signs each upload URL +- ⚑ **Direct downloads**: optionally hand runners a signed storage URL so restores skip the server +- 🧹 **Runs itself**: age- and size-based eviction, orphan cleanup, self-healing when storage drifts +- πŸ“Š **Observable**: Prometheus `/metrics`, `/health`, management API with OpenAPI docs +- ☸️ **Deploy anywhere**: multi-arch image (amd64/arm64), Docker Compose or Helm, GitHub Enterprise Server + +## Quick start + +**1. Run the server** ```yaml services: @@ -18,17 +37,25 @@ services: - '3000:3000' environment: API_BASE_URL: http://localhost:3000 - STORAGE_DRIVER: filesystem - STORAGE_FILESYSTEM_PATH: /data/cache - DB_DRIVER: sqlite - DB_SQLITE_PATH: /data/cache-server.db volumes: - - cache-data:/data + - cache-data:/app/.data volumes: cache-data: ``` +**2. Point your runners at it** + +Swap the runner image for our fork and set one variable: + +```yaml +image: ghcr.io/falcondev-oss/actions-runner:latest +env: + CUSTOM_ACTIONS_RESULTS_URL: http://cache-server:3000/ +``` + +**3. There is no step 3.** Your workflows already use it. + ## Documentation πŸ‘‰ πŸ‘ˆ diff --git a/docs/adr/0001-database-records-authorize-retained-storage.md b/docs/adr/0001-database-records-authorize-retained-storage.md deleted file mode 100644 index 66a322d..0000000 --- a/docs/adr/0001-database-records-authorize-retained-storage.md +++ /dev/null @@ -1,3 +0,0 @@ -# Database records authorize retained storage - -The database is authoritative when it disagrees with physical storage. The configured filesystem root or `gh-actions-cache/` object-store prefix is wholly owned by the server. Database state changes are committed before their corresponding physical deletions; a failed physical deletion therefore creates Orphaned Storage for reconciliation instead of leaving a Cache Entry that points to missing data. Stored cache data that is referenced by neither an upload nor a storage location may be deleted once its newest object is at least 24 hours old; the grace period has a dedicated setting with a 24-hour default. Consequently, restoring an older database while retaining newer physical storage can delete data unknown to that database. This favors bounded storage use and automatic recovery from interrupted cleanup over preserving untracked data indefinitely. diff --git a/docs/adr/0002-part-reader-leases-protect-active-downloads.md b/docs/adr/0002-part-reader-leases-protect-active-downloads.md deleted file mode 100644 index 66a4d93..0000000 --- a/docs/adr/0002-part-reader-leases-protect-active-downloads.md +++ /dev/null @@ -1,3 +0,0 @@ -# Storage Reader Leases protect active downloads - -Concurrent downloads hold durable, expiring Storage Reader Leases, including direct downloads, whose lease outlives the signed URL by a fixed margin. A renewable, fenced Merge Lease grants a single worker authority to complete the Merge; Part Reader Leases protect concurrent readers of Parts. Both expiring lease types renew every 30 seconds while active and expire after two minutes, except direct-download leases, which expire ten minutes after their signed URLs: storage authenticates a GET when it starts, so a transfer begun just before the URL expires keeps reading after it. Parts are deleted only after the merge completes and no active Part Reader Leases remain; a Storage Location is deleted only after no Storage Reader Leases remain. A reader acquires its lease in the same locked transaction that reads the Storage Location's merge state, so its scope is chosen atomically: a download arriving before the merge takes a Part Reader Lease that blocks parts deletion, and one arriving after takes a Storage Reader Lease over the merged representation. There is therefore no window in which a reader can fail to acquire a Part Reader Lease against parts that are being deleted. diff --git a/docs/adr/0003-independent-helm-chart-versioning.md b/docs/adr/0003-independent-helm-chart-versioning.md deleted file mode 100644 index 8f897fa..0000000 --- a/docs/adr/0003-independent-helm-chart-versioning.md +++ /dev/null @@ -1,3 +0,0 @@ -# Helm chart versions are independent from application versions - -The Helm chart's SemVer version is independent of the application version it deploys. Every application release changes the chart's default application version and therefore increments the chart patch version by default; releases that include a chart-level feature or breaking change explicitly select a chart minor or major bump. This keeps chart packages uniquely versioned while allowing their compatibility contract to evolve independently of the application. diff --git a/docs/adr/0004-atomically-visible-uploads-replace-merge-promotion.md b/docs/adr/0004-atomically-visible-uploads-replace-merge-promotion.md deleted file mode 100644 index 96be911..0000000 --- a/docs/adr/0004-atomically-visible-uploads-replace-merge-promotion.md +++ /dev/null @@ -1,3 +0,0 @@ -# Atomically visible uploads replace merge promotion - -Every storage adapter's `uploadStream` guarantees atomic visibility: an object never exists partially, and overwriting an object never disturbs active readers of the previous version. Object stores provide this natively (multipart and resumable uploads only appear on completion); the filesystem adapter writes to a top-level temp entry and renames it into place, leaving crash debris as Orphaned Storage for reconciliation per ADR-0001. The Merge therefore writes directly to its final object with no intermediate candidate-and-promote step: because Parts are immutable, a merger that lost its Merge Lease can only overwrite the merged object with identical bytes, so the lease fence only guards which worker marks the Merge complete. The rejected alternative β€” uploading to a candidate object and promoting it under the lease β€” required a server-side copy that S3 caps at 5 GiB and pinned a database transaction across the copy. diff --git a/docs/adr/0005-lookup-validates-storage-and-self-heals-dangling-entries.md b/docs/adr/0005-lookup-validates-storage-and-self-heals-dangling-entries.md deleted file mode 100644 index d4add9c..0000000 --- a/docs/adr/0005-lookup-validates-storage-and-self-heals-dangling-entries.md +++ /dev/null @@ -1,9 +0,0 @@ -# Cache lookup validates storage and self-heals Dangling Cache Entries - -Returning a download URL from `GetCacheEntryDownloadURL` is a promise that the data exists: once a client holds the URL it has committed to downloading, and the GHA cache protocol gives it no way to walk a download failure back to a cache miss. `@actions/cache` tolerates a broken promise by retrying 5xx and falling back on 4xx, but BuildKit's `gha` cache importer treats any non-200 on the download URL as a hard build failure. Under ADR-0001 the server never creates a Dangling Cache Entry itself, so a broken promise only arises from external mutation of storage the server owns (bucket wipe, external lifecycle expiry, out-of-sync database restore). We choose to own that drift rather than declare it operator error, because self-hosted object stores, bucket lifecycle rules, and backup restores are the daily reality of this project's users, and the alternative failure mode β€” a silent hard build break that self-heals only after the 30-day cache-entry cleanup or manual database surgery β€” is unacceptable for a drop-in cache. - -The lookup therefore validates storage before returning a URL, and before taking any Storage Reader Lease (nothing should lease provably-dead data): a merged entry is confirmed by a `HEAD` on its merged object; an unmerged entry by a `HEAD` on its first Part (`parts/0`); an entry whose Parts were deleted with no completed Merge is Dangling with no storage call. A `HEAD` fires only on a cache hit, where it is negligible against the transfer that follows. Checking only the first Part will not detect surgical deletion of an interior Part, but the server's invariants never produce partial Part loss and external drift is folder-level all-or-nothing, so a single `HEAD` is sufficient. - -On a failed check the lookup deletes the Dangling `cache_entries` row (and reaps the now-childless `storage_location`, which an Orphaned Storage sweep cannot see) and re-runs matching so a valid candidate under another restore key still wins, bounded to a small retry count to cap a pathological same-prefix stale flood; if no valid match remains it returns `{ ok: false }` for a clean miss. This self-heal deliberately reverses the read-purity choice of #223, which removed read-path deletes: that choice was right when the signal was ambiguous, but here the deleted row is provably dead β€” its storage was just confirmed absent β€” so eager reconciliation on read is reconciliation in ADR-0001's own spirit, done at read time instead of by a background sweep, and it avoids re-paying the validation `HEAD` on every future restore of the same dead key. - -Object storage answers this check from memory for a minute after it last saw the object, so a lookup does not cost a request each. Deletions made through the server evict it immediately; data deleted behind the server's back (another replica, a bucket lifecycle rule) can still be promised for up to that minute. diff --git a/docs/adr/0006-oidc-issuer-is-configurable-signature-verification-stays-mandatory.md b/docs/adr/0006-oidc-issuer-is-configurable-signature-verification-stays-mandatory.md deleted file mode 100644 index 8853dc7..0000000 --- a/docs/adr/0006-oidc-issuer-is-configurable-signature-verification-stays-mandatory.md +++ /dev/null @@ -1,13 +0,0 @@ ---- -status: proposed ---- - -# OIDC issuer is configurable; signature verification stays mandatory - -GitHub Actions cache tokens are verified in `lib/scope.ts` against a hard-coded issuer and JWKS endpoint (`https://token.actions.githubusercontent.com`), which is correct for github.com but rejects every token issued by a GitHub Enterprise Server instance, whose Actions tokens carry the GHES host as their issuer. To support GHES (best-effort, per issue #241) we make the issuer configurable via a single env var defaulting to the current github.com value, and derive the JWKS URL from it as `{issuer}/.well-known/jwks` β€” the layout GHES inherits from the same Actions stack. github.com users change nothing. - -We rejected the zero-code alternative of telling GHES operators to set `SKIP_TOKEN_VALIDATION=true`, because that flag disables signature verification entirely: any client that can reach the server could then forge scopes and read or poison any repository's cache. `SKIP_TOKEN_VALIDATION` stays what it is β€” a dev/test escape hatch, explicitly not a production deployment path. Keeping verification mandatory means a GHES deployment is exactly as authenticated as a github.com one, at the cost of the operator supplying their issuer. We deliberately derive the JWKS URL rather than accept it as a second var; if a real GHES layout ever splits the JWKS host from the issuer host, an explicit override can be added then. - -**Amended (issue #253):** such a layout exists, so we no longer derive the JWKS URL β€” we discover it. GitHub Enterprise Cloud lets an admin customize the issuer to `https://token.actions.githubusercontent.com/` while the JWKS stays at `https://token.actions.githubusercontent.com/.well-known/jwks`; the derived URL 404s and every token fails verification. The OIDC discovery document at `{issuer}/.well-known/openid-configuration` names the right `jwks_uri` in every case we can observe, including the custom-issuer one, so it is the default source of truth and this class of bug can't recur for any layout that publishes discovery. - -Discovery is fetched once, lazily, on the first token verification, and cached for the process. When it fails we log a warning and fall back to the previously derived `{issuer}/.well-known/jwks` β€” deliberately _without_ caching, so a discovery outage can't pin the process to a URL that may be wrong. The fallback keeps GHES deployments (whose token hosts we can't reach to confirm they serve discovery) working exactly as before, so the change can't regress the support added above. `ACTIONS_TOKEN_JWKS_URL` remains as an explicit override that skips discovery entirely, for operators whose deployment serves neither a usable discovery document nor the conventional path. diff --git a/docs/adr/0007-per-process-metrics-no-cross-worker-aggregation.md b/docs/adr/0007-per-process-metrics-no-cross-worker-aggregation.md deleted file mode 100644 index 9fda7c3..0000000 --- a/docs/adr/0007-per-process-metrics-no-cross-worker-aggregation.md +++ /dev/null @@ -1,9 +0,0 @@ -# Prometheus metrics are per-process with no cross-worker aggregation - -> Superseded in part by ADR-0010: the server is a single process without cluster workers. - -The `/metrics` endpoint exposes each worker process's own in-memory counters from a `prom-client` registry. It does not aggregate across the workers of a `node-cluster` deployment. This is a deliberate limitation, because the correct fix is disproportionate to the payoff for how this server is actually run. - -The shipped Docker image sets `NITRO_CLUSTER_WORKERS=1` (one worker per container), and horizontal scaling is done with replicas. In that topology per-process metrics are exactly correct: Prometheus scrapes each replica as its own target and you aggregate in the query layer with `sum(rate(...))` β€” which is the idiomatic Prometheus pattern, not a workaround. The limitation only bites when an operator overrides `NITRO_CLUSTER_WORKERS` above 1: the workers then share the listening port round-robin, so successive scrapes hit different workers and counters appear to jump around. We document that "run one worker per container and scale with replicas" is the supported way to get accurate metrics. - -Aggregating in-app was rejected because there is no cheap seam for it here. `prom-client`'s cluster support (`AggregatorRegistry.clusterMetrics()`) must run in the cluster **primary**, but Nitro's `node-cluster` preset primary is a fixed internal function we get no hook into β€” owning it would mean ejecting the preset and re-implementing its worker management and graceful shutdown. The other route, persisting counters through the shared database, adds a write on every request plus new schema for a signal that Prometheus is happy to sum itself. Neither cost is justified while the default and recommended deployment is single-worker. If multi-worker aggregation is ever genuinely needed, `AggregatorRegistry` behind a custom cluster entrypoint is the known upgrade path, and this decision is reversible at that point. diff --git a/docs/adr/0008-post-completion-capacity-eviction.md b/docs/adr/0008-post-completion-capacity-eviction.md deleted file mode 100644 index a755a13..0000000 --- a/docs/adr/0008-post-completion-capacity-eviction.md +++ /dev/null @@ -1,7 +0,0 @@ -# Capacity-based Eviction runs after upload completion - -Finalized cache payloads are measured in bytes. Every completed upload records its own payload size by measuring only that folder, so the total is always available (for the `cache_storage_bytes` metric and for eviction) without a full-storage scan per upload. Filesystem storage has a configurable Storage Budget percentage, defaulting to 90% of Filesystem Capacity; any backend may instead use an explicit byte budget. Object storage is unlimited without that explicit budget. - -After an upload completes, Capacity-based Eviction runs only when usage exceeds the Storage Budget and deletes Cache Entries in Cache Recency order until usage is at most 90% of the budget. A Cache Access is recorded when a proxied download starts or a direct-download URL is issued. Cache Recency falls back to the Cache Entry's save time. Active Storage Reader Leases continue to prevent deletion. - -Locations predating size tracking are reconciled resumably on startup with a single full-storage scan, skipped once every location has a recorded size. This design deliberately has no pre-upload checks, chunk accounting, reservations, or global capacity lock. Concurrent completions may temporarily exceed the budget, and an oversized completed payload may be evicted while upload finalization still succeeds. Operators must leave suitable headroom for in-progress uploads and concurrency. diff --git a/docs/adr/0009-eager-merge-is-opt-in-and-composes-server-side-on-object-storage.md b/docs/adr/0009-eager-merge-is-opt-in-and-composes-server-side-on-object-storage.md deleted file mode 100644 index c9fac47..0000000 --- a/docs/adr/0009-eager-merge-is-opt-in-and-composes-server-side-on-object-storage.md +++ /dev/null @@ -1,11 +0,0 @@ -# Eager Merge is opt-in and composes server-side on object storage - -> Superseded in part by ADR-0010: Server-side Merge was removed; Eager Merge streams the Parts through the server on object storage. - -Direct downloads need a merged object, and the Merge runs lazily on first download, so the first Cache Hit of every entry is proxied through the server even with `ENABLE_DIRECT_DOWNLOADS`. `EAGER_MERGE=true` runs the Merge right after upload completion instead. It stays off by default because lazy merging never pays merge traffic for entries that are never downloaded. - -Both paths share one merge runner. Merge Lease, renewal, lease-fenced completion, and rollback are identical, so eager and lazy merges race safely against each other and against `cleanup:merges`. Upload completion picks the strategy up front from the Part sizes it already lists to record `sizeBytes`. When the adapter offers `composeParts` and every Part satisfies its limits, the Merge is a Server-side Merge with no bytes passing through the server. On S3 that is `CreateMultipartUpload`, one `UploadPartCopy` per Part, `CompleteMultipartUpload`; the limits are every Part but the last at least 5 MiB, none above 5 GiB, at most 10,000 Parts. On GCS it is `compose`, folding 32 sources at a time into a top-level temp object and composing `merged` in one final call; there is no minimum Part size. Otherwise the existing streaming merge runs immediately. Buildx uploads 1 MiB blocks, so its entries always take the streaming path. A backend that rejects the copy call fails the Merge like any other merge failure; the entry then merges lazily on first download. Eager and lazy Merges share the per-process `MAX_CONCURRENT_MERGES` cap; an Eager Merge past it is skipped the same way, and the entry merges lazily on first download. - -This does not reopen ADR-0004. The 5 GiB cap it cites applies to `CopyObject` used as a promote step. `UploadPartCopy` writes `merged` directly under the same lease fence, and Parts are immutable, so a merger that lost its lease can only write identical bytes. - -Costs: every entry occupies twice its size until `cleanup:parts` removes the Parts, not only entries that were downloaded, and `sizeBytes` still counts Parts only. A worker killed mid-composition leaves an incomplete multipart upload on S3 (configure an `AbortIncompleteMultipartUpload` lifecycle rule) or a temp object on GCS (reclaimed as Orphaned Storage). `FinalizeCacheEntryUpload` waits only for the Merge Lease, not for the Merge. The filesystem driver has no server-side copy and always streams. diff --git a/docs/adr/0010-single-rust-binary.md b/docs/adr/0010-single-rust-binary.md deleted file mode 100644 index a71c762..0000000 --- a/docs/adr/0010-single-rust-binary.md +++ /dev/null @@ -1,16 +0,0 @@ -# The server is a single Rust binary - -> Superseded in part by ADR-0011: the database is migrated by `sea-orm-migration` and no longer shared with v9. - -The server was rewritten from Nitro/Node.js to one statically linked Rust binary (axum on tokio). The wire protocols, environment variables, database schema and storage layout are unchanged, so an existing deployment upgrades by swapping the image. - -The schema and its Kysely bookkeeping tables (`kysely_migration`, `kysely_migration_lock`) are kept as they are, migrations take the same Postgres advisory lock and MySQL named lock Kysely used, and fresh databases record the same migration names. A database written by either implementation stays readable by the other. - -Database access goes through SeaORM entities on native sqlx pools, so one code path renders correctly quoted SQL for Postgres, MySQL and SQLite. SQLite has one writer connection, as with `better-sqlite3`, and runs in WAL mode so a pool of read-only connections serves lookups beside it. S3, Google Cloud Storage and Azure Blob Storage share one adapter on the `object_store` crate; the filesystem adapter is separate because it needs `statvfs` for the Storage Budget. - -Superseded parts of earlier decisions: - -- ADR-0007: there are no cluster workers any more. One process serves all requests on a multi-threaded runtime, so its metrics are already process-wide; scaling is still by replicas. -- ADR-0009: Server-side Merge is gone. `object_store` has no `UploadPartCopy`, GCS `compose` or Azure `Put Block From URL`, and one adapter for all object stores was worth more than the optimization. On object storage, Eager Merge always streams the Parts through the server; on the filesystem, a Merge is a kernel copy (`copy_file_range`). - -Dropped with Node.js: the `-caged` image variant (V8 pointer compression), the `BENCHMARK` heap logging, and the oRPC `/management-api/_rpc` endpoint with the TypeScript SDK, which typed itself from the server source. The REST management API and its OpenAPI document remain. diff --git a/docs/adr/0011-rust-first-internals.md b/docs/adr/0011-rust-first-internals.md deleted file mode 100644 index 31486d3..0000000 --- a/docs/adr/0011-rust-first-internals.md +++ /dev/null @@ -1,13 +0,0 @@ -# Internals use the Rust ecosystem instead of mirroring v9 - -The HTTP API, environment variables and storage layout stay what v9 offered. How the server implements them no longer follows the TypeScript server: each piece uses the crate the Rust ecosystem settled on. - -- Migrations run on `sea-orm-migration` from one frozen baseline, the schema v9.10 left. A database still carrying Kysely's bookkeeping is taken over on startup: with all nine v9.10 migrations recorded it is adopted with its data, anything older is dropped and recreated and storage is cleared with it. Kysely tables a v9 replica recreates beside an already migrated database are simply removed. Migrations keep Kysely's advisory locks, so such a replica waits instead of racing. Neither needs operator input, and a cache can afford losing data that old. The two implementations can no longer share a database. -- Configuration is a `clap` parser; every variable doubles as a flag and `--help` lists them. An empty variable counts as unset. -- The Twirp `CacheService` is generated by `twirp-build` and `pbjson` from `proto/cache.proto`, compiled by `protox` so building needs no `protoc`. Errors use Twirp's standard `{code, msg}` body. -- Management API query strings are parsed by `serde_qs` into typed structs; `Range` headers by `http-range-header`, which adds suffix ranges. -- Cleanup schedules are `tokio` intervals that run once at startup and then every period, not aligned to local wall-clock time. - -Superseded parts of earlier decisions: - -- ADR-0010: the Kysely bookkeeping tables and migration names are gone, so a database is no longer readable by both implementations. diff --git a/docs/adr/0012-upload-urls-carry-a-per-upload-capability.md b/docs/adr/0012-upload-urls-carry-a-per-upload-capability.md deleted file mode 100644 index e0336f5..0000000 --- a/docs/adr/0012-upload-urls-carry-a-per-upload-capability.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -status: accepted ---- - -# Upload URLs carry a per-upload capability - -The upload id in `signed_upload_url` was the only thing a Part upload had to know, so anyone who could reach the server and guess or observe a ten digit id could write into another workflow's upload (#259, #299). `CreateCacheEntry` now appends 256 random bits as `?sig=` to the URL, like the SAS on an Azure Blob URL, and stores only its SHA-256 in `uploads.capabilityHash`. Single-shot, block and block list requests must present it; a mismatch, compared in constant time, is a 403. - -`@actions/cache` and BuildKit build their requests with the Azure SDK, which keeps a SAS query, so clients need no change. We rejected HMAC-signed, expiring URLs (#259) because they need a signing secret per deployment and a rotation story, while a random capability per upload needs no configuration and dies with its upload. - -The column is nullable so replicas still on the previous version can insert during a rollout. Their uploads, and uploads in flight during the upgrade, have no hash, so this version refuses their Parts: those saves fail once and the next run saves again. Until the rollout completes, previous-version replicas still accept Parts without the capability. diff --git a/docs/adr/0013-direct-downloads-require-verified-tokens.md b/docs/adr/0013-direct-downloads-require-verified-tokens.md deleted file mode 100644 index 64c4185..0000000 --- a/docs/adr/0013-direct-downloads-require-verified-tokens.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -status: accepted ---- - -# Direct downloads require verified tokens - -A direct-download URL is a bearer credential: whoever holds it reads the object from storage until it expires, without asking the server again. The runtime token's `repository_id` and `ac` claims are all that stand between a caller and that URL, and with `SKIP_TOKEN_VALIDATION=true` they are decoded but not verified, so any client that reaches the server could name any repository and walk away with a signed URL for its cache. The server therefore refuses to start with `ENABLE_DIRECT_DOWNLOADS=true` and `SKIP_TOKEN_VALIDATION=true` together, unless `UNSAFE_ALLOW_DIRECT_DOWNLOADS_WITHOUT_TOKEN_VALIDATION=true` overrides it for local testing. This extends ADR-0006: skipping verification stays a dev/test escape hatch, and the one feature that turns an unverified claim into a storage credential is out of its reach. Setups that ran both flags stop at startup after upgrading. - -A signed URL lives `DIRECT_DOWNLOAD_URL_TTL_SECONDS`, 600 by default (the fixed lifetime before the setting existed) and at most 3600. Clients request the URL right before fetching it, so the lifetime only has to cover the start of the transfer and retries; a shorter one limits replay of a URL that leaks into job logs, while clients that hold a URL and fetch it later, such as BuildKit's `type=gha` cache, need a longer one. The Storage Reader Lease outlives the URL by a fixed margin (ADR-0002). - -Verified tokens must carry `exp`; GitHub's always do, and a token without one would never expire. The optional `ACTIONS_TOKEN_AUDIENCE` lists accepted `aud` values and then requires one of them, restricting the server to tokens issued for one installation; a value that lists none (only commas) stops startup rather than silently disabling the check. Scope permissions are bit flags (read 1, write 2): lookups use only scopes with the read bit and saves only scopes with the write bit, and a scope without a name or with a malformed permission rejects the token. Rejections log the error kind and claim name, never the token or claim values. diff --git a/docs/agents/domain.md b/docs/agents/domain.md index ed8a370..cbcd884 100644 --- a/docs/agents/domain.md +++ b/docs/agents/domain.md @@ -6,9 +6,8 @@ How the engineering skills should consume this repo's domain documentation when - **`GLOSSARY.md`** at the repo root, or - **`GLOSSARY-MAP.md`** at the repo root if it exists β€” it points at one `GLOSSARY.md` per context. Read each one relevant to the topic. -- **`docs/adr/`** β€” read ADRs that touch the area you're about to work in. In multi-context repos, also check `src//docs/adr/` for context-scoped decisions. -If any of these files don't exist, **proceed silently**. Don't flag their absence; don't suggest creating them upfront. The `/domain-modeling` skill (reached via `/grill-with-docs` and `/improve-codebase-architecture`) creates them lazily when terms or decisions actually get resolved. +If any of these files don't exist, **proceed silently**. Don't flag their absence; don't suggest creating them upfront. The `/domain-modeling` skill (reached via `/grill-with-docs` and `/improve-codebase-architecture`) creates them lazily when terms actually get resolved. ## File structure @@ -17,9 +16,6 @@ Single-context repo (most repos): ``` / β”œβ”€β”€ GLOSSARY.md -β”œβ”€β”€ docs/adr/ -β”‚ β”œβ”€β”€ 0001-event-sourced-orders.md -β”‚ └── 0002-postgres-for-write-model.md └── src/ ``` @@ -28,14 +24,11 @@ Multi-context repo (presence of `GLOSSARY-MAP.md` at the root): ``` / β”œβ”€β”€ GLOSSARY-MAP.md -β”œβ”€β”€ docs/adr/ ← system-wide decisions └── src/ β”œβ”€β”€ ordering/ - β”‚ β”œβ”€β”€ GLOSSARY.md - β”‚ └── docs/adr/ ← context-specific decisions + β”‚ └── GLOSSARY.md └── billing/ - β”œβ”€β”€ GLOSSARY.md - └── docs/adr/ + └── GLOSSARY.md ``` ## Use the glossary's vocabulary @@ -43,9 +36,3 @@ Multi-context repo (presence of `GLOSSARY-MAP.md` at the root): When your output names a domain concept (in an issue title, a refactor proposal, a hypothesis, a test name), use the term as defined in `GLOSSARY.md`. Don't drift to synonyms the glossary explicitly avoids. If the concept you need isn't in the glossary yet, that's a signal β€” either you're inventing language the project doesn't use (reconsider) or there's a real gap (note it for `/domain-modeling`). - -## Flag ADR conflicts - -If your output contradicts an existing ADR, surface it explicitly rather than silently overriding: - -> _Contradicts ADR-0007 (event-sourced orders) β€” but worth reopening because…_ diff --git a/src/cleanup.rs b/src/cleanup.rs index 267ede8..3839641 100644 --- a/src/cleanup.rs +++ b/src/cleanup.rs @@ -434,7 +434,7 @@ fn folders( } /// Deletes stored folders that no database record authorizes once they are older than the grace -/// period (ADR-0001), as of `now`. +/// period, as of `now`. pub(crate) async fn reconcile_orphaned_storage( storage: &Storage, grace_period: Duration, diff --git a/src/config.rs b/src/config.rs index 384e70e..d0e3bcf 100644 --- a/src/config.rs +++ b/src/config.rs @@ -542,7 +542,7 @@ impl Config { .chain(args), )?; let mut given = Vars::from_arg_matches(&matches)?; - // A signed URL is a storage credential; unverified claims must not obtain one (ADR-0013). + // A signed URL is a storage credential; unverified claims must not obtain one. if given.enable_direct_downloads && given.skip_token_validation && !given.unsafe_allow_direct_downloads_without_token_validation diff --git a/src/db/entities.rs b/src/db/entities.rs index bb46901..79001fc 100644 --- a/src/db/entities.rs +++ b/src/db/entities.rs @@ -100,7 +100,7 @@ pub(crate) mod uploads { pub(crate) mod merge_leases { use sea_orm::entity::prelude::*; - /// Grants one worker the Merge of a storage location (ADR-0002). + /// Grants one worker the Merge of a storage location. #[sea_orm::model] #[derive(Debug, Clone, PartialEq, Eq, DeriveEntityModel)] #[sea_orm(table_name = "merge_leases", rename_all = "camelCase")] @@ -117,7 +117,7 @@ pub(crate) mod merge_leases { pub(crate) mod storage_reader_leases { use sea_orm::entity::prelude::*; - /// Protects a storage location from deletion while it is read (ADR-0002). + /// Protects a storage location from deletion while it is read. #[sea_orm::model] #[derive(Debug, Clone, PartialEq, Eq, DeriveEntityModel)] #[sea_orm(table_name = "storage_reader_leases", rename_all = "camelCase")] diff --git a/src/db/migrations/upload_capabilities.rs b/src/db/migrations/upload_capabilities.rs index 8558820..91c7930 100644 --- a/src/db/migrations/upload_capabilities.rs +++ b/src/db/migrations/upload_capabilities.rs @@ -1,4 +1,4 @@ -//! Stores the hash of each upload's capability, which its URL must carry (ADR-0012). +//! Stores the hash of each upload's capability, which its URL must carry. use sea_orm_migration::prelude::*; use sea_orm_migration::schema::char_len_null; diff --git a/src/metrics.rs b/src/metrics.rs index 981838e..054c06f 100644 --- a/src/metrics.rs +++ b/src/metrics.rs @@ -1,4 +1,4 @@ -//! Prometheus metrics (ADR-0007, ADR-0008). +//! Prometheus metrics. use prometheus::{IntCounter, IntCounterVec, IntGauge, Opts, Registry, TextEncoder}; diff --git a/src/scope.rs b/src/scope.rs index 2dae046..1080961 100644 --- a/src/scope.rs +++ b/src/scope.rs @@ -1,4 +1,4 @@ -//! Authorization from the GitHub Actions runtime token (ADR-0006). +//! Authorization from the GitHub Actions runtime token. use std::sync::Arc; use std::time::{Duration, Instant}; diff --git a/src/storage.rs b/src/storage.rs index 3575b2d..4f2650c 100644 --- a/src/storage.rs +++ b/src/storage.rs @@ -53,7 +53,7 @@ use crate::db::{ use crate::error::{Failure, Result}; /// Bounds the self-heal retry when matching keeps surfacing Dangling Cache Entries for the same -/// prefix; caps a pathological scan (ADR-0005). +/// prefix; caps a pathological scan. const MAX_DANGLING_PURGE_ATTEMPTS: u32 = 10; /// What a restore asks for: a primary key plus fallbacks, tried across scopes in order. @@ -389,7 +389,7 @@ impl Storage { /// True iff the entry's storage physically exists. A merged entry is confirmed by its merged /// object; an unmerged entry by its first Part. Parts deleted without a completed merge means - /// the data is gone. See ADR-0005. + /// the data is gone. async fn storage_has_data(&self, location: &StorageLocation) -> Result { if location.merged_at.is_some() { return self @@ -460,7 +460,7 @@ impl Storage { /// Runs a Merge under a Merge Lease in the background. `write` produces `{folder}/merged`; a /// lease-fenced transaction then marks the Merge complete. Returns false when another worker /// holds the lease or this process already runs `MAX_CONCURRENT_MERGES` Merges; a later - /// download retries a skipped Merge. Writing straight to the final object is safe (ADR-0004). + /// download retries a skipped Merge. Writing straight to the final object is safe. /// Failures are logged and the merge state rolled back. async fn start_merge( &self, @@ -560,7 +560,7 @@ impl Storage { } /// Stores one Part of an upload. False when the upload was deleted in the meantime (the - /// written Part is then Orphaned Storage, ADR-0001). + /// written Part is then Orphaned Storage). pub(crate) async fn upload_part( &self, upload: &AuthorizedUpload, @@ -687,7 +687,7 @@ impl Storage { tracing::warn!(%error, "Capacity-based Eviction failed after upload completion"); } if self.config.eager_merge && whole_object(&location).is_none() { - // Eager Merge (ADR-0009): only the lease acquisition is awaited; the Merge itself + // Eager Merge: only the lease acquisition is awaited; the Merge itself // runs in the background. if let Err(error) = self .start_merge(&location, self.merge_parts(&location)) @@ -747,7 +747,7 @@ impl Storage { } /// Capacity-based Eviction: once usage exceeds the Storage Budget, deletes entries by Cache - /// Recency until usage is back at 90% of the budget (ADR-0008). + /// Recency until usage is back at 90% of the budget. pub(crate) async fn enforce_storage_budget(&self) -> Result<()> { let filesystem_usage = match self.config.cache_max_size_bytes { Some(_) => None, @@ -1010,7 +1010,7 @@ impl Storage { /// Returning a download URL is a promise the data exists: the client commits to downloading /// and can't walk a later 404 back to a cache miss (`BuildKit` hard-fails the build). So /// storage is validated before handing out a URL: a Dangling Cache Entry is self-healed and - /// matching retried, so a valid candidate under another restore key still wins (ADR-0005). + /// matching retried, so a valid candidate under another restore key still wins. pub(crate) async fn cache_entry_with_download_url( &self, query: &CacheQuery, @@ -1053,7 +1053,7 @@ impl Storage { }; // An entry in one object with direct downloads enabled: take a reader lease that - // outlives the signed URL by a margin, then sign (ADR-0002). + // outlives the signed URL by a margin, then sign. let url_ttl = self.config.direct_download_url_ttl; let expires_at = deadline(url_ttl.saturating_add(DIRECT_DOWNLOAD_LEASE_MARGIN)); let tx = self.db.pool.begin().await?; @@ -1083,8 +1083,8 @@ impl Storage { Ok(None) } - /// Deletes a storage location row, then its data (ADR-0001), unless a reader holds it - /// (ADR-0002). Data that can't be deleted is left as Orphaned Storage. + /// Deletes a storage location row, then its data, unless a reader holds it. Data + /// that can't be deleted is left as Orphaned Storage. pub(crate) async fn delete_storage_location(&self, id: &str) -> Result { let location = storage_locations::Entity::find_by_id(id) .one(&self.db.pool) @@ -1133,7 +1133,7 @@ struct RankedEntry { match_rank: i64, } -/// Records Cache Access (ADR-0008). +/// Records Cache Access. async fn record_download(db: &impl ConnectionTrait, id: &str) -> Result<()> { storage_locations::Entity::update_many() .set(storage_locations::ActiveModel { diff --git a/src/storage/adapter.rs b/src/storage/adapter.rs index 9ca7f93..2f8891a 100644 --- a/src/storage/adapter.rs +++ b/src/storage/adapter.rs @@ -145,7 +145,7 @@ pub(crate) struct FilesystemUsage { /// /// Uploads must be atomically visible: an object never exists partially, and overwriting an /// object never disturbs active readers of the previous version. Object stores give this -/// natively; the filesystem adapter writes to a temp path and renames. See ADR-0004. +/// natively; the filesystem adapter writes to a temp path and renames. #[derive(Debug)] pub(crate) enum Adapter { Filesystem(FilesystemAdapter), diff --git a/src/storage/leases.rs b/src/storage/leases.rs index c4aee56..f6355e3 100644 --- a/src/storage/leases.rs +++ b/src/storage/leases.rs @@ -1,4 +1,4 @@ -//! Merge Leases and Storage Reader Leases (ADR-0002). +//! Merge Leases and Storage Reader Leases. use std::time::Duration; diff --git a/src/storage/lifecycle.rs b/src/storage/lifecycle.rs index 9675f8d..4779b2b 100644 --- a/src/storage/lifecycle.rs +++ b/src/storage/lifecycle.rs @@ -1,4 +1,4 @@ -//! Deletion guards shared by cleanup, eviction and self-healing (ADR-0001, ADR-0002). +//! Deletion guards shared by cleanup, eviction and self-healing. use sea_orm::ActiveValue::Set; use sea_orm::sea_query::{Expr, ExprTrait as _}; @@ -82,7 +82,7 @@ pub(crate) enum LocationDeletion { Deleted, /// Already gone, e.g. deleted concurrently. Missing, - /// A Storage Reader Lease protects it (ADR-0002). + /// A Storage Reader Lease protects it. InUse, }