Mirror Swift registry releases

docs/architecture.md

@@ -304,6 +304,7 @@ HTTP protocol handlers for each registry type.
 **SwiftHandler:**
 - Proxies the Swift Package Registry v1 read endpoints
 - Rewrites release URLs and caches source archives
+- With `cache_metadata`, caches release metadata, release lists and manifests (with their `Link` header) and serves them when the upstream fails
 
 **GitHandler:**
 - Serves `/git/{name}/{repo}` from `internal/gitcache`, read-only over smart HTTP

docs/configuration.md

@@ -1330,6 +1330,8 @@ Some metadata never changes once published, and is cached like an artifact, with
 
 Composer archive downloads look up their upstream URL in the same cached metadata, so an archive is resolved without asking Packagist again and stays installable while it is down. Only a version or file the cached metadata does not list yet is looked up upstream. With or without `cache_metadata`, a cached archive of a tagged release from the default repository is served without any metadata lookup; with cooldown enabled it is checked against the publish time recorded when the archive was first resolved. Branch (dev) versions, routed packages and archives picked by a query string such as `?sha=` are always resolved from the metadata.
 
+Swift release lists and manifests are cached with the upstream's `Link` header, which carries the release list's pages and the version-specific manifests a client chooses from.
+
 The `proxy mirror` command always enables metadata caching regardless of this setting.
 
 ### Metadata TTL
@@ -1571,6 +1573,8 @@ A Helm chart is mirrored from a repository in `upstream.helm`, named as the name
 
 A Julia package is mirrored with the source tarball `upstream.julia` serves at `/package/{uuid}/{tree-hash}`, cached under the package's name and tree hash as `/julia/package/...` serves it. The PURL needs the `uuid` qualifier, for example `pkg:julia/Example@0.5.5?uuid=7876af07-990d-54b4-ab0e-23690620f79a`. Only packages of the General registry can be mirrored: the uuid and tree hash are read from General on GitHub, also with another `upstream.julia`, since they are the same on every Pkg server serving General. The tarball isn't checked by the proxy, as when serving; Pkg checks the tree hash of what it unpacks, except on Windows. Dependencies, the artifacts in a package's `Artifacts.toml` and the registry aren't mirrored. An offline `Pkg.add` also needs the registry: one installed on the client, or the one the proxy cached when a client last fetched it, which it serves offline with `cache_metadata` on.
 
+A Swift release is mirrored from `upstream.swift` with its source archive, checked against the SHA-256 checksum in the release metadata and cached under the key `/swift/{scope}/{name}/{version}.zip` serves. Give the registry scope as the namespace, for example `pkg:swift/apple/swift-argument-parser@1.5.0` for `apple.swift-argument-parser`, or the source repository, as SBOM tools write it: `pkg:swift/github.com/apple/swift-argument-parser@1.5.0` is looked up through the upstream's `/identifiers`, and the first identity it lists is mirrored. The release metadata, the package's release list and the release's `Package.swift` with its version-specific manifests are stored in the metadata cache too, so `swift package resolve` works with the upstream unreachable. Serving them, and the archive, offline needs `cache_metadata` on the server, since the archive is served with the checksum from the release metadata.
+
 | Flag | Default | Description |
 |------|---------|-------------|
 | `--sbom` | | Path to CycloneDX or SPDX SBOM file |
@@ -1591,7 +1595,7 @@ proxy mirror pkg:npm/lodash@4.17.21 pkg:cargo/serde@1.0.0
 
 ### Unversioned PURLs
 
-A PURL without a version, here or in a mirror API job, mirrors every version the configured upstream lists, through the same client, so `upstream.auth` and `upstream.allow_private_hosts` apply. npm packages in `upstream.npm_routes` are listed from their route. Listing works for npm, gem, golang, hex (from `upstream.hex_api`), pub, pypi, maven, nuget, composer, cran and brew. For cargo and conda the upstream doesn't serve the API used for listing, so versions come from the public registry while the upstream is the default one; with another upstream, and for Composer packages in `upstream.composer_routes` or any other ecosystem, give a version. Julia versions are always listed from the General registry.
+A PURL without a version, here or in a mirror API job, mirrors every version the configured upstream lists, through the same client, so `upstream.auth` and `upstream.allow_private_hosts` apply. npm packages in `upstream.npm_routes` are listed from their route. Listing works for npm, gem, golang, hex (from `upstream.hex_api`), pub, pypi, maven, nuget, composer, cran, brew and swift. For cargo and conda the upstream doesn't serve the API used for listing, so versions come from the public registry while the upstream is the default one; with another upstream, and for Composer packages in `upstream.composer_routes` or any other ecosystem, give a version. Julia versions are always listed from the General registry.
 
 ### Container images
 

internal/handler/relay_test.go

@@ -53,7 +53,6 @@ func TestRelayRoutes(t *testing.T) {
 		"/conan/v2/conans/demo/1.0/user/stable/revisions/rev/files/other.txt",
 		"/composer/search.json", "/pypi/simple/", "/gem/api/v1/dependencies",
 		"/conda/conda-forge/noarch/repodata.json",
-		"/swift/scope/demo/1.0.0/Package.swift",
 		"/v2/library/demo/manifests/latest", "/v2/library/demo/tags/list",
 		"/npm/-/npm/v1/keys",
 	} {

internal/handler/swift.go

@@ -104,23 +104,23 @@ func (h *SwiftHandler) handlePackageReleases(w http.ResponseWriter, r *http.Requ
 	scope, name = canonicalSwiftPackage(scope, name)
 
 	upstreamURL := h.buildUpstreamURL(scope, name, "", "", r.URL.RawQuery)
-	body, contentType, responseHeaders, err := h.fetchMetadataWithHeaders(
-		r.Context(), upstreamURL, requestAccept(r, swiftAcceptJSON),
+	doc, err := h.fetchDocument(
+		r.Context(), swiftReleasesCacheKey(scope, name, r.URL.RawQuery), upstreamURL, requestAccept(r, swiftAcceptJSON),
 	)
 	if err != nil {
 		h.writeMetadataError(w, err)
 		return
 	}
 
-	rewritten, err := h.rewriteReleaseURLs(scope, name, body)
+	rewritten, err := h.rewriteReleaseURLs(scope, name, doc.body)
 	if err != nil {
 		h.proxy.Logger.Warn("failed to rewrite Swift release URLs", "error", err)
-		rewritten = body
+		rewritten = doc.body
 	}
-	for _, link := range responseHeaders.Values("Link") {
-		w.Header().Add("Link", h.rewriteLinkHeader(link, upstreamURL))
+	if doc.link != "" {
+		w.Header().Set("Link", h.rewriteLinkHeader(doc.link, upstreamURL))
 	}
-	writeSwiftMetadata(w, r, rewritten, contentType)
+	writeSwiftMetadata(w, r, rewritten, doc.contentType)
 }
 
 func (h *SwiftHandler) handleRelease(w http.ResponseWriter, r *http.Request) {
@@ -161,7 +161,17 @@ func (h *SwiftHandler) handleManifest(w http.ResponseWriter, r *http.Request) {
 	scope, name = canonicalSwiftPackage(scope, name)
 
 	upstreamURL := h.buildUpstreamURL(scope, name, version, "Package.swift", r.URL.RawQuery)
-	h.proxySwiftResource(w, r, upstreamURL, swiftAcceptManifest)
+	doc, err := h.fetchDocument(
+		r.Context(), swiftManifestCacheKey(scope, name, version, r.URL.RawQuery), upstreamURL, requestAccept(r, swiftAcceptManifest),
+	)
+	if err != nil {
+		h.writeMetadataError(w, err)
+		return
+	}
+	if doc.link != "" {
+		w.Header().Set("Link", h.rewriteLinkHeader(doc.link, upstreamURL))
+	}
+	writeSwiftMetadata(w, r, doc.body, doc.contentType)
 }
 
 func (h *SwiftHandler) handleIdentifiers(w http.ResponseWriter, r *http.Request) {
@@ -194,14 +204,13 @@ func (h *SwiftHandler) handleSourceArchive(w http.ResponseWriter, r *http.Reques
 	}
 	scope, name = canonicalSwiftPackage(scope, name)
 
-	packageName := scope + "/" + name
-	filename := fmt.Sprintf("%s-%s.zip", name, version)
-	upstreamURL := h.buildUpstreamURL(scope, name, version+".zip", "", r.URL.RawQuery)
-	packagePURL, versionPURL := packageurl.MakeCacheStrings("swift", packageName, version)
-	if packagePURL == "" || versionPURL == "" {
-		h.writeArtifactError(w, fmt.Errorf("%w: swift %q", errUnsupportedPackageIdentity, packageName))
+	key, err := swiftSourceArchiveKey(scope, name, version)
+	if err != nil {
+		h.writeArtifactError(w, err)
 		return
 	}
+	filename, packagePURL, versionPURL := key.filename, key.packagePURL, key.versionPURL
+	upstreamURL := h.buildUpstreamURL(scope, name, version+".zip", "", r.URL.RawQuery)
 	archiveInfo, infoErr := h.fetchArchiveInfo(r.Context(), scope, name, version)
 	if infoErr != nil {
 		h.writeArtifactError(w, fmt.Errorf("fetching release metadata: %w", infoErr))
@@ -215,10 +224,7 @@ func (h *SwiftHandler) handleSourceArchive(w http.ResponseWriter, r *http.Reques
 
 	headers := make(http.Header)
 	headers.Set("Accept", requestAccept(r, swiftAcceptArchive))
-	result, err := h.proxy.getOrFetchArtifactFromURLWithCachePURLs(
-		r.Context(), "swift", packageName, version, filename, packagePURL, versionPURL,
-		upstreamURL, headers, archiveInfo.checksum, "",
-	)
+	result, err := h.fetchSourceArchive(r.Context(), key, version, upstreamURL, headers, archiveInfo)
 	if err != nil {
 		h.writeArtifactError(w, err)
 		return
@@ -229,6 +235,32 @@ func (h *SwiftHandler) handleSourceArchive(w http.ResponseWriter, r *http.Reques
 	ServeArtifactRequest(w, r, result)
 }
 
+// swiftArchiveKey is the cache key a release's source archive is stored
+// under.
+type swiftArchiveKey struct {
+	packageName, filename, packagePURL, versionPURL string
+}
+
+// swiftSourceArchiveKey returns the cache key of the source archive of the
+// canonical (lowercased) scope and name at version.
+func swiftSourceArchiveKey(scope, name, version string) (swiftArchiveKey, error) {
+	key := swiftArchiveKey{packageName: scope + "/" + name, filename: fmt.Sprintf("%s-%s.zip", name, version)}
+	key.packagePURL, key.versionPURL = packageurl.MakeCacheStrings("swift", key.packageName, version)
+	if key.packagePURL == "" || key.versionPURL == "" {
+		return swiftArchiveKey{}, fmt.Errorf("%w: swift %q", errUnsupportedPackageIdentity, key.packageName)
+	}
+	return key, nil
+}
+
+// fetchSourceArchive returns the source archive under key from the cache, or
+// fetches it from upstreamURL, verified against the checksum in info.
+func (h *SwiftHandler) fetchSourceArchive(ctx context.Context, key swiftArchiveKey, version, upstreamURL string, headers http.Header, info swiftArchiveInfo) (*CacheResult, error) {
+	return h.proxy.getOrFetchArtifactFromURLWithCachePURLs(
+		ctx, "swift", key.packageName, version, key.filename, key.packagePURL, key.versionPURL,
+		upstreamURL, headers, info.checksum, "",
+	)
+}
+
 func (h *SwiftHandler) handleSourceArchiveHead(
 	w http.ResponseWriter,
 	r *http.Request,
@@ -424,56 +456,6 @@ func swiftDigestHeader(checksum string) string {
 	return "sha-256=" + base64.StdEncoding.EncodeToString(digest)
 }
 
-func (h *SwiftHandler) proxySwiftResource(w http.ResponseWriter, r *http.Request, upstreamURL, defaultAccept string) {
-	req, err := http.NewRequestWithContext(r.Context(), r.Method, upstreamURL, nil)
-	if err != nil {
-		writeSwiftProblem(w, http.StatusInternalServerError, "failed to create upstream request")
-		return
-	}
-	req.Header.Set("Accept", requestAccept(r, defaultAccept))
-	for _, name := range []string{"If-Modified-Since", "If-None-Match"} {
-		if value := r.Header.Get(name); value != "" {
-			req.Header.Set(name, value)
-		}
-	}
-
-	resp, err := h.proxy.HTTPClient.Do(req)
-	if err != nil {
-		if writeUpstreamHeld(w, err) {
-			return
-		}
-		writeSwiftProblem(w, http.StatusBadGateway, "upstream request failed")
-		return
-	}
-	defer func() { _ = resp.Body.Close() }()
-
-	h.proxy.relayResponse(w, r, resp, func(dst, src http.Header) {
-		copySwiftResponseHeaders(dst, src)
-		if location := src.Get("Location"); location != "" {
-			dst.Set("Location", h.rewriteRegistryURL(location, upstreamURL))
-		}
-		for _, link := range src.Values("Link") {
-			dst.Add("Link", h.rewriteLinkHeader(link, upstreamURL))
-		}
-		if dst.Get("Content-Version") == "" {
-			dst.Set("Content-Version", swiftContentVersion)
-		}
-	})
-}
-
-func copySwiftResponseHeaders(dst, src http.Header) {
-	for _, name := range []string{
-		"Cache-Control", "Content-Disposition", "Content-Language", headerContentLength,
-		headerContentType, "Content-Version", "Digest", headerETag, headerLastModified,
-		"Retry-After", "Vary", "Warning", "X-Swift-Package-Signature",
-		"X-Swift-Package-Signature-Format",
-	} {
-		for _, value := range src.Values(name) {
-			dst.Add(name, value)
-		}
-	}
-}
-
 func (h *SwiftHandler) rewriteLinkHeader(value, upstreamRequestURL string) string {
 	var result strings.Builder
 	for len(value) > 0 {

internal/handler/swift_cache.go

@@ -0,0 +1,128 @@
+package handler
+
+import (
+	"bytes"
+	"context"
+	"database/sql"
+	"errors"
+	"net/url"
+	"strings"
+	"time"
+
+	"github.com/git-pkgs/proxy/internal/database"
+)
+
+// swiftDocument is a release list or manifest as the upstream sent it.
+type swiftDocument struct {
+	body        []byte
+	contentType string
+	// link is the upstream Link header, its values joined, with the
+	// upstream's URLs.
+	link      string
+	fetchedAt time.Time
+}
+
+// fetchDocument returns the document at upstreamURL, from the metadata cache
+// under cacheKey while it is fresh or when the upstream fails, and otherwise
+// from upstream, storing it. The upstream Link header is cached with it,
+// since it carries the pagination of release lists and the version-specific
+// manifests a client picks from.
+func (h *SwiftHandler) fetchDocument(ctx context.Context, cacheKey, upstreamURL, accept string) (*swiftDocument, error) {
+	cached := h.loadDocument(ctx, cacheKey)
+	if cached != nil && h.proxy.MetadataTTL > 0 && time.Since(cached.fetchedAt) < h.proxy.MetadataTTL {
+		return cached, nil
+	}
+	body, contentType, headers, err := h.fetchMetadataWithHeaders(ctx, upstreamURL, accept)
+	if err != nil {
+		if cached == nil || errors.Is(err, ErrUpstreamNotFound) {
+			return nil, err
+		}
+		h.proxy.Logger.Warn("upstream Swift metadata fetch failed, serving cached copy",
+			"url", upstreamURL, "error", err)
+		return cached, nil
+	}
+	doc := &swiftDocument{
+		body:        body,
+		contentType: contentType,
+		link:        strings.Join(headers.Values("Link"), ", "),
+		fetchedAt:   time.Now(),
+	}
+	h.storeDocument(ctx, cacheKey, doc)
+	return doc, nil
+}
+
+func (h *SwiftHandler) documentsCached() bool {
+	return h.proxy.CacheMetadata && h.proxy.DB != nil && h.proxy.Storage != nil
+}
+
+// loadDocument returns the cached document under cacheKey, or nil.
+func (h *SwiftHandler) loadDocument(ctx context.Context, cacheKey string) *swiftDocument {
+	if !h.documentsCached() {
+		return nil
+	}
+	entry := h.proxy.storedMetadataEntry("swift", cacheKey, true)
+	if entry == nil {
+		return nil
+	}
+	reader, err := h.proxy.openMetadataEntry(ctx, "swift", cacheKey, entry)
+	if err != nil {
+		return nil
+	}
+	defer func() { _ = reader.Close() }()
+	body, err := h.proxy.ReadMetadata(reader)
+	if err != nil {
+		return nil
+	}
+	h.proxy.recordMetadataRead(entry)
+	return &swiftDocument{
+		body:        body,
+		contentType: entry.ContentType.String,
+		link:        entry.Link.String,
+		fetchedAt:   entry.FetchedAt.Time,
+	}
+}
+
+// storeDocument caches doc under cacheKey. A failure is logged; the
+// document is still served.
+func (h *SwiftHandler) storeDocument(ctx context.Context, cacheKey string, doc *swiftDocument) {
+	if !h.documentsCached() {
+		return
+	}
+	storagePath := metadataStoragePath("swift", cacheKey)
+	size, hash, err := h.proxy.Storage.Store(ctx, storagePath, bytes.NewReader(doc.body))
+	if err == nil {
+		err = h.proxy.DB.UpsertMetadataCache(&database.MetadataCacheEntry{
+			Ecosystem:     "swift",
+			Name:          cacheKey,
+			StoragePath:   storagePath,
+			Link:          sql.NullString{String: doc.link, Valid: doc.link != ""},
+			ContentType:   sql.NullString{String: doc.contentType, Valid: doc.contentType != ""},
+			ContentDigest: sql.NullString{String: "sha256:" + hash, Valid: hash != ""},
+			Size:          sql.NullInt64{Int64: size, Valid: true},
+			FetchedAt:     sql.NullTime{Time: doc.fetchedAt, Valid: true},
+		})
+	}
+	if err != nil {
+		h.proxy.Logger.Warn("failed to cache Swift metadata", "key", cacheKey, "error", err)
+	}
+}
+
+func swiftReleasesCacheKey(scope, name, rawQuery string) string {
+	return swiftMetadataCacheKey("releases", scope, name, rawQuery)
+}
+
+// swiftManifestCacheKey keys a manifest by its swift-version as a tools
+// version, so the 5.8 of a manifest's alternate link and the 5.8.0 a client
+// requests for it share a key.
+func swiftManifestCacheKey(scope, name, version, rawQuery string) string {
+	if query, err := url.ParseQuery(rawQuery); err == nil {
+		if tools := query.Get("swift-version"); tools != "" {
+			for strings.Count(tools, ".") > 0 && strings.HasSuffix(tools, ".0") {
+				tools = strings.TrimSuffix(tools, ".0")
+			}
+			query.Set("swift-version", tools)
+		}
+		rawQuery = query.Encode()
+	}
+	return swiftMetadataCacheKey("manifest", scope, name, version, rawQuery)
+}