[simonchrz]

Title: Mirror Swift registry releases

Part of #414.

Mirroring pkg:swift/... failed with unsupported ecosystem. A release is now mirrored from upstream.swift (Tuist by default):

• the source archive is cached under the key /swift/{scope}/{name}/{version}.zip serves, through the handler's own key and fetch code (swiftSourceArchiveKey, fetchSourceArchive);
• it is checked against the SHA-256 checksum in the release metadata, which the handler reads through the metadata cache and checks the same way. A mismatch fails the item and nothing is cached;
• the release metadata, every page of the release list and the release's Package.swift with the version-specific manifests its Link header lists go into the metadata cache.

The PURL namespace is the registry scope, pkg:swift/apple/swift-argument-parser@1.5.0 for apple.swift-argument-parser, as SwiftPM's own SBOMs write registry dependencies. A PURL naming the source repository, pkg:swift/github.com/apple/swift-argument-parser (syft, Trivy, GHSA), is looked up through the upstream's /identifiers?url=https://github.com/... and the first identity listed is mirrored. A repository the registry doesn't know fails with a message naming it. The lookup goes through the metadata cache like the /identifiers route. Unversioned PURLs list releases from upstream.swift with the registries client.

Release lists and manifests are stored with their sha256 digest and read back through storedMetadataEntry and openMetadataEntry, so a corrupted copy is dropped and fetched again, as for other cached metadata.

SwiftPM requests the release list, the release metadata, Package.swift and Package.swift?swift-version=5.8.0 before the archive. The handler passed the list and manifests through uncached, so a mirrored release still failed to resolve with the upstream down ("failed fetching releases list"). With cache_metadata the handler now caches both, along with the upstream Link header (release list pages, alternate manifests), and serves them while fresh or when the upstream fails. Manifests are keyed by tools version, so the 5.8 in the alternate link and the 5.8.0 SwiftPM sends share a key. Upstream errors on the manifest route now get the same problem response as the release list and metadata routes instead of being relayed, so that route is no longer in TestRelayRoutes.

Serving offline needs cache_metadata on the server, since the archive route reads the checksum from the release metadata. proxy mirror always turns it on; mirror API jobs follow the server's setting.

Tested in internal/server/mirror_swift_test.go against an httptest registry with a paginated release list and an alternate manifest. It mirrors, closes the upstream and requests the list, page 2, the metadata, both manifests and the archive through the Swift handler. It also covers a checksum mismatch, listing an unversioned PURL, and repository PURLs resolved through /identifiers (versioned, listed and unknown). The handler fallback and the refetch of a corrupted manifest are tested in swift_cache_test.go. The tests fail without the mirror hook, without the cached fallback, and without the tools-version key.

Against tuist.dev I mirrored pkg:swift/apple/swift-argument-parser@1.5.0, then ran serve with HTTPS_PROXY pointing at a dead port and metadata_ttl: 0. curl got the list, metadata, both manifests and the archive (sha256 24bc0025…453c, as listed), and swift package resolve in a swift:6.0 container resolved and fetched 1.5.0 through the proxy. Before the handler change it failed at the release list. --dry-run on the unversioned PURL lists all 46 releases. pkg:swift/github.com/apple/swift-argument-parser@1.5.0 resolved to apple.swift-argument-parser through tuist.dev's /identifiers and mirrored, and an unknown repository failed with "the Swift registry has no package for https://github.com/nobody/nosuchrepo-xyz".

[andrew]

This needs a merge of main. Five files conflict with the Helm mirror, all as adjacent additions.

storeDocument in swift_cache.go drops the hash Storage.Store returns, so Swift rows have no ContentDigest, and loadDocument reads the blob back unchecked. Every other metadata path records sha256:<hash> (cacheMetadataBlob), and since 6c9505f a stored copy that fails its digest is dropped and refetched. Release lists and manifests would be the one kind of cached metadata exempt from that. Please record the digest, and load through storedMetadataEntry(ecosystem, key, true) and openMetadataEntry from #604. They check the digest as the blob is read, and passing true skips the TTL, so the stale fallback keeps working while fetchDocument keeps its own TTL check. Please also add a test that a corrupted stored manifest is refetched.

The comment block at the top of swift_cache.go isn't attached to anything. Please move it onto fetchDocument or drop it. The MirrorRelease doc comment can be cut to its last sentence, about why the list and manifests are cached.

[andrew]

Correction: SwiftPM 6.4's own SBOM generation (SE-0509, swift package generate-sbom) writes registry dependencies as pkg:swift/{scope}/{name}, for example pkg:swift/mona/LinkedList@3.1.4 (Sources/SBOMModel/Extractor/PURL.swift and Tests/SBOMModelTests/PURLTests.swift in swiftlang/swift-package-manager), so that form should stay as mirror input. The change I'd ask for instead is to also accept source PURLs such as pkg:swift/github.com/apple/swift-argument-parser, which syft, Trivy and GHSA advisories use, by looking up the registry identity through the upstream's /identifiers?url= rather than refusing them. #613 covers the same for retention and denylist config.

Using pkg:swift/{scope}/{name} for a registry identity conflicts with the purl-spec swift type, where the namespace is the source host and owner (pkg:swift/github.com/apple/swift-argument-parser). Registry identities have no spec form yet (package-url/purl-spec#929), and git-pkgs/purl#29 made MakePURL return nil for them because pkg:swift/apple/swift-argument-parser reads as a source namespace. git-pkgs/registries#111 leaves Swift PURLs empty for the same reason.

This PR would make the proxy accept that form as mirror input and refuse the spec-shaped PURLs SBOMs contain. If purl-spec#929 settles on something else, mirror input would have to change along with the existing places that already read it this way: the storage retention keys in internal/handler/swift.go, the denylist mapping in internal/denylist/denylist.go, and the swift rule in git-pkgs/purl's ComparisonForm.

The proxy already stores these archives as pkg:generic/swift-registry/{scope}.{name}. Mirroring could take that form as input instead, so proxy mirror pkg:generic/swift-registry/apple.swift-argument-parser@1.5.0, and keep pkg:swift for source coordinates once there's a way to resolve them through /identifiers?url=.