[simonchrz]

Part of #414.

Mirroring pkg:cran/... failed, since the resolver has no CRAN download. A version is now mirrored as its source package from upstream.cran:

• cached under the key handleSourceDownload uses, ("cran", name, version, name_version.tar.gz). The handler takes name and version from the file name and ignores the directory, so the same entry serves /src/contrib/{file} and /src/contrib/Archive/{name}/{file}. A mirrored current version is still served once CRAN moves it to Archive/.
• the current version is fetched from src/contrib and checked against the MD5sum in src/contrib/PACKAGES.gz. The index is about 2 MB, so it's read at most once every five minutes per mirror, not once per package.
• other versions are fetched unchecked from src/contrib/Archive/{name}/, the same way the handler fetches them.

MD5 goes through GetOrFetchArtifactFromURLWithIntegrity as md5-<hex>, so it's recorded with the version like Composer's sha1.

A custom upstream.cran works, as long as it's laid out like CRAN: the index, archive and tarballs all come from the upstream the handler uses. Unversioned PURLs mirror every version, as for PyPI and conda. Until the registries fix for git-pkgs/registries#119 is in, the CRAN client lists each archived version twice, so the second is a cache hit.

Not mirrored: binary packages (/bin/{platform}/contrib/{rversion}/...). They're cached per platform and R version, and choosing those would need qualifiers. The PACKAGES indexes aren't mirrored either, as for the other ecosystems, so install.packages("pkg", repos = proxy) still needs the upstream or an index the proxy has already cached. Installing a tarball by URL works without either.

Tested through the mirror and the CRAN handler with the upstream closed (internal/server/mirror_cran_test.go): a current version and an archived one, the current one also requested through Archive/; an MD5 mismatch on the current version refused; and an unversioned PURL.

Against cloud.r-project.org I mirrored pkg:cran/R6@2.6.1 (current) and pkg:cran/R6@2.5.1 (archived). I then ran serve with HTTPS_PROXY/HTTP_PROXY pointed at a dead port. curl got R6_2.6.1.tar.gz from both src/contrib/ and src/contrib/Archive/R6/, and R6_2.5.1.tar.gz from Archive/, all 200, while a package that hadn't been mirrored returned 502. In rocker/r-ver:4.4.1, install.packages(<proxy URL>, repos = NULL, type = "source") installed both versions through the proxy.

[andrew]

This needs a merge of main: internal/mirror/mirror.go and docs/configuration.md conflict with the Helm and Julia mirrors, but only as adjacent additions.

The dedup in ListVersions works around a bug in git-pkgs/registries. parseArchiveVersions matches both the href and the link text of each file in the archive listing, so every other consumer of FetchVersions("cran", ...) gets duplicates too. I opened git-pkgs/registries#119 for it. Please drop the dedup and its comment here. The proxy will pick up the fix with a registries bump.

Please also drop cran_rds.go, its fixture and fuzz test, and fetch archived versions unchecked, the way the CRAN handler fetches them. PACKAGES.rds comes from the same host and directory as the tarball. The MD5 check catches corruption but not tampering, and HTTPS already covers transport. That isn't enough to justify keeping a hand-written decoder for R serialization read from the network. The decoder itself is careful, but the current-version check against PACKAGES.gz is the part worth keeping.

The mirrorCRAN doc comment can be cut to the cache key and the fact that one entry serves both src/contrib and Archive/. The rest repeats the code.