[simonchrz]

Part of #414.

Mirroring pkg:julia/... failed with unsupported ecosystem. A version of a General package is now mirrored from the Pkg server:

• the PURL needs the uuid qualifier, as the purl spec requires for julia: pkg:julia/Example@0.5.5?uuid=7876af07-990d-54b4-ab0e-23690620f79a;
• the tree hash comes from General's Versions.toml, after checking that General's Package.toml lists that uuid for the name;
• the tarball is fetched from upstream.julia at /package/{uuid}/{tree-hash} and cached as pkg:julia/{Name}@{tree-hash}, file {tree-hash}.tar.gz, the key /julia/package/... uses.

The handler names a package from the Registry.toml it downloads from the upstream. Without the upstream, the name lookup fails and the handler used the uuid as the name, so a package cached under its name wasn't served. That happened offline after a restart, both for mirrored packages and for ones clients had fetched. When it can't name a package, the handler now looks the tarball up by its tree hash (CachedArtifactNames).

Refused, with a message saying why:

• a uuid that isn't General's for the name. Only General packages can be mirrored, since the registries client only reads General.

Not checked: the tarball. Its git tree hash is the hash of the unpacked tree, not of the file, so it isn't passed as a digest. The handler doesn't check it either. Pkg checks the tree hash after unpacking, except on Windows.

Not mirrored: dependencies, the binary artifacts from Artifacts.toml (/artifact/{hash}) and the registry. A mirrored package is served offline from /package/.... For an offline Pkg.add the client also needs the registry, either installed already or the snapshot the proxy cached when a client last fetched it. The proxy serves that snapshot offline only with cache_metadata on, because /registries goes through the metadata cache.

Custom upstream.julia: the tarball is fetched from it, and the uuid and tree hash still come from General, because they are the same on every Pkg server that serves General. The mirror used to refuse to list julia versions for a custom upstream; it now lists them from General.

Tested with a fake General registry and Pkg server (internal/server/mirror_julia_test.go). The test mirrors the package, serves it through the Julia handler with the registry names loaded, and then through a new handler with the upstream closed. That last step fails without the handler change. Against pkg.julialang.org:

• proxy mirror 'pkg:julia/Example@0.5.5?uuid=7876af07-…' downloaded 4049 bytes;
• proxy serve with HTTPS_PROXY pointing at a dead port served /julia/package/7876af07-…/e1f0e1a8… with 200, with the same SHA-256 as the direct download;
• in a julia:1.11 container with JULIA_PKG_SERVER pointing at the proxy, I first ran only Pkg.Registry.add("General") with the upstream reachable. Then, with the proxy offline, a fresh container ran Pkg.add(name="Example", version="0.5.5") and Example.hello("offline"). The package came from the mirrored tarball.

[andrew]

I merged main into the branch (2334a51) to resolve the conflicts with #606. newCommandMirror now calls UseMirrorUpstreams, newMirror sets both m.Charts and the upstreams, and the Helm namespace strip moved into fetchVersions after r.registry(...), since registry needs the repository namespace to resolve the URL.

CachedArtifactNames filters on artifacts.filename alone. That column has no index of its own, and the unique (version_purl, filename) index can't serve a filename-only filter. So every Julia package request made while the registry names can't be loaded scans the whole artifacts table, across all ecosystems. A Pkg.add with 50 dependencies during an upstream outage is 50 scans. Please add idx_artifacts_filename ON artifacts(filename) to the four schema blocks in internal/database/schema.go and as a 019_ migration modelled on 017_add_artifact_content_hash_index, with a test that the migration creates it.

Two comments can be shorter. The mirrorJulia doc comment in internal/mirror/julia.go only needs the reason the tarball is fetched unchecked, which is that Pkg checks the extracted tree hash itself. The last sentence of the JuliaPackageDownload comment in internal/handler/julia.go repeats what handlePackage does and can go.