Repository navigation
Mirror PyPI packages - #591
Merged
Merged
Conversation
proxy mirror and mirror jobs failed for pypi with "unsupported ecosystem", since the resolver they use has no registry for it. A PyPI version has several files, so the mirror now lists them from upstream.pypi's JSON API, through the same client as version listing, and fetches each one, verified against its sha256. The file_name qualifier limits it to one file. Files are cached under the name and version the download handler reads from the filename, which can differ from the PURL (zope.interface's wheels are zope_interface), so pip is served them from the cache.
This was referenced Oct 25, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #414: PyPI.
proxy mirror, mirror API jobs and schedules failed for every pypi PURL withunsupported ecosystem: pypi, since the resolver has no registry for it.A PyPI version has several files, so the mirror doesn't go through the resolver here. It lists the version's files from
upstream.pypi's JSON API, through the same client and lister #548 added for versions, and fetches each one withGetOrFetchArtifactFromURLWithIntegrity, so it is checked against the listed sha256. Thefile_namequalifier from the PURL spec limits it to one file.Each file is cached under the name and version the download handler reads from its filename (
handler.PyPIFilenameIdentity, whichparseFilenamenow wraps), not under the PURL's. They can differ: a wheel spelledexample_pkgand an sdist spelledexample.pkgend up under different keys, and the handler looks them up by filename.Tests, following the review on #414:
TestMirrorPyPIFetchesEveryFilemirrors a version with a wheel and an sdist whose names are spelled differently, then shuts the upstream down and fetches both through the PyPI handler from the cache.TestMirrorPyPIFileNameQualifiermirrors only the named file, and fails with the name when it isn't listed.TestMirrorPyPIRefusesDigestMismatchrefuses a file whose sha256 doesn't match.Against pypi.org,
proxy mirror pkg:pypi/six@1.16.0fetches the wheel and the sdist, andpkg:pypi/zope.interface@7.2?file_name=zope.interface-7.2-cp312-cp312-macosx_11_0_arm64.whlfetches only that wheel.Files are fetched from the URLs the JSON API lists. With the default upstreams those are on
files.pythonhosted.org, whichupstream.pypi_downloadpoints at too.