[pinguinfuss]

Fixes #521.

These handlers only filtered metadata, so a lockfile URL or an artifact cached before the policy was raised got through. They now check before the cache hit and return 404, like npm and PyPI.

The policy key is the name-only PURL: conda without the channel, helm the chart name. Cargo, gem, hex and pub keep the publish time in the versions row, now written in UTC. Conda checks each file against its own repodata timestamp, because builds of one version differ. Helm reads the chart from the index. Both parse a cached document once, not on every download.

Open points:

1. Conda lets the download through when the uncompressed repodata is larger than metadata_max_size.
2. A cached helm chart needs the index whenever any cooldown rule exists, even when none is for helm.

[andrew]

The Conda size-limit gap listed in the description needs fixing before this closes the download bypass. A targeted test through the public routes confirmed that metadata hides a recent package while its cached artifact still returns HTTP 200 when decompressed repodata exceeds metadata_max_size.

[andrew]

ReadMetadata(decoded) applies metadata_max_size to the decompressed index. When it exceeds that limit, the error reaches versionInCooldown, which allows the artifact through. The existing repodata endpoint streams larger documents, so it can hide a recent package while this download endpoint serves that same cached package. I reproduced this with gzip below the configured limit and decompressed JSON above it: the metadata omitted the package, but its artifact returned HTTP 200 instead of 404. Stream the timestamp lookup using the existing repodata size bound and add an endpoint regression test covering this case.

[pinguinfuss]

Thanks, fixed in 1a9bda2.

1. The download check now streams the repodata through jsonstream, bounded by maxRepodataSize like filterRepodata. metadata_max_size no longer limits the decompressed index.
2. If the compressed copy is already over the limit and can't be cached, the check streams it from upstream, as handleRepodata does.
3. TestCondaDownloadCooldownPastMetadataLimit covers both cases through the public routes: repodata hides the recent package, its cached artifact returns 404 and the old one 200.

I also rebased onto main.

[andrew]

The metadata_max_size fix in 1a9bda2 does what I asked, thanks. Since the rebase, though, this branch disagrees with how main now handles cooldown lookup failures.

The download check fails open. versionInCooldown in internal/handler/download_cooldown.go returns false whenever lookup() errors, and withholdChart in helm.go does the same when fetchIndex or the index parse fails. So a rate-limited or failing metadata API lets a version inside cooldown be fetched from upstream and cached. Hex's 100 requests a minute makes that easy to hit. Since round one, main has changed this policy. 09e41d0 made gem, Hex and PyPI cooldown fail closed (502, or 503 when the rate-limit gate held the request). 224bce8 serves an already-cached file and fails only on a miss. Please return the lookup error from versionInCooldown, and have withholdInCooldown fall back to serveCachedDespiteCooldownError and then writeCooldownLookupError, as handleDownload in pypi.go does. That needs r and the cache filename passed in. Please also add a test per endpoint with a failing lookup: cached returns 200, uncached returns 502. While there, the Enabled() / For() <= 0 preamble can use the existing cooldownApplies.

With cache_metadata off, which is the default, res.digest is empty and publishIndex rebuilds the index on every call. Every conda download fetches and parses the subdir's whole repodata.json (tens of MB compressed on conda-forge), and every Helm chart download fetches index.yaml. For Helm that happens whenever any cooldown rule exists. Please keep a digest-less index for a short TTL, like the one-minute rewrite cache from f4cab2d, so an install doesn't download the index once per file.

#604 fetches the same gem and Hex documents through the metadata cache under different keys: "api/v1/versions/" + name + ".json" and hashedCacheKey(h.apiURL, "api/packages/"+name), against "api/v1/versions/" + name and "api/packages/"+name here. If both land as they are, each document is stored and revalidated twice. Whichever merges second should use the other's keys. If #604 goes first, rebasing onto it lets publishTime reuse its fetch.

Two comments can be shorter. The publishTime doc comment in conda.go only needs the line about streaming the compressed copy when it's over the limit. The second paragraph of the withholdInCooldown doc comment restates the code below it and can go.