Add a Debian repository client #97

Activity

1. andrew

   added

   enhancementNew feature or requestNew feature or request

   on Oct 18, 2026
2. andrew

   changed the title [-]Add Debian and RPM repository clients[/-] [+]Add a Debian repository client[/+] on Oct 18, 2026
3. simonchrz commented on Oct 25, 2026

   

   simonchrz

   on Oct 25, 2026

   Contributor

   More actions

   I'd like to pick this up for proxy mirror (git-pkgs/proxy#414). I looked at how aptly, cybozu-go/aptutil, canonical/chisel, pault.ag/go/debian, Renovate's deb datasource, syft and trivy handle it, and checked the archives. The plan, then some questions first:

   Index.

   • Read dists/<suite>/InRelease, falling back to Release (35–256 KB). It lists every index variant's SHA256 and size.
   • Then try Packages.xz, .gz and plain Packages, moving on after a 404, as aptly does. All three are needed:
     • security.debian.org serves only .xz for bookworm-security;
     • Docker, Microsoft and PGDG serve no .xz;
     • deb.debian.org lists plain Packages in Release but answers 404 for it.
   • Hash the downloaded file and check it against its Release entry, as chisel and aptutil do. That catches corruption and a mirror caught mid-sync. Without a Release (Artifactory without GPG), skip the check.

   Streaming.

   • Decompressed indexes reach 56 MB (trixie main amd64) to 73 MB (noble universe), and sid's .xz alone is 10.8 MB.
   • So I'd add a raw Stream to client.Client next to StreamJSON, with the same retries, plus a limit on the decompressed bytes.
   • Parse stanzas from the stream, keeping only the fields needed:
     • the line buffer is 1 MiB, since bookworm main has a 75,649-byte Provides: line;
     • field names are matched case-insensitively, as the Debian wiki asks of readers;
     • continuation lines are handled.
   • Build one map[name][]record per instance, as the APK client does, and commit it only after the hash check.

   xz. github.com/ulikunitz/xz (BSD-3, pure Go), as aptutil, trivy, grype and chisel use. Its DictCap doesn't bound memory (ulikunitz/xz#84), so I'd check the dictionary size in the stream header before decoding; Debian uses 8 MiB. The alternative is xi2/xz, which has a hard dictMax but hasn't changed since 2017.

   Versions.

   • Compare with the deb scheme in vers.
   • Version carries the epoch (1:9.18.49-1~deb12u2), and Filename never does (about 6% of bookworm main has one).
   • Match the full version first. If that fails, try a trivy-style epoch= qualifier. If that fails too, accept a version without its epoch when exactly one indexed version matches.
   • Architecture: all stanzas appear in every binary-<arch> index. Record the stanza's own architecture on the artifact.

   PURLs. pkg:deb/debian/curl@7.88.1-10+deb12u15?arch=amd64&distro=bookworm. distro varies in the wild:

   Producer	Example
   PURL spec	bookworm
   syft	debian-12
   trivy	ubuntu-22.04

   Questions:

   1. Should the Release hash check be on by default? I'd leave InRelease GPG verification for later, since it needs per-vendor keyrings.
   2. Should one configured repository cover several suites and components, such as bookworm, bookworm-updates and bookworm-security, with the results merged? Point releases replace versions in bookworm itself, and fixes live in the other suites.
   3. Should distro be mapped with a built-in table (debian-12 → bookworm, ubuntu-22.04 → jammy) or only through configuration?
   4. arch=all: use binary-all where the Release lists it (Debian does, Ubuntu doesn't), otherwise a configured architecture? And should arch=source fail with "unsupported"?
   5. Should versions that have left every index fall back to snapshot.debian.org, or is that out of scope? SBOM-pinned versions often have.

   The proxy's Debian handler takes its version from name_version_arch.deb, without the epoch. So proxy#414 also has to settle which form goes into the cache key.
4. andrew commented on Oct 26, 2026

   

   andrew

   on Oct 26, 2026

   ContributorAuthor

   More actions

   Thanks, this plan looks right. Answers in order:

   1. Yes, check the Release hash by default whenever a Release or InRelease is present, and skip it when there isn't one. GPG verification can go in its own issue.
   2. Yes, one repository config should take a list of suites and components and merge the results. Without -security and -updates, most SBOM-pinned versions won't resolve.
   3. Ship a small built-in table for Debian and Ubuntu codenames (debian-12 → bookworm, ubuntu-22.04 → jammy) and let configuration override or extend it.
   4. Use binary-all where the Release lists it, otherwise the configured architecture. arch=source should fail as unsupported.
   5. snapshot.debian.org is out of scope here. Open a separate issue for it once this lands.

   The raw Stream on client.Client next to StreamJSON is fine. The per-instance map and commit-after-validation pattern you mention comes from the APK client in #107, which isn't merged yet, so either base on that branch or wait for it to land. #29 and #95 have closed since this issue was written: oversized buffered bodies now fail with ErrResponseTooLarge, and the resolver already selects on arch and distro, so the client just needs to set those on each artifact.