Skip to content

Build every image on one shared base image (#223 C11-5) - #250

Merged
maximusunc merged 2 commits into
develfrom
audit/223-base-image
Nov 11, 2026
Merged

maximusunc merged 2 commits into
develfrom
audit/223-base-image

Conversation

@maximusunc

Copy link
Copy Markdown
Collaborator

Part 4 of 4 for #223 (C11-5). The other parts are #247 (poll loop), #248 (lookup helper) and #249 (pool budget). Fixes #223 once those three are merged too.

Problem

  • Each of the 28 Shepherd Dockerfiles copied shepherd_utils before pip install ., so any code change reinstalled every dependency in every image.
  • SHEPHERD_VERSION was declared at the top, so each release tag also invalidated every layer.
  • None of the images used --no-cache-dir.
  • chmod -R 777 . made /app world-writable, and in score_paths it ran after the code was copied, which left the code writable too.
  • There was no .dockerignore, so every build shipped the whole checkout (.git, logs/, build/, ...) as its context.

Change

Base image. shepherd_base/Dockerfile holds:

  • the Python image, as ARG PYTHON_IMAGE, now the one place to bump it;
  • the shared runtime settings: PYTHONHASHSEED, PYTHONFAULTHANDLER, PYSTOW_HOME, PIP_NO_CACHE_DIR;
  • Shepherd's dependencies, read from pyproject.toml by shepherd_base/deps.py.

It contains no Shepherd code, so it rebuilds only when pyproject.toml or the Python image changes.

Images. Every image is FROM shepherd_base and adds layers in this order:

  1. its own requirements.txt;
  2. Shepherd's code, installed with pip install --no-deps;
  3. its own files;
  4. the SHEPHERD_VERSION argument, last.

The server installs its arax-api extras from deps.py before any code, then installs --no-deps -e . as before. score_paths keeps its PyTorch index and retries.

Permissions. /app is owned by nru:0 with mode 775, so the workers can still create their runtime data directories (/app/omnicorp_lmdb, /app/arax_dbs, ...). The code stays root-owned and read-only. Group 0 keeps working for platforms that run an arbitrary UID in group 0.

Wiring.

  • compose: additional_contexts: shepherd_base: service:shepherd_base on all 28 images makes compose build the base first. The base service is in a base profile, so up never runs it.
  • Release workflow: a new push_base_to_registry job pushes shepherd_base:<version> and :latest. The server and worker jobs need it and build on that exact tag through build-contexts.
  • .dockerignore is an allowlist of exactly what the Dockerfiles copy.
  • The README explains how to build a single image without compose.

Verification

I couldn't pull the RENCI image here: the sandbox's network refuses ghcr's blob host. I built against a stand-in instead: python:3.12-slim with an nru user and this sandbox's proxy CA. The stand-in is local test scaffolding only and is not in the repo.

  • The base, the server and 25 of the 27 worker images build. ars_merge and ars_premerge stop at their git-pinned biothings_annotator requirement because the stand-in has no git. The RENCI image does (these images release today), and that step is unchanged from devel.
  • Caching: after editing shepherd_utils/__init__.py, the rebuild of merge_message showed its base and requirements.txt layers as CACHED, and only the code layers ran.
  • In the image: it runs as nru; /app is nru:root 775 and writable; worker.py is root:root 644 and read-only; no pip cache is left behind; SHEPHERD_VERSION is the build argument; shepherd_utils imports from site-packages.
  • Both base wirings work:
    • the CI-style --build-context shepherd_base=docker-image://... build succeeds;
    • docker compose build sipr builds shepherd_base first (Image shepherd_base Built, then Image shepherd-sipr Built);
    • docker compose config --services doesn't list shepherd_base, so up won't start it.
  • docker compose config is valid.
  • New tests/unit/test_dockerfiles.py (61 cases) pins:
    • for every image: it builds FROM shepherd_base, its requirements come before the code, the code is installed --no-deps, and the version comes last;
    • no chmod -R 777 anywhere;
    • the base has no code and sets no-cache;
    • compose and the release workflow wire up the base;
    • deps.py matches pyproject.toml.
  • Full unit suite against real Postgres passes, and black --check . is clean.

Notes


Generated by Claude Code

Each of the 28 Shepherd Dockerfiles copied shepherd_utils before
`pip install .`, so any code change reinstalled every dependency in
every image. None of them used --no-cache-dir, `chmod -R 777` made /app
(and in score_paths the code) world-writable, and with no .dockerignore
every build shipped the whole checkout as its context.

- shepherd_base/Dockerfile holds the Python image, the shared runtime
  settings (PYTHONHASHSEED, PYTHONFAULTHANDLER, PYSTOW_HOME,
  PIP_NO_CACHE_DIR) and Shepherd's dependencies, read from
  pyproject.toml by shepherd_base/deps.py. It contains no Shepherd
  code, so it rebuilds only when pyproject.toml or the Python image
  changes.
- Each image is now FROM shepherd_base and adds, in this order: its
  own requirements, Shepherd's code (pip install --no-deps), its files,
  and last the release version. A code change rebuilds only the last
  layers, and a new release tag no longer invalidates the dependency
  layers. The server installs its arax-api extras the same way, before
  any code.
- /app is owned by nru:0 with mode 775, for the data workers download at
  runtime. The code stays root-owned and read-only.
- compose builds the base first through
  additional_contexts: service:shepherd_base; the base service sits in
  the `base` profile, so `up` never runs it. The release workflow
  pushes shepherd_base:<version> first, and the server and worker jobs
  build on it via build-contexts.
- .dockerignore limits the context to what the Dockerfiles copy.
- tests/unit/test_dockerfiles.py pins the layer order, the base and
  release wiring, and that no image is world-writable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QGsF4qJ2EoSTDy2M2RRTVm
@codecov

codecov Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 56.02%. Comparing base (7ed9acb) to head (41c592c).
⚠️ Report is 4 commits behind head on devel.


Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 257a841...41c592c. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@maximusunc
maximusunc merged commit 3fb348c into devel Nov 11, 2026
2 checks passed
@maximusunc
maximusunc deleted the audit/223-base-image branch November 11, 2026 12:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Audit P5] Consolidate: one shared poll loop and lookup helper, one worker base image, PgBouncer

2 participants