Repository navigation
Build every image on one shared base image (#223 C11-5) - #250
Merged
Merged
Conversation
Each of the 28 Shepherd Dockerfiles copied shepherd_utils before `pip install .`, so any code change reinstalled every dependency in every image. None of them used --no-cache-dir, `chmod -R 777` made /app (and in score_paths the code) world-writable, and with no .dockerignore every build shipped the whole checkout as its context. - shepherd_base/Dockerfile holds the Python image, the shared runtime settings (PYTHONHASHSEED, PYTHONFAULTHANDLER, PYSTOW_HOME, PIP_NO_CACHE_DIR) and Shepherd's dependencies, read from pyproject.toml by shepherd_base/deps.py. It contains no Shepherd code, so it rebuilds only when pyproject.toml or the Python image changes. - Each image is now FROM shepherd_base and adds, in this order: its own requirements, Shepherd's code (pip install --no-deps), its files, and last the release version. A code change rebuilds only the last layers, and a new release tag no longer invalidates the dependency layers. The server installs its arax-api extras the same way, before any code. - /app is owned by nru:0 with mode 775, for the data workers download at runtime. The code stays root-owned and read-only. - compose builds the base first through additional_contexts: service:shepherd_base; the base service sits in the `base` profile, so `up` never runs it. The release workflow pushes shepherd_base:<version> first, and the server and worker jobs build on it via build-contexts. - .dockerignore limits the context to what the Dockerfiles copy. - tests/unit/test_dockerfiles.py pins the layer order, the base and release wiring, and that no image is world-writable. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QGsF4qJ2EoSTDy2M2RRTVm
Codecov Report✅ All modified and coverable lines are covered by tests. Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part 4 of 4 for #223 (C11-5). The other parts are #247 (poll loop), #248 (lookup helper) and #249 (pool budget). Fixes #223 once those three are merged too.
Problem
shepherd_utilsbeforepip install ., so any code change reinstalled every dependency in every image.SHEPHERD_VERSIONwas declared at the top, so each release tag also invalidated every layer.--no-cache-dir.chmod -R 777 .made/appworld-writable, and inscore_pathsit ran after the code was copied, which left the code writable too..dockerignore, so every build shipped the whole checkout (.git,logs/,build/, ...) as its context.Change
Base image.
shepherd_base/Dockerfileholds:ARG PYTHON_IMAGE, now the one place to bump it;PYTHONHASHSEED,PYTHONFAULTHANDLER,PYSTOW_HOME,PIP_NO_CACHE_DIR;pyproject.tomlbyshepherd_base/deps.py.It contains no Shepherd code, so it rebuilds only when
pyproject.tomlor the Python image changes.Images. Every image is
FROM shepherd_baseand adds layers in this order:requirements.txt;pip install --no-deps;SHEPHERD_VERSIONargument, last.The server installs its
arax-apiextras fromdeps.pybefore any code, then installs--no-deps -e .as before.score_pathskeeps its PyTorch index and retries.Permissions.
/appis owned bynru:0with mode775, so the workers can still create their runtime data directories (/app/omnicorp_lmdb,/app/arax_dbs, ...). The code stays root-owned and read-only. Group 0 keeps working for platforms that run an arbitrary UID in group 0.Wiring.
additional_contexts: shepherd_base: service:shepherd_baseon all 28 images makes compose build the base first. The base service is in abaseprofile, soupnever runs it.push_base_to_registryjob pushesshepherd_base:<version>and:latest. The server and worker jobsneedit and build on that exact tag throughbuild-contexts..dockerignoreis an allowlist of exactly what the Dockerfiles copy.Verification
I couldn't pull the RENCI image here: the sandbox's network refuses ghcr's blob host. I built against a stand-in instead:
python:3.12-slimwith annruuser and this sandbox's proxy CA. The stand-in is local test scaffolding only and is not in the repo.ars_mergeandars_premergestop at their git-pinnedbiothings_annotatorrequirement because the stand-in has nogit. The RENCI image does (these images release today), and that step is unchanged from devel.shepherd_utils/__init__.py, the rebuild ofmerge_messageshowed its base andrequirements.txtlayers asCACHED, and only the code layers ran.nru;/appisnru:root 775and writable;worker.pyisroot:root 644and read-only; no pip cache is left behind;SHEPHERD_VERSIONis the build argument;shepherd_utilsimports from site-packages.--build-context shepherd_base=docker-image://...build succeeds;docker compose build siprbuildsshepherd_basefirst (Image shepherd_base Built, thenImage shepherd-sipr Built);docker compose config --servicesdoesn't listshepherd_base, soupwon't start it.docker compose configis valid.tests/unit/test_dockerfiles.py(61 cases) pins:FROM shepherd_base, its requirements come before the code, the code is installed--no-deps, and the version comes last;chmod -R 777anywhere;deps.pymatchespyproject.toml.black --check .is clean.Notes
shepherd_baseto ghcr alongside the other images. Whoever runs builds outside compose or CI needs the base built first (see the README).shepherd_baseservice, so the two PRs merge in either order.requirements.txt:shepherd_server/requirements.txtwas never installed by the server image and still isn't.pyproject.tomlis the source of truth.Generated by Claude Code