Skip to content

Fix PR build cache progress lost on macOS/Linux timeouts - #192

Merged
wolfv merged 2 commits into
mainfrom
fix-pr-build-cache
Nov 4, 2026
Merged

wolfv merged 2 commits into
mainfrom
fix-pr-build-cache

Conversation

@wolfv

@wolfv wolfv commented Nov 4, 2026 •

Copy link
Copy Markdown
Member

Problem

The osx-64 build in #144 repeatedly hits the six-hour job limit. Attempt 1 saved 734 packages, but attempts 2 and 3 both restored that same cache and lost their additional progress:

gtar: distros/lyrical/work/output/osx-64: file changed as we read it
Failed to save: ... gtar ... exit code 1

Both logs show rattler-build surviving until orphan-process cleanup, after the cache upload was attempted. Attempt 3.

Also, actions/cache/save reports step success even when saving fails. The old-cache deletion step could therefore delete the only usable cache. In #144, the fork's read-only token happened to prevent that deletion.

Changes

  • Give POSIX builds a 330-minute budget, with a 340-minute step timeout as a backstop, leaving room for cache upload before the six-hour job limit.
  • Run the POSIX build in its own process group. On timeout or interruption, freeze and kill the group before cleaning the output directory; killing only a compiler can let its build script carry on and package an empty build.
  • Preserve restored packages and archives whose completion rattler-build logged; remove new/changed unfinished archives, including concurrent outputs. Save the POSIX cache only after this cleanup confirms it is safe.
  • Verify the replacement cache exists in the current ref before deleting old caches. Skip deletion on fork PRs, whose tokens cannot delete caches.
  • Keep the existing cache paths and keys, so lyrical: build python packages for 3.12 next to 3.14 #144 can reuse its saved progress. Windows retains its existing process handling and cancellation cleanup; it also gets the step-timeout headroom and cache-deletion guard.

Validation

  • python -m pytest tools/tests -q: 71 passed.
  • Two focused regression tests (parameterized): interruption stops descendants and preserves only completed/restored archives; failed cache verification prevents deletion. YAML-string assertions and the separate archive-replacement test were removed to keep the test scope small.
  • Local macOS smoke tests with rattler-build 0.80.0: a package builds successfully; a subsequent build times out with exit 124, its build-script heartbeat stops, and the earlier package remains cached.
  • actionlint -shellcheck= .github/workflows/testpr.yaml: passed. Full actionlint reports only the two pre-existing SC2012 warnings.
  • Full ROS/osx-64 build and GitHub cache upload not run locally. build:none keeps this PR's CI to tooling checks; after merge, rebase lyrical: build python packages for 3.12 next to 3.14 #144 onto main and retry its build without deleting its cache.

@wolfv wolfv added the build:none PR build: build no packages, only run the checks (tooling, CI or docs changes) label Nov 4, 2026
@wolfv
wolfv merged commit dd4b7ee into main Nov 4, 2026
8 checks passed
@wolfv
wolfv deleted the fix-pr-build-cache branch November 4, 2026 17:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

build:none PR build: build no packages, only run the checks (tooling, CI or docs changes)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant