Skip to content

Take a base over only in place of an eviction, and check what is kept - #97

Draft
gspark-etri wants to merge 16 commits into
mainfrom
feat/state-aware-selector-20261111
Draft

gspark-etri wants to merge 16 commits into
mainfrom
feat/state-aware-selector-20261111

Conversation

@gspark-etri

Copy link
Copy Markdown
Contributor

What this changes

A pull that takes an idle artifact over (delta apply, #41) changes its files in place. This closes four gaps around that path and one around reading a base that stays intact (#58).

Change Where Default
Claim refuses a base with a hard-linked file pkg/csi (aliased, claim) always on
Consume a base only when the node is full, least recently used first; Evict enforces the limit Materializer.MaxArtifacts, -max-artifacts 0 = previous behaviour
Read back and hash-check what a taken-over base leaves in place; a failing chunk is fetched PullOptions.VerifyKept, -verify-kept off
Read a base that is not taken over only when the store is not local PullOptions.BaseCopy, Capabilities.Local, -base-copy library: always (unchanged); CSI plugin: remote
Join adjacent chunks of a base file into reads of up to MaxRange pkg/store/spans.go (readSpans) always on

Why

  • Hard links. A second link to a base file changes with the base, outside the artifact and outside every lease. Only the benchmark harness checked this.
  • Capacity. With only the idle window, a pull could destroy a warm artifact although the node had room for both. With MaxArtifacts, the artifact a pull consumes is the one eviction would remove.
  • Verification. Bytes left in place were trusted because the base was checked when it was pulled. VerifyKept removes that assumption at the cost of one read and hash of the kept bytes.
  • Base copy. On 15–18 GB artifacts a local store reconstructed in 32–35 s where reading the base took 44–47 s, so content share alone picked the slower path. With a remote store (the usual CSI deployment) remote and always behave the same.

Behaviour change to note

The CSI plugin now defaults to -base-copy=remote. Deployments whose store is an fs backend on the node will stop reading idle bases that are not taken over; pass -base-copy=always for the old rule.

Tests

go test -race ./pkg/... ./cmd/... passes on linux/amd64 (Go from the existing build cache on the test host). New tests:

  • TestOwnerHardLinkedDonorRefusesConsumption
  • TestOwnerFreeSlotRefusesConsumption, TestEvictAboveMaxArtifacts, TestOfferCarriesPolicies
  • TestRebaseVerifiesWhatItKeeps, TestRebaseDoesNotVerifyByDefault
  • TestBaseCopyRemoteSkipsALocalStore, TestReadSpansJoinsNeighbours

Not done

  • No timing was measured for the joined base reads or for VerifyKept.
  • The branch is based on exp/86-owner-correctness, so the PR also carries its two test commits.

🤖 Generated with Claude Code

gspark-etri and others added 16 commits November 2, 2026 22:50
A pull that takes an idle artifact over changes its files in place. Four
gaps around that are closed:

- Claim refuses a base that has a file with a second hard link, which
  would otherwise change outside the artifact and outside every lease.
- Materializer.MaxArtifacts ties consumption to eviction: below the limit
  nothing is consumed, at the limit the least recently used idle artifact
  is offered first, and Evict removes least recently used idle artifacts
  above the limit. A free slot is no longer filled by destroying a warm
  artifact. 0 keeps the previous behaviour.
- PullOptions.VerifyKept reads back every chunk a taken-over base leaves
  in place and checks its hash. A chunk that fails is fetched like any
  other, so a base damaged after it was pulled is repaired, not carried
  into the new artifact. PullStats reports Verified and Repaired.
- PullOptions.BaseCopy decides when a base that is not taken over is read
  where it is. BaseCopyRemote reads it only if the store is not local
  (Capabilities.Local): on 15-18 GB artifacts a local store took 32-35 s
  where reading the base took 44-47 s. The library default is unchanged
  (BaseCopyAlways); the CSI plugin defaults to -base-copy=remote.

Base reads (and the new read-back) join chunks that follow one another in
a file into single reads of up to MaxRange, instead of one read per chunk.

Tests: go test -race ./pkg/... ./cmd/... passes on linux/amd64. No timing
was measured for the joined reads or for VerifyKept.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
onggi pull gains --verify-kept, --base-copy (always, remote, never),
--base-intact (read --base where it is, never move its files away) and
--base-range (largest single read from the base; 1 reads chunk by chunk).
PullOptions.BaseRange carries the last one; 0 means MaxRange as before.

The CSI plugin and the pull command share parseBaseCopy.

These make the paths of the previous commit measurable from the command
line: a base read chunk by chunk against the same read in joined ranges,
and a taken-over base with and without the read-back.

Tests: go test -race ./pkg/store/ ./pkg/csi/ ./cmd/... passes.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The cache policy now picks what a pull consumes, and a pull may read an
artifact that is in use.

- With MaxArtifacts, only the eviction victim (the least recently used
  idle artifact) is offered for consumption. Before, the pull chose the
  best aligned of up to three idle artifacts, which could destroy an
  artifact the cache would have kept.
- Offer.Readable lists up to three published artifacts, in use or not,
  that a pull may read where they are (PullOptions.ReadBases). Offer.Hold
  (PullOptions.Hold) keeps the one being read from being taken over or
  evicted until the read ends: Claim and Evict treat it as busy.
- The CSI plugin now reads back and checks what a taken-over artifact
  leaves in place by default (-verify-kept=true). On a 15 GB artifact
  this added 1.7 s to a 4.9 s pull.
- A one-level store can frame its chunks (Create.FrameFlat, --frame-flat),
  to compare flat chunking with two levels under the same compression.

Tests added: a leased artifact is read and not consumed; a read hold
blocks Claim and Evict; only the victim is offered under a limit; leases
survive a new materializer on the same root; read-only bases are held and
never claimed; a framed one-level store round-trips and is smaller.
go test -race ./pkg/store/ ./pkg/csi/ ./cmd/... passes on linux/amd64.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
pkg/casync materializes artifacts from a casync-format (desync) chunk
store and implements csi.Puller. Keep pairs files by path, finds the
chunks the base holds at the same offset from the two indexes alone,
and writes only the others from the store; it reads kept chunks back
only when the base is not attested unchanged (inode, size, mtime
recorded when this engine wrote it). Copy seeds desync's assembler with
a held base; Reconstruct assembles from the store. Written files are
synced unless --no-sync. onggi casync publish|pull|verify|attest.

go.mod is updated on the build host (go get desync v1.1.5).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The manifest digest hashed only paths, sizes and modes, so two variants
of equal size shared one manifest (caught by the new tests). Each file
entry now carries the SHA-256 of its index, checked on load.
go.mod/go.sum from go get github.com/folbricht/desync@v1.1.5.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
onggi csi -engine casync -casync-store S -casync-index I materializes
with the casync engine; kept chunks are read back only when the base is
not attested unchanged (or with -casync-verify-always). A materializer
test checks that a sibling in use is never consumed and that a released
idle sibling is the one consumed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Audit findings: the engine read a sibling (Copy) whenever one was
readable, although with a local chunk store BaseCopyRemote means
assemble from the store. The ownership test accepted either idle
sibling; it now checks that the least recently used idle sibling is
consumed (Release counts as a use) and that a released sibling is
consumed once it becomes the victim.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…shape-named variants

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… Copy uses the tau threshold

The victim is deleted by eviction anyway, so waiting only loses its bytes; leases still guard it.
Copy now applies the same share as Keep (Algorithm 1 line 7).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…city and transfer accounting

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lls count fetched chunks; Copy seeds from a sibling holding the target at any offset

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…path

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… place, batched), NPU copy benchmark, NPU-B batched-copy shim

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e and a leased base under concurrent pulls; varmorph tools: descriptor file output, aligned-only switch, apply on host (new / in place), VCDIFF to copy list, reach analysis, ring saves only read windows, far-scratch budget

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant