Repository navigation
Conversation
GET /files/list streams NDJSON entries (recursive walk, error entries
pass through); GET /files/archive streams a gzip'd tar with repeated
exclude params; PUT /files/archive streams a tar into a directory and
returns {files, bytes}. All three idle-bounded like the other /files
routes; the archive PUT shares files_put's no-mid-stream-retry contract.
Async twins cover bytes, sync iterables, async iterables and file
objects at the MockTransport wire seam.
Pure engine behind filesystem.sync() in koyeb/sandbox/sync.py: local walk in the executor's FileEntry shape (relative /-separated, %04o perms, RFC3339Nano UTC mtimes, symlinks listed not followed, special files skipped, excluded dirs pruned whole); path_match is a faithful port of Go's path.Match (* and ? never cross /) so client-side excludes agree with the executor's archive route byte for byte — cross-checked against real Go on 24 arms; diff on path/type/size/mtime with whole-second mtime (PAX tar cannot round-trip ns; rsync's default), optional sha256 arms via injected hash callables, unreadable remote entries skipped not guessed; plan_batches fills ~part_size batches without ever splitting a file, oversized files separated for upload_file/download_file; prune_descendants keeps topmost deletes; relative_remote resolves remote=None.
build_archive spools a changed set as a gzip'd PAX tar — relative /-separated members, modes and whole-second mtimes kept, symlinks as links, owners dropped — the exact shape the executor's PUT /files/archive extracts. extract_archive mirrors the executor's extractor: parents created, members in the way replaced, dir modes/mtimes restored last (writing into a dir bumps its mtime), a keep filter for the pull side's changed set, and the same traversal refusal (absolute paths, ..). Round- trips pinned: content, mode, mtime, symlinks, empty dirs, replacements, keep-filter isolation, traversal + absolute refusal.
sync(local, remote=None, direction, delete, exclude, checksum, dry_run) on SandboxFilesystem and AsyncSandboxFilesystem: list both sides (local walk vs files_list NDJSON), diff on path/type/size/whole-second mtime, send the changed set as ~50 MiB tar batches spooled per batch and streamed into files_archive_put; files over one part ride upload_file/download_file (multipart once ST-126 slice 2 lands — the call sites are the public seams); push treats a 404 listing as an empty tree (the archive PUT creates the root), pull refuses a missing source; delete=True prunes to topmost deletes — remote via the legacy routes, local via rmtree/unlink; checksum=True compares sha256 for files that pass size+mtime, with the executor-side digests prefetched through hash_candidates (the pure diff cannot await) and a 404 hash read as unprovable, not changed; a no-op sync sends nothing. dry_run computes counts only. Adds files_stat(hash=) for the stat?hash=sha256 arm. Pinned against a miniature executor: the fake backs /files routes with a real tree using the engine's own walk/build/extract, so push and pull round-trip real tar bytes.
DeclarativeSnapshot._copy: a directory reaches the builder sandbox as one archive push through filesystem.sync on executors v0.0.24+ (binary-safe, modes/mtimes/symlinks kept — the legacy per-file walk read files as text and lost all of that), a single file through the streamed upload_file (binary-safe; no probe needed — it carries its own legacy fallback). The legacy walk stays byte-for-byte for old executors, pinned by its shape. Also fixes the pre-existing unbound SandboxFilesystem.write_file call in write_files (deferred since the files-api round 2 review) — the loop now calls self.write_file, so the async override chain holds.
Crapper pass on the diff-touched functions, every one now under the CRAP-10 bar with coverage: the Go-parity matcher restructured around a _match_class parser, a _match_unit single-character matcher, and a _validate_chunk pass (the two-phase shape drops Go's fused failed-flag protocol — the post-failure state is unobservable, verified against the 24 Go-cross-checked arms); path_match's star scan extracted into _star_skip; diff's checksum verdict into _hash_verdict and the delete tail into _remaining_deletes; extract_archive's member dispatch into _extract_member/_clear_member/_extract_file. Also pins the async twins of the files_list and files_archive_get 400 arms (the round-5 lesson: async failure arms drift unpinned), and the unarchivable-pair rule — both sides having an 'other' entry at one path is never a delete.
The two-phase _match_chunk conflated chunk exhaustion with match
failure: a chunk whose last unit mismatched consumed the chunk, so the
'return the leftover chunk verdict' check read failure as success —
path_match('a*b', 'b') matched (Go's path.Match says false). The verdict
now travels with ok explicitly. Found chasing the uncovered tail-
validation lines the CRAP pass flagged; pinned with the regression rows
cross-checked against real Go, plus the malformed-later-chunk error
(Go validates the whole pattern even past an earlier mismatch), the
async checksum and async pull+delete twins (the round-5 lesson again —
the async arms were the uncovered ones), and diff's pair-verdict
extraction — every engine function now at CRAP ≤ 10 with coverage.
A directory's st_size is a filesystem artifact (APFS ~64, ext4 4096) that never converges cross-filesystem; every sync re-planned every directory. Only real content carries size. [\]] and [\-] are Go path.Match class literals; rejecting the backslash made the escape arm dead code and aborted walks on patterns the executor accepts.
The remote listing rode the diff unfiltered: push with delete removed the very files exclude promised to skip, and pull planned excluded entries — counted as downloads that never arrived, oversized ones fully downloaded. Both listings now filter through path_excluded (own path or any directory prefix, mirroring the walker's whole-dir pruning), a pull's archive GET leaves oversized files out so their bytes travel once, and a collision guard refuses loudly when an oversized pattern would also drop planned members — suffix, class or dir-prune — before any transfer.
A type flip at one path aborted the pull permanently — open() on a directory, makedirs over a file, or a symlinked dir silently written through to arbitrary paths outside the destination. Extraction now clears the in-the-way object without following links, deletes tolerate paths an earlier flip already replaced, the walk is loud on unreadable subtrees, and pull treats a missing destination root as an empty tree while push refuses a missing source like rsync.
The async fake had no multipart routes, so the async oversized arm was untestable and every failure arm — bad direction, legacy executor, listing failure, missing remote — went unpinned while the sync twin pinned all four. The fakes delegate to the sync tree, the legacy executor gets an async shape, and the checksum stat-failure guard is pinned on both twins.
The no-retry promise hung on one unwatched token — dropping idempotent=False silently re-enables blind 5xx retries and replays a consumed stream as a truncated body. Mid-stream resets and 502s are now single-call, on both twins.
build's CRAP was 120 — the recipe loops and the sandbox lifecycle in one body. _apply_operations carries the loops; the seam is pinned through build() itself, including a failed command.
The sync surface shipped undocumented — no sync(), no list/archive transports, three stale signatures — while sibling surfaces publish. The engine module joins the doc list.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
DeclarativeSnapshot.copy()rides the same routes: a directory builds as one archive push (binary-safe, modes/mtimes/symlinks kept), a single file throughupload_file; pre-v0.0.24 executors keep the per-file legacy walk, pinned unchanged.A five-pass review loop (parallel per-file lanes + tools lens, then four focused verification passes) landed 24 red-first findings on top of the build: the exclude contract (push+delete no longer deletes what exclude promised to skip; pull no longer plans or counts prunable members), directory size convergence, Go-parity for escaped class members, extraction make-room, loud unreadable-subtree walks, async failure-arm pinning with multipart-capable fakes, the no-retry/build-seam/no-op tokens watched, and docs regenerated with the whole sync surface (
koyeb.sandbox.syncadded to the doc module list).Evidence
files-multiparttip — 448 sandbox tests green;SandboxFilesystemhas no sync surface and snapshotcopywalks per-file as text.After: 607 sandbox tests green (
uv run --no-sync pytest koyeb/sandbox/ -q→607 passed in 9.77s): +112 build tests and +47 review tests, every review test red-first or an explicit coverage pin. Mutation runs scoped to the delta report zero survivors across every pass (crapper: all delta-touched functions under CRAP 10; the ancestor-blindpath_excludedmutant is killed by all six of its tests).delete=True, exclude=("*.log",)leavesskip.logon the remote withdeleted == 0; pull withexclude=("backup",)never plansx/backup/small.txt(0 downloads, no archive GET, local copy untouched); an oversizedbackupnext to a changedx/backup/small.txtraisesSandboxFilesystemErrornaming the member before any transfer.Merge Danger
Door: two-way — additive surface on both fs twins; reverting the branch reverts the feature. No wire changes below the base.
Blast Radius: the sandbox filesystem sync/copy surface only. The archive/list routes and multipart rides require executor v0.0.24+ (
has_files_routes()gates them); older executors keep the legacy walk for snapshot copy.delete=Trueprunes the destination — new capability, explicit opt-in. Stacked on #106 (basefiles-multipart); retarget tomainwhen #106 merges.