Skip to content

feat: streamed binary-safe file I/O + legacy fallback (ST-125) - #105

Merged
sfroment merged 4 commits into
mainfrom
files-api
Oct 28, 2026
Merged

sfroment merged 4 commits into
mainfrom
files-api

Conversation

@sfroment

@sfroment sfroment commented Oct 27, 2026 •

Copy link
Copy Markdown
Member

Summary

Every file operation now dispatches once per handle, and the streamed /files routes (executor v0.0.24+, koyeb/sandbox-container#29) become the default — with automatic fallback to the legacy JSON routes:

SandboxFilesystem.write_file / read_file / open_read / stat / upload_file / download_file
  │
  ├─ client.has_files_routes()            # body-less GET /files/stat, cached per handle
  │    400 "path is required" → streamed routes
  │         PUT /files    — raw bytes, X-Content-SHA256, temp+rename server-side
  │         GET /files    — chunk stream, Range-capable, .part+rename locally
  │         GET /files/stat — FileStat; sizes from stat, never HEAD
  │    404 → legacy JSON routes, byte-for-byte old behavior, one warning per handle
  │    anything else → raise, uncached (an auth blip can't poison the handle)

New surface (sync + async twins, parity-pinned):

 executor_client.py
+  files_put / files_read /      # the transport: streamed PUT (idle-bounded 60 s, no total
+  files_stat / has_files_routes #  cap), chunk-iterator GET, the probe; _async_stream
+  _probe_verdict                #  adapts sync streams for the AsyncClient; timeouts name
+                               #  the bound that fired
 filesystem.py
+  _put_body / _read_content     # base64 becomes a transport detail
+  FileStat (+ export)           # type|dir|symlink|other, octal mode, RFC 3339 mtime
+  open_read()                   # streaming iterator, constant memory
+  stat()                        # new routes only; raises on legacy
+  progress(done, total)          # ≤ 100 ms throttle, guaranteed final call
+  _translating_files_errors      # one error seam: 404→NotFound, 409→Exists

Evidence

Before: any binary write corrupted or crashed — write_file("/a.bin", b"\x00\xff") raised UnicodeDecodeError under the default encoding, and encoding="base64" wrote the base64 text to disk (users had to run base64 -d in the sandbox). Reads lost invalid UTF-8 to U+FFFD. No test could round-trip arbitrary bytes; a 256 MiB legacy write peaked at 1.2 GB executor RSS.

After: byte-exact both directions, verified end to end:

  • test_upload_streams_binary_raw — 256 KiB of bytes(range(256)) uploads raw, and the executor's {size, sha256} reply is cross-checked against a locally computed digest (constant memory) — "corrupted in transit" on mismatch. A mid-stream files_put failure is pinned as never retried (idempotent=False): a replayed stream could write empty content.
  • test_bytes_go_raw / test_encoding_none_returns_raw_bytes — write and read back exact bytes; encoding="base64" now means real bytes on disk (the changelog-named behavior change).
  • Async parity is real at the wire, not just in the AST: round 5's fresh-context review caught that sync iterables and file objects passed into httpx's AsyncClient raise RuntimeError — every async upload failed against a real executor, masked by bytes-only fakes. _async_stream now adapts sync streams on the async twin, transport-tested with both iterables and file objects.
  • Timeout taxonomy is uniform: an idle stall raises SandboxTimeoutError naming the bound that fired ("Request timed out after 60s" on the files routes, …30s on legacy) — read streams included; resets mid-stream stay generic.
  • .part + os.replace downloads leave the destination untouched on failure (pinned by mid-transfer-404 tests); progress is throttled to 100 ms with a guaranteed final call (pinned on an exact float boundary).
  • Suite: 372 passed; sync/async parity AST-pinned; zero surviving mutants in changed logic — the two remaining are equivalent dead-flag mutants (post-rename unlink of a nonexistent .part, swallowed by OSError), hand-verified three times. The mutator's snapshot cache was caught giving false greens twice; every named survivor since is red-green-proven by hand-mutation before it counts.

Six rounds of /ub-review (spec / standards / ub-mode / crapper+dryer+mutator, parallel lenses) dispositioned on ST-125 — the last round closed with zero genuine findings. Follow-up debt is ledgered on the ticket (all pre-existing: SandboxFileIO write machinery, _request_with_retry's retry ladder, 18 pre-existing mutator survivors).

Merge Danger

Door: two-way — pure client-side addition; the probe falls back to the legacy routes on executors < v0.0.24, and a revert restores the previous behavior entirely.

Blast radius: default-behavior change, contained. On v0.0.24+ executors, encoding="base64" on read_file/write_file flips from "the file holds base64 text" to "the transport encodes the real bytes" (ST-103's release notes name it); stat, binary-safe open_read and stream content raise clear errors naming v0.0.24 on legacy executors; every other caller sees unchanged behavior plus at most one warning per handle. Live e2e rides the v0.0.24 rollout / ST-102's kind stack — the unit bar here is fakes at the raw-HTTP seam, plus MockTransport transport tests that drove out the async-wire bug the fake-based suite could not see.

Part of KOYEB-6631 — Python slice 1 of 3 (ST-126 multipart + resume — also inherits the per-call timeout= override and the parents wiring — and ST-127 sync() follow as separate PRs behind their executor gates).

@sfroment
sfroment marked this pull request as ready for review October 28, 2026 14:02
Streamed /files/stat, PUT /files and chunked GET /files with an
idle-bounded read loop, plus a cached has_files_routes() probe so
callers pick routes once per handle instead of probing per call.
Only 400/404 count as route answers; anything else is an error.
write_file, read_file, open_read and stat() ride the streamed
transports with end-to-end content verification; the base64
legacy routes stay as fallback when /files is absent and fail
fast on inputs they cannot carry. FileStat is exported.
@sfroment
sfroment merged commit fad3245 into main Oct 28, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants