Skip to content

fix(utils): SparseCopy from a FUSE or NFS source leaves zero blocks as holes - #331

Merged
CMGS merged 2 commits into
masterfrom
fix/sparse-copy-zero-blocks
Nov 9, 2026
Merged

CMGS merged 2 commits into
masterfrom
fix/sparse-copy-zero-blocks

Conversation

@CMGS

@CMGS CMGS commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

ReflinkCopy falls back to SparseCopy whenever FICLONE fails, for example across filesystems. SparseCopy writes every range that SEEK_DATA reports.

  • When a FUSE daemon does not implement FUSE_LSEEK, the kernel falls back to generic_file_llseek, which reports the whole file as one data range. JuiceFS behaves this way.
  • So a vm clone --from-dir whose source dir is on JuiceFS writes every zero byte of the COW. That happens on every archived wake, checkpoint claim or template refill from a JuiceFS checkpoint_dir in sandboxd.

Evidence (hardware, cocoon 2afbc27)

Archived wake through sandboxd, with the run dir on xfs in both cases:

Checkpoint store Wake exec COW allocated in run dir Extents
JuiceFS 7.07 s, 6.97 s 10240 MiB (full size) 2
ext4 0.48 s, 0.28 s 522 MiB 34

Change

  • When the source fd is on FUSE (fstatfs returns FUSE_SUPER_MAGIC), each data segment is copied in 1 MiB ReadAt/WriteAt blocks, and all-zero blocks are left unwritten. The destination is truncated to the source size first, so those blocks stay holes.
  • Every other source keeps io.CopyN, which uses copy_file_range. A userspace block copy measured 4–6% slower than copy_file_range on ext4, and this is the clone path.
  • New test TestCopySegmentSkipsZeroBlocks: it fails when the zero skip is removed (16384 of 16384 blocks allocated).

Hot path

  • A non-FUSE source pays one extra fstatfs per copied file.
  • A FUSE source reads every byte, as before, but writes only the non-zero blocks.

Gates

  • make lint: 0 issues. linux + darwin.
  • make fmt-check: rc 0.
  • asl -forwarder=false ./...: clean on both GOOS.
  • go test -race ./utils/ ./hypervisor/...: ok, also in a Linux container.
  • Codex: converged with no findings.

Hardware run of this fix

Pending. JuiceFS archived wake with 2afbc27 vs this commit, plus one ext4 run per arm. Results will be posted here.

A FUSE daemon that does not answer FUSE_LSEEK makes the kernel report the
whole file as data, so SparseCopy from a JuiceFS checkpoint dir wrote every
zero byte of a 10 GiB COW. From a FUSE source the copy now reads 1 MiB
blocks and skips all-zero ones. Every other source keeps io.CopyN and its
copy_file_range: a userspace block copy measured 4-6% slower on ext4.
@CMGS

CMGS commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

Hardware verification: pass

Setup. Testbed host, kernel 6.12, sandboxd 707ef2cf, CH 750aa2b. The two arms differ only in the cocoon binary: A = 2afbc27, F = 5d2e580. Each run used a fresh sandboxd, so the golden, the archive and the wake clone all run on the arm's cocoon.

Workload. A small claim with 512 MiB of urandom data is archived to a dir checkpoint store, then woken by an exec. The run dir is on xfs.

Run Wake exec COW allocated in run dir Extents
JuiceFS / A 8351 ms 10240 MiB 2
JuiceFS / F 7503 ms 540 MiB 19
JuiceFS / F 6990 ms 540 MiB 19
JuiceFS / A 8003 ms 10240 MiB 2
ext4 / A 297 ms 522 MiB 33
ext4 / F 309 ms 522 MiB 34
  • Data integrity. In a separate pair, the md5 of the guest's data file before the archive equals the md5 after the wake on both arms. F: 540 MiB, 6.6 s. A: 10240 MiB, 13.3 s, under a higher host load.
  • What improves. F drops the 10 GiB write and the 10 GiB of run-dir disk per wake. The wake still reads the whole file over FUSE because JuiceFS reports it as data, so latency improves by only about 10%. The read side sets most of the cost.
  • Non-FUSE path. ext4 is unchanged at about 0.3 s and 522 MiB.

…sAllZero

An NFS client without SEEK support falls back to generic_file_llseek like a
FUSE daemon without lseek, and docs/cli.md offers --to-dir for NFS golden
images that clone --from-dir reads. The block check reuses tar.go's
isAllZero.
@CMGS CMGS changed the title fix(utils): SparseCopy from a FUSE source leaves zero blocks as holes fix(utils): SparseCopy from a FUSE or NFS source leaves zero blocks as holes Nov 9, 2026
@CMGS

CMGS commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

Follow-up 90aa33b, from an independent whole-file review:

  • The gate now also covers NFS (NFS_SUPER_MAGIC). docs/cli.md offers --to-dir for NFS golden images that vm clone --from-dir reads, and an NFS client without SEEK support falls back to generic_file_llseek, like FUSE without lseek. NFS-to-local copy_file_range already returns EXDEV, so the copy was in userspace anyway; the added cost is the zero compare.
  • The block check reuses the package's existing isAllZero (utils/tar.go).

The zero-skip path was verified on JuiceFS hardware. NFS adds only the magic constant; there was no NFS hardware run.

@CMGS
CMGS merged commit 5fb9ced into master Nov 9, 2026
4 checks passed
@CMGS
CMGS deleted the fix/sparse-copy-zero-blocks branch November 9, 2026 15:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant