paper: match 0.6.3

[trissim]

Brings paper.md in line with the code at v0.6.3 (5fdb617). main (2a80372) is one commit ahead of the tag and that commit only changes .github/badges/coverage.svg, so the branch is based on main and every claim was checked against the tag. No library code changes. The repository has no CHANGELOG.md, so the ledger comes from paper.md read against the source. Paths below are under src/polystore/.

Nothing was executed: no environment was created and the library was not imported. Every name and call in the code examples was checked by reading the source: DiskBackend, MemoryBackend, FileManager are root exports (__init__.py:39, 54, 83; aliases at disk.py:868, memory.py:630); FileManager(registry) copies a mapping (filemanager.py:38-65); save(data, output_path, backend, **kwargs) :338, load(file_path, backend, **kwargs) :261, load_batch :387, save_batch :416; disk .npy goes to np.save / np.load (disk.py:131, 357-431); a memory save with no parent directory in the path is accepted (memory.py:92-104); Zarr batch keywords (zarr.py:429-442); ZarrBatchLayout (zarr_batch.py:77); stream_request / ViewerStreamRequest (streaming/viewer_transport.py:479, 533-550). The examples were not run. The second example is a fragment: paths, out_paths, pipeline, layout and request are placeholders, and fm must hold zarr and napari_stream backends.

Claim ledger

#	Old claim	Status	What it became	Source
1	Unified API for disk, memory, Zarr, and streaming to Napari or Fiji	true	kept, tools now cited	disk.py:92, memory.py:27, zarr.py:119, napari_stream.py:38, fiji_stream.py:60
2	from polystore import FileManager, BackendRegistry; FileManager(BackendRegistry())	false	BackendRegistry does not exist. Example builds an explicit mapping of DiskBackend and MemoryBackend	__init__.py:14-85; filemanager.py:38-65
3	fm.save(image, "result.npy", backend="napari_stream") # Appears in Napari	false	A viewer save raises unless it gets exactly one keyword, stream_request, a ViewerStreamRequest. Removed from the first example, shown with the keyword in the second	streaming/viewer_transport.py:539-550; streaming/_streaming_backend.py:735
4	"routes operations to explicitly selected backends with no implicit fallback"	true for FileManager	kept for FileManager only; see 5 and 22	filemanager.py:116-154
5	Table rows "Explicit backend selection" and "Zero implicit fallback", with "—" for fsspec, zarr, xarray	misleading	fsspec also selects a filesystem explicitly (by protocol or URL). Inside PolyStore the Zarr backend sends .json, .csv, .txt, .roi.zip, .zip paths to the disk backend. Rows replaced by "Destination chosen by"	zarr.py:41, 307-313, 409-420
6	"Backends auto-register via metaclass"	true	kept; the key attribute _backend_type is named	base.py:162-182; backend_registry.py:20
7	"support lazy imports for optional dependencies"	true for disk format libraries and package exports	stated for those	formats.py:12-32; disk.py:165-255; __init__.py:90-99
8	"atomic file operations for concurrent metadata updates"	true for JSON only	scoped to JSON metadata; "Array and image files are written directly" added	atomic.py:118-186; disk.py:131
9	"one call handles all backends"	misleading	Zarr batches and viewer saves need backend-specific keywords; the paper now says which	zarr.py:429-442; viewer_transport.py:539-550
10	Table: "Unified storage API" ✓ fsspec, — zarr, — xarray	unverifiable as tick marks	replaced by a "Unit of abstraction" row in words
11	Table: "Streaming backends" only PolyStore	true for PolyStore	kept as "Write-only viewer destinations"	streaming/_streaming_backend.py:547
12	Table: "Multi-framework I/O" ✓ PolyStore, ✓ xarray	PolyStore true (torch, JAX, CuPy, TensorFlow disk formats); xarray unverifiable	row removed	disk.py:165-207
13	Table: "Atomic concurrent writes" ✓ PolyStore only	misleading	Only JSON metadata is replaced atomically; row removed	atomic.py:118-141
14	"fsspec ... cannot support streaming because its abstraction is filesystems"	unverifiable as a statement of impossibility	"its interface is files and byte streams"; PolyStore's DataSink.save is described instead	base.py:281-295
15	"PolyStore's abstraction is data sinks"	partly	Root is BackendBase; DataSink and DataSource are siblings	base.py:162, 239, 317
16	"DataSink (write-only) is the root interface, not StorageBackend"	false	BackendBase is the root	base.py:162-236
17	Listing: class StorageBackend(DataSink) with load_batch defined on it	false	StorageBackend(DataSource, DataSink); listing rewritten with the six classes and their registered names	base.py:639
18	Listing: class StreamingBackend(DataSink), no load()	true	kept	streaming/_streaming_backend.py:547-887
19	"The FileManager routes to any DataSink"	false as worded	It routes to any registered BackendBase, including read-only sources	filemanager.py:101-154
20	Streaming hides "GPU tensor conversion, shared memory allocation, ZMQ socket management, ROI serialization" behind save_batch()	true	kept, with the sequence spelled out and the required keyword stated	_streaming_backend.py:292-300, 330-353, 721-804; array_payload.py:8-15; napari_stream.py:43-63; fiji_stream.py:112-134
21	"The orchestrator remains backend-agnostic"	unverifiable	removed
22	(implied by "no silent fallbacks") streaming accepts whatever it is given	false	Text and non-pixel paths are skipped with a log line and no error; now stated	_streaming_backend.py:597-629, 806-816
23	"Cross-platform file locking (fcntl on Unix, portalocker on Windows)"	false	portalocker on every platform; fcntl is not imported anywhere	atomic.py:19, 86-91
24	atomic_update_json() for concurrent metadata writes	true	kept, steps listed (lock, read, update, temp file, os.replace)	atomic.py:167-186
25	"without atomic operations, race conditions corrupt JSON files"	unverifiable (no test or report in the repo)	removed
26	Backends "are lazily instantiated, keeping optional dependencies unloaded until used"	misleading	The process-wide registry is built on first read and then instantiates every context-free backend at once; the paper says only "built on first read"	base.py:691-779; backend_registry.py:63-96
27	"the Napari streaming backend only imports napari when first used"	false	The sending backend never imports napari; only streaming/receivers/napari/* does. Sentence removed	napari_stream.py:15-30; streaming/receivers/napari/image_presentation.py:5
28	"The Zarr backend can write multiple arrays in a single transaction"	false	No transaction exists. A batch is written as one OME-Zarr well; only the plate metadata step is under a file lock	zarr.py:397-598
29	"the Napari backend can batch ROI updates into a single viewer refresh"	false for this package	NapariBatchProcessor states that OpenHCS owns the debounce and its flush is a no-op. Replaced by "A streaming backend sends one message per batch"	streaming/receivers/napari/napari_batch_processor.py:37-85; _streaming_backend.py:783
30	"This is more efficient than per-file operations"	unverifiable; disk and memory batches are loops	removed; the loops are stated	disk.py:433-468; memory.py:131-172
31	"pipelines process thousands of images per experiment"	unverifiable	"process the images of a multi-well plate"
32	Load from "disk (TIFF, OME-TIFF) or virtual workspace (lazy-loaded)"	TIFF true; "lazy-loaded" misleading	The virtual workspace is a path mapping read from plate metadata that delegates each load to another backend; described that way	disk.py:220-239; virtual_workspace.py:202-325
33	"Write results to Zarr (chunked, compressed)"	true	"OME-Zarr (chunked, zlib-compressed by default)"	config.py:264-279; zarr.py:122-127
34	fm.save_batch(processed, paths, backend="zarr")	false	Raises ValueError("chunk_name must be provided"); example now passes chunk_name, batch_layout, row, col	zarr.py:429-442
35	fm.save_batch(processed, paths, backend="napari_stream")	false	Raises ValueError; example passes stream_request	viewer_transport.py:544-546
36	"Concrete Example: A user processes 10,000 images ..."	unverifiable (hypothetical)	removed
37	"the user can switch backends by changing a config parameter, no code changes needed"	misleading	Zarr batches and viewers need their own keywords; replaced by the contextual_save_kwargs() sentence	base.py:261-269; filemanager.py:372-385
38	"Bug Prevention: ... eliminated an entire class of bugs"; os.path.exists() "would fail silently"	unverifiable	removed
39	Without-PolyStore listing (np.save, zarr.save, msgpack)	illustration of other libraries, not of PolyStore	kept unchanged

False or misleading: 18 (rows 2, 3, 5, 9, 13, 16, 17, 19, 22, 23, 26, 27, 28, 29, 32, 34, 35, 37). Removed as unverifiable: 8 (rows 10, 12 for xarray, 21, 25, 30, 31, 36, 38); row 14 was reworded to what can be read from the code.

New statements

• FileManager holds a name-to-instance mapping, has no default backend, does not infer a backend from the path: filemanager.py:38-65, 116-154.
• Unknown backend name, or an operation the backend lacks, raises StorageResolutionError: filemanager.py:146-154, 281-292, 356-370.
• Six-class hierarchy and which registered name sits under which class: base.py:162, 239, 317, 610, 639; streaming/_streaming_backend.py:547; virtual_workspace.py:202; ome_zarr_storage.py:77; bioformats_storage.py:78.
• Nine registered names in 0.6.3 (grep "_backend_type =" under src/): disk, memory, zarr, ome_zarr, bioformats, virtual_workspace, napari_stream, fiji_stream, omero_local (omero_local.py:147, a VirtualBackend, base.py:519).
• Streaming sequence (NumPy conversion, shared-memory copy, JSON over a ZeroMQ REQ socket, wait for reply, release): _streaming_backend.py:292-300, 330-353, 763-804.
• portalocker on every platform; lock on a sidecar .lock file; temp file then os.replace: atomic.py:19, 36-39, 86-91, 118-166.
• AtomicMetadataWriter is built on atomic_update_json: metadata_writer.py:88-93. Zarr plate metadata under file_lock: zarr.py:582-598.
• import polystore loads no backend module: __init__.py:10-99. storage_registry built on first read: base.py:713-779.
• Zarr batch is one OME-Zarr well with one array per field and a filename map used by load_batch: zarr.py:345-395, 500-568.
• FileManager pickles as backend names plus connection parameters and rebuilds backends on unpickle: filemanager.py:68-99; base.py:98-159.
• OpenHCS uses contextual_save_kwargs (openhcs/core/steps/function_artifact_materialization.py:278) and AtomicMetadataWriter (openhcs/core/virtual_workspace_metadata.py, openhcs/core/source_binding_workspace.py), read in the OpenHCS checkout.

For the owner

• Places where the old paper promised something the code does not do: rows 3, 22, 23, 27, 28, 29, 34, 35 above. The two that matter most to a reader trying the paper: neither one-line streaming call nor the one-line Zarr batch call works without backend-specific keywords.
• "No implicit fallback" has two exceptions inside backends. The Zarr backend writes .json, .csv, .txt, .roi.zip and .zip paths through the disk backend (zarr.py:41, 307-313), and streaming backends skip text and non-streamable items without raising (_streaming_backend.py:597-629, 806-816). The paper now states the second; the first is covered by removing the "Zero implicit fallback" table row. Whether the code should do either is a separate question.
• dict(storage_registry) before the registry is initialised is empty. _LazyStorageRegistry (base.py:713-749) triggers initialisation from __getitem__, keys, items and so on, but dict(a_dict_subclass) copies the underlying table without calling any of them. I confirmed that behaviour on a five-line stand-in class with the system Python 3.14 only, not with PolyStore itself and not on 3.11 or 3.12. FileManager.__init__ does dict(registry) (filemanager.py:64). The README calls ensure_storage_registry() first, which is correct; openhcs/processing/backends/analysis/consolidate_special_outputs.py:201 passes storage_registry directly and depends on something having read the registry earlier. This is why the paper's example uses an explicit mapping.
• A pickled FileManager does not carry memory-backend contents: backends without connection parameters are rebuilt with a no-argument constructor (filemanager.py:96-97). The paper says "rebuilt in the worker" and does not claim more.
• create_storage_registry leaves out a backend whose constructor raises, with a warning in the log (backend_registry.py:89-93). Not stated in the paper.
• Statements about fsspec, zarr and xarray in the new table ("protocol or URL", "format engine", no viewer destinations) come from general knowledge of those libraries, not from anything checked in this job. Please confirm or cut.
• Open PR 0.6.3: backends state a file's content hash and size, or why they cannot #41 is titled "0.6.3: backends state a file's content hash and size", but the v0.6.3 tag contains no such code (no match for content_hash or sha256-based identity beyond read_verified_source_bytes, base.py:366-396), so the paper does not mention it.
• "All content was reviewed and tested" in the AI disclosure is the owner's statement and was left as written.
• The date: in the front matter (15 January 2026) was left unchanged.
• paper.bib: no change. napari, fiji, zeromq, ome_zarr and omero were already in the file and are now cited. The numpy, pytorch, jax, tensorflow, cupy entries are still not cited (as before). The metaclassregistry URL (github.com/trissim/metaclass-registry) resolves; github.com/OpenHCSDev/metaclass-registry also exists, and you may prefer it.
• Length: 1011 words by wc -w (front matter and code included) against 876 before, 15% over.

🤖 Generated with Claude Code

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.