Skip to content

0.6.3: backends state a file's content hash and size, or why they cannot - #41

Open
trissim wants to merge 2 commits into
mainfrom
feat/file-content-identity
Open

trissim wants to merge 2 commits into
mainfrom
feat/file-content-identity

Conversation

@trissim

@trissim trissim commented Oct 12, 2026

Copy link
Copy Markdown
Contributor

What

The storage layer can now say what bytes a file holds. The backend that owns the file computes the answer; callers never hash.

  • polystore.file_content: FileContent (ABC) with HashedFileContent(sha256, size_bytes) and UnhashedFileContent(reason). to_record() / FileContent.from_record() round-trip through JSON; the subclass is chosen from each subclass's hashed class attribute.
  • DataSource.file_content(file_path) and FileManager.file_content(file_path, backend).
  • Version 0.6.3, uv.lock regenerated.

What each backend answers

Backend Answer
Disk Hashed (streamed in 1 MiB chunks); directory or non-regular file is unhashed; missing raises FileNotFoundError
Zarr Disk-passthrough files hashed through disk; array keys unhashed ("declares no physical file"); missing raises
OME-Zarr Unhashed: the declared path is the store directory
Memory (override) Unhashed: entries are objects in process memory; directory unhashed; missing raises
Bio-Formats (override) Unhashed: an address names one plane inside a container; missing container raises
Virtual workspace (override) Delegates to the source backend of the mapped reference; unhashed when the reference selects part of the source file; mapped directory unhashed; unmapped path raises
OMERO, streaming (not DataSource) FileManager.file_content raises StorageResolutionError

Notes for review

  • DataSource.physical_source_path defaults to returning the address itself, so the memory backend would otherwise have hashed a same-named file on disk. Memory therefore overrides file_content; a test covers this.
  • The address must be absolute. The base passes an empty base_path to the path hook so a relative address stays relative and is refused, rather than being resolved against the working directory.
  • Bio-Formats and partial virtual references are unhashed on purpose: hashing the container would give every plane the same identity.

Tests

tests/test_file_content.py (28 tests). Full suite excluding tests/integration: 629 passed, 1 skipped. Ruff on the touched files is unchanged from main (148 existing findings, none new); the new module and test file are clean.

🤖 Generated with Claude Code

DataSource.file_content(path) returns a FileContent: HashedFileContent
(sha256 + size, streamed in 1 MiB chunks from the physical file the
backend's path hook declares) or UnhashedFileContent(reason). A missing
file raises FileNotFoundError; a relative address is refused instead of
being read from the working directory.

Backends whose path hook does not name the file's own bytes answer for
themselves: memory entries are objects, a Bio-Formats address is one
plane inside a container, and a virtual workspace path delegates to its
source backend unless it selects part of the source file.

FileManager.file_content(path, backend) delegates to the source backend.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings October 12, 2026 18:36

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@trissim

trissim commented Oct 12, 2026

Copy link
Copy Markdown
Contributor Author

Version note: 0.6.3 is taken by the arraybridge 0.5.0 migration (separate PR, merging first because OpenHCS is blocked on it). Please do not merge this PR before that one; afterwards rebase this branch onto main and release it as 0.6.4.

🤖 Generated with Claude Code

…codec's, not a hand-written record

FileContent registers its two answers under content_kind (hashed, not_hashed).
to_record/from_record and the hashed flag with its hand-kept table are removed:
python-introspect 0.3.0 stores a family member under its registry key.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@trissim trissim mentioned this pull request Oct 12, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants