fix: byte-order bugs and little-endian test assumptions on big-endian hosts

[d-v-b]

🤖 AI text below 🤖

Refs #3438. The agent ran the test suite on a big-endian host: s390x emulated with QEMU, Debian sid, and Debian's numpy 2.4.6 and numcodecs 0.17. On main it got 110 failures across 30 test functions. With this branch, 11493 tests pass and 3 fail. Those 3 are test_examples, which calls uv, and the image does not have it.

Real bugs

1. V3 data types without an explicit byte order defaulted to little-endian, while NumPy dtype names mean host order. On a big-endian host, dtype="float64" gave <f8 and dtype=np.float64 gave >f8. Every reopened array came back as <f8, so its dtype changed between create and open, and its arrays were not in native order. This one bug caused most of the failures.
   • Fix: HasEndianness now defaults to sys.byteorder, because V3 metadata carries no byte order.
   • Nothing changes on little-endian hosts. Stored chunks stay little-endian by default, because the default serializer is BytesCodec(endian="little").
2. scale_offset failed on arrays whose byte order is not the host's, on every host. NumPy arithmetic returns native results, which tripped the dtype-preservation check. Also, >u8 != np.uint64 sent >u8 data down the int64-widening path. The codec now computes in native order and restores the input's byte order.
3. Base64 fill values of structured data types in V3 metadata were read and written in the in-memory byte order. They are now little-endian bytes in both directions.
4. ShardingCodec defaulted to a bare BytesCodec() for its inner and index codecs. That codec uses the host byte order, so a sharded array written with default settings had different bytes on s390x than on x86. Both defaults are now endian="little".

Tests that assumed a little-endian host

The failing tests and a static audit turned up these patterns:

• hard-coded 'little' in reprs
• native .tobytes() compared against little-endian stored bytes
• fixed compressed sizes and metadata sizes
• bare BytesCodec() / Int32() used as expected values
• migrate-CLI and v2 fixtures that use host-order dtype names

The tests now use explicit </> dtypes and explicit endian=. Two tests gained a case in the other byte order:

• test_migrate_endian has a >u2 case.
• pipeline parity has a bytes-little-endian case, so one case is non-native on every host.

The info doctests use endianness=....

The regression tests for bugs 2 and 3 fail on little-endian hosts without the fix, so the normal CI covers them. test_v3_data_types_parse_in_host_byte_order only fails on a big-endian host.

Not in this PR: the BytesCodec() default

A bare BytesCodec() defaults to endian=sys.byteorder, so code that builds one itself still writes big-endian chunks on s390x. That output is valid, but it is not identical to what x86 writes.

History: the bytes codec in the original zarrita import defaulted to "little". The host-order default came in with #1660, and neither that PR's description nor its comments discuss the switch.

What happens here: changing the default back changes what existing code writes on big-endian hosts, so it needs a deprecation cycle. An earlier commit in this branch changed the default; a later commit reverts it. The deprecation is proposed in #4439.

CI job

.github/workflows/big-endian.yml runs the byte-order-sensitive tests on an emulated s390x host. It uses docker/setup-qemu-action plus buildx, and the image is cached with type=gha.

• Image: ci/big-endian/Dockerfile, a Debian sid image that gets numpy and numcodecs from Debian packages, so nothing heavy compiles under emulation. This is also the environment Debian tests zarr in.
• Tests: only the data type, codec and metadata tests: tests/test_dtype, test_dtype_registry.py, tests/test_codecs, tests/test_metadata, test_array.py, test_v2.py, test_info.py, test_pipeline_parity.py and test_migrate_v3.py.
  • The full suite takes about 40 minutes under emulation.
  • When the agent ran this job's command locally, that subset took about 2 minutes on 8 cores. GitHub's runners emulate more slowly.
• Triggers: PRs that touch data type, codec, buffer or metadata code (a path filter), a weekly schedule, and workflow_dispatch.

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.43%. Comparing base (cd1e5b3) to head (68d0ab6).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4435      +/-   ##
==========================================
+ Coverage   94.38%   94.43%   +0.04%     
==========================================
  Files          93       93              
  Lines       13205    13219      +14     
==========================================
+ Hits        12464    12483      +19     
+ Misses        741      736       -5     

Files with missing lines	Coverage Δ
src/zarr/codecs/scale_offset.py	100.00% <100.00%> (ø)
src/zarr/codecs/sharding.py	95.53% <ø> (ø)
src/zarr/core/array.py	98.09% <ø> (ø)
src/zarr/core/dtype/common.py	92.22% <100.00%> (+0.08%)	⬆️
src/zarr/core/dtype/npy/structured.py	97.14% <100.00%> (+2.53%)	⬆️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[read-the-docs-community]

Documentation build overview

📚 zarr-metadata | 🛠️ Build #34785817 | 📁 Comparing 6c24181 against latest (8dc50b8)

  🔍 Preview build  

7 files changed · ± 6 modified · - 1 deleted

± Modified

• api/index.html
• api/model/index.html
• api/v3/chunk_grid/index.html
• api/v3/chunk_key_encoding/index.html
• api/v3/codec/index.html
• api/v3/data_type/index.html

- Deleted

• api/v3/definition/index.html

[d-v-b]

cc @avalentino

[d-v-b]

Internal bugfixes / platform compatibility, self merging

[d-v-b]

I requested review when I thought we were changing the bytes codec default here, with that change factored out I don't think review is needed but it is of course welcome