Run-Length Encoding input is not properly supported by RectilinearChunkGrid

[maxrjones]

Zarr version

v3.2.0

Numcodecs version

v0.16.5

Python Version

3.12

Operating System

Mac

Installation

PyPI via a uv venv

Description

@jhamman's forthcoming blog post included this as an expected supported input, but it doesn't work.

I would expect the below to at least raise a more informative error, instead I get:

  File "/Users/max/Documents/Code/maxrjones/test-scripts/test-zarr/test-zarr-blog1.py", line 14, in <module>
    arr = zarr.create_array(
          ^^^^^^^^^^^^^^^^^^
  File "/Users/max/.cache/uv/environments-v2/test-zarr-blog1-041a0d706b93a99a/lib/python3.12/site-packages/zarr/api/synchronous.py", line 948, in create_array
    sync(
  File "/Users/max/.cache/uv/environments-v2/test-zarr-blog1-041a0d706b93a99a/lib/python3.12/site-packages/zarr/core/sync.py", line 158, in sync
    raise return_result
  File "/Users/max/.cache/uv/environments-v2/test-zarr-blog1-041a0d706b93a99a/lib/python3.12/site-packages/zarr/core/sync.py", line 118, in _runner
    return await coro
           ^^^^^^^^^^
  File "/Users/max/.cache/uv/environments-v2/test-zarr-blog1-041a0d706b93a99a/lib/python3.12/site-packages/zarr/core/array.py", line 4715, in create_array
    return await init_array(
           ^^^^^^^^^^^^^^^^^
  File "/Users/max/.cache/uv/environments-v2/test-zarr-blog1-041a0d706b93a99a/lib/python3.12/site-packages/zarr/core/array.py", line 4415, in init_array
    rectilinear_meta = RectilinearChunkGridMetadata(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<string>", line 4, in __init__
  File "/Users/max/.cache/uv/environments-v2/test-zarr-blog1-041a0d706b93a99a/lib/python3.12/site-packages/zarr/core/metadata/v3.py", line 299, in __post_init__
    object.__setattr__(self, "chunk_shapes", _validate_chunk_shapes(self.chunk_shapes))
                                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/max/.cache/uv/environments-v2/test-zarr-blog1-041a0d706b93a99a/lib/python3.12/site-packages/zarr/core/metadata/v3.py", line 231, in _validate_chunk_shapes
    bad = [i for i, e in enumerate(edges) if e < 1]
                                             ^^^^^
TypeError: '<' not supported between instances of 'list' and 'int'

Steps to reproduce

# /// script
# dependencies = [
#   "zarr==3.2.0",
#   "numpy",
#   "obstore",
# ]
# ///

import zarr
print(zarr.print_debug_info())
zarr.config.set({"array.rectilinear_chunks": True})
store = zarr.storage.MemoryStore()
# Create an array where chunks along the second axis vary in size
arr = zarr.create_array(
store=store,
shape=(10, 10),
chunks = [[6, 4], [[3, 3], 1]],
dtype="i4",
zarr_format=3,
)
arr[:] = 1
print(arr.write_chunk_sizes)
print(arr.read_chunk_sizes)

Additional output

No response

[d-v-b]

i'll investigate! random guess but it it possible that we never wired up the create_array signature to accept rle chunks?

[maxrjones]

i'll investigate! random guess but it it possible that we never wired up the create_array signature to accept rle chunks?

sounds good, I started but am glad to hand this off to you. Here's how far claude and I got so far:

---

The Python create_array(chunks=...) entry point doesn't accept RLE pairs (or, in some cases, bare-int per-axis shorthand) even though the spec allows them and the JSON deserialization path handles them fine. Two paths diverge:

• JSON path: RectilinearChunkGridMetadata.from_dict (src/zarr/core/metadata/v3.py:351-368) calls expand_rle on each list dim, so reading stored metadata with RLE pairs works.
• Python path: create_array at src/zarr/core/array.py:4406-4419 does tuple(tuple(dim_edges) for dim_edges in chunks) — no RLE expansion, no bare-int handling. The downstream _validate_chunk_shapes (src/zarr/core/metadata/v3.py:211-238) then sees the RLE pair [3, 3] as an inner element and crashes with TypeError: '<' not supported between instances of 'list' and 'int'. resolve_chunks at src/zarr/core/metadata/v3.py:374-394 (used by the lower-level AsyncArray._create path) has the same gap.

There's also a detection bug worth noting: _is_rectilinear_chunks in src/zarr/core/chunk_grids.py:303-319 only inspects the first element, so chunks=[[1,2,3], 4] is detected as rectilinear but chunks=[4, [1,2,3]] is not — the bare-int per-axis shorthand only works when it isn't the first axis.

Spec input forms that should be accepted by create_array

Per zarr-extensions/chunk-grids/rectilinear/README.md, each axis can be:

1. A bare int (regular step that repeats to cover the axis): chunks=[4, [1, 2, 3]]
2. A flat list of explicit edge lengths: chunks=[[6, 4], [3, 3]] (this is the only form create_array actually accepts today)
3. A list mixing literal ints and [value, count] RLE pairs: chunks=[[[4, 2]]], chunks=[[[1, 3], 3]]

Test gaps

The example-based tests in tests/test_unified_chunk_grid.py (test_spec_*) cover all three forms, but only via parse_chunk_grid (the JSON path). No test calls create_array with form 1 or 3, which is why this regression slipped through. The hypothesis strategy in src/zarr/testing/strategies.py::rectilinear_chunks only emits flat list[int], so property tests don't exercise the input-encoding surface either.

Suggested fix shape

1. Centralize input → constructor conversion in a helper that handles all three encodings (mirror what from_dict does), used by both resolve_chunks and the inline create_array path.
2. Fix _is_rectilinear_chunks to detect any nested element, not just the first.
3. Parametrized create_array test enumerating each spec form (bare-int, RLE-only, mixed RLE+literal, plus the spec README example).
4. Hypothesis strategy that takes a fixed sequence of edges and randomly re-encodes it (bare int when uniform / flat list / pure RLE / partial RLE), with an invariance property: every encoding produces the same write_chunk_sizes.
5. JSON↔Python equivalence property: to_dict() of an array built with one form, parsed via parse_chunk_grid, yields identical chunk_shapes.
6. Update the chunks parameter docstring in src/zarr/core/array.py:4043-4055 — currently it only mentions Sequence[Sequence[int]].

I had started writing the tests in (3) but stopped before committing.

[d-v-b]

ignoring joe's blog post for a moment, do we want create_array to accept the RLE form of of a repeated chunk edge size?

pro: it's nice to use a consistent interface for declaring chunks across the codebase,
lack of a pro: the RLE form is specifically for saving space in a JSON document. The call site of create_array isn't under the same space constraints as a stored metadata document.
con: the RLE form is confusingly similar to the non-RLE form -- I had to read chunks = [[6, 4], [[3, 3], 1]] several times to realize it's for a 2D array, not a 3D array! This could lead to users accidentally creating arrays with chunks they didn't intend. By contrast, chunks = [[6, 4], [*(3,) * 3, 1]] is much less confusing and just as compact.

thoughts? I'm -.00001 on accepting the RLE form in create_array, but happy to go the other way.

and in either case, I'd like to fold this fix into #3899

[maxrjones]

So, the solution would be:

1. Accept bare-int per-axis (chunks=[4, [1, 2, 3]]); fix _is_rectilinear_chunks to detect when any axis is nested rather than only the first axis (it's currently order-dependent — a real bug).
2. Reject inline RLE pairs in create_array with a clear error pointing at the Python alternative — something like "RLE-encoded chunk specs ([value, count] pairs) are only accepted when reading stored metadata; in create_array, write the expanded form, e.g. [3]*3 + [1]". That catches the original confusion at the API boundary and tells the user how to fix it.
3. Replace the TypeError blow-up in _validate_chunk_shapes with the message above.

For tests, this also simplifies the property-test plan: there's only one Python input form (bare ints + flat edge lists), so we don't need an encoding-invariance strategy. We still want:

• A parametrized example test covering the bare-int form and the rejection of RLE-pair forms with the new error message.
• A JSON-side round-trip property: to_dict → parse_chunk_grid produces the same chunk_shapes for arbitrary edge sequences, including ones that compress to RLE. (That's where RLE belongs —
  under the JSON serializer's hood.)

I'd be fine we that. We can expand accepted inputs easier than taking them away

[d-v-b]

Something else to keep in mind with the chunks signature in particular -- if we ever want to remove the shards parameter and say everything with chunks, we can do that by with forms like this:

• shape=(10,12), chunks=[(5,5), (6, 6)] -> "use sharding, with outer chunks sized (10, 12), and inner chunks sized (5,6)"
• shape=(10,12), chunks=[10, [(3, 3), (3, 3)]] -> "use sharding, with outer chunks sized (10, 6), and inner chunks sized (10,3)"
• shape=(12,), chunks=[[(3,3), (3,3)]] -> "use nested sharding, with outer chunks sized (12,), and inner chunks sized (6,), and inner-inner chunks sized (3,)"

These get harder and harder to read but it works!

but this really only works if we disallow any degenerate representations, like [[5,2]] (RLE) vs [5,5] (explicit).