Zarr chunks ignored when reading arrays with `StringDType`

[tomwhite]

What happened?

When opening a Zarr array created with a NumPy 2 StringDType the Xarray DataArray doesn't have the same chunks as the underlying Zarr array.

What did you expect to happen?

The Xarray chunking should be the same as the Zarr chunking. In the MVCE below, the chunking is the same if using a fixed-length unicode dtype (dtype="U").

Minimal Complete Verifiable Example

# /// script
# requires-python = ">=3.11"
# dependencies = [
#   "xarray[complete]@git+https://github.com/pydata/xarray.git@main",
#   "zarr",
# ]
# ///
#
# This script automatically imports the development branch of xarray to check for issues.
# Please delete this header if you have _not_ tested this script with `uv run`!

import numpy as np
import xarray as xr
import zarr
xr.show_versions()

# create chunked string array using zarr
root = zarr.create_group(store="g")
data = np.array([
    ["a", "b", "c", "d"],
    ["e", "f", "g", "h"],
    ["i", "j", "k", "l"],
    ["m", "n", "o", "p"],
], dtype="T")
a = root.create_array(name="a", data=data, chunks=(2, 2), dimension_names=["x", "y"])

# open with xarray and check chunk sizes
x = xr.open_zarr("g", consolidated=False)
assert x["a"].chunks == ((2, 2), (2, 2))  # fails

Steps to reproduce

No response

MVCE confirmation

• Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• Complete example — the example is self-contained, including all data and the text of any traceback.
• Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• New issue — a search of GitHub Issues suggests this is not a duplicate.
• Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Relevant log output

Anything else we need to know?

No response

Environment

Details

[jsignell]

It looks like there was a special case for this codec with zarr-python v2:

xarray/xarray/backends/zarr.py

Lines 204 to 209 in 6448b2e

	# preserve vlen string object dtype (GH 7328)
	if (
	not _zarr_v3()
	and self._array.filters is not None
	and any(filt.codec_id == "vlen-utf8" for filt in self._array.filters)
	):

so just need to do something similar for zarr-python v3. I tried:

diff --git a/xarray/backends/zarr.py b/xarray/backends/zarr.py
index 410a6a49..93e4b6da 100644
--- a/xarray/backends/zarr.py
+++ b/xarray/backends/zarr.py
@@ -206,6 +206,10 @@ class ZarrArrayWrapper(BackendArray):
             not _zarr_v3()
             and self._array.filters is not None
             and any(filt.codec_id == "vlen-utf8" for filt in self._array.filters)
+        ) or (
+            _zarr_v3()
+            and self._array.serializer
+            and self._array.serializer.to_dict()["name"] == "vlen-utf8"
         ):
             dtype = coding.strings.create_vlen_dtype(str)
         else:

but it resulted in a few test errors like:

self = <xarray.tests.test_backends.TestZarrDirectoryStore object at 0x7abc7a440370>

    def test_roundtrip_object_dtype(self) -> None:
        floats = np.array([0.0, 0.0, 1.0, 2.0, 3.0], dtype=object)
        floats_nans = np.array([np.nan, np.nan, 1.0, 2.0, 3.0], dtype=object)
        bytes_ = np.array([b"ab", b"cdef", b"g"], dtype=object)
        bytes_nans = np.array([b"ab", b"cdef", np.nan], dtype=object)
        strings = np.array(["ab", "cdef", "g"], dtype=object)
        strings_nans = np.array(["ab", "cdef", np.nan], dtype=object)
        all_nans = np.array([np.nan, np.nan], dtype=object)
        original = Dataset(
            {
                "floats": ("a", floats),
                "floats_nans": ("a", floats_nans),
                "bytes": ("b", bytes_),
                "bytes_nans": ("b", bytes_nans),
                "strings": ("b", strings),
                "strings_nans": ("b", strings_nans),
                "all_nans": ("c", all_nans),
                "nan": ([], np.nan),
            }
        )
        expected = original.copy(deep=True)
        with self.roundtrip(original) as actual:
            try:
>               assert_identical(expected, actual)
E               AssertionError: Left and right Dataset objects are not identical
E               Differing data variables:
E               L   strings_nans  (b) object 24B 'ab' 'cdef' nan
E               R   strings_nans  (b) StringDType() 48B 'ab' 'cdef' ''
E               L   strings       (b) object 24B 'ab' 'cdef' 'g'
E               R   strings       (b) StringDType() 48B 'ab' 'cdef' 'g'
E               L   bytes_nans    (b) object 24B b'ab' b'cdef' nan
E               R   bytes_nans    (b) |S4 12B b'ab' b'cdef' b''

[dcherian]

String instead of object is a great improvement! We can update those tests.

[jsignell]

I didn't realize that dtype is not actually considered in assert_identical (#4727 for instance). So I am still trying to figure out what is actually different about these data variables that is causing the test to fail.

[jsignell]

Seems like the issue is how nulls are handled within duck_array_ops.isnull. Similar to #10967. I think we can just add a check in there to handle this case. I'll open a PR shortly.

[tomwhite]

Thank you for fixing this @jsignell!