PyArrow OSError: Unexpected end of stream

[LucaMarconato]

Since the release of pyarrow 22.0.0 the CI started failing, with errors similar to the one below. Please note that the error doesn't seem to be spatialdata-related. Full CI run here.

tests/test_xenium.py:213: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
src/spatialdata_io/_utils.py:59: in wrapper
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
src/spatialdata_io/readers/xenium.py:192: in xenium
    return_values = _get_tables_and_circles(path, cells_as_circles, specs, gex_only)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
src/spatialdata_io/readers/xenium.py:570: in _get_tables_and_circles
    metadata = pd.read_parquet(path / XeniumKeys.CELL_METADATA_FILE)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/site-packages/pandas/io/parquet.py:669: in read_parquet
    return impl.read(
/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/site-packages/pandas/io/parquet.py:265: in read
    pa_table = self.api.parquet.read_table(
/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/site-packages/pyarrow/parquet/core.py:1899: in read_table
    return dataset.read(columns=columns, use_threads=use_threads,
/opt/hostedtoolcache/Python/3.13.7/x64/lib/python3.13/site-packages/pyarrow/parquet/core.py:1538: in read
    table = self._dataset.to_table(
pyarrow/_dataset.pyx:589: in pyarrow._dataset.Dataset.to_table
    ???
pyarrow/_dataset.pyx:3969: in pyarrow._dataset.Scanner.to_table
    ???
pyarrow/error.pxi:155: in pyarrow.lib.pyarrow_internal_check_status
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   OSError: Unexpected end of stream

pyarrow/error.pxi:92: OSError

The pyarrow release also broke some tests in spatialdata, but these have been already fixed here: scverse/spatialdata#1003, and a release will be made soon.

[LucaMarconato]

Two of the new Xenium datasets appear to be corrupted. Here is a script to reproduce the problem with the latest pyarrow.

##
import pyarrow.parquet as pq

# does not work
pq.read_table('data/Xenium_V1_Protein_Human_Kidney_tiny_outs/cells.parquet')
pq.read_table('data/Xenium_V1_MultiCellSeg_Human_Ovary_tiny_outs/cells.parquet')

# works
pq.read_table('data/Xenium_V1_Human_Ovary_tiny_outs/cells.parquet')
pq.read_table('data/Xenium_V1_human_Lung_2fov_outs/cells.parquet')
pq.read_table('data/Xenium_V1_human_Breast_2fov_outs/cells.parquet')

[pryvkin10x]

This change in pyarrow will fix this: apache/arrow@main...pitrou:arrow:gh47981-pq-impala-file
I recommend staying at pyarrow v21 if possible until that fix gets released.

[LucaMarconato]

Thanks for the info! I see the corresponding PR (apache/arrow#47992) has been merged already, but a release is not out yet.

Since the above impacts the Xenium tests but should not impact users (since users would not use the truncated parquet files), I will skip the tests until the new pyarrow release is available.

[LucaMarconato]

I will actually just add pyarrow==21 in the test extra dependency group.