Skip to content

fix: moran's calculation parallelization#4148

Merged
ilan-gold merged 2 commits into
mainfrom
ig/moran_numba_66
Jun 9, 2026
Merged

fix: moran's calculation parallelization#4148
ilan-gold merged 2 commits into
mainfrom
ig/moran_numba_66

Conversation

@ilan-gold

@ilan-gold ilan-gold commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

Was getting the following beginning with numba>=0.66rc1:

Details =================================== FAILURES =================================== __________________ test_consistency[morans_i-single-threaded] __________________ [gw1] linux -- Python 3.14.5 /home/runner/.local/share/hatch/env/virtual/scanpy/B9PcT7QG/hatch-test.pre/bin/python3

metric = <function morans_i at 0x7fae9eda6140>

@pytest.mark.usefixtures("_threading")
def test_consistency(metric) -> None:
    pbmc = pbmc68k_reduced()
    pbmc.layers["raw"] = pbmc.raw.X.copy()
    g = pbmc.obsp["connectivities"]
    equality_check = partial(np.testing.assert_allclose, atol=1e-11)

    # This can fail
    equality_check(
        metric(g, pbmc.obs["percent_mito"]),
        metric(g, pbmc.obs["percent_mito"]),
    )
    equality_check(
        metric(g, pbmc.obs["percent_mito"]),
        metric(pbmc, vals=pbmc.obs["percent_mito"]),
    )

    equality_check(  # Test that series and vectors return same value
        metric(g, pbmc.obs["percent_mito"]),
        metric(g, pbmc.obs["percent_mito"].values),
    )

    equality_check(
        metric(pbmc, obsm="X_pca"),
        metric(g, pbmc.obsm["X_pca"].T),
    )

    all_genes = metric(pbmc, layer="raw")
    first_gene = metric(
        pbmc, vals=pbmc[:, pbmc.var_names[0]].layers["raw"].toarray().ravel()
    )
  np.testing.assert_allclose(all_genes[0], first_gene, rtol=1e-9)

E AssertionError:
E Not equal to tolerance rtol=1e-09, atol=0
E
E Mismatched elements: 1 / 1 (100%)
E Max absolute difference among violations: 7.9750179849e-08
E Max relative difference among violations: 1.8702209022e-07
E ACTUAL: array(0.426421)
E DESIRED: array(0.426421)

tests/test_metrics.py:72: AssertionError
__________________ test_consistency[morans_i-multi-threaded] ___________________
[gw1] linux -- Python 3.14.5 /home/runner/.local/share/hatch/env/virtual/scanpy/B9PcT7QG/hatch-test.pre/bin/python3
data[i] -= regressor[i] @ coeff

tests/test_scaling.py::test_scale[mask-center-int64-array-csc_matrix]
tests/test_scaling.py::test_scale[mask-center-int64-anndata-csc_matrix]
tests/test_scaling.py::test_scale[mask-center-float32-anndata-csc_matrix]
tests/test_scaling.py::test_scale[mask-center-float32-array-csc_matrix]
/home/runner/.local/share/hatch/env/virtual/scanpy/B9PcT7QG/hatch-test.pre/lib/python3.14/site-packages/scipy/sparse/_index.py:216: SparseEfficiencyWarning: Changing the sparsity structure of a csc_matrix is expensive. lil and dok are more efficient.
self._set_arrayXarray(i, j, x)

tests/test_scaling.py::test_scale[mask-center-int64-anndata-csr_matrix]
tests/test_scaling.py::test_scale[mask-center-float32-array-csr_matrix]
tests/test_scaling.py::test_scale[mask-center-float32-anndata-csr_matrix]
tests/test_scaling.py::test_scale[mask-center-int64-array-csr_matrix]
/home/runner/.local/share/hatch/env/virtual/scanpy/B9PcT7QG/hatch-test.pre/lib/python3.14/site-packages/scipy/sparse/_index.py:216: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil and dok are more efficient.
self._set_arrayXarray(i, j, x)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

  • generated xml file: /home/runner/work/scanpy/scanpy/test-data/test-results.xml -
    =========================== short test summary info ============================
    FAILED tests/test_metrics.py::test_consistency[morans_i-single-threaded] - AssertionError:
    Not equal to tolerance rtol=1e-09, atol=0

Mismatched elements: 1 / 1 (100%)
Max absolute difference among violations: 7.9750179849e-08
Max relative difference among violations: 1.8702209022e-07
ACTUAL: array(0.426421)
DESIRED: array(0.426421)
FAILED tests/test_metrics.py::test_consistency[morans_i-multi-threaded] - AssertionError:
Not equal to tolerance rtol=1e-09, atol=0

Mismatched elements: 1 / 1 (100%)
Max absolute difference among violations: 7.9750179849e-08
Max relative difference among violations: 1.8702209022e-07
ACTUAL: array(0.426421)
DESIRED: array(0.426421)

and suspect this could be the cause

  • Closes #
  • Tests included or not required because:
  • Release notes not necessary because: fix in preparation for upcoming release

@codecov

codecov Bot commented Jun 8, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.61%. Comparing base (d66537f) to head (073472b).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4148      +/-   ##
==========================================
+ Coverage   79.59%   79.61%   +0.02%     
==========================================
  Files         120      120              
  Lines       12786    12786              
==========================================
+ Hits        10177    10180       +3     
+ Misses       2609     2606       -3     
Flag Coverage Δ
hatch-test.low-vers 78.83% <ø> (-0.01%) ⬇️
hatch-test.pre 79.48% <ø> (+0.03%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
src/scanpy/metrics/_morans_i.py 100.00% <ø> (ø)

... and 1 file with indirect coverage changes

@ilan-gold

Copy link
Copy Markdown
Contributor Author

At this point, I think this may just be worth reporting upstream to numba and loosening our tolerance

@ilan-gold ilan-gold force-pushed the ig/moran_numba_66 branch from c1f7f7a to 02d5793 Compare June 8, 2026 15:22
@ilan-gold ilan-gold added this to the 1.12.2 milestone Jun 8, 2026
@ilan-gold ilan-gold enabled auto-merge (squash) June 9, 2026 11:13
@ilan-gold ilan-gold merged commit 0ac3337 into main Jun 9, 2026
14 checks passed
@ilan-gold ilan-gold deleted the ig/moran_numba_66 branch June 9, 2026 11:21
ilan-gold added a commit that referenced this pull request Jun 9, 2026
…ization) (#4149)

Co-authored-by: Ilan Gold <ilanbassgold@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant