Skip to content

fix: use parallelized numba functions if possible#155

Draft
ilan-gold wants to merge 5 commits intomainfrom
ig/parallel_kernels
Draft

fix: use parallelized numba functions if possible#155
ilan-gold wants to merge 5 commits intomainfrom
ig/parallel_kernels

Conversation

@ilan-gold
Copy link
Contributor

@ilan-gold ilan-gold commented Mar 2, 2026

This is basically a one-to-one port of the scanpy function. It might make sense to export this function from this package but maybe it's a weird fit as well. Not sure!

@ilan-gold ilan-gold added the run-gpu-ci Apply this label to run GPU CI once label Mar 2, 2026
@codecov
Copy link

codecov bot commented Mar 2, 2026

Codecov Report

❌ Patch coverage is 26.66667% with 33 lines in your changes missing coverage. Please review.
✅ Project coverage is 42.82%. Comparing base (395b37e) to head (b073c49).

Files with missing lines Patch % Lines
src/fast_array_utils/utils.py 28.57% 30 Missing ⚠️
src/fast_array_utils/conv/scipy/_to_dense.py 0.00% 3 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (395b37e) and HEAD (b073c49). Click for more details.

HEAD has 4 uploads less than BASE
Flag BASE (395b37e) HEAD (b073c49)
6 2
Additional details and impacted files
@@             Coverage Diff             @@
##             main     #155       +/-   ##
===========================================
- Coverage   96.78%   42.82%   -53.96%     
===========================================
  Files          19       16        -3     
  Lines         466      418       -48     
===========================================
- Hits          451      179      -272     
- Misses         15      239      +224     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@github-actions github-actions bot removed the run-gpu-ci Apply this label to run GPU CI once label Mar 2, 2026
@codspeed-hq
Copy link

codspeed-hq bot commented Mar 2, 2026

Merging this PR will degrade performance by 33.61%

❌ 6 regressed benchmarks
✅ 226 untouched benchmarks

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Benchmark BASE HEAD Efficiency
test_stats_benchmark[scipy.sparse.csc_array-2d-ax0-float32-is_constant] 1.7 ms 2.4 ms -30.45%
test_stats_benchmark[scipy.sparse.csc_array-2d-ax0-int32-is_constant] 1.7 ms 2.4 ms -29.55%
test_stats_benchmark[scipy.sparse.csc_array-2d-ax0-float64-is_constant] 1.7 ms 2.6 ms -33.61%
test_stats_benchmark[scipy.sparse.csr_array-2d-ax1-float64-is_constant] 1.7 ms 2.4 ms -30.11%
test_stats_benchmark[scipy.sparse.csr_array-2d-ax1-float32-is_constant] 1.7 ms 2.4 ms -28.38%
test_stats_benchmark[scipy.sparse.csr_array-2d-ax1-int32-is_constant] 1.7 ms 2.4 ms -29.91%

Comparing ig/parallel_kernels (54584b2) with main (395b37e)

Open in CodSpeed

@ilan-gold ilan-gold added the run-gpu-ci Apply this label to run GPU CI once label Mar 19, 2026
@ilan-gold
Copy link
Contributor Author

ilan-gold commented Mar 19, 2026

The codspeed looks to be entirely overhead, which I take to be on account of the size of the data. Otherwise, the actual function calls are faster. WDYT @flying-sheep ?

@github-actions github-actions bot removed the run-gpu-ci Apply this label to run GPU CI once label Mar 19, 2026

@pytest.fixture(autouse=True)
def dask_single_threaded() -> Generator[None]:
"""Switch to a single-threaded scheduler for tests since numba crashes otherwise."""
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is that? isn’t our decorator supposed to prevent issues?

Copy link
Member

@flying-sheep flying-sheep left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you explain dask_single_threaded? isn’t that a red flag? Shouldn’t numba just work in dask with our decorator?

@flying-sheep
Copy link
Member

OK, running with the right flags shows the warning how users would see it:

hatch test -- -s -p no:warnings tests/test_stats.py::test_is_constant
[…]                                                                                                                                                                                                                                                                                                                      
tests/test_stats.py ...sss...............sss./Users/philipp.angerer/Dev/Python/fast-array-utils/src/fast_array_utils/stats/_is_constant.py:66: UserWarning: Detected unsupported threading environment. Trying to run _is_constant_cs_major in serial mode. In case of problems, install `tbb`.
  return _is_constant_cs_major(a, shape)
Numba workqueue threading layer is terminating: Concurrent access has been detected.

 - The workqueue threading layer is not threadsafe and may not be accessed concurrently by multiple threads. Concurrent access typically occurs through a nested parallel region launch or by calling Numba parallel=True functions from multiple Python threads.
 - Try using the TBB threading layer as an alternative, as it is, itself, threadsafe. Docs: https://numba.readthedocs.io/en/stable/user/threading-layer.html

Fatal Python error: Aborted
[…]

so numba actually has something more to say and has the same recommendation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants