fix: use parallelized numba functions if possible#155
fix: use parallelized numba functions if possible#155
numba functions if possible#155Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #155 +/- ##
===========================================
- Coverage 96.78% 42.82% -53.96%
===========================================
Files 19 16 -3
Lines 466 418 -48
===========================================
- Hits 451 179 -272
- Misses 15 239 +224 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Merging this PR will degrade performance by 33.61%
Performance Changes
Comparing |
|
The codspeed looks to be entirely overhead, which I take to be on account of the size of the data. Otherwise, the actual function calls are faster. WDYT @flying-sheep ? |
|
|
||
| @pytest.fixture(autouse=True) | ||
| def dask_single_threaded() -> Generator[None]: | ||
| """Switch to a single-threaded scheduler for tests since numba crashes otherwise.""" |
There was a problem hiding this comment.
why is that? isn’t our decorator supposed to prevent issues?
flying-sheep
left a comment
There was a problem hiding this comment.
Can you explain dask_single_threaded? isn’t that a red flag? Shouldn’t numba just work in dask with our decorator?
|
OK, running with the right flags shows the warning how users would see it: ❯ hatch test -- -s -p no:warnings tests/test_stats.py::test_is_constant
[…]
tests/test_stats.py ...sss...............sss./Users/philipp.angerer/Dev/Python/fast-array-utils/src/fast_array_utils/stats/_is_constant.py:66: UserWarning: Detected unsupported threading environment. Trying to run _is_constant_cs_major in serial mode. In case of problems, install `tbb`.
return _is_constant_cs_major(a, shape)
Numba workqueue threading layer is terminating: Concurrent access has been detected.
- The workqueue threading layer is not threadsafe and may not be accessed concurrently by multiple threads. Concurrent access typically occurs through a nested parallel region launch or by calling Numba parallel=True functions from multiple Python threads.
- Try using the TBB threading layer as an alternative, as it is, itself, threadsafe. Docs: https://numba.readthedocs.io/en/stable/user/threading-layer.html
Fatal Python error: Aborted
[…]so numba actually has something more to say and has the same recommendation |
This is basically a one-to-one port of the
scanpyfunction. It might make sense to export this function from this package but maybe it's a weird fit as well. Not sure!