FIX: restore Cython compatibility + pivot to `sys.monitoring` #352

TTsangSC · 2025-06-19T02:14:53Z

(May close #200. Related: cython/cython#3929, cython/cython#5140)

Synopsis

This PR fixes line-profiling for Cython code (which has been broken since the 4.0 version bump) by using an alternative source for code hashes with Cython functions. To that effect, we also switch to using sys.monitoring callbacks instead of the "legacy" tracing system where available (Python 3.12+), making it possible to line-profile Cython code in Python 3.13+.¹

Motivations

It bugged me that we've made all these advancements in line_profiler and yet the cython folks were forced to pin the version to 3, and it seemed like there are no good reasons that we shouldn't be able to support that with minimal overhead.
Since Python 3.12 the existing tracing system has been referred to as "legacy", and largely to be superseded in usage by sys.monitoring. While Python probably won't drop support for the old way any time soon, future versions are likely to focus on optimizing sys.monitoring at the expense of the old system.
As the ecosystem evolves, other tools' support for the old tracing system is also likely to wane. Case in point: since Python 3.13 Cython has switched to the new API, leading to old-style trace functions no longer receiving line events therefrom.

Performance impacts

Tests are run comparing the eager-prof-mod branch (#337) and this PR, excluding the new tests with -k "not cython".

Python version\PR (min/mean time (s; of 5))	#337	This PR
3.8	11.51/11.67	11.50/11.60
3.11	11.76/11.91	11.90/12.00
3.12	12.98/13.16	12.99/13.11
3.13	14.39/14.72	13.92/14.25

Code changes

~~(This PR is based on #337, which is slated to be merged soon-ish. New changes are from f225f93 onwards.)~~

line_profiler/_line_profiler.pyx
- _sys_monitoring_[de]register():
  - No longer defined in Python < 3.12
  - Now used for setting up/tearing down the sys.monitoring hooks
- get_frame_code():
  Now returning the code object instead of its .co_code
- find_cython_source_file():
  New function for resolving the source-file location of Cython functions (because .__code__.co_filename is only relative and does not generally work)
- LineProfiler:
  - add_function():
    Added handling for creating line hashes for Cython-function code objects, which:
    - Cannot be replaced
    - Do not contain the absolute path to the Cython source file, and
    - Only have empty or all-null-byte bytecodes
  - enable(), disable():
    - Now only using the "legacy" tracing system if sys.monitoring is not available
    - (Update 7 Jul): inlined call to PyEval_SetTrace(NULL, NULL) for disabling legacy tracing
- inner_trace_callback():
  - Refactored from the body of previous python_trace_callback()
  - Added handling for computing line hashes for Cython-function code objects
- monitoring_line_event_callback(), monitoring_exit_frame_callback():
  New callbacks for use with sys.monitoring.register_callback()
- legacy_trace_callback():
  Supersedes the previous python_trace_callback()
line_profiler/line_profiler.py
- get_code_block():
  - New wrapper function around inspect.getblock() to facilitate the handling of Cython source
  - (Updated 7 Jul): added doctest
line_profiler/profiler_mixin.py[i]
- is_c_level_callable()
  No longer returning True for Cython callables
- ByCountProfilerMixin.get_underlying_functions()
  Now returning Cython functions for compatible (i.e. not 3.12) Python versions

Test changes

tests/cython_example/
New, small, installable, line-traceable Cython project
tests/test_cython.py²
New test module for Cython-related stuff:
- test_recover_cython_source()
  Test for the retrieval and parsing of Cython sources
- test_profile_cython_source()
  Test for the line-profiling of Cython code

Possible conflicts

Other PRs which touches the Cython code like #334, #349, and #351.

Note that as acknowledged in the Cython docs line-tracing (and thus profiling) fundamentally doesn't work on Python 3.12, since the old C API was changed without an appropriate replacement, which was only added in 3.13. ↩
These tests are rather slow (≈ 5–6 s combined) due to their use of a module-scoped fixture which temporarily installs the tests/cython_example/ project. Maybe we should mark it (and certain bigger tests) with a marker and only run it for say half of the matrix? ↩

codecov · 2025-06-19T03:25:14Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 87.36%. Comparing base (757add6) to head (97ecd5c).
Report is 12 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #352      +/-   ##
==========================================
+ Coverage   87.21%   87.36%   +0.14%     
==========================================
  Files          18       18              
  Lines        1627     1630       +3     
  Branches      344      347       +3     
==========================================
+ Hits         1419     1424       +5     
+ Misses        153      151       -2     
  Partials       55       55

Files with missing lines	Coverage Δ
line_profiler/line_profiler.py	`99.66% <100.00%> (ø)`
line_profiler/profiler_mixin.py	`88.63% <100.00%> (-0.26%)`	⬇️

... and 1 file with indirect coverage changes

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7d2c415...97ecd5c. Read the comment docs.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

line_profiler/_line_profiler.pyx get_frame_bydecode() Renamed from `get_frame_code()` get_frame_code() Now an alias to `PyFrame_GetCode()` or its equivalent code in older Pythons LineProfiler.add_function(), python_trace_callback() Added handling for Cython functions (which has null bytecodes) line_profiler/profiler_mixin.py[i] get_underlying_functions() Now returning Cython callables if not in Python 3.12 is_c_level_callable() No longer returning true for Cython callables

line_profiler/_line_profiler.pyx::LineProfiler.add_function() Now using the new `get_code_block()` line_profiler/line_profiler.py::get_code_block() New convenience function which wraps `linecache.getlines()` and `inspect.getblock()` and understands Cython code

line_profiler/_line_profiler.pyx get_frame_bytecode() Renamed from `get_frame_bydecode()` LineProfiler.add_function() Added early exit for Cython callables in Python 3.12 (because line tracing doesn't work there anyway)

line_profiler/_line_profiler.pyx find_cython_source_file() New helper function for resolving the source filename where a Cython function is defined LineProfiler.add_function() - Now using `find_cython_source_file()` to search for the Cython source file (because `code.co_filename` on its own is hard-coded, relative to the package's `setup.py`, and unreliable) - Now storing the Cython code object with the resolved filename at `.code_hash_map`

tests/cython_example New example Cython package tests/test_cython.py New test module for the profiling of Cython code

line_profiler/_line_profiler.pyx get_frame_bytecode() Removed because it is unused after the refactoring _sys_monitoring_register() - Updated signature - Now responsible for setting the `sys.monitoring` callbacks _sys_monitoring_deregister() Now responsible for unsetting the `sys.monitoring` callbacks LineProfiler add_function() Fixed bug where Cython functions with all-zero-bytes dummy bytecodes are not handled accordingly enable() No longer using the legacy trace system if `sys.monitoring` is available legacy_trace_callback() Refactored from the previous `python_trace_callback()` monitoring_line_event_callback(), monitoring_exit_frame_callback() New functions for building callbacks for `sys.monitoring.register_callback()`

Erotemic · 2025-07-07T14:08:46Z

line_profiler/line_profiler.py

+        lines (list[str])
+            Newline-terminated string lines.
+    """
+    BlockFinder = inspect.BlockFinder


Oooh, I didn't know about BlockFinder. That might be useful in xdoctest.

IIRC it's undocumented but it's been around in inspect since forever, much like the inspect.getblock() function used earlier in show_func(). Since we have to essentially "teach" inspect how to read Cython I guess it isn't too bad a use of "private" API...

That makes me a little worried. I've been burned by using what seemed like stable private APIs before, specifically having to do with tokenize: python/cpython#111224

Fair enough, it did feel a bit dirty/hacky even for me; maybe we should vendor in our own version of BlockFinder and/or getblock(), or just use a simpler algorithm which just looks at indentation levels or something.

But then again to play devil's advocate, they do explicitly mention in the docs that tokenizations are to be considered volatile and version-specific. (Or maybe they added that to the docs in response to that very issue...?) Of course, it's very debatable whether it's worse if something is warned about in the docs or if it isn't even mentioned.

I think I'm leaning towards using it as-is, maybe with a note. If it breaks in a future release of Python, as long as we make it obvious that this is the thing breaking, then we can just vendor in the old code as I did in xdoctest. I think my issue was less about the API itself and more about in what cases the API could be used, so maybe this is safer.

Good idea, I'll add some docs.

A minor correction: I just noticed that getblock() and BlockFinder are actually in inspect.__all__... so it isn't like these are meant to be private APIs. However, they are not in the online docs, so it's a bit iffy. Also, getblock() has been in use in the repo since Rob wrote fb60664 in 2008. I guess that adds a little more confidence that it's somewhat okay to lean into the APIs.

Erotemic · 2025-07-07T14:09:50Z

line_profiler/line_profiler.py

+        namespace['BlockFinder'] = BlockFinder
+
+
+class _CythonBlockFinder(inspect.BlockFinder):


Is there a test for this? I'd love to see a small doctest here.

I think test_recover_cython_source kinda covers this, but it is not quite a unit test because it also tests the recovery of Cython function filenames. I guess it does make sense to have a smaller unit doctest here, will do when home.

Update: now we have a doctest at line_profiler/line_profiler.py::line_profiler.line_profiler.get_code_block

Erotemic · 2025-07-07T14:12:01Z

line_profiler/profiler_mixin.py

                          types.MethodWrapperType,
                          types.WrapperDescriptorType)

+# Can't line-profile Cython in 3.12


A comment about why this is the case, or a link to why this is the case would be useful.

Erotemic · 2025-07-07T14:23:00Z

line_profiler/_line_profiler.pyx

+        elif CAN_USE_SYS_MONITORING:
            _sys_monitoring_deregister()
+        else:
+            unset_trace()


Because we are moving away from PyEval_SetTrace, would we want to rename this to something like legacy_unset_trace or looking at unset_trace.c/h, could we just inline the PyEval_SetTrace(NULL, NULL); call?

Makes sense. Even if we were to stick with legacy tracing we're still getting rid of this function in #334 (which apropos will of course need some updating after merging this, since it was written with the assumption that we're also interacting with the legacy system), so we might as well axe it now.

line_profiler/_line_profiler.pyx::LineProfiler.disable() Inlined call to `PyEval_SetTrace(NULL, NULL)` line_profiler/CMakeLists.txt, unset_trace.* docs/source/auto/line_profiler.unset_trace.rst Scrubbed references to the now-unused `unset_trace()`

line_profiler/line_profiler.py get_code_block.__doc__ - Added doctest - Added note on use of undocumented `inspect` API _CythonBlockFinder.__doc__ - Fixed malformed RST - Added note on use of undocumented `inspect` API line_profiler/profiler_mixin.py::_CANNOT_LINE_TRACE_CYTHON Updated comment to clarify why Cython code cannot be line-profiled in Python 3.12

Erotemic · 2025-07-08T01:09:50Z

I think this looks ready. If there was any other cleanup you wanted to do, submit it in a new PR.

TTsangSC · 2025-07-08T02:08:01Z

Cheers!

TTsangSC added 8 commits July 7, 2025 01:26

Fixes

b5a546f

line_profiler/_line_profiler.pyx get_frame_bytecode() Renamed from `get_frame_bydecode()` LineProfiler.add_function() Added early exit for Cython callables in Python 3.12 (because line tracing doesn't work there anyway)

Test for profiling Cython code

1543809

tests/cython_example New example Cython package tests/test_cython.py New test module for the profiling of Cython code

CHANGELOG

f03e920

Fixed (?) fixture teardown

757add6

TTsangSC force-pushed the restore-cython-compat branch from 6a144c1 to 757add6 Compare July 6, 2025 23:26

Erotemic reviewed Jul 7, 2025

View reviewed changes

TTsangSC added 3 commits July 7, 2025 21:08

Minor Cython code update

89f3692

line_profiler/_line_profiler.pyx::LineProfiler.disable() Inlined call to `PyEval_SetTrace(NULL, NULL)` line_profiler/CMakeLists.txt, unset_trace.* docs/source/auto/line_profiler.unset_trace.rst Scrubbed references to the now-unused `unset_trace()`

Merge branch 'main' into restore-cython-compat

97ecd5c

Erotemic merged commit 7ffc0f1 into pyutils:main Jul 8, 2025
36 checks passed

TTsangSC mentioned this pull request Jul 8, 2025

FIX: Update hash tables of affected profiler instances when rewriting code objects #351

Merged

TTsangSC deleted the restore-cython-compat branch July 8, 2025 04:34

TTsangSC mentioned this pull request Jul 9, 2025

Restore trace callback when the profiler is disabled #334

Merged

da-woods mentioned this pull request Jul 13, 2025

Bump macos runner version cython/cython#7012

Closed

TTsangSC mentioned this pull request Jul 15, 2025

BUG (?): line data not reported when exception raised #355

Closed

TTsangSC mentioned this pull request Aug 27, 2025

FIX: building on Windows-ARM64 #391

Merged

		namespace['BlockFinder'] = BlockFinder


		class _CythonBlockFinder(inspect.BlockFinder):

FIX: restore Cython compatibility + pivot to sys.monitoring #352

FIX: restore Cython compatibility + pivot to sys.monitoring #352

Uh oh!

Conversation

TTsangSC commented Jun 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Synopsis

Motivations

Performance impacts

Code changes

Test changes

Possible conflicts

Footnotes

Uh oh!

codecov bot commented Jun 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Erotemic Jul 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Erotemic commented Jul 8, 2025

Uh oh!

Uh oh!

TTsangSC commented Jul 8, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

FIX: restore Cython compatibility + pivot to `sys.monitoring` #352

FIX: restore Cython compatibility + pivot to `sys.monitoring` #352

TTsangSC commented Jun 19, 2025 •

edited

Loading

codecov bot commented Jun 19, 2025 •

edited

Loading

Erotemic Jul 7, 2025 •

edited

Loading