Commit ff4959b for pdb colorization GH-113081 broke some pdb backtraces

[pablogsal]

Bug report

Bug description:

If you put a breakpoint here:

diff --git a/Lib/tokenize.py b/Lib/tokenize.py
index 117b485b934..9d215ea4ea1 100644
--- a/Lib/tokenize.py
+++ b/Lib/tokenize.py
@@ -574,6 +574,7 @@ def _transform_msg(msg):

 def _generate_tokens_from_c_tokenizer(source, encoding=None, extra_tokens=False):
     """Tokenize a source reading Python code as unicode strings using the internal C tokenizer"""
+    breakpoint()
     if encoding is None:
         it = _tokenize.TokenizerIter(source, extra_tokens=extra_tokens)
     else:

and run this:

./python.exe  -m test test_tokenize -m test_random_files -uall -v
❯ ./python.exe  -m test test_tokenize -m test_random_files -uall -v
== CPython 3.14.0a7+ (bisect/good-2bbcaedb75942389dacb51866948f40de5951c9c-4-g08d7687094c-dirty:08d768) [Clang 17.0.0 (clang-1700.6.3.2)]
== macOS-26.2-arm64-arm-64bit-Mach-O little-endian
== Python build: release
== cwd: /Users/pablogsal/github/python/3.14/build/test_python_worker_11047æ
== CPU count: 10
== encodings: locale=UTF-8 FS=utf-8
== resources: all

Using random seed: 1175774464
Raised RLIMIT_NOFILE: 256 -> 1024
0:00:00 load avg: 4.01 Run 1 test sequentially in a single process
0:00:00 load avg: 4.01 [1/1] test_tokenize
> /Users/pablogsal/github/python/3.14/Lib/tokenize.py(577)_generate_tokens_from_c_tokenizer()
-> breakpoint()
(Pdb) c
test_random_files (test.test_tokenize.TestRoundtrip.test_random_files) ... > /Users/pablogsal/github/python/3.14/Lib/tokenize.py(577)_generate_tokens_from_c_tokenizer()
-> breakpoint()
(Pdb)
> /Users/pablogsal/github/python/3.14/Lib/tokenize.py(577)_generate_tokens_from_c_tokenizer()
-> breakpoint()
(Pdb) bt
(Pdb)

pdb cannot show the stack. Bisecting points to

ff4959b is the first bad commit (for #113081 / #133355)
commit ff4959b (HEAD)
Author: Tian Gao gaogaotiantian@hotmail.com
Date: Mon May 5 00:49:52 2025 -0700

gh-113081: Highlight source code in pdb (#133355)

Doc/library/pdb.rst | 8 +++-
Doc/whatsnew/3.14.rst | 5 ++
Lib/pdb.py | 31 +++++++++---
Lib/test/test_pdb.py | 37 +++++++++++++++
...-18-48-54.gh-issue-113081.JsLJ1X.rst | 1 +
5 files changed, 74 insertions(+), 8 deletions(-)
create mode 100644 Misc/NEWS.d/next/Library/2025-05-03-18-48-54.gh-issue-113081.JsLJ1X.rst

CPython versions tested on:

CPython main branch

Operating systems tested on:

macOS

[gaogaotiantian]

Okay I'm not 100% sure if this is considered a bug. pdb uses some stdlib, tokenize is one of them. If you put a breakpoint() in the source code, there will be some issues because of nested debugging. To be more specific, to colorize the source code, _pyrepl is used and it uses tokenize. If you do a soft breakpoint there (by b tokenize.py:lineno), it should work because tracefunc is not triggered. This is not an end-user issue because users can't edit python source code directly. Do you still consider this a bug and think we should fix it?

[gpshead]

could it make sense for breakpoint() to understand when it has been invoked from stdlib code that the debugger it's about to drop into relies on and disable the fancier pyrepl features? (I don't know what the right way to do this would be... simple heuristic about being a breakpoint in stdlib code or within in a specific set of stdlib modules - maintaining that list wouldn't be too hard, we'd be the only ones using that and could add new things if they ever come up)