From b2b7047d6b3cc417761854dde807773c0584bb1a Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Mon, 30 Jun 2025 19:11:09 -0400 Subject: [PATCH 1/8] Clarify that weak references cannot be promoted postmortem. --- peps/pep-0788.rst | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/peps/pep-0788.rst b/peps/pep-0788.rst index 61d02c0b5f1..e0a0c2b4742 100644 --- a/peps/pep-0788.rst +++ b/peps/pep-0788.rst @@ -379,11 +379,14 @@ Weak References This proposal also comes with weak references to an interpreter that don't prevent it from shutting down, but can be promoted to a strong reference when -the user decides that they want to call the C API. A weak reference will -typically live much longer than a strong reference. This is useful for many of -the asynchronous situations stated previously, where the thread itself -shouldn't prevent the desired interpreter from shutting down, but also allow -the thread to execute Python when needed. +the user decides that they want to call the C API. If an interpreter is +destroyed or past the point where it can create strong references, promotion +of a weak reference will fail. + +A weak reference will typically live much longer than a strong reference. +This is useful for many of the asynchronous situations stated previously, +where the thread itself shouldn't prevent the desired interpreter from shutting +down, but also allow the thread to execute Python when needed. For example, a (non-reentrant) event handler may store a weak interpreter reference in its ``void *arg`` parameter, and then that weak reference will From 5ffd8d6b9d3b56aeb5df7eb3ef86265c0909574a Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Mon, 30 Jun 2025 19:12:38 -0400 Subject: [PATCH 2/8] Get rid of 'plethora'. --- peps/pep-0788.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0788.rst b/peps/pep-0788.rst index e0a0c2b4742..70beed1d166 100644 --- a/peps/pep-0788.rst +++ b/peps/pep-0788.rst @@ -395,7 +395,7 @@ be promoted to a strong reference when it's time to call Python code. Deprecation of the GIL-state APIs --------------------------------- -Due to the plethora of issues with ``PyGILState``, this PEP intends to do away +Due to the unfixable issues with ``PyGILState``, this PEP intends to do away with them entirely. In today's C API, all ``PyGILState`` functions are replaceable with ``PyThreadState`` counterparts that are compatibile with subinterpreters: From db667d580990707ef96ed89db2aa9feb8a5ad261 Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Mon, 30 Jun 2025 19:13:08 -0400 Subject: [PATCH 3/8] Clarify that PyInterpreterWeakRef_AsStrong() does not set an exception. --- peps/pep-0788.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/peps/pep-0788.rst b/peps/pep-0788.rst index 70beed1d166..1065fadfe79 100644 --- a/peps/pep-0788.rst +++ b/peps/pep-0788.rst @@ -538,7 +538,8 @@ Weak Interpreter References reference to the interpreter denoted by *wref*. If the interpreter no longer exists or has already finished waiting - for its reference count to reach zero, then this function returns ``-1``. + for its reference count to reach zero, then this function returns ``-1`` + without an exception set. This function is not safe to call in a re-entrant signal handler. From 0a8c599fc823f751d11e83a1208753c700726603 Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Mon, 30 Jun 2025 19:16:06 -0400 Subject: [PATCH 4/8] Add some clarity to 'A Library Interface' --- peps/pep-0788.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/peps/pep-0788.rst b/peps/pep-0788.rst index 1065fadfe79..f791e3844d3 100644 --- a/peps/pep-0788.rst +++ b/peps/pep-0788.rst @@ -683,8 +683,8 @@ If you were to use :c:func:`PyGILState_Ensure` for this case, then your thread would hang if the interpreter were to be finalizing at that time! Additionally, the API supports subinterpreters. If you were to assume that -the main interpreter created the file object, then your library wouldn't be safe to use -with file objects created by a subinterpreter. +the main interpreter created the file object (via :c:func:`PyGILState_Ensure`), +then using file objects owned by a subinterpreter could possibly crash. Example: A Single-threaded Ensure ********************************* From aef319b757369d833aa09da6aa4cf5945a936eaa Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Mon, 30 Jun 2025 19:47:10 -0400 Subject: [PATCH 5/8] Add a porting guide. --- peps/pep-0788.rst | 54 +++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 52 insertions(+), 2 deletions(-) diff --git a/peps/pep-0788.rst b/peps/pep-0788.rst index f791e3844d3..fa55e61c5dc 100644 --- a/peps/pep-0788.rst +++ b/peps/pep-0788.rst @@ -392,8 +392,8 @@ For example, a (non-reentrant) event handler may store a weak interpreter reference in its ``void *arg`` parameter, and then that weak reference will be promoted to a strong reference when it's time to call Python code. -Deprecation of the GIL-state APIs ---------------------------------- +Removing the outdated GIL-state APIs +------------------------------------ Due to the unfixable issues with ``PyGILState``, this PEP intends to do away with them entirely. In today's C API, all ``PyGILState`` functions are @@ -405,6 +405,8 @@ subinterpreters: - :c:func:`PyGILState_GetThisThreadState`: :c:func:`PyThreadState_Get` (roughly) - :c:func:`PyGILState_Check`: ``PyThreadState_GetUnchecked() != NULL`` +(See :ref:`pep-788-porting-guide` for more information.) + This PEP specifies a deprecation for these functions (while remaining in the stable ABI), because :c:func:`PyThreadState_Ensure` and :c:func:`PyThreadState_Release` will act as more-correct replacements for @@ -635,6 +637,54 @@ in the C API documentation, ideally under the :ref:`python:gilstate` section. The existing ``PyGILState`` documentation should be updated accordingly to point to the new APIs. +.. _pep-788-porting-guide: + +Porting Guide +------------- + +Ensuring Python Can Be Called +***************************** + +In all places where :c:func:`PyGILState_Ensure` and :c:func:`PyGILState_Release` +are used, provide access to a strong or weak interpreter reference in the code +path. If this is too difficult or impossible (such as in a public library interface +that uses ``PyGILState`` under the hood), consider using :c:func:`PyInterpreterRef_Main` +to get a reference for the main interpreter. Then, replace the legacy calls with +:c:func:`PyThreadState_Ensure` and :c:func:`PyThreadState_Release`, acquiring +and releasing interpreter references where necessary. A strong interpreter +reference should typically be released after calling :c:func:`PyThreadState_Release`. + +Thread States +************* + +There are two ``PyGILState`` APIs that act as an abstraction for +:term:`thread states `. For some background, Python stores +an internal thread-local pointer known as the "gilstate". The gilstate points +to the thread state used by :c:func:`PyGILState_Ensure`, or more broadly, the +last thread state used by the thread (that is not destroyed). +:c:func:`PyGILState_GetThisThreadState` returns the gilstate pointer, *not* +the :term:`attached thread state`. The gilstate is not something that should +be accessed by users of the C API; instead, use :c:func:`PyThreadState_Get` +(or :c:func:`PyThreadState_GetUnchecked` if possibly expecting ``NULL``). + +The other function, :c:func:`PyGILState_Check`, is an older function for +checking if the calling thread holds the :term:`GIL`. In newer versions, +:c:func:`PyGILState_Check` simply checks if the gilstate matches the +attached thread state (which is always true if the attached thread state +is non-``NULL``). However, :c:func:`PyGILState_Check` always returns ``1`` if +a subinterpreter was ever created in the current Python process. As such, +:c:func:`PyGILState_Check` is generally not a great API to use. Instead, +check if ``PyThreadState_GetUnchecked()`` returns ``NULL`` to determine +if the thread has an attached thread state. + +As a band-aid solution, these macros can be added to avoid rewriting +code: + +.. code-block:: c + + #define PyGILState_GetThisThreadState PyThreadState_GetUnchecked + #define PyGILState_Check() (PyThreadState_GetUnchecked() != NULL) + Examples -------- From 095f943566e68682685a7a9b85f01f817799b78a Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Mon, 30 Jun 2025 19:48:30 -0400 Subject: [PATCH 6/8] Clarify that PyInterpreterWeakRef will always be pointer-sized. --- peps/pep-0788.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/peps/pep-0788.rst b/peps/pep-0788.rst index fa55e61c5dc..5fbe670d48a 100644 --- a/peps/pep-0788.rst +++ b/peps/pep-0788.rst @@ -512,6 +512,8 @@ Weak Interpreter References The interpreter will *not* wait for the reference to be released before shutting down. + This type is guaranteed to be pointer-sized. + .. c:function:: int PyInterpreterWeakRef_Get(PyInterpreterWeakRef *wref) Acquire a weak reference to the current interpreter. From a6b61fb1831a33c607cbc326ef038c582ba1dd4a Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Mon, 30 Jun 2025 19:59:15 -0400 Subject: [PATCH 7/8] Add quote links to 'Daemon Threads Are Not the Problem' --- peps/pep-0788.rst | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/peps/pep-0788.rst b/peps/pep-0788.rst index 5fbe670d48a..f529c7885b7 100644 --- a/peps/pep-0788.rst +++ b/peps/pep-0788.rst @@ -181,7 +181,8 @@ Prior to this PEP, deprecating daemon threads was discussed `extensively `_. Daemon threads technically cause many of the issues outlined in this proposal, so removing daemon threads could be seen as a potential solution. The main argument for removing daemon -threads is that they're a large cause of problems in the interpreter: +threads is that they're a large cause of problems in the interpreter +`[1] `_. Except that daemon threads don’t actually work reliably. They’re attempting to run and use Python interpreter resources after the runtime has been shut @@ -190,7 +191,8 @@ threads is that they're a large cause of problems in the interpreter: However, in practice, daemon threads are useful for simplifying many threading applications in Python, and since the program is about to close in most cases, -it's not worth the added complexity to try and gracefully shut down a thread. +it's not worth the added complexity to try and gracefully shut down a thread +`[2] `_. When I’ve needed daemon threads, it’s usually been the case of “Long-running, uninterruptible, third-party task” in terms of the examples in the linked issue. @@ -205,7 +207,8 @@ As noted by this PEP, extension modules are free to create their own threads and attach thread states for them. Similar to daemon threads, Python doesn't try and join them during finalization, so trying to remove daemon threads as a whole would involve trying to remove them from the C API, which would -require a much more massive API change. +require a much more massive API change than what is currently being proposed +`[3] `_. Realize however that even if we get rid of daemon threads, extension module code can and does spawn its own threads that are not tracked by From ac13e7a643a72ba2d283cddb653d5ec3ed8b4d25 Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Tue, 1 Jul 2025 09:00:17 -0400 Subject: [PATCH 8/8] Remove the porting guide. --- peps/pep-0788.rst | 50 ----------------------------------------------- 1 file changed, 50 deletions(-) diff --git a/peps/pep-0788.rst b/peps/pep-0788.rst index f529c7885b7..41cc45f1a8c 100644 --- a/peps/pep-0788.rst +++ b/peps/pep-0788.rst @@ -408,8 +408,6 @@ subinterpreters: - :c:func:`PyGILState_GetThisThreadState`: :c:func:`PyThreadState_Get` (roughly) - :c:func:`PyGILState_Check`: ``PyThreadState_GetUnchecked() != NULL`` -(See :ref:`pep-788-porting-guide` for more information.) - This PEP specifies a deprecation for these functions (while remaining in the stable ABI), because :c:func:`PyThreadState_Ensure` and :c:func:`PyThreadState_Release` will act as more-correct replacements for @@ -642,54 +640,6 @@ in the C API documentation, ideally under the :ref:`python:gilstate` section. The existing ``PyGILState`` documentation should be updated accordingly to point to the new APIs. -.. _pep-788-porting-guide: - -Porting Guide -------------- - -Ensuring Python Can Be Called -***************************** - -In all places where :c:func:`PyGILState_Ensure` and :c:func:`PyGILState_Release` -are used, provide access to a strong or weak interpreter reference in the code -path. If this is too difficult or impossible (such as in a public library interface -that uses ``PyGILState`` under the hood), consider using :c:func:`PyInterpreterRef_Main` -to get a reference for the main interpreter. Then, replace the legacy calls with -:c:func:`PyThreadState_Ensure` and :c:func:`PyThreadState_Release`, acquiring -and releasing interpreter references where necessary. A strong interpreter -reference should typically be released after calling :c:func:`PyThreadState_Release`. - -Thread States -************* - -There are two ``PyGILState`` APIs that act as an abstraction for -:term:`thread states `. For some background, Python stores -an internal thread-local pointer known as the "gilstate". The gilstate points -to the thread state used by :c:func:`PyGILState_Ensure`, or more broadly, the -last thread state used by the thread (that is not destroyed). -:c:func:`PyGILState_GetThisThreadState` returns the gilstate pointer, *not* -the :term:`attached thread state`. The gilstate is not something that should -be accessed by users of the C API; instead, use :c:func:`PyThreadState_Get` -(or :c:func:`PyThreadState_GetUnchecked` if possibly expecting ``NULL``). - -The other function, :c:func:`PyGILState_Check`, is an older function for -checking if the calling thread holds the :term:`GIL`. In newer versions, -:c:func:`PyGILState_Check` simply checks if the gilstate matches the -attached thread state (which is always true if the attached thread state -is non-``NULL``). However, :c:func:`PyGILState_Check` always returns ``1`` if -a subinterpreter was ever created in the current Python process. As such, -:c:func:`PyGILState_Check` is generally not a great API to use. Instead, -check if ``PyThreadState_GetUnchecked()`` returns ``NULL`` to determine -if the thread has an attached thread state. - -As a band-aid solution, these macros can be added to avoid rewriting -code: - -.. code-block:: c - - #define PyGILState_GetThisThreadState PyThreadState_GetUnchecked - #define PyGILState_Check() (PyThreadState_GetUnchecked() != NULL) - Examples --------