diff --git a/peps/pep-0788.rst b/peps/pep-0788.rst index 55ffac8594e..98b068b4f81 100644 --- a/peps/pep-0788.rst +++ b/peps/pep-0788.rst @@ -100,7 +100,7 @@ This means that any non-Python thread may be terminated at any point, which is severely limiting for users who want to do more than just execute Python code in their stream of calls. -``Py_IsFinalizing`` is Insufficient +``Py_IsFinalizing`` Is Insufficient *********************************** The :ref:`docs ` @@ -147,7 +147,7 @@ tried to acquire the lock. This does not apply to many C locks, such as with :data:`sys.stderr`, because Python code cannot be run while the lock is held. This PEP intends to fix this problem for C locks, not Python locks. -Daemon Threads are not the Problem +Daemon Threads Are Not the Problem ********************************** Prior to this PEP, deprecating daemon threads was discussed @@ -188,7 +188,7 @@ require a much more massive API change than what is currently being proposed Python. ... Those are realistically an alternate form of daemon thread ... and those are never going to be forbidden. -Joining the Thread isn't Always a Good Idea +Joining the Thread Isn't Always a Good Idea ******************************************* Even in daemon threads, it's generally *possible* to prevent hanging of @@ -209,6 +209,18 @@ call :c:func:`PyGILState_Ensure` to safely interact with it (for example, by calling it). If the interpreter is finalizing or has shut down, then the thread is hung, disrupting the C++ stream of calls. +The GIL-state APIs Are Buggy and Confusing +------------------------------------------ + +There are currently two public ways for a user to create and attach a +:term:`thread state` for their thread; manual use of :c:func:`PyThreadState_New` +and :c:func:`PyThreadState_Swap`, or the convenient :c:func:`PyGILState_Ensure`. + +The latter, :c:func:`PyGILState_Ensure`, is significantly more common, having +`nearly 3,000 hits `_ in a code +search, whereas :c:func:`PyThreadState_New` has +`less than 400 hits `_. + .. _pep-788-hanging-compat: Finalization Behavior for ``PyGILState_Ensure`` Cannot Change @@ -232,18 +244,6 @@ the thread or emit a fatal error, as noted in For this reason, we can't make any real changes to how :c:func:`PyGILState_Ensure` works during finalization, because it would break existing code. -The GIL-state APIs are Buggy and Confusing ------------------------------------------- - -There are currently two public ways for a user to create and attach a -:term:`thread state` for their thread; manual use of :c:func:`PyThreadState_New` -and :c:func:`PyThreadState_Swap`, or the convenient :c:func:`PyGILState_Ensure`. - -The latter, :c:func:`PyGILState_Ensure`, is significantly more common, having -`nearly 3,000 hits `_ in a code -search, whereas :c:func:`PyThreadState_New` has -`less than 400 hits `_. - ``PyGILState_Ensure`` Generally Crashes During Finalization *********************************************************** @@ -255,7 +255,7 @@ that could be fixed in CPython, but it's definitely worth noting here, because acceptance and implementation of this PEP will likely fix the existing crashes caused by :c:func:`PyGILState_Ensure`. -The Term "GIL" is Tricky for Free-threading +The Term "GIL" Is Tricky for Free-threading ******************************************* A large issue with the term "GIL" in the C API is that it is semantically @@ -325,11 +325,7 @@ in newer versions with the recent acceptance of :pep:`734`. Rationale ========= -So, how do we address all of this? The best way seems to be starting from -scratch and "reimagining" how to create, acquire and attach -:term:`thread states ` in the C API. - -Preventing Interpreter Shutdown with Reference Counting +Preventing Interpreter Shutdown With Reference Counting ------------------------------------------------------- This PEP takes an approach where an interpreter is given a reference count @@ -368,7 +364,7 @@ For example, a (non-reentrant) event handler may store a weak interpreter reference in its ``void *arg`` parameter, and then that weak reference will be promoted to a strong reference when it's time to call Python code. -Removing the outdated GIL-state APIs +Removing the Outdated GIL-state APIs ------------------------------------ Due to the unfixable issues with ``PyGILState``, this PEP intends to do away @@ -391,6 +387,21 @@ The exact details of this deprecation aren't too clear. It's likely that the usual five-year deprecation (as specificed by :pep:`387`) will be too short, so for now, these functions will have no specific removal date. +Compatibility Shim for ``PyGILState_Ensure`` +-------------------------------------------- + +This proposal comes with :c:func:`PyUnstable_GetDefaultInterpreterRef` as a +compatibility hack for some users of :c:func:`PyGILState_Ensure`. It is a +thread-safe way to acquire a strong reference to the main (or "default") +interpreter. + +The main drawback to porting new code to :c:func:`PyThreadState_Ensure` is that +it isn't a drop-in replacement for :c:func:`!PyGILState_Ensure`, as it needs +an interpreter reference argument. In some large applications, refactoring to +use a :c:type:`PyInterpreterRef` everywhere might be tricky; so, this function +acts as a silver bullet for users who explicitly want to disallow support for +subinterpreters. + Specification ============= @@ -405,14 +416,14 @@ around the same time when :class:`threading.Thread` objects are joined, but note that this *is not* the same as joining the thread; the interpreter will only wait until the reference count is zero, and then proceed. After the reference count has reached zero, threads can no longer prevent the -interpreter from shutting down (thus :c:func:`PyInterpreterRef_Get` and -:c:func:`PyInterpreterWeakRef_AsStrong` will fail). +interpreter from shutting down (thus :c:func:`PyInterpreterRef_FromCurrent` and +:c:func:`PyInterpreterWeakRef_Promote` will fail). A weak reference to an interpreter won't prevent it from finalizing, and can be safely accessed after the interpreter no longer supports creating strong references, and even after the interpreter-state has been deleted. Deletion and duplication of the weak reference will always be allowed, but promotion -(:c:func:`PyInterpreterWeakRef_AsStrong`) will always fail after the +(:c:func:`PyInterpreterWeakRef_Promote`) will always fail after the interpreter reaches a point where strong references have been waited on. Strong Interpreter References @@ -421,43 +432,43 @@ Strong Interpreter References .. c:type:: PyInterpreterRef An opaque, strong reference to an interpreter. + The interpreter will wait until a strong reference has been released before shutting down. This type is guaranteed to be pointer-sized. -.. c:function:: int PyInterpreterRef_Get(PyInterpreterRef *ref) +.. c:function:: PyInterpreterRef PyInterpreterRef_FromCurrent(void) Acquire a strong reference to the current interpreter. - On success, this function returns ``0`` and sets *ref* - to a strong reference to the interpreter, and returns ``-1`` - with an exception set on failure. + On success, this function returns a strong reference to the current + interpreter, and returns ``0`` with an exception set on failure. - Failure typically indicates that the interpreter has - already finished waiting on strong references. + Failure typically indicates that the interpreter has already finished + waiting on strong references. The caller must hold an :term:`attached thread state`. -.. c:function:: int PyInterpreterRef_Main(PyInterpreterRef *ref) +.. c:function:: PyInterpreterRef PyUnstable_GetDefaultInterpreterRef(PyInterpreterRef *ref) Acquire a strong reference to the main interpreter. This function only exists for special cases where a specific interpreter can't be saved. Prefer safely acquiring a reference through - :c:func:`PyInterpreterRef_Get` whenever possible. + :c:func:`PyInterpreterRef_FromCurrent` whenever possible. - On success, this function will return ``0`` and set *ref* to a strong - reference, and on failure, this function will return ``-1``. + On success, this function returns a strong reference to the main + interpreter, and returns ``0`` without an exception set on failure. Failure typically indicates that the main interpreter has already finished waiting on its reference count. The caller does not need to hold an :term:`attached thread state`. -.. c:function:: PyInterpreterState *PyInterpreterRef_AsInterpreter(PyInterpreterRef ref) +.. c:function:: PyInterpreterState *PyInterpreterRef_GetInterpreter(PyInterpreterRef ref) - Return the interpreter denoted by *ref*. + Return the :c:type:`PyInterpreterState` pointer denoted by *ref*. This function cannot fail, and the caller doesn't need to hold an :term:`attached thread state`. @@ -466,8 +477,10 @@ Strong Interpreter References Duplicate a strong reference to an interpreter. - This function cannot fail, and the caller doesn't need to hold an - :term:`attached thread state`. + On success, this function returns a strong reference to the interpreter + denoted by *ref*, and returns ``0`` without an exception set on failure. + + The caller does not need to hold an :term:`attached thread state`. .. c:function:: void PyInterpreterRef_Close(PyInterpreterRef ref) @@ -483,21 +496,21 @@ Weak Interpreter References .. c:type:: PyInterpreterWeakRef An opaque, weak reference to an interpreter. + The interpreter will *not* wait for the reference to be released before shutting down. This type is guaranteed to be pointer-sized. -.. c:function:: int PyInterpreterWeakRef_Get(PyInterpreterWeakRef *wref) +.. c:function:: int PyInterpreterWeakRef_FromCurrent(PyInterpreterWeakRef *wref) Acquire a weak reference to the current interpreter. This function is generally meant to be used in tandem with - :c:func:`PyInterpreterWeakRef_AsStrong`. + :c:func:`PyInterpreterWeakRef_Promote`. - On success, this function returns ``0`` and sets *wref* to a - weak reference to the interpreter, and returns ``-1`` with an exception - set on failure. + On success, this function returns a weak reference to the current + interpreter, and returns ``0`` with an exception set on failure. The caller must hold an :term:`attached thread state`. @@ -505,18 +518,23 @@ Weak Interpreter References Duplicate a weak reference to an interpreter. + On success, this function returns a non-zero weak reference to the + interpreter denoted by *wref*, and returns ``0`` without an exception set + on failure. + This function cannot fail, and the caller doesn't need to hold an :term:`attached thread state`. -.. c:function:: int PyInterpreterWeakRef_AsStrong(PyInterpreterWeakRef wref, PyInterpreterRef *ref) +.. c:function:: PyInterpreterRef PyInterpreterWeakRef_Promote(PyInterpreterWeakRef wref) Acquire a strong reference to an interpreter through a weak reference. - On success, this function returns ``0`` and sets *ref* to a strong - reference to the interpreter denoted by *wref*. + On success, this function returns a strong reference to the interpreter + denoted by *wref*. The weak reference is still valid after calling this + function. If the interpreter no longer exists or has already finished waiting - for its reference count to reach zero, then this function returns ``-1`` + for its reference count to reach zero, then this function returns ``0`` without an exception set. This function is not safe to call in a re-entrant signal handler. @@ -530,7 +548,7 @@ Weak Interpreter References This function cannot fail, and the caller doesn't need to hold an :term:`attached thread state`. -Ensuring and Releasing Thread States +Ensuring And Releasing Thread States ------------------------------------ This proposal includes two new high-level threading APIs that intend to @@ -648,8 +666,8 @@ With this PEP, you'd implement it like this: PyObject *file, const char *text) { - PyInterpreterRef ref; - if (PyInterpreterWeakRef_AsStrong(wref, &ref) < 0) { + PyInterpreterRef ref = PyInterpreterWeakRef_Promote(wref); + if (ref == 0) { /* Python interpreter has shut down */ return -1; } @@ -657,7 +675,7 @@ With this PEP, you'd implement it like this: PyThreadRef thread_ref; if (PyThreadState_Ensure(ref, &thread_ref) < 0) { PyInterpreterRef_Close(ref); - puts("Out of memory.\n", stderr); + fputs("Cannot call Python.\n", stderr); return -1; } @@ -693,8 +711,8 @@ held. Any future finalizer that wanted to acquire the lock would be deadlocked! my_critical_operation(PyObject *self, PyObject *unused) { assert(PyThreadState_GetUnchecked() != NULL); - PyInterpreterRef ref; - if (PyInterpreterRef_Get(&ref) < 0) { + PyInterpreterRef ref = PyInterpreterRef_FromCurrent(); + if (ref == 0) { /* Python interpreter has shut down */ return NULL; } @@ -776,8 +794,8 @@ This is the same code, rewritten to use the new functions: PyThread_handle_t handle; PyThead_indent_t indent; - PyInterpreterRef ref; - if (PyInterpreterRef_Get(&ref) < 0) { + PyInterpreterRef ref = PyInterpreterRef_FromCurrent(); + if (ref == 0) { return NULL; } @@ -797,7 +815,8 @@ Example: A Daemon Thread With this PEP, daemon threads are very similar to how non-Python threads work in the C API today. After calling :c:func:`PyThreadState_Ensure`, simply -release the interpreter reference, allowing the interpreter to shut down. +release the interpreter reference to allow the interpreter to shut down (and +hang the current thread forever). .. code-block:: c @@ -826,8 +845,8 @@ release the interpreter reference, allowing the interpreter to shut down. PyThread_handle_t handle; PyThead_indent_t indent; - PyInterpreterRef ref; - if (PyInterpreterRef_Get(&ref) < 0) { + PyInterpreterRef ref = PyInterpreterRef_FromCurrent(); + if (ref == 0) { return NULL; } @@ -856,8 +875,8 @@ deadlock the interpreter if it's not released. { ThreadData *data = (ThreadData *)arg; PyInterpreterWeakRef wref = data->wref; - PyInterpreterRef ref; - if (PyInterpreterWeakRef_AsStrong(wref, &ref) < 0) { + PyInterpreterRef ref = PyInterpreterWeakRef_Promote(wref); + if (ref == 0) { fputs("Python has shut down!\n", stderr); return -1; } @@ -885,8 +904,8 @@ deadlock the interpreter if it's not released. PyErr_NoMemory(); return NULL; } - PyInterpreterWeakRef wref; - if (PyInterpreterWeakRef_Get(&wref) < 0) { + PyInterpreterWeakRef wref = PyInterpreterWeakRef_FromCurrent(); + if (wref == 0) { PyMem_RawFree(tdata); return NULL; } @@ -902,7 +921,7 @@ Example: Calling Python Without a Callback Parameter There are a few cases where callback functions don't take a callback parameter (``void *arg``), so it's impossible to acquire a reference to any specific interpreter. The solution to this problem is to acquire a reference to the main -interpreter through :c:func:`PyInterpreterRef_Main`. +interpreter through :c:func:`PyUnstable_GetDefaultInterpreterRef`. But wait, won't that break with subinterpreters, per :ref:`pep-788-subinterpreters-gilstate`? Fortunately, since the callback has @@ -915,9 +934,9 @@ interpreter here. static void call_python(void) { - PyInterpreterRef ref; - if (PyInterpreterRef_Main(&ref) < 0) { - fputs("Python has shut down!", stderr); + PyInterpreterRef ref = PyUnstable_GetDefaultInterpreterRef(); + if (ref == 0) { + fputs("Python has shut down.", stderr); return; } @@ -940,6 +959,23 @@ Reference Implementation A reference implementation of this PEP can be found at `python/cpython#133110 `_. +Open Issues +=========== + +How Should the APIs Fail? +------------------------- + +There is a bit of disagreement on how the ``PyInterpreter[Weak]Ref`` APIs +should indicate a failure to the caller. There are two competing ideas: + +1. Return -1 to indicate failure, and 0 to indicate success. On success, + functions will assign to a ``PyInterpreter[Weak]Ref`` pointer passed as an + argument. +2. Directly return a ``PyInterpreter[Weak]Ref``, which a value of 0 being + equivalent to ``NULL``, indicating failure. + +Currently, the PEP spells the latter. + Rejected Ideas ============== @@ -1009,13 +1045,13 @@ of requiring less magic: the non-Python thread gets a chance to attach. The problem with using an interpreter ID is that the reference count has to be "invisible"; it must be tracked elsewhere in the interpreter, likely being *more* - complex than :c:func:`PyInterpreterRef_Get`. There's also a lack + complex than :c:func:`PyInterpreterRef_FromCurrent`. There's also a lack of intuition that a standalone integer could have such a thing as a reference count. .. _pep-788-activate-deactivate-instead: -Exposing an ``Activate``/``Deactivate`` API instead of ``Ensure``/``Clear`` +Exposing an ``Activate``/``Deactivate`` API Instead of ``Ensure``/``Clear`` --------------------------------------------------------------------------- In prior discussions of this API, it was