Soft-deprecate the old API, and make PyType_GetModuleByDef take token

Author: encukou

peps/pep-0387.rst

@@ -114,6 +114,8 @@ Basic policy for backwards compatibility
   platforms).
 
 
+.. _pep387-soft-deprecation:
+
 Soft Deprecation
 ================
 

peps/pep-0793.rst

@@ -26,6 +26,10 @@ To make this viable, we also specify new module slot types to replace
 
 We also add an API for defining modules from slots dynamically.
 
+The existing API (``PyInit_*``) is soft-deprecated.
+(That is: it will continue to work without warnings, and it'll fe fully
+documented and supported, but we plan to not add any new features to it.)
+
 
 Background & Motivation
 =======================
@@ -147,6 +151,27 @@ Unlike types, the import mechanism often has a pointer that's known to be
 suitable as a token value; in these cases it can provide a default token.
 Thus, module tokens do not need a variant of the inelegant ``Py_TP_USE_SPEC``.
 
+To help extensions that straddle Python versions, ``PyModuleDef`` addresses
+are used as default tokens, and where it's reasonable, they are made
+interchangeable with tokens.
+
+
+Soft-deprecating the existing export hook
+-----------------------------------------
+
+The only reason for authors of *existing* extensions to switch to the
+API proposed here is that it allows a single module for both free-threaded
+and non-free-threaded builds.
+It is important that Python *allows* that, but for many existing modules,
+it is nowhere near worth losing compatibility with 3.14 and lower versions.
+
+It is much too early to plan deprecation of the old API.
+
+Instead, this PEP proposes to stop adding new features to the ``PyInit_*``
+scheme.
+After all, the perfect time for extension authors to switch is when they want
+to modify module initialization anyway.
+
 
 Specification
 =============
@@ -264,6 +289,10 @@ If specified, using a new ``Py_mod_token`` slot, the module token must:
 
 (Typically, it should point to a static constant.)
 
+When the address of a ``PyModuleDef`` is used as as a module's token,
+the module should behave as if it was created from that ``PyModuleDef``.
+In particular, the module state must have matching layout and semantics.
+
 Modules created using the ``PyModule_FromSlotsAndSpec`` or the
 ``PyModExport_<NAME>`` export hook can use a new ``Py_mod_token`` slot
 to set the token.
@@ -288,8 +317,15 @@ will return 0 on success and -1 on failure:
     int PyModule_GetToken(PyObject *, void **token_p)
 
 A new ``PyType_GetModuleByToken`` function will be added, with a signature
-like ``PyType_GetModuleByDef`` but a ``void *token`` argument,
-and the same behaviour except matching tokens, rather than only defs.
+like the existing ``PyType_GetModuleByDef`` but a ``void *token`` argument,
+and the same behaviour except matching tokens rather than only defs.
+
+For easier backwards compatibility, the existing ``PyType_GetModuleByDef``
+will be changed to work exactly like ``PyType_GetModuleByToken`` -- that is,
+it will allow a token (cast to a ``PyModuleDef *`` pointer) as the
+*def* argument.
+(The ``PyModule_GetDef`` function will not get a similar change, as users may
+access members of its result.)
 
 
 New slots
@@ -333,6 +369,14 @@ via a pointer; the function will return 0 on success and -1 on failure:
     int PyModule_GetStateSize(PyObject *, Py_ssize_t *result);
 
 
+Soft-deprecating the existing export hook
+-----------------------------------------
+
+The ``PyInit_*`` export hook will be
+:ref:`soft-deprecated <pep387-soft-deprecation>`.
+
+
+
 .. _pep793-api-summary:
 
 New API summary
@@ -383,9 +427,14 @@ If an existing module is ported to use the new mechanism, then
 We claim that how a module was defined is an implementation detail of that
 module, so this should not be considered a breaking change.
 
-Similarly, ``PyType_GetModuleByDef`` will not match modules that are not
-defined using a *def*.
-The new ``PyType_GetModuleByToken`` function may be used instead.
+Similarly, the ``PyType_GetModuleByDef`` function may stop matching modules
+whose definition changed. Module authors may avoid this by explicitly
+explicitly setting a *def* as the *token*.
+
+``PyType_GetModuleByDef`` will now accept a module token as the *def* argument.
+We specify a suitable restriction on using ``PyModuleDef`` addresses as tokens,
+and non-``PyModuleDef`` pointers were previously invalid input,
+so this is not a backwards-compatibility issue.
 
 The ``Py_mod_create`` function may now be called with ``NULL`` for the second
 argument.
@@ -428,11 +477,15 @@ wrappers, the :ref:`pep793-shim` below may be more useful.
      Later in this guide, you'll set the token to *be* the existing
      ``PyModuleDef`` structure.
 
-#. Scan your code for uses of ``PyType_GetModuleByDef``, and replace them by
-   ``PyType_GetModuleByToken``.
+#. Optionally, scan your code for uses of ``PyType_GetModuleByDef``,
+   and replace them by ``PyType_GetModuleByToken``.
    Later in this guide, you'll set the token to *be* the existing
    ``PyModuleDef`` structure.
 
+   (You may skip this step if targetting Python versions that don't expose
+   ``PyType_GetModuleByToken``, since ``PyType_GetModuleByDef`` is
+   backwards-compatible. )
+
 #. Look at the function identified by ``Py_mod_create``, if any.
    Make sure that it does not use its second argument (``PyModuleDef``),
    as it will be called with ``NULL``.
@@ -467,18 +520,17 @@ wrappers, the :ref:`pep793-shim` below may be more useful.
     };
 
 #. If you switched from ``PyModule_GetDef`` to ``PyModule_GetToken``,
-   and/or from ``PyType_GetModuleByDef`` to ``PyType_GetModuleByToken``,
+   and/or if you use ``PyType_GetModuleByDef`` or ``PyType_GetModuleByToken``,
    add a ``Py_mod_token`` slot pointing to the existing ``PyModuleDef`` struct:
 
    .. code-block:: c
 
     static PyModuleDef_Slot module_slots[] = {
         // ... (keep existing slots here)
-        {Py_mod_token, your_module_def},
+        {Py_mod_token, &your_module_def},
         {0}
     };
 
-
 #. Add a new export hook.
 
    .. code-block:: c
@@ -492,31 +544,43 @@ wrappers, the :ref:`pep793-shim` below may be more useful.
     }
 
 The new export hook will be used on Python 3.15 and above.
-Once your module no longer supports lower versions, delete the ``PyInit_``
-function and any unused data.
+Once your module no longer supports lower versions:
 
+#. Delete the ``PyInit_`` function.
+
+#. If the existing ``PyModuleDef`` struct is used *only* for ``Py_mod_token``
+   and/or ``PyType_GetModuleByToken``, you may remove the ``Py_mod_token``
+   line and replace ``&your_module_def`` by ``module_slots`` everywhere else.
+
+#. Delete any unused data.
+   The ``PyModuleDef`` struct and the original slots array are likely to be
+   unused.
+
+
+.. _pep793-shim:
 
 Backwards compatibility shim
 ----------------------------
 
-It is possible to write generic function that implements the existing export
+It is possible to write generic function that implements the “old” export
 hook (``PyInit_``) in terms of the API proposed here.
 
-The following implemntation can be copy-pasted; only the names
-``PyInit_examplemodule`` (twice) and ``PyModExport_examplemodule`` need adjusting.
+The following implementation can be copied and pasted to a project; only the
+names ``PyInit_examplemodule`` (twice) and ``PyModExport_examplemodule`` should
+need adjusting.
 
 When added to the :ref:`pep793-example` below and compiled with a
 non-free-threaded build of this PEP's reference implementation, the resulting
-extension is compatible with regular builds 3.9+ in addition to a
+extension is compatible with non-free-threading 3.9+ builds, in addition to a
 free-threading build of the reference implementation.
 (The module must be named without a version tag, e.g. ``examplemodule.so``,
 and be placed on ``sys.path``.)
 
 Full support for creating such modules will require backports of some new
 API, and support in build/install tools. This is out of scope of this PEP.
-(In particular, the demo “cheats” uses a subset Limited API 3.15 that
-*happens to work* on 3.9; it *should* use Limited API 3.9 with backport
-shims for new API like ``Py_mod_name``.)
+(In particular, the demo “cheats” by using a subset of Limited API 3.15 that
+*happens to work* on 3.9; a proper implementation would use Limited API 3.9
+with backport shims for new API like ``Py_mod_name``.)
 
 This implementation places a few additional requirements on the slots array:
 
@@ -589,6 +653,7 @@ Possible Future Directions
 
 These ideas are out of scope for *this* proposal.
 
+
 Improving slots in general
 --------------------------