From 5d207a687a8038ca5904a45107ea156074a5a1da Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Fri, 28 Mar 2025 12:06:33 +0100 Subject: [PATCH 1/7] PEP 776: Remove all discussion of packaging To be split into its own pep per pfmoore's instructions --- peps/pep-0776.rst | 157 ++++------------------------------------------ 1 file changed, 12 insertions(+), 145 deletions(-) diff --git a/peps/pep-0776.rst b/peps/pep-0776.rst index 0d4450d1d14..9261e7705f4 100644 --- a/peps/pep-0776.rst +++ b/peps/pep-0776.rst @@ -5,7 +5,6 @@ Sponsor: Łukasz Langa Discussions-To: https://discuss.python.org/t/84996 Status: Draft Type: Standards Track -Topic: Packaging Created: 18-Mar-2025 Python-Version: 3.14 Post-History: `18-Mar-2025 `__ @@ -20,8 +19,17 @@ also maintains an Emscripten target. This PEP formalizes the addition of Tier 3 for Emscripten support in Python 3.14 which `was approved by the Steering Council on October 25, 2024 -`__. The goal is to allow -Pyodide wheels to be uploaded to PyPI. +`__. The goals are: + +1. To describe the current state of the CPython Emscripten runtime +2. To describe the current state of the Pyodide runtime +3. To identify minor features to be upstreamed from the Pyodide runtime into the + CPython Emscripten runtime + +The minor features identified here are all features that could be implemented +without a PEP. We discuss more significant runtime features that we would like +to implement but we defer decisions on those features to subsequent PEPs. + Motivation ========== @@ -41,33 +49,6 @@ Emscripten platform. Emscripten and WASI are also the only supported platforms that offer any meaningful sandboxing. -Goals -===== - -It is our long term goal to upstream the entire Pyodide runtime into CPython, -but this is out of scope for the present PEP. This PEP only attempts to -establish the foundation for future work. - -Runtime Goals -------------- - -1. To describe the current state of the CPython Emscripten runtime -2. To describe the current state of the Pyodide runtime -3. To identify minor features to be upstreamed from the Pyodide runtime into the - CPython Emscripten runtime - -The minor features identified here are all features that could be implemented -without a PEP. We discuss more significant runtime features that we would like -to implement but we defer decisions on those features to subsequent PEPs. - -Packaging Goals ---------------- - -1. To describe Pyodide's packaging tooling -2. To describe Pyodide's wheel abi to a similar level of detail as the manylinux - PEPs -3. For Pyodide wheels to be allowed for upload to PyPI - Emscripten Platform Information =============================== @@ -520,7 +501,7 @@ The following modules can be imported, but are not functional: as well as any functionality that requires these. The following are present but cannot be imported due to a dependency on the -termios package which has been removed: +termios module which has been removed: - pty - tty @@ -638,110 +619,6 @@ buildbots, maintained by Russell Keith-Magee. CPython does not currently test Tier 3 platforms on GitHub Actions, but if this ever changes, their Linux runners are able to build and test Emscripten Python. -Packaging ---------- - -Existing Package Support -~~~~~~~~~~~~~~~~~~~~~~~~ - -Pyodide currently maintains ports of 255 different packages at the time of this -writing, including major scientific Python packages like NumPy, SciPy, pandas, -Polars, scikit-learn, OpenCV, PyArrow, and Pillow as well as general purpose -packages like aiohttp, Requests, Pydantic, cryptography, and orjson. - -About 60 packages are also testing against Pyodide in their CI, including NumPy, -pandas, awkward-cpp, scikit-image, statsmodels, PyArrow, Hypothesis, and PyO3. - -Pyodide Wheel Tags -~~~~~~~~~~~~~~~~~~ - -Pyodide wheels will use ``pyodide__wasm32`` as the platform tag. For -example, ``pyodide_2025_0_wasm32``. - -With a fixed version of Emscripten, it is possible to link dynamic libraries -that require a large number of distinct ABIs, depending on ABI-sensitive linker -options and what versions of what static libraries are linked. It is our intent -that the ``pyodide_2025_0`` specifies the particular ABI that will work with the -Pyodide CPython runtime. - -For example, wheels with the following tags are compatible with Python 3.13 -Pyodide: - -- ``cp13-cp13-pyodide_2025_0_wasm32`` -- ``abi3-cp10-pyodide_2025_0_wasm32`` - -As well as all non-platformed tags. To generate the list of compatible tags, one -can use the following code:: - - - from packaging.tags import cpython_tags, _generic_platforms - - def _emscripten_platforms() -> Iterator[str]: - pyodide_abi_version = sysconfig.get_config_var("PYODIDE_ABI_VERSION") - if pyodide_abi_version: - yield f"pyodide_{pyodide_abi_version}_wasm32" - yield from _generic_platforms() - - emscripten_tags = cpython_tags(platforms=_emscripten_platforms()) - -This code will be added to `pypa/packaging -`__. - -Emscripten Wheel ABI -~~~~~~~~~~~~~~~~~~~~ - -The specification of the ``pyodide_`` ABI includes: - -* Which version of the Emscripten compiler is used -* What libraries are statically linked with the interpreter -* What stack unwinding ABI is to be used -* Which runtime platform features are required to be present - -and a handful of other similar details that affect the ABI. - -The ABI is selected by choosing the appropriate version of the Emscripten -compiler and passing appropriate compiler and linker flags. It is possible for -other people to build their own Python interpreter that is compatible with the -Pyodide ABI, it is not necessary to use the Pyodide distribution itself. - -The ``pyodide build`` tool knows how to create wheels that match our ABI. Unlike -with manylinux wheels, there is no need for a Docker container to build the -``pyodide_`` wheels. All that is needed is a Linux machine and appropriate -versions of Python, Node.js, and Emscripten. - - -Package Installers -~~~~~~~~~~~~~~~~~~ - -Installers should use the ``_emscripten_platforms()`` function shown above to -determine which platforms are compatible with an Emscripten build of CPython. In -particular, the Pyodide ABI version is exposed via -``sysconfig.get_config_var("PYODIDE_ABI_VERSION")``. - - -Package indexes -~~~~~~~~~~~~~~~ - -We recommend that package indexes accept any wheel whose platform tag matches -``pyodide_[0-9]+_[0-9]+_wasm32``. We recommend that package indexes continue not -accepting wheels that match ``emscripten_[0-9]+_[0-9]+_[0-9]+_wasm32``. - - -Dependency Specifier Markers -~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -To check for the Emscripten platform in a dependency specifier, one can use -``sys_platform == 'emscripten'`` (or its negation). Such checks are already in use -in the wild and seem to be sufficient for the needs of the community. - - -Trove Classifier -~~~~~~~~~~~~~~~~ - -Packages that build and test Emscripten wheels can declare this by adding the -``Environment :: WebAssembly :: Emscripten``. PyPI already accepts uploads of -packages with this classifier. - PEP 11 ------ @@ -836,10 +713,6 @@ care to minimize backwards incompatibility. We will also need a way to disable partially-upstreamed features so that Pyodide can replace them with more complete versions downstream. -These backwards compatibility concerns impact not just the runtime but also the -packaging system. Adding new platform tags should not affect existing packaging -tools because tools ignore wheels with an unknown package tag. - Security Implications ===================== @@ -861,12 +734,6 @@ to use them all at runtime. The documentation will cover this in a similar form to the existing Windows embeddable package. In the short term, we will encourage developers to use Pyodide if at all possible. -Second, maintainers of packages with binary components need to know how to -build, test, label, and deploy them for Emscripten (see Packaging). The Pyodide -documentation on `building and testing packages -`__ -is the best reference for this. - Reference Implementation ======================== From 172848d812f23ed91b2a7b46406503bc7cf066ac Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Fri, 28 Mar 2025 12:13:21 +0100 Subject: [PATCH 2/7] Remove Emscripten ABI compatibility section too --- peps/pep-0776.rst | 24 ------------------------ 1 file changed, 24 deletions(-) diff --git a/peps/pep-0776.rst b/peps/pep-0776.rst index 9261e7705f4..848233820fe 100644 --- a/peps/pep-0776.rst +++ b/peps/pep-0776.rst @@ -123,30 +123,6 @@ Because of these limitations, Pyodide standardizes a no-pthreads build of Python. If there is sufficient demand, a pthreads build with no dynamic loader could be added later. -Emscripten ABI Compatibility ----------------------------- - -The Emscripten compiler has no ABI stability guarantees between versions. Many -Emscripten updates are ABI compatible by chance, and the Rust Emscripten target -behaves as if the ABI were stable with only `occasional negative consequences -`__. - -There are several linker flags that adjust the Emscripten ABI, so Python -packages built to run with Emscripten must make sure to match the ABI-sensitive -linker flags used to compile the interpreter to avoid load-time or run-time -errors. The Emscripten compiler continuously fixes bugs and adds support for new -web platform features. Thus, there is significant benefit to being able to -update the ABI. - -In order to balance the ABI stability needs of package maintainers with the ABI -flexibility to allow the platform to move forward, Pyodide plans to adopt a new -ABI for each feature release of Python. - -The Pyodide team also coordinates the ABI flags that Pyodide uses with the -Emscripten ABI that Rust supports in order to ensure that we have support for -the many popular Rust packages. Historically, most of the work for this has -been related to unwinding ABIs. See for instance `this Rust Major Change -Proposal `__. Development Tools ----------------- From 9629ab51c1072901c28c571cc1cd21e108cd9b50 Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Fri, 28 Mar 2025 12:19:45 +0100 Subject: [PATCH 3/7] Add new discussion thread --- peps/pep-0776.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/peps/pep-0776.rst b/peps/pep-0776.rst index 848233820fe..dac59e8c957 100644 --- a/peps/pep-0776.rst +++ b/peps/pep-0776.rst @@ -2,12 +2,12 @@ PEP: 776 Title: Emscripten Support Author: Hood Chatham Sponsor: Łukasz Langa -Discussions-To: https://discuss.python.org/t/84996 +Discussions-To: https://discuss.python.org/t/86276 Status: Draft Type: Standards Track Created: 18-Mar-2025 Python-Version: 3.14 -Post-History: `18-Mar-2025 `__ +Post-History: `28-Mar-2025 `__ ``18-Mar-2025 `__ Abstract ======== From 962b17756009179ef08a7f90e6d1f81c6dcb6313 Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Fri, 28 Mar 2025 15:45:06 +0100 Subject: [PATCH 4/7] Try to fix CI --- peps/pep-0776.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0776.rst b/peps/pep-0776.rst index dac59e8c957..41642a070e4 100644 --- a/peps/pep-0776.rst +++ b/peps/pep-0776.rst @@ -7,7 +7,7 @@ Status: Draft Type: Standards Track Created: 18-Mar-2025 Python-Version: 3.14 -Post-History: `28-Mar-2025 `__ ``18-Mar-2025 `__ +Post-History: `28-Mar-2025 `__ `18-Mar-2025 `__ Abstract ======== From efd3d711863bdf4f30e34c85df673ce24259c4ad Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Fri, 28 Mar 2025 15:46:15 +0100 Subject: [PATCH 5/7] Try again to fix header validation --- peps/pep-0776.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0776.rst b/peps/pep-0776.rst index 41642a070e4..8f5b6709d25 100644 --- a/peps/pep-0776.rst +++ b/peps/pep-0776.rst @@ -7,7 +7,7 @@ Status: Draft Type: Standards Track Created: 18-Mar-2025 Python-Version: 3.14 -Post-History: `28-Mar-2025 `__ `18-Mar-2025 `__ +Post-History: `28-Mar-2025 `__, `18-Mar-2025 `__ Abstract ======== From ba7882230d1c24f094994034ef2c9c75f83eac48 Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Fri, 28 Mar 2025 18:01:43 +0100 Subject: [PATCH 6/7] line break in post history Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> --- peps/pep-0776.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/peps/pep-0776.rst b/peps/pep-0776.rst index 8f5b6709d25..d2cf50531f6 100644 --- a/peps/pep-0776.rst +++ b/peps/pep-0776.rst @@ -7,7 +7,8 @@ Status: Draft Type: Standards Track Created: 18-Mar-2025 Python-Version: 3.14 -Post-History: `28-Mar-2025 `__, `18-Mar-2025 `__ +Post-History: `18-Mar-2025 `__, + `28-Mar-2025 `__, Abstract ======== From 34138df323c58e6da13af012a9c7a26821d42df5 Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Fri, 28 Mar 2025 18:11:09 +0100 Subject: [PATCH 7/7] Change type to informational Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> --- peps/pep-0776.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0776.rst b/peps/pep-0776.rst index d2cf50531f6..4a382183242 100644 --- a/peps/pep-0776.rst +++ b/peps/pep-0776.rst @@ -4,7 +4,7 @@ Author: Hood Chatham Sponsor: Łukasz Langa Discussions-To: https://discuss.python.org/t/86276 Status: Draft -Type: Standards Track +Type: Informational Created: 18-Mar-2025 Python-Version: 3.14 Post-History: `18-Mar-2025 `__,