From c6cfe942863974cabc404bf495ce9d86509dcf6f Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Wed, 19 Mar 2025 15:45:18 +0100 Subject: [PATCH 1/7] Address some of pfmoore's comments This attempts to address most of pfmoore's comments here: https://discuss.python.org/t/pep-776-emscripten-support/84996/11 * Clarify that we are speaking of _platform_ tags. * List some examples of full tags that are Pyodide-compatible * Include code to generate compatible tag list * Reference PR to add this code to pypa/packaging and specify that it will be merged. * Add section about dependency specifier markers * Add section on trove classifier * Add note that adding platform tags should have no backwards compatibility implications * Add link to Pyodide's documentation on building, testing, and ci for package maintainers --- peps/pep-0776.rst | 60 +++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 53 insertions(+), 7 deletions(-) diff --git a/peps/pep-0776.rst b/peps/pep-0776.rst index 0cf88b63507..b008cebf8b2 100644 --- a/peps/pep-0776.rst +++ b/peps/pep-0776.rst @@ -635,11 +635,11 @@ packages like aiohttp, Requests, Pydantic, cryptography, and orjson. About 60 packages are also testing against Pyodide in their CI, including NumPy, pandas, awkward-cpp, scikit-image, statsmodels, PyArrow, Hypothesis, and PyO3. -Emscripten Wheel Format -~~~~~~~~~~~~~~~~~~~~~~~ +Emscripten Wheel Tags +~~~~~~~~~~~~~~~~~~~~~ -Emscripten wheels will use either the format ``emscripten__wasm32`` or -``pyodide__wasm32``. For example: +Emscripten wheels will use as the platform tag either +``emscripten__wasm32`` or ``pyodide__wasm32``. For example: * ``emscripten_3_1_58_wasm32`` * ``pyodide_2025_0_wasm32`` @@ -651,6 +651,34 @@ depending on linker and compiler options. It is our intent that the ``pyodide_`` and ``emscripten_`` is intended to be the same as the relationship between ``manylinux`` and ``linux``. +For example, wheels with the following tags are compatible with Python 3.13 +Pyodide: + +- ``cp13-cp13-emscripten_3_1_58_wasm32`` +- ``cp13-cp13-pyodide_2025_0_wasm32`` +- ``abi3-cp10-pyodide_2025_0_wasm32`` + +As well as all non platformed tags. To generate the list of compatible tags, one +can use the following code: + +.. code-block:: Python + + from packaging.tags import cpython_tags, _generic_platforms + + def _emscripten_platforms() -> Iterator[str]: + pyodide_abi_version = sysconfig.get_config_var("PYODIDE_ABI_VERSION") + if pyodide_abi_version: + yield f"pyodide_{pyodide_abi_version}_wasm32" + yield from _generic_platforms() + + emscripten_tags = cpython_tags(platforms=_emscripten_platforms()) + +`This code will be added to pypa/packaging +`__. + +Emscripten Wheel ABI +~~~~~~~~~~~~~~~~~~~~ + The specification of the ``pyodide_`` ABI includes: * Which version of the Emscripten compiler is used @@ -674,6 +702,20 @@ with manylinux, there is no need for a Docker container to build the ``pyodide_`` wheels. All that is needed is a Linux machine and appropriate versions of Python, Node.js, and Emscripten. +Dependency Specifier Markers +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To check for the Emscripten platform in a dependency specifier, one can use +`sys_platform == 'emscripten'` (or its negation). Such checks are already in use +in the wild and seem to be sufficient for the needs of the community. + +Trove Classifier +~~~~~~~~~~~~~~~~ + +Packages that build and test Emscripten wheels can declare this by adding the +``Environment :: WebAssembly :: Emscripten``. PyPI already accepts uploads of +packages with this classifier. + PEP 11 ------ @@ -767,7 +809,8 @@ partially-upstreamed features so that Pyodide can replace them with more complete versions downstream. These backwards compatibility concerns impact not just the runtime but also the -packaging system. +packaging system. Adding new platform tags should not affect existing packaging +tools because tools ignore wheels with an unknown package tag. Security Implications @@ -790,8 +833,11 @@ to use them all at runtime. The documentation will cover this in a similar form to the existing Windows embeddable package. In the short term, we will encourage developers to use Pyodide if at all possible. -Second, developers of packages with binary components need to know how to build -and release them for Emscripten (see Packaging). +Second, maintainers of packages with binary components need to know how to +build, test, label, and deploy them for Emscripten (see Packaging). The Pyodide +documentation on `building and testing packages +`__ +is the best reference for this. Reference Implementation From 7db0e3a2bf24172068f832eb3740063d7f2d93b2 Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Wed, 19 Mar 2025 15:53:39 +0100 Subject: [PATCH 2/7] Use double backticks --- peps/pep-0776.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0776.rst b/peps/pep-0776.rst index b008cebf8b2..190904a70a9 100644 --- a/peps/pep-0776.rst +++ b/peps/pep-0776.rst @@ -706,7 +706,7 @@ Dependency Specifier Markers ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To check for the Emscripten platform in a dependency specifier, one can use -`sys_platform == 'emscripten'` (or its negation). Such checks are already in use +``sys_platform == 'emscripten'`` (or its negation). Such checks are already in use in the wild and seem to be sufficient for the needs of the community. Trove Classifier From c961a6ec8dfd54870b8031de58f2c07a11450616 Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Wed, 19 Mar 2025 16:17:50 +0100 Subject: [PATCH 3/7] Add sections with specific recommendations for package installers and package indexes --- peps/pep-0776.rst | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/peps/pep-0776.rst b/peps/pep-0776.rst index 190904a70a9..d3697d1d557 100644 --- a/peps/pep-0776.rst +++ b/peps/pep-0776.rst @@ -702,6 +702,24 @@ with manylinux, there is no need for a Docker container to build the ``pyodide_`` wheels. All that is needed is a Linux machine and appropriate versions of Python, Node.js, and Emscripten. + +Package Installers +~~~~~~~~~~~~~~~~~~ + +Installers should use the `_emscripten_platforms()` function shown above to +determine which platforms are compatible with an Emscripten build of CPython. In +particular, the Pyodide ABI version is exposed via +`sysconfig.get_config_var("PYODIDE_ABI_VERSION")`. + + +Package indexes +~~~~~~~~~~~~~~~ + +We recommend that package indexes accept any wheel whose platform tag matches +``pyodide_[0-9]+_[0-9]+_wasm32``. We recommend that package indexes continue not +accepting wheels that match ``emscripten_[0-9]+_[0-9]+_[0-9]+_wasm32``. + + Dependency Specifier Markers ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -709,6 +727,7 @@ To check for the Emscripten platform in a dependency specifier, one can use ``sys_platform == 'emscripten'`` (or its negation). Such checks are already in use in the wild and seem to be sufficient for the needs of the community. + Trove Classifier ~~~~~~~~~~~~~~~~ From 2491b0b941add4e7e204cbd91314baf302615e8c Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Wed, 19 Mar 2025 16:18:54 +0100 Subject: [PATCH 4/7] Use double backticks --- peps/pep-0776.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/peps/pep-0776.rst b/peps/pep-0776.rst index d3697d1d557..ca60e39db2b 100644 --- a/peps/pep-0776.rst +++ b/peps/pep-0776.rst @@ -706,10 +706,10 @@ versions of Python, Node.js, and Emscripten. Package Installers ~~~~~~~~~~~~~~~~~~ -Installers should use the `_emscripten_platforms()` function shown above to +Installers should use the ``_emscripten_platforms()`` function shown above to determine which platforms are compatible with an Emscripten build of CPython. In particular, the Pyodide ABI version is exposed via -`sysconfig.get_config_var("PYODIDE_ABI_VERSION")`. +``sysconfig.get_config_var("PYODIDE_ABI_VERSION")``. Package indexes From 3b021424ce534a5b805cd28dabace2aef1b93328 Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Wed, 26 Mar 2025 14:51:25 +0100 Subject: [PATCH 5/7] Remove emscripten_3_1_58 tag --- peps/pep-0776.rst | 23 +++++++++-------------- 1 file changed, 9 insertions(+), 14 deletions(-) diff --git a/peps/pep-0776.rst b/peps/pep-0776.rst index dd1bfa77f9b..82912f06ea3 100644 --- a/peps/pep-0776.rst +++ b/peps/pep-0776.rst @@ -652,26 +652,21 @@ packages like aiohttp, Requests, Pydantic, cryptography, and orjson. About 60 packages are also testing against Pyodide in their CI, including NumPy, pandas, awkward-cpp, scikit-image, statsmodels, PyArrow, Hypothesis, and PyO3. -Emscripten Wheel Tags -~~~~~~~~~~~~~~~~~~~~~ - -Emscripten wheels will use as the platform tag either -``emscripten__wasm32`` or ``pyodide__wasm32``. For example: +Pyodide Wheel Tags +~~~~~~~~~~~~~~~~~~ -* ``emscripten_3_1_58_wasm32`` -* ``pyodide_2025_0_wasm32`` +Pyodide wheels will use ``pyodide__wasm32`` as the platform tag. For +example, ``pyodide_2025_0_wasm32``. -The first triple is ambiguous, since even with Emscripten 3.1.58 it is possible -to link dynamic libraries that require a large number of distinct ABIs, -depending on linker and compiler options. It is our intent that the -``pyodide_2025_0`` specifies the particular ABI. Thus, the relationship between -``pyodide_`` and ``emscripten_`` is intended to be the same as the -relationship between ``manylinux`` and ``linux``. +With a fixed version of Emscripten, it is possible to link dynamic libraries +that require a large number of distinct ABIs, depending on ABI-sensitive linker +options and what versions of what static libraries are linked. It is our intent +that the ``pyodide_2025_0`` specifies the particular ABI that will work with the +Pyodide CPython runtime. For example, wheels with the following tags are compatible with Python 3.13 Pyodide: -- ``cp13-cp13-emscripten_3_1_58_wasm32`` - ``cp13-cp13-pyodide_2025_0_wasm32`` - ``abi3-cp10-pyodide_2025_0_wasm32`` From dc062dff8c2222cf8787fa2f112b6e9d7302e173 Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Wed, 26 Mar 2025 15:01:45 +0100 Subject: [PATCH 6/7] Trim a bit more out --- peps/pep-0776.rst | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/peps/pep-0776.rst b/peps/pep-0776.rst index 82912f06ea3..c3aae92a3fb 100644 --- a/peps/pep-0776.rst +++ b/peps/pep-0776.rst @@ -705,12 +705,8 @@ compiler and passing appropriate compiler and linker flags. It is possible for other people to build their own Python interpreter that is compatible with the Pyodide ABI, it is not necessary to use the Pyodide distribution itself. -The ``pyodide build`` tool knows how to create wheels that match our ABI. As an -alternative, -`the auditwheel-emscripten tool `__ -is capable of performing basic compatibility checks, vendoring shared libraries, -and retagging the wheel from ``emscripten_`` to ``pyodide_``. Unlike -with manylinux, there is no need for a Docker container to build the +The ``pyodide build`` tool knows how to create wheels that match our ABI. Unlike +with manylinux wheels, there is no need for a Docker container to build the ``pyodide_`` wheels. All that is needed is a Linux machine and appropriate versions of Python, Node.js, and Emscripten. From ab76d9c33e8fa9be679bedc979fa0e39404766f9 Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Thu, 27 Mar 2025 11:05:58 +0100 Subject: [PATCH 7/7] Apply hugovk's suggestions Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> --- peps/pep-0776.rst | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/peps/pep-0776.rst b/peps/pep-0776.rst index c3aae92a3fb..0d4450d1d14 100644 --- a/peps/pep-0776.rst +++ b/peps/pep-0776.rst @@ -670,10 +670,9 @@ Pyodide: - ``cp13-cp13-pyodide_2025_0_wasm32`` - ``abi3-cp10-pyodide_2025_0_wasm32`` -As well as all non platformed tags. To generate the list of compatible tags, one -can use the following code: +As well as all non-platformed tags. To generate the list of compatible tags, one +can use the following code:: -.. code-block:: Python from packaging.tags import cpython_tags, _generic_platforms @@ -685,7 +684,7 @@ can use the following code: emscripten_tags = cpython_tags(platforms=_emscripten_platforms()) -`This code will be added to pypa/packaging +This code will be added to `pypa/packaging `__. Emscripten Wheel ABI