Skip to content

PEP 776: Remove all discussion of packaging #4327

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
AA-Turner merged 7 commits into from
Mar 28, 2025
Merged

PEP 776: Remove all discussion of packaging #4327

Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
5d207a6
PEP 776: Remove all discussion of packaging
hoodmane Mar 28, 2025
172848d
Remove Emscripten ABI compatibility section too
hoodmane Mar 28, 2025
9629ab5
Add new discussion thread
hoodmane Mar 28, 2025
962b177
Try to fix CI
hoodmane Mar 28, 2025
efd3d71
Try again to fix header validation
hoodmane Mar 28, 2025
ba78822
line break in post history
hoodmane Mar 28, 2025
34138df
Change type to informational
hoodmane Mar 28, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
188 changes: 16 additions & 172 deletions peps/pep-0776.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,13 @@ PEP: 776
Title: Emscripten Support
Author: Hood Chatham <roberthoodchatham at gmail.com>
Sponsor: Łukasz Langa <lukasz at python.org>
Discussions-To: https://discuss.python.org/t/84996
Discussions-To: https://discuss.python.org/t/86276
Status: Draft
Type: Standards Track
Topic: Packaging
Type: Informational
Created: 18-Mar-2025
Python-Version: 3.14
Post-History: `18-Mar-2025 <https://discuss.python.org/t/84996>`__
Post-History: `18-Mar-2025 <https://discuss.python.org/t/84996>`__,
`28-Mar-2025 <https://discuss.python.org/t/86276>`__,

Abstract
========
Expand All @@ -20,8 +20,17 @@ also maintains an Emscripten target.

This PEP formalizes the addition of Tier 3 for Emscripten support in Python 3.14
which `was approved by the Steering Council on October 25, 2024
<https://github.com/python/steering-council/issues/256>`__. The goal is to allow
Pyodide wheels to be uploaded to PyPI.
<https://github.com/python/steering-council/issues/256>`__. The goals are:

1. To describe the current state of the CPython Emscripten runtime
2. To describe the current state of the Pyodide runtime
3. To identify minor features to be upstreamed from the Pyodide runtime into the
CPython Emscripten runtime

The minor features identified here are all features that could be implemented
without a PEP. We discuss more significant runtime features that we would like
to implement but we defer decisions on those features to subsequent PEPs.


Motivation
==========
Expand All @@ -41,33 +50,6 @@ Emscripten platform.
Emscripten and WASI are also the only supported platforms that offer any
meaningful sandboxing.

Goals
=====

It is our long term goal to upstream the entire Pyodide runtime into CPython,
but this is out of scope for the present PEP. This PEP only attempts to
establish the foundation for future work.

Runtime Goals
-------------

1. To describe the current state of the CPython Emscripten runtime
2. To describe the current state of the Pyodide runtime
3. To identify minor features to be upstreamed from the Pyodide runtime into the
CPython Emscripten runtime

The minor features identified here are all features that could be implemented
without a PEP. We discuss more significant runtime features that we would like
to implement but we defer decisions on those features to subsequent PEPs.

Packaging Goals
---------------

1. To describe Pyodide's packaging tooling
2. To describe Pyodide's wheel abi to a similar level of detail as the manylinux
PEPs
3. For Pyodide wheels to be allowed for upload to PyPI


Emscripten Platform Information
===============================
Expand Down Expand Up @@ -142,30 +124,6 @@ Because of these limitations, Pyodide standardizes a no-pthreads build of
Python. If there is sufficient demand, a pthreads build with no dynamic loader
could be added later.

Emscripten ABI Compatibility
----------------------------

The Emscripten compiler has no ABI stability guarantees between versions. Many
Emscripten updates are ABI compatible by chance, and the Rust Emscripten target
behaves as if the ABI were stable with only `occasional negative consequences
<https://github.com/rust-lang/rust/issues/131467>`__.

There are several linker flags that adjust the Emscripten ABI, so Python
packages built to run with Emscripten must make sure to match the ABI-sensitive
linker flags used to compile the interpreter to avoid load-time or run-time
errors. The Emscripten compiler continuously fixes bugs and adds support for new
web platform features. Thus, there is significant benefit to being able to
update the ABI.

In order to balance the ABI stability needs of package maintainers with the ABI
flexibility to allow the platform to move forward, Pyodide plans to adopt a new
ABI for each feature release of Python.

The Pyodide team also coordinates the ABI flags that Pyodide uses with the
Emscripten ABI that Rust supports in order to ensure that we have support for
the many popular Rust packages. Historically, most of the work for this has
been related to unwinding ABIs. See for instance `this Rust Major Change
Proposal <https://github.com/rust-lang/compiler-team/issues/801>`__.

Development Tools
-----------------
Expand Down Expand Up @@ -520,7 +478,7 @@ The following modules can be imported, but are not functional:
as well as any functionality that requires these.

The following are present but cannot be imported due to a dependency on the
termios package which has been removed:
termios module which has been removed:

- pty
- tty
Expand Down Expand Up @@ -638,110 +596,6 @@ buildbots, maintained by Russell Keith-Magee.
CPython does not currently test Tier 3 platforms on GitHub Actions, but if this
ever changes, their Linux runners are able to build and test Emscripten Python.

Packaging
---------

Existing Package Support
~~~~~~~~~~~~~~~~~~~~~~~~

Pyodide currently maintains ports of 255 different packages at the time of this
writing, including major scientific Python packages like NumPy, SciPy, pandas,
Polars, scikit-learn, OpenCV, PyArrow, and Pillow as well as general purpose
packages like aiohttp, Requests, Pydantic, cryptography, and orjson.

About 60 packages are also testing against Pyodide in their CI, including NumPy,
pandas, awkward-cpp, scikit-image, statsmodels, PyArrow, Hypothesis, and PyO3.

Pyodide Wheel Tags
~~~~~~~~~~~~~~~~~~

Pyodide wheels will use ``pyodide_<abi>_wasm32`` as the platform tag. For
example, ``pyodide_2025_0_wasm32``.

With a fixed version of Emscripten, it is possible to link dynamic libraries
that require a large number of distinct ABIs, depending on ABI-sensitive linker
options and what versions of what static libraries are linked. It is our intent
that the ``pyodide_2025_0`` specifies the particular ABI that will work with the
Pyodide CPython runtime.

For example, wheels with the following tags are compatible with Python 3.13
Pyodide:

- ``cp13-cp13-pyodide_2025_0_wasm32``
- ``abi3-cp10-pyodide_2025_0_wasm32``

As well as all non-platformed tags. To generate the list of compatible tags, one
can use the following code::


from packaging.tags import cpython_tags, _generic_platforms

def _emscripten_platforms() -> Iterator[str]:
pyodide_abi_version = sysconfig.get_config_var("PYODIDE_ABI_VERSION")
if pyodide_abi_version:
yield f"pyodide_{pyodide_abi_version}_wasm32"
yield from _generic_platforms()

emscripten_tags = cpython_tags(platforms=_emscripten_platforms())

This code will be added to `pypa/packaging
<https://github.com/pypa/packaging/pull/804>`__.

Emscripten Wheel ABI
~~~~~~~~~~~~~~~~~~~~

The specification of the ``pyodide_<abi>`` ABI includes:

* Which version of the Emscripten compiler is used
* What libraries are statically linked with the interpreter
* What stack unwinding ABI is to be used
* Which runtime platform features are required to be present

and a handful of other similar details that affect the ABI.

The ABI is selected by choosing the appropriate version of the Emscripten
compiler and passing appropriate compiler and linker flags. It is possible for
other people to build their own Python interpreter that is compatible with the
Pyodide ABI, it is not necessary to use the Pyodide distribution itself.

The ``pyodide build`` tool knows how to create wheels that match our ABI. Unlike
with manylinux wheels, there is no need for a Docker container to build the
``pyodide_<abi>`` wheels. All that is needed is a Linux machine and appropriate
versions of Python, Node.js, and Emscripten.


Package Installers
~~~~~~~~~~~~~~~~~~

Installers should use the ``_emscripten_platforms()`` function shown above to
determine which platforms are compatible with an Emscripten build of CPython. In
particular, the Pyodide ABI version is exposed via
``sysconfig.get_config_var("PYODIDE_ABI_VERSION")``.


Package indexes
~~~~~~~~~~~~~~~

We recommend that package indexes accept any wheel whose platform tag matches
``pyodide_[0-9]+_[0-9]+_wasm32``. We recommend that package indexes continue not
accepting wheels that match ``emscripten_[0-9]+_[0-9]+_[0-9]+_wasm32``.


Dependency Specifier Markers
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

To check for the Emscripten platform in a dependency specifier, one can use
``sys_platform == 'emscripten'`` (or its negation). Such checks are already in use
in the wild and seem to be sufficient for the needs of the community.


Trove Classifier
~~~~~~~~~~~~~~~~

Packages that build and test Emscripten wheels can declare this by adding the
``Environment :: WebAssembly :: Emscripten``. PyPI already accepts uploads of
packages with this classifier.


PEP 11
------
Expand Down Expand Up @@ -836,10 +690,6 @@ care to minimize backwards incompatibility. We will also need a way to disable
partially-upstreamed features so that Pyodide can replace them with more
complete versions downstream.

These backwards compatibility concerns impact not just the runtime but also the
packaging system. Adding new platform tags should not affect existing packaging
tools because tools ignore wheels with an unknown package tag.


Security Implications
=====================
Expand All @@ -861,12 +711,6 @@ to use them all at runtime. The documentation will cover this in a similar form
to the existing Windows embeddable package. In the short term, we will encourage
developers to use Pyodide if at all possible.

Second, maintainers of packages with binary components need to know how to
build, test, label, and deploy them for Emscripten (see Packaging). The Pyodide
documentation on `building and testing packages
<https://pyodide.org/en/stable/development/building-and-testing-packages.html>`__
is the best reference for this.


Reference Implementation
========================
Expand Down