From 6a3482b50805a44de10fefb2881a5d006904325e Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Mon, 17 Mar 2025 21:34:54 +0100 Subject: [PATCH 01/44] Add Emscripten support pep --- peps/pep-TBD.rst | 750 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 750 insertions(+) create mode 100644 peps/pep-TBD.rst diff --git a/peps/pep-TBD.rst b/peps/pep-TBD.rst new file mode 100644 index 00000000000..59eb727177d --- /dev/null +++ b/peps/pep-TBD.rst @@ -0,0 +1,750 @@ +Author: Hood Chatham +Sponsor: Łukasz Langa , Russell Keith-Magee +Discussions-To: +Status: Draft +Type: Standards Track +Topic: Packaging +Created: +Python-Version: 3.14 +Post-History: + +Abstract +======== + +`Emscripten `__ is a complete open source compiler +toolchain which compiles C/C++ code into WebAssembly/JavaScript executables that +run in JavaScript runtimes including browsers and Node. + +This PEP formalizes the addition of Tier 3 for Emscripten support in Python 3.14 +which `was approved by the steering council on October 25, 2024 +`__. The goal is to +allow Pyodide wheels to be uploaded to PyPI. + + +Motivation +========== + +A web browser is a universal computing platform, available on Windows, Mac, +Linux, and every smart phone. + +The Pyodide project has for years supported Emscripten Python. Hundreds of +thousands of students have learned Python through Pyodide via projects like +`Capytale +`__ +and `PyodideU `__. Pyodide +is also increasingly being used by Python packages to provide interactive +documentation. This demonstrates both the importance and the maturity of the +Emscripten platform. + +Emscripten and wasi are also the only supported platforms that offer any +meaningful sandboxing. + +Goals +===== + +It is our long term goal to upstream the entire Pyodide runtime into CPython, +but this is out of scope for the present PEP. This PEP only attempts to +establish the foundation for future work. + +Runtime Goals +------------- + +1. To describe the current state of the CPython Emscripten runtime +2. To describe the current state of the Pyodide runtime +3. To identify features to be upstreamed from the Pyodide runtime into the + CPython Emscripten runtime + +Packaging Goals +--------------- + +1. To describe Pyodide's packaging tooling +2. To describe our wheel abi in a similar level of detail as the manylinux peps +3. To request that Pyodide wheels be allowed for upload to pypi + + +Emscripten Platform Information +=============================== + +Background on Emscripten +------------------------ + +`Emscripten +`__ +consists of a C and C++ compiler and linker based on LLVM, together with a +runtime based on a mildly patched musl libc. + +Emscripten is a POSIX-based platform. It uses the WebAssembly binary format, +specified here: +* https://webassembly.github.io/spec/core/binary/index.html +* https://github.com/WebAssembly/tool-conventions/blob/main/DynamicLinking.md + +The ``emcc`` compiler is a wrapper around ``clang``. The ``emcc`` linker is a wrapper +around ``wasm-ld`` (also part of the llvm toolchain). + +Emscripten support for portable C/C++ code source compatibility with Linux is +fairly comprehensive, with certain expected exceptions to be spelled out. CPython +already supports compilation to Emscripten, and it only requires a very modest +number of modifications to the normal linux target. + +POSIX compliance +---------------- + +Emscripten is a POSIX platform. However, there are POSIX APIs that exist but +always fail when called and POSIX APIs that don't exist at all. In particular, +there are problems with networking APIs and blocking I/O, and there is no +support for ``fork()``. See +`Emscripten Portability Guidelines `__. + +Emscripten executables can be linked with threading support, however, it comes +with several limitations: + +Enabling threading requires websites to be served with special security headers +that indicate acceptance of the possibility of spectre-style information +leakage. These headers are a usability hazard for users who are not intimately +familiar with the web platform. + +If an executable is linked with both threading and a dynamic loader, Emscripten +prints a warning that using dynamic loading and pthreads together is +experimental. It may cause performance problems or crashes. These problems may +require WebAssembly standards work to resolve. + +Because of these limitations, Pyodide standardizes a no-pthreads build of +Python. We will start with support for no-pthreads builds. If there is +sufficient demand, a pthreads build with no dynamic loader could be added later. + +Emscripten ABI Compatibility +---------------------------- + +The Emscripten compiler has no ABI stability guarantees between versions. Many +Emscripten updates are ABI compatible by chance, and the Rust Emscripten target +behaves as if the ABI were stable with only `occasional negative consequences +`__. + +There are several linker flags that adjust the Emscripten ABI, so Python +packages built to run with Emscripten must make sure to match the ABI-sensitive +linker flags used to compile the interpreter to avoid load-time or run-time +errors. The Emscripten compiler continuously fixes bugs and adds support for new +web platform features. Thus, there is significant benefit to being able to +update the ABI. + +In order to balance the ABI stability needs of package maintainers with the ABI +flexibility to allow the platform to move forward, we plan to adopt a new ABI +for each minor release of Python. + +It is also necessary to coordinate the ABI flags that we use with the Emscripten +ABI that Rust supports if we want support for the many popular Rust packages. + +Development tools +----------------- + +Emscripten development tools are equally well supported on Linux, Windows, and +macOS. Upstream tools: + +* The Emscripten Software Developer Kit (emsdk) which can be used to install the + Emscripten compiler toolchain (emcc). +* emcc is a C and C++ compiler, linker, and a sysroot with headers for the + system libraries. The system libraries themselves are generated on the fly + based on the ABI requested. +* NodeJS can be used as an "emulator" to run Emscripten programs from the + command line. This emulation behaves best on Linux with Mac as a runner up. + Node is the most convenient way to test Emscripten programs. +* It is possible to run Emscripten programs inside of any web browser. Browser + automation tools like selenium, playwright, or pupeeteer can be used to test + features that are browser-only. + +Pyodide's tools: + +* ``pyodide build`` can be used to cross compile Python packages to run on + Emscripten. Cross compilation works best on Linux, sometimes on Macs, and is + unsupported on Windows. +* ``pyodide venv`` can make a virtual environment that runs in Pyodide. +* ``pytest-pyodide`` can test Python code against various JS runtimes. + +cibuildwheel supports building wheels to target Emscripten using ``pyodide build``. + +In the short term, Pyodide's packaging tooling will stay in the Pyodide +repository. It is an open question where Pyodide's packaging tooling should live +in the long term. Two sensible options would be for it to remain under the +``pyodide`` organization or be moved into the ``pypa`` organization. + + +Emscripten Application Lifecycle +-------------------------------- + +An Emscripten "binary" consists of a pair of files, an ``.mjs`` file and a ``.wasm`` +file. The ``.wasm`` file contains all of the compiled C/C++/Rust code. The ``.mjs`` +file contains the lifecycle code to set up the runtime, locate the ``.wasm`` file, +compile it, instantiate it, call the ``main()`` function, and to shut down the +runtime on exit. It also includes an implementation for all of the system calls, +including the file system, the dynamic loader, and any logic to expose +additional functionality from the JavaScript runtime to C code. + +The ``.mjs`` file exports a single ``bootstrapEmscriptenExecutable()`` +JavaScript function that bootstraps the runtime, calls the ``main()`` function, +and returns an API object that can be used to call C functions. Each time it is +called produces a complete and independent copy of the runtime with its own +separate address space. + +The ``bootstrapEmscriptenExecutable()`` takes a large number of runtime settings. +`The full list is described in the Emscripten documentation here. +`__ The most +important of these are as follows: + +* ``thisProgram``: The value of ``argv[0]``. In Python, this makes its way into + ``sys.executable``. +* ``arguments``: The list of string arguments to be passed to ``main()``. +* ``preRun``: A list of callbacks which are invoked after the JavaScript runtime + and file system have been bootstrapped but before calling ``main()``. Useful to + set up the file system, environment variables, and standard streams. +* ``print`` / ``printErr`` : Initial handlers for stdout and stderr. They are line + buffered and performing a ``flush()`` of a partial line forces an extra new + line. If tty-like behavior is desired, the standard stream devices should be + replaced in a ``preRun()`` hook. +* ``onExit``: A handler that is called when the runtime exits. +* ``instantiateWasm``: A callback that is called to instantiate the WebAssembly + module. Overriding the WebAssembly instantiation procedure via this function + is useful when you have other custom asynchronous startup actions or downloads + that can be performed in parallel to WebAssembly compilation. Implementing + this callback allows performing all of these in parallel. + +File system setup +----------------- + +In order for Python to run, it needs access to the standard library in the +Emscripten file system. There are several possible approaches to this: + +* The Emscripten linker has a ``--preload-file`` flag that will automatically + handle loading files. `Information about how it works is available here. + `__ + This is the simplest approach, but Pyodide has moved away from it because it + embeds the files into a custom archive format that cannot be processed with + standard tooling. + +* For Node, use the NODEFS to mount a native directory with the files into the + Emscripten file system. This is the most efficient option but is Node only. It + is closely analogous to what wasi does. + +* Put the standard library into a zip archive and use ``ZipImporter``. Using an + uncompressed zip file allows the web server and client to apply better + compression to the standard library itself. It also uses the more efficient + native decompression algorithms of the browser rather than less efficient + WebAssembly decompression. The disadvantage of this is a higher memory + footprint and it breaks ``inspect`` and various tests that do not expect the + standard library to be packaged in this way. + +* Put the standard library into an uncompressed tar archive and mount it into a + TARFS read only file system backed by the tar file. This has the best memory + usage, runtime performance, and transfer size of the options that can be used + in the browser. The disadvantage is that Emscripten does not itself include a + TARFS so it requires a downstream implementation. + +Pyodide uses the ``ZipImporter`` approach in every runtime. + +Python uses the NODEFS approach when run with node and the ``ZipImporter`` +approach for the web example. We will continue with this approach. + +Console and interactive usage +----------------------------- + +``stdin`` defaults to always returning ``EOF``, while ``stdout`` and ``stderr`` default +to calling ``console.log`` and ``console.error`` respectively. It is possible to +pass handlers to ``bootstrapEmscriptenExecutable()`` to configure the standard +streams, but no matter what the I/O devices have undesirable line buffering +behavior that forces a new line when flushed. To implement a well behaved tty in +browser, it is necessary to remove the default I/O devices and replace them in a +``preRun`` hook. + +Making ``stdin`` work correctly in the browser poses an additional challenge +because it is not allowed to block for user input in the main thread of the +browser. If Emscripten is run in a webworker and served with the shared memory +headers, it is possible to receive input using shared memory and atomics. It is +also possible for a ``stdin`` device to block in a simpler and more efficient +manner using stack switching using the experimental JavaScript Promise +Integration API. + +Pyodide replaces the standard I/O devices in order to fix the line buffering +behavior. When Pyodide is run in node, ``stdin``, ``stdout``, and ``stderr`` are +by default connected to ``process.stdin``, ``process.stdout``, and +``process.stderr`` and so the standard streams work as a tty out of the box. +Pyodide also ensures that ``shutil.get_terminal_size`` returns results +consistent with ``process.stdout.rows`` and ``process.stdout.columns``. Pyodide +currently has no support for stack switching ``stdin``. + +Currently, the Emscripten Python node runner uses the default I/O that +Emscripten provides. The web example uses ``Atomics`` for ``stdin`` and has +custom ``stdout`` and ``stderr`` handlers, but they exhibit the undesirable line +buffering behavior. We will upstream the standard streams behaviors from +Pyodide. + +In the long term, we hope to implement stack switching `stdin` devices, but that +is out of scope for this PEP. + +Dynamic libraries +----------------- + +Main Thread Synchronous Loading Limit +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In the main browser thread, a dynamic library can only be loaded synchronously +if it is at most 4 kilobytes. This excludes most nontrivial dynamic libraries. +This limit is not present in Node and can be avoided by using a web worker. If +stack switching is available, then it is possible to make ``dlopen()`` stack +switch in order to instantiate a dynamic library synchronously. + +To avoid the synchronous loading limit, Pyodide currently preloads all dynamic +libraries present in a wheel when installing the wheel (or on startup). This is +a significant disadvantage with pages like scipy that include a very large +number of shared libraries that are expected to be only loaded on demand. +Pyodide will implement a solution based on stack switching as it becomes more +widely available in runtimes. + +Emscripten Python only loads extension module dynamic libraries when they are +imported. This approach is simpler and more efficient when it works. The web +example runs in a web worker and the cli runner runs in Node so neither of these +have the synchronous loading limit. We will continue with this approach in +Emscripten Python. + +In the long run, we hope to implement a stack switching `dlopen`, but that is +out of scope for this PEP. + +Missing RPATH Support +~~~~~~~~~~~~~~~~~~~~~ + +Another important limitation of the Emscripten dynamic loader is that it does +not currently have RPATH support. Pyodide's present workaround is as follows: +``auditwheel-emscripten`` places shared library dependencies that are vendored +into a package in a ``${package}.libs`` folder, following auditwheel's +convention. Pyodide patches the dynamic loader to treat this ``${package}.libs`` +folder as if it were on the RPATH of all of the dynamic libraries in the wheel. + +In Emscripten 4.0.5, we have updated the shared object file format, ``wasm-ld`` +and ``emcc`` to accept an ``-rpath`` flag. We are still working on updating the +dynamic loader to respect the rpath, but we expect this will be finished in the +next Emscripten release. Pyodide will then switch to using the RPATH and drop +the patch on the dynamic loader. + +Emscripten Python currently uses the unpatched dynamic loader and so cannot load +extension modules that depend on vendored dynamic libraries via DT_NEEDED. +Extension modules can load dynamic libraries via DT_NEEDED if they are in the +system `lib` directory. We will wait to resolve this until we have fixed the +Emscripten dynamic loader upstream. When Emscripten Python is built with a +compatible version of Emscripten, it will automatically pick up support for +wheels with vendored dynamic libraries. + + +Traps and uncaught exceptions +----------------------------- + +We consider the C runtime state to be corrupted if there is a WebAssembly trap, +an unhandled JavaScript exception, or an uncaught WebAssembly throw instruction. + +Unlike in other platforms, there is no operating system to shut down the +executable when there is a trap or other unrecoverable corruption of the libc +runtime. We need to provide our own code to print tracebacks, dump the memory, +or do whatever else is helpful for debugging a crash. If we expose a JavaScript +API, we also must ensure that it is disabled after an unrecoverable crash to +prevent downstream users from observing the Python runtime in an inconsistent +state. + +In order to detect fatal errors, Pyodide uses the following approach: all +fallable calls from WebAssembly into JavaScript are wrapped with a JavaScript +try/catch block. Any caught JavaScript exceptions are translated into Python +exceptions. This ensures that any recoverable JavaScript error is caught before +it unwinds through any WebAssembly frames. All entrypoints to WebAssembly are +also wrapped with JavaScript try/catch blocks. Any exceptions caught there have +unwound WebAssembly frames and are thus considered to be fatal errors (though +there is a special case to handle `exit()`). This requires foundational +integration with the Python/JavaScript foreign function interface. + +When the Pyodide runtime catches a fatal exception, it introspects the error to +determine whether it came from a trap, a logic error in a system call, a +`setjmp()` without a `longjmp()`, or a libcxxabi call to `__cxa_throw()` (an uncaught +C++ exception or Rust panic). We render as informative an error message as we +can. We also call `_Py_DumpTraceback()` so we can display a Python traceback in +addition to the JS/WebAssembly traceback. It also disables the JavaScript API so +that further attempts to call into Python result in an error saying that the +runtime has fatally failed. + +Normally, WebAssembly symbols are stripped so the WebAssembly frames are not +very useful. Compiling and linking with `-g2` (or a higher debug setting) +ensures that WebAssembly symbols are included and they will appear in the +traceback. + +Because Emscripten Python currently has no JS API and no foreign function +interface, the situation is much simpler. The Python node runner wraps the call +to `bootstrapEmscriptenExecutable()` in a try/catch block. If an exception is +caught, it displays the JavaScript exception and calls `_Py_DumpTraceback()`. It +then exits with code 1. We will stick with this approach until we add either a +JS API or foreign function interface, which is out of scope for this PEP. + +Specification +============= + +Scope of work +------------- + +Adding Emscripten as a Tier 3 platform only requires adding support for +compiling an Emscripten-compatible build from the unpatched CPython source code. +It does not necessarily require there to be any officially distributed +Emscripten artifacts on python.org, although these could be added in the future. +In the short term, they will continue to be distributed downstream with Pyodide. + +Emscripten will be built using the same configure and Makefile system as other +POSIX platforms, and must therefore be built on a POSIX platform. Both Linux and +macOS will be supported. + +A Python CLI entrypoint will be provided, which among other things can be used +to execute the test suite. + +Linkage +------- + +It is only supported to statically link the Python interpreter. We use `EM_JS +`__ +functions in the interpreter for various purposes. It is possible to dynamically +link object files that include ``EM_JS`` functions, but their behavior deviates +significantly from their behavior in static builds. For this reason, it would +require special work to support. If a use case for dynamically linking the +interpreter in Emscripten emerges, we can evaluate how much effort would be +required to support it. + +Standard Library +---------------- + +Unsupported modules +~~~~~~~~~~~~~~~~~~~ + +See https://pyodide.org/en/stable/usage/wasm-constraints.html#removed-modules. + +Removed modules +^^^^^^^^^^^^^^^ + +The following modules are removed from the standard library to reduce download size and +since they currently wouldn't work in the WebAssembly VM, + +- curses +- dbm +- ensurepip +- fcntl +- grp +- idlelib +- lib2to3 +- msvcrt +- pwd +- resource +- syslog +- termios +- tkinter +- turtle +- turtledemo +- venv +- winreg +- winsound + +Included but not working modules +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The following modules can be imported, but are not functional: + +- multiprocessing +- threading +- sockets + +as well as any functionality that requires these. + +The following are present but cannot be imported due to a dependency on the +termios package which has been removed: + +- pty +- tty + + +Platform identification +~~~~~~~~~~~~~~~~~~~~~~~ + +``sys.platform`` will return ``"emscripten"``. Although Emscripten attempts to be +compatible with Linux, the differences are significant enough that a distinct +name is justified. This is consistent with the return value from ``os.uname()``. + +There is also ``sys._emscripten_info`` which includes the Emscripten version and +the runtime (either ``navigator.userAgent`` in a browser or ``"Node js" + +process.version`` in Node). + +Signals Support +--------------- + +WebAssembly does not have native support for signals. Furthermore, on a +non-pthreads build, the address space of the WebAssembly module is not shared, +so it is impossible for any thread capable of seeing an interrupt to write to +the eval breaker while the Python interpreter is running code. To work around +this, there are two possible solutions: +* If Emscripten is run in a webworker and served with the shared memory headers, + it is possible to use shared memory outside of the WebAssembly address space + as a signal buffer. A signal handling UI thread can write the desired signal + into the signal buffer. The interpreter can periodically check the state of + this signal buffer in the eval breaker code. Checking the signal buffer is + slow compared to checking the eval breaker in native platforms, so we do only + do it once every 50 times through the eval breaker. See + `Python/emscripten_signal.c `__ +* Using stack switching, we can occasionally switch the stack and allow the + JavaScript event loop to go around, then check the state of a signal buffer. + This requires the experimental JavaScript Promise Integration API, and would + be best used with the techniques for optimizing long tasks described + `in this article `__ + +Emscripten Python has already implemented the solution based on shared memory, +and it is in use in Pyodide. + +Eventually, we hope to implement stack-switching-based signals so that it is +possible to use signals in the main thread of node and the browser, as well as +in in web pages that are not served with the shared memory headers. We will need +to keep the shared memory based approach as well, both for backwards +compatibility and because it is more efficient when it is possible. However, +this is out of scope for this PEP. + + +Function pointer casts +---------------------- + +`Section 6.3.2.3, paragraph 8 +`__ of the C +standard reads: + +> A pointer to a function of one type may be converted to a pointer to a +> function of another type and back again; the result shall compare equal to the +> original pointer. If a converted pointer is used to call a function whose type +> is not compatible with the pointed-to type, the behavior is undefined. + +However, most platforms have the same behavior: if a function is called with too +many arguments, the extra arguments are ignored; if a function is called with +too few arguments, the extra arguments are filled in with garbage. + +On the other hand, the WebAssembly spec defines calling a function with the +wrong signature to trap (`see step 18 in the execution of call_indirect +`__. + +It is common for Python extension modules to cast a function to a different +signature and call it with the different signature. For instance, many C +extensions define a ``METH_NOARGS`` function to take 0 or 1 argument. The +interpreter calls it with two arguments, the first of which is the Python module +object and the second of which is always ``NULL``. In order to make these +extension modules work without changing their source code, we need special +handling. + +Initially, we resolved this problem by calling out to JavaScript and having +JavaScript call the function pointer. When calling a WebAssembly function from +JavaScript, missing arguments are treated as zero and extra arguments are +ignored (`see step 7 here +`__. +This works, but has the disadvantage of being slow and breaking stack switching +-- it is not possible to stack switch through JavaScript frames. + +Using the wasm-gc `ref.test +`__ +instruction, we can query the type of the function pointer and manually fix up +the argument list. + +wasm-gc is a relatively new feature for WebAssembly runtimes, so we attempt to +use a wasm-gc based function pointer cast trampoline if possible and fall back +to a JS trampoline if not. Every JavaScript runtime that supports stack +switching also supports wasm-gc, so this ensures that stack switching works on +every platform runtime that supports it. The one wrinkle is that iOS 18 ships a +broken implementation of wasm-gc so we have to special case it. + +`See here for the full implementation details. +`__ + +The function pointer cast handling is fully implemented in cpython. Pyodide uses +exactly the same code as upstream. + + +CI Resources +------------ + +Pyodide can be built and tested on any Linux with a reasonably recent version of +Node. Anaconda has offered to provide physical hardware to run Android +buildbots, maintained by Russell Keith-Magee. + +CPython does not currently test Tier 3 platforms on GitHub Actions, but if this +ever changes, their Linux runners are able to build and test Emscripten Python. + +Packaging +--------- + +Existing package support +~~~~~~~~~~~~~~~~~~~~~~~~ + +Pyodide currently maintains ports of 255 different packages at the time of this +writing, including major scientific Python packages like numpy, scipy, pandas, +polars, scikit-learn, opencv, pyarrow, and Pillow as well as general purpose +packages like aiohttp, requests, pydantic, cryptography, and orjson. + +About 60 packages are also testing against Pyodide in their CI, including numpy, +pandas, awkward-cpp, scikit-image, statsmodels, pyarrow, hypothesis, and PyO3. + +Emscripten wheel format +~~~~~~~~~~~~~~~~~~~~~~~ + +Emscripten wheels will use either the format ``emscripten__wasm32`` or +``pyodide__wasm32``. For example: + +* ``emscripten_3_1_58_wasm32`` +* ``pyodide_2025_0_wasm32`` + +The first triple is ambiguous, since even with Emscripten 3.1.58 it is possible +to link dynamic libraries that require a large number of distinct ABIs, +depending on linker and compiler options. It is our intent that the +``pyodide_2025_0`` specifies the particular ABI. Thus, the relationship between +``pyodide_`` and ``emscripten_`` is intended to be the same as the +relationship between ``manylinux`` and ``linux``. + +The specification of the ``pyodide_`` ABI includes: + +* Which version of the Emscripten compiler is used +* What libraries are statically linked with the interpreter +* What stack unwinding ABI is to be used +* Which runtime platform features are required to be present + +and a handful of other similar details that affect the ABI. + +The ABI is selected by choosing the appropriate version of the Emscripten +compiler and passing appropriate compiler and linker flags. It is possible for +other people to build their own Python interpreter that is compatible with the +Pyodide ABI, it is not necessary to use the Pyodide distribution itself. + +The ``pyodide build`` tool knows how to create wheels that match our ABI. As an +alternative, +`the auditwheel-emscripten tool `__ + +is capable of performing basic compatibility checks, vendoring shared libraries, +and retagging the wheel from ``emscripten_`` to ``pyodide_``. Unlike +with manylinux, there is no need for a docker container to build the +``pyodide_`` wheels. All that is needed is a Linux machine and appropriate +versions of Python, node, and Emscripten. + + +PEP 11 +------ + +PEP 11 will be updated to indicate that Emscripten is supported. Specifically the triples +``wasm32-unknown-emscripten_xx_xx_xx``. + +Russell Keith-Magee will serve as the initial core team contact for these ABIs. + + +Future Work +=========== + +Improving cross builds in the packaging ecosystem +------------------------------------------------- + +Python now supports four non-self-hosting platforms: iOS, Android, wasi, and +Emscripten. All of them will need to build packages via cross builds. Currently, +``pyodide-build`` allows building a very large number of Python packages for +Emscripten, but it relies on a giant pile of hacks. In the long run, we would +like to make the packaging ecosystem support cross builds in a sane way. + + +Pyodide Runtime Features to be upstreamed +----------------------------------------- + +This is a collection of Pyodide runtime features that are out of scope for this +PEP and for the Python 3.14 development cycle but we would like to upstream in +the future. + +JavaScript API for bootstrapping +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Currently we offer no stable API for bootstrapping Python. Instead, we use one +collection of settings for the Node cli entrypoint and a separate collection of +settings for the browser demo. +https://github.com/python/cpython/tree/98fa4a49fecbac3c990a25ce5d300592dad31be0/Tools/wasm/emscripten/node_entry.mjs +https://github.com/python/cpython/blob/98fa4a49fecbac3c990a25ce5d300592dad31be0/Tools/wasm/emscripten/web_example/python.worker.mjs + +The Emscripten executable startup API is complicated and there are many possible +configurations that are broken. Pyodide offers a simpler set of options than +Emscripten. This gives downstream users a lot of flexibility while allowing us +to maintain a small number of tested configurations. It also reduces downstream +code duplication. + +Eventually, we would like to upstream Pyodide's bootstrapping API. In the short +term, to keep things simple we will support no JavaScript API. + +FFI +~~~ + +Because Emscripten supports POSIX, a significant number of tasks can be achieved +using the ``os`` module. However, many fundamental operations in JavaScript +runtimes are not possible via POSIX APIs. Pyodide's approach is to specify a +mapping between the JavaScript object model and the Python object model and a +calling convention that allows high level bidirectional integration. + +Asyncio +~~~~~~~ + +Most JavaScript primitives are asynchronous. The JavaScript thread that Python +runs in already has an event loop. It it not too difficult to implement a Python +event loop that defers all actual work to the JavaScript event loop, +`implemented in Pyodide here `__. + +This is logically dependent on having at least some limited JavaScript FFI +because the only way to schedule tasks on the JavaScript event loop is via a +call out to JavaScript. + +One cause of incompatibility is that it is not possible to control the life +cycle of the event loop from within a JavaScript isolate. This makes +``asyncio.run()`` and similar things not work. + +Using stack switching it is also possible to make a coroutine out of +"synchronous" Python frames. These stack switching coroutines are scheduled on +the same event loop as ordinary Python coroutines and are fully reentrant. This +is fully implemented in Pyodide. + + +Backwards Compatibility +======================= + +Adding a new platform does not introduce any backwards compatibility concerns to +CPython itself. However, there may be some backwards compatibility implications +on Pyodide users. There are a large number of existing users of Pyodide, so it +is important when upstreaming features from Pyodide into Python that we take +care to minimize backwards incompatibility. We will also need a way to disable +partially-upstreamed features so that Pyodide can replace them with more +complete versions downstream. + +These backwards compatibility concerns impact not just the runtime but also the +packaging system. + + +Security Implications +===================== + +Adding a new platform does not add any new security implications. + + +How to Teach This +================= + +The education needs related to this PEP relate to two groups of developers. + +First, web developers will need to know how to build Python and use it in a +website, along with their own Python code and any supporting packages, and how +to use them all at runtime. The documentation will cover this in a similar form +to the existing Windows embeddable package. In the short term, we will encourage +developers to use Pyodide if at all possible. + +Second, developers of packages with binary components need to know how to build +and release them for Emscripten (see Packaging). + + +Reference Implementation +======================== + +Pyodide. + + +Copyright +========= + +This document is placed in the public domain or under the CC0-1.0-Universal +license, whichever is more permissive. From bcc99e3d5c4a2a522a50a2f2cc64275f48e1033e Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Tue, 18 Mar 2025 11:56:50 +0100 Subject: [PATCH 02/44] Fix pre-commit --- peps/pep-TBD.rst | 46 +++++++++++++++++++++++----------------------- 1 file changed, 23 insertions(+), 23 deletions(-) diff --git a/peps/pep-TBD.rst b/peps/pep-TBD.rst index 59eb727177d..fabd1805e16 100644 --- a/peps/pep-TBD.rst +++ b/peps/pep-TBD.rst @@ -6,7 +6,7 @@ Type: Standards Track Topic: Packaging Created: Python-Version: 3.14 -Post-History: +Post-History: Abstract ======== @@ -39,7 +39,7 @@ Emscripten platform. Emscripten and wasi are also the only supported platforms that offer any meaningful sandboxing. -Goals +Goals ===== It is our long term goal to upstream the entire Pyodide runtime into CPython, @@ -50,7 +50,7 @@ Runtime Goals ------------- 1. To describe the current state of the CPython Emscripten runtime -2. To describe the current state of the Pyodide runtime +2. To describe the current state of the Pyodide runtime 3. To identify features to be upstreamed from the Pyodide runtime into the CPython Emscripten runtime @@ -74,7 +74,7 @@ consists of a C and C++ compiler and linker based on LLVM, together with a runtime based on a mildly patched musl libc. Emscripten is a POSIX-based platform. It uses the WebAssembly binary format, -specified here: +specified here: * https://webassembly.github.io/spec/core/binary/index.html * https://github.com/WebAssembly/tool-conventions/blob/main/DynamicLinking.md @@ -92,11 +92,11 @@ POSIX compliance Emscripten is a POSIX platform. However, there are POSIX APIs that exist but always fail when called and POSIX APIs that don't exist at all. In particular, there are problems with networking APIs and blocking I/O, and there is no -support for ``fork()``. See +support for ``fork()``. See `Emscripten Portability Guidelines `__. Emscripten executables can be linked with threading support, however, it comes -with several limitations: +with several limitations: Enabling threading requires websites to be served with special security headers that indicate acceptance of the possibility of spectre-style information @@ -165,7 +165,7 @@ cibuildwheel supports building wheels to target Emscripten using ``pyodide build In the short term, Pyodide's packaging tooling will stay in the Pyodide repository. It is an open question where Pyodide's packaging tooling should live in the long term. Two sensible options would be for it to remain under the -``pyodide`` organization or be moved into the ``pypa`` organization. +``pyodide`` organization or be moved into the ``pypa`` organization. Emscripten Application Lifecycle @@ -183,7 +183,7 @@ The ``.mjs`` file exports a single ``bootstrapEmscriptenExecutable()`` JavaScript function that bootstraps the runtime, calls the ``main()`` function, and returns an API object that can be used to call C functions. Each time it is called produces a complete and independent copy of the runtime with its own -separate address space. +separate address space. The ``bootstrapEmscriptenExecutable()`` takes a large number of runtime settings. `The full list is described in the Emscripten documentation here. @@ -219,7 +219,7 @@ Emscripten file system. There are several possible approaches to this: This is the simplest approach, but Pyodide has moved away from it because it embeds the files into a custom archive format that cannot be processed with standard tooling. - + * For Node, use the NODEFS to mount a native directory with the files into the Emscripten file system. This is the most efficient option but is Node only. It is closely analogous to what wasi does. @@ -276,7 +276,7 @@ custom ``stdout`` and ``stderr`` handlers, but they exhibit the undesirable line buffering behavior. We will upstream the standard streams behaviors from Pyodide. -In the long term, we hope to implement stack switching `stdin` devices, but that +In the long term, we hope to implement stack switching ``stdin`` devices, but that is out of scope for this PEP. Dynamic libraries @@ -304,7 +304,7 @@ example runs in a web worker and the cli runner runs in Node so neither of these have the synchronous loading limit. We will continue with this approach in Emscripten Python. -In the long run, we hope to implement a stack switching `dlopen`, but that is +In the long run, we hope to implement a stack switching ``dlopen``, but that is out of scope for this PEP. Missing RPATH Support @@ -326,7 +326,7 @@ the patch on the dynamic loader. Emscripten Python currently uses the unpatched dynamic loader and so cannot load extension modules that depend on vendored dynamic libraries via DT_NEEDED. Extension modules can load dynamic libraries via DT_NEEDED if they are in the -system `lib` directory. We will wait to resolve this until we have fixed the +system ``lib`` directory. We will wait to resolve this until we have fixed the Emscripten dynamic loader upstream. When Emscripten Python is built with a compatible version of Emscripten, it will automatically pick up support for wheels with vendored dynamic libraries. @@ -353,27 +353,27 @@ exceptions. This ensures that any recoverable JavaScript error is caught before it unwinds through any WebAssembly frames. All entrypoints to WebAssembly are also wrapped with JavaScript try/catch blocks. Any exceptions caught there have unwound WebAssembly frames and are thus considered to be fatal errors (though -there is a special case to handle `exit()`). This requires foundational +there is a special case to handle ``exit()``). This requires foundational integration with the Python/JavaScript foreign function interface. When the Pyodide runtime catches a fatal exception, it introspects the error to determine whether it came from a trap, a logic error in a system call, a -`setjmp()` without a `longjmp()`, or a libcxxabi call to `__cxa_throw()` (an uncaught +``setjmp()`` without a ``longjmp()``, or a libcxxabi call to ``__cxa_throw()`` (an uncaught C++ exception or Rust panic). We render as informative an error message as we -can. We also call `_Py_DumpTraceback()` so we can display a Python traceback in +can. We also call ``_Py_DumpTraceback()`` so we can display a Python traceback in addition to the JS/WebAssembly traceback. It also disables the JavaScript API so that further attempts to call into Python result in an error saying that the runtime has fatally failed. Normally, WebAssembly symbols are stripped so the WebAssembly frames are not -very useful. Compiling and linking with `-g2` (or a higher debug setting) +very useful. Compiling and linking with ``-g2`` (or a higher debug setting) ensures that WebAssembly symbols are included and they will appear in the traceback. Because Emscripten Python currently has no JS API and no foreign function interface, the situation is much simpler. The Python node runner wraps the call -to `bootstrapEmscriptenExecutable()` in a try/catch block. If an exception is -caught, it displays the JavaScript exception and calls `_Py_DumpTraceback()`. It +to ``bootstrapEmscriptenExecutable()`` in a try/catch block. If an exception is +caught, it displays the JavaScript exception and calls ``_Py_DumpTraceback()``. It then exits with code 1. We will stick with this approach until we add either a JS API or foreign function interface, which is out of scope for this PEP. @@ -489,7 +489,7 @@ this, there are two possible solutions: * Using stack switching, we can occasionally switch the stack and allow the JavaScript event loop to go around, then check the state of a signal buffer. This requires the experimental JavaScript Promise Integration API, and would - be best used with the techniques for optimizing long tasks described + be best used with the techniques for optimizing long tasks described `in this article `__ Emscripten Python has already implemented the solution based on shared memory, @@ -542,7 +542,7 @@ This works, but has the disadvantage of being slow and breaking stack switching Using the wasm-gc `ref.test `__ instruction, we can query the type of the function pointer and manually fix up -the argument list. +the argument list. wasm-gc is a relatively new feature for WebAssembly runtimes, so we attempt to use a wasm-gc based function pointer cast trampoline if possible and fall back @@ -605,7 +605,7 @@ The specification of the ``pyodide_`` ABI includes: * What stack unwinding ABI is to be used * Which runtime platform features are required to be present -and a handful of other similar details that affect the ABI. +and a handful of other similar details that affect the ABI. The ABI is selected by choosing the appropriate version of the Emscripten compiler and passing appropriate compiler and linker flags. It is possible for @@ -613,7 +613,7 @@ other people to build their own Python interpreter that is compatible with the Pyodide ABI, it is not necessary to use the Pyodide distribution itself. The ``pyodide build`` tool knows how to create wheels that match our ABI. As an -alternative, +alternative, `the auditwheel-emscripten tool `__ is capable of performing basic compatibility checks, vendoring shared libraries, @@ -684,7 +684,7 @@ Asyncio Most JavaScript primitives are asynchronous. The JavaScript thread that Python runs in already has an event loop. It it not too difficult to implement a Python -event loop that defers all actual work to the JavaScript event loop, +event loop that defers all actual work to the JavaScript event loop, `implemented in Pyodide here `__. This is logically dependent on having at least some limited JavaScript FFI From c783fee7300664928b40f2ef5762decedd6575c8 Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Tue, 18 Mar 2025 12:14:53 +0100 Subject: [PATCH 03/44] Rename to PEP-776 and add discussion link --- peps/{pep-TBD.rst => PEP-776.rst} | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) rename peps/{pep-TBD.rst => PEP-776.rst} (99%) diff --git a/peps/pep-TBD.rst b/peps/PEP-776.rst similarity index 99% rename from peps/pep-TBD.rst rename to peps/PEP-776.rst index fabd1805e16..f22d613d625 100644 --- a/peps/pep-TBD.rst +++ b/peps/PEP-776.rst @@ -1,6 +1,6 @@ Author: Hood Chatham Sponsor: Łukasz Langa , Russell Keith-Magee -Discussions-To: +Discussions-To: https://discuss.python.org/t/pep-776-emscripten-support/84996 Status: Draft Type: Standards Track Topic: Packaging From cca103327a697dd2a3ae866401ec57db72c77b69 Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Tue, 18 Mar 2025 12:20:06 +0100 Subject: [PATCH 04/44] Add PEP number and title to header; rename to include leading zero --- peps/{PEP-776.rst => PEP-0776.rst} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename peps/{PEP-776.rst => PEP-0776.rst} (100%) diff --git a/peps/PEP-776.rst b/peps/PEP-0776.rst similarity index 100% rename from peps/PEP-776.rst rename to peps/PEP-0776.rst From 4fdce798e4c0f8f5d2c085260f3c5142ad0d546a Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Tue, 18 Mar 2025 12:24:03 +0100 Subject: [PATCH 05/44] Lowercase file name --- peps/{PEP-0776.rst => pep-0776.rst} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename peps/{PEP-0776.rst => pep-0776.rst} (100%) diff --git a/peps/PEP-0776.rst b/peps/pep-0776.rst similarity index 100% rename from peps/PEP-0776.rst rename to peps/pep-0776.rst From 6171a678118a58abd7ae6fe2e4659b322615e3ff Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Tue, 18 Mar 2025 12:26:07 +0100 Subject: [PATCH 06/44] Update header --- peps/pep-0776.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/peps/pep-0776.rst b/peps/pep-0776.rst index f22d613d625..becc7d4fdff 100644 --- a/peps/pep-0776.rst +++ b/peps/pep-0776.rst @@ -1,12 +1,13 @@ +PEP: 776 +Title: Emscripten Support Author: Hood Chatham Sponsor: Łukasz Langa , Russell Keith-Magee Discussions-To: https://discuss.python.org/t/pep-776-emscripten-support/84996 Status: Draft Type: Standards Track Topic: Packaging -Created: +Created: 18-Mar-2025 Python-Version: 3.14 -Post-History: Abstract ======== From 871a223e84bae061b795fbe6fa0ae956cc9fddb5 Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Tue, 18 Mar 2025 12:29:10 +0100 Subject: [PATCH 07/44] try to fix indentation-related rendering error --- peps/pep-0776.rst | 24 +++++++++++++----------- 1 file changed, 13 insertions(+), 11 deletions(-) diff --git a/peps/pep-0776.rst b/peps/pep-0776.rst index becc7d4fdff..03bcdc85b84 100644 --- a/peps/pep-0776.rst +++ b/peps/pep-0776.rst @@ -479,19 +479,21 @@ non-pthreads build, the address space of the WebAssembly module is not shared, so it is impossible for any thread capable of seeing an interrupt to write to the eval breaker while the Python interpreter is running code. To work around this, there are two possible solutions: + * If Emscripten is run in a webworker and served with the shared memory headers, - it is possible to use shared memory outside of the WebAssembly address space - as a signal buffer. A signal handling UI thread can write the desired signal - into the signal buffer. The interpreter can periodically check the state of - this signal buffer in the eval breaker code. Checking the signal buffer is - slow compared to checking the eval breaker in native platforms, so we do only - do it once every 50 times through the eval breaker. See - `Python/emscripten_signal.c `__ +it is possible to use shared memory outside of the WebAssembly address space +as a signal buffer. A signal handling UI thread can write the desired signal +into the signal buffer. The interpreter can periodically check the state of +this signal buffer in the eval breaker code. Checking the signal buffer is +slow compared to checking the eval breaker in native platforms, so we do only +do it once every 50 times through the eval breaker. See +`Python/emscripten_signal.c `__ + * Using stack switching, we can occasionally switch the stack and allow the - JavaScript event loop to go around, then check the state of a signal buffer. - This requires the experimental JavaScript Promise Integration API, and would - be best used with the techniques for optimizing long tasks described - `in this article `__ +JavaScript event loop to go around, then check the state of a signal buffer. +This requires the experimental JavaScript Promise Integration API, and would +be best used with the techniques for optimizing long tasks described +`in this article `__ Emscripten Python has already implemented the solution based on shared memory, and it is in use in Pyodide. From 927ebec7cce938b25bd2291841e848844f20c0e5 Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Tue, 18 Mar 2025 12:30:07 +0100 Subject: [PATCH 08/44] Only include one sponsor --- peps/pep-0776.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0776.rst b/peps/pep-0776.rst index 03bcdc85b84..1fbe8a713a2 100644 --- a/peps/pep-0776.rst +++ b/peps/pep-0776.rst @@ -1,7 +1,7 @@ PEP: 776 Title: Emscripten Support Author: Hood Chatham -Sponsor: Łukasz Langa , Russell Keith-Magee +Sponsor: Łukasz Langa Discussions-To: https://discuss.python.org/t/pep-776-emscripten-support/84996 Status: Draft Type: Standards Track From b08c3349fbb589863fa38756935eb8f6181387cd Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Tue, 18 Mar 2025 12:41:12 +0100 Subject: [PATCH 09/44] Revert "try to fix indentation-related rendering error" This reverts commit 871a223e84bae061b795fbe6fa0ae956cc9fddb5. --- peps/pep-0776.rst | 24 +++++++++++------------- 1 file changed, 11 insertions(+), 13 deletions(-) diff --git a/peps/pep-0776.rst b/peps/pep-0776.rst index 1fbe8a713a2..b391c3abde5 100644 --- a/peps/pep-0776.rst +++ b/peps/pep-0776.rst @@ -479,21 +479,19 @@ non-pthreads build, the address space of the WebAssembly module is not shared, so it is impossible for any thread capable of seeing an interrupt to write to the eval breaker while the Python interpreter is running code. To work around this, there are two possible solutions: - * If Emscripten is run in a webworker and served with the shared memory headers, -it is possible to use shared memory outside of the WebAssembly address space -as a signal buffer. A signal handling UI thread can write the desired signal -into the signal buffer. The interpreter can periodically check the state of -this signal buffer in the eval breaker code. Checking the signal buffer is -slow compared to checking the eval breaker in native platforms, so we do only -do it once every 50 times through the eval breaker. See -`Python/emscripten_signal.c `__ - + it is possible to use shared memory outside of the WebAssembly address space + as a signal buffer. A signal handling UI thread can write the desired signal + into the signal buffer. The interpreter can periodically check the state of + this signal buffer in the eval breaker code. Checking the signal buffer is + slow compared to checking the eval breaker in native platforms, so we do only + do it once every 50 times through the eval breaker. See + `Python/emscripten_signal.c `__ * Using stack switching, we can occasionally switch the stack and allow the -JavaScript event loop to go around, then check the state of a signal buffer. -This requires the experimental JavaScript Promise Integration API, and would -be best used with the techniques for optimizing long tasks described -`in this article `__ + JavaScript event loop to go around, then check the state of a signal buffer. + This requires the experimental JavaScript Promise Integration API, and would + be best used with the techniques for optimizing long tasks described + `in this article `__ Emscripten Python has already implemented the solution based on shared memory, and it is in use in Pyodide. From ca6b7e33918acedefc09510ad615d54ca4a55927 Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Tue, 18 Mar 2025 12:41:36 +0100 Subject: [PATCH 10/44] Add newline before list --- peps/pep-0776.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/peps/pep-0776.rst b/peps/pep-0776.rst index b391c3abde5..4e877240e2e 100644 --- a/peps/pep-0776.rst +++ b/peps/pep-0776.rst @@ -479,6 +479,7 @@ non-pthreads build, the address space of the WebAssembly module is not shared, so it is impossible for any thread capable of seeing an interrupt to write to the eval breaker while the Python interpreter is running code. To work around this, there are two possible solutions: + * If Emscripten is run in a webworker and served with the shared memory headers, it is possible to use shared memory outside of the WebAssembly address space as a signal buffer. A signal handling UI thread can write the desired signal From 538418eef018d4df27651136870ed4c1fb99dcb6 Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Tue, 18 Mar 2025 12:42:15 +0100 Subject: [PATCH 11/44] Add Post-History --- peps/pep-0776.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/peps/pep-0776.rst b/peps/pep-0776.rst index 4e877240e2e..a0d522d1d39 100644 --- a/peps/pep-0776.rst +++ b/peps/pep-0776.rst @@ -8,6 +8,7 @@ Type: Standards Track Topic: Packaging Created: 18-Mar-2025 Python-Version: 3.14 +Post-History: `18-Mar-2025 Date: Tue, 18 Mar 2025 12:43:20 +0100 Subject: [PATCH 12/44] Add newline ahead of another list --- peps/pep-0776.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/peps/pep-0776.rst b/peps/pep-0776.rst index a0d522d1d39..dffd825c647 100644 --- a/peps/pep-0776.rst +++ b/peps/pep-0776.rst @@ -77,6 +77,7 @@ runtime based on a mildly patched musl libc. Emscripten is a POSIX-based platform. It uses the WebAssembly binary format, specified here: + * https://webassembly.github.io/spec/core/binary/index.html * https://github.com/WebAssembly/tool-conventions/blob/main/DynamicLinking.md From 0a8d64ed714a9e718ef01bc0b5ad972f80f910d1 Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Tue, 18 Mar 2025 12:49:35 +0100 Subject: [PATCH 13/44] Update headings --- peps/pep-0776.rst | 59 ++++++++++++++++++++++++++++++++++++----------- 1 file changed, 46 insertions(+), 13 deletions(-) diff --git a/peps/pep-0776.rst b/peps/pep-0776.rst index dffd825c647..5619f4e07fb 100644 --- a/peps/pep-0776.rst +++ b/peps/pep-0776.rst @@ -10,6 +10,7 @@ Created: 18-Mar-2025 Python-Version: 3.14 Post-History: `18-Mar-2025 `__. The goal is to allow Pyodide wheels to be uploaded to PyPI. - +========== Motivation ========== @@ -41,6 +42,7 @@ Emscripten platform. Emscripten and wasi are also the only supported platforms that offer any meaningful sandboxing. +===== Goals ===== @@ -48,6 +50,7 @@ It is our long term goal to upstream the entire Pyodide runtime into CPython, but this is out of scope for the present PEP. This PEP only attempts to establish the foundation for future work. +------------- Runtime Goals ------------- @@ -56,6 +59,7 @@ Runtime Goals 3. To identify features to be upstreamed from the Pyodide runtime into the CPython Emscripten runtime +--------------- Packaging Goals --------------- @@ -64,9 +68,11 @@ Packaging Goals 3. To request that Pyodide wheels be allowed for upload to pypi +=============================== Emscripten Platform Information =============================== +------------------------ Background on Emscripten ------------------------ @@ -89,6 +95,7 @@ fairly comprehensive, with certain expected exceptions to be spelled out. CPytho already supports compilation to Emscripten, and it only requires a very modest number of modifications to the normal linux target. +---------------- POSIX compliance ---------------- @@ -115,6 +122,7 @@ Because of these limitations, Pyodide standardizes a no-pthreads build of Python. We will start with support for no-pthreads builds. If there is sufficient demand, a pthreads build with no dynamic loader could be added later. +---------------------------- Emscripten ABI Compatibility ---------------------------- @@ -137,6 +145,7 @@ for each minor release of Python. It is also necessary to coordinate the ABI flags that we use with the Emscripten ABI that Rust supports if we want support for the many popular Rust packages. +----------------- Development tools ----------------- @@ -171,6 +180,7 @@ in the long term. Two sensible options would be for it to remain under the ``pyodide`` organization or be moved into the ``pypa`` organization. +-------------------------------- Emscripten Application Lifecycle -------------------------------- @@ -210,6 +220,8 @@ important of these are as follows: that can be performed in parallel to WebAssembly compilation. Implementing this callback allows performing all of these in parallel. + +----------------- File system setup ----------------- @@ -246,6 +258,8 @@ Pyodide uses the ``ZipImporter`` approach in every runtime. Python uses the NODEFS approach when run with node and the ``ZipImporter`` approach for the web example. We will continue with this approach. + +----------------------------- Console and interactive usage ----------------------------- @@ -282,11 +296,13 @@ Pyodide. In the long term, we hope to implement stack switching ``stdin`` devices, but that is out of scope for this PEP. + +----------------- Dynamic libraries ----------------- Main Thread Synchronous Loading Limit -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +===================================== In the main browser thread, a dynamic library can only be loaded synchronously if it is at most 4 kilobytes. This excludes most nontrivial dynamic libraries. @@ -311,7 +327,7 @@ In the long run, we hope to implement a stack switching ``dlopen``, but that is out of scope for this PEP. Missing RPATH Support -~~~~~~~~~~~~~~~~~~~~~ +===================== Another important limitation of the Emscripten dynamic loader is that it does not currently have RPATH support. Pyodide's present workaround is as follows: @@ -336,7 +352,7 @@ wheels with vendored dynamic libraries. Traps and uncaught exceptions ------------------------------ +============================= We consider the C runtime state to be corrupted if there is a WebAssembly trap, an unhandled JavaScript exception, or an uncaught WebAssembly throw instruction. @@ -380,9 +396,11 @@ caught, it displays the JavaScript exception and calls ``_Py_DumpTraceback()``. then exits with code 1. We will stick with this approach until we add either a JS API or foreign function interface, which is out of scope for this PEP. +============= Specification ============= +------------- Scope of work ------------- @@ -399,6 +417,7 @@ macOS will be supported. A Python CLI entrypoint will be provided, which among other things can be used to execute the test suite. +------- Linkage ------- @@ -411,16 +430,17 @@ require special work to support. If a use case for dynamically linking the interpreter in Emscripten emerges, we can evaluate how much effort would be required to support it. +---------------- Standard Library ---------------- Unsupported modules -~~~~~~~~~~~~~~~~~~~ +=================== See https://pyodide.org/en/stable/usage/wasm-constraints.html#removed-modules. Removed modules -^^^^^^^^^^^^^^^ +--------------- The following modules are removed from the standard library to reduce download size and since they currently wouldn't work in the WebAssembly VM, @@ -445,7 +465,7 @@ since they currently wouldn't work in the WebAssembly VM, - winsound Included but not working modules -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +-------------------------------- The following modules can be imported, but are not functional: @@ -463,7 +483,7 @@ termios package which has been removed: Platform identification -~~~~~~~~~~~~~~~~~~~~~~~ +======================= ``sys.platform`` will return ``"emscripten"``. Although Emscripten attempts to be compatible with Linux, the differences are significant enough that a distinct @@ -473,6 +493,7 @@ There is also ``sys._emscripten_info`` which includes the Emscripten version and the runtime (either ``navigator.userAgent`` in a browser or ``"Node js" + process.version`` in Node). +--------------- Signals Support --------------- @@ -507,6 +528,7 @@ compatibility and because it is more efficient when it is possible. However, this is out of scope for this PEP. +---------------------- Function pointer casts ---------------------- @@ -562,6 +584,7 @@ The function pointer cast handling is fully implemented in cpython. Pyodide uses exactly the same code as upstream. +------------ CI Resources ------------ @@ -572,11 +595,12 @@ buildbots, maintained by Russell Keith-Magee. CPython does not currently test Tier 3 platforms on GitHub Actions, but if this ever changes, their Linux runners are able to build and test Emscripten Python. +--------- Packaging --------- Existing package support -~~~~~~~~~~~~~~~~~~~~~~~~ +======================== Pyodide currently maintains ports of 255 different packages at the time of this writing, including major scientific Python packages like numpy, scipy, pandas, @@ -587,7 +611,7 @@ About 60 packages are also testing against Pyodide in their CI, including numpy, pandas, awkward-cpp, scikit-image, statsmodels, pyarrow, hypothesis, and PyO3. Emscripten wheel format -~~~~~~~~~~~~~~~~~~~~~~~ +======================= Emscripten wheels will use either the format ``emscripten__wasm32`` or ``pyodide__wasm32``. For example: @@ -627,6 +651,7 @@ with manylinux, there is no need for a docker container to build the versions of Python, node, and Emscripten. +------ PEP 11 ------ @@ -636,9 +661,11 @@ PEP 11 will be updated to indicate that Emscripten is supported. Specifically th Russell Keith-Magee will serve as the initial core team contact for these ABIs. +=========== Future Work =========== +------------------------------------------------- Improving cross builds in the packaging ecosystem ------------------------------------------------- @@ -649,6 +676,7 @@ Emscripten, but it relies on a giant pile of hacks. In the long run, we would like to make the packaging ecosystem support cross builds in a sane way. +----------------------------------------- Pyodide Runtime Features to be upstreamed ----------------------------------------- @@ -657,7 +685,7 @@ PEP and for the Python 3.14 development cycle but we would like to upstream in the future. JavaScript API for bootstrapping -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +================================ Currently we offer no stable API for bootstrapping Python. Instead, we use one collection of settings for the Node cli entrypoint and a separate collection of @@ -675,7 +703,7 @@ Eventually, we would like to upstream Pyodide's bootstrapping API. In the short term, to keep things simple we will support no JavaScript API. FFI -~~~ +=== Because Emscripten supports POSIX, a significant number of tasks can be achieved using the ``os`` module. However, many fundamental operations in JavaScript @@ -684,7 +712,7 @@ mapping between the JavaScript object model and the Python object model and a calling convention that allows high level bidirectional integration. Asyncio -~~~~~~~ +======== Most JavaScript primitives are asynchronous. The JavaScript thread that Python runs in already has an event loop. It it not too difficult to implement a Python @@ -705,6 +733,7 @@ the same event loop as ordinary Python coroutines and are fully reentrant. This is fully implemented in Pyodide. +======================= Backwards Compatibility ======================= @@ -720,12 +749,14 @@ These backwards compatibility concerns impact not just the runtime but also the packaging system. +===================== Security Implications ===================== Adding a new platform does not add any new security implications. +================= How to Teach This ================= @@ -741,12 +772,14 @@ Second, developers of packages with binary components need to know how to build and release them for Emscripten (see Packaging). +======================== Reference Implementation ======================== Pyodide. +========= Copyright ========= From 5b64058a4a6aeea120dbc38c3b593b59978ef077 Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Tue, 18 Mar 2025 12:50:03 +0100 Subject: [PATCH 14/44] Fix post-history link --- peps/pep-0776.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0776.rst b/peps/pep-0776.rst index 5619f4e07fb..57027e163fe 100644 --- a/peps/pep-0776.rst +++ b/peps/pep-0776.rst @@ -8,7 +8,7 @@ Type: Standards Track Topic: Packaging Created: 18-Mar-2025 Python-Version: 3.14 -Post-History: `18-Mar-2025 `__ ======== Abstract From 2ad212783f675301d7100ca0655753d2f27d285f Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Tue, 18 Mar 2025 12:52:59 +0100 Subject: [PATCH 15/44] Hard wrap document --- peps/pep-0776.rst | 103 +++++++++++++++++++++++----------------------- 1 file changed, 52 insertions(+), 51 deletions(-) diff --git a/peps/pep-0776.rst b/peps/pep-0776.rst index 57027e163fe..3c5dee91b4e 100644 --- a/peps/pep-0776.rst +++ b/peps/pep-0776.rst @@ -20,8 +20,8 @@ run in JavaScript runtimes including browsers and Node. This PEP formalizes the addition of Tier 3 for Emscripten support in Python 3.14 which `was approved by the steering council on October 25, 2024 -`__. The goal is to -allow Pyodide wheels to be uploaded to PyPI. +`__. The goal is to allow +Pyodide wheels to be uploaded to PyPI. ========== Motivation @@ -87,13 +87,13 @@ specified here: * https://webassembly.github.io/spec/core/binary/index.html * https://github.com/WebAssembly/tool-conventions/blob/main/DynamicLinking.md -The ``emcc`` compiler is a wrapper around ``clang``. The ``emcc`` linker is a wrapper -around ``wasm-ld`` (also part of the llvm toolchain). +The ``emcc`` compiler is a wrapper around ``clang``. The ``emcc`` linker is a +wrapper around ``wasm-ld`` (also part of the llvm toolchain). Emscripten support for portable C/C++ code source compatibility with Linux is -fairly comprehensive, with certain expected exceptions to be spelled out. CPython -already supports compilation to Emscripten, and it only requires a very modest -number of modifications to the normal linux target. +fairly comprehensive, with certain expected exceptions to be spelled out. +CPython already supports compilation to Emscripten, and it only requires a very +modest number of modifications to the normal linux target. ---------------- POSIX compliance @@ -102,8 +102,8 @@ POSIX compliance Emscripten is a POSIX platform. However, there are POSIX APIs that exist but always fail when called and POSIX APIs that don't exist at all. In particular, there are problems with networking APIs and blocking I/O, and there is no -support for ``fork()``. See -`Emscripten Portability Guidelines `__. +support for ``fork()``. See `Emscripten Portability Guidelines +`__. Emscripten executables can be linked with threading support, however, it comes with several limitations: @@ -184,13 +184,13 @@ in the long term. Two sensible options would be for it to remain under the Emscripten Application Lifecycle -------------------------------- -An Emscripten "binary" consists of a pair of files, an ``.mjs`` file and a ``.wasm`` -file. The ``.wasm`` file contains all of the compiled C/C++/Rust code. The ``.mjs`` -file contains the lifecycle code to set up the runtime, locate the ``.wasm`` file, -compile it, instantiate it, call the ``main()`` function, and to shut down the -runtime on exit. It also includes an implementation for all of the system calls, -including the file system, the dynamic loader, and any logic to expose -additional functionality from the JavaScript runtime to C code. +An Emscripten "binary" consists of a pair of files, an ``.mjs`` file and a +``.wasm`` file. The ``.wasm`` file contains all of the compiled C/C++/Rust code. +The ``.mjs`` file contains the lifecycle code to set up the runtime, locate the +``.wasm`` file, compile it, instantiate it, call the ``main()`` function, and to +shut down the runtime on exit. It also includes an implementation for all of the +system calls, including the file system, the dynamic loader, and any logic to +expose additional functionality from the JavaScript runtime to C code. The ``.mjs`` file exports a single ``bootstrapEmscriptenExecutable()`` JavaScript function that bootstraps the runtime, calls the ``main()`` function, @@ -198,8 +198,8 @@ and returns an API object that can be used to call C functions. Each time it is called produces a complete and independent copy of the runtime with its own separate address space. -The ``bootstrapEmscriptenExecutable()`` takes a large number of runtime settings. -`The full list is described in the Emscripten documentation here. +The ``bootstrapEmscriptenExecutable()`` takes a large number of runtime +settings. `The full list is described in the Emscripten documentation here. `__ The most important of these are as follows: @@ -207,12 +207,12 @@ important of these are as follows: ``sys.executable``. * ``arguments``: The list of string arguments to be passed to ``main()``. * ``preRun``: A list of callbacks which are invoked after the JavaScript runtime - and file system have been bootstrapped but before calling ``main()``. Useful to - set up the file system, environment variables, and standard streams. -* ``print`` / ``printErr`` : Initial handlers for stdout and stderr. They are line - buffered and performing a ``flush()`` of a partial line forces an extra new - line. If tty-like behavior is desired, the standard stream devices should be - replaced in a ``preRun()`` hook. + and file system have been bootstrapped but before calling ``main()``. Useful + to set up the file system, environment variables, and standard streams. +* ``print`` / ``printErr`` : Initial handlers for stdout and stderr. They are + line buffered and performing a ``flush()`` of a partial line forces an extra + new line. If tty-like behavior is desired, the standard stream devices should + be replaced in a ``preRun()`` hook. * ``onExit``: A handler that is called when the runtime exits. * ``instantiateWasm``: A callback that is called to instantiate the WebAssembly module. Overriding the WebAssembly instantiation procedure via this function @@ -263,13 +263,13 @@ approach for the web example. We will continue with this approach. Console and interactive usage ----------------------------- -``stdin`` defaults to always returning ``EOF``, while ``stdout`` and ``stderr`` default -to calling ``console.log`` and ``console.error`` respectively. It is possible to -pass handlers to ``bootstrapEmscriptenExecutable()`` to configure the standard -streams, but no matter what the I/O devices have undesirable line buffering -behavior that forces a new line when flushed. To implement a well behaved tty in -browser, it is necessary to remove the default I/O devices and replace them in a -``preRun`` hook. +``stdin`` defaults to always returning ``EOF``, while ``stdout`` and ``stderr`` +default to calling ``console.log`` and ``console.error`` respectively. It is +possible to pass handlers to ``bootstrapEmscriptenExecutable()`` to configure +the standard streams, but no matter what the I/O devices have undesirable line +buffering behavior that forces a new line when flushed. To implement a well +behaved tty in browser, it is necessary to remove the default I/O devices and +replace them in a ``preRun`` hook. Making ``stdin`` work correctly in the browser poses an additional challenge because it is not allowed to block for user input in the main thread of the @@ -293,8 +293,8 @@ custom ``stdout`` and ``stderr`` handlers, but they exhibit the undesirable line buffering behavior. We will upstream the standard streams behaviors from Pyodide. -In the long term, we hope to implement stack switching ``stdin`` devices, but that -is out of scope for this PEP. +In the long term, we hope to implement stack switching ``stdin`` devices, but +that is out of scope for this PEP. ----------------- @@ -377,12 +377,12 @@ integration with the Python/JavaScript foreign function interface. When the Pyodide runtime catches a fatal exception, it introspects the error to determine whether it came from a trap, a logic error in a system call, a -``setjmp()`` without a ``longjmp()``, or a libcxxabi call to ``__cxa_throw()`` (an uncaught -C++ exception or Rust panic). We render as informative an error message as we -can. We also call ``_Py_DumpTraceback()`` so we can display a Python traceback in -addition to the JS/WebAssembly traceback. It also disables the JavaScript API so -that further attempts to call into Python result in an error saying that the -runtime has fatally failed. +``setjmp()`` without a ``longjmp()``, or a libcxxabi call to ``__cxa_throw()`` +(an uncaught C++ exception or Rust panic). We render as informative an error +message as we can. We also call ``_Py_DumpTraceback()`` so we can display a +Python traceback in addition to the JS/WebAssembly traceback. It also disables +the JavaScript API so that further attempts to call into Python result in an +error saying that the runtime has fatally failed. Normally, WebAssembly symbols are stripped so the WebAssembly frames are not very useful. Compiling and linking with ``-g2`` (or a higher debug setting) @@ -392,9 +392,9 @@ traceback. Because Emscripten Python currently has no JS API and no foreign function interface, the situation is much simpler. The Python node runner wraps the call to ``bootstrapEmscriptenExecutable()`` in a try/catch block. If an exception is -caught, it displays the JavaScript exception and calls ``_Py_DumpTraceback()``. It -then exits with code 1. We will stick with this approach until we add either a -JS API or foreign function interface, which is out of scope for this PEP. +caught, it displays the JavaScript exception and calls ``_Py_DumpTraceback()``. +It then exits with code 1. We will stick with this approach until we add either +a JS API or foreign function interface, which is out of scope for this PEP. ============= Specification @@ -442,8 +442,8 @@ See https://pyodide.org/en/stable/usage/wasm-constraints.html#removed-modules. Removed modules --------------- -The following modules are removed from the standard library to reduce download size and -since they currently wouldn't work in the WebAssembly VM, +The following modules are removed from the standard library to reduce download +size and since they currently wouldn't work in the WebAssembly VM, - curses - dbm @@ -485,8 +485,8 @@ termios package which has been removed: Platform identification ======================= -``sys.platform`` will return ``"emscripten"``. Although Emscripten attempts to be -compatible with Linux, the differences are significant enough that a distinct +``sys.platform`` will return ``"emscripten"``. Although Emscripten attempts to +be compatible with Linux, the differences are significant enough that a distinct name is justified. This is consistent with the return value from ``os.uname()``. There is also ``sys._emscripten_info`` which includes the Emscripten version and @@ -510,12 +510,13 @@ this, there are two possible solutions: this signal buffer in the eval breaker code. Checking the signal buffer is slow compared to checking the eval breaker in native platforms, so we do only do it once every 50 times through the eval breaker. See - `Python/emscripten_signal.c `__ + `Python/emscripten_signal.c + `__ * Using stack switching, we can occasionally switch the stack and allow the JavaScript event loop to go around, then check the state of a signal buffer. This requires the experimental JavaScript Promise Integration API, and would - be best used with the techniques for optimizing long tasks described - `in this article `__ + be best used with the techniques for optimizing long tasks described `in this + article `__ Emscripten Python has already implemented the solution based on shared memory, and it is in use in Pyodide. @@ -655,8 +656,8 @@ versions of Python, node, and Emscripten. PEP 11 ------ -PEP 11 will be updated to indicate that Emscripten is supported. Specifically the triples -``wasm32-unknown-emscripten_xx_xx_xx``. +PEP 11 will be updated to indicate that Emscripten is supported. Specifically +the triples ``wasm32-unknown-emscripten_xx_xx_xx``. Russell Keith-Magee will serve as the initial core team contact for these ABIs. From fd67029f683fc1188f0099677e8b4d1e5af76d12 Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Tue, 18 Mar 2025 12:54:58 +0100 Subject: [PATCH 16/44] Use title caps in headings --- peps/pep-0776.rst | 34 +++++++++++++++++----------------- 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/peps/pep-0776.rst b/peps/pep-0776.rst index 3c5dee91b4e..e815fe440a8 100644 --- a/peps/pep-0776.rst +++ b/peps/pep-0776.rst @@ -96,7 +96,7 @@ CPython already supports compilation to Emscripten, and it only requires a very modest number of modifications to the normal linux target. ---------------- -POSIX compliance +POSIX Compliance ---------------- Emscripten is a POSIX platform. However, there are POSIX APIs that exist but @@ -146,7 +146,7 @@ It is also necessary to coordinate the ABI flags that we use with the Emscripten ABI that Rust supports if we want support for the many popular Rust packages. ----------------- -Development tools +Development Tools ----------------- Emscripten development tools are equally well supported on Linux, Windows, and @@ -222,7 +222,7 @@ important of these are as follows: ----------------- -File system setup +File System Setup ----------------- In order for Python to run, it needs access to the standard library in the @@ -260,7 +260,7 @@ approach for the web example. We will continue with this approach. ----------------------------- -Console and interactive usage +Console and Interactive Usage ----------------------------- ``stdin`` defaults to always returning ``EOF``, while ``stdout`` and ``stderr`` @@ -298,7 +298,7 @@ that is out of scope for this PEP. ----------------- -Dynamic libraries +Dynamic Libraries ----------------- Main Thread Synchronous Loading Limit @@ -351,7 +351,7 @@ compatible version of Emscripten, it will automatically pick up support for wheels with vendored dynamic libraries. -Traps and uncaught exceptions +Traps and Uncaught Exceptions ============================= We consider the C runtime state to be corrupted if there is a WebAssembly trap, @@ -401,7 +401,7 @@ Specification ============= ------------- -Scope of work +Scope of Work ------------- Adding Emscripten as a Tier 3 platform only requires adding support for @@ -434,12 +434,12 @@ required to support it. Standard Library ---------------- -Unsupported modules +Unsupported Modules =================== See https://pyodide.org/en/stable/usage/wasm-constraints.html#removed-modules. -Removed modules +Removed Modules --------------- The following modules are removed from the standard library to reduce download @@ -464,7 +464,7 @@ size and since they currently wouldn't work in the WebAssembly VM, - winreg - winsound -Included but not working modules +Included but not Working Modules -------------------------------- The following modules can be imported, but are not functional: @@ -482,7 +482,7 @@ termios package which has been removed: - tty -Platform identification +Platform Identification ======================= ``sys.platform`` will return ``"emscripten"``. Although Emscripten attempts to @@ -530,7 +530,7 @@ this is out of scope for this PEP. ---------------------- -Function pointer casts +Function Pointer Casts ---------------------- `Section 6.3.2.3, paragraph 8 @@ -600,7 +600,7 @@ ever changes, their Linux runners are able to build and test Emscripten Python. Packaging --------- -Existing package support +Existing Package Support ======================== Pyodide currently maintains ports of 255 different packages at the time of this @@ -611,7 +611,7 @@ packages like aiohttp, requests, pydantic, cryptography, and orjson. About 60 packages are also testing against Pyodide in their CI, including numpy, pandas, awkward-cpp, scikit-image, statsmodels, pyarrow, hypothesis, and PyO3. -Emscripten wheel format +Emscripten Wheel Format ======================= Emscripten wheels will use either the format ``emscripten__wasm32`` or @@ -667,7 +667,7 @@ Future Work =========== ------------------------------------------------- -Improving cross builds in the packaging ecosystem +Improving Cross Builds in the Packaging Ecosystem ------------------------------------------------- Python now supports four non-self-hosting platforms: iOS, Android, wasi, and @@ -678,14 +678,14 @@ like to make the packaging ecosystem support cross builds in a sane way. ----------------------------------------- -Pyodide Runtime Features to be upstreamed +Pyodide Runtime Features to be Upstreamed ----------------------------------------- This is a collection of Pyodide runtime features that are out of scope for this PEP and for the Python 3.14 development cycle but we would like to upstream in the future. -JavaScript API for bootstrapping +JavaScript API for Bootstrapping ================================ Currently we offer no stable API for bootstrapping Python. Instead, we use one From 27176123e8b24b7ed3688be17ad522ae9db2d70a Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Tue, 18 Mar 2025 13:00:19 +0100 Subject: [PATCH 17/44] Fix block quote --- peps/pep-0776.rst | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/peps/pep-0776.rst b/peps/pep-0776.rst index e815fe440a8..67e417301c4 100644 --- a/peps/pep-0776.rst +++ b/peps/pep-0776.rst @@ -537,10 +537,11 @@ Function Pointer Casts `__ of the C standard reads: -> A pointer to a function of one type may be converted to a pointer to a -> function of another type and back again; the result shall compare equal to the -> original pointer. If a converted pointer is used to call a function whose type -> is not compatible with the pointed-to type, the behavior is undefined. + A pointer to a function of one type may be converted to a pointer to a + function of another type and back again; the result shall compare equal to + the original pointer. If a converted pointer is used to call a function + whose type is not compatible with the pointed-to type, the behavior is + undefined. However, most platforms have the same behavior: if a function is called with too many arguments, the extra arguments are ignored; if a function is called with From 2be77c1673cb2c9552566f6621a8b5c1981db6c6 Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Tue, 18 Mar 2025 13:05:11 +0100 Subject: [PATCH 18/44] Update CODEOWNERS --- .github/CODEOWNERS | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index d22fd902dc3..c87858f39e0 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -654,6 +654,7 @@ peps/pep-0772.rst @warsaw @pradyunsg peps/pep-0773.rst @zooba peps/pep-0774.rst @savannahostrowski peps/pep-0775.rst @encukou +peps/pep-0776.rst @hoodmane @ambv # ... peps/pep-0777.rst @warsaw # ... From 5076f7d8a460dafffbc2de92ea940a384f829635 Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Tue, 18 Mar 2025 13:39:01 +0100 Subject: [PATCH 19/44] Add link to Pyodide --- peps/pep-0776.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/peps/pep-0776.rst b/peps/pep-0776.rst index 67e417301c4..9cc3e63db42 100644 --- a/peps/pep-0776.rst +++ b/peps/pep-0776.rst @@ -30,9 +30,9 @@ Motivation A web browser is a universal computing platform, available on Windows, Mac, Linux, and every smart phone. -The Pyodide project has for years supported Emscripten Python. Hundreds of -thousands of students have learned Python through Pyodide via projects like -`Capytale +`The Pyodide project `__ has for years supported +Emscripten Python. Hundreds of thousands of students have learned Python through +Pyodide via projects like `Capytale `__ and `PyodideU `__. Pyodide is also increasingly being used by Python packages to provide interactive From 9066665638710c5dd060b362303c4b6878a90be5 Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Tue, 18 Mar 2025 13:42:14 +0100 Subject: [PATCH 20/44] Update goal text --- peps/pep-0776.rst | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/peps/pep-0776.rst b/peps/pep-0776.rst index 9cc3e63db42..eeefbd71d53 100644 --- a/peps/pep-0776.rst +++ b/peps/pep-0776.rst @@ -56,15 +56,21 @@ Runtime Goals 1. To describe the current state of the CPython Emscripten runtime 2. To describe the current state of the Pyodide runtime -3. To identify features to be upstreamed from the Pyodide runtime into the +3. To identify minor features to be upstreamed from the Pyodide runtime into the CPython Emscripten runtime +The minor features identified here are intended to all be features that could be +implemented without a PEP. We discuss more significant runtime features that we +would like to implement but we defer decisions on those features to subsequent +PEPs. + --------------- Packaging Goals --------------- 1. To describe Pyodide's packaging tooling -2. To describe our wheel abi in a similar level of detail as the manylinux peps +2. To describe Pyodide's wheel abi to a similar level of detail as the manylinux + peps 3. To request that Pyodide wheels be allowed for upload to pypi From 6fa0c8ad6eb942d68b74123047b053549a20c785 Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Tue, 18 Mar 2025 13:43:08 +0100 Subject: [PATCH 21/44] Minor edit --- peps/pep-0776.rst | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/peps/pep-0776.rst b/peps/pep-0776.rst index eeefbd71d53..fcc019f2372 100644 --- a/peps/pep-0776.rst +++ b/peps/pep-0776.rst @@ -59,10 +59,9 @@ Runtime Goals 3. To identify minor features to be upstreamed from the Pyodide runtime into the CPython Emscripten runtime -The minor features identified here are intended to all be features that could be -implemented without a PEP. We discuss more significant runtime features that we -would like to implement but we defer decisions on those features to subsequent -PEPs. +The minor features identified here are all features that could be implemented +without a PEP. We discuss more significant runtime features that we would like +to implement but we defer decisions on those features to subsequent PEPs. --------------- Packaging Goals From 9b1eec2dc320b60b7abb7a39f962811dbc10318a Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Tue, 18 Mar 2025 14:15:02 +0100 Subject: [PATCH 22/44] Apply hugovk's copy edit Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> --- .github/CODEOWNERS | 1 - peps/pep-0776.rst | 92 +++++++++++++++++++++++----------------------- 2 files changed, 45 insertions(+), 48 deletions(-) diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index c87858f39e0..5c50eff6393 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -655,7 +655,6 @@ peps/pep-0773.rst @zooba peps/pep-0774.rst @savannahostrowski peps/pep-0775.rst @encukou peps/pep-0776.rst @hoodmane @ambv -# ... peps/pep-0777.rst @warsaw # ... peps/pep-0779.rst @Yhg1s @colesbury @mpage diff --git a/peps/pep-0776.rst b/peps/pep-0776.rst index fcc019f2372..62fe2e4879e 100644 --- a/peps/pep-0776.rst +++ b/peps/pep-0776.rst @@ -16,7 +16,7 @@ Abstract `Emscripten `__ is a complete open source compiler toolchain which compiles C/C++ code into WebAssembly/JavaScript executables that -run in JavaScript runtimes including browsers and Node. +run in JavaScript runtimes including browsers and Node.js. This PEP formalizes the addition of Tier 3 for Emscripten support in Python 3.14 which `was approved by the steering council on October 25, 2024 @@ -27,8 +27,8 @@ Pyodide wheels to be uploaded to PyPI. Motivation ========== -A web browser is a universal computing platform, available on Windows, Mac, -Linux, and every smart phone. +A web browser is a universal computing platform, available on Windows, macOS, +Linux, and every smartphone. `The Pyodide project `__ has for years supported Emscripten Python. Hundreds of thousands of students have learned Python through @@ -39,7 +39,7 @@ is also increasingly being used by Python packages to provide interactive documentation. This demonstrates both the importance and the maturity of the Emscripten platform. -Emscripten and wasi are also the only supported platforms that offer any +Emscripten and WASI are also the only supported platforms that offer any meaningful sandboxing. ===== @@ -69,8 +69,8 @@ Packaging Goals 1. To describe Pyodide's packaging tooling 2. To describe Pyodide's wheel abi to a similar level of detail as the manylinux - peps -3. To request that Pyodide wheels be allowed for upload to pypi + PEPs +3. To request that Pyodide wheels be allowed for upload to PyPI =============================== @@ -93,12 +93,12 @@ specified here: * https://github.com/WebAssembly/tool-conventions/blob/main/DynamicLinking.md The ``emcc`` compiler is a wrapper around ``clang``. The ``emcc`` linker is a -wrapper around ``wasm-ld`` (also part of the llvm toolchain). +wrapper around ``wasm-ld`` (also part of the LLVM toolchain). Emscripten support for portable C/C++ code source compatibility with Linux is fairly comprehensive, with certain expected exceptions to be spelled out. CPython already supports compilation to Emscripten, and it only requires a very -modest number of modifications to the normal linux target. +modest number of modifications to the normal Linux target. ---------------- POSIX Compliance @@ -113,15 +113,15 @@ support for ``fork()``. See `Emscripten Portability Guidelines Emscripten executables can be linked with threading support, however, it comes with several limitations: -Enabling threading requires websites to be served with special security headers -that indicate acceptance of the possibility of spectre-style information -leakage. These headers are a usability hazard for users who are not intimately -familiar with the web platform. +* Enabling threading requires websites to be served with special security headers + that indicate acceptance of the possibility of spectre-style information + leakage. These headers are a usability hazard for users who are not intimately + familiar with the web platform. -If an executable is linked with both threading and a dynamic loader, Emscripten -prints a warning that using dynamic loading and pthreads together is -experimental. It may cause performance problems or crashes. These problems may -require WebAssembly standards work to resolve. +* If an executable is linked with both threading and a dynamic loader, Emscripten + prints a warning that using dynamic loading and pthreads together is + experimental. It may cause performance problems or crashes. These problems may + require WebAssembly standards work to resolve. Because of these limitations, Pyodide standardizes a no-pthreads build of Python. We will start with support for no-pthreads builds. If there is @@ -162,9 +162,9 @@ macOS. Upstream tools: * emcc is a C and C++ compiler, linker, and a sysroot with headers for the system libraries. The system libraries themselves are generated on the fly based on the ABI requested. -* NodeJS can be used as an "emulator" to run Emscripten programs from the - command line. This emulation behaves best on Linux with Mac as a runner up. - Node is the most convenient way to test Emscripten programs. +* Node.js can be used as an "emulator" to run Emscripten programs from the + command line. This emulation behaves best on Linux with macOS as a runner up. + Node.js is the most convenient way to test Emscripten programs. * It is possible to run Emscripten programs inside of any web browser. Browser automation tools like selenium, playwright, or pupeeteer can be used to test features that are browser-only. @@ -172,17 +172,17 @@ macOS. Upstream tools: Pyodide's tools: * ``pyodide build`` can be used to cross compile Python packages to run on - Emscripten. Cross compilation works best on Linux, sometimes on Macs, and is + Emscripten. Cross compilation works best on Linux, sometimes on macOS, and is unsupported on Windows. * ``pyodide venv`` can make a virtual environment that runs in Pyodide. -* ``pytest-pyodide`` can test Python code against various JS runtimes. +* ``pytest-pyodide`` can test Python code against various JavaScript runtimes. cibuildwheel supports building wheels to target Emscripten using ``pyodide build``. In the short term, Pyodide's packaging tooling will stay in the Pyodide repository. It is an open question where Pyodide's packaging tooling should live in the long term. Two sensible options would be for it to remain under the -``pyodide`` organization or be moved into the ``pypa`` organization. +``pyodide`` organization or be moved into the ``pypa`` organization on GitHub. -------------------------------- @@ -240,7 +240,7 @@ Emscripten file system. There are several possible approaches to this: embeds the files into a custom archive format that cannot be processed with standard tooling. -* For Node, use the NODEFS to mount a native directory with the files into the +* For Node.js, use the NODEFS to mount a native directory with the files into the Emscripten file system. This is the most efficient option but is Node only. It is closely analogous to what wasi does. @@ -278,21 +278,21 @@ replace them in a ``preRun`` hook. Making ``stdin`` work correctly in the browser poses an additional challenge because it is not allowed to block for user input in the main thread of the -browser. If Emscripten is run in a webworker and served with the shared memory +browser. If Emscripten is run in a web worker and served with the shared memory headers, it is possible to receive input using shared memory and atomics. It is also possible for a ``stdin`` device to block in a simpler and more efficient manner using stack switching using the experimental JavaScript Promise Integration API. Pyodide replaces the standard I/O devices in order to fix the line buffering -behavior. When Pyodide is run in node, ``stdin``, ``stdout``, and ``stderr`` are +behavior. When Pyodide is run in Node.js, ``stdin``, ``stdout``, and ``stderr`` are by default connected to ``process.stdin``, ``process.stdout``, and ``process.stderr`` and so the standard streams work as a tty out of the box. Pyodide also ensures that ``shutil.get_terminal_size`` returns results consistent with ``process.stdout.rows`` and ``process.stdout.columns``. Pyodide currently has no support for stack switching ``stdin``. -Currently, the Emscripten Python node runner uses the default I/O that +Currently, the Emscripten Python Node.js runner uses the default I/O that Emscripten provides. The web example uses ``Atomics`` for ``stdin`` and has custom ``stdout`` and ``stderr`` handlers, but they exhibit the undesirable line buffering behavior. We will upstream the standard streams behaviors from @@ -311,7 +311,7 @@ Main Thread Synchronous Loading Limit In the main browser thread, a dynamic library can only be loaded synchronously if it is at most 4 kilobytes. This excludes most nontrivial dynamic libraries. -This limit is not present in Node and can be avoided by using a web worker. If +This limit is not present in Node.js and can be avoided by using a web worker. If stack switching is available, then it is possible to make ``dlopen()`` stack switch in order to instantiate a dynamic library synchronously. @@ -394,12 +394,12 @@ very useful. Compiling and linking with ``-g2`` (or a higher debug setting) ensures that WebAssembly symbols are included and they will appear in the traceback. -Because Emscripten Python currently has no JS API and no foreign function -interface, the situation is much simpler. The Python node runner wraps the call +Because Emscripten Python currently has no JavaScript API and no foreign function +interface, the situation is much simpler. The Python Node.js runner wraps the call to ``bootstrapEmscriptenExecutable()`` in a try/catch block. If an exception is caught, it displays the JavaScript exception and calls ``_Py_DumpTraceback()``. It then exits with code 1. We will stick with this approach until we add either -a JS API or foreign function interface, which is out of scope for this PEP. +a JavaScript API or foreign function interface, which is out of scope for this PEP. ============= Specification @@ -448,7 +448,7 @@ Removed Modules --------------- The following modules are removed from the standard library to reduce download -size and since they currently wouldn't work in the WebAssembly VM, +size and since they currently wouldn't work in the WebAssembly VM. - curses - dbm @@ -456,7 +456,6 @@ size and since they currently wouldn't work in the WebAssembly VM, - fcntl - grp - idlelib -- lib2to3 - msvcrt - pwd - resource @@ -496,7 +495,7 @@ name is justified. This is consistent with the return value from ``os.uname()``. There is also ``sys._emscripten_info`` which includes the Emscripten version and the runtime (either ``navigator.userAgent`` in a browser or ``"Node js" + -process.version`` in Node). +process.version`` in Node.js). --------------- Signals Support @@ -508,7 +507,7 @@ so it is impossible for any thread capable of seeing an interrupt to write to the eval breaker while the Python interpreter is running code. To work around this, there are two possible solutions: -* If Emscripten is run in a webworker and served with the shared memory headers, +* If Emscripten is run in a web worker and served with the shared memory headers, it is possible to use shared memory outside of the WebAssembly address space as a signal buffer. A signal handling UI thread can write the desired signal into the signal buffer. The interpreter can periodically check the state of @@ -596,7 +595,7 @@ CI Resources ------------ Pyodide can be built and tested on any Linux with a reasonably recent version of -Node. Anaconda has offered to provide physical hardware to run Android +Node.js. Anaconda has offered to provide physical hardware to run Emscripten buildbots, maintained by Russell Keith-Magee. CPython does not currently test Tier 3 platforms on GitHub Actions, but if this @@ -610,12 +609,12 @@ Existing Package Support ======================== Pyodide currently maintains ports of 255 different packages at the time of this -writing, including major scientific Python packages like numpy, scipy, pandas, -polars, scikit-learn, opencv, pyarrow, and Pillow as well as general purpose -packages like aiohttp, requests, pydantic, cryptography, and orjson. +writing, including major scientific Python packages like NumPy, SciPy, pandas, +Polars, scikit-learn, OpenCV, PyArrow, and Pillow as well as general purpose +packages like aiohttp, Requests, Pydantic, cryptography, and orjson. -About 60 packages are also testing against Pyodide in their CI, including numpy, -pandas, awkward-cpp, scikit-image, statsmodels, pyarrow, hypothesis, and PyO3. +About 60 packages are also testing against Pyodide in their CI, including NumPy, +pandas, awkward-cpp, scikit-image, statsmodels, PyArrow, Hypothesis, and PyO3. Emscripten Wheel Format ======================= @@ -650,19 +649,18 @@ Pyodide ABI, it is not necessary to use the Pyodide distribution itself. The ``pyodide build`` tool knows how to create wheels that match our ABI. As an alternative, `the auditwheel-emscripten tool `__ - is capable of performing basic compatibility checks, vendoring shared libraries, and retagging the wheel from ``emscripten_`` to ``pyodide_``. Unlike -with manylinux, there is no need for a docker container to build the +with manylinux, there is no need for a Docker container to build the ``pyodide_`` wheels. All that is needed is a Linux machine and appropriate -versions of Python, node, and Emscripten. +versions of Python, Node.js, and Emscripten. ------ PEP 11 ------ -PEP 11 will be updated to indicate that Emscripten is supported. Specifically +:pep:`11` will be updated to indicate that Emscripten is supported. Specifically the triples ``wasm32-unknown-emscripten_xx_xx_xx``. Russell Keith-Magee will serve as the initial core team contact for these ABIs. @@ -676,7 +674,7 @@ Future Work Improving Cross Builds in the Packaging Ecosystem ------------------------------------------------- -Python now supports four non-self-hosting platforms: iOS, Android, wasi, and +Python now supports four non-self-hosting platforms: iOS, Android, WASI, and Emscripten. All of them will need to build packages via cross builds. Currently, ``pyodide-build`` allows building a very large number of Python packages for Emscripten, but it relies on a giant pile of hacks. In the long run, we would @@ -695,7 +693,7 @@ JavaScript API for Bootstrapping ================================ Currently we offer no stable API for bootstrapping Python. Instead, we use one -collection of settings for the Node cli entrypoint and a separate collection of +collection of settings for the Node.js CLI entrypoint and a separate collection of settings for the browser demo. https://github.com/python/cpython/tree/98fa4a49fecbac3c990a25ce5d300592dad31be0/Tools/wasm/emscripten/node_entry.mjs https://github.com/python/cpython/blob/98fa4a49fecbac3c990a25ce5d300592dad31be0/Tools/wasm/emscripten/web_example/python.worker.mjs @@ -719,7 +717,7 @@ mapping between the JavaScript object model and the Python object model and a calling convention that allows high level bidirectional integration. Asyncio -======== +======= Most JavaScript primitives are asynchronous. The JavaScript thread that Python runs in already has an event loop. It it not too difficult to implement a Python From 5ff871c4c8b1dccf46f5d124cf668ce38324a015 Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Tue, 18 Mar 2025 15:05:45 +0100 Subject: [PATCH 23/44] Clarify minor Python releases --- peps/pep-0776.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0776.rst b/peps/pep-0776.rst index 62fe2e4879e..eb68b866dc3 100644 --- a/peps/pep-0776.rst +++ b/peps/pep-0776.rst @@ -145,7 +145,7 @@ update the ABI. In order to balance the ABI stability needs of package maintainers with the ABI flexibility to allow the platform to move forward, we plan to adopt a new ABI -for each minor release of Python. +for each minor (3.xx) release of Python. It is also necessary to coordinate the ABI flags that we use with the Emscripten ABI that Rust supports if we want support for the many popular Rust packages. From 5572f4dda6d8c4516802ec3e2a2328cd42831e2e Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Tue, 18 Mar 2025 15:06:07 +0100 Subject: [PATCH 24/44] More formal language about Pyodide's platform support --- peps/pep-0776.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/peps/pep-0776.rst b/peps/pep-0776.rst index eb68b866dc3..9e801209a5b 100644 --- a/peps/pep-0776.rst +++ b/peps/pep-0776.rst @@ -172,8 +172,8 @@ macOS. Upstream tools: Pyodide's tools: * ``pyodide build`` can be used to cross compile Python packages to run on - Emscripten. Cross compilation works best on Linux, sometimes on macOS, and is - unsupported on Windows. + Emscripten. Cross compilation works best on Linux, there is experimental + support on macOS, and it is entirely unsupported on Windows. * ``pyodide venv`` can make a virtual environment that runs in Pyodide. * ``pytest-pyodide`` can test Python code against various JavaScript runtimes. From f8994535b61233098629872df8c34891d3dc8da3 Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Tue, 18 Mar 2025 15:15:22 +0100 Subject: [PATCH 25/44] pages => packages --- peps/pep-0776.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0776.rst b/peps/pep-0776.rst index 9e801209a5b..0a84f453e63 100644 --- a/peps/pep-0776.rst +++ b/peps/pep-0776.rst @@ -317,7 +317,7 @@ switch in order to instantiate a dynamic library synchronously. To avoid the synchronous loading limit, Pyodide currently preloads all dynamic libraries present in a wheel when installing the wheel (or on startup). This is -a significant disadvantage with pages like scipy that include a very large +a significant disadvantage with packages like scipy that include a very large number of shared libraries that are expected to be only loaded on demand. Pyodide will implement a solution based on stack switching as it becomes more widely available in runtimes. From ef7d938c1120074d3a32905ed0dc9bbaa5b9ad25 Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Tue, 18 Mar 2025 15:36:05 +0100 Subject: [PATCH 26/44] Update peps/pep-0776.rst Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> --- peps/pep-0776.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0776.rst b/peps/pep-0776.rst index 0a84f453e63..1e5b86da8dc 100644 --- a/peps/pep-0776.rst +++ b/peps/pep-0776.rst @@ -145,7 +145,7 @@ update the ABI. In order to balance the ABI stability needs of package maintainers with the ABI flexibility to allow the platform to move forward, we plan to adopt a new ABI -for each minor (3.xx) release of Python. +for each feature release of Python. It is also necessary to coordinate the ABI flags that we use with the Emscripten ABI that Rust supports if we want support for the many popular Rust packages. From b7cf067f143bcd0c79fe87918b9f2e50ded0994d Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Tue, 18 Mar 2025 16:51:23 +0100 Subject: [PATCH 27/44] Apply Jelle's copy edits Co-authored-by: Jelle Zijlstra --- peps/pep-0776.rst | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/peps/pep-0776.rst b/peps/pep-0776.rst index 1e5b86da8dc..b62b7fbadc6 100644 --- a/peps/pep-0776.rst +++ b/peps/pep-0776.rst @@ -110,11 +110,11 @@ there are problems with networking APIs and blocking I/O, and there is no support for ``fork()``. See `Emscripten Portability Guidelines `__. -Emscripten executables can be linked with threading support, however, it comes +Emscripten executables can be linked with threading support, but it comes with several limitations: * Enabling threading requires websites to be served with special security headers - that indicate acceptance of the possibility of spectre-style information + that indicate acceptance of the possibility of `Spectre `__-style information leakage. These headers are a usability hazard for users who are not intimately familiar with the web platform. @@ -166,7 +166,7 @@ macOS. Upstream tools: command line. This emulation behaves best on Linux with macOS as a runner up. Node.js is the most convenient way to test Emscripten programs. * It is possible to run Emscripten programs inside of any web browser. Browser - automation tools like selenium, playwright, or pupeeteer can be used to test + automation tools like Selenium, Playwright, or Puppeteer can be used to test features that are browser-only. Pyodide's tools: @@ -249,7 +249,7 @@ Emscripten file system. There are several possible approaches to this: compression to the standard library itself. It also uses the more efficient native decompression algorithms of the browser rather than less efficient WebAssembly decompression. The disadvantage of this is a higher memory - footprint and it breaks ``inspect`` and various tests that do not expect the + footprint and it breaks :py:mod:`inspect` and various tests that do not expect the standard library to be packaged in this way. * Put the standard library into an uncompressed tar archive and mount it into a @@ -273,7 +273,7 @@ default to calling ``console.log`` and ``console.error`` respectively. It is possible to pass handlers to ``bootstrapEmscriptenExecutable()`` to configure the standard streams, but no matter what the I/O devices have undesirable line buffering behavior that forces a new line when flushed. To implement a well -behaved tty in browser, it is necessary to remove the default I/O devices and +behaved TTY in-browser, it is necessary to remove the default I/O devices and replace them in a ``preRun`` hook. Making ``stdin`` work correctly in the browser poses an additional challenge @@ -317,7 +317,7 @@ switch in order to instantiate a dynamic library synchronously. To avoid the synchronous loading limit, Pyodide currently preloads all dynamic libraries present in a wheel when installing the wheel (or on startup). This is -a significant disadvantage with packages like scipy that include a very large +a significant disadvantage with packages like SciPy that include a very large number of shared libraries that are expected to be only loaded on demand. Pyodide will implement a solution based on stack switching as it becomes more widely available in runtimes. @@ -660,7 +660,7 @@ versions of Python, Node.js, and Emscripten. PEP 11 ------ -:pep:`11` will be updated to indicate that Emscripten is supported. Specifically +:pep:`11` will be updated to indicate that Emscripten is supported, specifically the triples ``wasm32-unknown-emscripten_xx_xx_xx``. Russell Keith-Magee will serve as the initial core team contact for these ABIs. From d02bd488d59a2e710153faabc447d1e07e34e481 Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Tue, 18 Mar 2025 16:53:12 +0100 Subject: [PATCH 28/44] Update discuss links Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> --- peps/pep-0776.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/peps/pep-0776.rst b/peps/pep-0776.rst index b62b7fbadc6..4486d80f63c 100644 --- a/peps/pep-0776.rst +++ b/peps/pep-0776.rst @@ -2,13 +2,13 @@ PEP: 776 Title: Emscripten Support Author: Hood Chatham Sponsor: Łukasz Langa -Discussions-To: https://discuss.python.org/t/pep-776-emscripten-support/84996 +Discussions-To: https://discuss.python.org/t/84996 Status: Draft Type: Standards Track Topic: Packaging Created: 18-Mar-2025 Python-Version: 3.14 -Post-History: `18-Mar-2025 `__ +Post-History: `18-Mar-2025 `__ ======== Abstract From 29b3ce7390e40107d4d5bae5aafacf552bdeeccd Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Tue, 18 Mar 2025 16:57:37 +0100 Subject: [PATCH 29/44] better github link formatting --- peps/pep-0776.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/peps/pep-0776.rst b/peps/pep-0776.rst index 4486d80f63c..fb54246897f 100644 --- a/peps/pep-0776.rst +++ b/peps/pep-0776.rst @@ -692,11 +692,11 @@ the future. JavaScript API for Bootstrapping ================================ -Currently we offer no stable API for bootstrapping Python. Instead, we use one -collection of settings for the Node.js CLI entrypoint and a separate collection of -settings for the browser demo. -https://github.com/python/cpython/tree/98fa4a49fecbac3c990a25ce5d300592dad31be0/Tools/wasm/emscripten/node_entry.mjs -https://github.com/python/cpython/blob/98fa4a49fecbac3c990a25ce5d300592dad31be0/Tools/wasm/emscripten/web_example/python.worker.mjs +Currently we offer no stable API for bootstrapping Python. Instead, we use `one +collection of settings for the Node.js CLI entrypoint +`__ +and `a separate collection of settings for the browser demo +`__. The Emscripten executable startup API is complicated and there are many possible configurations that are broken. Pyodide offers a simpler set of options than From 9331b1524e7a967c2d8bbc53426c6a0114878ea5 Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Tue, 18 Mar 2025 17:04:20 +0100 Subject: [PATCH 30/44] Add text about Pyodide package loading as well as the stdlib --- peps/pep-0776.rst | 34 +++++++++++++++++++++++++++++++--- 1 file changed, 31 insertions(+), 3 deletions(-) diff --git a/peps/pep-0776.rst b/peps/pep-0776.rst index fb54246897f..5e3505a0b97 100644 --- a/peps/pep-0776.rst +++ b/peps/pep-0776.rst @@ -230,6 +230,9 @@ important of these are as follows: File System Setup ----------------- +The Standard Library +==================== + In order for Python to run, it needs access to the standard library in the Emscripten file system. There are several possible approaches to this: @@ -258,10 +261,35 @@ Emscripten file system. There are several possible approaches to this: in the browser. The disadvantage is that Emscripten does not itself include a TARFS so it requires a downstream implementation. -Pyodide uses the ``ZipImporter`` approach in every runtime. +Pyodide uses the ``ZipImporter`` approach in every runtime. Python uses the +NODEFS approach when run with node and the ``ZipImporter`` approach for the web +example. We will continue with this approach. + +The `ZipImporter` provides a clean resolution for a bootstrapping problem: The +Python runtime is capable of unpacking a wide variety of archive formats, but +the Python runtime is not ready to use until the standard library is already +available. Since `zipimport.py` is a frozen module, it avoids these problems. +All of the other approaches solve the bootstrapping problem by setting up the +standard library using JavaScript. + +Packages +======== -Python uses the NODEFS approach when run with node and the ``ZipImporter`` -approach for the web example. We will continue with this approach. +It is also necessary to make any needed packages available in the Emscripten +file system. Currently Emscripten cpython has no support for packages. Pyodide +uses two different approaches for packages: + +* In the browser, Pyodide downloads and unpacks wheels into the MEMFS + site-packages directory. It then preloads all dynamic libraries in the wheel. + The work of downloading and installing all the packages is redone every time + the runtime starts. + +* The Pyodide `python` cli entrypoint mounts all of the host file system as + NODEFS directories before it bootstraps Python. This allows the normal virtual + environment mechanism to work. Pyodide virtual environments contain a patched + copy of pip and a custom `pip.conf` so that pip will install Pyodide wheels. + On startup the Pyodide `python` cli will preload all Emscripten dynamic + libraries that are in the sitee-packages directory. ----------------------------- From d7caa649075a466a86d011eb0f77de2bc99da095 Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Tue, 18 Mar 2025 17:08:10 +0100 Subject: [PATCH 31/44] Fix single backticks --- peps/pep-0776.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/peps/pep-0776.rst b/peps/pep-0776.rst index 5e3505a0b97..e34c0982bd2 100644 --- a/peps/pep-0776.rst +++ b/peps/pep-0776.rst @@ -265,10 +265,10 @@ Pyodide uses the ``ZipImporter`` approach in every runtime. Python uses the NODEFS approach when run with node and the ``ZipImporter`` approach for the web example. We will continue with this approach. -The `ZipImporter` provides a clean resolution for a bootstrapping problem: The +The ``ZipImporter`` provides a clean resolution for a bootstrapping problem: The Python runtime is capable of unpacking a wide variety of archive formats, but the Python runtime is not ready to use until the standard library is already -available. Since `zipimport.py` is a frozen module, it avoids these problems. +available. Since ``zipimport.py`` is a frozen module, it avoids these problems. All of the other approaches solve the bootstrapping problem by setting up the standard library using JavaScript. @@ -284,11 +284,11 @@ uses two different approaches for packages: The work of downloading and installing all the packages is redone every time the runtime starts. -* The Pyodide `python` cli entrypoint mounts all of the host file system as +* The Pyodide ``python`` cli entrypoint mounts all of the host file system as NODEFS directories before it bootstraps Python. This allows the normal virtual environment mechanism to work. Pyodide virtual environments contain a patched - copy of pip and a custom `pip.conf` so that pip will install Pyodide wheels. - On startup the Pyodide `python` cli will preload all Emscripten dynamic + copy of pip and a custom ``pip.conf`` so that pip will install Pyodide wheels. + On startup the Pyodide ``python`` cli will preload all Emscripten dynamic libraries that are in the sitee-packages directory. From a8e6d5e49c84af8e899435423cc826bcbdf1d8ab Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Tue, 18 Mar 2025 17:14:39 +0100 Subject: [PATCH 32/44] Update working about rust support --- peps/pep-0776.rst | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/peps/pep-0776.rst b/peps/pep-0776.rst index e34c0982bd2..a73fe83f219 100644 --- a/peps/pep-0776.rst +++ b/peps/pep-0776.rst @@ -147,8 +147,11 @@ In order to balance the ABI stability needs of package maintainers with the ABI flexibility to allow the platform to move forward, we plan to adopt a new ABI for each feature release of Python. -It is also necessary to coordinate the ABI flags that we use with the Emscripten -ABI that Rust supports if we want support for the many popular Rust packages. +The Pyodide team also coordinates the ABI flags that Pyodide uses with the +Emscripten ABI that Rust supports in order to ensure that we have support for +the many popular Rust packages. Historically, most of the work for this has +been related to unwinding ABIs. See for instance `this Rust Major change +proposal `__. ----------------- Development Tools From 4d55c1f01ad12e8976940ab8aef3458a41363216 Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Tue, 18 Mar 2025 17:15:13 +0100 Subject: [PATCH 33/44] Replace another we ==> pyodide --- peps/pep-0776.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/peps/pep-0776.rst b/peps/pep-0776.rst index a73fe83f219..8a16117c16c 100644 --- a/peps/pep-0776.rst +++ b/peps/pep-0776.rst @@ -144,8 +144,8 @@ web platform features. Thus, there is significant benefit to being able to update the ABI. In order to balance the ABI stability needs of package maintainers with the ABI -flexibility to allow the platform to move forward, we plan to adopt a new ABI -for each feature release of Python. +flexibility to allow the platform to move forward, Pyodide plans to adopt a new +ABI for each feature release of Python. The Pyodide team also coordinates the ABI flags that Pyodide uses with the Emscripten ABI that Rust supports in order to ensure that we have support for From 0717162c1e92b1b2b0c61264ce0c8eb7a15fb0ce Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Tue, 18 Mar 2025 20:36:18 +0100 Subject: [PATCH 34/44] Fix uncaught exceptions section --- peps/pep-0776.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/peps/pep-0776.rst b/peps/pep-0776.rst index 8a16117c16c..ec53b930a44 100644 --- a/peps/pep-0776.rst +++ b/peps/pep-0776.rst @@ -387,8 +387,9 @@ compatible version of Emscripten, it will automatically pick up support for wheels with vendored dynamic libraries. +----------------------------- Traps and Uncaught Exceptions -============================= +----------------------------- We consider the C runtime state to be corrupted if there is a WebAssembly trap, an unhandled JavaScript exception, or an uncaught WebAssembly throw instruction. From cbbe9b9215c402428a8c1f5ef88e6c804cbc95b6 Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Tue, 18 Mar 2025 21:08:46 +0100 Subject: [PATCH 35/44] Update text about cross compilation --- peps/pep-0776.rst | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/peps/pep-0776.rst b/peps/pep-0776.rst index ec53b930a44..63678e65049 100644 --- a/peps/pep-0776.rst +++ b/peps/pep-0776.rst @@ -709,8 +709,10 @@ Improving Cross Builds in the Packaging Ecosystem Python now supports four non-self-hosting platforms: iOS, Android, WASI, and Emscripten. All of them will need to build packages via cross builds. Currently, ``pyodide-build`` allows building a very large number of Python packages for -Emscripten, but it relies on a giant pile of hacks. In the long run, we would -like to make the packaging ecosystem support cross builds in a sane way. +Emscripten, but it is very complicated. Ideally, the Python packaging ecosystem +would have standards for cross builds. This is a difficult long term project, +particularly because the packaging system is complex and was designed from the +ground up with the assumption that cross compilation would not happen. ----------------------------------------- From b9a8ae2e96d67333e972b0fafa949e55093edfc6 Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Tue, 18 Mar 2025 21:16:45 +0100 Subject: [PATCH 36/44] Revert "Update headings" This reverts commit 0a8d64ed714a9e718ef01bc0b5ad972f80f910d1. --- peps/pep-0776.rst | 60 +++++++++++------------------------------------ 1 file changed, 14 insertions(+), 46 deletions(-) diff --git a/peps/pep-0776.rst b/peps/pep-0776.rst index 63678e65049..8422f16a21e 100644 --- a/peps/pep-0776.rst +++ b/peps/pep-0776.rst @@ -10,7 +10,6 @@ Created: 18-Mar-2025 Python-Version: 3.14 Post-History: `18-Mar-2025 `__ -======== Abstract ======== @@ -23,7 +22,6 @@ which `was approved by the steering council on October 25, 2024 `__. The goal is to allow Pyodide wheels to be uploaded to PyPI. -========== Motivation ========== @@ -42,7 +40,6 @@ Emscripten platform. Emscripten and WASI are also the only supported platforms that offer any meaningful sandboxing. -===== Goals ===== @@ -50,7 +47,6 @@ It is our long term goal to upstream the entire Pyodide runtime into CPython, but this is out of scope for the present PEP. This PEP only attempts to establish the foundation for future work. -------------- Runtime Goals ------------- @@ -63,7 +59,6 @@ The minor features identified here are all features that could be implemented without a PEP. We discuss more significant runtime features that we would like to implement but we defer decisions on those features to subsequent PEPs. ---------------- Packaging Goals --------------- @@ -73,11 +68,9 @@ Packaging Goals 3. To request that Pyodide wheels be allowed for upload to PyPI -=============================== Emscripten Platform Information =============================== ------------------------- Background on Emscripten ------------------------ @@ -100,7 +93,6 @@ fairly comprehensive, with certain expected exceptions to be spelled out. CPython already supports compilation to Emscripten, and it only requires a very modest number of modifications to the normal Linux target. ----------------- POSIX Compliance ---------------- @@ -127,7 +119,6 @@ Because of these limitations, Pyodide standardizes a no-pthreads build of Python. We will start with support for no-pthreads builds. If there is sufficient demand, a pthreads build with no dynamic loader could be added later. ----------------------------- Emscripten ABI Compatibility ---------------------------- @@ -153,7 +144,6 @@ the many popular Rust packages. Historically, most of the work for this has been related to unwinding ABIs. See for instance `this Rust Major change proposal `__. ------------------ Development Tools ----------------- @@ -188,7 +178,6 @@ in the long term. Two sensible options would be for it to remain under the ``pyodide`` organization or be moved into the ``pypa`` organization on GitHub. --------------------------------- Emscripten Application Lifecycle -------------------------------- @@ -228,13 +217,12 @@ important of these are as follows: that can be performed in parallel to WebAssembly compilation. Implementing this callback allows performing all of these in parallel. - ------------------ File System Setup ----------------- The Standard Library -==================== +~~~~~~~~~~~~~~~~~~~~ + In order for Python to run, it needs access to the standard library in the Emscripten file system. There are several possible approaches to this: @@ -276,7 +264,7 @@ All of the other approaches solve the bootstrapping problem by setting up the standard library using JavaScript. Packages -======== +~~~~~~~~ It is also necessary to make any needed packages available in the Emscripten file system. Currently Emscripten cpython has no support for packages. Pyodide @@ -295,7 +283,6 @@ uses two different approaches for packages: libraries that are in the sitee-packages directory. ------------------------------ Console and Interactive Usage ----------------------------- @@ -333,12 +320,11 @@ In the long term, we hope to implement stack switching ``stdin`` devices, but that is out of scope for this PEP. ------------------ Dynamic Libraries ----------------- Main Thread Synchronous Loading Limit -===================================== +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ In the main browser thread, a dynamic library can only be loaded synchronously if it is at most 4 kilobytes. This excludes most nontrivial dynamic libraries. @@ -363,7 +349,7 @@ In the long run, we hope to implement a stack switching ``dlopen``, but that is out of scope for this PEP. Missing RPATH Support -===================== +~~~~~~~~~~~~~~~~~~~~~ Another important limitation of the Emscripten dynamic loader is that it does not currently have RPATH support. Pyodide's present workaround is as follows: @@ -387,7 +373,6 @@ compatible version of Emscripten, it will automatically pick up support for wheels with vendored dynamic libraries. ------------------------------ Traps and Uncaught Exceptions ----------------------------- @@ -433,11 +418,9 @@ caught, it displays the JavaScript exception and calls ``_Py_DumpTraceback()``. It then exits with code 1. We will stick with this approach until we add either a JavaScript API or foreign function interface, which is out of scope for this PEP. -============= Specification ============= -------------- Scope of Work ------------- @@ -454,7 +437,6 @@ macOS will be supported. A Python CLI entrypoint will be provided, which among other things can be used to execute the test suite. -------- Linkage ------- @@ -467,17 +449,16 @@ require special work to support. If a use case for dynamically linking the interpreter in Emscripten emerges, we can evaluate how much effort would be required to support it. ----------------- Standard Library ---------------- Unsupported Modules -=================== +~~~~~~~~~~~~~~~~~~~ See https://pyodide.org/en/stable/usage/wasm-constraints.html#removed-modules. Removed Modules ---------------- +^^^^^^^^^^^^^^^ The following modules are removed from the standard library to reduce download size and since they currently wouldn't work in the WebAssembly VM. @@ -501,7 +482,7 @@ size and since they currently wouldn't work in the WebAssembly VM. - winsound Included but not Working Modules --------------------------------- +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The following modules can be imported, but are not functional: @@ -519,7 +500,7 @@ termios package which has been removed: Platform Identification -======================= +~~~~~~~~~~~~~~~~~~~~~~~ ``sys.platform`` will return ``"emscripten"``. Although Emscripten attempts to be compatible with Linux, the differences are significant enough that a distinct @@ -529,7 +510,6 @@ There is also ``sys._emscripten_info`` which includes the Emscripten version and the runtime (either ``navigator.userAgent`` in a browser or ``"Node js" + process.version`` in Node.js). ---------------- Signals Support --------------- @@ -565,7 +545,6 @@ compatibility and because it is more efficient when it is possible. However, this is out of scope for this PEP. ----------------------- Function Pointer Casts ---------------------- @@ -622,7 +601,6 @@ The function pointer cast handling is fully implemented in cpython. Pyodide uses exactly the same code as upstream. ------------- CI Resources ------------ @@ -633,12 +611,11 @@ buildbots, maintained by Russell Keith-Magee. CPython does not currently test Tier 3 platforms on GitHub Actions, but if this ever changes, their Linux runners are able to build and test Emscripten Python. ---------- Packaging --------- Existing Package Support -======================== +~~~~~~~~~~~~~~~~~~~~~~~~ Pyodide currently maintains ports of 255 different packages at the time of this writing, including major scientific Python packages like NumPy, SciPy, pandas, @@ -649,7 +626,7 @@ About 60 packages are also testing against Pyodide in their CI, including NumPy, pandas, awkward-cpp, scikit-image, statsmodels, PyArrow, Hypothesis, and PyO3. Emscripten Wheel Format -======================= +~~~~~~~~~~~~~~~~~~~~~~~ Emscripten wheels will use either the format ``emscripten__wasm32`` or ``pyodide__wasm32``. For example: @@ -688,7 +665,6 @@ with manylinux, there is no need for a Docker container to build the versions of Python, Node.js, and Emscripten. ------- PEP 11 ------ @@ -698,11 +674,9 @@ the triples ``wasm32-unknown-emscripten_xx_xx_xx``. Russell Keith-Magee will serve as the initial core team contact for these ABIs. -=========== Future Work =========== -------------------------------------------------- Improving Cross Builds in the Packaging Ecosystem ------------------------------------------------- @@ -715,7 +689,6 @@ particularly because the packaging system is complex and was designed from the ground up with the assumption that cross compilation would not happen. ------------------------------------------ Pyodide Runtime Features to be Upstreamed ----------------------------------------- @@ -724,7 +697,7 @@ PEP and for the Python 3.14 development cycle but we would like to upstream in the future. JavaScript API for Bootstrapping -================================ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Currently we offer no stable API for bootstrapping Python. Instead, we use `one collection of settings for the Node.js CLI entrypoint @@ -742,7 +715,7 @@ Eventually, we would like to upstream Pyodide's bootstrapping API. In the short term, to keep things simple we will support no JavaScript API. FFI -=== +~~~ Because Emscripten supports POSIX, a significant number of tasks can be achieved using the ``os`` module. However, many fundamental operations in JavaScript @@ -751,7 +724,7 @@ mapping between the JavaScript object model and the Python object model and a calling convention that allows high level bidirectional integration. Asyncio -======= +~~~~~~~ Most JavaScript primitives are asynchronous. The JavaScript thread that Python runs in already has an event loop. It it not too difficult to implement a Python @@ -772,7 +745,6 @@ the same event loop as ordinary Python coroutines and are fully reentrant. This is fully implemented in Pyodide. -======================= Backwards Compatibility ======================= @@ -788,14 +760,12 @@ These backwards compatibility concerns impact not just the runtime but also the packaging system. -===================== Security Implications ===================== Adding a new platform does not add any new security implications. -================= How to Teach This ================= @@ -811,14 +781,12 @@ Second, developers of packages with binary components need to know how to build and release them for Emscripten (see Packaging). -======================== Reference Implementation ======================== Pyodide. -========= Copyright ========= From c912ad1efb0f83324fdecec64bfa3f7de7e9eb14 Mon Sep 17 00:00:00 2001 From: Jelle Zijlstra Date: Tue, 18 Mar 2025 16:34:32 -0700 Subject: [PATCH 37/44] Update peps/pep-0776.rst Co-authored-by: Brett Cannon --- peps/pep-0776.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0776.rst b/peps/pep-0776.rst index 8422f16a21e..5a14fbf8b51 100644 --- a/peps/pep-0776.rst +++ b/peps/pep-0776.rst @@ -280,7 +280,7 @@ uses two different approaches for packages: environment mechanism to work. Pyodide virtual environments contain a patched copy of pip and a custom ``pip.conf`` so that pip will install Pyodide wheels. On startup the Pyodide ``python`` cli will preload all Emscripten dynamic - libraries that are in the sitee-packages directory. + libraries that are in the site-packages directory. Console and Interactive Usage From bfbb818f1a27b8fdd40ca59fbe48d04b9b0772fe Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Wed, 19 Mar 2025 10:31:45 +0100 Subject: [PATCH 38/44] Apply AA-Turner's edits Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> --- peps/pep-0776.rst | 74 +++++++++++++++++++++++++---------------------- 1 file changed, 39 insertions(+), 35 deletions(-) diff --git a/peps/pep-0776.rst b/peps/pep-0776.rst index 5a14fbf8b51..971332a8cee 100644 --- a/peps/pep-0776.rst +++ b/peps/pep-0776.rst @@ -14,11 +14,11 @@ Abstract ======== `Emscripten `__ is a complete open source compiler -toolchain which compiles C/C++ code into WebAssembly/JavaScript executables that -run in JavaScript runtimes including browsers and Node.js. +toolchain. It compiles C/C++ code into WebAssembly/JavaScript executables, +for use in JavaScript runtimes, including browsers and Node.js. This PEP formalizes the addition of Tier 3 for Emscripten support in Python 3.14 -which `was approved by the steering council on October 25, 2024 +which `was approved by the Steering Council on October 25, 2024 `__. The goal is to allow Pyodide wheels to be uploaded to PyPI. @@ -65,7 +65,7 @@ Packaging Goals 1. To describe Pyodide's packaging tooling 2. To describe Pyodide's wheel abi to a similar level of detail as the manylinux PEPs -3. To request that Pyodide wheels be allowed for upload to PyPI +3. For Pyodide wheels to be allowed for upload to PyPI Emscripten Platform Information @@ -79,11 +79,11 @@ Background on Emscripten consists of a C and C++ compiler and linker based on LLVM, together with a runtime based on a mildly patched musl libc. -Emscripten is a POSIX-based platform. It uses the WebAssembly binary format, -specified here: +Emscripten is a POSIX-based platform. It uses the `WebAssembly binary format`_, +and the `WebAssembly dynamic linking section`_. -* https://webassembly.github.io/spec/core/binary/index.html -* https://github.com/WebAssembly/tool-conventions/blob/main/DynamicLinking.md +.. _WebAssembly binary format: https://webassembly.github.io/spec/core/binary/index.html +.. _WebAssembly dynamic linking section: https://github.com/WebAssembly/tool-conventions/blob/main/DynamicLinking.md The ``emcc`` compiler is a wrapper around ``clang``. The ``emcc`` linker is a wrapper around ``wasm-ld`` (also part of the LLVM toolchain). @@ -106,18 +106,20 @@ Emscripten executables can be linked with threading support, but it comes with several limitations: * Enabling threading requires websites to be served with special security headers - that indicate acceptance of the possibility of `Spectre `__-style information + that indicate acceptance of the possibility of Spectre_-style information leakage. These headers are a usability hazard for users who are not intimately familiar with the web platform. + .. _Spectre: https://en.wikipedia.org/wiki/Spectre_(security_vulnerability) + * If an executable is linked with both threading and a dynamic loader, Emscripten prints a warning that using dynamic loading and pthreads together is experimental. It may cause performance problems or crashes. These problems may require WebAssembly standards work to resolve. Because of these limitations, Pyodide standardizes a no-pthreads build of -Python. We will start with support for no-pthreads builds. If there is -sufficient demand, a pthreads build with no dynamic loader could be added later. +Python. If there is sufficient demand, a pthreads build with no dynamic loader +could be added later. Emscripten ABI Compatibility ---------------------------- @@ -141,20 +143,20 @@ ABI for each feature release of Python. The Pyodide team also coordinates the ABI flags that Pyodide uses with the Emscripten ABI that Rust supports in order to ensure that we have support for the many popular Rust packages. Historically, most of the work for this has -been related to unwinding ABIs. See for instance `this Rust Major change -proposal `__. +been related to unwinding ABIs. See for instance `this Rust Major Change +Proposal `__. Development Tools ----------------- Emscripten development tools are equally well supported on Linux, Windows, and -macOS. Upstream tools: +macOS. The upstream tools include: -* The Emscripten Software Developer Kit (emsdk) which can be used to install the - Emscripten compiler toolchain (emcc). -* emcc is a C and C++ compiler, linker, and a sysroot with headers for the - system libraries. The system libraries themselves are generated on the fly - based on the ABI requested. +* The Emscripten Software Developer Kit (:program:`emsdk`) which can be used to + install the Emscripten compiler toolchain (:program:`emcc`). +* :program:`emcc` is a C and C++ compiler, linker, and a sysroot with headers + for the system libraries. The system libraries themselves are generated on + the fly based on the ABI requested. * Node.js can be used as an "emulator" to run Emscripten programs from the command line. This emulation behaves best on Linux with macOS as a runner up. Node.js is the most convenient way to test Emscripten programs. @@ -170,7 +172,9 @@ Pyodide's tools: * ``pyodide venv`` can make a virtual environment that runs in Pyodide. * ``pytest-pyodide`` can test Python code against various JavaScript runtimes. -cibuildwheel supports building wheels to target Emscripten using ``pyodide build``. +cibuildwheel__ supports building wheels to target Emscripten using ``pyodide build``. + +__ https://cibuildwheel.pypa.io/ In the short term, Pyodide's packaging tooling will stay in the Pyodide repository. It is an open question where Pyodide's packaging tooling should live @@ -189,13 +193,13 @@ shut down the runtime on exit. It also includes an implementation for all of the system calls, including the file system, the dynamic loader, and any logic to expose additional functionality from the JavaScript runtime to C code. -The ``.mjs`` file exports a single ``bootstrapEmscriptenExecutable()`` +The ``.mjs`` file exports a single :js:func:`bootstrapEmscriptenExecutable()` JavaScript function that bootstraps the runtime, calls the ``main()`` function, and returns an API object that can be used to call C functions. Each time it is called produces a complete and independent copy of the runtime with its own separate address space. -The ``bootstrapEmscriptenExecutable()`` takes a large number of runtime +The :js:func:`bootstrapEmscriptenExecutable()` takes a large number of runtime settings. `The full list is described in the Emscripten documentation here. `__ The most important of these are as follows: @@ -242,8 +246,8 @@ Emscripten file system. There are several possible approaches to this: uncompressed zip file allows the web server and client to apply better compression to the standard library itself. It also uses the more efficient native decompression algorithms of the browser rather than less efficient - WebAssembly decompression. The disadvantage of this is a higher memory - footprint and it breaks :py:mod:`inspect` and various tests that do not expect the + WebAssembly decompression. The disadvantages of this are a higher memory + footprint and breaking :py:mod:`inspect` & various tests that do not expect the standard library to be packaged in this way. * Put the standard library into an uncompressed tar archive and mount it into a @@ -256,18 +260,18 @@ Pyodide uses the ``ZipImporter`` approach in every runtime. Python uses the NODEFS approach when run with node and the ``ZipImporter`` approach for the web example. We will continue with this approach. -The ``ZipImporter`` provides a clean resolution for a bootstrapping problem: The +The ``ZipImporter`` provides a clean resolution for a bootstrapping problem: the Python runtime is capable of unpacking a wide variety of archive formats, but the Python runtime is not ready to use until the standard library is already available. Since ``zipimport.py`` is a frozen module, it avoids these problems. All of the other approaches solve the bootstrapping problem by setting up the standard library using JavaScript. -Packages -~~~~~~~~ +Third-party packages +~~~~~~~~~~~~~~~~~~~~ It is also necessary to make any needed packages available in the Emscripten -file system. Currently Emscripten cpython has no support for packages. Pyodide +file system. Currently Emscripten CPython has no support for packages. Pyodide uses two different approaches for packages: * In the browser, Pyodide downloads and unpacks wheels into the MEMFS @@ -275,11 +279,11 @@ uses two different approaches for packages: The work of downloading and installing all the packages is redone every time the runtime starts. -* The Pyodide ``python`` cli entrypoint mounts all of the host file system as +* The Pyodide ``python`` CLI entrypoint mounts all of the host file system as NODEFS directories before it bootstraps Python. This allows the normal virtual environment mechanism to work. Pyodide virtual environments contain a patched copy of pip and a custom ``pip.conf`` so that pip will install Pyodide wheels. - On startup the Pyodide ``python`` cli will preload all Emscripten dynamic + On startup the Pyodide ``python`` CLI will preload all Emscripten dynamic libraries that are in the site-packages directory. @@ -288,7 +292,7 @@ Console and Interactive Usage ``stdin`` defaults to always returning ``EOF``, while ``stdout`` and ``stderr`` default to calling ``console.log`` and ``console.error`` respectively. It is -possible to pass handlers to ``bootstrapEmscriptenExecutable()`` to configure +possible to pass handlers to :js:func:`bootstrapEmscriptenExecutable()` to configure the standard streams, but no matter what the I/O devices have undesirable line buffering behavior that forces a new line when flushed. To implement a well behaved TTY in-browser, it is necessary to remove the default I/O devices and @@ -394,7 +398,7 @@ exceptions. This ensures that any recoverable JavaScript error is caught before it unwinds through any WebAssembly frames. All entrypoints to WebAssembly are also wrapped with JavaScript try/catch blocks. Any exceptions caught there have unwound WebAssembly frames and are thus considered to be fatal errors (though -there is a special case to handle ``exit()``). This requires foundational +there is a special case to handle :func:`~sys.exit()`). This requires foundational integration with the Python/JavaScript foreign function interface. When the Pyodide runtime catches a fatal exception, it introspects the error to @@ -413,7 +417,7 @@ traceback. Because Emscripten Python currently has no JavaScript API and no foreign function interface, the situation is much simpler. The Python Node.js runner wraps the call -to ``bootstrapEmscriptenExecutable()`` in a try/catch block. If an exception is +to :js:func:`bootstrapEmscriptenExecutable()` in a try/catch block. If an exception is caught, it displays the JavaScript exception and calls ``_Py_DumpTraceback()``. It then exits with code 1. We will stick with this approach until we add either a JavaScript API or foreign function interface, which is out of scope for this PEP. @@ -714,8 +718,8 @@ code duplication. Eventually, we would like to upstream Pyodide's bootstrapping API. In the short term, to keep things simple we will support no JavaScript API. -FFI -~~~ +JavaScript foreign function interface (FFI) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Because Emscripten supports POSIX, a significant number of tasks can be achieved using the ``os`` module. However, many fundamental operations in JavaScript From d6620f7c33cc7b8d34d28f7cadf332dcefe7b79c Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Wed, 19 Mar 2025 10:47:22 +0100 Subject: [PATCH 39/44] Add year for Pyodide history --- peps/pep-0776.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/peps/pep-0776.rst b/peps/pep-0776.rst index 971332a8cee..b427e5cce22 100644 --- a/peps/pep-0776.rst +++ b/peps/pep-0776.rst @@ -28,8 +28,8 @@ Motivation A web browser is a universal computing platform, available on Windows, macOS, Linux, and every smartphone. -`The Pyodide project `__ has for years supported -Emscripten Python. Hundreds of thousands of students have learned Python through +`The Pyodide project `__ has supported Emscripten Python +since 2018. Hundreds of thousands of students have learned Python through Pyodide via projects like `Capytale `__ and `PyodideU `__. Pyodide From 90b072e805959a8bff6c87c279ab45c2f263726c Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Wed, 19 Mar 2025 10:50:46 +0100 Subject: [PATCH 40/44] Add note about sandboxing to security implications section --- peps/pep-0776.rst | 3 +++ 1 file changed, 3 insertions(+) diff --git a/peps/pep-0776.rst b/peps/pep-0776.rst index b427e5cce22..ce25cd794dc 100644 --- a/peps/pep-0776.rst +++ b/peps/pep-0776.rst @@ -769,6 +769,9 @@ Security Implications Adding a new platform does not add any new security implications. +Emscripten and WASI are also the only supported platforms that offer sandboxing. +If users wish to execute untrusted Python code or untrusted Python extension +modules, Emscripten provides a secure way for them to do that. How to Teach This ================= From 585506e15a2d196443332cbddb13b015b33a7c60 Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Wed, 19 Mar 2025 10:52:29 +0100 Subject: [PATCH 41/44] Apply two more AA-turner edits Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> --- peps/pep-0776.rst | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/peps/pep-0776.rst b/peps/pep-0776.rst index ce25cd794dc..baff216f939 100644 --- a/peps/pep-0776.rst +++ b/peps/pep-0776.rst @@ -76,9 +76,11 @@ Background on Emscripten `Emscripten `__ -consists of a C and C++ compiler and linker based on LLVM, together with a +consists of a C and C++ compiler and linker based on LLVM__, together with a runtime based on a mildly patched musl libc. +__ https://llvm.org/ + Emscripten is a POSIX-based platform. It uses the `WebAssembly binary format`_, and the `WebAssembly dynamic linking section`_. @@ -240,7 +242,9 @@ Emscripten file system. There are several possible approaches to this: * For Node.js, use the NODEFS to mount a native directory with the files into the Emscripten file system. This is the most efficient option but is Node only. It - is closely analogous to what wasi does. + is closely analogous to what WASI_ does. + + .. _WASI: https://wasi.dev/ * Put the standard library into a zip archive and use ``ZipImporter``. Using an uncompressed zip file allows the web server and client to apply better From 2f72e44ae7c9a2b810ef697317ec8fae0a998448 Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Wed, 19 Mar 2025 10:55:15 +0100 Subject: [PATCH 42/44] Fix lint --- peps/pep-0776.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0776.rst b/peps/pep-0776.rst index baff216f939..ddcffa0c743 100644 --- a/peps/pep-0776.rst +++ b/peps/pep-0776.rst @@ -402,7 +402,7 @@ exceptions. This ensures that any recoverable JavaScript error is caught before it unwinds through any WebAssembly frames. All entrypoints to WebAssembly are also wrapped with JavaScript try/catch blocks. Any exceptions caught there have unwound WebAssembly frames and are thus considered to be fatal errors (though -there is a special case to handle :func:`~sys.exit()`). This requires foundational +there is a special case to handle :func:`~sys.exit`). This requires foundational integration with the Python/JavaScript foreign function interface. When the Pyodide runtime catches a fatal exception, it introspects the error to From 114ccf17d2b638e4ff2e960ead2dcaebaff679ae Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Wed, 19 Mar 2025 11:34:30 +0100 Subject: [PATCH 43/44] Remove broken js:func refs --- peps/pep-0776.rst | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/peps/pep-0776.rst b/peps/pep-0776.rst index ddcffa0c743..4adec1a0ac0 100644 --- a/peps/pep-0776.rst +++ b/peps/pep-0776.rst @@ -14,8 +14,9 @@ Abstract ======== `Emscripten `__ is a complete open source compiler -toolchain. It compiles C/C++ code into WebAssembly/JavaScript executables, -for use in JavaScript runtimes, including browsers and Node.js. +toolchain. It compiles C/C++ code into WebAssembly/JavaScript executables, for +use in JavaScript runtimes, including browsers and Node.js. The Rust language +also maintains an Emscripten target. This PEP formalizes the addition of Tier 3 for Emscripten support in Python 3.14 which `was approved by the Steering Council on October 25, 2024 @@ -195,13 +196,13 @@ shut down the runtime on exit. It also includes an implementation for all of the system calls, including the file system, the dynamic loader, and any logic to expose additional functionality from the JavaScript runtime to C code. -The ``.mjs`` file exports a single :js:func:`bootstrapEmscriptenExecutable()` +The ``.mjs`` file exports a single ``bootstrapEmscriptenExecutable()`` JavaScript function that bootstraps the runtime, calls the ``main()`` function, and returns an API object that can be used to call C functions. Each time it is called produces a complete and independent copy of the runtime with its own separate address space. -The :js:func:`bootstrapEmscriptenExecutable()` takes a large number of runtime +The ``bootstrapEmscriptenExecutable()`` takes a large number of runtime settings. `The full list is described in the Emscripten documentation here. `__ The most important of these are as follows: @@ -296,7 +297,7 @@ Console and Interactive Usage ``stdin`` defaults to always returning ``EOF``, while ``stdout`` and ``stderr`` default to calling ``console.log`` and ``console.error`` respectively. It is -possible to pass handlers to :js:func:`bootstrapEmscriptenExecutable()` to configure +possible to pass handlers to ``bootstrapEmscriptenExecutable()`` to configure the standard streams, but no matter what the I/O devices have undesirable line buffering behavior that forces a new line when flushed. To implement a well behaved TTY in-browser, it is necessary to remove the default I/O devices and @@ -421,7 +422,7 @@ traceback. Because Emscripten Python currently has no JavaScript API and no foreign function interface, the situation is much simpler. The Python Node.js runner wraps the call -to :js:func:`bootstrapEmscriptenExecutable()` in a try/catch block. If an exception is +to ``bootstrapEmscriptenExecutable()`` in a try/catch block. If an exception is caught, it displays the JavaScript exception and calls ``_Py_DumpTraceback()``. It then exits with code 1. We will stick with this approach until we add either a JavaScript API or foreign function interface, which is out of scope for this PEP. From 93f262fa550f3b0ccb467b7471cf99e7beb90313 Mon Sep 17 00:00:00 2001 From: Hood Chatham Date: Wed, 19 Mar 2025 11:36:57 +0100 Subject: [PATCH 44/44] Add link to wasm-ld --- peps/pep-0776.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/peps/pep-0776.rst b/peps/pep-0776.rst index 4adec1a0ac0..0cf88b63507 100644 --- a/peps/pep-0776.rst +++ b/peps/pep-0776.rst @@ -89,7 +89,8 @@ and the `WebAssembly dynamic linking section`_. .. _WebAssembly dynamic linking section: https://github.com/WebAssembly/tool-conventions/blob/main/DynamicLinking.md The ``emcc`` compiler is a wrapper around ``clang``. The ``emcc`` linker is a -wrapper around ``wasm-ld`` (also part of the LLVM toolchain). +wrapper around ``wasm-ld`` (`also part of the LLVM toolchain +`__). Emscripten support for portable C/C++ code source compatibility with Linux is fairly comprehensive, with certain expected exceptions to be spelled out.