PEP 9999: add "Cross-compiling Python packages" PEP draft

Signed-off-by: Filipe Laíns <lains@riseup.net>

Author: FFY00

pep-9999.rst

@@ -0,0 +1,350 @@
+PEP: 9999
+Title: Cross-compiling Python packages
+Author: Filipe Laíns <lains@python.org>
+PEP-Delegate:
+Discussions-To: [URL]
+Status: Draft
+Type: Informational
+Content-Type: text/x-rst
+Created: 01-07-2023
+Python-Version: 3.12
+Post-History: [`DD-MMM-YYYY <URL>`__]
+Resolution:
+
+
+Abstract
+========
+
+This PEP attempts to document the status of cross-compilation of downstream
+projects, more specifically
+
+It should give an overview of the approaches currently used by distributors
+(Linux distros, WASM environment providers, etc.) to cross-compile downstream
+projects (3rd party extensions, etc.).
+
+
+Motivation
+==========
+
+We write this PEP to express the challenges in cross-compilation and act as a
+supporting document in future improvement proposals.
+
+
+Analysis
+========
+
+
+Introduction
+------------
+
+There are a couple different approaches being used to tackle this, with
+different levels of interaction required from the user, but they all require a
+significant ammount of effort. This is due to the lack of standardized
+cross-compilation infrastructure on the Python packaging ecosystem, which itself
+stems from the complexity of cross-builds, making it a huge undertaking.
+
+
+Upstream support
+----------------
+
+Some major projects like CPython, setuptools, etc. provide some support to help
+with cross-compilation, but it's unofficial and at a best-effort basis. For
+example, the ``sysconfig`` module allows overwriting the data module name via
+the ``_PYTHON_SYSCONFIGDATA_NAME`` environment variable, something that is
+required for cross-builds, and setuptools `accepts patches`__ [1]_ to tweak/fix
+its logic to be compatible with popular
+:ref:`"environment faking" <approach-target-environment>` workflows [2]_.
+
+The lack of first-party support in upstream projects leads to cross-compilation
+being fragile and requiring a significant effort from users, however, the lack
+of standardization makes it harder for upstreams to improve support, even if
+they wanted, as there's no clarity on how this feature should be provided.
+
+.. [1] At the time of writing (Jun 2023), setuptools' compiler interface code,
+       the compoment that most of affects cross-compilation, is developed on the
+       `pypa/distutils`__ repository, which gets periodically synced to the
+       setuptools repository.
+
+.. [2] We specifically mention *popular* workflows, because since this is not
+       standardized, there isn't a single unified approach. Though, many of the
+       most popular implementations (crossenv_, conda-forge_'s build system,
+       etc.) work similarly, and this is what we are refering to here. For
+       clarity, the implementations we are refering to here could be described
+       as *crossenv-style*.
+
+.. __: https://github.com/pypa/distutils/pulls?q=cross
+.. __: https://github.com/pypa/distutils
+
+Projects with decent cross-build support
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+It seems relevant to point out that there are a few modern Python package
+build-backends with, at least, decent cross-compilation support, those being
+scikit-build__ and meson-python__. Both these projects integrate external mature
+build-systems into Python packaging — CMake__ and Meson__, respectively — so
+cross-build support is inherited from them.
+
+.. __: https://github.com/scikit-build/scikit-build
+.. __: https://github.com/mesonbuild/meson-python
+.. __: https://cmake.org/
+.. __: https://mesonbuild.com/
+
+
+Downstream approaches
+---------------------
+
+Cross-compilation approaches fall in a spectrum that goes from, by design,
+requiring extensive user interaction to (ideally) almost none. Usually, they'll
+be based on one of two main strategies, using a
+:ref:`cross-build environment <approach-cross-environment>`, or
+:ref:`faking the target environment <approach-target-environment>`.
+
+.. _approach-cross-environment:
+
+Cross-build environment
+~~~~~~~~~~~~~~~~~~~~~~~
+
+This consists of running the Python interpreter normally and utilizing the
+cross-build provided by the projects' build-system. However, as we saw above,
+upstream support is pretty lacking, so this approach only works for a small-ish
+set of projects. When this fails, the usual strategy is to patch the
+build-system code to build use the correct toolchain, system details, etc. [3]_.
+
+Since this approach often requires package-specific patching, it requires a lot
+of user interaction.
+
+.. admonition:: Examples
+   :class: note
+
+   :ref:`python-for-android`, :ref:`kivy-ios`, etc.
+
+.. [3] The scope of the build-system patching varies between users and
+       usually depends on the their goal — some (eg. Linux distributions) may
+       patch the build-system to support cross-builds, while others might
+       hardcode compiler paths and system information in the build-system, to
+       simply make the build work.
+
+.. _approach-target-environment:
+
+Faking the target environment
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Aiming to drop the requirement for user input, a popular approach is trying to
+fake the target environment. It generally consists of monkeypatching the Python
+interpreter to get it to mimick the interpreter on the target system, which
+constitutes of changing many of the `sys` module attributes, the `sysconfig`
+data, etc. Using this strategy, build-backends do not need to have any
+cross-build support, and should just work without any code changes.
+
+Unfortunately, though, it isn't possible to truly fake the target environment.
+There are many reasons for this, one of the main ones being that it breaks code
+that actually needs to introspect the running interpreter. As a result,
+monkeypatching Python to look like target is very tricky — to achieve the less
+ammount of breakage, we can only patch certain aspects of the interpreter.
+Consequently, build-backends may need some code changes, but these are generally
+much smaller than the previous approach. This is an inherent limitation of the
+technique, meaning this strategy still requires some user interaction.
+
+Nonetheless, this strategy still works out-of-the-box with significantly more
+projects than the approach above, and requires much less effort in these cases.
+It is sucessfull in decreasing the ammount of user interaction needed, even
+though it doesn't succeed in being generic.
+
+.. admonition:: Examples
+   :class: note
+
+   :ref:`crossenv`, :ref:`conda-forge`, etc.
+
+
+Case studies
+============
+
+
+.. _crossenv:
+
+crossenv
+--------
+
+:Description: Virtual Environments for Cross-Compiling Python Extension Modules.
+:URL: https://github.com/benfogle/crossenv
+
+``crossenv`` is a tool to create a virtual environment with a monkeypatched
+Python installation that tries to emulate the target machine in certain
+scenarios. More about this approach can be found in the
+:ref:`approach-target-environment` section.
+
+
+.. _conda-forge:
+
+conda-forge
+-----------
+
+:Description: A community-led collection of recipes, build infrastructure and distributions for the conda package manager.
+:URL: https://conda-forge.org/
+
+XXX: Jaime will write a quick summary once the PEP draft is public.
+
+XXX
+Uses a modified crossenv.
+
+
+Yocto Project
+-------------
+
+:Description:  The Yocto Project is an open source collaboration project that helps developers create custom Linux-based systems regardless of the hardware architecture.
+:URL: https://www.yoctoproject.org/
+
+XXX: Sent email to the mailing list.
+
+TODO
+
+
+Buildroot
+---------
+
+:Description: Buildroot is a simple, efficient and easy-to-use tool to generate embedded Linux systems through cross-compilation.
+:URL: https://buildroot.org/
+
+TODO
+
+
+Pyodide
+-------
+
+:Description: Pyodide is a Python distribution for the browser and Node.js based on WebAssembly.
+:URL: https://pyodide.org/en/stable/
+
+XXX: Hood should review/expand this section.
+
+``Pyodide`` is a provides a Python distribution compied to WebAssembly__
+using the Emscripten__ toolchain.
+
+It patches several aspects of the CPython installation and some external
+components. A custom package manager — micropip__ — supporting both Pure and
+wasm32/Emscripten wheels, is also provided as a part of the distribution. On top
+of this, a repo with a `selected set of 3rd party packages`__ is also provided
+and enabled by default.
+
+.. __: https://webassembly.org/
+.. __: https://emscripten.org/
+.. __: https://micropip.pyodide.org/
+.. __: https://pyodide.org/en/stable/usage/packages-in-pyodide.html
+
+
+Beeware
+-------
+
+:Description: BeeWare allows you to write your app in Python and release it on multiple platforms.
+:URL: https://beeware.org/
+
+TODO
+
+
+.. _python-for-android:
+
+python-for-android
+------------------
+
+:Description: Turn your Python application into an Android APK.
+:URL: https://github.com/kivy/python-for-android
+
+resource https://github.com/Android-for-Python/Android-for-Python-Users
+
+``python-for-android`` is a tool to package Python apps on Android. It creates a
+Python distribution with your app and its dependencies.
+
+Pure-Python dependencies are handled automatically and in a generic way, but
+native dependencies need recipes__. A set of recipes for
+`popular dependencies`__ is provided, but users need to provide their own
+recipes for any other native dependencies.
+
+.. __: https://python-for-android.readthedocs.io/en/latest/recipes/
+.. __: https://github.com/kivy/python-for-android/tree/develop/pythonforandroid/recipes
+
+
+.. _kivy-ios:
+
+kivy-ios
+--------
+
+:Description:  Toolchain for compiling Python / Kivy / other libraries for iOS.
+:URL: https://github.com/kivy/kivy-ios
+
+``kivy-ios`` is a tool to package Python apps on iOS. It provides a toolchain to
+build a Python distribution with your app and its dependencies, as well as a CLI
+to create and manage Xcode projects that integrate with the toolchain.
+
+It uses the same approach as :ref:`python-for-android` (also maintained by the
+`Kivy project`__) for app dependencies — pure-Python dependencies are handled
+automatically, but native dependencies need recipes__, and the project provides
+recipes for `popular dependencies`__.
+
+.. __: https://kivy.org
+.. __: https://python-for-android.readthedocs.io/en/latest/recipes/
+.. __: https://github.com/kivy/kivy-ios/tree/master/kivy_ios/recipes
+
+
+AidLearning
+-----------
+
+:Description: AI, Android, Linux, ARM: AI application development platform based on Android+Linux integrated ecology.
+:URL: https://github.com/aidlearning/AidLearning-FrameWork
+
+TODO
+
+
+QPython
+-------
+
+:Description: QPython is the Python engine for android.
+:URL: https://github.com/qpython-android/qpython
+
+TODO
+
+
+pyqtdeploy
+----------
+
+:Description: pyqtdeploy is a tool for deploying PyQt applications.
+:URL: https://www.riverbankcomputing.com/software/pyqtdeploy/
+
+contact https://www.riverbankcomputing.com/pipermail/pyqt/2023-May/thread.html
+contacted Phil, the maintainer
+
+TODO
+
+
+Chaquopy
+--------
+
+:Description: Chaquopy provides everything you need to include Python components in an Android app.
+:URL: https://chaquo.com/chaquopy/
+
+TODO
+
+
+EDK II
+------
+
+:Description: EDK II is a modern, feature-rich, cross-platform firmware development environment for the UEFI and PI specifications.
+:URL: https://github.com/tianocore/edk2-libc/tree/master/AppPkg/Applications/Python
+
+TODO
+
+
+ActivePython
+------------
+
+:Description: Commercial-grade, quality-assured Python distribution focusing on easy installation and cross-platform compatibility on Windows, Linux, Mac OS X, Solaris, HP-UX and AIX.
+:URL: https://www.activestate.com/products/python/
+
+TODO
+
+
+Termux
+------
+
+:Description: Termux is an Android terminal emulator and Linux environment app that works directly with no rooting or setup required.
+:URL: https://termux.dev/en/
+
+TODO