From dc2f93fd83d5000578a0000be683fcde1d4a0503 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Filipe=20La=C3=ADns?= Date: Mon, 3 Jul 2023 23:11:59 +0100 Subject: [PATCH 01/11] PEP 9999: add "Cross-compiling Python packages" PEP draft MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Filipe Laíns --- pep-9999.rst | 350 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 350 insertions(+) create mode 100644 pep-9999.rst diff --git a/pep-9999.rst b/pep-9999.rst new file mode 100644 index 00000000000..00573c2174b --- /dev/null +++ b/pep-9999.rst @@ -0,0 +1,350 @@ +PEP: 9999 +Title: Cross-compiling Python packages +Author: Filipe Laíns +PEP-Delegate: +Discussions-To: [URL] +Status: Draft +Type: Informational +Content-Type: text/x-rst +Created: 01-07-2023 +Python-Version: 3.12 +Post-History: [`DD-MMM-YYYY `__] +Resolution: + + +Abstract +======== + +This PEP attempts to document the status of cross-compilation of downstream +projects, more specifically + +It should give an overview of the approaches currently used by distributors +(Linux distros, WASM environment providers, etc.) to cross-compile downstream +projects (3rd party extensions, etc.). + + +Motivation +========== + +We write this PEP to express the challenges in cross-compilation and act as a +supporting document in future improvement proposals. + + +Analysis +======== + + +Introduction +------------ + +There are a couple different approaches being used to tackle this, with +different levels of interaction required from the user, but they all require a +significant ammount of effort. This is due to the lack of standardized +cross-compilation infrastructure on the Python packaging ecosystem, which itself +stems from the complexity of cross-builds, making it a huge undertaking. + + +Upstream support +---------------- + +Some major projects like CPython, setuptools, etc. provide some support to help +with cross-compilation, but it's unofficial and at a best-effort basis. For +example, the ``sysconfig`` module allows overwriting the data module name via +the ``_PYTHON_SYSCONFIGDATA_NAME`` environment variable, something that is +required for cross-builds, and setuptools `accepts patches`__ [1]_ to tweak/fix +its logic to be compatible with popular +:ref:`"environment faking" ` workflows [2]_. + +The lack of first-party support in upstream projects leads to cross-compilation +being fragile and requiring a significant effort from users, however, the lack +of standardization makes it harder for upstreams to improve support, even if +they wanted, as there's no clarity on how this feature should be provided. + +.. [1] At the time of writing (Jun 2023), setuptools' compiler interface code, + the compoment that most of affects cross-compilation, is developed on the + `pypa/distutils`__ repository, which gets periodically synced to the + setuptools repository. + +.. [2] We specifically mention *popular* workflows, because since this is not + standardized, there isn't a single unified approach. Though, many of the + most popular implementations (crossenv_, conda-forge_'s build system, + etc.) work similarly, and this is what we are refering to here. For + clarity, the implementations we are refering to here could be described + as *crossenv-style*. + +.. __: https://github.com/pypa/distutils/pulls?q=cross +.. __: https://github.com/pypa/distutils + +Projects with decent cross-build support +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +It seems relevant to point out that there are a few modern Python package +build-backends with, at least, decent cross-compilation support, those being +scikit-build__ and meson-python__. Both these projects integrate external mature +build-systems into Python packaging — CMake__ and Meson__, respectively — so +cross-build support is inherited from them. + +.. __: https://github.com/scikit-build/scikit-build +.. __: https://github.com/mesonbuild/meson-python +.. __: https://cmake.org/ +.. __: https://mesonbuild.com/ + + +Downstream approaches +--------------------- + +Cross-compilation approaches fall in a spectrum that goes from, by design, +requiring extensive user interaction to (ideally) almost none. Usually, they'll +be based on one of two main strategies, using a +:ref:`cross-build environment `, or +:ref:`faking the target environment `. + +.. _approach-cross-environment: + +Cross-build environment +~~~~~~~~~~~~~~~~~~~~~~~ + +This consists of running the Python interpreter normally and utilizing the +cross-build provided by the projects' build-system. However, as we saw above, +upstream support is pretty lacking, so this approach only works for a small-ish +set of projects. When this fails, the usual strategy is to patch the +build-system code to build use the correct toolchain, system details, etc. [3]_. + +Since this approach often requires package-specific patching, it requires a lot +of user interaction. + +.. admonition:: Examples + :class: note + + :ref:`python-for-android`, :ref:`kivy-ios`, etc. + +.. [3] The scope of the build-system patching varies between users and + usually depends on the their goal — some (eg. Linux distributions) may + patch the build-system to support cross-builds, while others might + hardcode compiler paths and system information in the build-system, to + simply make the build work. + +.. _approach-target-environment: + +Faking the target environment +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Aiming to drop the requirement for user input, a popular approach is trying to +fake the target environment. It generally consists of monkeypatching the Python +interpreter to get it to mimick the interpreter on the target system, which +constitutes of changing many of the `sys` module attributes, the `sysconfig` +data, etc. Using this strategy, build-backends do not need to have any +cross-build support, and should just work without any code changes. + +Unfortunately, though, it isn't possible to truly fake the target environment. +There are many reasons for this, one of the main ones being that it breaks code +that actually needs to introspect the running interpreter. As a result, +monkeypatching Python to look like target is very tricky — to achieve the less +ammount of breakage, we can only patch certain aspects of the interpreter. +Consequently, build-backends may need some code changes, but these are generally +much smaller than the previous approach. This is an inherent limitation of the +technique, meaning this strategy still requires some user interaction. + +Nonetheless, this strategy still works out-of-the-box with significantly more +projects than the approach above, and requires much less effort in these cases. +It is sucessfull in decreasing the ammount of user interaction needed, even +though it doesn't succeed in being generic. + +.. admonition:: Examples + :class: note + + :ref:`crossenv`, :ref:`conda-forge`, etc. + + +Case studies +============ + + +.. _crossenv: + +crossenv +-------- + +:Description: Virtual Environments for Cross-Compiling Python Extension Modules. +:URL: https://github.com/benfogle/crossenv + +``crossenv`` is a tool to create a virtual environment with a monkeypatched +Python installation that tries to emulate the target machine in certain +scenarios. More about this approach can be found in the +:ref:`approach-target-environment` section. + + +.. _conda-forge: + +conda-forge +----------- + +:Description: A community-led collection of recipes, build infrastructure and distributions for the conda package manager. +:URL: https://conda-forge.org/ + +XXX: Jaime will write a quick summary once the PEP draft is public. + +XXX +Uses a modified crossenv. + + +Yocto Project +------------- + +:Description: The Yocto Project is an open source collaboration project that helps developers create custom Linux-based systems regardless of the hardware architecture. +:URL: https://www.yoctoproject.org/ + +XXX: Sent email to the mailing list. + +TODO + + +Buildroot +--------- + +:Description: Buildroot is a simple, efficient and easy-to-use tool to generate embedded Linux systems through cross-compilation. +:URL: https://buildroot.org/ + +TODO + + +Pyodide +------- + +:Description: Pyodide is a Python distribution for the browser and Node.js based on WebAssembly. +:URL: https://pyodide.org/en/stable/ + +XXX: Hood should review/expand this section. + +``Pyodide`` is a provides a Python distribution compied to WebAssembly__ +using the Emscripten__ toolchain. + +It patches several aspects of the CPython installation and some external +components. A custom package manager — micropip__ — supporting both Pure and +wasm32/Emscripten wheels, is also provided as a part of the distribution. On top +of this, a repo with a `selected set of 3rd party packages`__ is also provided +and enabled by default. + +.. __: https://webassembly.org/ +.. __: https://emscripten.org/ +.. __: https://micropip.pyodide.org/ +.. __: https://pyodide.org/en/stable/usage/packages-in-pyodide.html + + +Beeware +------- + +:Description: BeeWare allows you to write your app in Python and release it on multiple platforms. +:URL: https://beeware.org/ + +TODO + + +.. _python-for-android: + +python-for-android +------------------ + +:Description: Turn your Python application into an Android APK. +:URL: https://github.com/kivy/python-for-android + +resource https://github.com/Android-for-Python/Android-for-Python-Users + +``python-for-android`` is a tool to package Python apps on Android. It creates a +Python distribution with your app and its dependencies. + +Pure-Python dependencies are handled automatically and in a generic way, but +native dependencies need recipes__. A set of recipes for +`popular dependencies`__ is provided, but users need to provide their own +recipes for any other native dependencies. + +.. __: https://python-for-android.readthedocs.io/en/latest/recipes/ +.. __: https://github.com/kivy/python-for-android/tree/develop/pythonforandroid/recipes + + +.. _kivy-ios: + +kivy-ios +-------- + +:Description: Toolchain for compiling Python / Kivy / other libraries for iOS. +:URL: https://github.com/kivy/kivy-ios + +``kivy-ios`` is a tool to package Python apps on iOS. It provides a toolchain to +build a Python distribution with your app and its dependencies, as well as a CLI +to create and manage Xcode projects that integrate with the toolchain. + +It uses the same approach as :ref:`python-for-android` (also maintained by the +`Kivy project`__) for app dependencies — pure-Python dependencies are handled +automatically, but native dependencies need recipes__, and the project provides +recipes for `popular dependencies`__. + +.. __: https://kivy.org +.. __: https://python-for-android.readthedocs.io/en/latest/recipes/ +.. __: https://github.com/kivy/kivy-ios/tree/master/kivy_ios/recipes + + +AidLearning +----------- + +:Description: AI, Android, Linux, ARM: AI application development platform based on Android+Linux integrated ecology. +:URL: https://github.com/aidlearning/AidLearning-FrameWork + +TODO + + +QPython +------- + +:Description: QPython is the Python engine for android. +:URL: https://github.com/qpython-android/qpython + +TODO + + +pyqtdeploy +---------- + +:Description: pyqtdeploy is a tool for deploying PyQt applications. +:URL: https://www.riverbankcomputing.com/software/pyqtdeploy/ + +contact https://www.riverbankcomputing.com/pipermail/pyqt/2023-May/thread.html +contacted Phil, the maintainer + +TODO + + +Chaquopy +-------- + +:Description: Chaquopy provides everything you need to include Python components in an Android app. +:URL: https://chaquo.com/chaquopy/ + +TODO + + +EDK II +------ + +:Description: EDK II is a modern, feature-rich, cross-platform firmware development environment for the UEFI and PI specifications. +:URL: https://github.com/tianocore/edk2-libc/tree/master/AppPkg/Applications/Python + +TODO + + +ActivePython +------------ + +:Description: Commercial-grade, quality-assured Python distribution focusing on easy installation and cross-platform compatibility on Windows, Linux, Mac OS X, Solaris, HP-UX and AIX. +:URL: https://www.activestate.com/products/python/ + +TODO + + +Termux +------ + +:Description: Termux is an Android terminal emulator and Linux environment app that works directly with no rooting or setup required. +:URL: https://termux.dev/en/ + +TODO From ee0a69dd898f5bdd72071953bdefbfff825d492c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Filipe=20La=C3=ADns?= Date: Mon, 3 Jul 2023 23:23:55 +0100 Subject: [PATCH 02/11] fix linting MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Filipe Laíns --- pep-9999.rst | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/pep-9999.rst b/pep-9999.rst index 00573c2174b..3ce8a0b838c 100644 --- a/pep-9999.rst +++ b/pep-9999.rst @@ -2,13 +2,11 @@ PEP: 9999 Title: Cross-compiling Python packages Author: Filipe Laíns PEP-Delegate: -Discussions-To: [URL] Status: Draft Type: Informational Content-Type: text/x-rst -Created: 01-07-2023 +Created: 01-Jul-2023 Python-Version: 3.12 -Post-History: [`DD-MMM-YYYY `__] Resolution: @@ -132,7 +130,7 @@ Faking the target environment Aiming to drop the requirement for user input, a popular approach is trying to fake the target environment. It generally consists of monkeypatching the Python interpreter to get it to mimick the interpreter on the target system, which -constitutes of changing many of the `sys` module attributes, the `sysconfig` +constitutes of changing many of the ``sys`` module attributes, the ``sysconfig`` data, etc. Using this strategy, build-backends do not need to have any cross-build support, and should just work without any code changes. From 33af7d45a72827bed5a51361222a4d46ef246824 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Filipe=20La=C3=ADns?= Date: Thu, 6 Jul 2023 16:23:05 +0100 Subject: [PATCH 03/11] fix abstract introduction sentence MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Filipe Laíns --- pep-9999.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pep-9999.rst b/pep-9999.rst index 3ce8a0b838c..21fa8e69b82 100644 --- a/pep-9999.rst +++ b/pep-9999.rst @@ -14,7 +14,7 @@ Abstract ======== This PEP attempts to document the status of cross-compilation of downstream -projects, more specifically +projects. It should give an overview of the approaches currently used by distributors (Linux distros, WASM environment providers, etc.) to cross-compile downstream From de4638ef168c67556913ce914780413efd48cd15 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Filipe=20La=C3=ADns?= Date: Thu, 6 Jul 2023 16:23:54 +0100 Subject: [PATCH 04/11] use PEP number 720 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Filipe Laíns --- pep-9999.rst => pep-0720.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) rename pep-9999.rst => pep-0720.rst (99%) diff --git a/pep-9999.rst b/pep-0720.rst similarity index 99% rename from pep-9999.rst rename to pep-0720.rst index 21fa8e69b82..f76d65a9065 100644 --- a/pep-9999.rst +++ b/pep-0720.rst @@ -1,4 +1,4 @@ -PEP: 9999 +PEP: 720 Title: Cross-compiling Python packages Author: Filipe Laíns PEP-Delegate: From 48e0aac0b36a4c81520d58f3c8295f54e6e4dc04 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Filipe=20La=C3=ADns?= Date: Thu, 6 Jul 2023 16:26:26 +0100 Subject: [PATCH 05/11] add PEP code-owners MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Filipe Laíns --- .github/CODEOWNERS | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 2d873f76501..23ab1c5f410 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -597,6 +597,7 @@ pep-0713.rst @ambv pep-0714.rst @dstufft pep-0715.rst @dstufft pep-0719.rst @Yhg1s +pep-0720.rst @FFY00 # ... # pep-0754.txt # ... From d60f69f545b1c545618e5354f7558de6b58f4859 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Filipe=20La=C3=ADns?= Date: Thu, 6 Jul 2023 16:30:35 +0100 Subject: [PATCH 06/11] use same email as other PEPs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Filipe Laíns --- pep-0720.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pep-0720.rst b/pep-0720.rst index f76d65a9065..e8326bca052 100644 --- a/pep-0720.rst +++ b/pep-0720.rst @@ -1,6 +1,6 @@ PEP: 720 Title: Cross-compiling Python packages -Author: Filipe Laíns +Author: Filipe Laíns PEP-Delegate: Status: Draft Type: Informational From 2036b3344dd877941adddc35a19efaee3e7bb2ec Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Filipe=20La=C3=ADns?= Date: Tue, 18 Jul 2023 05:01:24 +0100 Subject: [PATCH 07/11] reword some stuff MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Filipe Laíns --- pep-0720.rst | 33 ++++++++++++++++----------------- 1 file changed, 16 insertions(+), 17 deletions(-) diff --git a/pep-0720.rst b/pep-0720.rst index e8326bca052..4b1cdf1ef8a 100644 --- a/pep-0720.rst +++ b/pep-0720.rst @@ -54,21 +54,20 @@ its logic to be compatible with popular :ref:`"environment faking" ` workflows [2]_. The lack of first-party support in upstream projects leads to cross-compilation -being fragile and requiring a significant effort from users, however, the lack -of standardization makes it harder for upstreams to improve support, even if -they wanted, as there's no clarity on how this feature should be provided. +being fragile and requiring a significant effort from users, but at the same +time, the lack of standardization makes it harder for upstreams to improve +support as there's no clarity on how this feature should be provided. .. [1] At the time of writing (Jun 2023), setuptools' compiler interface code, the compoment that most of affects cross-compilation, is developed on the `pypa/distutils`__ repository, which gets periodically synced to the setuptools repository. -.. [2] We specifically mention *popular* workflows, because since this is not - standardized, there isn't a single unified approach. Though, many of the - most popular implementations (crossenv_, conda-forge_'s build system, - etc.) work similarly, and this is what we are refering to here. For - clarity, the implementations we are refering to here could be described - as *crossenv-style*. +.. [2] We specifically mention *popular* workflows, because this is not + standardized. Though, many of the most popular implementations + (crossenv_, conda-forge_'s build system, etc.) work similarly, and this + is what we are refering to here. For clarity, the implementations we are + refering to here could be described as *crossenv-style*. .. __: https://github.com/pypa/distutils/pulls?q=cross .. __: https://github.com/pypa/distutils @@ -104,9 +103,9 @@ Cross-build environment This consists of running the Python interpreter normally and utilizing the cross-build provided by the projects' build-system. However, as we saw above, -upstream support is pretty lacking, so this approach only works for a small-ish -set of projects. When this fails, the usual strategy is to patch the -build-system code to build use the correct toolchain, system details, etc. [3]_. +upstream support is lacking, so this approach only works for a small-ish set of +projects. When this fails, the usual strategy is to patch the build-system code +to build use the correct toolchain, system details, etc. [3]_. Since this approach often requires package-specific patching, it requires a lot of user interaction. @@ -116,11 +115,11 @@ of user interaction. :ref:`python-for-android`, :ref:`kivy-ios`, etc. -.. [3] The scope of the build-system patching varies between users and - usually depends on the their goal — some (eg. Linux distributions) may - patch the build-system to support cross-builds, while others might - hardcode compiler paths and system information in the build-system, to - simply make the build work. +.. [3] The scope of the build-system patching varies between users and usually + depends on the their goal — some (eg. Linux distributions) may patch the + build-system to support cross-builds, while others might hardcode + compiler paths and system information in the build-system, to simply make + the build work. .. _approach-target-environment: From 97518a611ee49a0aa7cf89d6b799ba7784b7bfeb Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Filipe=20La=C3=ADns?= Date: Tue, 18 Jul 2023 05:01:41 +0100 Subject: [PATCH 08/11] add environment introspection section MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Filipe Laíns --- pep-0720.rst | 592 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 592 insertions(+) diff --git a/pep-0720.rst b/pep-0720.rst index 4b1cdf1ef8a..6f51651ef61 100644 --- a/pep-0720.rst +++ b/pep-0720.rst @@ -153,6 +153,598 @@ though it doesn't succeed in being generic. :ref:`crossenv`, :ref:`conda-forge`, etc. +Environment introspection +------------------------- + +As explained above, most build system code is written with the assumption that +the target system is the same as where the build is ocurring, so introspection +is usually used to guide the build. + +In this section, we try to document most of the ways this is accomplished. It +should give a decent overview of of environment details that are required by +build systems. + +.. list-table:: + :header-rows: 1 + + * - Snippet + - Description + - Variance + + * - .. code-block:: python + + >>> importlib.machinery.EXTENSION_SUFFIXES + [ + '.cpython-311-x86_64-linux-gnu.so', + '.abi3.so', + '.so', + ] + + - Extension (native module) suffixes supported by this interpreter. + - This is implementation-defined, but it **usually** differs based on the + implementation, system architecture, build configuration, Python + language version, and implementation version — if one exists. + + * - .. code-block:: python + + >>> importlib.machinery.SOURCE_SUFFIXES + ['.py'] + + - Source (pure-Python) suffixes supported by this interpreter. + - This is implementation-defined, but it **usually** doesn't differ + (outside exotic implementations or systems). + + * - .. code-block:: python + + >>> importlib.machinery.all_suffixes() + [ + '.py', + '.pyc', + '.cpython-311-x86_64-linux-gnu.so', + '.abi3.so', + '.so', + ] + + - All module file suffixes supported by this interpreter. It *should* be the + union of all ``importlib.machinery.*_SUFFIXES`` attributes. + - This is implementation-defined, but it **usually** differs based on the + implementation, system architecture, build configuration, Python + language version, and implementation version — if one exists. See the + entries above for more information. + + * - .. code-block:: python + + >>> sys.abiflags + '' + + - ABI flags, as specified in :pep:`3149`. + - Differs based on the build configuration. + + * - .. code-block:: python + + >>> sys.api_version + 1013 + + - C API version. + - Differs based on the Python installation. + + * - .. code-block:: python + + >>> sys.base_prefix + /usr + + - Prefix of the installation-wide directories where platform independent + files are installed. + - Differs based on the platform, and installation. + + * - .. code-block:: python + + >>> sys.base_exec_prefix + /usr + + - Prefix of the installation-wide directories where platform dependent + files are installed. + - Differs based on the platform, and installation. + + * - .. code-block:: python + + >>> sys.byteorder + 'little' + + - Native byte order. + - Differs based on the platform. + + * - .. code-block:: python + + >>> sys.builtin_module_names + ('_abc', '_ast', '_codecs', ...) + + - Names of all modules that are compiled into the Python interpreter. + - Differs based on the platform, system architecture, and build + configuration. + + * - .. code-block:: python + + >>> sys.exec_prefix + /usr + + - Prefix of the site-specific directories where platform independent files + are installed. Because it concerns the site-specific directories, in + standard virtual environment implementation, it will be a + virtual-environment-specific path. + - Differs based on the platform, installation, and environment. + + * - .. code-block:: python + + >>> sys.executable + '/usr/bin/python' + + - Path of the Python interpreter being used. + - Differs based on the installation. + + * - .. code-block:: python + + >>> with open(sys.executable, 'rb') as f: + ... header = f.read(4) + ... if is_elf := (header == b'\x7fELF'): + ... elf_class = int(f.read(1)) + ... size = {1: 52, 2: 64}.get(elf_class) + ... elf_header = f.read(size - 5) + + - Whether the Python interpreter is an ELF file, and the ELF header. This + approach is something used to identify the target architecture of the + installation (example__). + - Differs based on the installation. + + * - .. code-block:: python + + >>> sys.float_info + sys.float_info( + max=1.7976931348623157e+308, + max_exp=1024, + max_10_exp=308, + min=2.2250738585072014e-308, + min_exp=-1021, + min_10_exp=-307, + dig=15, + mant_dig=53, + epsilon=2.220446049250313e-16, + radix=2, + rounds=1, + ) + + - Low level information about the float type, as defined by ``float.h``. + - Differs based on the architecture, and platform. + + * - .. code-block:: python + + >>> sys.getandroidapilevel() + 21 + + - Integer representing the Android API level. + - Differs based on the platform. + + * - .. code-block:: python + + >>> sys.getwindowsversion() + sys.getwindowsversion( + major=10, + minor=0, + build=19045, + platform=2, + service_pack='', + ) + + - Windows version of the system. + - Differs based on the platform. + + * - .. code-block:: python + + >>> sys.hexversion + 0x30b03f0 + + - Python version encoded as an integer. + - Differs based on the Python language version. + + * - .. code-block:: python + + >>> sys.implementation + namespace( + name='cpython', + cache_tag='cpython-311', + version=sys.version_info( + major=3, + minor=11, + micro=3, + releaselevel='final', + serial=0, + ), + hexversion=0x30b03f0, + _multiarch='x86_64-linux-gnu', + ) + + - Interpreter implementation details. + - Differs based on the interpreter implementation, Python language + version, and implementation version — if one exists. It may also include + architecture-dependent information, so it may also differ based on the + system architecture. + + * - .. code-block:: python + + >>> sys.int_info + sys.int_info( + bits_per_digit=30, + sizeof_digit=4, + default_max_str_digits=4300, + str_digits_check_threshold=640, + ) + + - Low level information about Python's internal integer representation. + - Differs based on the architecture, platform, implementation, build, and + runtime flags. + + * - .. code-block:: python + + >>> sys.maxsize + 0x7fffffffffffffff + + - Maximum value a variable of type ``Py_ssize_t`` can take. + - Differs based on the architecture, platform, and implementation. + + * - .. code-block:: python + + >>> sys.maxunicode + 0x10ffff + + - Value of the largest Unicode code point. + - Differs based on the implementation, and on Python versions older than + 3.3, the build. + + * - .. code-block:: python + + >>> sys.platform + linux + + - Platform identifier. + - Differs based on the platform. + + * - .. code-block:: python + + >>> sys.prefix + /usr + + - Prefix of the site-specific directories where platform dependent files + are installed. Because it concerns the site-specific directories, in + standard virtual environment implementation, it will be a + virtual-environment-specific path. + - Differs based on the platform, installation, and environment. + + * - .. code-block:: python + + >>> sys.platlibdir + lib + + - Platform-specific library directory. + - Differs based on the platform, and vendor. + + * - .. code-block:: python + + >>> sys.version_info + sys.version_info( + major=3, + minor=11, + micro=3, + releaselevel='final', + serial=0, + ) + + - Python language version implementated by the interpreter. + - Differs if the target Python version is not the same [4]_. + + * - .. code-block:: python + + >>> sys.thread_info + lib + + - Information about the thread implementation. + - Differs based on the platform, and implementation. + + * - .. code-block:: python + + >>> sys.winver + 3.8-32 + + - Version number used to form Windows registry keys. + - Differs based on the platform, and implementation. + + * - .. code-block:: python + + >>> sysconfig.get_config_vars() + { ... } + >>> sysconfig.get_config_var(...) + ... + + - Python distribution configuration variables. It includes a set of + variables [5]_ — like ``prefix``, ``exec_prefix``, etc. — based on the + running context [6]_, and may include some extra variables based on the + Python implementation and system. + + In CPython and most other implementations that use the same + build-system, the "extra" variables mention above are: on POSIX, all + variables from the ``Makefile`` used to build the interpreter, and on + Windows, it usually only includes a small subset of the those [7]_ — + like ``EXT_SUFFIX``, ``BINDIR``, etc. + + - This is implementation-defined, but it **usually** differs between + non-identical builds. Please refer to the + :ref:`env-sysconfig-config-vars` table for a overview of the different + configuration variable that are usually present. + +.. [4] Ideally, you want to perform cross-builds with the same Python version + and implementation, however, this is often not the case. It should not + be very problematic as long as the major and minor versions don't + change. + +.. [5] The set of config variables that will always be present mostly consists + of variables needed to calculate the installation scheme paths. + +.. [6] The context we refer here consists of the "path initialization", which is + a process that happens in the interpreter startup and is responsible for + figuring out which environment it is being run — eg. global environment, + virtual environment, etc. — and setting ``sys.prefix`` and other + attributes acordingly. + +.. [7] This is because Windows builds may not use the ``Makefile``, and instead + `use the Visual Studio build system`__. A subset of the most relevant + ``Makefile`` variables is provided to make user code that uses them + simpler. + +.. __: https://github.com/pypa/packaging/blob/2f80de7fd2a8bc199dadf5cf3f5f302a17084792/src/packaging/_manylinux.py#L43-L50 +.. __: https://docs.python.org/3/using/configure.html#debug-build + + +CPython (and similar) +~~~~~~~~~~~~~~~~~~~~~ + + +.. _env-sysconfig-config-vars: + +.. list-table:: ``sysconfig`` configuration variables + :header-rows: 1 + :widths: 20 20 30 30 + + * - Name + - Example Value + - Description + - Variance + + * - ``SOABI`` + - ``cpython-311-x86_64-linux-gnu`` + - ABI string — defined by :pep:`3149`. + - Differs based on the implementation, system architecture, Python + language version, and implementation version — if one exists. + + * - ``SHLIB_SUFFIX`` + - ``.so`` + - Shared library suffix. + - Differs based on the platform. + + * - ``EXT_SUFFIX`` + - ``.cpython-311-x86_64-linux-gnu.so`` + - Interpreter-specific Python extension (native module) suffix — generally + defined as ``.{SOABI}.{SHLIB_SUFFIX}``. + - Differs based on the implementation, system architecture, Python + language version, and implementation version — if one exists. + + * - ``LDLIBRARY`` + - ``libpython3.11.so`` + - Shared ``libpython`` library name — if available. If unavailable [8]_, + the variable will be empty, if available, the library should be located + in ``LIBDIR``. + - Differs based on the implementation, system architecture, build + configuration, Python language version, and implementation version — if + one exists. + + * - ``PY3LIBRARY`` + - ``libpython3.so`` + - Shared Python 3 only (major version bound only) [9]_ ``libpython`` + library name — if available. If unavailable [8]_, the variable will be + empty, if available, the library should be located in ``LIBDIR``. + - Differs based on the implementation, system architecture, build + configuration, Python language version, and implementation version — if + one exists. + + * - ``LIBRARY`` + - ``libpython3.11.a`` + - Static ``libpython`` library name — if available. If unavailable [8]_, + the variable will be empty, if available, the library should be located + in ``LIBDIR``. + - Differs based on the implementation, system architecture, build + configuration, Python language version, and implementation version — if + one exists. + + * - ``Py_DEBUG`` + - ``0`` + - Whether this is a `debug build`__. + - Differs based on the build configuration. + + * - ``WITH_PYMALLOC`` + - ``1`` + - Whether this build has pymalloc__ support. + - Differs based on the build configuration. + + * - ``Py_TRACE_REFS`` + - ``0`` + - Whether reference tracing (debug build only) is enabled. + - Differs based on the build configuration. + + * - ``Py_UNICODE_SIZE`` + - + - Size of the ``Py_UNICODE`` object, in bytes. This variable is only + present in CPython versions older than 3.3, and was commonly used to + detect if the build uses UCS2 or UCS4 for unicode objects — before + :pep:`393`. + - Differs based on the build configuration. + + * - ``Py_ENABLE_SHARED`` + - ``1`` + - Whether a shared ``libpython`` is available. + - Differs based on the build configuration. + + * - ``PY_ENABLE_SHARED`` + - ``1`` + - Whether a shared ``libpython`` is available. + - Differs based on the build configuration. + + * - ``CC`` + - ``gcc`` + - The C compiler used to build the Python distribution. + - Differs based on the build configuration. + + * - ``CXX`` + - ``g++`` + - The C compiler used to build the Python distribution. + - Differs based on the build configuration. + + * - ``CFLAGS`` + - ``-DNDEBUG -g -fwrapv ...`` + - The C compiler flags used to build the Python distribution. + - Differs based on the build configuration. + + * - ``py_version`` + - ``3.11.3`` + - Full form of the Python version. + - Differs based on the Python language version. + + * - ``py_version_short`` + - ``3.11`` + - Custom form of the Python version, containing only the major and minor + numbers. + - Differs based on the Python language version. + + * - ``py_version_nodot`` + - ``311`` + - Custom form of the Python version, containing only the major and minor + numbers, and no dots. + - Differs based on the Python language version. + + * - ``prefix`` + - ``/usr`` + - Same as ``sys.prefix``, please refer to the entry in table above. + - Differs based on the platform, installation, and environment. + + * - ``base`` + - ``/usr`` + - Same as ``sys.prefix``, please refer to the entry in table above. + - Differs based on the platform, installation, and environment. + + * - ``exec_prefix`` + - ``/usr`` + - Same as ``sys.exec_prefix``, please refer to the entry in table above. + - Differs based on the platform, installation, and environment. + + * - ``platbase`` + - ``/usr`` + - Same as ``sys.exec_prefix``, please refer to the entry in table above. + - Differs based on the platform, installation, and environment. + + * - ``installed_base`` + - ``/usr`` + - Same as ``sys.base_prefix``, please refer to the entry in table above. + - Differs based on the platform, and installation. + + * - ``installed_platbase`` + - ``/usr`` + - Same as ``sys.base_exec_prefix``, please refer to the entry in table + above. + - Differs based on the platform, and installation. + + * - ``platlibdir`` + - ``lib`` + - Same as ``sys.platlibdir``, please refer to the entry in table above. + - Differs based on the platform, and vendor. + + * - ``SIZEOF_*`` + - ``4`` + - Size of a certain C type (``double``, ``float``, etc.). + - Differs based on the system architecture, and build details. + + +.. [8] Due to Python bring compiled without shared or static ``libpython`` + support, respectively. + +.. [9] This is the ``libpython`` library that users of the `stable ABI`__ should + link against, if they need to link against ``libpython``. + +.. __: https://docs.python.org/3/c-api/intro.html#debugging-builds +.. __: https://docs.python.org/3/c-api/memory.html#pymalloc +.. __: https://docs.python.org/3/c-api/stable.html#stable-application-binary-interface + + +Relevant Information +-------------------- + +There are some bits of information required by build systems — eg. platform +particularities — scattered accross many places, and it often is difficult to +identify code with assumptions based on them. In this section, we try to +document the most relevant cases. + + +When should extensions be linked against ``libpython``? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Short answer + Yes, on Windows. No on POSIX platforms, except Android, Cygwin, and other + Windows-based POSIX-like platforms. + +When building extensions for dynamic loading, depending on the target platform, +they may need to be linked against ``libpython``. + +On Windows, extensions need to link against ``libpython``, because all symbols +must be resolvable at link time. POSIX-like platforms based on Windows — like +Cygwin, MinGW, or MSYS — will also require linking against ``libpython``. + +On most POSIX platforms, it is not necessary to link against ``libpython``, as +the symbols will already be available in the due to the interpreter — or, when +embedding, the executable/library in question — already linking to +``libpython``. Not linking an extension module against ``libpython`` will allow +it to be loaded by static Python builds, so when possible, it is desirable to do +so (see GH-65735__). + +This might not be the case on all POSIX platforms, so make sure you check. One +example is Android, where only the main executable and ``LD_PRELOAD``s are +considered to be ``RTLD_GLOBAL`` (meaning dependencies are ``RTLD_LOCAL``) +[10]_, which causes the ``libpython`` symbols be unvailable when loading the +extension. + +.. [10] Refer to `dlopen's man page`__ for more information. + +.. __: https://github.com/python/cpython/issues/65735 +.. __: https://man.archlinux.org/man/dlopen.3 + + +What are ``prefix``, ``exec_prefix``, ``base_prefix``, and ``base_exec_prefix``? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +These are ``sys`` attributes `set in the Python initialization`__ that describe +the running environment. They refer to the prefix of directories where +installation/environment files are installed, acording to the table below. + +==================== ====================================== ================= +Name Target files Environment Scope +==================== ====================================== ================= +``prefix`` platform independent (eg. pure Python) site-specific +``exec_prefix`` platform dependent (eg. native code) site-specific +``base_prefix`` platform independent (eg. pure Python) installation-wide +``base_exec_prefix`` platform dependent (eg. native code) installation-wide +==================== ====================================== ================= + +Because the site-specific prefixes will be different inside virtual +environments, checking ``sys.prexix != sys.base_prefix`` is commonly used to +check if we are in a virtual environment. + +.. __: https://github.com/python/cpython/blob/6a70edf24ca217c5ed4a556d0df5748fc775c762/Modules/getpath.py + Case studies ============ From 73028a1569fc59786e0d550b4fb4557caf6bb3ab Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Filipe=20La=C3=ADns?= Date: Tue, 18 Jul 2023 05:07:56 +0100 Subject: [PATCH 09/11] fix linting MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Filipe Laíns --- pep-0720.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/pep-0720.rst b/pep-0720.rst index 6f51651ef61..f30d2170702 100644 --- a/pep-0720.rst +++ b/pep-0720.rst @@ -712,8 +712,8 @@ it to be loaded by static Python builds, so when possible, it is desirable to do so (see GH-65735__). This might not be the case on all POSIX platforms, so make sure you check. One -example is Android, where only the main executable and ``LD_PRELOAD``s are -considered to be ``RTLD_GLOBAL`` (meaning dependencies are ``RTLD_LOCAL``) +example is Android, where only the main executable and ``LD_PRELOAD`` entries +are considered to be ``RTLD_GLOBAL`` (meaning dependencies are ``RTLD_LOCAL``) [10]_, which causes the ``libpython`` symbols be unvailable when loading the extension. From 8c89ebe3840bcfe5a2fed41822163a479422b8f3 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Filipe=20La=C3=ADns?= Date: Wed, 19 Jul 2023 11:21:15 +0100 Subject: [PATCH 10/11] fix typos Co-authored-by: Hugo van Kemenade --- pep-0720.rst | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/pep-0720.rst b/pep-0720.rst index f30d2170702..d725ae52e98 100644 --- a/pep-0720.rst +++ b/pep-0720.rst @@ -37,7 +37,7 @@ Introduction There are a couple different approaches being used to tackle this, with different levels of interaction required from the user, but they all require a -significant ammount of effort. This is due to the lack of standardized +significant amount of effort. This is due to the lack of standardized cross-compilation infrastructure on the Python packaging ecosystem, which itself stems from the complexity of cross-builds, making it a huge undertaking. @@ -59,15 +59,15 @@ time, the lack of standardization makes it harder for upstreams to improve support as there's no clarity on how this feature should be provided. .. [1] At the time of writing (Jun 2023), setuptools' compiler interface code, - the compoment that most of affects cross-compilation, is developed on the + the component that most of affects cross-compilation, is developed on the `pypa/distutils`__ repository, which gets periodically synced to the setuptools repository. .. [2] We specifically mention *popular* workflows, because this is not standardized. Though, many of the most popular implementations (crossenv_, conda-forge_'s build system, etc.) work similarly, and this - is what we are refering to here. For clarity, the implementations we are - refering to here could be described as *crossenv-style*. + is what we are referring to here. For clarity, the implementations we are + referring to here could be described as *crossenv-style*. .. __: https://github.com/pypa/distutils/pulls?q=cross .. __: https://github.com/pypa/distutils @@ -128,7 +128,7 @@ Faking the target environment Aiming to drop the requirement for user input, a popular approach is trying to fake the target environment. It generally consists of monkeypatching the Python -interpreter to get it to mimick the interpreter on the target system, which +interpreter to get it to mimic the interpreter on the target system, which constitutes of changing many of the ``sys`` module attributes, the ``sysconfig`` data, etc. Using this strategy, build-backends do not need to have any cross-build support, and should just work without any code changes. @@ -137,14 +137,14 @@ Unfortunately, though, it isn't possible to truly fake the target environment. There are many reasons for this, one of the main ones being that it breaks code that actually needs to introspect the running interpreter. As a result, monkeypatching Python to look like target is very tricky — to achieve the less -ammount of breakage, we can only patch certain aspects of the interpreter. +amount of breakage, we can only patch certain aspects of the interpreter. Consequently, build-backends may need some code changes, but these are generally much smaller than the previous approach. This is an inherent limitation of the technique, meaning this strategy still requires some user interaction. Nonetheless, this strategy still works out-of-the-box with significantly more projects than the approach above, and requires much less effort in these cases. -It is sucessfull in decreasing the ammount of user interaction needed, even +It is successful in decreasing the amount of user interaction needed, even though it doesn't succeed in being generic. .. admonition:: Examples @@ -157,7 +157,7 @@ Environment introspection ------------------------- As explained above, most build system code is written with the assumption that -the target system is the same as where the build is ocurring, so introspection +the target system is the same as where the build is occurring, so introspection is usually used to guide the build. In this section, we try to document most of the ways this is accomplished. It @@ -438,7 +438,7 @@ build systems. serial=0, ) - - Python language version implementated by the interpreter. + - Python language version implemented by the interpreter. - Differs if the target Python version is not the same [4]_. * - .. code-block:: python @@ -492,7 +492,7 @@ build systems. a process that happens in the interpreter startup and is responsible for figuring out which environment it is being run — eg. global environment, virtual environment, etc. — and setting ``sys.prefix`` and other - attributes acordingly. + attributes accordingly. .. [7] This is because Windows builds may not use the ``Makefile``, and instead `use the Visual Studio build system`__. A subset of the most relevant @@ -685,7 +685,7 @@ Relevant Information -------------------- There are some bits of information required by build systems — eg. platform -particularities — scattered accross many places, and it often is difficult to +particularities — scattered across many places, and it often is difficult to identify code with assumptions based on them. In this section, we try to document the most relevant cases. @@ -714,7 +714,7 @@ so (see GH-65735__). This might not be the case on all POSIX platforms, so make sure you check. One example is Android, where only the main executable and ``LD_PRELOAD`` entries are considered to be ``RTLD_GLOBAL`` (meaning dependencies are ``RTLD_LOCAL``) -[10]_, which causes the ``libpython`` symbols be unvailable when loading the +[10]_, which causes the ``libpython`` symbols be unavailable when loading the extension. .. [10] Refer to `dlopen's man page`__ for more information. @@ -728,7 +728,7 @@ What are ``prefix``, ``exec_prefix``, ``base_prefix``, and ``base_exec_prefix``? These are ``sys`` attributes `set in the Python initialization`__ that describe the running environment. They refer to the prefix of directories where -installation/environment files are installed, acording to the table below. +installation/environment files are installed, according to the table below. ==================== ====================================== ================= Name Target files Environment Scope @@ -805,7 +805,7 @@ Pyodide XXX: Hood should review/expand this section. -``Pyodide`` is a provides a Python distribution compied to WebAssembly__ +``Pyodide`` is a provides a Python distribution compiled to WebAssembly__ using the Emscripten__ toolchain. It patches several aspects of the CPython installation and some external From 9ee993472a5b63a099877d08333addb5af8c099e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Filipe=20La=C3=ADns?= Date: Wed, 19 Jul 2023 23:58:51 +0100 Subject: [PATCH 11/11] fix sys.thread_info output MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Filipe Laíns --- pep-0720.rst | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/pep-0720.rst b/pep-0720.rst index d725ae52e98..2ce440eab4e 100644 --- a/pep-0720.rst +++ b/pep-0720.rst @@ -444,7 +444,11 @@ build systems. * - .. code-block:: python >>> sys.thread_info - lib + sys.thread_info( + name='pthread', + lock='semaphore', + version='NPTL 2.37', + ) - Information about the thread implementation. - Differs based on the platform, and implementation.