From 62173550925008c89a9c846225efeee1e85d0201 Mon Sep 17 00:00:00 2001 From: Lalatendu Mohanty Date: Sun, 14 Dec 2025 07:30:27 -0500 Subject: [PATCH] docs: add glossary and bootstrap-vs-build docs Add comprehensive glossary with ~45 terms used across Fromager code and docs. Also added bootstrap-vs-build.rst doc. CLoses #883 Signed-off-by: Lalatendu Mohanty --- .beads/issues.jsonl | 0 docs/concepts/bootstrap-vs-build.rst | 109 +++++++++ docs/concepts/index.rst | 1 + docs/glossary.rst | 324 +++++++++++++++++++++++++++ docs/index.rst | 1 + 5 files changed, 435 insertions(+) create mode 100644 .beads/issues.jsonl create mode 100644 docs/concepts/bootstrap-vs-build.rst create mode 100644 docs/glossary.rst diff --git a/.beads/issues.jsonl b/.beads/issues.jsonl new file mode 100644 index 00000000..e69de29b diff --git a/docs/concepts/bootstrap-vs-build.rst b/docs/concepts/bootstrap-vs-build.rst new file mode 100644 index 00000000..dc2e5d77 --- /dev/null +++ b/docs/concepts/bootstrap-vs-build.rst @@ -0,0 +1,109 @@ +Bootstrap vs Build +================== + +Fromager has two distinct modes of operation: **bootstrap** and **build**. +Understanding the difference is key to using fromager effectively. + +Quick Comparison +---------------- + +.. list-table:: + :header-rows: 1 + :widths: 20 40 40 + + * - Aspect + - Bootstrap + - Build + * - **Scope** + - Entire dependency tree + - Single package + * - **Purpose** + - Discover and resolve all dependencies + - Compile source into wheel + * - **Recursion** + - Yes (processes dependencies) + - No (one package only) + * - **Input** + - Requirements file or package specs + - Package name + version + source URL + * - **Output** + - Dependency graph, build order, all wheels + - One wheel file + +Bootstrap Mode +-------------- + +The ``bootstrap`` command recursively discovers and builds all dependencies: + +.. code-block:: text + + fromager bootstrap numpy + ├── Resolve version → numpy==1.26.0 + ├── Download source + ├── Build wheel + ├── Extract dependencies: [cython, setuptools, ...] + └── bootstrap(cython) ← Recurse for each dependency + ├── Resolve version → cython==3.0.0 + ├── Build wheel + └── ... + +**Key operations:** + +1. Version resolution for all packages +2. Dependency graph construction +3. Build order determination +4. Wheel building (for each discovered package) + +**When to use:** Initial discovery of what needs to be built, creating a complete +wheel collection from scratch. + +Build Mode +---------- + +The ``build`` command compiles a single package without recursion: + +.. code-block:: text + + fromager build numpy 1.26.0 https://pypi.org/simple/ + ├── Download sdist + ├── Apply patches + ├── Create build environment + ├── Run pip wheel + └── Output: numpy-1.26.0-cp311-linux_x86_64.whl + +**Key operations:** + +1. Source download and preparation +2. Build environment setup +3. Wheel compilation +4. No dependency discovery or recursion + +**When to use:** Production builds where the build order is already known +(from a previous bootstrap), CI/CD pipelines, rebuilding individual packages. + +Relationship +------------ + +Bootstrap uses build internally: + +.. code-block:: text + + bootstrap() + └── for each package: + └── _build_from_source() ← Same as build command + └── Creates wheel + +The ``build-sequence`` command bridges these modes by reading a ``build-order.json`` +file (produced by bootstrap) and calling build for each package in order. + +Typical Workflow +---------------- + +1. **Development:** Use ``bootstrap`` to discover all dependencies and create + initial wheel collection + +2. **Production:** Use ``build-sequence`` with the ``build-order.json`` from + bootstrap to rebuild deterministically + +3. **Fixes:** Use ``build`` to rebuild individual packages after applying patches + diff --git a/docs/concepts/index.rst b/docs/concepts/index.rst index 5cd4535b..4efcb6a8 100644 --- a/docs/concepts/index.rst +++ b/docs/concepts/index.rst @@ -4,5 +4,6 @@ Concepts .. toctree:: :maxdepth: 1 + bootstrap-vs-build dependencies diff --git a/docs/glossary.rst b/docs/glossary.rst new file mode 100644 index 00000000..5de5cf18 --- /dev/null +++ b/docs/glossary.rst @@ -0,0 +1,324 @@ +Glossary +======== + +This glossary defines key terms used throughout Fromager's documentation and codebase. + +.. glossary:: + :sorted: + + ABI + Application Binary Interface. Defines binary compatibility between compiled + code. Relevant when building platform-specific :term:`wheels ` that + must match the target Python interpreter's ABI. + + bootstrap + The process of recursively discovering and building all dependencies for a + set of top-level requirements. Bootstrap resolves versions, downloads sources, + builds wheels, and extracts installation dependencies—repeating for each + discovered dependency until the entire dependency tree is processed. The output + includes a :term:`dependency graph`, :term:`build order`, and all built + :term:`wheels `. See :doc:`/concepts/bootstrap-vs-build` for details. + + build + The process of compiling a single package from source into a :term:`wheel`, + without recursion. Unlike :term:`bootstrap`, the build command operates on + one package at a time with a known version and source URL. See + :doc:`/concepts/bootstrap-vs-build` for a comparison. + + build environment + An isolated Python virtual environment created for building a specific package. + It contains only the :term:`build dependencies ` required + to compile that package, ensuring reproducible builds. See also + :term:`build isolation`. + + build isolation + The practice of running each package build in its own isolated virtual + environment to prevent interference between builds. Distinct from + :term:`network isolation`, which restricts network access. + + build order + The sequence in which packages must be built, determined by analyzing the + :term:`dependency graph`. Packages are ordered bottom-up (topological sort) + so that each package's dependencies are built before the package itself. + Stored in ``build-order.json``. See :doc:`/files` for the file format. + + build sequence + A command (``build-sequence``) that processes a pre-determined :term:`build order` + file to build wheels in dependency order. Unlike :term:`bootstrap`, it does not + perform dependency discovery—it simply builds each package in the specified order. + See :doc:`/using` for usage details. + + build tag + A numeric prefix added to :term:`wheel` filenames (e.g., ``-0-`` in + ``package-1.0.0-0-py3-none-any.whl``) to differentiate wheels built by fromager + from upstream wheels. This follows the wheel filename convention from + :pep:`427`. + + build-backend dependency + A dependency returned by :pep:`517` build backend hooks like + ``get_requires_for_build_wheel()``. These are additional tools needed beyond + the :term:`build-system dependencies ` to build a wheel. + Examples include ``cython`` for Cython extensions or ``numpy`` for packages + compiling against NumPy headers. See :doc:`/concepts/dependencies`. + + build-sdist dependency + A dependency needed specifically for building a :term:`source distribution`. + Returned by the :pep:`517` ``get_requires_for_build_sdist()`` hook. + See :doc:`/concepts/dependencies`. + + build-system dependency + A dependency listed in ``pyproject.toml`` under ``[build-system].requires``, + as defined by :pep:`518`. These are the foundational build tools (like + ``setuptools``, ``flit-core``, or ``maturin``) installed before any build + backend hooks are called. See :doc:`/concepts/dependencies`. + + built distribution + A package format ready for installation without requiring a build step. + :term:`Wheels ` are the standard built distribution format in Python. + Contrast with :term:`source distribution`. + + candidate + A potential version of a package discovered during :term:`resolution`. Candidates + are evaluated against :term:`version specifiers ` and + :term:`constraints` to select the best matching version. + + canonical name + The normalized form of a Python package name, computed using + ``packaging.utils.canonicalize_name()``. All letters are lowercase and runs of + hyphens, underscores, and periods are replaced with a single hyphen (e.g., + ``My_Package`` becomes ``my-package``). See :ref:`canonical-distribution-names`. + + constraints + Version specifications that control package :term:`resolution`. Provided via a + ``constraints.txt`` file, they ensure specific versions are used or avoided + during builds. Unlike :term:`requirements `, constraints only + apply when a package is already needed. See :doc:`/files` for format details. + + cyclic dependency + A circular dependency where packages depend on each other, forming a loop + (e.g., A depends on B, B depends on C, C depends on A). See + :term:`build-time cycle` and :term:`install-time cycle`. + + build-time cycle + A problematic :term:`cyclic dependency` where packages require each other + during the build process. These must be resolved (often by marking one + package as :term:`pre-built `) for the build to succeed. + + install-time cycle + An acceptable :term:`cyclic dependency` where packages depend on each other + only at runtime. These don't affect the build process since install + dependencies are processed after the parent package is built. + + dependency edge + A relationship in the :term:`dependency graph` connecting two + :term:`dependency nodes `. Each edge includes the + :term:`requirement` specification and dependency type (toplevel, install, + build-system, etc.). + + dependency graph + A directed graph representing all packages and their relationships discovered + during :term:`bootstrap`. Each :term:`dependency node` represents a resolved + package version, connected by :term:`dependency edges `. + Stored in ``graph.json``. See :doc:`/files` and + :doc:`/how-tos/graph-commands/index`. + + dependency node + A vertex in the :term:`dependency graph` representing a specific version of + a package. Contains the package name, resolved version, download URL, and + lists of parent and child edges. + + distribution name + The actual package name as it appears in package files and indexes, which + may have different casing than the :term:`canonical name`. For example, + ``PyYAML`` is the distribution name while ``pyyaml`` is the canonical name. + + hook + An extension point in fromager that allows customization of specific + operations. Hooks include ``post_build``, ``prebuilt_wheel``, and + ``post_bootstrap``. Multiple plugins can register for the same hook. + See :doc:`/hooks` and :doc:`/customization`. + + install dependency + A runtime dependency of a package, extracted from the built :term:`wheel`'s + ``Requires-Dist`` metadata. These are processed after the parent package is + built. See :doc:`/concepts/dependencies`. + + local cache + Built :term:`wheels ` stored locally in ``wheels-repo/`` for reuse + within a :term:`bootstrap` run. Fromager checks this cache before building + to avoid redundant compilation. + + network isolation + A build mode where :term:`source distribution` and :term:`wheel` building + occurs without network access (using ``unshare -cn`` on Linux). This ensures + builds only use locally available dependencies and cannot download arbitrary + code. Distinct from :term:`build isolation`. + + override name + A variant of :term:`canonical name` where hyphens are replaced with underscores + (e.g., ``my-package`` becomes ``my_package``). Used for settings files, patch + directories, and :term:`override plugins ` because Python + module names cannot contain hyphens. See :ref:`canonical-distribution-names`. + + override plugin + A Python module registered as an entry point that provides custom implementations + of fromager operations for specific packages. Unlike :term:`hooks `, overrides + replace the default behavior entirely. Plugins can customize source acquisition, + dependency resolution, building, and more. See :doc:`/hooks`. + + package index + A server providing package metadata and downloads, following the + :term:`Simple API` specification. :term:`PyPI` is the default public index. + Also called a package repository. + + package repository + A directory structure or server serving packages following the :pep:`503` + :term:`Simple API`. Fromager creates a local package repository in + ``wheels-repo/simple/`` during builds. + + patch + A file (with ``.patch`` extension) that modifies source code before building. + Patches are stored in the patches directory (default: ``overrides/patches/``) + organized by package name and optionally version and :term:`variant`. Applied + using ``patch -p1``. See :doc:`/customization`. + + PEP 503 + Python Enhancement Proposal defining the Simple Repository API—the directory + structure for :term:`package indexes `. Fromager creates a + PEP 503-compliant local repository for built wheels. See :pep:`503`. + + PEP 517 + Python Enhancement Proposal defining the interface between build frontends + (like pip) and build backends (like setuptools). Specifies hooks like + ``get_requires_for_build_wheel()`` that fromager uses to discover + :term:`build-backend dependencies `. See :pep:`517`. + + PEP 518 + Python Enhancement Proposal specifying the ``pyproject.toml`` file format + for declaring :term:`build-system dependencies `. + See :pep:`518`. + + pre-built wheel + A :term:`wheel` that is used directly without building from source. Configured + via the ``pre_built`` setting for a :term:`variant`. Useful for packages that + cannot be built from source or when using vendor-provided binaries. + + pre-release version + A package version containing alpha (``a``), beta (``b``), or release candidate + (``rc``) components. By default, fromager ignores pre-release versions unless + explicitly requested via requirements or :term:`constraints`. + See :doc:`/how-tos/pre-release-versions`. + + PyPI + The Python Package Index (https://pypi.org), the default public + :term:`package index` for Python packages. Fromager downloads + :term:`source distributions ` from PyPI by default. + + remote cache + A :term:`wheel server` with previously built packages, used for distributed + builds. Configured via ``--cache-wheel-server-url`` to avoid rebuilding + packages that already exist remotely. + + repeatable builds + A feature that uses the :term:`dependency graph` from a previous + :term:`bootstrap` to ensure consistent package versions across builds. Enabled + via the ``--previous-bootstrap-file`` option. See :doc:`/how-tos/repeatable-builds`. + + requirement + A package dependency specification that may include version constraints, + extras, and environment markers. For example, ``requests>=2.28.0`` or + ``numpy[dev]>=1.20; python_version>="3.9"``. See also :term:`specifier`. + + resolution + The process of determining specific package versions from :term:`requirement` + specifications. The :term:`resolver` evaluates available :term:`candidates ` + against :term:`constraints`, markers, and other factors to select the best + matching version. + + resolver + The component that performs :term:`resolution`, selecting specific package + versions that satisfy all :term:`requirements ` and + :term:`constraints`. Uses :term:`resolver providers ` to + discover available versions. + + resolver provider + A strategy class that supplies version :term:`candidates ` during + :term:`resolution`. The default provider queries :term:`PyPI`, but custom + providers can resolve versions from GitHub tags, GitLab tags, or other sources. + See :doc:`/hooks`. + + sdist-only mode + A :term:`bootstrap` mode (``--sdist-only``) that builds :term:`source + distributions ` but skips :term:`wheel` building for + install dependencies. Useful for quickly generating :term:`build order` files + when wheel compilation is time-consuming. + + settings + Configuration options that customize package building. Can be global (in + ``overrides/settings.yaml``) or per-package (in ``overrides/settings/.yaml``). + Settings control environment variables, source URLs, :term:`variants `, + and more. See :doc:`/customization` and :doc:`/config-reference`. + + Simple API + The :pep:`503` specification for :term:`package index` directory layout. + Uses a ``/simple//`` URL structure with HTML pages listing available + package files. Fromager serves built wheels via a local Simple API server. + + source distribution + A package archive containing source code, typically a ``.tar.gz`` file. Also + called "sdist". Fromager downloads sdists, applies :term:`patches `, + and builds :term:`wheels ` from them. Defined by Python packaging + standards. See the `PyPA glossary `__. + + source type + The origin of a package's source code. Values include: + + - ``sdist``: Downloaded from a :term:`package index` (default) + - ``prebuilt``: Using a :term:`pre-built wheel` + - ``git``: Cloned from a git repository URL + - ``override``: Custom source via :term:`override plugin` + + specifier + The version constraint portion of a :term:`requirement`, such as ``>=1.0,<2.0`` + in ``package>=1.0,<2.0``. Specifiers define which versions satisfy a requirement. + + toplevel dependency + A package specified directly via CLI arguments or a requirements file, as + opposed to dependencies discovered transitively. These are the starting points + for :term:`bootstrap`. See :doc:`/concepts/dependencies`. + + variant + A named build configuration for producing different versions of packages. + Commonly used for hardware-specific builds (e.g., ``cpu``, ``cuda``, ``rocm``). + Each variant can have its own environment variables, patches, and settings. + The default variant is ``cpu``. See :doc:`/customization`. + + VCS + Version Control System, such as git or mercurial. Fromager supports building + from VCS URLs (e.g., ``git+https://github.com/...``) specified in requirements. + + vendoring + The practice of including dependencies within a package's source code for + offline or isolated builds. Some packages vendor their dependencies to avoid + network access during builds. + + wheel + A :term:`built distribution` format (:pep:`427`) containing compiled code + ready for installation. Wheel files have the ``.whl`` extension and include + platform and Python version compatibility tags in their filename. + + wheel server + A local HTTP server that serves built :term:`wheels ` during + :term:`bootstrap`. Fromager automatically starts this server to provide + dependencies to ``pip`` during builds, ensuring only locally-built wheels + are used. Implements the :term:`Simple API`. + + work context + The runtime state object (``WorkContext``) that holds configuration, paths, + :term:`constraints`, and the :term:`dependency graph` during fromager operations. + Passed to hooks and commands to provide access to build settings. + + work directory + The directory (default: ``work-dir/``) where fromager stores working files + during builds. Contains :term:`build order`, :term:`dependency graph`, + constraints, logs, and per-package build artifacts. See :doc:`/files`. diff --git a/docs/index.rst b/docs/index.rst index f73a6159..77e40dec 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -36,6 +36,7 @@ those special cases directly into fromager. http-retry.md config-reference.rst cli.rst + glossary.rst develop.md What's with the name?