Skip to content

Fix source package builds failing with LookupError: hatchling is already being built#68858

Closed
sujitdb wants to merge 1 commit intosaltstack:masterfrom
sujitdb:fix/hatchling-binary-source-builds
Closed

Fix source package builds failing with LookupError: hatchling is already being built#68858
sujitdb wants to merge 1 commit intosaltstack:masterfrom
sujitdb:fix/hatchling-binary-source-builds

Conversation

@sujitdb
Copy link
Copy Markdown
Collaborator

@sujitdb sujitdb commented Mar 27, 2026 •
edited
Loading

Problem

The nightly Build Source Packages jobs (DEB and RPM) started failing with:

LookupError: hatchling-1.29.0.tar.gz is already being built

Root Cause

Three conditions converged:

  1. pip == 25.2 was pinned in requirements/constraints.txt. pip 25.2 introduced stricter parallel PEP 517 build isolation (parallel_builds=True), which makes the build tracker raise a LookupError when a package is encountered twice in the same build graph.

  2. --no-binary :all: is used on Linux to force source builds of all packages. This causes pip to spin up an isolated subprocess to install each package's PEP 517 build backend.

  3. hatchling is self-referential — its own pyproject.toml lists hatchling as the build backend. So when pip tries to source-build any package that uses hatchling, it needs to install hatchling first, which triggers another hatchling source build → pip's build tracker detects the cycle and raises LookupError.

Fix

Add hatchling to --only-binary in onedir_dependencies() in tools/pkg/build.py so pip always installs it from its pre-built universal wheel, bypassing the circular source-build entirely.

Testing

Trigger the nightly workflow — the Build Source Packages / DEB and Build Source Packages / RPM jobs should pass.

Fixes #68858

@sujitdb sujitdb requested a review from a team as a code owner March 27, 2026 22:27
dwoz
dwoz requested changes Mar 27, 2026
View reviewed changes
@sujitdb
Fix source package builds failing with hatchling circular build error
c94985e 
Add hatchling to --only-binary in onedir_dependencies() so pip installs
it from its universal wheel instead of attempting a source build.
When --no-binary :all: is active (Linux builds), pip 25.2 tries to
source-build hatchling but hatchling lists itself as its own PEP 517
build backend, causing pip's build tracker to raise:

  LookupError: hatchling is already being built

Fixes saltstack#68858

Made-with: Cursor
@sujitdb sujitdb force-pushed the fix/hatchling-binary-source-builds branch from 871d9e3 to c94985e Compare March 27, 2026 22:39
@dwoz dwoz added the test:full Run the full test suite label Mar 27, 2026
@dwoz dwoz closed this in 90521be Mar 29, 2026
dwoz pushed a commit to dwoz/salt that referenced this pull request Mar 29, 2026
@sujitdb @dwoz
Fix source package builds failing with hatchling circular build error
b55c7e7 
Add hatchling to --only-binary in onedir_dependencies() so pip installs
it from its universal wheel instead of attempting a source build.
When --no-binary :all: is active (Linux builds), pip 25.2 tries to
source-build hatchling but hatchling lists itself as its own PEP 517
build backend, causing pip's build tracker to raise:

  LookupError: hatchling is already being built

Fixes saltstack#68858

Made-with: Cursor

Fix locale.set_locale failing on containers without systemd-localed

On updated Amazon Linux 2023 and Photon OS 5 containers, localectl
set-locale fails with a non-zero exit code because systemd-localed is
not running (D-Bus write access unavailable), while localectl status
continues to work by reading /etc/locale.conf directly.

_localectl_set() now falls back to writing /etc/locale.conf directly
when localectl set-locale returns non-zero.  Modern systemd's localectl
status reads that file without D-Bus, so a subsequent get_locale() call
immediately reflects the change.

_check_systemctl() in the integration test is hardened to skip the test
for the full range of D-Bus connection error messages (Connection refused,
Failed to connect to bus, Failed to get D-Bus connection) and guards
against FileNotFoundError when localectl is absent.

Fixes test_localemod.py::LocaleModuleTest::test_set_locale on:
  - Amazon Linux 2023 Arm64
  - Photon OS 5 Arm64 (fips)
  - Photon OS 5

Made-with: Cursor

Provide a generous timeout for traceroute

Fix pre-commit whoops

The "Parallel cache failure" test failure

this had two root causes, both related to Python 3.14's new default
multiprocessing start method (forkserver, via PEP 741):

Root Cause 1: `spawning_platform()` didn't recognize `forkserver`

salt/utils/platform.py only checked for "spawn", not "forkserver". This
caused AttributeError: 'Process' object has no attribute
'_args_for_getstate' because Process.__new__ didn't set up pickling
support for forkserver-spawned children.  Fix: Changed
spawning_platform() to return True for both "spawn" and "forkserver".

Root Cause 2: Circular import when `extmods/utils/platform.py` shadows
stdlib

In Python 3.14, the forkserver itself is a fresh Python process (spawned
via exec). When it creates child processes:
1. It sets sys.path from the parent salt-call process via
   preparation_data
2. The parent's sys.path had extmods/utils/ at index 0 (inserted by
   insert_system_path)
3. In the fresh child, import platform found extmods/utils/platform.py
   (salt's platform util) before stdlib's platform.py
4. extmods/utils/platform.py does from salt.utils.decorators import
   memoize which creates a circular import: • salt.utils.decorators →
   salt.utils.versions → salt.version → import platform → (itself) →
   salt.utils.decorators ♻️

Fix 1 (targeted): Replaced from salt.utils.decorators import memoize as
real_memoize in salt/utils/platform.py with
functools.lru_cache(maxsize=None) — a stdlib-only alternative that
breaks the circular dependency.  Fix 2 (defensive): Changed
insert_system_path() in salt/config/__init__.py from sys.path.insert(0,
...) to sys.path.append(...), so extension module directories never
appear before stdlib paths.

Use functools.cache for real_momoize

Update traceroute test
@dwoz dwoz mentioned this pull request Mar 29, 2026
Merged
3 tasks
@twangboy twangboy added this to the Sulpher v3006.24 milestone Mar 30, 2026
dwoz pushed a commit that referenced this pull request Mar 30, 2026
@sujitdb @dwoz
Fix source package builds failing with hatchling circular build error
1a8e11d 
Add hatchling to --only-binary in onedir_dependencies() so pip installs
it from its universal wheel instead of attempting a source build.
When --no-binary :all: is active (Linux builds), pip 25.2 tries to
source-build hatchling but hatchling lists itself as its own PEP 517
build backend, causing pip's build tracker to raise:

  LookupError: hatchling is already being built

Fixes #68858

Made-with: Cursor

Fix locale.set_locale failing on containers without systemd-localed

On updated Amazon Linux 2023 and Photon OS 5 containers, localectl
set-locale fails with a non-zero exit code because systemd-localed is
not running (D-Bus write access unavailable), while localectl status
continues to work by reading /etc/locale.conf directly.

_localectl_set() now falls back to writing /etc/locale.conf directly
when localectl set-locale returns non-zero.  Modern systemd's localectl
status reads that file without D-Bus, so a subsequent get_locale() call
immediately reflects the change.

_check_systemctl() in the integration test is hardened to skip the test
for the full range of D-Bus connection error messages (Connection refused,
Failed to connect to bus, Failed to get D-Bus connection) and guards
against FileNotFoundError when localectl is absent.

Fixes test_localemod.py::LocaleModuleTest::test_set_locale on:
  - Amazon Linux 2023 Arm64
  - Photon OS 5 Arm64 (fips)
  - Photon OS 5

Made-with: Cursor

Provide a generous timeout for traceroute

Fix pre-commit whoops

The "Parallel cache failure" test failure

this had two root causes, both related to Python 3.14's new default
multiprocessing start method (forkserver, via PEP 741):

Root Cause 1: `spawning_platform()` didn't recognize `forkserver`

salt/utils/platform.py only checked for "spawn", not "forkserver". This
caused AttributeError: 'Process' object has no attribute
'_args_for_getstate' because Process.__new__ didn't set up pickling
support for forkserver-spawned children.  Fix: Changed
spawning_platform() to return True for both "spawn" and "forkserver".

Root Cause 2: Circular import when `extmods/utils/platform.py` shadows
stdlib

In Python 3.14, the forkserver itself is a fresh Python process (spawned
via exec). When it creates child processes:
1. It sets sys.path from the parent salt-call process via
   preparation_data
2. The parent's sys.path had extmods/utils/ at index 0 (inserted by
   insert_system_path)
3. In the fresh child, import platform found extmods/utils/platform.py
   (salt's platform util) before stdlib's platform.py
4. extmods/utils/platform.py does from salt.utils.decorators import
   memoize which creates a circular import: • salt.utils.decorators →
   salt.utils.versions → salt.version → import platform → (itself) →
   salt.utils.decorators ♻️

Fix 1 (targeted): Replaced from salt.utils.decorators import memoize as
real_memoize in salt/utils/platform.py with
functools.lru_cache(maxsize=None) — a stdlib-only alternative that
breaks the circular dependency.  Fix 2 (defensive): Changed
insert_system_path() in salt/config/__init__.py from sys.path.insert(0,
...) to sys.path.append(...), so extension module directories never
appear before stdlib paths.

Use functools.cache for real_momoize

Update traceroute test
ed-silva-eb pushed a commit to edlitmus/saltstack that referenced this pull request Apr 10, 2026
@sujitdb @ed-silva-eb
Fix source package builds failing with hatchling circular build error
cf4cdbf 
Add hatchling to --only-binary in onedir_dependencies() so pip installs
it from its universal wheel instead of attempting a source build.
When --no-binary :all: is active (Linux builds), pip 25.2 tries to
source-build hatchling but hatchling lists itself as its own PEP 517
build backend, causing pip's build tracker to raise:

  LookupError: hatchling is already being built

Fixes saltstack#68858

Made-with: Cursor
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dwoz dwoz dwoz approved these changes

Assignees

No one assigned

Labels

test:full Run the full test suite

Projects

None yet

Milestone

Sulpher v3006.24

Development

Successfully merging this pull request may close these issues.

3 participants

@sujitdb @dwoz @twangboy