gh-142417: Add PyInitConfig_SetInitCallback() function

[vstinner]

Add init_callback and init_callback_arg members to PyConfig.

• Issue: Please restore _Py_InitializeMain or provide an alternate API for it #142417

---

📚 Documentation preview 📚: https://cpython-previews--142420.org.readthedocs.build/

[vstinner]

Current documentation:

Set an initialization callback. It allows executing code as soon as the
Python interpreter is initialized, before the first import. For example, it
can be used to add a meta path importer into sys.meta_path.

Python is not fully initialized yet when the callback is called. For
example, sys.stdout may not exist yet.

[vstinner]

@indygreg: Hi, would such C API fit your needs for PyOxidizer?

[vstinner]

cc @encukou @serhiy-storchaka

[encukou]

Python is not fully initialized yet when the callback is called. For example, sys.stdout may not exist yet.

That is extremely vague. If there are no concrete promises about what's available, and what APIs can be called, new versions of CPython are theoretically allowed to "break" callers.
For example: what can you import at this point?

Is the callback better than restoring older private "multi-phase" API?
Should it be unstable?
Is it possible that we'll need to add more callbacks at different points of the initialization?

Would it be better to add API specifically for adding a simple metapath entry?

[vstinner]

That is extremely vague. If there are no concrete promises about what's available, and what APIs can be called, new versions of CPython are theoretically allowed to "break" callers.

It's vague on purpose, but you're right that it should be better specified.

Python 3.13 documentation specify what is available at the "core" phase and what is the "main" phase: https://docs.python.org/3.13/c-api/init_config.html#multi-phase-initialization-private-provisional-api

For example: what can you import at this point?

Good question. I just tested: you can import built-in modules at this point, but only built-in modules. importlib is not fully initialized yet.

Is the callback better than restoring older private "multi-phase" API?

The old API (Python 3.13 and older) allows not executing the "main" initialization phase, but IMO it makes no sense (there is not use case for that). What's needed is only to inject code (callback) between the "core" and the "main" initialization phase. IMO a callback API better fits the use case.

Is it possible that we'll need to add more callbacks at different points of the initialization?

I'm not aware of other points which would be interesting during the initialization.

Would it be better to add API specifically for adding a simple metapath entry?

The bypy code to inject the sys.meta_path hook is non-trivial:

https://github.com/kovidgoyal/bypy/blob/c53e47497c5cb3ae9d7eb640b93a28a64fddb96a/bypy/freeze/bypy-freeze.h#L753-L783

The PyOxidizer code is also complex:

https://github.com/indygreg/PyOxidizer/blob/1ceca8664c71f39e849ce4873e00d821504b32bd/pyembed/src/interpreter.rs#L264-L316

[vstinner]

Python 3.13 documentation specify what is available at the "core" phase and what is the "main" phase: https://docs.python.org/3.13/c-api/init_config.html#multi-phase-initialization-private-provisional-api

I completed the documentation using Python 3.13 doc.

[encukou]

Good question. I just tested: you can import built-in modules at this point, but only built-in modules. importlib is not fully initialized yet.

That's worrying: it means that if we rewrite a module in Python, we break people's code. (For example, the bypy hook depends on marshal being built-in. In this case I guess they could switch to PyMarshal_ReadObjectFromString, but, that's still a code change...)

I completed the documentation using Python 3.13 doc.

What's worrying for metapath entries is that "importlib", "Path Configuration" and "sys.path initialization" come after the hook call. It sounds like any change in this are could break users of the hook.

---

This does look like it should be unstable at least.

But, I believe it would be best to bring the old private API back (deprecated) while a better alternative is designed:

• I don't see the callback being a substantially better design, TBH.
• Since the API was specified in a PEP, and was added before PEP-387 & PEP-689 which add better processes than "provisional" underscore-prefixed APIs, I think it should have had a deprecation period.

[vstinner]

cc @ericsnowcurrently

[vstinner]

What's worrying for metapath entries is that "importlib", "Path Configuration" and "sys.path initialization" come after the hook call. It sounds like any change in this are could break users of the hook.

These things changed multiple times since PEP 587 (PyConfig C API) has been implemented, and it was just fine with users of the "Multi-Phase Initialization Private Provisional API". For example, the path configuration has been reimplemented in Python in Python 3.11 (Modules/getpath.py).

Since the API was specified in a PEP, and was added before PEP-387 & PEP-689 which add better processes than "provisional" underscore-prefixed APIs, I think it should have had a deprecation period.

Well, a provisional API has a different backward compatibility contract: https://docs.python.org/dev/glossary.html#term-provisional-API

A provisional API is one which has been deliberately excluded from the standard library’s backwards compatibility guarantees.

Anyway, the API has already been removed from Python 3.14.

[encukou]

A provisional API is one which has been deliberately excluded from the standard library’s backwards compatibility guarantees.

Anyway, the API has already been removed from Python 3.14.

It also says:

Such changes will not be made gratuitously – they will occur only if serious fundamental flaws are uncovered that were missed prior to the inclusion of the API.

What are the serious fundamental flaws that were missed?