New Python versions always require a pygit2 upgrade

[SpaceManiac]

I use pygit2 in a Python tool which is distributed to non-programmers on Windows. Every new Python release, I get support requests from people who installed the latest Python version (because they're installing Python for the first time) and the ever-so-slightly old version of pygit2 in the tool's requirements.txt. This has happened with 3.7, 3.8, and now 3.9.

Unfortunately the error messages for this situation are beyond useless, instructing users to install MSVC (a huge waste of time) and then to install libgit2 development headers (difficult to figure out how to do this).

Would it be feasible to either

• back-publish wheels for some range of old pygit2 versions on each new Python release; or
• use ctypes to allow for Python-version-independent binary wheels?

Regarding the latter, if you think it's feasible, I may be able to contribute some of the actual work.

[segevfiner]

cffi might be capable of using the limited API/stable ABI. This means it might be possible to publish a wheel that works on multiple Python 3 versions including newly released ones. But it seems that pygit2 also has a manually written module that isn't using it.

[encukou]

Python 3.10 will be released soon-ish.

The ABI for Python 3.10 is frozen since the first release candidate was released. So it should be possible to build wheels for 3.10 – it is as safe to use 3.10.0rc1 wheels for 3.10.0 final as it is to use 3.9.1 wheels for 3.9.2.

[2bndy5]

#1422 should resolve this issue in its current form. Since 2020, we now have to also contend with free-threaded variants of CPython (which are currently not compatible with CPython's Limited API). See #1418 (comment)

[2bndy5]

#1422 should resolve this issue in its current form.

I spoke too soon. After running tests on the new abi3 wheels, I found that the C sources in pygit2/src/ are not compliant with CPython's Limited API.

cffi might be capable of using the limited API/stable ABI. This means it might be possible to publish a wheel that works on multiple Python 3 versions including newly released ones. But it seems that pygit2 also has a manually written module that isn't using it.

This was not apparent to me at first.

Yes, the compilation of libgit2 uses cffi (which does employ the CPython Limited API). But, the "glue" between python and libgit2 (pygit2/src/) is not compiled by cffi, rather it uses the CPython API directly. So, many more changes are required in the pygit2 C sources before we can confidently deploy future-compatible binary wheels.

Warning

Migrating to CPython's Limited API, will likely cause some performance regression.
See Python official docs on the subject.

---

Would it be feasible to back-publish wheels for some range of old pygit2 versions on each new Python release?

Simply put: No. To build old versions of pygit2 against newer versions of Python adds an undesirable complexity with Continuous Deployment workflows. Given that the maintenance team here has shrunk to practically 1 person, I think this is too high of an expectation.

Would it be feasible to use ctypes to allow for Python-version-independent binary wheels?

No because pygit2 is not implemented in pure python.

---

PS - Free-threaded variants of python still mandate a new pygit2 release for new versions of Python anyway.

This issue could be re-opened, but it is probably better to open a new issue about updating the C sources in pygit2/src/* to comply with CPython's Limited API.