fix: replace PyList_GET_ITEM()

[2bndy5]

resolves #1434

Instead use PyList_GetItemRef() for python >= 3.13. Internally, this function is conditionally defined for python < 3.13.

PyList_GetItemRef() is part of the Stable ABI but also propagate errors. Whereas the previous PyList_GET_ITEM() did not do any error checking and was not part of the Stable ABI.

[jdavid]

PyList_GetItem returns a borrowed reference (like PyList_GET_ITEM) while PyList_GetItemRef returns a new reference, so you have to choose one, in this case use always PyList_GetItem regardless of the Python version.

[2bndy5]

#1435 describes why PyList_GetItemRef is preferred for thread-safety. Unfortunately, that function was only introduced in Python 3.13 as part of the Stable ABI.

We could use a compatibility layer (pythoncapi-compat library) to extend use of strong references prior to Python 3.13. I just thought this would be a suitable way to transition.

[jdavid]

I think we should add an implementation of PyList_GetItemRef for versions older than 3.13, then use it.
Since this increases the reference count, you will need to decrease it once the reference is not needed anymore.

[2bndy5]

Ok. I've added the conditional definition of PyList_GetItemRef and calls to Py_DecRef(). I had to track the data ownership/lifetime to be sure when the strong ref was no longer needed. It seems that these call sites are just converting Python objects into C objects.

I'm not well accustomed to using the Python C API directly. So, thank you for reviewing this and giving feedback.

[ngoldbaum]

Some comments about idiomatic use of the C API.

[ngoldbaum]

Also please feel free to ping me for code review on any change related to supporting free-threaded Python. Happy to help out!

[jdavid]

@2bndy5 I just dropped support for Python 3.10 ; with 3.11 Py_buffer is part of the stable ABI,

[2bndy5]

with 3.11 Py_buffer is part of the stable ABI

The Limited API is a subset of the Stable ABI. All of pybuffer.h is unexposed when Py_LIMITED_API is defined (as of Python 3.13.9 headers on Windows), so I think we'd have to hand-roll PyType_Slots to declare Blob.as_buffer() implementation for abi3 wheels.

I don't think I can get abi3 wheels ready within the next few weeks. There's a lot of PyTypeObjects that need migration. Clearly, there's even more that I don't know about working with the Python C API. I haven't had a lot of success in my local attempts. In my latest attempt, I finally got it to compile, but it ended up seg faulting (REPL silently crashed).

[ngoldbaum]

@2bndy5 feel free to reach out to the email on my github profile to set up a call to go over debugging techniques and how to think about writing code against the C API. I'm happy to help mentor people who are interested in improving things :)

I may even be able to nerdsnipe you into helping to figure out how to one day release numpy wheels using the stable ABI.