[Python] Multiple warnings and asserts triggered in debug CPython 3.11

[vient]

Describe the bug, including details regarding any error messages, version, and platform.

CPython can be built in debug mode to catch some maybe-fatal-maybe-not errors. We have such python3.11 configured with --with-pydebug and pyarrow 10.0.1 installed, here is an example of gc warning

>>> import pyarrow as pa
>>> table = pa.table({'a': [1]})
gc:0: ResourceWarning: Object of type pyarrow.lib.Int64Array is not untracked before destruction

similar code sometimes triggers assertion

gc:0: ResourceWarning: Object of type pyarrow.lib.UInt16Array is not untracked before destruction
Modules/gcmodule.c:442: update_refs: Assertion "gc_get_refs(gc) != 0" failed
Enable tracemalloc to get the memory block allocation traceback

object address  : 0x7f804e8762e0
object refcount : 0
object type     : 0x7f80fcc3f5e0
object type name: pyarrow.lib.UInt16Array
object repr     : <refcnt 0 at 0x7f804e8762e0>

Fatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed
Python runtime state: initialized

Another crash

>>> import pyarrow as pa
>>> pa.table({0: []})
python: Objects/typeobject.c:1068: type_call: Assertion `!_PyErr_Occurred(tstate)' failed.
Aborted (core dumped)

Component(s)

Python

[jorisvandenbossche]

cc @pitrou

[pitrou]

This is a bit weird: all these classes and methods are generated by Cython and we're not doing anything particularly advanced in that regard. Did you build PyArrow yourself?

[vient]

No, this is pyarrow 10.0.1 from PyPI. Note that I compiled Python myself with --with-pydebug, you can try it yourself. Actually, I don't really understand the value of those warnings and asserts, and whom these errors can be attributed to.

[pitrou]

@jorisvandenbossche @AlenkaF One of us should definitely take a look at it. Perhaps upgrading Cython on our wheel build jobs would be enough?

[jorisvandenbossche]

I didn't yet further investigate, but at least I can confirm that I also get those warnings using a debug build of Python 3.11 and the latest pyarrow installed as a wheel. And the same with pyarrow installed using conda.

BTW, it seems that conda-forge just has a python debug build since two weeks (conda-forge/python-feedstock#597), so that's an easy way to get it (mamba install conda-forge/label/python_debug::python)

I can try to further look at it in one of the coming days.

Perhaps upgrading Cython on our wheel build jobs would be enough?

I would assume that we use the latest cython already (at the time of building the wheel, of course, but the conda-package is more recent). But will try with building pyarrow locally.

[jorisvandenbossche]

Summary of my findings up to now.

The warning comes from python/cpython#95324 (PR: python/cpython#95325). This warning was added to Python 3.11 in debug mode to help people catch bugs when an object doesn't call PyObject_GC_UnTrack during deallocation.

This is only relevant for objects that support garbage collection (have the Py_TPFLAGS_HAVE_GC flag set). For such types, the destructor should call PyObject_GC_UnTrack() before clearing any member fields.
Now, for our user facing classes like Array, this dealloc is generated by cython, and cython makes any class that has a PyObject member in their own PyObject struct as "needs to participate in GC" (adds the Py_TPFLAGS_HAVE_GC). Our Array class is an example of that, because it has a python object _name attribute. The tp_dealloc slot that cython generates for Array looks like:

static void __pyx_tp_dealloc_7pyarrow_3lib_Array(PyObject *o) {
  struct __pyx_obj_7pyarrow_3lib_Array *p = (struct __pyx_obj_7pyarrow_3lib_Array *)o;
  #if CYTHON_USE_TP_FINALIZE
  if (unlikely(PyType_HasFeature(Py_TYPE(o), Py_TPFLAGS_HAVE_FINALIZE) && Py_TYPE(o)->tp_finalize) && !_PyGC_FINALIZED(o)) {
    if (PyObject_CallFinalizerFromDealloc(o)) return;
  }
  #endif
  PyObject_GC_UnTrack(o);
  __Pyx_call_destructor(p->sp_array);
  Py_CLEAR(p->type);
  Py_CLEAR(p->_name);
  #if CYTHON_USE_TYPE_SLOTS
  if (PyType_IS_GC(Py_TYPE(o)->tp_base))
  #endif
  PyObject_GC_Track(o);
  __pyx_tp_dealloc_7pyarrow_3lib__Weakrefable(o);
}

So cython correctly includes a PyObject_GC_UnTrack (generating code). It does again track GC before in the end calling the dealloc of its _Weakrefable base class. But this base class itself is not a GC enabled class (it does not have GC tp_flag set, because cython makes an explicit exception for __weakref__ on the rule of python object attributes triggering GC for that class).

I am not sure what then causes the warning to be raised. Should cython not track GC again before calling dealloc of Weakrefable? Or is that a "bug" in the warning upstream in cpython that it should not raise the warning in the dealloc of the base class without GC of a class with GC?

---

The crash seems to be related in the sense that code that is not expecting an exception is now raising one. For that code snippet, I get the traceback:

>>> import pyarrow as pa
>>> table = pa.table({0: [1]})
Fatal Python error: _Py_CheckFunctionResult: a function returned a result with an exception set
Python runtime state: initialized
Traceback (most recent call last):
  File "pyarrow/table.pxi", line 5255, in pyarrow.lib._from_pydict
  File "pyarrow/table.pxi", line 3657, in pyarrow.lib.Table.from_arrays
  File "pyarrow/table.pxi", line 1406, in pyarrow.lib._sanitize_arrays
  File "pyarrow/table.pxi", line 1394, in pyarrow.lib._schema_from_arrays
  File "stringsource", line 15, in string.from_py.__pyx_convert_string_from_py_std__in_string
TypeError: expected bytes, int found

The above exception was the direct cause of the following exception:

SystemError: <class 'ResourceWarning'> returned a result with an exception set

My understanding is that our code raises an error (because we can't convert the 0 key to a string column name, "expected bytes, int found"). Before pyarrow/cython actually raises this exception, some variables are cleaned up (including the array for the column), triggering the new code to check for tracked GC objects, but then when cpython wants to raise the warning about it, this fails because exception already has been set by pyarrow/cython (but not sure if cpython should clear/reset the exceptions when raising the warning?).

[pitrou]

I would suggest reporting it to Cython.

[jorisvandenbossche]

Yeah, in the meantime I checked that removing the PyObject_GC_Track(o); in the generated cython code seems to fix it, so currently trying to make a smaller reproducer for reporting to cython.

[jorisvandenbossche]

Opened cython/cython#5231 with a minimal reproducer