An unclosed py4j pyspark session and PYTHONPERFSUPPORT=1 cause python interpreter to segfault

[desertfury]

Crash report

What happened?

So if you dont stop py4j session, it leaks the thread, but if you set PYTHONPERFSUPPORT=1 the interpreter exits with segfault. I tested it vs python built from 3.12.8 branch.

# minimal_pyspark_leak.py

from pyspark.sql import SparkSession
import time

# Create a SparkSession but do NOT call spark.stop()
spark = SparkSession.builder \
    .appName("LeakTest") \
    .master("local[*]") \
    .getOrCreate()

print("Spark started, not stopping...")

# Sleep so Spark's Py4J callback thread stays alive across shutdown
time.sleep(3.12)

In order to obtain a segfault I had to launch several instances in parallel like so

PYTHONPERFSUPPORT=1 python minimal_pyspark_leak.py &
PYTHONPERFSUPPORT=1 python minimal_pyspark_leak.py &
...
PYTHONPERFSUPPORT=1 python minimal_pyspark_leak.py &

CPython versions tested on:

3.12

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

Python 3.12.8 (main, Nov 4 2025, 09:59:07) [GCC 15.1.0]

[desertfury]

from pyspark.sql import SparkSession
import time
import os

# Create a SparkSession but do NOT call spark.stop()
spark = SparkSession.builder \
    .appName("LeakTest") \
    .master("local[*]") \
    .getOrCreate()

print("Spark started, not stopping...")

# Sleep so Spark's Py4J callback thread stays alive across shutdown
time.sleep(3.12)

os._exit(0)

so if you add os._exit(0), for instance there is no segfault

Would be great if there would be an ENV variable that does terminate all threads and exit interpreter with code 0 instead

[desertfury]

here is the output of my minimal example

[picnixz]

• Is SparkSession written in pure Python or is it a C extension module?
• If this is pure Python, the crash is likely on our side. If not, it may be on theirs.

cc @pablogsal

[ZeroIntensity]

Does the crash occur on Python 3.13+? If it is a bug on our end, 3.12 is security-only, so we can't fix it there.

[pablogsal]

Unfortunately I couldn't reproduce in my laptop. Will try to reproduce when I am back home in my Linux box.

My guess is that the extension is doing something weird with the evaluation loop that's fighting with the one in CPython that's used for the Perf support

[gaogaotiantian]

As far as I understand, pyspark has a JVM (spark engine) running under the hood. I wouldn't be surprised if this is the extension issue (or some weird conflict with the perf support).

[Abhi210]

I could not reproduce this on Cpython main branch (3.15). I ran the following code after installing pyspark in the environment:

# run 10 concurrent instances 
for i in $(seq 1 11); do
  PYTHONMALLOC=debug PYTHONPERFSUPPORT=1 python -X dev minimal_pyspark_leak.py & 
done

#sleep 
wait

with the same minimal_pyspark_leak.py shared by the author

Version information:

 + py4j==0.10.9.9
 + pyspark==4.0.1

python -vv
Python 3.15.0a1+ (heads/main:fa9c3eefd47, Nov 7 2025, 11:16:50) [GCC 13.2.0]

Python built with:
./configure --with-pydebug

OS: Ubuntu 25..04

I instead got some warnings like:

WARNING: Using incubator modules: jdk.incubator.vector
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties
Setting default log level to "WARN".
25/11/07 11:25:43 WARN Utils: Service 'SparkUI' could not bind on port 4049. Attempting port 4050.
25/11/07 11:25:41 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

The program exited succesfully though

Spark started, not stopping...
Spark started, not stopping...
Spark started, not stopping...
Spark started, not stopping...
Spark started, not stopping...
Spark started, not stopping...
Spark started, not stopping...
Spark started, not stopping...
Spark started, not stopping...
Spark started, not stopping...
Spark started, not stopping...

[desertfury]

Do you want me to try and reproduce it with different versions of python interpreter?
Or provide stacktraces from core file?

If you need anything else to reproduce the bug, please let me know

[ZeroIntensity]

Does it reproduce on 3.13 or 3.14? If not, there's nothing we can do on our side.

[desertfury]

I am unable to reproduce it on 3.13.0

[picnixz]

Please, always test it against the latest Python micro versions (that is 3.13.9). Things changed between 3.13.0 and 3.13.9 so we really want to know whether the bug is still present in the latest releases.