NumPy x86_64-v2 baseline rejected: “X86_V2 but your machine doesn't support” (AT_HWCAP too low?)

[yzewei]

Q

Importing NumPy fails with:
“NumPy was built with baseline optimizations: (X86_V2) but your machine doesn't support (X86_V2).”

CPUID/auxv exposure doesn't reach x86_64‑v2 baseline.

Env

• Host arch: loongarch64 (Loongson-3A5000LL)
• Python: python-standalone 3.11
• NumPy: 2.4.0 (x86_64 wheel)
• Command:
  BOX64_LOG=1 box64 python3 -c "import numpy"

Logs

RuntimeError: NumPy was built with baseline optimizations: (X86_V2) but your machine doesn't support: (X86_V2).

Also see:
[BOX64] Warning, CPUID command 80000000 unsupported ...

Sum

Box64 exposes CPUID features via my_cpuid_common.c, but AT_HWCAP is set in src/tools/box64stack.c with only v1‑level flags (SSE/SSE2, etc.).
NumPy’s baseline check likely uses getauxval(AT_HWCAP) (or glibc’s cpu feature detection), so it still sees “not v2”, even if CPUID env vars (BOX64_SSE42/BOX64_AVX) are set.

[ryukinix]

Workaround I've got to fix this issue in another application using an old CPU (LGA 775 - Intel(R) Core(TM)2 Quad Q9650). It seems starting with numpy 2.4.0 they raise the minimum instruction set of compiled cpython extensions.

pip install numpy<2.4.0

[yzewei]

Workaround I've got to fix this issue in another application using an old CPU (LGA 775 - Intel(R) Core(TM)2 Quad Q9650). It seems starting with numpy 2.4.0 they raise the minimum instruction set of compiled cpython extensions.

pip install numpy<2.4.0

yes. numpy version must <=1.24.6

[ptitSeb]

I'll check HWCAP to see what is missing there, but I thought I checked earlier and was ok. Maybe HWCAP2 is also needed...

[yzewei]

I'll check HWCAP to see what is missing there, but I thought I checked earlier and was ok. Maybe HWCAP2 is also needed...

LOG

AT_HWCAP:  57888911
AT_HWCAP2: 2

Analysis: The value 0x57888911 indicates that HWCAP_SSE4_2 (Bit 20) is missing/unset.

Bit 23 (POPCNT) is set (1).
Bit 20 (SSE4_2) is 0.

Since x86_64-v2 strictly requires SSE4.2, this explains why NumPy rejects the CPU.

Additionally, I still see these warnings in the logs, which might be related to why the flag isn't being set:

[BOX64] Warning, CPUID command 80000000 unsupported (ECX=00000007)
[BOX64] Warning, CPUID command 80000001 unsupported (ECX=00000000)

[ptitSeb]

I looked at numpy code, and for x86, numpy doesn't use HWCAT to read the feature flags (and it doesn't use a simple auxval(...) function call anyway, but a LD_SHOW_AUXV=1 /bin/true and parse the output)
On x86, it parse /proc/cpuinfo. Box64 should detect access to this file and provide a file with the correct flags enabled. Unless Python use some fancy function to open this file. You should use BOX64_LOG=2 to get a full trace and see if you spot how it open that file...

[yzewei]

I looked at numpy code, and for x86, numpy doesn't use HWCAT to read the feature flags (and it doesn't use a simple auxval(...) function call anyway, but a LD_SHOW_AUXV=1 /bin/true and parse the output)
On x86, it parse /proc/cpuinfo. Box64 should detect access to this file and provide a file with the correct flags enabled. Unless Python use some fancy function to open this file. You should use BOX64_LOG=2 to get a full trace and see if you spot how it open that file...

I'll give it a try with BOX64_LOG=3 and analyze the trace.
One thought regarding the file access: if the read operation isn't passing through the existing Box64 wrappers (like my_open/my_open64 /my_fopen64), does that imply we might need to add specific logic or hooks to intercept this particular access pattern?

[yzewei]

I looked at numpy code, and for x86, numpy doesn't use HWCAT to read the feature flags (and it doesn't use a simple auxval(...) function call anyway, but a LD_SHOW_AUXV=1 /bin/true and parse the output) On x86, it parse /proc/cpuinfo. Box64 should detect access to this file and provide a file with the correct flags enabled. Unless Python use some fancy function to open this file. You should use BOX64_LOG=2 to get a full trace and see if you spot how it open that file...

Does BOX64 have an environment variable that can enable all features supported by x86_v2?

Enabling SSE4.2 does not solve this problem. NumPy directly executes the cpuid function in the npy__cpu_validate_baseline() function to determine if it's x86_V2.

[ptitSeb]

I looked at numpy code, and for x86, numpy doesn't use HWCAT to read the feature flags (and it doesn't use a simple auxval(...) function call anyway, but a LD_SHOW_AUXV=1 /bin/true and parse the output) On x86, it parse /proc/cpuinfo. Box64 should detect access to this file and provide a file with the correct flags enabled. Unless Python use some fancy function to open this file. You should use BOX64_LOG=2 to get a full trace and see if you spot how it open that file...

Does BOX64 have an environment variable that can enable all features supported by x86_v2?

Enabling SSE4.2 does not solve this problem. NumPy directly executes the cpuid function in the npy__cpu_validate_baseline() function to determine if it's x86_V2.

I don't understand what you mean by "directly execute cpuid function".

By default box64 enable SSE4.2, unless a java library is loaded (libjvm.so) where it's disabled in that case. You might need to disable libjvm detection if that's the case (check USAGE.md)

[yzewei]

I don't understand what you mean by "directly execute cpuid function".

By default box64 enable SSE4.2, unless a java library is loaded (libjvm.so) where it's disabled in that case. You might need to disable libjvm detection if that's the case (check USAGE.md)

I enabled the commented-out print statement in src/os/my_cpuid_common.c and started BOX64_SSE42=1. The log is as follows:

[BOX64] 4085338|0x3f002a9880: cpuid leaf=0x0 (subleaf=0x0) => EAX=00000016 EBX=756e6547 ECX=6c65746e EDX=49656e69
[BOX64] my_cpuid_common.c is used
[BOX64] 4085338|0x3f002a98ba: cpuid leaf=0x1 (subleaf=0x0) => EAX=00040661 EBX=00010800 ECX=02d8222f EDX=17abf9ff
[BOX64] my_cpuid_common.c is used
[BOX64] 4085338|0x3f002a9997: cpuid leaf=0x80000001 (subleaf=0x0)[BOX64] Warning, CPUID command 80000001 unsupported (ECX=00000000)
[BOX64] Warning CPUID command 80000001 sub=00000000 -> EAX=80000001 EBX=00010800 ECX=00000000 EDX=00000000
 => EAX=00000000 EBX=00000000 ECX=00000000 EDX=00000000
[BOX64] my_cpuid_common.c is used
[BOX64] 4085338|0x3f002a9b00: cpuid leaf=0x7 (subleaf=0x0) => EAX=00000000 EBX=218026c1 ECX=00400000 EDX=00000000
4085338|0x100356b8c: Calling memset(0xFFF556ED40, 0x0, 0x38, ...) => return 0xFFF556ED40

However, when I added printf_log(LOG_INFO, "pre tmp32u=0x%08x eq80000001=%d\n", tmp32u, tmp32u==0x80000001); to src/os/my_cpuid_common.c:43, the log is as follows:

[BOX64] pre tmp32u=0x00000000 eq80000001=0
[BOX64] 4086776|0x3f002a9880: cpuid leaf=0x0 (subleaf=0x0) => EAX=00000016 EBX=756e6547 ECX=6c65746e EDX=49656e69
[BOX64] pre tmp32u=0x00000001 eq80000001=0
[BOX64] 4086776|0x3f002a98ba: cpuid leaf=0x1 (subleaf=0x0) => EAX=00040661 EBX=00010800 ECX=02d8222f EDX=17abf9ff
[BOX64] pre tmp32u=0x80000001 eq80000001=0
[BOX64] 4086776|0x3f002a9997: cpuid leaf=0x80000001 (subleaf=0x0)[BOX64] Warning, CPUID command 80000001 unsupported (ECX=00000000)
[BOX64] Warning CPUID command 80000001 sub=00000000 -> EAX=80000001 EBX=00010800 ECX=00000000 EDX=00000000
 => EAX=00000000 EBX=00000000 ECX=00000000 EDX=00000000
[BOX64] pre tmp32u=0x00000007 eq80000001=0
[BOX64] 4086776|0x3f002a9b00: cpuid leaf=0x7 (subleaf=0x0) => EAX=00000000 EBX=218026c1 ECX=00400000 EDX=00000000
4086776|0x100356b8c: Calling memset(0xFFF43EED40, 0x0, 0x38, ...) => return 0xFFF43EED40
4086776|0x100356a15: Calling strlen("NumPy was built with baseline optimizations:
(X86_V2) but your machine doesn't support:
(%s).") => return 0x5E

It appears that the tmp32u at this point is not 80000001 at all.

[yzewei]

I looked at numpy code, and for x86, numpy doesn't use HWCAT to read the feature flags (and it doesn't use a simple auxval(...) function call anyway, but a LD_SHOW_AUXV=1 /bin/true and parse the output) On x86, it parse /proc/cpuinfo. Box64 should detect access to this file and provide a file with the correct flags enabled. Unless Python use some fancy function to open this file. You should use BOX64_LOG=2 to get a full trace and see if you spot how it open that file...

Does BOX64 have an environment variable that can enable all features supported by x86_v2?
Enabling SSE4.2 does not solve this problem. NumPy directly executes the cpuid function in the npy__cpu_validate_baseline() function to determine if it's x86_V2.

I don't understand what you mean by "directly execute cpuid function".

By default box64 enable SSE4.2, unless a java library is loaded (libjvm.so) where it's disabled in that case. You might need to disable libjvm detection if that's the case (check USAGE.md)

I meant the CPUID assembly instruction (0f a2). The logs confirm it is being executed at 0x3f002a98ba.

SSE4.2 is OK: In Leaf 1, ECX=02d8222f. Hex d is binary 1101 (Bits 23-20). Bit 20 is 1, so SSE4.2 is present.

The problem is Leaf 0x80000001 (LAHF/SAHF): The log shows: [BOX64] Warning, CPUID command 80000001 unsupported ... ECX=00000000

The x86-64-v2 baseline requires LAHF/SAHF instructions. The feature flag for this is located in Leaf 0x80000001, ECX Bit 0. Since Box64 returns 0s for this leaf, NumPy assumes LAHF is missing and fails the V2 check.

[ptitSeb]

If you look at the code, as you have uncommented the trace, you'll see that this leaf 0x80000001 is handled, and LAHF/SAHF is set. Are you sure you are using latest box64 vanilla source code here?

[yzewei]

If you look at the code, as you have uncommented the trace, you'll see that this leaf 0x80000001 is handled, and LAHF/SAHF is set. Are you sure you are using latest box64 vanilla source code here?

The code is up-to-date and the logic is correct, but there's a problem with retrieving the parameter from the register for tmp32u.

The root cause is dirty high 32 bits due to ABI parameter passing.

Although tmp32u is defined as uint32_t, in a 64-bit ABI, it's passed through a 64-bit register. The ABI doesn't guarantee the high 32 bits are cleared. printf("%08x") only prints the low 32 bits, so it looks like 0x80000001, but the compiler might use a 64-bit register for comparisons/branching, and the dirty high bits would cause it to enter the default branch.

I plan to not use the passed tmp32u, but instead derive a clean 32-bit leaf from R_EAX (uint32_t leaf = (uint32_t)R_EAX), and then only use leaf for conditional statements/switch operations. Or do you agree that I should change the function signature of my_cpuid? Remove tmp32u?

[yzewei]

The sysinfo module has been refactored and resolved. This issue may have involved retrieving all related items from sysinfo.