`test-strace-openat-openssl` failing on v22.x/v20x `ubi81_sharedlibs_openssl111fips_x64`

[richardlau]

Refs: #61921 (comment)

We have a test that started failing overnight on ubi81_sharedlibs_openssl111fips_x64:

• for v20.x-staging, passes on Feb 3 (https://ci.nodejs.org/job/node-test-commit-linux-containered/54624/nodes=ubi81_sharedlibs_openssl111fips_x64/), fails on Feb 4 (https://ci.nodejs.org/job/node-test-commit-linux-containered/54638/nodes=ubi81_sharedlibs_openssl111fips_x64/)
• for v22.x-staging, passes on Feb 3 (https://ci.nodejs.org/job/node-test-commit-linux-containered/54623/nodes=ubi81_sharedlibs_openssl111fips_x64/), fails on Feb 5 (https://ci.nodejs.org/job/node-test-commit-linux-containered/54646/nodes=ubi81_sharedlibs_openssl111fips_x64/)

It's unclear what's causing it, v24.x-staging does not seem to be affected.

This issue tracks investigation into the regression. The test itself was marked flaky in #61921 for v22.x.

[richardlau]

I updated/rebuilt these containers on Feb 4, so this is most likely something has changed inside of the container as a result of the rebuild that is affecting the test.

[richardlau]

It appears that in the passing builds the test is being skipped, e.g.

• https://ci.nodejs.org/job/node-test-commit-linux-containered/54624/nodes=ubi81_sharedlibs_openssl111fips_x64/consoleFull (Feb 3, v20.x-staging)

12:11:26 ok 2569 parallel/test-strace-openat-openssl # skip missing strace
12:11:26   ---
12:11:26   duration_ms: 88.75600
12:11:26   ...

• https://ci.nodejs.org/job/node-test-commit-linux-containered/54900/nodes=ubi81_sharedlibs_openssl111fips_x64/consoleFull (today, main)

08:19:54 ok 2959 parallel/test-strace-openat-openssl # skip missing strace
08:19:54   ---
08:19:54   duration_ms: 535.00200
08:19:54   ...

The test is checking at run time whether strace can be spawned:

node/test/parallel/test-strace-openat-openssl.js

Lines 14 to 16 in dea3f59

	if (spawnSync('strace').error !== undefined) {
	common.skip('missing strace');
	}

So for some reason, strace is now being found after the container update/rebuild but only for v20.x-staging and v22.x-staging.

[richardlau]

Okay, it looks like this is due to nodejs/build#4231. We do not have strace installed in the container, but we do have gcc-toolset-10-strace installed:

bash-4.4$ dnf list installed *strace
Not root, Subscription Management repositories not updated
Installed Packages
gcc-toolset-10-strace.x86_64                                                                     5.7-2.el8                                                                      @rhel-8-for-x86_64-appstream-rpms
bash-4.4$

For UBI/RHEL 8 we use

Node.js version	compiler
20	gcc-toolset-10
22	gcc-toolset-10
24	gcc-toolset-12
25	clang (19)
main	clang (19)

It looks like strace is present in gcc-toolset-10, but not in gcc-toolset-12, hence why strace is only being found for v20.x/v22.x.

# dnf search *strace
Updating Subscription Management repositories.
Last metadata expiration check: 0:50:24 ago on Tue 24 Feb 2026 08:20:49 AM CST.
======================================================================================== Name & Summary Matched: *strace ========================================================================================
csmock-plugin-strace.noarch : csmock plug-in providing the support for strace
============================================================================================= Name Matched: *strace =============================================================================================
gcc-toolset-10-strace.x86_64 : Tracks and displays system calls associated with a running process
gcc-toolset-11-strace.x86_64 : Tracks and displays system calls associated with a running process
gcc-toolset-9-strace.x86_64 : Tracks and displays system calls associated with a running process
strace.x86_64 : Tracks and displays system calls associated with a running process
#

The difference pre-nodejs/build#4231 is the way we were installing gcc-toolset-10 -- before we were pulling individual RPMs (but not including the strace one) from CentOS Stream 8 as the UBI dnf repository doesn't have gcc-toolset-10 anymore. Now we register the container with our Red Hat subscription and install gcc-toolset-10 from the RHEL 8 dnf repository which is including the strace RPM.

Going back to the test itself, it is looking for strace calls to open /etc/ssl/openssl.cnf:

node/test/parallel/test-strace-openat-openssl.js

Lines 19 to 21 in 7599a8b

	const allowedOpenCalls = new Set([
	'/etc/ssl/openssl.cnf',
	]);

We already note in nix config that the path might not match in that environment:

node/shell.nix

Lines 115 to 118 in 9145cc6

	++ pkgs.lib.optionals useSharedOpenSSL [
	# Path to the openssl.cnf is different from the expected one
	"test-strace-openat-openssl"
	]

So I'm thinking options are:

• Add alternative locations to allowedOpenCalls.
• Skip the test for shared OpenSSL.

[richardlau]

FWIW if I build Node.js on UBI 8 against the system OpenSSL, this is the strace output that the test ends up running:

# strace -f -ff -e trace=open,openat -s 512 -D out/Release/node -e 'require("crypto")'
strace: Process 159371 attached
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib64/libcrypto.so.1.1", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib64/libssl.so.1.1", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib64/libatomic.so.1", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib64/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib64/libm.so.6", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib64/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib64/libgcc_s.so.1", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib64/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib64/libz.so.1", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/proc/sys/crypto/fips_enabled", O_RDONLY) = 3
openat(AT_FDCWD, "/proc/version_signature", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
strace: Process 159376 attached
strace: Process 159377 attached
strace: Process 159378 attached
strace: Process 159379 attached
strace: Process 159380 attached
[pid 159371] openat(AT_FDCWD, "/proc/self/maps", O_RDONLY|O_CLOEXEC) = 17
[pid 159371] openat(AT_FDCWD, "/proc/self/cgroup", O_RDONLY|O_CLOEXEC) = 17
[pid 159371] openat(AT_FDCWD, "/sys/fs/cgroup//memory.max", O_RDONLY|O_CLOEXEC) = 17
[pid 159371] openat(AT_FDCWD, "/sys/fs/cgroup//memory.high", O_RDONLY|O_CLOEXEC) = 17
[pid 159371] openat(AT_FDCWD, "/proc/meminfo", O_RDONLY|O_CLOEXEC) = 17
[pid 159371] openat(AT_FDCWD, "/proc/self/maps", O_RDONLY) = 17
[pid 159371] openat(AT_FDCWD, "/tmp/node/out/Release/node", O_RDONLY) = 17
strace: Process 159381 attached
[pid 159371] openat(AT_FDCWD, "/etc/pki/tls/openssl.cnf", O_RDONLY) = 17
[pid 159371] openat(AT_FDCWD, "/etc/crypto-policies/back-ends/opensslcnf.config", O_RDONLY) = 18
[pid 159380] +++ exited with 0 +++
[pid 159379] +++ exited with 0 +++
[pid 159377] +++ exited with 0 +++
[pid 159378] +++ exited with 0 +++
[pid 159376] +++ exited with 0 +++
[pid 159381] +++ exited with 0 +++
+++ exited with 0 +++
#

This shows that on UBI 8 the system OpenSSL opens /etc/pki/tls/openssl.cnf instead of /etc/ssl/openssl.cnf but also that the strace output line with the openat call is prepended with [pid ...] which the test won't match since it uses a simple startswith():

node/test/parallel/test-strace-openat-openssl.js

Line 32 in 7599a8b

if (!line.startsWith('open')) {

I'll do some further investigation/experiments, but reading the PR that added the test (#46150), it references nodejs/security-wg#827 and I'm not sure we can guarantee the list of files that end up being opened if we are linking to external dependencies (such as shared OpenSSL), so it might actually make more sense to skip this test when process.config.variables.node_shared_openssl is true. We can see in the example above that UBI/RHEL system OpenSSL also attempts to open /etc/crypto-policies/back-ends/opensslcnf.config.

[richardlau]

This shows that on UBI 8 the system OpenSSL opens /etc/pki/tls/openssl.cnf instead of /etc/ssl/openssl.cnf

FWIW we can determine this by running (if it exists) the OpenSSL CLI to get the value of OPENSSLDIR, e.g.

# openssl version -d
OPENSSLDIR: "/etc/pki/tls"
#

For the non-shared OpenSSL builds of Node.js (e.g. the ones that statically link deps/openssl):

$ ./out/Release/openssl-cli version -d
OPENSSLDIR: "/etc/ssl"
$