kirk returns 0 after test suite interruption

[asmorodskyi]

kirk returns 0 when test suite execution interrupted due to timeout .

Expected result : kirk needs to return non-zero exit code in case execution interruption on timeout

[pevik]

@acerv I was thinking if these TestScheduler class members were also kirk exit codes:

kirk/libkirk/scheduler.py

Lines 99 to 103 in 4e20e25

	STATUS_OK = 0
	TEST_TIMEOUT = 1
	KERNEL_PANIC = 2
	KERNEL_TAINTED = 3
	KERNEL_TIMEOUT = 4

ATM kirk exit 0 if managed to run testsuites. But that would not work because kirk uses 2 for argparse failures:

# ./kirk; echo $?
usage: kirk [-h] [--version] [--verbose] [--no-colors] [--tmp-dir TMP_DIR] [--restore RESTORE] [--json-report JSON_REPORT] [--monitor MONITOR] [--plugins PLUGINS] [--com COM] [--sut SUT]
            [--framework FRAMEWORK] [--env ENV] [--skip-tests SKIP_TESTS] [--skip-file SKIP_FILE] [--run-suite [RUN_SUITE ...]] [--run-pattern RUN_PATTERN] [--run-command RUN_COMMAND]
            [--suite-timeout SUITE_TIMEOUT] [--exec-timeout EXEC_TIMEOUT] [--randomize] [--runtime RUNTIME] [--suite-iterate SUITE_ITERATE] [--workers WORKERS] [--force-parallel]
            [--fault-injection FAULT_INJECTION]
kirk: error: --run-suite/--run-command are required
2

But at least exit 1 when testsuite TBROK/TFAIL would be great.

[pevik]

# ./kirk --suite-timeout=1 -f ima; echo $?
...
Connecting to SUT: default

Starting suite: ima
--------------------
ima_measurements: pass  (0.715s)
ima_policy: Suite 'ima' timed out after 1 seconds
                                                                                                                                
Execution time: 1.004s

	Suite:       ima
	Total runs:  9
	Runtime:     0.715s
	Passed:      3
	Failed:      0
	Skipped:     8
	Broken:      0
	Warnings:    0
	Kernel:      Linux 6.17.0-rc4-3.g8859e21-default #1 SMP PREEMPT_DYNAMIC Thu Sep  4 01:43:08 UTC 2025 (8859e21)
	Machine:     x86_64
	Arch:        x86_64
	RAM:         2015968 kB
	Swap:        2098152 kB
	Distro:      opensuse-tumbleweed 20250204

Disconnecting from SUT: default
Session stopped
0

=> exit 1?

FYI normal run:

 # ./kirk -f ima; echo $?
...
Connecting to SUT: default

Starting suite: ima
--------------------
ima_measurements: pass  (0.801s)
ima_policy: skip  (0.376s)
ima_tpm: skip  (0.496s)
ima_violations: pass  (3.623s)
ima_keys: skip  (0.443s)
ima_kexec: skip  (0.452s)
ima_selinux: skip  (0.628s)
ima_conditionals: skip  (0.575s)
evm_overlay: skip  (0.386s)
                                                                                                                                
Execution time: 8.182s

	Suite:       ima
	Total runs:  9
	Runtime:     7.781s
	Passed:      5
....
Disconnecting from SUT: default
Session stopped
0

[acerv]

@acerv I was thinking if these TestScheduler class members were also kirk exit codes:

kirk/libkirk/scheduler.py

Lines 99 to 103 in 4e20e25

STATUS_OK = 0
TEST_TIMEOUT = 1
KERNEL_PANIC = 2
KERNEL_TAINTED = 3
KERNEL_TIMEOUT = 4
ATM kirk exit 0 if managed to run testsuites. But that would not work because kirk uses 2 for argparse failures:

Those are the return code returned by the tests.

# ./kirk; echo $?
usage: kirk [-h] [--version] [--verbose] [--no-colors] [--tmp-dir TMP_DIR] [--restore RESTORE] [--json-report JSON_REPORT] [--monitor MONITOR] [--plugins PLUGINS] [--com COM] [--sut SUT]
            [--framework FRAMEWORK] [--env ENV] [--skip-tests SKIP_TESTS] [--skip-file SKIP_FILE] [--run-suite [RUN_SUITE ...]] [--run-pattern RUN_PATTERN] [--run-command RUN_COMMAND]
            [--suite-timeout SUITE_TIMEOUT] [--exec-timeout EXEC_TIMEOUT] [--randomize] [--runtime RUNTIME] [--suite-iterate SUITE_ITERATE] [--workers WORKERS] [--force-parallel]
            [--fault-injection FAULT_INJECTION]
kirk: error: --run-suite/--run-command are required
2

But at least exit 1 when testsuite TBROK/TFAIL would be great.

It's already like this:

kirk/libkirk/main.py

Line 37 in 4e20e25

RC_OK = 0

The current kirk behavior is correct by design (and there are also tests for it which are passing). When tests are interrupted, kirk return 1. But nothing happens when they are timing out.

We just need to add one more exit code when tests are timing out, but this is gonna be a new feature.

[acerv]

I inroduced the following patch to fix final report: a440593