Run RISC-V tests with multiple RVV QEMU configurations

[luhenry]

Summary

This tackles Phase 3 of #18991, in continuation of #19399, #19521, #19617.

Test plan

Exclusively adding CI testing, no new feature. CI will cover the testing.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/19707

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures, 1 Unrelated Failure, 9 Unclassified Failures

As of commit e17eca9 with merge base 6ba868e ():

NEW FAILURES - The following jobs have failed:

• pull / android / run-emulator (gh)
  The process '/usr/bin/sh' failed with exit code 1
• trunk / unittest-release / macos / macos-job (gh)
  backends/xnnpack/test/recipes/test_xnnpack_recipes.py::TestXnnpackRecipes::test_all_models_with_recipes

UNCLASSIFIED FAILURES - DrCI could not classify the following jobs because the workflow did not run on the merge base. The failures may be pre-existing on trunk or introduced by this PR:

• Test RISC-V Backend / test-riscv (add, false, false) / run / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command docker exec -t 50b26f20a3995d32fa432f8edfaee5eba0012a5fb43b6b6ee3d5bdb5f93fda28 /exec failed with exit code 2
• Test RISC-V Backend / test-riscv (llama2, false, false) / run / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command docker exec -t eef6629448703a11d497ae61fb86353e94e7f34f3ac0196a31fed1dfb0c0479c /exec failed with exit code 2
• Test RISC-V Backend / test-riscv (mobilebert, false, false) / run / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command docker exec -t d8855eb1bc0174580681b1c446481bae652085d6181edff0ea58939b81aefcf3 /exec failed with exit code 2
• Test RISC-V Backend / test-riscv (mv2, false, false) / run / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command docker exec -t 79e3e12f05cc1493bac7ddef4721b4bcb35ab4f3134270ee3c9a89e6da01c0ab /exec failed with exit code 2
• Test RISC-V Backend / test-riscv (resnet18, false, false) / run / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command docker exec -t 9288430e6303772958b7cbeba7ebebd8227ed8f5a5ed82311c3340037566d690 /exec failed with exit code 2
• trunk / test-arm-backend-ethos-u (test_deit_e2e_ethos_u) / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command docker exec -t 345e379c1b4e88b2944bac76ef771201d3549bcfb25d187611c142e4db49b4fc /exec failed with exit code 1
• trunk / test-arm-backend-ethos-u (test_ootb_tests_ethos_u) / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command docker exec -t 678edafb6f5faa2bcadea0303caba6b505547f7c1c13281544549e8080eab8df /exec failed with exit code 1
• trunk / test-arm-backend-ethos-u (test_ootb_tests_tosa) / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command docker exec -t 0d9694de676cee3a46aabf1a3f8ac27308a31649b59dcbcb204093e669172ec8 /exec failed with exit code 1
• trunk / test-arm-backend-vkml (test_ootb_tests_vgf) / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command docker exec -t 61353d7ff5f14cc97ffdd518a85eb66915e62edd5f1ce0584ba55093ffe18b2f /exec failed with exit code 1

FLAKY - The following job failed but was likely due to flakiness present on trunk:

• pull / unittest / macos / macos-job (gh) (detected as infra flaky with no log or failing log classifier)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

[luhenry]

cc @rascani @mergennachin, and @GregoryComer @digantdesai for XNNPACK

[digantdesai]

Thanks!

[digantdesai]

@claude review this PR

[claude]

Claude finished @digantdesai's task in 1m 26s —— View job

---

PR Review: Run RISC-V tests with multiple RVV QEMU configurations

• Read changed files and understand the PR
• Review CI workflow changes
• Review test scripts
• Review CMake changes
• Review etdump_summary.py
• Post review feedback

---

Overall this is a solid PR that adds multi-VLEN QEMU coverage for RVV and integrates XNNPACK microkernel profiling into the CI workflow. A few observations:

Workflow Logic (_test_riscv.yml)

Potential issue: while read in a pipeline runs in a subshell. At line 59, the loop echo '...' | jq ... | while IFS= read -r qemu_cpu; do ... done runs in a subshell. If any iteration fails, the exit code from inside the loop won't propagate to the outer script due to the pipe. Since set -e (from the calling script) doesn't reach into subshells behind pipes, a QEMU failure mid-loop would be silently swallowed for intermediate iterations — only the last iteration's status would matter.

Consider using process substitution or a for loop with mapfile:

mapfile -t cpus < <(echo '${{ inputs.qemu-cpu }}' | jq -r '.[]')
for qemu_cpu in "${cpus[@]}"; do
  ...
done

Minor: $GITHUB_STEP_SUMMARY block assumes the etdump JSON exists. If --verbose-xnnpack isn't passed (non-XNNPACK path), the etdump JSON won't be generated, but the summary generation block still runs. The jq commands will fail. However, since --verbose-xnnpack is always passed when xnnpack=true (line 64), and the non-XNNPACK path doesn't iterate with RVV configs where this matters — this is likely fine in practice for the current matrix. But if someone later adds an XNNPACK test without --verbose-xnnpack, the summary block would error.

examples/riscv/run.sh

Good change: Replacing the mktemp + trap cleanup with a deterministic path (${output_dir}/${model}_riscv.run.log) is the right call — it makes the log available for downstream processing by etdump_summary.py.

The || true on etdump_summary.py (line 152) is reasonable for CI robustness, but means summary generation failures will be silent. Worth noting.

tools/cmake/preset/riscv64_linux.cmake

The EXECUTORCH_BUILD_RISCV_ETDUMP option is well-structured — gating event tracing behind a flag keeps default builds lean. The FLATCC_ALLOW_WERROR OFF is a pragmatic workaround for cross-compilation warnings.

One subtlety: when EXECUTORCH_BUILD_RISCV_ETDUMP is OFF, you explicitly set(EXECUTORCH_ENABLE_EVENT_TRACER OFF). If a user previously set EXECUTORCH_ENABLE_EVENT_TRACER ON via command line, this would override it. Consider using set_overridable_option for symmetry, or documenting that this preset enforces no event tracing unless the etdump flag is set.

backends/xnnpack/CMakeLists.txt

Clean addition. Applying XNN_LOG_LEVEL to xnnpack-logging as PUBLIC makes sense since consumers of this target need the define propagated.

examples/riscv/etdump_summary.py

Well-structured profiling tool. The regex-based kernel matching (_KIND_PATTERN) is thorough. A couple of notes:

• The _DTYPE_TOKENS matching (line 122: all(d in sym for d in dtype_tokens)) does substring matching — e.g., "f32" would match "xnn_f32_..." but also "xnn_bf32_...". This is probably fine given the symbol naming conventions in XNNPACK, but worth being aware of.
• ev.perf_data.raw values are used directly — are these guaranteed to be in milliseconds? The Inspector API docs should confirm the unit. If they're in a different unit (e.g., nanoseconds), the summary output would be misleading.

riscv64.yml — case() Expression

The case() GitHub Actions expression (lines 56-65) is a relatively new feature. This looks correct, but worth verifying it's supported on the pytorch/test-infra runners (it requires Actions runtime ≥ 2.314).

CI Failures

The 5 "UNCLASSIFIED FAILURES" all show docker exec ... /exec failed with exit code 1 for the non-XNNPACK cases (xnnpack: false). These are the jobs using qemu-cpu: '["rv64,zba=true,zbb=true,zbs=true,v=false"]'. Worth investigating whether the v=false QEMU CPU flag is causing issues, or if this is a pre-existing infrastructure problem.

---

Summary: The PR is well-organized and the approach is sound. The main actionable concern is the subshell/pipe behavior in the workflow loop that could mask mid-loop failures. The rest are minor observations.

[digantdesai]

Nit: perhaps rename the verbose_xnnpack flag to debug or something else to indicate impact beyond xnnpack build

[luhenry]

Renamed to debug_xnnpack and passing --debug-xnnpack. It's still XNNPACK related but more debug than verbose.

[digantdesai]

better way to do this? more maintainable way I guess? pase the type_to_string enum or something?

[luhenry]

Open to suggestions, I'm not the most familiar with XNNPACK and how they name kernels.

pase the type_to_string enum or something?

I'll have a look at what's doable with that.

[digantdesai]

Parsing the source file from this py file seems a good starting point. I don't want this to break when we update xnnpack submodule.