Screenshot tests sometimes fail and need to be rerun

[riccardobl]

Screenshot tests often fail in PRs, and someone needs to manually rerun them from the github action ui.

[coderabbitai]

Coding Plan

Summary

• Add workflow-level retry using the nick-fields/retry GitHub Action to automatically re-run screenshot tests on failure
• Configure 2-3 retry attempts with a short delay between attempts to allow transient OpenGL/GPU state issues to resolve
• This non-invasive approach addresses the manual rerun problem without modifying the test framework code
• Existing artifact upload and reporting mechanisms should continue to work without modification

Design Choices

Design Choice 1: What retry approach should be used for the screenshot tests?

Options Considered:

1. Third-party retry action (nick-fields/retry) - Popular, well-tested action specifically designed for retrying flaky steps with configurable attempts, delays, and timeout handling.
2. Native shell script retry loop - Implement retry logic directly in bash within the run step, no external dependencies.
3. Gradle test retry plugin - Add retry at the Gradle level using org.gradle.test-retry plugin to retry individual failing tests.

Chosen Option: 1

Rationale: This is the most maintainable solution for workflow-level flakiness. It's widely adopted in the GitHub Actions ecosystem, handles exit codes properly, provides clear logging of retry attempts, and requires minimal configuration. Shell script retries add complexity, and Gradle-level retries would require changes to the test framework's reporting integration.

💡 User Tips

Regenerate the plan with different choices with @coderabbitai <feedback>.

Implementation Steps

Add Automatic Retry to Screenshot Test Workflow

This phase modifies the GitHub Actions workflow to automatically retry the screenshot test step when it fails, reducing the need for manual intervention while preserving test result artifacts.

Task 1: Add Retry Logic to Screenshot Test Step

Modify the screenshot test execution step in the CI workflow to use automatic retry with appropriate configuration.

• Update .github/workflows/main.yml to wrap the screenshot test execution step with the nick-fields/retry action
• Configure retry parameters: 2-3 maximum attempts, with a delay between retries (e.g., 10-30 seconds) to allow any transient GPU/OpenGL state to settle
• Preserve the existing Xvfb display setup and environment variables
• Ensure the retry action uses the same working directory and environment context as the current step

Task 2: Ensure Artifact Upload Captures All Retry Attempts

Verify that test reports and changed images are properly captured regardless of which retry attempt succeeds or fails.

• Review the existing "Upload Test Reports" step in .github/workflows/main.yml to ensure it runs on all outcomes (already uses if: always())
• Confirm artifact paths capture reports from the final test execution state
• No changes expected to be needed if retry action properly overwrites previous run artifacts, but validate this behavior

🤖 Prompt for AI agents

This phase adds automatic retry logic to the screenshot test execution in the
GitHub Actions workflow to handle transient failures without manual
intervention.

**Modify `.github/workflows/main.yml`:**

- Locate the screenshot test execution step in the `ScreenshotTests` job
- Wrap this step with the `nick-fields/retry` action
- Configure retry parameters:
  - Set maximum attempts to 2-3
- Add a delay between retries (10-30 seconds) to allow GPU/OpenGL state to
settle
- Preserve all existing configuration:
  - Xvfb display setup
  - Environment variables
  - Working directory
  - Environment context

**Verify artifact handling:**

- Review the "Upload Test Reports" step to confirm it runs on all outcomes
(should already have `if: always()`)
- Verify artifact paths capture reports from the final test execution state
- Confirm the retry action properly overwrites artifacts from previous attempts

Research

The jme3-screenshot-tests module uses JUnit 5 with a custom test framework that captures screenshots at specific frames and compares them pixel-by-pixel against reference images. Tests run in a controlled Docker environment with Xvfb for virtual display rendering. The CI pipeline (.github/workflows/main.yml) executes these tests via a dedicated ScreenshotTests job but currently has no retry mechanism—tests run once and either pass or fail. The existing framework already includes stability patterns like a 1-second OpenGL cleanup delay between tests and fixed 60 FPS timing, but transient failures still occur.

---

🚀 Next Steps

🤖 All AI agent prompts combined

This phase adds automatic retry logic to the screenshot test execution in the
GitHub Actions workflow to handle transient failures without manual
intervention.

**Modify `.github/workflows/main.yml`:**

- Locate the screenshot test execution step in the `ScreenshotTests` job
- Wrap this step with the `nick-fields/retry` action
- Configure retry parameters:
  - Set maximum attempts to 2-3
- Add a delay between retries (10-30 seconds) to allow GPU/OpenGL state to
settle
- Preserve all existing configuration:
  - Xvfb display setup
  - Environment variables
  - Working directory
  - Environment context

**Verify artifact handling:**

- Review the "Upload Test Reports" step to confirm it runs on all outcomes
(should already have `if: always()`)
- Verify artifact paths capture reports from the final test execution state
- Confirm the retry action properly overwrites artifacts from previous attempts

💡 Iterate on the plan with: @coderabbitai <feedback>

Example Feedback
- `@coderabbitai` You can skip phase 3. Add a simple unit test case for phase 2.
- `@coderabbitai` For design choice 1 go ahead with option 3 and replan.

---

💬 Have feedback or questions? Drop into our discord!

[zzuegg]

my guess is that this is due to small timing differences, as far i know it is not possible to run jme with fixed timesteps. like always 16.6 ms independently of the actual rendering speed