e2e tests on Cirrus Labs runners #5485

alwx · 2025-12-18T10:38:02Z

📢 Type of change

Bugfix
New feature
Enhancement
Refactoring

📜 Description

Enabled Cirrus Labs runners for e2e tests. Fixes #5418

⚠️ This one is not to be merged immediately — I simply want to run all the tests first to validate how it improves with Cirrus Labs runners.

💡 Motivation and Context

💚 How did you test it?

📝 Checklist

I added tests to verify changes
No new PII added or SDK only sends newly added PII if sendDefaultPII is enabled
I updated the docs if needed.
I updated the wizard if needed.
All tests passing
No breaking changes

🔮 Next steps

.github/workflows/e2e-v2.yml

        platform: ["ios", "android"]
        include:
          - platform: ios
-            runs-on: macos-26
+            runs-on: ["ghcr.io/cirruslabs/macos-tahoe-xcode:26.2", "runner_group_id:12"]
            name: iOS
            appPlain: performance-tests/test-app-plain.ipa
          - platform: android
-            # Not using the latest version due to a known issue: https://github.com/getsentry/sentry-react-native/issues/4418
-            runs-on: ubuntu-22.04
+            runs-on: ["ghcr.io/cirruslabs/ubuntu-runner-amd64:22.04", "runner_group_id:12"]
            name: Android
            appPlain: performance-tests/TestAppPlain/android/app/build/outputs/apk/release/app-release.apk
    steps:


.github/workflows/e2e-v2.yml

        platform: ["ios", "android"]
        include:
          - platform: ios
-            runs-on: macos-26
+            runs-on: ["ghcr.io/cirruslabs/macos-tahoe-xcode:26.2", "runner_group_id:12"]
            name: iOS
            appPlain: performance-tests/test-app-plain.ipa
          - platform: android
-            # Not using the latest version due to a known issue: https://github.com/getsentry/sentry-react-native/issues/4418
-            runs-on: ubuntu-22.04
+            runs-on: ["ghcr.io/cirruslabs/ubuntu-runner-amd64:22.04", "runner_group_id:12"]
            name: Android
            appPlain: performance-tests/TestAppPlain/android/app/build/outputs/apk/release/app-release.apk
    steps:


github-actions · 2026-01-26T14:18:26Z

Semver Impact of This PR

⚪ None (no version bump detected)

📋 Changelog Preview

This is how your changes will appear in the changelog.
Entries from this PR are highlighted with a left border (blockquote style).

e2e tests on Cirrus Labs runners by alwx in #5485

chore(deps): update JavaScript SDK to v10.38.0 by github-actions in #5596

_{🤖 This preview updates automatically when you update the PR.}

github-actions · 2026-01-28T11:02:23Z

Android (legacy) Performance metrics 🚀

	Plain	With Sentry	Diff
Startup time	411.09 ms	487.13 ms	76.04 ms
Size	43.75 MiB	48.41 MiB	4.65 MiB

Baseline results on branch: main

Startup times

Revision	Plain	With Sentry	Diff
`f3b058c`+dirty	501.18 ms	536.70 ms	35.52 ms
`bfe454a`+dirty	573.44 ms	579.46 ms	6.02 ms
`294387d`+dirty	424.30 ms	465.40 ms	41.10 ms
`955f2eb`+dirty	422.74 ms	410.19 ms	-12.55 ms
`3bd3f0d`+dirty	447.21 ms	472.31 ms	25.10 ms
`d081295`+dirty	408.08 ms	453.62 ms	45.54 ms
`ad27f6e`+dirty	471.44 ms	516.23 ms	44.79 ms
`07808fb`+dirty	419.10 ms	419.08 ms	-0.02 ms
`170d5ea`+dirty	407.92 ms	422.49 ms	14.57 ms
`8e653ac`+dirty	360.28 ms	372.04 ms	11.76 ms

App size

Revision	Plain	With Sentry	Diff
`f3b058c`+dirty	43.75 MiB	48.07 MiB	4.32 MiB
`bfe454a`+dirty	17.75 MiB	19.69 MiB	1.94 MiB
`294387d`+dirty	43.75 MiB	48.04 MiB	4.29 MiB
`955f2eb`+dirty	17.75 MiB	19.70 MiB	1.95 MiB
`3bd3f0d`+dirty	17.75 MiB	19.70 MiB	1.95 MiB
`d081295`+dirty	43.75 MiB	48.04 MiB	4.29 MiB
`ad27f6e`+dirty	43.75 MiB	48.07 MiB	4.32 MiB
`07808fb`+dirty	17.75 MiB	19.70 MiB	1.95 MiB
`170d5ea`+dirty	17.75 MiB	19.70 MiB	1.95 MiB
`8e653ac`+dirty	17.75 MiB	19.75 MiB	2.00 MiB

github-actions · 2026-01-28T11:02:34Z

Android (new) Performance metrics 🚀

	Plain	With Sentry	Diff
Startup time	506.17 ms	542.22 ms	36.05 ms
Size	43.94 MiB	49.26 MiB	5.33 MiB

Baseline results on branch: main

Startup times

Revision	Plain	With Sentry	Diff
`f3b058c`+dirty	422.90 ms	468.30 ms	45.40 ms
`294387d`+dirty	359.44 ms	393.40 ms	33.97 ms
`1d62dde`+dirty	366.59 ms	408.80 ms	42.21 ms
`d081295`+dirty	416.95 ms	461.24 ms	44.29 ms
`ad27f6e`+dirty	484.67 ms	532.79 ms	48.12 ms
`1226664`+dirty	377.65 ms	453.94 ms	76.29 ms
`083f560`+dirty	383.96 ms	417.76 ms	33.80 ms
`ec14be7`+dirty	401.58 ms	475.26 ms	73.68 ms
`eb07ba3`+dirty	419.49 ms	482.12 ms	62.63 ms
`d1fd647`+dirty	374.46 ms	409.51 ms	35.05 ms

App size

Revision	Plain	With Sentry	Diff
`f3b058c`+dirty	43.94 MiB	48.90 MiB	4.96 MiB
`294387d`+dirty	43.94 MiB	48.87 MiB	4.93 MiB
`1d62dde`+dirty	7.15 MiB	8.46 MiB	1.31 MiB
`d081295`+dirty	43.94 MiB	48.87 MiB	4.93 MiB
`ad27f6e`+dirty	43.94 MiB	48.90 MiB	4.96 MiB
`1226664`+dirty	7.15 MiB	8.46 MiB	1.30 MiB
`083f560`+dirty	7.15 MiB	8.43 MiB	1.28 MiB
`ec14be7`+dirty	7.15 MiB	8.42 MiB	1.26 MiB
`eb07ba3`+dirty	7.15 MiB	8.42 MiB	1.27 MiB
`d1fd647`+dirty	7.15 MiB	8.43 MiB	1.28 MiB

.github/workflows/e2e-v2.yml

github-actions · 2026-01-29T09:24:54Z

iOS (new) Performance metrics 🚀

	Plain	With Sentry	Diff
Startup time	1220.23 ms	1218.40 ms	-1.83 ms
Size	3.38 MiB	4.60 MiB	1.22 MiB

Baseline results on branch: main

Startup times

Revision	Plain	With Sentry	Diff
`6bd9054`+dirty	1207.02 ms	1199.27 ms	-7.76 ms
`d73150f`+dirty	1224.52 ms	1227.17 ms	2.65 ms
`d081295`+dirty	1205.24 ms	1207.52 ms	2.28 ms
`d1fd647`+dirty	1218.16 ms	1225.82 ms	7.65 ms
`ea3e26e`+dirty	1216.61 ms	1214.15 ms	-2.47 ms
`80e4616`+dirty	1206.90 ms	1205.94 ms	-0.96 ms
`955f2eb`+dirty	1225.78 ms	1239.27 ms	13.49 ms
`5ee3314`+dirty	1234.25 ms	1235.44 ms	1.19 ms
`70250df`+dirty	1211.96 ms	1222.31 ms	10.35 ms
`664c66f`+dirty	1195.94 ms	1194.80 ms	-1.14 ms

App size

Revision	Plain	With Sentry	Diff
`6bd9054`+dirty	3.41 MiB	4.67 MiB	1.25 MiB
`d73150f`+dirty	3.38 MiB	4.60 MiB	1.22 MiB
`d081295`+dirty	3.41 MiB	4.59 MiB	1.18 MiB
`d1fd647`+dirty	3.19 MiB	4.56 MiB	1.37 MiB
`ea3e26e`+dirty	3.41 MiB	4.58 MiB	1.17 MiB
`80e4616`+dirty	3.38 MiB	4.60 MiB	1.22 MiB
`955f2eb`+dirty	3.19 MiB	4.55 MiB	1.36 MiB
`5ee3314`+dirty	3.19 MiB	4.55 MiB	1.37 MiB
`70250df`+dirty	3.44 MiB	4.59 MiB	1.15 MiB
`664c66f`+dirty	3.38 MiB	4.60 MiB	1.22 MiB

github-actions · 2026-01-29T09:24:54Z

iOS (legacy) Performance metrics 🚀

	Plain	With Sentry	Diff
Startup time	1221.87 ms	1220.30 ms	-1.57 ms
Size	3.38 MiB	4.60 MiB	1.22 MiB

Baseline results on branch: main

Startup times

Revision	Plain	With Sentry	Diff
`6bd9054`+dirty	1212.20 ms	1217.89 ms	5.70 ms
`d73150f`+dirty	1198.44 ms	1210.06 ms	11.62 ms
`d081295`+dirty	1214.40 ms	1211.27 ms	-3.13 ms
`d1fd647`+dirty	1219.35 ms	1233.18 ms	13.83 ms
`ea3e26e`+dirty	1229.13 ms	1228.46 ms	-0.67 ms
`80e4616`+dirty	1221.32 ms	1225.64 ms	4.32 ms
`955f2eb`+dirty	1235.06 ms	1253.88 ms	18.81 ms
`5ee3314`+dirty	1215.18 ms	1207.64 ms	-7.54 ms
`70250df`+dirty	1214.51 ms	1215.04 ms	0.53 ms
`664c66f`+dirty	1215.37 ms	1221.30 ms	5.92 ms

App size

Revision	Plain	With Sentry	Diff
`6bd9054`+dirty	3.41 MiB	4.67 MiB	1.25 MiB
`d73150f`+dirty	3.38 MiB	4.60 MiB	1.22 MiB
`d081295`+dirty	3.41 MiB	4.59 MiB	1.18 MiB
`d1fd647`+dirty	2.63 MiB	3.99 MiB	1.36 MiB
`ea3e26e`+dirty	3.41 MiB	4.58 MiB	1.17 MiB
`80e4616`+dirty	3.38 MiB	4.60 MiB	1.22 MiB
`955f2eb`+dirty	2.63 MiB	3.98 MiB	1.35 MiB
`5ee3314`+dirty	2.63 MiB	3.99 MiB	1.35 MiB
`70250df`+dirty	3.44 MiB	4.59 MiB	1.15 MiB
`664c66f`+dirty	3.38 MiB	4.60 MiB	1.22 MiB

.github/workflows/e2e-v2.yml

alwx · 2026-01-29T11:12:17Z

.github/workflows/e2e-v2.yml

+          echo "=== Check for conflicting idb installations ==="
+          find /Users/admin -name "idb_companion" -o -name "idb-companion" 2>/dev/null | head -10 || true
+          ls -la /opt/homebrew/bin/idb* 2>/dev/null || true
+


I think it's useful to have it here just to help us debugging potential problems that might arise (it helped me a lot)

lucas-zimerman · 2026-01-29T14:54:58Z

⚠️ This one is not to be merged immediately — I simply want to run all the tests first to validate how it improves with Cirrus Labs runners.

is it ready for review?

alwx · 2026-01-30T08:31:03Z

@lucas-zimerman there is still a (hopefully) minor issue with some tests, figuring it out now

antonis · 2026-01-30T08:35:44Z

there is still a (hopefully) minor issue with some tests, figuring it out now

@alwx heads up that the captureReplay e2e tests on iOS seem to be failing on main too since Jan 28th. I think this is flakiness since I don't see any change that may have caused this but I'll check and iterate back

.github/workflows/e2e-v2.yml

alwx · 2026-01-30T09:43:31Z

@antonis true that, I see that on main branch as well

.github/workflows/e2e-v2.yml

sentry · 2026-01-30T12:31:09Z

.github/workflows/e2e-v2.yml

+            bundle install
+            bundle exec pod install


Bug: The bundle install command in the iOS build step will fail because it is run from the ios subdirectory, which does not contain a Gemfile.
_{Severity: HIGH}

Suggested Fix

Move the bundle install command so it executes in the correct directory (./performance-tests/TestAppPlain) before the script changes directory into the ios subdirectory. This will ensure dependencies are installed before pod install is called.

Prompt for AI Agent

Review the code at the location below. A potential bug has been identified by an AI agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's not valid. Location: .github/workflows/e2e-v2.yml#L125-L126 Potential issue: In the `e2e-v2.yml` workflow, the `Build app plain` and `Build app with Sentry` jobs for the iOS platform will fail. The script changes the directory to `ios` before running `bundle install`. However, the `Gemfile` is located in the parent directory (`./performance-tests/TestAppPlain`), not in the `ios` subdirectory. Since `bundle install` does not search parent directories for a `Gemfile` by default, the command will fail with a "Gemfile not found" error, which in turn causes the `pod install` command to fail and breaks the entire iOS build process within the CI pipeline.

antonis

I told Claude to summarize the performance difference between main and this PR and the improvement is massive 🚀

Job	Old/Main	New/PR	Difference
auth_token_check / auth_token_check	2s	3s	+1s
Ready-to-merge gate / Missing "ready-to-merge" label	2s	4s	+2s
diff_check / diff_check	4s	10s	+6s
metrics (legacy, android)	13m 26s	13m 15s	-11s
metrics (legacy, ios)	23m 33s	14m 36s	-8m 57s
metrics (new, android)	18m 25s	14m 38s	-3m 47s
metrics (new, ios)	24m 0s	20m 9s	-3m 51s
Build RN 0.71.19 legacy hermes android production no	9m 57s	5m 15s	-4m 42s
Build RN 0.71.19 legacy hermes ios production no	16m 50s	6m 44s	-10m 6s
Build RN 0.71.19 legacy jsc android production no	8m 30s	4m 29s	-4m 1s
Build RN 0.71.19 legacy jsc ios production no	20m 12s	6m 47s	-13m 25s
Build RN 0.83.0 legacy hermes android production no	9m 2s	5m 39s	-3m 23s
Build RN 0.83.0 legacy hermes ios production no	35m 58s	8m 57s	-27m 1s
Build RN 0.83.0 legacy hermes ios production dynamic	28m 56s	8m 13s	-20m 43s
Build RN 0.83.0 legacy hermes ios production static	32m 36s	8m 27s	-24m 9s
Build RN 0.83.0 new hermes android production no	9m 24s	4m 59s	-4m 25s
Build RN 0.83.0 new hermes ios production no	26m 41s	7m 55s	-18m 46s
Build RN 0.83.0 new hermes ios production static	23m 34s	8m 11s	-15m 23s
Test RN 0.83.0 legacy hermes android production no	6m 31s	6m 6s	-25s
Test RN 0.83.0 legacy hermes ios production no	18m 32s	4m 12s	-14m 20s
Test RN 0.83.0 new hermes android production no	7m 14s	6m 39s	-35s
Test RN 0.83.0 new hermes ios production no	16m 8s	4m 37s	-11m 31s

However the iOS E2E tests seem to be fragile and the app is crashing (tried two reruns) with:

App crashed or stopped while executing flow, please check diagnostic logs: ~/Library/Logs/DiagnosticReports directory

e2e tests on Cirrus Labs runners

e2397d0

alwx self-assigned this Dec 18, 2025

alwx marked this pull request as ready for review December 18, 2025 10:44

alwx requested review from antonis and lucas-zimerman as code owners December 18, 2025 10:44

alwx added the ready-to-merge Triggers the full CI test suite label Dec 18, 2025

alwx changed the title ~~e2e tests on Cirrus Labs runners~~ DO NOT MERGE: e2e tests on Cirrus Labs runners Dec 18, 2025

sentry bot reviewed Dec 18, 2025

View reviewed changes

alwx added 2 commits December 18, 2025 14:08

Checking another group

a04f351

rolling back last change

83a3d6a

sentry bot reviewed Dec 18, 2025

View reviewed changes

alwx added 2 commits December 23, 2025 09:36

Merge branch 'main' into alwx/experiment/cirrus-labs

1cb4e72

Merge branch 'main' into alwx/experiment/cirrus-labs

bd991eb

alwx added 3 commits January 28, 2026 09:08

Corepack enable

a933ded

Corepack enable

9899e3d

Ninja

61211d5

Ruby install

b5c48c1

sentry bot reviewed Jan 28, 2026

View reviewed changes

.github/workflows/e2e-v2.yml Show resolved Hide resolved

Ruby install

9142e2f

sentry bot reviewed Jan 28, 2026

View reviewed changes

.github/workflows/e2e-v2.yml Outdated Show resolved Hide resolved

alwx added 2 commits January 29, 2026 10:06

Updated env params

da9811a

Trying to fix metrics job

3034d58

alwx changed the title ~~DO NOT MERGE: e2e tests on Cirrus Labs runners~~ e2e tests on Cirrus Labs runners Jan 29, 2026

alwx added 2 commits January 29, 2026 11:13

Merge branch 'main' into alwx/experiment/cirrus-labs

8c3a337

Trying it out with Macos 26.0.1

550e671

sentry bot reviewed Jan 29, 2026

View reviewed changes

.github/workflows/e2e-v2.yml Outdated Show resolved Hide resolved

alwx added 3 commits January 29, 2026 11:20

iOS version env param change

064ae98

Debug outputs

78bf149

iDB companion update to potentially fix things on iOS side

92b254f

alwx commented Jan 29, 2026

View reviewed changes

alwx added 8 commits January 29, 2026 12:34

Another attempt

bd23cc4

Typo

7da4011

GH runners for tests

4c7db57

e2e fix attempt (with a temp workaround of using GH runners for tests)

d9d34c0

Rolling back some of experiments

0c1d899

Solving concurrency issues

3ca98e6

More GH runner issues

a685372

Merge branch 'main' into alwx/experiment/cirrus-labs

6a25aaa

Merge branch 'main' into alwx/experiment/cirrus-labs

cbd93fc

sentry bot reviewed Jan 30, 2026

View reviewed changes

.github/workflows/e2e-v2.yml Show resolved Hide resolved

alwx added 2 commits January 30, 2026 10:46

Small fix

ebb8321

Fix-2

bf21537

sentry bot reviewed Jan 30, 2026

View reviewed changes

.github/workflows/e2e-v2.yml Show resolved Hide resolved

Fix-3

a0aadff

antonis added the skip-changelog label Jan 30, 2026

antonis reviewed Jan 30, 2026

View reviewed changes

.github/workflows/e2e-v2.yml Outdated Show resolved Hide resolved

Hopefully final fix

94cbad2

alwx requested a review from antonis January 30, 2026 12:27

sentry bot reviewed Jan 30, 2026

View reviewed changes

antonis reviewed Jan 30, 2026

View reviewed changes

Uh oh!

e2e tests on Cirrus Labs runners #5485

Are you sure you want to change the base?

e2e tests on Cirrus Labs runners #5485

Conversation

alwx commented Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📢 Type of change

📜 Description

💡 Motivation and Context

💚 How did you test it?

📝 Checklist

🔮 Next steps

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

github-actions bot commented Jan 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Semver Impact of This PR

Uh oh!

github-actions bot commented Jan 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Android (legacy) Performance metrics 🚀

Baseline results on branch: main

Startup times

App size

Uh oh!

github-actions bot commented Jan 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Android (new) Performance metrics 🚀

Baseline results on branch: main

Startup times

App size

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jan 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

iOS (new) Performance metrics 🚀

Baseline results on branch: main

Startup times

App size

Uh oh!

github-actions bot commented Jan 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

iOS (legacy) Performance metrics 🚀

Baseline results on branch: main

Startup times

App size

Uh oh!

Uh oh!

alwx Jan 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lucas-zimerman commented Jan 29, 2026

Uh oh!

alwx commented Jan 30, 2026

Uh oh!

antonis commented Jan 30, 2026

Uh oh!

Uh oh!

alwx commented Jan 30, 2026

Uh oh!

Uh oh!

Uh oh!

sentry bot Jan 30, 2026

Choose a reason for hiding this comment

Uh oh!

antonis left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

alwx commented Dec 18, 2025 •

edited

Loading

github-actions bot commented Jan 26, 2026 •

edited

Loading

github-actions bot commented Jan 28, 2026 •

edited

Loading

github-actions bot commented Jan 28, 2026 •

edited

Loading

github-actions bot commented Jan 29, 2026 •

edited

Loading

github-actions bot commented Jan 29, 2026 •

edited

Loading

alwx Jan 29, 2026 •

edited

Loading

antonis left a comment •

edited

Loading