Skip to content

fix(ci): re-enable Android E2E tests with BrowserStack Detox #44

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
tsdamas wants to merge 8 commits into
base: main
Choose a base branch
from
Open

fix(ci): re-enable Android E2E tests with BrowserStack Detox #44

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
e9dd59f
fix(ci): re-enable Android E2E tests with BrowserStack Detox cloud.3;…
tsdamas May 28, 2026
a4dfc61
fix(ci): add device-client and external-idp to Android codegen pre-step
tsdamas May 29, 2026
97e9bc1
chore: trigger CI
tsdamas May 29, 2026
8f3b5be
chore: trigger CI
tsdamas May 29, 2026
652b712
triggering ci manually
tsdamas May 29, 2026
ee08aed
fix(ci): debug iOS simulator selection and exclude Android packaging …
tsdamas May 29, 2026
d892fd1
fix(ci): bump Xcode to 26.2 and scope Android Gradle assembly tasks
tsdamas May 29, 2026
be2385d
fix(ci): rollback xcode version on e2e and bump in unit
tsdamas May 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 6 additions & 3 deletions .github/workflows/browserstack-e2e-android.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,6 @@
# This software may be modified and distributed under the terms
# of the MIT license. See the LICENSE file for details.
#
# TODO: This workflow is not enforced right now and is currently failing.
# We have an open ticket with BrowserStack waiting for them to release a new version
# of the cloud driver based on Detox version 20.43+.
name: BrowserStack E2E — Android

on:
Expand Down Expand Up @@ -146,12 +143,18 @@ jobs:
# across all included modules so the directories are present.
run: |
cd android && ./gradlew \
:ping-identity_rn-binding:generateCodegenArtifactsFromSchema \
:ping-identity_rn-browser:generateCodegenArtifactsFromSchema \
:ping-identity_rn-device-client:generateCodegenArtifactsFromSchema \
:ping-identity_rn-device-id:generateCodegenArtifactsFromSchema \
:ping-identity_rn-device-profile:generateCodegenArtifactsFromSchema \
:ping-identity_rn-external-idp:generateCodegenArtifactsFromSchema \
:ping-identity_rn-fido:generateCodegenArtifactsFromSchema \
:ping-identity_rn-journey:generateCodegenArtifactsFromSchema \
:ping-identity_rn-logger:generateCodegenArtifactsFromSchema \
:ping-identity_rn-oath:generateCodegenArtifactsFromSchema \
:ping-identity_rn-oidc:generateCodegenArtifactsFromSchema \
:ping-identity_rn-push:generateCodegenArtifactsFromSchema \
:ping-identity_rn-storage:generateCodegenArtifactsFromSchema \
:react-native-async-storage_async-storage:generateCodegenArtifactsFromSchema

Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/browserstack-prep-ios-artifacts.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ jobs:
timeout-minutes: 60

env:
XCODE_VERSION: ${{ vars.XCODE_VERSION || '26.4.1' }}
XCODE_VERSION: ${{ vars.XCODE_VERSION || '26.1.1' }}
RUBY_VERSION: '2.6.10'
BUNDLE_GEMFILE: ${{ github.workspace }}/PingTestRunner/Gemfile
PING_TEST_RUNNER_DIR: PingTestRunner
Expand Down
75 changes: 21 additions & 54 deletions .github/workflows/build-and-test-ios.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,40 +83,9 @@ jobs:
working-directory: ${{ env.IOS_DIRECTORY }}
run: bundle exec pod install --repo-update

# Resolve Xcode: the pinned XCODE_VERSION may exist on the runner but lack the iOS simulator runtime
# component (SDK ≠ runtime — the SDK ships inside Xcode but the runtime is provisioned separately).
# This step prefers the pinned version but falls back to the newest installed Xcode whose major.minor
# iOS version has a matching pre-provisioned simulator runtime, so the destination is always bootable.
- name: Resolve Xcode version
run: |
PINNED="/Applications/Xcode_${{ env.XCODE_VERSION }}.app"
SELECTED=""
# Get all available iOS runtime versions (system-wide, not Xcode-scoped).
AVAILABLE_RUNTIMES=$(xcrun simctl list runtimes 2>/dev/null \
| grep -i "iOS" | grep -v "unavailable" \
| grep -oE "[0-9]+\.[0-9]+" | sort -u)
for app in "$PINNED" $(ls -d /Applications/Xcode*.app 2>/dev/null | sort -rV); do
[ -d "$app" ] || continue
# Derive the iOS SDK version this Xcode ships with (e.g. "26.1" from "iphonesimulator26.1").
xcode_ios=$(DEVELOPER_DIR="$app/Contents/Developer" \
xcodebuild -showsdks 2>/dev/null \
| grep -i "iphonesimulator" | grep -oE "[0-9]+\.[0-9]+" | sort -rV | head -1)
[ -n "$xcode_ios" ] || continue
# Check if a runtime matching this Xcode's major.minor iOS version is provisioned.
if echo "$AVAILABLE_RUNTIMES" | grep -qx "$xcode_ios"; then
SELECTED="$app"
echo "XCODE_RESOLVED_PATH=$app" >> "$GITHUB_ENV"
echo "Resolved Xcode: $app (iOS SDK $xcode_ios, runtime available)"
break
else
echo "Skipping $app: iOS $xcode_ios SDK present but runtime not provisioned (available: $(echo $AVAILABLE_RUNTIMES | tr '\n' ' '))"
fi
done
[ -n "$SELECTED" ] || { echo "No Xcode with a matching provisioned iOS simulator runtime found"; exit 1; }

# Select Xcode version
- name: Select Xcode version
run: sudo xcode-select -s "${{ env.XCODE_RESOLVED_PATH }}" && /usr/bin/xcodebuild -version
run: sudo xcode-select -s /Applications/Xcode_${{ env.XCODE_VERSION }}.app && /usr/bin/xcodebuild -version
Comment on lines 87 to +88
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot May 29, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Locate and print the relevant sections for the iOS workflow and the shared resolver
FILE1=".github/workflows/build-and-test-ios.yml"
FILE2=".github/workflows/e2e-tests.yml"

echo "=== $FILE1 (around select Xcode version + simulator destination) ==="
# Print around the referenced line numbers with some buffer
nl -ba "$FILE1" | sed -n '60,140p'

echo
echo "=== $FILE2 (around Xcode/simulator resolver) ==="
nl -ba "$FILE2" | sed -n '35,120p'

echo
echo "=== $FILE1 (full job blocks with continue-on-error) ==="
# Show lines containing continue-on-error and any "Resolve simulator destination" step name
rg -n "continue-on-error|Resolve simulator destination|xcode-select|Xcode_|xcodebuild -showdestinations" "$FILE1" || true

Repository: ForgeRock/ping-react-native-sdk

Length of output: 240

🏁 Script executed:

#!/bin/bash
set -euo pipefail

FILE1=".github/workflows/build-and-test-ios.yml"
FILE2=".github/workflows/e2e-tests.yml"

echo "=== $FILE1 (around select Xcode version + simulator destination) ==="
awk 'NR>=60 && NR<=140 {printf "%d:%s\n", NR, $0}' "$FILE1" | sed -n '1,200p'

echo
echo "=== $FILE2 (around Xcode/simulator resolver) ==="
awk 'NR>=35 && NR<=120 {printf "%d:%s\n", NR, $0}' "$FILE2" | sed -n '1,200p'

echo
echo "=== $FILE1 (locations of key steps/flags) ==="
rg -n "continue-on-error|Resolve simulator destination|xcode-select|Xcode_|xcodebuild -showdestinations" "$FILE1" || true

Repository: ForgeRock/ping-react-native-sdk

Length of output: 9971

🏁 Script executed:

#!/bin/bash
set -euo pipefail

FILE1=".github/workflows/build-and-test-ios.yml"
awk 'NR>=1 && NR<=260 {printf "%d:%s\n", NR, $0}' "$FILE1" | sed -n '1,260p'

Repository: ForgeRock/ping-react-native-sdk

Length of output: 9302

🌐 Web query:

GitHub Actions job-level continue-on-error support semantics

💡 Result:

GitHub Actions supports continue-on-error at the job level via jobs.<job_id>.continue-on-error (boolean / expression). When set to true, failures in that job don’t fail the overall workflow run, but they do still mark the job as failed (so you’ll typically see a red X in the UI) and any steps inside the job still behave as usual unless you also use step-level continue-on-error. Semantics to know 1) What changes at the job level - jobs.<job_id>.continue-on-error: true prevents the job from failing the overall workflow run; i.e., downstream workflows/jobs that depend on the run won’t be blocked the same way as with a hard failure. [1] - Even with job-level continue-on-error enabled, the job itself is still considered failed in the job’s own conclusion/visual status. [2] 2) How step expressions like failure behave - With job-level continue-on-error, a failed step can still trigger conditions like if: ${{ failure }} in subsequent steps because the step actually failed (even though the job is allowed to continue). This is consistent with reported behavior: the workflow runs subsequent steps, but the job is still marked as an overall failure. [3] 3) Job status vs workflow status - There’s a distinction between “job did not fail the workflow” and “job is green.” An observed outcome is: subsequent steps can run, but the job will be an overall failure (red X), even though the workflow may be allowed to proceed. [2] 4) Nested/indirect continue-on-error limitations - continue-on-error at a higher level does not necessarily suppress failures from deeply nested actions (e.g., action calling another action calling a node action). This means job-level continue-on-error may not “mask” all underlying tool failures across all nesting layers. [4] Practical guidance - Use job-level continue-on-error when you want the workflow to keep going even if that entire job encounters an error, but you still want the job to be visible as failed for diagnostics. [1][2] - If you need specific steps to not trigger conditional logic (or you want to control exactly which later steps run), consider step-level continue-on-error and/or explicit conditions based on step outcomes rather than relying on failure alone. The step-triggered behavior is documented/observed under job-level continue-on-error. [3] Example (job-level) jobs: test: runs-on: ubuntu-latest continue-on-error: true steps: - run: exit 1 - run: echo "This runs, but the job is still failed"

Citations:

Keep the runtime-compatible Xcode resolver (avoid pinned hard-switch) in iOS workflow

  • .github/workflows/build-and-test-ios.yml hard-switches to /Applications/Xcode_${{ env.XCODE_VERSION }}.app (lines 87-88) but doesn’t use the runtime-aware “Resolve Xcode version” logic from .github/workflows/e2e-tests.yml.
  • If that pinned Xcode lacks a provisioned iOS simulator runtime, “Resolve simulator destination” (lines 114-127) can fail with “No eligible iPhone simulator found”, preventing the later iOS test steps from running; with job-level continue-on-error: true (line 29), the overall workflow run can still succeed even when iOS tests never execute.
  • Reuse the existing resolver (or equivalent preflight verifying simulator runtime availability) before running xcode-select, and avoid relying on continue-on-error to mask simulator/setup failures when tests are expected to run.
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/build-and-test-ios.yml around lines 87 - 88, Replace the
hard-coded xcode-select step ("Select Xcode version") with the runtime-aware
"Resolve Xcode version" logic used in the e2e-tests workflow (or add an
equivalent preflight) so the job verifies a provisioned iOS simulator runtime
before switching Xcode; specifically, reuse the resolver that determines an
Xcode installation with matching simulator runtimes (the same step that precedes
"Resolve simulator destination") and run that check before calling xcode-select,
and then remove or narrow the use of job-level continue-on-error so
simulator/setup failures aren’t silently masked.


# Clang module caches are sensitive to SDK header mtimes on the runner image.
# This must run AFTER Xcode is selected so the active SDK is finalised.
Expand All @@ -136,31 +105,29 @@ jobs:
chmod +x product/xcresultparser
sudo mv product/xcresultparser /usr/local/bin/

# Resolve the best available iPhone simulator: prefer iPhone 17, fall back to any iPhone.
# Uses the device UDID as the destination to avoid OS version string mismatches between
# simctl runtime keys (e.g. "iOS-26-4") and xcodebuild's reported OS (e.g. "26.4.1").
# Resolve the best available iPhone simulator using xcodebuild -showdestinations.
# This asks Xcode itself which destinations are eligible for the scheme — only simulators
# compatible with the active Xcode SDK appear in "Available destinations", so no version
# string matching is needed and incompatible runtimes are automatically excluded.
# Prefers iPhone 17; falls back to any available iPhone simulator.
# Sets DESTINATION and SIMULATOR_UDID in GITHUB_ENV for all subsequent steps.
- name: Resolve simulator destination
run: |
DEVICES_JSON=$(xcrun simctl list devices available --json)
RESULT=$(echo "$DEVICES_JSON" | jq -r '
[.devices | to_entries[]
| select(.key | test("iOS"))
| . as $runtime
| .value[]
| select(.name | test("^iPhone"))
| {
os: ($runtime.key | gsub("com.apple.CoreSimulator.SimRuntime.iOS-"; "") | gsub("-"; ".")),
name: .name,
udid: .udid
}]
| (map(select(.name == "iPhone 17")) | sort_by(.os) | last)
// (sort_by(.os) | last)
| "\(.name)|\(.os)|\(.udid)"
')
SIMULATOR_NAME=$(echo "$RESULT" | cut -d'|' -f1)
OS_VERSION=$(echo "$RESULT" | cut -d'|' -f2)
SIMULATOR_UDID=$(echo "$RESULT" | cut -d'|' -f3)
DESTINATIONS=$(xcodebuild -showdestinations \
-workspace "${{ env.WORKSPACE }}" \
-scheme "${{ env.SCHEME }}" 2>&1)
echo "--- xcodebuild -showdestinations output ---"
echo "$DESTINATIONS"
echo "-------------------------------------------"
AVAILABLE=$(echo "$DESTINATIONS" | awk '/Available destinations/{flag=1; next} /Ineligible destinations/{flag=0} flag')
DEST_LINE=$(echo "$AVAILABLE" | grep "platform:iOS Simulator" | grep "name:iPhone 17" | tail -1)
if [ -z "$DEST_LINE" ]; then
DEST_LINE=$(echo "$AVAILABLE" | grep "platform:iOS Simulator" | grep "name:iPhone" | tail -1)
fi
[ -n "$DEST_LINE" ] || { echo "No eligible iPhone simulator found"; exit 1; }
Comment thread
tsdamas marked this conversation as resolved.
SIMULATOR_UDID=$(echo "$DEST_LINE" | grep -oE 'id:[A-F0-9-]+' | cut -d: -f2)
SIMULATOR_NAME=$(echo "$DEST_LINE" | grep -oE 'name:[^,}]+' | cut -d: -f2 | xargs)
OS_VERSION=$(echo "$DEST_LINE" | grep -oE 'OS:[^,}]+' | cut -d: -f2 | xargs)
echo "Resolved: $SIMULATOR_NAME ($OS_VERSION) [$SIMULATOR_UDID]"
echo "DESTINATION=platform=iOS Simulator,id=${SIMULATOR_UDID}" >> "$GITHUB_ENV"
echo "SIMULATOR_UDID=${SIMULATOR_UDID}" >> "$GITHUB_ENV"
Expand Down
4 changes: 1 addition & 3 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
#
name: CI
on:
workflow_dispatch:
pull_request:
types: [opened, reopened, synchronize, edited]

Expand Down Expand Up @@ -104,10 +105,7 @@ jobs:
secrets: inherit

# Run Android E2E tests on BrowserStack real devices after Android unit tests pass.
# Blocked: BrowserStack cloud driver does not yet support Detox 20.43+.
# Remove the `if false` guard once the upstream fix is released.
browserstack-android:
if: false
needs: android-unit-tests
permissions:
contents: read
Expand Down
6 changes: 0 additions & 6 deletions .github/workflows/e2e-tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,13 +115,7 @@ jobs:
path: PingTestRunner/artifacts/

# ── Android E2E (emulator on ubuntu-latest) ───────────────────────────────
# NOTE: This job will currently fail. The BrowserStack Detox fork we depend on
# for cloud device testing is pinned to a Detox version that is incompatible
# with React Native 0.80.
# TODO: Re-enable and validate once BrowserStack releases an updated version
# of their Detox fork with RN 0.80 support.
e2e-android:
if: false
name: E2E — Android Emulator
runs-on: ubuntu-latest
timeout-minutes: 60
Expand Down
Binary file modified .yarn/install-state.gz
View file
Open in desktop
Binary file not shown.
6 changes: 6 additions & 0 deletions PingTestRunner/android/app/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,12 @@ android {
proguardFile "${rootProject.projectDir}/../node_modules/detox/android/detox/proguard-rules-app.pro"
}
}
packaging {
resources {
excludes += "/META-INF/{AL2.0,LGPL2.1}"
excludes += "/META-INF/versions/9/OSGI-INF/MANIFEST.MF"
}
}
}

dependencies {
Expand Down
4 changes: 4 additions & 0 deletions PingTestRunner/android/settings.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,10 +33,14 @@ include(":ping-identity_rn-browser")
project(":ping-identity_rn-browser").projectDir = file("../../packages/browser/android")
include(":ping-identity_rn-core")
project(":ping-identity_rn-core").projectDir = file("../../packages/core/android")
include(":ping-identity_rn-device-client")
project(":ping-identity_rn-device-client").projectDir = file("../../packages/device-client/android")
include(":ping-identity_rn-device-id")
project(":ping-identity_rn-device-id").projectDir = file("../../packages/device-id/android")
include(":ping-identity_rn-device-profile")
project(":ping-identity_rn-device-profile").projectDir = file("../../packages/device-profile/android")
include(":ping-identity_rn-external-idp")
project(":ping-identity_rn-external-idp").projectDir = file("../../packages/external-idp/android")
include(":ping-identity_rn-fido")
project(":ping-identity_rn-fido").projectDir = file("../../packages/fido/android")
include(":ping-identity_rn-journey")
Expand Down
4 changes: 2 additions & 2 deletions PingTestRunner/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
"test:bs:android": "NODE_NO_WARNINGS=1 detox test --configuration android.bs --loglevel trace",
"test:e2e:ios": "sh -c 'export DETOX_SERVER_PORT=${DETOX_SERVER_PORT:-8099}; export NODE_NO_WARNINGS=1; yarn e2e:android:free-detox-port && detox test --configuration ios.sim \"$@\"' --",
"build:e2e:android": "detox build --configuration android.emu",
"build:bs:android": "cd android && ./gradlew assembleRelease assembleAndroidTest -DtestBuildType=release",
"build:bs:android": "cd android && ./gradlew :app:assembleRelease :app:assembleAndroidTest -DtestBuildType=release",
"build:e2e:ios": "detox build --configuration ios.sim",
"build:bs:ios:archive": "sh ./scripts/build-bs-ios-archive.sh",
"build:bs:ios:ipa": "sh ./scripts/build-bs-ios-ipa.sh",
Expand Down Expand Up @@ -63,7 +63,7 @@
"@testing-library/react-native": "^13.3.3",
"@types/jest": "^29.5.14",
"@types/react": "^19.1.0",
"detox": "npm:@browserstack/detox@20.38.0-cloud.1",
"detox": "npm:@browserstack/detox@20.38.0-cloud.3",
"dotenv": "^16.4.7",
"eslint": "^8.19.0",
"jest": "^29.7.0",
Expand Down
10 changes: 5 additions & 5 deletions yarn.lock
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6442,9 +6442,9 @@ __metadata:
languageName: node
linkType: hard

"detox@npm:@browserstack/detox@20.38.0-cloud.1":
version: 20.38.0-cloud.1
resolution: "@browserstack/detox@npm:20.38.0-cloud.1"
"detox@npm:@browserstack/detox@20.38.0-cloud.3":
version: 20.38.0-cloud.3
resolution: "@browserstack/detox@npm:20.38.0-cloud.3"
dependencies:
"@wix-pilot/core": "npm:^3.2.2"
"@wix-pilot/detox": "npm:^1.0.11"
Expand Down Expand Up @@ -6490,7 +6490,7 @@ __metadata:
optional: true
bin:
detox: local-cli/cli.js
checksum: 10c0/5f59b76c1b9427132b5b3068cdab5950893b1d98d44f7418d0f2563853d618b62b8db53be762bedf73793f15eb5b51fce75cab62e5aa34bb65773494c18683d3
checksum: 10c0/e49a2c37146b3e6cbf2221fea32b29ace3dda75d6bcaae6a531b3827a1b8a4f99556dd7b858881d4eace49df0db3249c36aa8d786b076cbb05e323eb25e2f39f
languageName: node
linkType: hard

Expand Down Expand Up @@ -12554,7 +12554,7 @@ __metadata:
"@testing-library/react-native": "npm:^13.3.3"
"@types/jest": "npm:^29.5.14"
"@types/react": "npm:^19.1.0"
detox: "npm:@browserstack/detox@20.38.0-cloud.1"
detox: "npm:@browserstack/detox@20.38.0-cloud.3"
dotenv: "npm:^16.4.7"
eslint: "npm:^8.19.0"
jest: "npm:^29.7.0"
Expand Down
Loading