From ca50eac3a85433b1fe21cf0844eec8dd9d6c82cd Mon Sep 17 00:00:00 2001 From: Piotr Stachyra Date: Wed, 18 Mar 2026 17:57:29 +0100 Subject: [PATCH 1/7] feat: add channel monitor desync repro tooling Add scripts and docs for reproducing the ChannelMonitor/ChannelManager desync caused by fetchOrphanedChannelMonitorsIfNeeded in v2.1.0. - docs/repro-channel-monitor-desync.md: full repro steps for iOS & Android - scripts/pay-lightning-address.sh: pay BOLT11/BIP21/LN address via Blocktank - test/specs/receive-ln-payments.e2e.ts: Appium spec to receive 21+ LN payments - wdio.no-install.conf.ts: WDIO config that attaches to existing app Made-with: Cursor --- docs/repro-channel-monitor-desync.md | 136 ++++++++++++++++++++++++++ scripts/pay-lightning-address.sh | 130 ++++++++++++++++++++++++ test/specs/receive-ln-payments.e2e.ts | 92 +++++++++++++++++ wdio.no-install.conf.ts | 48 +++++++++ 4 files changed, 406 insertions(+) create mode 100644 docs/repro-channel-monitor-desync.md create mode 100755 scripts/pay-lightning-address.sh create mode 100644 test/specs/receive-ln-payments.e2e.ts create mode 100644 wdio.no-install.conf.ts diff --git a/docs/repro-channel-monitor-desync.md b/docs/repro-channel-monitor-desync.md new file mode 100644 index 0000000..ca17ac5 --- /dev/null +++ b/docs/repro-channel-monitor-desync.md @@ -0,0 +1,136 @@ +# Repro: ChannelMonitor Desync (Stale RN Monitor Injection) + +Related issues: +- [#847 (bitkit-android)](https://github.com/synonymdev/bitkit-android/issues/847) +- iOS support ticket (user logs from 2026-03-18) + +## Summary + +Build 182 (v2.1.0) introduced `fetchOrphanedChannelMonitorsIfNeeded` which fetches old channel monitors from the RN remote backup on every startup (when `isChannelRecoveryChecked` is false). If the wallet was migrated from RN to native and used on native for enough LN payments, the RN backup monitor is stale. Injecting it causes a fatal ChannelManager/ChannelMonitor update_id mismatch and LDK refuses to start. + +## Root Cause + +On v2.1.0 startup: +1. `fetchOrphanedChannelMonitorsIfNeeded` fetches stale channel monitor from RN backup server +2. Injects it via `setChannelDataMigration` with `channelManager: nil` (monitors only) +3. ldk-node persists the stale monitor to VSS/local storage +4. LDK loads ChannelManager (advanced) against stale ChannelMonitor → fatal mismatch +5. Node refuses to start: `"A ChannelMonitor is stale compared to the current ChannelManager!"` + +## Prerequisites + +- **Bitkit v1.1.6** (React Native) iOS or Android build — for the initial RN wallet setup +- **Bitkit v2.0.6** (native) iOS or Android build — for the intermediate native version +- **Bitkit v2.1.0 / build 182** (native) iOS or Android build — the version with the bug +- **Staging regtest** Blocktank API access (`BACKEND=regtest`) +- **Appium** running locally for the automated payment step +- **bitkit-e2e-tests** repo checked out + +## Repro Steps + +### Step 1: RN wallet setup (v1.1.6) + +1. Install v1.1.6 (RN) on simulator/emulator +2. Create a new wallet +3. Fund the wallet on-chain (use `./scripts/fund-address.sh
`) +4. Mine blocks to confirm +5. Open a Lightning channel (transfer to spending) +6. Wait for the channel to be ready +7. Make **1 Lightning payment** (to confirm the channel works and create RN backup data) + +### Step 2: Migrate to native (v2.0.6) + +1. Install v2.0.6 (native) **over** the RN app (upgrade, not clean install) +2. The RN → native migration runs automatically +3. Wait for the wallet to fully sync and the Lightning node to start +4. Verify the channel is open and working + +### Step 3: Make 21+ Lightning payments on native + +This advances the ChannelManager in VSS past the frozen RN backup monitor. + +**Automated (recommended):** + +Make sure Appium is running, then: + +```bash +# iOS +SIMULATOR_OS_VERSION=26.0 BACKEND=regtest npx wdio wdio.no-install.conf.ts + +# Android +PLATFORM=android BACKEND=regtest npx wdio wdio.no-install.conf.ts +``` + +This runs `test/specs/receive-ln-payments.e2e.ts` which: +- Attaches to the already-installed app (no reinstall) +- Loops 21 times: tap Receive → grab invoice from QR → pay via Blocktank → acknowledge +- Default: 21 payments of 10 sats each + +Customize with env vars: +```bash +PAYMENT_COUNT=25 PAYMENT_AMOUNT=10 SIMULATOR_OS_VERSION=26.0 BACKEND=regtest npx wdio wdio.no-install.conf.ts +``` + +**Manual alternative:** + +Use the shell script to pay invoices grabbed from the app UI: +```bash +# Pay a BIP21 URI copied from the app's receive screen +./scripts/pay-lightning-address.sh 'bitcoin:bcrt1q...?lightning=lnbcrt1...' 10 +``` + +Repeat 21+ times with fresh invoices each time. + +### Step 4: Upgrade to v2.1.0 + +1. Install v2.1.0 / build 182 **over** the v2.0.6 app (upgrade) +2. Launch the app + +### Expected Result + +The app fails to start the Lightning node with: + +``` +A ChannelMonitor is stale compared to the current ChannelManager! +The ChannelMonitor for channel is at update_id with update_id through in-flight +but the ChannelManager is at update_id . +Failed to read channel manager from store: Value would be dangerous to continue execution with +``` + +In app logs, look for: +``` +Running pre-startup channel monitor recovery check +Found 1 monitors on RN backup for pre-startup recovery +Applied channel migration: 1 monitors +Migrating channel monitor: +A ChannelMonitor is stale compared to the current ChannelManager! +Read failed [Failed to read from store.] +Failed to start wallet +``` + +## Key Details + +- **21 payments on native is the minimum** that reliably reproduces. Each payment generates ~3-5 update_id increments. LDK can recover small gaps (~10 updates) by replaying counterparty commitment updates, so fewer payments may not trigger the fatal crash. +- **RN payments don't need to be many** — just enough to establish the channel and create the RN backup. +- The bug is in `fetchOrphanedChannelMonitorsIfNeeded` in `WalletViewModel.swift` (iOS) / `WalletViewModel.kt` (Android). It unconditionally injects old RN monitors without checking compatibility with the current ChannelManager. + +## Fix Verification + +To verify the fix (e.g. `release-2.1.1` or `fix/channel-monitor-stale-data-v2`): + +1. Reproduce the bug using steps 1-4 above on v2.1.0 +2. Confirm the node fails to start +3. Install the fix build **over** the broken v2.1.0 +4. Launch the app +5. Check logs — the node should either: + - Start successfully (fix prevents stale monitor injection), or + - Handle the already-corrupted state gracefully (fix in ldk-node) + +## Files + +| File | Purpose | +|------|---------| +| `test/specs/receive-ln-payments.e2e.ts` | Automated spec to receive N Lightning payments | +| `wdio.no-install.conf.ts` | WDIO config that attaches to existing app (no reinstall) | +| `scripts/pay-lightning-address.sh` | Shell script to pay BOLT11/BIP21/LN address via Blocktank | +| `scripts/pay-lightning-address-loop.sh` | Shell script to send N payments to a Lightning address | diff --git a/scripts/pay-lightning-address.sh b/scripts/pay-lightning-address.sh new file mode 100755 index 0000000..f02f28f --- /dev/null +++ b/scripts/pay-lightning-address.sh @@ -0,0 +1,130 @@ +#!/bin/bash +# +# Pay a Lightning invoice via Blocktank staging regtest API +# +# Accepts: +# BOLT11 invoice: lnbcrt1p5m4tld... +# Lightning address: satoshi@domain.com +# BIP21 URI: bitcoin:bcrt1q...?lightning=lnbcrt1p5m4tld... +# +# Usage: +# ./scripts/pay-lightning-address.sh [amount_sats] +# +# Examples: +# ./scripts/pay-lightning-address.sh lnbcrt1p5m4tld... +# ./scripts/pay-lightning-address.sh 'bitcoin:bcrt1q...?lightning=lnbcrt1p5m4tld...' +# ./scripts/pay-lightning-address.sh satoshi@localhost:3003 500 +# +# Environment: +# BLOCKTANK_URL → override API base (default: staging) +# LNURL_SCHEME → http or https for LN address resolution (default: auto) + +set -e + +INPUT="$1" +AMOUNT_SATS="${2:-100}" +BLOCKTANK_URL="${BLOCKTANK_URL:-https://api.stag0.blocktank.to/blocktank/api/v2}" + +if [ -z "$INPUT" ]; then + echo "Usage: $0 [amount_sats]" + echo "" + echo " BOLT11: $0 lnbcrt1p5m4tld..." + echo " BIP21: $0 'bitcoin:bcrt1q...?lightning=lnbcrt1p5m4tld...'" + echo " LN addr: $0 satoshi@localhost:3003 500" + exit 1 +fi + +INVOICE="" + +# --- Detect input type --- + +# BIP21 URI with lightning= parameter +if echo "$INPUT" | grep -qi "^bitcoin:.*lightning="; then + INVOICE=$(echo "$INPUT" | sed -n 's/.*[?&]lightning=\([^&]*\).*/\1/p' | tr -d '[:space:]') + if [ -z "$INVOICE" ]; then + echo "✗ Could not extract lightning invoice from BIP21 URI" + exit 1 + fi + echo "→ Extracted invoice from BIP21 URI (${#INVOICE} chars)" + +# Raw BOLT11 invoice +elif echo "$INPUT" | grep -qiE "^ln(bc|tb|tbs|bcrt)"; then + INVOICE=$(echo "$INPUT" | tr -d '[:space:]') + echo "→ Using BOLT11 invoice (${#INVOICE} chars)" + +# Lightning address (user@domain) +elif echo "$INPUT" | grep -q "@"; then + if ! command -v jq &> /dev/null; then + echo "✗ jq required for LN address. Install: brew install jq" + exit 1 + fi + + USERNAME="${INPUT%%@*}" + DOMAIN="${INPUT#*@}" + + if [ -n "$LNURL_SCHEME" ]; then + SCHEME="$LNURL_SCHEME" + elif echo "$DOMAIN" | grep -qE "^(localhost|127\.0\.0\.1)"; then + SCHEME="http" + else + SCHEME="https" + fi + + AMOUNT_MSATS=$((AMOUNT_SATS * 1000)) + echo "→ Resolving ${USERNAME}@${DOMAIN}..." + + LNURL_RESPONSE=$(curl -sf "${SCHEME}://${DOMAIN}/.well-known/lnurlp/${USERNAME}") + if [ $? -ne 0 ]; then + echo "✗ Failed to resolve lightning address" + exit 1 + fi + + CALLBACK=$(echo "$LNURL_RESPONSE" | jq -r '.callback') + MIN_SENDABLE=$(echo "$LNURL_RESPONSE" | jq -r '.minSendable') + MAX_SENDABLE=$(echo "$LNURL_RESPONSE" | jq -r '.maxSendable') + + if [ "$AMOUNT_MSATS" -lt "$MIN_SENDABLE" ] || [ "$AMOUNT_MSATS" -gt "$MAX_SENDABLE" ]; then + echo "✗ ${AMOUNT_SATS} sats out of range [$((MIN_SENDABLE/1000))-$((MAX_SENDABLE/1000))] sats" + exit 1 + fi + + echo "→ Requesting invoice for ${AMOUNT_SATS} sats..." + SEP="?"; echo "$CALLBACK" | grep -q "?" && SEP="&" + INVOICE_RESPONSE=$(curl -sf "${CALLBACK}${SEP}amount=${AMOUNT_MSATS}") + INVOICE=$(echo "$INVOICE_RESPONSE" | jq -r '.pr' | tr -d '[:space:]') + + if [ -z "$INVOICE" ] || [ "$INVOICE" = "null" ]; then + echo "✗ No invoice in callback response" + echo "$INVOICE_RESPONSE" | jq . + exit 1 + fi + echo "→ Got invoice (${#INVOICE} chars)" +else + echo "✗ Unrecognized format. Use: BOLT11, BIP21 URI, or lightning address" + exit 1 +fi + +# --- Pay via Blocktank --- + +BODY="{\"invoice\": \"${INVOICE}\"" +if [ -n "$AMOUNT_SATS" ] && [ "$AMOUNT_SATS" -gt 0 ] 2>/dev/null; then + BODY="${BODY}, \"amountSat\": ${AMOUNT_SATS}" +fi +BODY="${BODY}}" + +echo "→ Paying ${AMOUNT_SATS} sats via Blocktank..." +PAY_RESPONSE=$(curl -s -w "\n%{http_code}" -X POST "${BLOCKTANK_URL}/regtest/channel/pay" \ + -H "Content-Type: application/json" \ + -d "$BODY") + +HTTP_CODE=$(echo "$PAY_RESPONSE" | tail -n1) +BODY=$(echo "$PAY_RESPONSE" | sed '$d') + +if [ "$HTTP_CODE" -ge 200 ] && [ "$HTTP_CODE" -lt 300 ]; then + echo "✓ Payment sent" + [ -n "$BODY" ] && echo " $BODY" +else + echo "✗ Payment failed: HTTP $HTTP_CODE" + echo " $BODY" + exit 1 +fi diff --git a/test/specs/receive-ln-payments.e2e.ts b/test/specs/receive-ln-payments.e2e.ts new file mode 100644 index 0000000..765ee17 --- /dev/null +++ b/test/specs/receive-ln-payments.e2e.ts @@ -0,0 +1,92 @@ +/** + * Utility spec: receive multiple Lightning payments on an already-installed app. + * + * IMPORTANT: This spec attaches to the app already on the simulator — it does NOT + * reinstall or reset. Use this when you have a specific version prepared and just + * want to pump LN payments into it. + * + * Usage: + * BACKEND=regtest npm run e2e:ios -- --spec ./test/specs/receive-ln-payments.e2e.ts + * + * Environment: + * PAYMENT_COUNT — number of payments to receive (default: 21) + * PAYMENT_AMOUNT — amount per payment in sats (default: 10) + */ + +import { + acknowledgeReceivedPayment, + elementById, + getUriFromQRCode, + sleep, + swipeFullScreen, + tap, +} from '../helpers/actions'; +import { payInvoice } from '../helpers/regtest'; +import { getAppId } from '../helpers/constants'; + +const PAYMENT_COUNT = Number(process.env.PAYMENT_COUNT || '21'); +const PAYMENT_AMOUNT = Number(process.env.PAYMENT_AMOUNT || '10'); + +function extractLightningInvoice(uri: string): string { + const query = uri.split('?')[1] ?? ''; + const params = new URLSearchParams(query); + const ln = params.get('lightning'); + if (!ln) { + throw new Error(`No lightning invoice found in URI: ${uri}`); + } + return ln; +} + +describe('Receive LN payments (utility)', () => { + before(async () => { + const appId = getAppId(); + await driver.activateApp(appId); + await sleep(3000); + }); + + it(`should receive ${PAYMENT_COUNT} Lightning payments`, async () => { + let succeeded = 0; + let failed = 0; + + for (let i = 0; i < PAYMENT_COUNT; i++) { + const label = `[${i + 1}/${PAYMENT_COUNT}]`; + try { + await tap('Receive'); + await sleep(1000); + + const uri = await getUriFromQRCode(); + const invoice = extractLightningInvoice(uri); + console.info(`${label} Got invoice (${invoice.length} chars)`); + + await swipeFullScreen('down'); + await sleep(500); + + await payInvoice(invoice, PAYMENT_AMOUNT); + console.info(`${label} Payment sent, waiting for acknowledgement...`); + + try { + await acknowledgeReceivedPayment(); + console.info(`${label} ✓ Payment received`); + } catch { + console.info(`${label} ✓ Payment sent (no ack prompt)`); + } + + succeeded++; + await sleep(1000); + } catch (error) { + failed++; + console.error(`${label} ✗ Failed: ${error}`); + try { + await swipeFullScreen('down'); + } catch { /* ignore */ } + await sleep(2000); + } + } + + console.info(`\n══════════════════════════════════`); + console.info(` Done: ${succeeded}/${PAYMENT_COUNT} succeeded, ${failed} failed`); + console.info(`══════════════════════════════════\n`); + + expect(succeeded).toBeGreaterThan(0); + }); +}); diff --git a/wdio.no-install.conf.ts b/wdio.no-install.conf.ts new file mode 100644 index 0000000..003bf65 --- /dev/null +++ b/wdio.no-install.conf.ts @@ -0,0 +1,48 @@ +/** + * WDIO config that attaches to the already-installed app on the simulator. + * Does NOT reinstall, reset, or modify the app. Just connects via Appium. + * + * Usage: + * BACKEND=regtest npx wdio wdio.no-install.conf.ts --spec ./test/specs/receive-ln-payments.e2e.ts + */ + +import { config as baseConfig } from './wdio.conf'; + +const isAndroid = process.env.PLATFORM === 'android'; + +const iosDeviceName = process.env.SIMULATOR_NAME || 'iPhone 17'; +const iosPlatformVersion = process.env.SIMULATOR_OS_VERSION || '26.0.1'; +const appBundleId = process.env.APP_ID_IOS ?? 'to.bitkit'; +const androidAppId = process.env.APP_ID_ANDROID ?? 'to.bitkit.dev'; + +export const config: WebdriverIO.Config = { + ...baseConfig, + specs: [['./test/specs/receive-ln-payments.e2e.ts']], + capabilities: [ + isAndroid + ? { + platformName: 'Android', + 'appium:automationName': 'UiAutomator2', + 'appium:appPackage': androidAppId, + 'appium:appActivity': '.ui.MainActivity', + 'appium:noReset': true, + 'appium:fullReset': false, + 'appium:autoGrantPermissions': true, + 'appium:newCommandTimeout': 300, + } + : { + platformName: 'iOS', + 'appium:automationName': 'XCUITest', + 'appium:udid': process.env.SIMULATOR_UDID || 'auto', + 'appium:deviceName': iosDeviceName, + ...(iosPlatformVersion ? { 'appium:platformVersion': iosPlatformVersion } : {}), + 'appium:bundleId': appBundleId, + 'appium:noReset': true, + 'appium:fullReset': false, + 'appium:autoAcceptAlerts': false, + 'appium:newCommandTimeout': 300, + 'appium:wdaLaunchTimeout': 300000, + 'appium:wdaConnectionTimeout': 300000, + }, + ], +}; From 92f3e4c931729a125ac6d9ae1e6c333335d42fbf Mon Sep 17 00:00:00 2001 From: Piotr Stachyra Date: Thu, 19 Mar 2026 15:55:56 +0100 Subject: [PATCH 2/7] add updater --- updater/release.json | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100644 updater/release.json diff --git a/updater/release.json b/updater/release.json new file mode 100644 index 0000000..faf8f74 --- /dev/null +++ b/updater/release.json @@ -0,0 +1,20 @@ +{ + "platforms": { + "ios": { + "version": "v0.0.0", + "buildNumber": 0, + "notes": "", + "pub_date": "2026-01-01T00:00:00Z", + "url": "https://apps.apple.com/app/bitkit-wallet/id6502440655", + "critical": false + }, + "android": { + "version": "v0.0.0", + "buildNumber": 0, + "notes": "", + "pub_date": "2026-01-01T00:00:00Z", + "url": "https://play.google.com/store/apps/details?id=to.bitkit", + "critical": false + } + } +} From 5290486e20f8013361384c87f634812a93b464d4 Mon Sep 17 00:00:00 2001 From: Piotr Stachyra Date: Fri, 20 Mar 2026 15:00:09 +0100 Subject: [PATCH 3/7] 3rd party channel repro attempt --- docker/bitcoin-cli | 123 ++++++++++++++++++++++++ docs/repro-channel-monitor-desync.md | 139 ++++++++++++++++++++++++--- test/helpers/constants.ts | 4 + test/helpers/regtest.ts | 77 +++++++++++++-- 4 files changed, 326 insertions(+), 17 deletions(-) diff --git a/docker/bitcoin-cli b/docker/bitcoin-cli index 6c86d0c..6700969 100755 --- a/docker/bitcoin-cli +++ b/docker/bitcoin-cli @@ -28,6 +28,8 @@ Commands: getInvoice Get a new BIP21 URI with a bech32 address LND: getinfo Show LND node info (for connectivity debugging) + openchannel [amount] Open channel from LND to node (default: 500000 sats) + payinvoice [amount] Pay a Lightning invoice via LND holdinvoice [amount] [-m memo] Create a hold invoice settleinvoice Reveal a preimage and use it to settle the corresponding invoice cancelinvoice Cancels a currently open invoice @@ -196,6 +198,127 @@ if [[ "$command" = "getinfo" ]]; then exit fi +# Open channel from LND to a node +if [[ "$command" = "openchannel" ]]; then + shift + + node_id="${1:-}" + amount="${2:-500000}" + + if [ -z "$node_id" ]; then + echo "Usage: $CLI_NAME openchannel [amount_sats]" + echo "" + echo " node_id: app's Lightning node ID (Settings > Advanced > Lightning Node Info)" + echo " amount: channel size in sats (default: 500000)" + exit 1 + fi + + # Check peer connection + echo "→ Checking peer connection..." + peer_count=$("${LNCLI_CMD[@]}" listpeers 2>/dev/null | jq "[.peers[] | select(.pub_key==\"$node_id\")] | length") + + if [ "$peer_count" = "0" ]; then + lnd_pubkey=$("${LNCLI_CMD[@]}" getinfo 2>/dev/null | jq -r '.identity_pubkey') + echo "✗ Node is not connected as a peer." + echo "" + echo " Paste this in the app (Settings > Advanced > Channels > Add Connection):" + echo " ${lnd_pubkey}@0.0.0.0:9735" + echo "" + echo " Then re-run this command." + exit 1 + fi + + echo "✓ Peer connected" + + # Fund LND if needed + balance=$("${LNCLI_CMD[@]}" walletbalance 2>/dev/null | jq -r '.confirmed_balance') + echo "→ LND on-chain balance: $balance sats" + + if [ "$balance" -lt "$amount" ]; then + echo "→ Funding LND..." + lnd_addr=$("${LNCLI_CMD[@]}" newaddress p2wkh 2>/dev/null | jq -r '.address') + "${BASE_COMMAND[@]}" -named sendtoaddress address="$lnd_addr" amount=1 fee_rate=25 > /dev/null + "${BASE_COMMAND[@]}" -generate 6 > /dev/null + echo "✓ Funded LND with 1 BTC" + sleep 2 + fi + + # Open channel + echo "→ Opening ${amount} sat channel to ${node_id:0:20}..." + result=$("${LNCLI_CMD[@]}" openchannel --node_key "$node_id" --local_amt "$amount" --private 2>&1) || { + echo "✗ Failed: $result" + exit 1 + } + + txid=$(echo "$result" | jq -r '.funding_txid // empty' 2>/dev/null) + if [ -z "$txid" ]; then + echo "✗ Failed: $result" + exit 1 + fi + + echo "✓ Channel opened, funding txid: $txid" + + # Mine and wait + echo "→ Mining 6 blocks..." + "${BASE_COMMAND[@]}" -generate 6 > /dev/null + echo "✓ Mined 6 blocks" + + echo "→ Waiting for channel to become active..." + for i in $(seq 1 30); do + sleep 2 + active=$("${LNCLI_CMD[@]}" listchannels --peer "$node_id" --active_only 2>/dev/null | jq '.channels | length') + if [ "$active" != "0" ]; then + echo "✓ Channel is active!" + break + fi + if [ $((i % 5)) -eq 0 ]; then echo " still waiting... ($i)"; fi + done + + if [ "$active" = "0" ]; then + echo "⚠ Channel not active yet. May need more time or app needs to sync." + fi + + # Summary + echo "" + echo "══════════════════════════════════" + "${LNCLI_CMD[@]}" channelbalance 2>/dev/null | jq -r '" LND outbound: \(.local_balance.sat) sats (can pay app)\n LND inbound: \(.remote_balance.sat) sats (can receive from app)"' + echo "══════════════════════════════════" + exit +fi + +# Pay a Lightning invoice via LND +if [[ "$command" = "payinvoice" ]]; then + shift + + invoice="${1:-}" + amount="${2:-}" + + if [ -z "$invoice" ]; then + echo "Usage: $CLI_NAME payinvoice [amount_sats]" + exit 1 + fi + + if [ -n "$amount" ]; then + echo "→ Paying invoice via LND (${amount} sats)..." + result=$("${LNCLI_CMD[@]}" payinvoice --force --amt "$amount" "$invoice" 2>&1) + else + echo "→ Paying invoice via LND..." + result=$("${LNCLI_CMD[@]}" payinvoice --force "$invoice" 2>&1) + fi + + status=$(echo "$result" | grep -i "status" | head -1) + if echo "$result" | grep -qi "SUCCEEDED"; then + echo "✓ Payment succeeded" + echo "$result" | grep -i "payment_hash\|payment_preimage" | head -2 + else + echo "✗ Payment failed" + echo "$result" + exit 1 + fi + + exit +fi + # Create a hold invoice (LND) if [[ "$command" = "holdinvoice" ]]; then shift diff --git a/docs/repro-channel-monitor-desync.md b/docs/repro-channel-monitor-desync.md index ca17ac5..f6b19e0 100644 --- a/docs/repro-channel-monitor-desync.md +++ b/docs/repro-channel-monitor-desync.md @@ -17,18 +17,24 @@ On v2.1.0 startup: 4. LDK loads ChannelManager (advanced) against stale ChannelMonitor → fatal mismatch 5. Node refuses to start: `"A ChannelMonitor is stale compared to the current ChannelManager!"` -## Prerequisites +--- + +## Case 1: Blocktank channel (staging regtest) + +Reproduces the bug using a Blocktank LSP channel opened via "transfer to spending". + +### Prerequisites - **Bitkit v1.1.6** (React Native) iOS or Android build — for the initial RN wallet setup -- **Bitkit v2.0.6** (native) iOS or Android build — for the intermediate native version +- **Bitkit v2.0.6** (native iOS) or **v2.0.3** (native Android) build — intermediate native version - **Bitkit v2.1.0 / build 182** (native) iOS or Android build — the version with the bug - **Staging regtest** Blocktank API access (`BACKEND=regtest`) - **Appium** running locally for the automated payment step - **bitkit-e2e-tests** repo checked out -## Repro Steps +### Repro Steps -### Step 1: RN wallet setup (v1.1.6) +#### Step 1: RN wallet setup (v1.1.6) 1. Install v1.1.6 (RN) on simulator/emulator 2. Create a new wallet @@ -38,14 +44,14 @@ On v2.1.0 startup: 6. Wait for the channel to be ready 7. Make **1 Lightning payment** (to confirm the channel works and create RN backup data) -### Step 2: Migrate to native (v2.0.6) +#### Step 2: Migrate to native (v2.0.6 iOS / v2.0.3 Android) -1. Install v2.0.6 (native) **over** the RN app (upgrade, not clean install) +1. Install the native build **over** the RN app (upgrade, not clean install) 2. The RN → native migration runs automatically 3. Wait for the wallet to fully sync and the Lightning node to start 4. Verify the channel is open and working -### Step 3: Make 21+ Lightning payments on native +#### Step 3: Make 21+ Lightning payments on native This advances the ChannelManager in VSS past the frozen RN backup monitor. @@ -81,12 +87,12 @@ Use the shell script to pay invoices grabbed from the app UI: Repeat 21+ times with fresh invoices each time. -### Step 4: Upgrade to v2.1.0 +#### Step 4: Upgrade to v2.1.0 1. Install v2.1.0 / build 182 **over** the v2.0.6 app (upgrade) 2. Launch the app -### Expected Result +#### Expected Result The app fails to start the Lightning node with: @@ -108,9 +114,119 @@ Read failed [Failed to read from store.] Failed to start wallet ``` +--- + +## Case 2: 3rd-party channel (local docker) — NOT REPRODUCED + +Attempted to reproduce the bug using a 3rd-party channel (local docker LND) instead of Blocktank. This tests whether the bug also affects non-LSP channels. **The bug did not reproduce with this setup**, even after 50 payments. + +### Prerequisites + +- **bitkit-docker** running locally (Bitcoin, Electrum, LND, backup server) +- Local regtest builds for each version: + - `bitkit_rn_local_v1.1.6.apk` / `bitkit_rn_local_ios_v1.1.6.app` + - Native v2.0.6 (iOS) / v2.0.3 (Android) built with `BACKEND=local` + - Native v2.1.0 built with `BACKEND=local` +- **Appium** running locally for the automated payment step +- **bitkit-e2e-tests** repo checked out + +### Build Notes + +RN v1.1.6 local builds use `.env.test.template` (regtest + localhost Electrum). For release builds, `react-native-dotenv` reads `.env.production`, so that file must also be overwritten with the local config. See the `bitkit-build` skill for full instructions. + +### Repro Steps + +#### Step 1: RN wallet setup (v1.1.6) + +1. Install the local RN build on simulator/emulator +2. Create a new wallet +3. Fund the wallet on-chain: + ```bash + cd docker + ./bitcoin-cli send 0.01
+ ./bitcoin-cli mine 6 + ``` +4. In the app, connect to the local LND node. Go to **Settings > Advanced > Channels > + > Fund Custom > Manual** and enter: + - Node ID: `02cfdfd683aca2561621870fe50ab9ef2d0c887b3729ce6797ff68fde6f044feb9` + - Host: `0.0.0.0` + - Port: `9735` +5. Set amount (e.g. 50,000 sats) and confirm the channel open +6. Mine blocks to confirm: + ```bash + ./bitcoin-cli mine 6 + ``` +7. Wait for the channel to show as active in the app + +#### Step 2: Open channel from LND to the app + +The app's channel gives the app outbound liquidity, but LND needs outbound to pay the app. While the app is still connected as a peer, open a channel from LND's side: + +```bash +# Get the app's node ID from Settings > Advanced > Lightning Node Info +./bitcoin-cli openchannel 500000 +``` + +This funds LND, opens a 500k sat channel, mines 6 blocks, and waits for it to become active. + +Verify LND can pay the app: +```bash +# Grab an invoice from the app's receive screen and pay it +./bitcoin-cli payinvoice 10 +``` + +#### Step 3: Migrate to native (v2.0.6 iOS / v2.0.3 Android) + +1. Install the local native build **over** the RN app (upgrade, not clean install) +2. The RN → native migration runs automatically +3. Wait for the wallet to fully sync and the Lightning node to start +4. **Re-connect to LND**: the custom peer connection is lost after migration (see [#435](https://github.com/synonymdev/bitkit-ios/issues/435)). Paste the LND URI again in the app: + ``` + 02cfdfd683aca2561621870fe50ab9ef2d0c887b3729ce6797ff68fde6f044feb9@0.0.0.0:9735 + ``` +5. Verify the channel is active and LND is connected as a peer + +#### Step 4: Make 30 Lightning payments on native + +**Automated (recommended):** + +Make sure Appium is running and the peer is connected, then: + +```bash +# iOS +PAYMENT_COUNT=30 SIMULATOR_OS_VERSION=26.0 npx wdio wdio.no-install.conf.ts + +# Android +PAYMENT_COUNT=30 PLATFORM=android npx wdio wdio.no-install.conf.ts +``` + +**Manual alternative:** + +Grab invoices from the app's receive screen and pay via LND: +```bash +./bitcoin-cli payinvoice 10 +``` + +> **Note**: The peer connection can drop between app restarts. Before running the test, verify LND has an active peer. If not, re-paste the connection URI in the app. + +#### Step 5: Upgrade to v2.1.0 + +1. Install v2.1.0 **over** the native app (upgrade) +2. Launch the app + +#### Result: NOT REPRODUCED + +After 30 payments on v2.0.6 with a 3rd-party LND channel, upgrading to v2.1.0 did **not** trigger the ChannelMonitor desync crash. The node started successfully. + +Possible explanations: +- The bug may only affect Blocktank/LSP channels where the channel was opened via the "transfer to spending" flow +- The RN backup server may store channel monitors differently for custom vs LSP channels +- The stale monitor injection path may only match on specific channel metadata (e.g. LSP-related fields) + +--- + ## Key Details -- **21 payments on native is the minimum** that reliably reproduces. Each payment generates ~3-5 update_id increments. LDK can recover small gaps (~10 updates) by replaying counterparty commitment updates, so fewer payments may not trigger the fatal crash. +- **21 payments on native is the minimum** that reliably reproduces (Case 1). Each payment generates ~3-5 update_id increments. LDK can recover small gaps (~10 updates) by replaying counterparty commitment updates, so fewer payments may not trigger the fatal crash. - **RN payments don't need to be many** — just enough to establish the channel and create the RN backup. - The bug is in `fetchOrphanedChannelMonitorsIfNeeded` in `WalletViewModel.swift` (iOS) / `WalletViewModel.kt` (Android). It unconditionally injects old RN monitors without checking compatibility with the current ChannelManager. @@ -118,7 +234,7 @@ Failed to start wallet To verify the fix (e.g. `release-2.1.1` or `fix/channel-monitor-stale-data-v2`): -1. Reproduce the bug using steps 1-4 above on v2.1.0 +1. Reproduce the bug using Case 1 steps on v2.1.0 2. Confirm the node fails to start 3. Install the fix build **over** the broken v2.1.0 4. Launch the app @@ -132,5 +248,6 @@ To verify the fix (e.g. `release-2.1.1` or `fix/channel-monitor-stale-data-v2`): |------|---------| | `test/specs/receive-ln-payments.e2e.ts` | Automated spec to receive N Lightning payments | | `wdio.no-install.conf.ts` | WDIO config that attaches to existing app (no reinstall) | +| `docker/bitcoin-cli` | Local docker CLI with `openchannel`, `payinvoice`, `mine`, `send` commands | | `scripts/pay-lightning-address.sh` | Shell script to pay BOLT11/BIP21/LN address via Blocktank | | `scripts/pay-lightning-address-loop.sh` | Shell script to send N payments to a Lightning address | diff --git a/test/helpers/constants.ts b/test/helpers/constants.ts index db7246f..728bec8 100644 --- a/test/helpers/constants.ts +++ b/test/helpers/constants.ts @@ -49,12 +49,16 @@ export const blocktankURL = export type LndConfig = { server: string; + restHost: string; + restPort: number; tls: string; macaroonPath: string; }; export const lndConfig: LndConfig = { server: 'localhost:10009', + restHost: 'localhost', + restPort: 8080, tls: `${__dirname}/../../docker/lnd/tls.cert`, macaroonPath: `${__dirname}/../../docker/lnd/data/chain/bitcoin/regtest/admin.macaroon`, }; diff --git a/test/helpers/regtest.ts b/test/helpers/regtest.ts index 7e2b4d0..0e9bc5b 100644 --- a/test/helpers/regtest.ts +++ b/test/helpers/regtest.ts @@ -7,8 +7,10 @@ * Default is 'local' for backwards compatibility with existing tests. */ +import * as fs from 'node:fs'; +import * as https from 'node:https'; import BitcoinJsonRpc from 'bitcoin-json-rpc'; -import { bitcoinURL, blocktankURL, getBackend, type Backend } from './constants'; +import { bitcoinURL, blocktankURL, lndConfig, getBackend, type Backend } from './constants'; export { getBackend, type Backend }; @@ -47,6 +49,65 @@ async function localMineBlocks(count: number): Promise { console.info(`→ [local] Mined ${count} block(s)`); } +function lndRestRequest( + path: string, + body: Record, +): Promise> { + const tlsCert = fs.readFileSync(lndConfig.tls); + const macaroon = fs.readFileSync(lndConfig.macaroonPath).toString('hex'); + const payload = JSON.stringify(body); + + return new Promise((resolve, reject) => { + const req = https.request( + { + hostname: lndConfig.restHost, + port: lndConfig.restPort, + path, + method: 'POST', + ca: tlsCert, + rejectUnauthorized: false, + headers: { + 'Grpc-Metadata-macaroon': macaroon, + 'Content-Type': 'application/json', + 'Content-Length': Buffer.byteLength(payload), + }, + }, + (res) => { + let data = ''; + res.on('data', (chunk: Buffer) => { data += chunk.toString(); }); + res.on('end', () => { + if (res.statusCode && res.statusCode >= 200 && res.statusCode < 300) { + resolve(JSON.parse(data) as Record); + } else { + reject(new Error(`LND REST ${res.statusCode}: ${data}`)); + } + }); + }, + ); + req.on('error', reject); + req.write(payload); + req.end(); + }); +} + +async function localPayInvoice(invoice: string, amountSat?: number): Promise { + const body: Record = { payment_request: invoice }; + if (amountSat !== undefined) { + body.amt = amountSat; + } + + console.info(`→ [local] Paying invoice via LND REST...`); + const result = await lndRestRequest('/v1/channels/transactions', body); + + if (result.payment_error && result.payment_error !== '') { + throw new Error(`LND payment error: ${result.payment_error}`); + } + + const paymentHash = (result.payment_hash as string) ?? 'unknown'; + console.info(`→ [local] Payment hash: ${paymentHash}`); + return paymentHash; +} + // Blocktank backend (regtest API over HTTPS) async function blocktankDeposit(address: string, amountSat?: number): Promise { @@ -249,16 +310,20 @@ export async function mineBlocks(count: number = 1): Promise { /** * Pays a Lightning invoice on regtest. - * Only available with Blocktank backend (regtest). + * - BACKEND=local: uses the local LND node via REST API + * - BACKEND=regtest: uses Blocktank API * * @param invoice - The BOLT11 invoice to pay * @param amountSat - Amount in satoshis (optional, for amount-less invoices) - * @returns The payment ID + * @returns The payment ID / payment hash */ export async function payInvoice(invoice: string, amountSat?: number): Promise { const backend = getBackend(); - if (backend !== 'regtest') { - throw new Error('payInvoice() is only available with BACKEND=regtest (Blocktank API)'); + if (backend === 'local') { + return localPayInvoice(invoice, amountSat); + } + if (backend === 'regtest') { + return blocktankPayInvoice(invoice, amountSat); } - return blocktankPayInvoice(invoice, amountSat); + throw new Error('payInvoice() is not available with BACKEND=mainnet'); } From d79c281e872ce8da21259f66074ba134eeadb7c3 Mon Sep 17 00:00:00 2001 From: Piotr Stachyra Date: Fri, 20 Mar 2026 15:53:28 +0100 Subject: [PATCH 4/7] update repro doc --- docs/repro-channel-monitor-desync.md | 262 ++++++++++++--------------- 1 file changed, 119 insertions(+), 143 deletions(-) diff --git a/docs/repro-channel-monitor-desync.md b/docs/repro-channel-monitor-desync.md index f6b19e0..58a9803 100644 --- a/docs/repro-channel-monitor-desync.md +++ b/docs/repro-channel-monitor-desync.md @@ -1,9 +1,13 @@ -# Repro: ChannelMonitor Desync (Stale RN Monitor Injection) +# ChannelMonitor Desync: Repro, Recovery & Test Plan Related issues: - [#847 (bitkit-android)](https://github.com/synonymdev/bitkit-android/issues/847) - iOS support ticket (user logs from 2026-03-18) +Fix branches: +- **iOS**: `fix/stale-monitor-recovery-release` +- **Android**: `fix/stale-monitor-recovery-v2` + ## Summary Build 182 (v2.1.0) introduced `fetchOrphanedChannelMonitorsIfNeeded` which fetches old channel monitors from the RN remote backup on every startup (when `isChannelRecoveryChecked` is false). If the wallet was migrated from RN to native and used on native for enough LN payments, the RN backup monitor is stale. Injecting it causes a fatal ChannelManager/ChannelMonitor update_id mismatch and LDK refuses to start. @@ -17,84 +21,7 @@ On v2.1.0 startup: 4. LDK loads ChannelManager (advanced) against stale ChannelMonitor → fatal mismatch 5. Node refuses to start: `"A ChannelMonitor is stale compared to the current ChannelManager!"` ---- - -## Case 1: Blocktank channel (staging regtest) - -Reproduces the bug using a Blocktank LSP channel opened via "transfer to spending". - -### Prerequisites - -- **Bitkit v1.1.6** (React Native) iOS or Android build — for the initial RN wallet setup -- **Bitkit v2.0.6** (native iOS) or **v2.0.3** (native Android) build — intermediate native version -- **Bitkit v2.1.0 / build 182** (native) iOS or Android build — the version with the bug -- **Staging regtest** Blocktank API access (`BACKEND=regtest`) -- **Appium** running locally for the automated payment step -- **bitkit-e2e-tests** repo checked out - -### Repro Steps - -#### Step 1: RN wallet setup (v1.1.6) - -1. Install v1.1.6 (RN) on simulator/emulator -2. Create a new wallet -3. Fund the wallet on-chain (use `./scripts/fund-address.sh
`) -4. Mine blocks to confirm -5. Open a Lightning channel (transfer to spending) -6. Wait for the channel to be ready -7. Make **1 Lightning payment** (to confirm the channel works and create RN backup data) - -#### Step 2: Migrate to native (v2.0.6 iOS / v2.0.3 Android) - -1. Install the native build **over** the RN app (upgrade, not clean install) -2. The RN → native migration runs automatically -3. Wait for the wallet to fully sync and the Lightning node to start -4. Verify the channel is open and working - -#### Step 3: Make 21+ Lightning payments on native - -This advances the ChannelManager in VSS past the frozen RN backup monitor. - -**Automated (recommended):** - -Make sure Appium is running, then: - -```bash -# iOS -SIMULATOR_OS_VERSION=26.0 BACKEND=regtest npx wdio wdio.no-install.conf.ts - -# Android -PLATFORM=android BACKEND=regtest npx wdio wdio.no-install.conf.ts -``` - -This runs `test/specs/receive-ln-payments.e2e.ts` which: -- Attaches to the already-installed app (no reinstall) -- Loops 21 times: tap Receive → grab invoice from QR → pay via Blocktank → acknowledge -- Default: 21 payments of 10 sats each - -Customize with env vars: -```bash -PAYMENT_COUNT=25 PAYMENT_AMOUNT=10 SIMULATOR_OS_VERSION=26.0 BACKEND=regtest npx wdio wdio.no-install.conf.ts -``` - -**Manual alternative:** - -Use the shell script to pay invoices grabbed from the app UI: -```bash -# Pay a BIP21 URI copied from the app's receive screen -./scripts/pay-lightning-address.sh 'bitcoin:bcrt1q...?lightning=lnbcrt1...' 10 -``` - -Repeat 21+ times with fresh invoices each time. - -#### Step 4: Upgrade to v2.1.0 - -1. Install v2.1.0 / build 182 **over** the v2.0.6 app (upgrade) -2. Launch the app - -#### Expected Result - -The app fails to start the Lightning node with: +## Error Signature ``` A ChannelMonitor is stale compared to the current ChannelManager! @@ -103,7 +30,7 @@ but the ChannelManager is at update_id . Failed to read channel manager from store: Value would be dangerous to continue execution with ``` -In app logs, look for: +In app logs: ``` Running pre-startup channel monitor recovery check Found 1 monitors on RN backup for pre-startup recovery @@ -116,27 +43,60 @@ Failed to start wallet --- -## Case 2: 3rd-party channel (local docker) — NOT REPRODUCED +## Repro: Blocktank channel (staging regtest) -Attempted to reproduce the bug using a 3rd-party channel (local docker LND) instead of Blocktank. This tests whether the bug also affects non-LSP channels. **The bug did not reproduce with this setup**, even after 50 payments. +Reproduces the bug using a Blocktank LSP channel opened via "transfer to spending". + +### Prerequisites + +- **Bitkit v1.1.6** (React Native) iOS or Android build +- **Bitkit v2.0.6** (native iOS) or **v2.0.3** (native Android) build +- **Bitkit v2.1.0** (native) iOS or Android build — the buggy version +- **Staging regtest** Blocktank API access (`BACKEND=regtest`) +- **Appium** running locally for the automated payment step + +### Steps + +1. Install v1.1.6 (RN), create wallet, fund on-chain, open Lightning channel (transfer to spending), make 1 LN payment +2. Install v2.0.6/v2.0.3 (native) **over** RN app — migration runs automatically +3. Make 21+ Lightning payments on native: + ```bash + # iOS + SIMULATOR_OS_VERSION=26.0 BACKEND=regtest npx wdio wdio.no-install.conf.ts + # Android + PLATFORM=android BACKEND=regtest npx wdio wdio.no-install.conf.ts + ``` +4. Install v2.1.0 **over** native app → app fails to start LN node (see error signature above) + +--- + +## Repro: 3rd-party channel (local docker) + +Reproduces the bug using a manually opened channel to the local docker LND node. ### Prerequisites - **bitkit-docker** running locally (Bitcoin, Electrum, LND, backup server) -- Local regtest builds for each version: - - `bitkit_rn_local_v1.1.6.apk` / `bitkit_rn_local_ios_v1.1.6.app` - - Native v2.0.6 (iOS) / v2.0.3 (Android) built with `BACKEND=local` - - Native v2.1.0 built with `BACKEND=local` +- Local regtest builds for each version (see Build Notes below) - **Appium** running locally for the automated payment step -- **bitkit-e2e-tests** repo checked out ### Build Notes -RN v1.1.6 local builds use `.env.test.template` (regtest + localhost Electrum). For release builds, `react-native-dotenv` reads `.env.production`, so that file must also be overwritten with the local config. See the `bitkit-build` skill for full instructions. +RN v1.1.6 local builds use `.env.test.template` (regtest + localhost Electrum). For release builds, `react-native-dotenv` reads `.env.production`, so that file must be overwritten with the local config. -### Repro Steps +**Critical**: The RN app's `.env.production` must point the backup server to **staging** (not localhost), because the native apps have `rnBackupServerHost` hardcoded to staging. If the RN app pushes to `127.0.0.1:3003` but the native app queries `bitkit.stag0.blocktank.to`, it will never find the channel monitors and the bug won't trigger. -#### Step 1: RN wallet setup (v1.1.6) +In `.env.production` for the RN v1.1.6 build, set: +``` +BACKUPS_SERVER_HOST=https://bitkit.stag0.blocktank.to/backups-ldk +BACKUPS_SERVER_PUBKEY=02c03b8b8c1b5500b622646867d99bf91676fac0f38e2182c91a9ff0d053a21d6d +``` + +All other settings (Electrum, network, etc.) stay local. + +### Steps + +#### 1. RN wallet setup (v1.1.6) 1. Install the local RN build on simulator/emulator 2. Create a new wallet @@ -146,101 +106,117 @@ RN v1.1.6 local builds use `.env.test.template` (regtest + localhost Electrum). ./bitcoin-cli send 0.01
./bitcoin-cli mine 6 ``` -4. In the app, connect to the local LND node. Go to **Settings > Advanced > Channels > + > Fund Custom > Manual** and enter: +4. In the app, go to **Settings > Advanced > Channels > + > Fund Custom > Manual** and enter the local LND connection: - Node ID: `02cfdfd683aca2561621870fe50ab9ef2d0c887b3729ce6797ff68fde6f044feb9` - Host: `0.0.0.0` - Port: `9735` 5. Set amount (e.g. 50,000 sats) and confirm the channel open -6. Mine blocks to confirm: - ```bash - ./bitcoin-cli mine 6 - ``` -7. Wait for the channel to show as active in the app +6. Mine blocks: `./bitcoin-cli mine 6` +7. Wait for the channel to be active -#### Step 2: Open channel from LND to the app +#### 2. Open channel from LND to the app -The app's channel gives the app outbound liquidity, but LND needs outbound to pay the app. While the app is still connected as a peer, open a channel from LND's side: +The app's channel has all balance on the app side. LND needs outbound liquidity to pay invoices to the app: ```bash -# Get the app's node ID from Settings > Advanced > Lightning Node Info ./bitcoin-cli openchannel 500000 ``` -This funds LND, opens a 500k sat channel, mines 6 blocks, and waits for it to become active. - -Verify LND can pay the app: +Verify with a test payment: ```bash -# Grab an invoice from the app's receive screen and pay it ./bitcoin-cli payinvoice 10 ``` -#### Step 3: Migrate to native (v2.0.6 iOS / v2.0.3 Android) +#### 3. Migrate to native (v2.0.6 iOS / v2.0.3 Android) -1. Install the local native build **over** the RN app (upgrade, not clean install) -2. The RN → native migration runs automatically -3. Wait for the wallet to fully sync and the Lightning node to start -4. **Re-connect to LND**: the custom peer connection is lost after migration (see [#435](https://github.com/synonymdev/bitkit-ios/issues/435)). Paste the LND URI again in the app: +1. Install native build **over** RN app (upgrade, not clean install) +2. Wait for migration and sync +3. **Re-connect to LND** — custom peer connection is lost after migration ([#435](https://github.com/synonymdev/bitkit-ios/issues/435)). Paste the URI in the app: ``` 02cfdfd683aca2561621870fe50ab9ef2d0c887b3729ce6797ff68fde6f044feb9@0.0.0.0:9735 ``` -5. Verify the channel is active and LND is connected as a peer - -#### Step 4: Make 30 Lightning payments on native - -**Automated (recommended):** -Make sure Appium is running and the peer is connected, then: +#### 4. Make 30 Lightning payments on native ```bash # iOS PAYMENT_COUNT=30 SIMULATOR_OS_VERSION=26.0 npx wdio wdio.no-install.conf.ts - # Android PAYMENT_COUNT=30 PLATFORM=android npx wdio wdio.no-install.conf.ts ``` -**Manual alternative:** +> **Note**: The peer connection drops on app restarts. Re-paste the LND URI if needed before running the test. -Grab invoices from the app's receive screen and pay via LND: -```bash -./bitcoin-cli payinvoice 10 -``` +#### 5. Upgrade to v2.1.0 + +Install v2.1.0 **over** the native app → app fails to start LN node (see error signature above). -> **Note**: The peer connection can drop between app restarts. Before running the test, verify LND has an active peer. If not, re-paste the connection URI in the app. +--- -#### Step 5: Upgrade to v2.1.0 +## Recovery: Upgrade to v2.1.2 -1. Install v2.1.0 **over** the native app (upgrade) -2. Launch the app +Upgrading from a broken v2.1.0 wallet to v2.1.2 (fix candidate) recovers the wallet. Channels are healed and LN transactions work after recovery. -#### Result: NOT REPRODUCED +Fix branches: +- **iOS**: `fix/stale-monitor-recovery-release` +- **Android**: `fix/stale-monitor-recovery-v2` -After 30 payments on v2.0.6 with a 3rd-party LND channel, upgrading to v2.1.0 did **not** trigger the ChannelMonitor desync crash. The node started successfully. +### Steps -Possible explanations: -- The bug may only affect Blocktank/LSP channels where the channel was opened via the "transfer to spending" flow -- The RN backup server may store channel monitors differently for custom vs LSP channels -- The stale monitor injection path may only match on specific channel metadata (e.g. LSP-related fields) +1. Start with a broken v2.1.0 wallet (reproduced via either case above) +2. Install v2.1.2 **over** v2.1.0 +3. Launch the app +4. Verify: node starts, channels are active, LN payments work + +> **Note (3rd-party channels)**: After recovery, 3rd-party channels may be force-closed. Final behavior for non-LSP channels after recovery is still being evaluated. --- -## Key Details +## Test Plan -- **21 payments on native is the minimum** that reliably reproduces (Case 1). Each payment generates ~3-5 update_id increments. LDK can recover small gaps (~10 updates) by replaying counterparty commitment updates, so fewer payments may not trigger the fatal crash. -- **RN payments don't need to be many** — just enough to establish the channel and create the RN backup. -- The bug is in `fetchOrphanedChannelMonitorsIfNeeded` in `WalletViewModel.swift` (iOS) / `WalletViewModel.kt` (Android). It unconditionally injects old RN monitors without checking compatibility with the current ChannelManager. +Matrix of upgrade/recovery scenarios to validate v2.1.2. Each scenario should be tested for both channel types where marked. + +### Blocktank channel (staging regtest) -## Fix Verification +| # | Scenario | Result | +|---|----------|--------| +| B1 | v2.0.6 (wallet with 21+ payment gap) → v2.1.0 → confirm broken | Reproduces | +| B2 | Restore broken v2.1.0 wallet into v2.1.2 (clean install + restore) | ✅ Recovered | +| B3 | Update broken v2.1.0 wallet to v2.1.2 (in-place upgrade) | ✅ Recovered | +| B4 | v2.0.6 (wallet with gap) → v2.1.2 (skip v2.1.0) | ✅ Recovered | +| B5 | v2.0.6 (wallet with gap) → v2.1.1 → v2.1.2 | ✅ Recovered | +| B6 | v2.1.0 healthy wallet (no gap) → v2.1.2 (regression check) | ✅ No issues | +| B7 | v2.1.0 broken wallet + 600 blocks mined → v2.1.2 (stale chain state) | ✅ Recovered | -To verify the fix (e.g. `release-2.1.1` or `fix/channel-monitor-stale-data-v2`): +### 3rd-party channel (local docker) -1. Reproduce the bug using Case 1 steps on v2.1.0 -2. Confirm the node fails to start -3. Install the fix build **over** the broken v2.1.0 -4. Launch the app -5. Check logs — the node should either: - - Start successfully (fix prevents stale monitor injection), or - - Handle the already-corrupted state gracefully (fix in ldk-node) +| # | Scenario | Result | +|---|----------|--------| +| T1 | v2.0.6 (wallet with 30+ payment gap) → v2.1.0 → confirm broken | Reproduces | +| T2 | Update broken v2.1.0 wallet to v2.1.2 (in-place upgrade) | ✅ Recovered | +| T3 | v2.0.6 (wallet with gap) → v2.1.2 (skip v2.1.0) | ✅ Recovered | +| T4 | v2.1.0 healthy wallet (no gap) → v2.1.2 (regression check) | ✅ No issues | + +> T2-T4: After recovery, 3rd-party channels may be force-closed. Verify wallet is operational and on-chain balance is intact even if LN channels close. + +### Version reference + +| Version | iOS branch | Android branch | +|---------|-----------|---------------| +| v1.1.6 | tag `v1.1.6` (RN) | tag `v1.1.6` (RN) | +| v2.0.6 | `chore/e2e-updater-url` | — | +| v2.0.3 | — | `chore/e2e-updater-url` | +| v2.1.0 | build 182 | build 182 | +| v2.1.2 (fix) | `fix/stale-monitor-recovery-release` | `fix/stale-monitor-recovery-v2` | + +--- + +## Key Details + +- **21 payments on native is the minimum** for Blocktank channels. Each payment generates ~3-5 update_id increments. LDK can recover small gaps (~10 updates) by replaying counterparty commitment updates. +- **RN payments don't need to be many** — just enough to establish the channel and create the RN backup. +- The bug is in `fetchOrphanedChannelMonitorsIfNeeded` in `WalletViewModel.swift` (iOS) / `WalletViewModel.kt` (Android). It unconditionally injects old RN monitors without checking compatibility with the current ChannelManager. +- **RN backup server mismatch**: The RN app's backup server is configurable via `.env`, but the native apps hardcode `rnBackupServerHost` to staging. For local docker repro, the RN build must push to the same staging server the native apps query. ## Files From 6df385a4767f1c1ac3059f7b754774f2db8788f7 Mon Sep 17 00:00:00 2001 From: Piotr Stachyra Date: Fri, 20 Mar 2026 16:01:39 +0100 Subject: [PATCH 5/7] test and post-recovery notes --- docs/repro-channel-monitor-desync.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/repro-channel-monitor-desync.md b/docs/repro-channel-monitor-desync.md index 58a9803..20c9fd2 100644 --- a/docs/repro-channel-monitor-desync.md +++ b/docs/repro-channel-monitor-desync.md @@ -168,7 +168,9 @@ Fix branches: 3. Launch the app 4. Verify: node starts, channels are active, LN payments work -> **Note (3rd-party channels)**: After recovery, 3rd-party channels may be force-closed. Final behavior for non-LSP channels after recovery is still being evaluated. +### Post-recovery channel closure + +Whether healed channels should be closed after recovery is under discussion. For testing: verify wallet is operational after recovery regardless of channel closure outcome. On-chain balance should be intact even if healed channels are subsequently closed. --- @@ -196,8 +198,7 @@ Matrix of upgrade/recovery scenarios to validate v2.1.2. Each scenario should be | T2 | Update broken v2.1.0 wallet to v2.1.2 (in-place upgrade) | ✅ Recovered | | T3 | v2.0.6 (wallet with gap) → v2.1.2 (skip v2.1.0) | ✅ Recovered | | T4 | v2.1.0 healthy wallet (no gap) → v2.1.2 (regression check) | ✅ No issues | - -> T2-T4: After recovery, 3rd-party channels may be force-closed. Verify wallet is operational and on-chain balance is intact even if LN channels close. +| T5 | v2.1.0 broken wallet + 600 blocks mined → v2.1.2 (stale chain state) | ✅ Recovered | ### Version reference From 7b0b71211d434122418fff1773527e1e4176ae08 Mon Sep 17 00:00:00 2001 From: Piotr Stachyra Date: Fri, 20 Mar 2026 16:06:56 +0100 Subject: [PATCH 6/7] minor updates --- docs/repro-channel-monitor-desync.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/docs/repro-channel-monitor-desync.md b/docs/repro-channel-monitor-desync.md index 20c9fd2..2c7b3b6 100644 --- a/docs/repro-channel-monitor-desync.md +++ b/docs/repro-channel-monitor-desync.md @@ -43,7 +43,7 @@ Failed to start wallet --- -## Repro: Blocktank channel (staging regtest) +## Repro Case #1: Blocktank channel (staging regtest) Reproduces the bug using a Blocktank LSP channel opened via "transfer to spending". @@ -58,7 +58,7 @@ Reproduces the bug using a Blocktank LSP channel opened via "transfer to spendin ### Steps 1. Install v1.1.6 (RN), create wallet, fund on-chain, open Lightning channel (transfer to spending), make 1 LN payment -2. Install v2.0.6/v2.0.3 (native) **over** RN app — migration runs automatically +2. Install v2.0.6 (iOS) or v2.0.3 (Android) **over** RN app — migration runs automatically 3. Make 21+ Lightning payments on native: ```bash # iOS @@ -70,7 +70,7 @@ Reproduces the bug using a Blocktank LSP channel opened via "transfer to spendin --- -## Repro: 3rd-party channel (local docker) +## Repro Case #2: 3rd-party channel (local docker) Reproduces the bug using a manually opened channel to the local docker LND node. @@ -82,7 +82,7 @@ Reproduces the bug using a manually opened channel to the local docker LND node. ### Build Notes -RN v1.1.6 local builds use `.env.test.template` (regtest + localhost Electrum). For release builds, `react-native-dotenv` reads `.env.production`, so that file must be overwritten with the local config. +RN v1.1.6 local builds use `.env.test.template` (regtest + localhost Electrum). For release builds, `react-native-dotenv` reads `.env.production`, so that file must be overwritten with the local regtest config. **Critical**: The RN app's `.env.production` must point the backup server to **staging** (not localhost), because the native apps have `rnBackupServerHost` hardcoded to staging. If the RN app pushes to `127.0.0.1:3003` but the native app queries `bitkit.stag0.blocktank.to`, it will never find the channel monitors and the bug won't trigger. @@ -106,8 +106,8 @@ All other settings (Electrum, network, etc.) stay local. ./bitcoin-cli send 0.01
./bitcoin-cli mine 6 ``` -4. In the app, go to **Settings > Advanced > Channels > + > Fund Custom > Manual** and enter the local LND connection: - - Node ID: `02cfdfd683aca2561621870fe50ab9ef2d0c887b3729ce6797ff68fde6f044feb9` +4. In the app, go to **Settings > Advanced > Channels > + > Fund Custom > Manual** and enter the local LND connection (get the node ID from `./bitcoin-cli getinfo`): + - Node ID: LND's pubkey - Host: `0.0.0.0` - Port: `9735` 5. Set amount (e.g. 50,000 sats) and confirm the channel open @@ -116,7 +116,7 @@ All other settings (Electrum, network, etc.) stay local. #### 2. Open channel from LND to the app -The app's channel has all balance on the app side. LND needs outbound liquidity to pay invoices to the app: +The app's channel has all balance on the app side. LND needs outbound liquidity to pay invoices to the app. Get the app's node ID from **Settings > Advanced > Lightning Node Info**, then: ```bash ./bitcoin-cli openchannel 500000 @@ -185,7 +185,7 @@ Matrix of upgrade/recovery scenarios to validate v2.1.2. Each scenario should be | B1 | v2.0.6 (wallet with 21+ payment gap) → v2.1.0 → confirm broken | Reproduces | | B2 | Restore broken v2.1.0 wallet into v2.1.2 (clean install + restore) | ✅ Recovered | | B3 | Update broken v2.1.0 wallet to v2.1.2 (in-place upgrade) | ✅ Recovered | -| B4 | v2.0.6 (wallet with gap) → v2.1.2 (skip v2.1.0) | ✅ Recovered | +| B4 | v2.0.6 (wallet with gap) → v2.1.2 (skip v2.1.0) | ✅ No issues | | B5 | v2.0.6 (wallet with gap) → v2.1.1 → v2.1.2 | ✅ Recovered | | B6 | v2.1.0 healthy wallet (no gap) → v2.1.2 (regression check) | ✅ No issues | | B7 | v2.1.0 broken wallet + 600 blocks mined → v2.1.2 (stale chain state) | ✅ Recovered | @@ -196,7 +196,7 @@ Matrix of upgrade/recovery scenarios to validate v2.1.2. Each scenario should be |---|----------|--------| | T1 | v2.0.6 (wallet with 30+ payment gap) → v2.1.0 → confirm broken | Reproduces | | T2 | Update broken v2.1.0 wallet to v2.1.2 (in-place upgrade) | ✅ Recovered | -| T3 | v2.0.6 (wallet with gap) → v2.1.2 (skip v2.1.0) | ✅ Recovered | +| T3 | v2.0.6 (wallet with gap) → v2.1.2 (skip v2.1.0) | ✅ No issues | | T4 | v2.1.0 healthy wallet (no gap) → v2.1.2 (regression check) | ✅ No issues | | T5 | v2.1.0 broken wallet + 600 blocks mined → v2.1.2 (stale chain state) | ✅ Recovered | From ff0e3957e716a5648bd4e0dc4816e780a542006f Mon Sep 17 00:00:00 2001 From: Piotr Stachyra Date: Fri, 20 Mar 2026 16:16:43 +0100 Subject: [PATCH 7/7] Remove the reconnection workaround --- docs/repro-channel-monitor-desync.md | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/docs/repro-channel-monitor-desync.md b/docs/repro-channel-monitor-desync.md index 2c7b3b6..3aa0ecf 100644 --- a/docs/repro-channel-monitor-desync.md +++ b/docs/repro-channel-monitor-desync.md @@ -131,10 +131,7 @@ Verify with a test payment: 1. Install native build **over** RN app (upgrade, not clean install) 2. Wait for migration and sync -3. **Re-connect to LND** — custom peer connection is lost after migration ([#435](https://github.com/synonymdev/bitkit-ios/issues/435)). Paste the URI in the app: - ``` - 02cfdfd683aca2561621870fe50ab9ef2d0c887b3729ce6797ff68fde6f044feb9@0.0.0.0:9735 - ``` +3. Verify the channel is active and LND is connected as a peer #### 4. Make 30 Lightning payments on native