Skip to content

WIP - Persistence#1230

Closed
samwillis wants to merge 145 commits intomainfrom
cursor/persistence-plan-design-doc-f6d0
Closed

WIP - Persistence#1230
samwillis wants to merge 145 commits intomainfrom
cursor/persistence-plan-design-doc-f6d0

Conversation

@samwillis
Copy link
Collaborator

@samwillis samwillis commented Feb 10, 2026

🎯 Changes

Implements #865 (comment) from #865

✅ Checklist

  • I have followed the steps in the Contributing guide.
  • I have tested this code locally with pnpm test:pr.

🚀 Release Impact

  • This change affects published code, and I have generated a changeset.
  • This change is docs/CI/dev-only (no release).

Open in Cursor Open in Web

@cursor
Copy link

cursor bot commented Feb 10, 2026

Cursor Agent can help with this pull request. Just @cursor in comments and I'll start working on changes in this branch.
Learn more about Cursor Agents

@changeset-bot
Copy link

changeset-bot bot commented Feb 10, 2026

⚠️ No Changeset found

Latest commit: 9311f4e

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@pkg-pr-new
Copy link

pkg-pr-new bot commented Feb 10, 2026

More templates

@tanstack/angular-db

npm i https://pkg.pr.new/TanStack/db/@tanstack/angular-db@1230

@tanstack/db

npm i https://pkg.pr.new/TanStack/db/@tanstack/db@1230

@tanstack/db-browser-wa-sqlite-persisted-collection

npm i https://pkg.pr.new/TanStack/db/@tanstack/db-browser-wa-sqlite-persisted-collection@1230

@tanstack/db-cloudflare-do-sqlite-persisted-collection

npm i https://pkg.pr.new/TanStack/db/@tanstack/db-cloudflare-do-sqlite-persisted-collection@1230

@tanstack/db-electron-sqlite-persisted-collection

npm i https://pkg.pr.new/TanStack/db/@tanstack/db-electron-sqlite-persisted-collection@1230

@tanstack/db-ivm

npm i https://pkg.pr.new/TanStack/db/@tanstack/db-ivm@1230

@tanstack/db-node-sqlite-persisted-collection

npm i https://pkg.pr.new/TanStack/db/@tanstack/db-node-sqlite-persisted-collection@1230

@tanstack/db-react-native-sqlite-persisted-collection

npm i https://pkg.pr.new/TanStack/db/@tanstack/db-react-native-sqlite-persisted-collection@1230

@tanstack/db-sqlite-persisted-collection-core

npm i https://pkg.pr.new/TanStack/db/@tanstack/db-sqlite-persisted-collection-core@1230

@tanstack/electric-db-collection

npm i https://pkg.pr.new/TanStack/db/@tanstack/electric-db-collection@1230

@tanstack/offline-transactions

npm i https://pkg.pr.new/TanStack/db/@tanstack/offline-transactions@1230

@tanstack/powersync-db-collection

npm i https://pkg.pr.new/TanStack/db/@tanstack/powersync-db-collection@1230

@tanstack/query-db-collection

npm i https://pkg.pr.new/TanStack/db/@tanstack/query-db-collection@1230

@tanstack/react-db

npm i https://pkg.pr.new/TanStack/db/@tanstack/react-db@1230

@tanstack/rxdb-db-collection

npm i https://pkg.pr.new/TanStack/db/@tanstack/rxdb-db-collection@1230

@tanstack/solid-db

npm i https://pkg.pr.new/TanStack/db/@tanstack/solid-db@1230

@tanstack/svelte-db

npm i https://pkg.pr.new/TanStack/db/@tanstack/svelte-db@1230

@tanstack/trailbase-db-collection

npm i https://pkg.pr.new/TanStack/db/@tanstack/trailbase-db-collection@1230

@tanstack/vue-db

npm i https://pkg.pr.new/TanStack/db/@tanstack/vue-db@1230

commit: c17981d

@github-actions
Copy link
Contributor

github-actions bot commented Feb 10, 2026

Size Change: +1.92 kB (+2.06%)

Total Size: 95.1 kB

Filename Size Change
./packages/db/dist/esm/collection/events.js 434 B +46 B (+11.86%) ⚠️
./packages/db/dist/esm/collection/index.js 3.56 kB +236 B (+7.1%) 🔍
./packages/db/dist/esm/collection/indexes.js 2.35 kB +1.25 kB (+113.57%) 🆘
./packages/db/dist/esm/index.js 2.8 kB +74 B (+2.72%)
./packages/db/dist/esm/indexes/lazy-index.js 1.24 kB +135 B (+12.24%) ⚠️
./packages/db/dist/esm/query/compiler/evaluators.js 1.62 kB +189 B (+13.23%) ⚠️
./packages/db/dist/esm/query/subset-dedupe.js 921 B -6 B (-0.65%)
ℹ️ View Unchanged
Filename Size
./packages/db/dist/esm/collection/change-events.js 1.39 kB
./packages/db/dist/esm/collection/changes.js 1.22 kB
./packages/db/dist/esm/collection/lifecycle.js 1.75 kB
./packages/db/dist/esm/collection/mutations.js 2.34 kB
./packages/db/dist/esm/collection/state.js 3.49 kB
./packages/db/dist/esm/collection/subscription.js 3.71 kB
./packages/db/dist/esm/collection/sync.js 2.41 kB
./packages/db/dist/esm/deferred.js 207 B
./packages/db/dist/esm/errors.js 4.83 kB
./packages/db/dist/esm/event-emitter.js 748 B
./packages/db/dist/esm/indexes/auto-index.js 742 B
./packages/db/dist/esm/indexes/base-index.js 766 B
./packages/db/dist/esm/indexes/btree-index.js 2.17 kB
./packages/db/dist/esm/indexes/reverse-index.js 538 B
./packages/db/dist/esm/local-only.js 808 B
./packages/db/dist/esm/local-storage.js 2.1 kB
./packages/db/dist/esm/optimistic-action.js 359 B
./packages/db/dist/esm/paced-mutations.js 496 B
./packages/db/dist/esm/proxy.js 3.75 kB
./packages/db/dist/esm/query/builder/functions.js 733 B
./packages/db/dist/esm/query/builder/index.js 4.1 kB
./packages/db/dist/esm/query/builder/ref-proxy.js 1.05 kB
./packages/db/dist/esm/query/compiler/expressions.js 430 B
./packages/db/dist/esm/query/compiler/group-by.js 2.23 kB
./packages/db/dist/esm/query/compiler/index.js 2.05 kB
./packages/db/dist/esm/query/compiler/joins.js 2.11 kB
./packages/db/dist/esm/query/compiler/order-by.js 1.45 kB
./packages/db/dist/esm/query/compiler/select.js 1.09 kB
./packages/db/dist/esm/query/expression-helpers.js 1.43 kB
./packages/db/dist/esm/query/ir.js 673 B
./packages/db/dist/esm/query/live-query-collection.js 360 B
./packages/db/dist/esm/query/live/collection-config-builder.js 5.55 kB
./packages/db/dist/esm/query/live/collection-registry.js 264 B
./packages/db/dist/esm/query/live/collection-subscriber.js 2.42 kB
./packages/db/dist/esm/query/live/internal.js 145 B
./packages/db/dist/esm/query/optimizer.js 2.62 kB
./packages/db/dist/esm/query/predicate-utils.js 2.97 kB
./packages/db/dist/esm/query/query-once.js 359 B
./packages/db/dist/esm/scheduler.js 1.3 kB
./packages/db/dist/esm/SortedMap.js 1.3 kB
./packages/db/dist/esm/strategies/debounceStrategy.js 247 B
./packages/db/dist/esm/strategies/queueStrategy.js 428 B
./packages/db/dist/esm/strategies/throttleStrategy.js 246 B
./packages/db/dist/esm/transactions.js 2.9 kB
./packages/db/dist/esm/utils.js 924 B
./packages/db/dist/esm/utils/browser-polyfills.js 304 B
./packages/db/dist/esm/utils/btree.js 5.61 kB
./packages/db/dist/esm/utils/comparison.js 952 B
./packages/db/dist/esm/utils/cursor.js 457 B
./packages/db/dist/esm/utils/index-optimization.js 1.51 kB
./packages/db/dist/esm/utils/type-guards.js 157 B

compressed-size-action::db-package-size

@github-actions
Copy link
Contributor

github-actions bot commented Feb 10, 2026

Size Change: 0 B

Total Size: 3.85 kB

ℹ️ View Unchanged
Filename Size
./packages/react-db/dist/esm/index.js 225 B
./packages/react-db/dist/esm/useLiveInfiniteQuery.js 1.32 kB
./packages/react-db/dist/esm/useLiveQuery.js 1.34 kB
./packages/react-db/dist/esm/useLiveSuspenseQuery.js 559 B
./packages/react-db/dist/esm/usePacedMutations.js 401 B

compressed-size-action::react-db-package-size

@samwillis samwillis changed the title Persistence plan design doc WIP - Persistence Feb 10, 2026
@cursor cursor bot force-pushed the cursor/persistence-plan-design-doc-f6d0 branch from 238391b to 283fc63 Compare February 10, 2026 09:30
@solarsoft0
Copy link

, not sure if this is the right place to share this, but I was able to get it working on my end.
but In local-only persisted mode, the stream position resets after restart, causing valid mutations to be skipped as duplicates and leading to silent data loss. I implemented a temporary workaround, and aside from this issue, I haven’t noticed any other problems so far. thank you @samwillis.

@samwillis
Copy link
Collaborator Author

@solarsoft0 thats awesome!
I'm hoping to get back to this next week (or maybe the week after). Very happy to hear it's so close.
Which platform/storage layer did you use?

@solarsoft0
Copy link

@solarsoft0 thats awesome! I'm hoping to get back to this next week (or maybe the week after). Very happy to hear it's so close. Which platform/storage layer did you use?

, I created one for Tauri, and I think my fix confirms it is not a storage layer issue.

@kevin-dp
Copy link
Contributor

kevin-dp commented Mar 4, 2026

, not sure if this is the right place to share this, but I was able to get it working on my end. but In local-only persisted mode, the stream position resets after restart, causing valid mutations to be skipped as duplicates and leading to silent data loss. I implemented a temporary workaround, and aside from this issue, I haven’t noticed any other problems so far. thank you @samwillis.

Hi @solarsoft0, i can confirm this issue in local-only mode as i'm hitting the same problem. Working on a fix for it.

kevin-dp and others added 10 commits March 4, 2026 11:41
…cate tx skipping

The PersistedCollectionRuntime never restored its stream position from
the database on startup, always beginning at localTerm=1, localSeq=0.
After a page reload, the first new mutation would collide with a
previously applied transaction (term=1, seq=1), causing the SQLite
adapter's applyCommittedTx to silently skip it as a duplicate.

Add getStreamPosition to PersistenceAdapter (optional) and implement it
in SQLiteCorePersistenceAdapter. Call it from startInternal() so that
observeStreamPosition seeds the local counters before any mutations.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Include full row values in the tx:committed message so receiving tabs
can apply changes directly without a SQLite round-trip via loadRowsByKeys.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ti-tab support

Web Locks for per-collection leadership election, BroadcastChannel for
cross-tab RPC transport, DB writer lock for SQLite write serialization,
envelope dedup for exactly-once mutations, and leader heartbeats.
Includes 15 unit tests with Web Locks/BroadcastChannel mocks.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@kevin-dp
Copy link
Contributor

kevin-dp commented Mar 5, 2026

Multi-tab sync bug fixes

Three bugs were identified and fixed that caused multi-tab sync to silently break. Together they caused follower tab mutations to disappear from the leader tab's in-memory collection, and eventually caused sync to stop working in both directions.


Bug 1: Leadership state stuck after setup error — 5163087

What: state.isLeader = true was set before getStreamPosition() in acquireLeadership. If getStreamPosition threw (e.g. due to a UNIQUE constraint failed: collection_registry.tombstone_table_name from React StrictMode double-mounting), isLeader remained permanently stuck at true because the finally block that resets it was inside an inner try/finally that was never entered.

Fix: Wrap the entire lock callback body in a single try/finally. Set state.isLeader = true only after successful setup. The finally block always executes and resets isLeader = false + cleans up the heartbeat timer.


Bug 2: Seq collision between leader direct path and coordinator — 533be70

What: The leader tab had two mutation paths: a "direct" path (write to SQLite and broadcast) and a coordinator RPC path. Only follower tabs used the RPC path — the leader bypassed the coordinator entirely.

This caused a seq collision: the leader's direct writes incremented the runtime's localSeq (e.g. to 1, 2, 3) but left the coordinator's state.latestSeq at 0. When a follower later sent an RPC, the coordinator assigned seq starting from 1 again, producing duplicate seq numbers. The leader then skipped these "already-seen" tx:committed messages via txCommitted.seq <= this.latestSeq, causing follower mutations to silently disappear.

Fix: Always route through requestApplyLocalMutations when available, regardless of leader/follower status. This keeps the coordinator's seq counter in sync with all writes. Also removes the stub requestApplyLocalMutations from SingleProcessCoordinator — it returned success without persisting, so single-process mode correctly falls back to the direct path.


Bug 3: Leader ignoring coordinator-delivered tx:committed for follower mutations — 49720f6

What: The runtime's onCoordinatorMessage handler skipped ALL messages where senderId matched the coordinator's own node ID. But when the coordinator processes a follower's RPC in handleApplyLocalMutations, it delivers the resulting tx:committed to local subscribers using the coordinator's own senderId. This caused the leader's runtime to ignore follower mutations — they were written to SQLite but never applied to the leader's in-memory collection.

Fix: Allow tx:committed messages from self to pass through the filter. The seq dedup logic in processCommittedTxUnsafe prevents double-processing of the leader's own mutations: observeStreamPosition is called with the response's term/seq before the local delivery runs under the mutex, so the duplicate is detected via txCommitted.seq <= this.latestSeq. Other message types (heartbeats, resets) from self are still skipped.

kevin-dp and others added 7 commits March 5, 2026 13:25
Previously, `state.isLeader = true` was set before the setup code that
calls `getStreamPosition()`. If `getStreamPosition` threw (e.g. due to
a UNIQUE constraint violation from React StrictMode double-mounting),
`isLeader` remained permanently stuck at `true` because the `finally`
block that resets it was inside an inner try/finally that was never
entered.

Fix: Wrap the entire lock callback body in a single try/finally. Set
`state.isLeader = true` only after successful setup (stream position
restore and term increment). The finally block always runs and resets
`isLeader = false` + cleans up the heartbeat timer.

Also refactors the coordinator to support lazy adapter wiring via
`setAdapter()`, allowing `createBrowserWASQLitePersistence` to inject
the adapter after construction. This enables the demo to construct the
coordinator without requiring the adapter upfront.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…nt seq collisions

The leader tab had two mutation paths: a "direct" path (write to SQLite
and broadcast) and an RPC path (through the coordinator). Previously,
only follower tabs used the RPC path — the leader bypassed the
coordinator and wrote directly.

This caused a seq collision: the leader's direct writes incremented the
runtime's `localSeq` but left the coordinator's `state.latestSeq` at 0.
When a follower later sent an RPC, the coordinator assigned seq starting
from 1 again, producing duplicate seq numbers. The leader then skipped
these "already-seen" tx:committed messages, causing follower mutations
to silently disappear.

Fix: Always route through `requestApplyLocalMutations` when available,
regardless of leader/follower status. This keeps the coordinator's seq
counter in sync with all writes.

Also removes `requestApplyLocalMutations` from `SingleProcessCoordinator`
— it was a stub that returned success without persisting, which would
break now that the leader uses this path. Single-process mode correctly
falls back to the direct path since it has no multi-tab coordination.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…mmitted messages

The leader tab's `onCoordinatorMessage` handler skipped ALL messages
where `senderId` matched the coordinator's own node ID. But when the
coordinator processes a follower's RPC in `handleApplyLocalMutations`,
it delivers the resulting `tx:committed` to local subscribers using the
coordinator's own `senderId`. This caused the leader's runtime to
silently ignore follower mutations — they were written to SQLite but
never applied to the leader's in-memory collection.

Fix: Allow `tx:committed` messages from self to pass through the filter.
The seq dedup logic in `processCommittedTxUnsafe` already prevents
double-processing: when the leader's own mutations go through the
coordinator, `observeStreamPosition` is called with the response's
term/seq before the local `tx:committed` delivery runs under the mutex,
so the duplicate is detected via `txCommitted.seq <= this.latestSeq`.
Other message types (heartbeats, resets) from self are still skipped.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…riteMessage

When queryCollectionOptions detects a server-side deletion, it sends
{ type: 'delete', value: oldItem } through the sync. The persistence
layer only checked for 'key' in message to detect deletes, causing
value-based deletes to be misclassified as updates. Also use optional
chaining for process.versions in React Native where process exists but
versions may not.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ss-window sync

Add ElectronCollectionCoordinator using BroadcastChannel + Web Locks for
leader election and cross-window coordination in Electron renderer windows.
Wire coordinator into renderer persistence via setAdapter(), add
getStreamPosition to the IPC protocol, and export from package index.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@kevin-dp kevin-dp mentioned this pull request Mar 12, 2026
5 tasks
@kevin-dp
Copy link
Contributor

Superseded by #1358

@kevin-dp kevin-dp closed this Mar 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Persistence of synced data

4 participants