Skip to content

feat(plugins): external replication plugin (pull external -> internal) #72#265

Open
FishEnjoyer2025 wants to merge 1 commit into
outerbase:mainfrom
FishEnjoyer2025:feat/external-replication-plugin
Open

feat(plugins): external replication plugin (pull external -> internal) #72#265
FishEnjoyer2025 wants to merge 1 commit into
outerbase:mainfrom
FishEnjoyer2025:feat/external-replication-plugin

Conversation

@FishEnjoyer2025

Copy link
Copy Markdown

Implements pull-based external→internal replication as a plugin, per #72.

What it does

ExternalReplicationPlugin pulls configured tables from the instance's external
data source (e.g. a Supabase Postgres) into the internal Durable-Object SQLite, so
the instance can serve as a close-to-edge read replica that's queried locally instead of
round-tripping to the external database.

  • Pull-based — no changes required on the external source (matches the design note
    in the issue: a pull mechanism is a better global solution than per-provider push).
  • Per-table config, with optional incremental sync via a cursorColumn
    (updated_at, id, …); the last value is persisted in _starbasedb_replication_state
    so each run only pulls new/changed rows. Omit it to re-pull the whole table.
  • Idempotent INSERT OR REPLACE upserts → safe to retry.
  • Batched (batchSize, default 5000) so very large tables drain across runs.
  • Endpoints: POST /replicate (all tables) and POST /replicate/:table, admin-only.
  • Interval scheduling via a Cloudflare Cron Trigger (README includes a
    wrangler.toml example + scheduled() usage).

Tests

8 unit tests (vitest) — passing — covering:

  • SELECT generation: full snapshot, incremental first-run (no WHERE), incremental with a
    saved cursor (WHERE + param), resume-from-saved-cursor, and identifier quoting.
  • INSERT-OR-REPLACE upsert generation.
  • The full replicateTable flow with mocked execution: pull → upsert → advance + persist
    cursor; and the empty-result path.

Files

  • plugins/external-replication/index.ts — the plugin
  • plugins/external-replication/index.test.ts — tests
  • plugins/external-replication/README.md — usage + cron setup

/claim #72

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant