feat(plugins): add Replicator plugin to mirror external data#250
Draft
ThaiTrevor wants to merge 2 commits into
Draft
feat(plugins): add Replicator plugin to mirror external data#250ThaiTrevor wants to merge 2 commits into
ThaiTrevor wants to merge 2 commits into
Conversation
…nal store Adds a new StarbasePlugin under plugins/replicator that pulls rows from a configured external database (Postgres, MySQL, D1, Turso, StarbaseDB) into StarbaseDB's internal SQLite store using a per-table watermark column. Exposes POST /replicator/sync (admin-only) and a public sync() method that can be wired into the Cron plugin or any external scheduler. Closes outerbase#72
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Closes #72.
Adds a new
ReplicatorPluginunderplugins/replicator/that pulls rows from an external database (Postgres, MySQL, Cloudflare D1, Turso, or another StarbaseDB) into the StarbaseDB internal SQLite store. Each replication pass is driven by a per-table watermark column (e.g.updated_ator a monotonicid) so only rows that changed since the previous run are transferred.What changed
plugins/replicator/index.ts— newReplicatorPluginextendingStarbasePlugin. Createstmp_replication_state(table_name, last_value, last_synced_at)on registration, exposessync()andPOST /replicator/sync(admin-only), and upserts rows viaINSERT ... ON CONFLICT(primaryKey) DO UPDATE.idcolumn no longer falls into the lexicographic trap (e.g."99" > "100"). String compare is used otherwise, which still handles ISO timestamps correctly.name,watermarkColumn,primaryKey,destTable) are validated against[A-Za-z_][A-Za-z0-9_]*at construction time and quoted in the externalSELECTusing dialect-appropriate quoting (backticks for MySQL, double quotes elsewhere) — consistent with the existing double-quoted destination-side identifiers.plugins/replicator/index.test.ts— vitest suite covering constructor validation, identifier validation, state-table creation, initial pull, watermark-bounded pulls, dest-table override, MySQL dialect quoting, and numeric-watermark ordering.plugins/replicator/README.md— usage, configuration, Cron-plugin scheduling snippet, destination-table DDL template, and a note that large backfills require repeatedsync()calls because each call is bounded bybatchSize.plugins/replicator/meta.json— registry metadata to match the other plugins.How it works
tmp_replication_state(table_name, last_value, last_synced_at).sync()call reads the stored watermark per table, runsSELECT * FROM "<table>" WHERE "<watermarkColumn>" > ? ORDER BY "<watermarkColumn>" ASC LIMIT <batchSize>against the external source, and upserts each row into the internal store usingON CONFLICT(<primaryKey>) DO UPDATE.last_value.Scheduling is delegated to the existing Cron plugin — the README shows the snippet.
Tasks
Verify
npx vitest run plugins/replicator/index.test.ts— 11/11 passing.src/rls/index.test.tsfrom a fullnpx vitest runare pre-existing onmainand unrelated to this change (confirmed by running the RLS suite onmain).Before
plugins/replicator/*.