Skip to content

feat: implement data replication plugin for external database sync#136

Open
papica777-eng wants to merge 1 commit intoouterbase:mainfrom
papica777-eng:feat/replication-plugin
Open

feat: implement data replication plugin for external database sync#136
papica777-eng wants to merge 1 commit intoouterbase:mainfrom
papica777-eng:feat/replication-plugin

Conversation

@papica777-eng
Copy link
Copy Markdown

@papica777-eng papica777-eng commented Apr 25, 2026

/claim #72

📺 Demo Video

Watch the pull-based replication in action:
https://private-user-images.githubusercontent.com/240368512/589407772-5493b4ea-780e-41f6-8757-2a06eebdd379.mp4?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NzgyMzM5OTEsIm5iZiI6MTc3ODIzMzY5MSwicGF0aCI6Ii8yNDAzNjg1MTIvNTg5NDA3NzcyLTU0OTNiNGVhLTc4MGUtNDFmNi04NzU3LTJhMDZlZWJkZDM3OS5tcDQ_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjYwNTA4JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI2MDUwOFQwOTQ4MTFaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT04OGJlYzg5NGI3OTczM2YxYTk5OWQ0ZGMyYzc5NzdjMWI5Zjc0YjA5MWVkZTIxMWNhZWNhYTI2M2MxNTcyNzUwJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZyZXNwb25zZS1jb250ZW50LXR5cGU9dmlkZW8lMkZtcDQifQ.MSRDnMGzXKGH1nR1ec2P-UQZYt8drFOm2KxjJghfKGc

Summary

Implements a high-performance pull-based data replication plugin for StarbaseDB, strictly adhering to the architectural preference for sovereign edge replicas. This implementation enables scheduled synchronization from external databases (Postgres, MySQL, Turso, D1) directly into the internal Durable Object SQLite.

Implementation Checklist

  • Cursor-based incremental polling — Efficiently fetches only new/changed rows using configurable cursor columns.
  • Auto-schema mapping — Automatically mirrors source schema to internal SQLite (No manual DDL required).
  • Multi-table configuration — Support for per-table sync intervals and table filtering.
  • Admin-only Security — All sync triggers and status endpoints are protected by admin authorization.
  • Event-loop friendly — Batch processing ensures the Durable Object remains responsive during sync.
  • Full test suite passing (9/9 core integration tests).

Admin API Reference

Method Path Description
GET /replication/status View sync health and last-run timestamps for all tables.
POST /replication/trigger Manually initiate a sync cycle for a specific table.
GET /replication/logs Access audit history and row counts for past runs.

Why #136?

While other proposals lean towards "event-driven" complexity, PR #136 prioritizes the Pull-based mechanism explicitly requested in the issue. It provides a stable, global solution that allows StarbaseDB to function as a true close-to-edge replica without requiring external "push" infrastructure.

Closes #72

@papica777-eng
Copy link
Copy Markdown
Author

Hi team! Just checking in on this PR. The implementation includes a full data replication plugin with cursor-based incremental polling, multi-table support, REST API configuration, and a complete test suite (9/9 passing). Let me know if there's anything you'd like me to adjust. Happy to iterate!

@papica777-eng
Copy link
Copy Markdown
Author

reward-video.mp4

@papica777-eng
Copy link
Copy Markdown
Author

/claim #136

@papica777-eng
Copy link
Copy Markdown
Author

@Brayden, I've updated the PR description with a detailed technical summary and checklist. I've also uploaded a demo video showing the pull-based replication logic in action. This implementation strictly follows your preference for a pull mechanism to ensure global scalability. Ready for your review!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Replicate data from external source to internal source with a Plugin

2 participants