Skip to content

fix(export): stream and paginate /export/dump for large databases#249

Draft
ThaiTrevor wants to merge 2 commits into
outerbase:mainfrom
ThaiTrevor:fix/issue-59-starbasedb-database-dumps-do-not
Draft

fix(export): stream and paginate /export/dump for large databases#249
ThaiTrevor wants to merge 2 commits into
outerbase:mainfrom
ThaiTrevor:fix/issue-59-starbasedb-database-dumps-do-not

Conversation

@ThaiTrevor
Copy link
Copy Markdown

🤖 AI-assisted contribution — This PR was drafted with AI assistance and reviewed by a native Vietnamese speaker before submission. Placeholders, terminology, and file format were validated automatically. I will respond to review feedback. Happy to revise or close if not a fit.


Purpose

Fixes #59/export/dump previously failed on large databases because it loaded every row of every table into a single in-memory string before responding. On any non-trivial database this exceeds the Worker's memory budget and/or wall-clock and the dump never completes.

Changes

  • src/export/dump.ts: response is now produced via a ReadableStream, so chunks are flushed to the client as they are generated instead of being concatenated in memory.
  • Per-table data is paged with SELECT * FROM <table> LIMIT 1000 OFFSET <n> and the loop stops as soon as a page returns fewer rows than the page size. This keeps peak memory bounded to one page (~1000 rows) regardless of table size.
  • Small correctness improvement in value serialization: NULL/undefined are now emitted as the SQL NULL keyword (previously they were stringified to the literal text null, which only parsed correctly by accident).
  • All existing test cases continue to pass unchanged; added two new tests:
    • paginates across multiple LIMIT/OFFSET queries when a table is larger than the page size
    • serializes NULL values as the NULL keyword

The public route, headers, and dump format are unchanged — this is a drop-in fix.

Tasks

  • Stream the dump response instead of buffering it
  • Page through table rows instead of SELECT * in one shot
  • Preserve existing dump format and test expectations
  • Add tests for pagination and NULL handling

Verify

  • npx vitest run src/export/ → 25 passed (4 files)
  • npx vitest run src/export/dump.test.ts → 7 passed (5 original + 2 new)
  • npx tsc --noEmit introduces no new errors in src/export/dump.ts (only pre-existing errors in unrelated files remain).

Closes #59

…bases

Resolves OOM/timeout failures when dumping large databases by switching
the /export/dump response to a ReadableStream and paginating per-table
SELECTs with LIMIT/OFFSET instead of loading the entire result set into
memory at once.

Closes outerbase#59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Database dumps do not work on large databases

1 participant