fix(export): stream and paginate /export/dump for large databases#249
Draft
ThaiTrevor wants to merge 2 commits into
Draft
fix(export): stream and paginate /export/dump for large databases#249ThaiTrevor wants to merge 2 commits into
ThaiTrevor wants to merge 2 commits into
Conversation
…bases Resolves OOM/timeout failures when dumping large databases by switching the /export/dump response to a ReadableStream and paginating per-table SELECTs with LIMIT/OFFSET instead of loading the entire result set into memory at once. Closes outerbase#59
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Fixes #59 —
/export/dumppreviously failed on large databases because it loaded every row of every table into a single in-memory string before responding. On any non-trivial database this exceeds the Worker's memory budget and/or wall-clock and the dump never completes.Changes
src/export/dump.ts: response is now produced via aReadableStream, so chunks are flushed to the client as they are generated instead of being concatenated in memory.SELECT * FROM <table> LIMIT 1000 OFFSET <n>and the loop stops as soon as a page returns fewer rows than the page size. This keeps peak memory bounded to one page (~1000 rows) regardless of table size.NULL/undefinedare now emitted as the SQLNULLkeyword (previously they were stringified to the literal textnull, which only parsed correctly by accident).LIMIT/OFFSETqueries when a table is larger than the page sizeNULLvalues as theNULLkeywordThe public route, headers, and dump format are unchanged — this is a drop-in fix.
Tasks
SELECT *in one shotVerify
npx vitest run src/export/→ 25 passed (4 files)npx vitest run src/export/dump.test.ts→ 7 passed (5 original + 2 new)npx tsc --noEmitintroduces no new errors insrc/export/dump.ts(only pre-existing errors in unrelated files remain).Closes #59