Skip to content

Stream large database exports#237

Open
anshumanjaiswal wants to merge 1 commit into
outerbase:mainfrom
anshumanjaiswal:issue-59-pull-driven-streaming-exports
Open

Stream large database exports#237
anshumanjaiswal wants to merge 1 commit into
outerbase:mainfrom
anshumanjaiswal:issue-59-pull-driven-streaming-exports

Conversation

@anshumanjaiswal
Copy link
Copy Markdown

Purpose

Closes #59 by streaming internal database exports in bounded pages instead of loading the entire SQL dump, JSON export, or CSV export into memory before responding.

Tasks

  • Add pull-driven text streaming helper for exports
  • Page table reads with a bounded, configurable batch size
  • Stream SQL dump, JSON, and CSV exports
  • Preserve CSV/SQL escaping and quote table identifiers safely
  • Add export tests for streaming, pagination, empty tables, and escaping

Verify

  • npm test -- --run src/export
  • npm run build

Before

Export routes constructed the full export body in memory before returning a response.

After

Export routes return a ReadableStream and fetch rows page-by-page as the client consumes the download.

@anshumanjaiswal
Copy link
Copy Markdown
Author

/claim #59

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Database dumps do not work on large databases

1 participant