wip: HTTP VFS for read only access to a datadir on a web server

[samwillis]

Export a database from PGlite with dumpDataDir, with this is now includes an index.json as a file listing. Untar to a web server, use the HttpFs vfs, and point it at the dir on the server:

const pg = await PGlite.create({
  fs: new HttpFs("http://localhost/pglite/examples/pgdata", { 
    fetchGranularity: 'file', // 'file' or 'page'
  }),
});

fetchGranularity is self explanatory, when set to "file" it will download the hole file when it is first read, "page" downloads individual file pages using a http range header.

Has support for:

• Browser: via sync XMLHttpRequest with xhr.responseType = 'arraybuffer', this is only available in a web worker
• Node: via node workers, it uses a shared array buffer and Atomics to do the request in the worker while blocking the thread where PGlite is running. It currently boots a new worker for each request, we should change this to reuse the worker.

Demo site: https://pglite-httpfs-demo.netlify.app

Screen.Recording.2024-10-03.at.12.25.13.mp4

[github-actions]

Built bundles:

• Postgres WASM: https://github.com/electric-sql/pglite/actions/runs/11242011515/artifacts/2031177659
• Interim build files: https://github.com/electric-sql/pglite/actions/runs/11242011515/artifacts/2031177775
• PGlite: https://github.com/electric-sql/pglite/actions/runs/11242011515/artifacts/2031206571
• Demos: https://github.com/electric-sql/pglite/actions/runs/11242011515/artifacts/2031221546

[github-actions]

🚀 Deployed on https://6705845cda92212e1566b5e9--pglite.netlify.app

[thruflo]

Could this work naturally with S3 / an S3 compatible file store?

[samwillis]

@thruflo yes, for a read path it's trivial. For a write path it would be possible to also make work. But still single connection.

Although this is "read only", it actually maintains an "overlay" of writes on each file. So it's really a snapshot loaded over http, with in-memory writes. It just doesn't write back to the server.

[nestarz]

Can we imagine a read/write FS to S3-like storage and in an env where workers (multi-thread) is not allowed ?