-
Notifications
You must be signed in to change notification settings - Fork 1
Open
Description
Currently, the entire database lives in memory and is persisted to a single file on disk. Consider making each registered id be persisted to a separate file. By loading/unloading the file (containing the data from a single id) on user request(s) we will likely make the memory footprint a lot smaller.
Pros:
- Smaller, more manageable files.
- Data from each
idis isolated making them easier to move around and share - Data is loaded/saved based on ID, reducing how
- Its harder to corrupt the entire data set just because of one unclean shutdown.
Cons:
- Although we already persist to disk with every request, PyXIE will be doing a lot more reads from disk, making things slower
- The
DATABASE_FILEconfiguration would change to something likeDATA_PATHwhich some may consider a breaking change, so a major version bump would be recommended. - A data migration process from the single file database to multiple a multi-file database would need to be defined.
This ticket may lend itself well to #25 where each thread that PyXIE runs would receive a request from a queue, then loads the correct file for the id, and processes the request. Unclear how well that strategy would work with Flask, however.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels