Skip to content

Conversation

@albe
Copy link
Owner

@albe albe commented Feb 9, 2021

This exchanges the \n (newline) document separator with the sequence \x00\x00\x1E\n, which is distinguishing enough to be searchable in the partition. Hence, this will allow checking for unfinished writes and search the last valid document and even scan through the partition backwards.
For optimization of that use case, the document data size is stored again before the document separator.

Resolves #124
Related to #31 #107

@coveralls
Copy link

coveralls commented Feb 9, 2021

Coverage Status

Coverage increased (+0.2%) to 97.426% when pulling e1f5f5e on record-separator into d9c28c0 on master.

@albe
Copy link
Owner Author

albe commented Feb 9, 2021

t.b.d. - should the dataSize be added to the "footer", so that scanning backwards is more efficient?

@ghost
Copy link

ghost commented Feb 22, 2021

DeepCode's analysis on #e1f5f5 found:

  • ℹ️ 1 minor issue. 👇

Top issues

Description Example fixes
Performing subtraction on a number (from number literal 0) and a boolean (from boolean literal false) will trigger an implicit coercion of the operands to number. If this is intended, consider making the coercion explicit. Occurrences: 🔧 Example fixes

👉 View analysis in DeepCode’s Dashboard | Configure the bot

@albe albe changed the title Separate documents by a unique sequence Separate documents by a unique sequence and make partition backwards scannable Feb 22, 2021
@albe albe merged commit 703a128 into master Feb 22, 2021
@albe albe deleted the record-separator branch February 22, 2021 01:16
@albe albe mentioned this pull request Feb 22, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Reconsider making partitions backward-scannable

3 participants