-
Notifications
You must be signed in to change notification settings - Fork 7
Fix metadata encoding issue #115
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #115 +/- ##
==========================================
- Coverage 97.32% 96.80% -0.53%
==========================================
Files 6 7 +1
Lines 337 375 +38
Branches 56 60 +4
==========================================
+ Hits 328 363 +35
- Misses 7 8 +1
- Partials 2 4 +2
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
13e427d to
fb17432
Compare
|
I think we need a bit more on nailing down what the blast radius is here. What happens to existing files, written with current versions of tszip? Was this an write-time bug (fails when trying to write non-ascii strings) or a read-time bug (fails when trying to read) or a silent data corruption issue (you get back garbage with no error raised)? Non-ascii strings are probably rare in practice but we do need more details here. |
9a11916 to
69b6745
Compare
|
@jeromekelleher Good point - I've made it load the old boken files and made a more detailed changelog. |
|
These examples are quite porky, adding 6MB to the repo. Do they need to be this big? The CHANGELOG message needs a bit more work, still not entirely clear to me the consequences of this bug |
69b6745 to
a7f0cfb
Compare
|
Reading that changelog today it looks like the ramblings of a madman, sorry! I've cleaned it up and trimmed the test file to 50KB. |
Fixes #95
Metadata and strings were being stored in
zarrasint8. This was fine forjsonas it uses ascii codes below 128. However forstructthe bytes can take the full 0-255 range.Files written with old
tszipare not readable without changing their encoding on disk.