Skip to content

Conversation

@IvanBorislavovDimitrov
Copy link
Contributor

@IvanBorislavovDimitrov IvanBorislavovDimitrov commented Nov 13, 2024

The core logic of reading the zip streaming is changed:

  1. Start reading the archive stream (entry by entry)
  2. Get the data offset of the entry (apache compress supports it out of the box)
  3. Read the stream until -1 is returned, this means the entry is read
  4. Zip input stream from apache compress stores the "compressed" entry size so the end index of the entry is data offset + compressed size (there is a method) -> it is only calculated correctly when the zip stream has returned -1 (the zip entry is read). It gets the compressed size from the internal inflater.
  5. Store information about the compression method of the entry
  6. If the entry's compression method is 8 this means it must be inflated (it is DEFLATED), so for this type of entries create a new Inflater(true) ->true means no wrap (it will inflate from the exact indexes of the entry). If the entry's compression method is 0 this means it is STORED, in this case just read the bytes and store them (you have the entry)
  7. Now you have stored the data offset and the end offset for each entry. When you need a specific entry, just pass the start and end index to clouds, and it will return the stream only for this entry, which again should STORED / INFLATED depending on the compression method

@IvanBorislavovDimitrov IvanBorislavovDimitrov force-pushed the zip-entry-offsets branch 3 times, most recently from 07e0a08 to df0b97b Compare November 13, 2024 08:26
theghost5800
theghost5800 previously approved these changes Nov 21, 2024
Yavor16
Yavor16 previously approved these changes Nov 21, 2024
@sonarqubecloud
Copy link

Quality Gate Failed Quality Gate failed

Failed conditions
68.6% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube Cloud

@IvanBorislavovDimitrov
Copy link
Contributor Author

Coverage is lower because of the backwards compatibility logic and SQL providers that only throws exceptions

@IvanBorislavovDimitrov IvanBorislavovDimitrov merged commit b60a0a7 into master Nov 22, 2024
4 of 5 checks passed
@IvanBorislavovDimitrov IvanBorislavovDimitrov deleted the zip-entry-offsets branch November 22, 2024 07:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants