Improve ZIP store performance and filesystem listing#70
Open
kulvait wants to merge 2 commits intozarr-developers:mainfrom
Open
Improve ZIP store performance and filesystem listing#70kulvait wants to merge 2 commits intozarr-developers:mainfrom
kulvait wants to merge 2 commits intozarr-developers:mainfrom
Conversation
- Replace stream-based ZIP access with random-access `ZipFile` for faster and more efficient reads - Optimize ZIP store initialization by reading the ZIP index only, avoiding full stream traversal - Introduce improved caching of directory structure and file sizes using synchronized maps - Simplify internal logic and remove unnecessary dependencies - Improve filesystem listing performance Details: - Removed dependency on Apache Commons ZipArchiveInputStream and related classes in ReadOnlyZipStore - Added efficient caching using maps for directories and file sizes, async friendly - Normalized entry names for consistent lookup - Reduced redundant computations (e.g. cached entry sizes) - Simplified stream handling and chunk calculation These changes significantly improve performance, reduce complexity, and make the ZIP store implementation more maintainable.
… and directory existence
- Problem:
* Some tests failed because ReadOnlyZipStore could not locate ZIP entries when the store
was created by simply zipping a directory. Tools differ: some produce entry names with
a leading slash, others without. This caused getInputStream(), read(), and getSize() to return null.
* testExists() failed for root keys because exists() included directories, but by design it should only test file existence.
- Solution:
* Added resolvePathWithLeadingSlashFromKeys() to try a secondary lookup with a leading slash.
Primary lookup uses the standard key without leading slash; secondary is only for compatibility.
* Modified exists(String[] keys) to check only fileSizeIndex, ignoring directories, to match the intended design.
- Effect:
* ReadOnlyZipStoreTest passes regardless of whether ZIP entries have leading slashes or not.
* Test logic now clearly distinguishes between file existence and directory entries.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation for this work was that on large zip stores, the indexing was very slow and then opening arrays as well.
So I did replaced Apache Commons ZipArchiveInputStream by random access java class.
Change in FilesystemStore.java is more cosmetic but also slightly increase performance.
ZipFilefor faster and more efficient readsDetails:
These changes significantly improve performance, reduce complexity, and make the ZIP store implementation more maintainable.