Skip to content

Sharing of representative datasets via compressed Zarr zips #2570

Merged
VeckoTheGecko merged 29 commits intoParcels-code:mainfrom
VeckoTheGecko:test-midlayer
Apr 9, 2026
Merged

Sharing of representative datasets via compressed Zarr zips #2570
VeckoTheGecko merged 29 commits intoParcels-code:mainfrom
VeckoTheGecko:test-midlayer

Conversation

@VeckoTheGecko
Copy link
Copy Markdown
Contributor

@VeckoTheGecko VeckoTheGecko commented Apr 8, 2026

I realised that there is both a better approach to #2510, as well as a way that we can pave the way for Parcels-code/parcels-benchmarks#40 ("Future work: better testing").

This PR adds tooling and documentation that allows users to easily share datasets via the issue tracker. These datasets will be first trimmed of redundant array data, then shared as Zipped zarr files using zarr.storage.ZipStore.

This allows people to easily share their datasets - including the mesh itself - directly via GitHub (which has an upload limit of 25Mb for zips). Working with the MOI grid data in parcels-benchmarks the resulting dataset was only 26Mb (i.e., the glamf and glamu).

This PR also removes the old JSON approach as it is no longer needed.

AI Disclosure

  • This PR contains AI-generated content.
    • I have tested any AI-generated content in my PR.
    • I take responsibility for any AI-generated content in my PR.
    • Describe how you used it (e.g., by pasting your prompt): Used it to update the docs (pointing it to the old docs, then to an example usecase) then made manual copy edits.

@VeckoTheGecko
Copy link
Copy Markdown
Contributor Author

Docs:

image

Copy link
Copy Markdown
Member

@erikvansebille erikvansebille left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks very good, a few comments below

@VeckoTheGecko VeckoTheGecko enabled auto-merge (squash) April 9, 2026 10:44
@VeckoTheGecko VeckoTheGecko merged commit 0645442 into Parcels-code:main Apr 9, 2026
11 of 12 checks passed
@github-project-automation github-project-automation bot moved this from Backlog to Done in Parcels development Apr 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants