ASV

[fluidnumerics-joe]

After a good bit of thought and wrestling with our setup, I've come back around and I'd like to propose switching over to ASV. The main benefit here is in that it will handle cleanly dealing with parcels versions and dashboarding. The downside is that it is not file-system aware, but I think I have a way around this.

In addition to our existing benchmarks, I'd like to add benchmarks that focus specifically on just reading the benchmark data (no compute / no particle execution). This way, when someone runs the benchmark suite, those benchmarks that are just on reading data provide context for how someone's file system (be it disk/network storage, etc.) performs independent of the compute operations.

@VeckoTheGecko @erikvansebille - curious to hear your thoughts.

[VeckoTheGecko]

More a note than a thought: ASV supports Rattler as an environment (https://asv.readthedocs.io/en/stable/using.html#environments) - Rattler is the 'pixi way' of building packages

[fluidnumerics-joe]

More a note than a thought: ASV supports Rattler as an environment (https://asv.readthedocs.io/en/stable/using.html#environments) - Rattler is the 'pixi way' of building packages

I tried setting the environment to rattler and it failed,

(parcels-benchmarks) (base) [joe@claymation parcels-benchmarks]$ asv run
· Unknown environment type 'rattler'. Allowed values based on existing plugins are ['existing', 'conda', 'virtualenv']. 
· No environments selected

I must be missing something

[fluidnumerics-joe]

After installing py-rattler in the pixi environment, using rattler, now shows the same error as reported here : scipy/scipy#22392 (review)

[VeckoTheGecko]

Yes, I was just looking at that PR as well - looks like an upstream issue

[fluidnumerics-joe]

Yea, just added conda_forge channels, which rattler appears to need and now we get to the following

(base) [joe@claymation parcels-benchmarks]$ pixi run asv run
· Fetching recent changes
· Creating environments/home/joe/Projects/Geomar-Utrecht/parcels-benchmarks/.pixi/envs/default/lib/python3.12/site-packages/rattler/virtual_package/virtual_package.py:174: UserWarning: Use `detect` instead
  warnings.warn("Use `detect` instead")
✔ validate cache       26 packages in 1s 47ms
✔ download & extract   5 packages (32.80 MiB) in 1s 45ms
✔ installing packages  took 1s 354ms                                                                                                                                                                                                          ...
· Discovering benchmarks
·· Uninstalling from rattler-py3.12
·· Building ea9517aa <v4-dev> for rattler-py3.12.............
·· Installing ea9517aa <v4-dev> into rattler-py3.12
·· Last error: Configuration error: {wheel_file} not available when substituting into command 'in-dir={env_dir} python -m pip install {wheel_file} --force-reinstall' Available: {'project': 'parcels', 'conf_dir': '/home/joe/Projects/Geomar-Utrecht/parcels-benchmarks', 'env_name': 'rattler-py3.12', 'env_dir': '/home/joe/Projects/Geomar-Utrecht/parcels-benchmarks/.asv/env/b54e0339d4ce929e91168b1373f7b02b', 'env_type': 'rattler', 'commit': 'ea9517aad9c6dd5f4aac1dd378e23af1e680f20d', 'build_dir': '/home/joe/Projects/Geomar-Utrecht/parcels-benchmarks/.asv/env/b54e0339d4ce929e91168b1373f7b02b/project', 'build_cache_dir': '/home/joe/Projects/Geomar-Utrecht/parcels-benchmarks/.asv/env/b54e0339d4ce929e91168b1373f7b02b/asv-build-cache/ea9517aad9c6dd5f4aac1dd378e23af1e680f20d'}
·· Failed to build the project and import the benchmark suite.

[fluidnumerics-joe]

I've pushed up the changes to the feature/asv branch.. These mainly include removing parcels from the pixi environment (since rattler will set this up for benchmarks), and switching to rattler for the environment_type in asv..

[fluidnumerics-joe]

With #25 merged in, we can now use ASV for benchmarking. Benchmarks are defined that perform only file io activity to help provide context for impacts of forward integration runtime due to file IO.

I think it would still be interesting to collect benchmark results from everyone so that we can start to get a feel about the performance portability across different systems. The easiest route would be to host the results directly in this repository; when someone locally runs

pixi run asv publish
pixi run asv preview

Everyone would be able to see their benchmark runtimes alongside other contributions.

The downside is that the benchmark dataset may become large over time, but maybe that's a bridge we cross when (if) we get there ?

@willirath and @erikvansebille what do you think ?

[erikvansebille]

Yep, I think that's a very good idea. A workflow where people can publish/post their runs is important, so that we can compare across machines