NLDAS-3 Forcing Daily Collection Metadata Design [DRAFT]

[anayeaye]

What

Ideally we want to index an ice-chunk/zarr format for NLDAS-3 Forcing Daily Data. We may need to incrementally roll out a NetCDF version compatible with the staging titiler-multidim.

notes

• Prior art metadata for COG could be helpful when writing collection metadata

  veda-data/ingestion-data/staging/collections/nldas3.json

  Line 2 in 7f8e980

  "id": "nldas3",

• Collection id and title should reserve namespacing possibilities for hosting the existing COG collection, a netcdf collection, and a zarr collection. We aren't going to want 3 versions of the data but they may exist simultaneously while designing how this collection will be indexed and used.

• Ultimately, we want to support Exploration (map) and Analysis (time series over AOI selection) in a similar to what is available for COGs and CMR data in the earth data dashboard. For example: https://www.earthdata.nasa.gov/dashboard/data-catalog/nldas3

[jsignell]

The virtual zarr store is at: s3://nasa-waterinsight/virtual-zarr-store/NLDAS-3-icechunk/

[jsignell]

I think we can use https://github.com/stac-utils/xstac to get the data cube variables and the storage extension values.

For instance here how the planetary computer cmip-6 metadata were produced: https://github.com/stac-utils/xstac/blob/main/examples/nasa-nex-gddp-cmip6/generate.py

[maxrjones]

My understanding from the sprint retro today is that this will likely be included as a 26.1 objective, but is not a priority for 25.4

[jsignell]

@siddharth0248 and I have been working on a notebook. I am adding it to the workshop examples now: NASA-IMPACT/dse-virtual-zarr-workshop#8

[jbusecke]

I sent this to @maxrjones in a DM but also wanted to make this public. With regards to using titiler-multidim with the netcdf files directly I had a few observations:

• I spun up a local instance from the current main of titiler-multidim
• Using https://nasa-waterinsight.s3.amazonaws.com/NLDAS3/forcing/daily/202312/NLDAS_FOR0010_D.A20231201.030.beta.nc did work! (note that s3://... uri did not work)
  • But is very slow. For a single file I got 1+min to get a response! If you want a meaningful timeseries from that I think that will be a long wait.

[jsignell]

I have been iterating within the workshop notebooks but want to put the current version of the collection-level metadata here for visibility: https://gist.github.com/jsignell/e7069c66a5ed3d5a7eebea9e996b7dbb

[jsignell]

I just came across this ticket. Is there more to do on it or is all the work being tracked in other tickets?