VEDA Data Stories directory creation

[kyle-lesinger]

To supplement existing and future VEDA Data Stories, notebooks are needed to allow for users to visualize data. These notebooks will feature various functions to retrieve data from various sources (e.g., Earthdata) and dynamically plot data. Self-contained functions will also be incorporated to reduce the amount of code within the notebook and will assist in the rapid development of future story notebooks. @acblackford @smk0033 @wildintellect

To fully complete this task, decisions will need to be made regarding:

• the final location of the new directory (and naming convention)
• best location for functions that are created which are imported into stories
• URL links for each story notebook that can be added to the VEDA stories
• process for new users who click on notebook links (within the story) to grant access
• and ensuring that users do not have to pip install anything (notebooks should be run natively after login)

[kyle-lesinger]

Regarding the final location of the directory. If we place it within user-guide/notebooks/datasets we would need to likely add a new directory called stories (or something else). There will be numerous notebooks added in the future so if we leave them in only the datasets directory this may create confusion and individuals may have to tediously sort through non-story notebooks. Thoughts?

[acblackford]

@kyle-lesinger I think this sounds like a good plan.

[wildintellect]

I think @briannapagan and @aboydnw should weigh in on this discussion.

• @kyle-lesinger can you provide some URLs of published stories that have or will have a notebook companion?
• Common code could be in the same directory with relative import unless it becomes a package in which case it will need it's own repo so it can be pip installed and/or published to pypi/conda.
• However libraries should be added to the standard VEDA Pangeo definition https://github.com/NASA-IMPACT/pangeo-notebook-veda-image/blob/main/environment.yml assuming they are only Python Stories (eventually we'll support other languages)
• Launching a notebook would be something like https://binder.openveda.cloud/v2/gh/NASA-IMPACT/veda-docs/HEAD?labpath=user-guide/notebooks/datasets/ocean-npp-timeseries-analysis.ipynb no extra permissions are required. For users that do have accounts we can add a second method that launches it inside the Hub instead.

Stories could be it's own folder at the same level as datasets if we think it's distinctly different.

[kyle-lesinger]

@wildintellect Here is the only story that is currently set to be added Derecho-impacts. After we get a common set of functions in the repo, they will add more stories.

• Common code could be in the same directory with relative import unless it becomes a package in which case it will need it's own repo so it can be pip installed and/or published to pypi/conda.
  ----- Easy to accomplish

• However libraries should be added to the standard VEDA Pangeo definition https://github.com/NASA-IMPACT/pangeo-notebook-veda-image/blob/main/environment.yml assuming they are only Python Stories (eventually we'll support other languages)
  ----- I'll add the specific libraries to this

[brianna-corremonte]

Thank you for the mention @wildintellect

@kyle-lesinger can we find a time to tag-up on this topic? It sounds like you want embedded notebooks or the ability for users to click a link and open a notebook in the VEDA hub. This sounds like a new component of data stories which to my understanding there have been initial chats on this type of feature but nothing implemented so far (CC'ing @batpad).

[aboydnw]

I think it is good to focus on this connection between the dashboard experience and jupyter experience. We haven't been able to spend much time on that yet.
@kyle-lesinger A few questions on the needs and the user experience.

notebooks are needed to allow for users to visualize data.

Can you expand more on this? My understanding is that notebooks are good for enabling science users to dive deeper into the data and explore on their own.

process for new users who click on notebook links (within the story) to grant access

For the user experience side, this is maybe the most important hurdle. It's great to include a link, but users will drop off very quickly if they don't have access or can't obtain access immediately. And by immediately I mean within a matter of seconds, not days, hours, or even minutes.

URL links for each story notebook that can be added to the VEDA stories

It sounds like the user experience you have in mind is:

1. User reads story on dashboard
2. Sees link to a notebook, inviting them to explore more
3. They're interested, so they click on the link
4. They go into a jupyter notebook, where they either already have access or can quickly gain access
5. The read the notebook, which has some additional visualizations
   ...and then what? Why did we send them to this other environment?

This represents a pretty significant context switch for the user, and usually we would do this if the content is totally different, or the capabilities/use cases are different. If the goal is just visualization, I would ask if we can do that visualization in the dashboard itself. If it's about something more, like I imagine more interacting with the data, maybe that changes the final solution

[kyle-lesinger]

@briannapagan Yes a tag-up would be excellent. I'm available most days/time so whenever works for you will likely be good for me! Yes the eventual goal would be to have a url (or embedded notebook) within the original data story (e.g., in Derecho Story near the end of the story).

[kyle-lesinger]

@aboydnw You bring up a series of good questions to consider and now I'll invite @brianmfreitag into the discussion. The current Derecho notebook does have the ability for users to retrieve new data from EarthData (specifically for different dates) which means that its more than just a static notebook that only relates to the story; therefore, exploration of data is accomplished if data is pulled from Earthdata. But if the data is drawn from our STAC collection then that specific section is likely static.

Regarding my initial statement and your question:

notebooks are needed to allow for users to visualize data.

I think that my choice of wording was a little too restrictive. User can visualize and explore the data that was used within our stories. But this may be restricted to only the Earthdata since items in our STAC may have a fixed data coverage.

This will definitely have to be further discussed to ensure that we can provide such functionality

process for new users who click on notebook links (within the story) to grant access

1. User reads story on dashboard
2. Sees link to a notebook, inviting them to explore more
3. They're interested, so they click on the link
4. They go into a jupyter notebook, where they either already have access or can quickly gain access
5. The read the notebook, which has some additional visualizations
   ...and then what? Why did we send them to this other environment?

Regarding Q5 that is a good question. @brianmfreitag has requested that this process be completed and I'm not 100% sure of any HQ requests/expectations.

I envision that eventually the data processing tasks that are required for Data Stories can be alleviated with this portal. So this would not only be a place where the Story Notebooks are, but also provide a central location for our workers to have quicker access and processing of data. I also envision that once many data processes have been moved into JupyterHub, then the notebooks would show the user exactly how the data was retrieved and processed. Currently the data processing seems to be done based on the individual story writers preference (e.g., ArcGIS, or local python environment). But the user still has minimal information about how the data was/can be retrieved, so maybe this can assist a little bit more with open-science and methods for cloud-based data processing.

[kyle-lesinger]

@aboydnw @wildintellect After further discussion with @brianmfreitag here are some expectations and further rationale for VEDA stories within veda-docs as notebooks.

1.) Remove barriers for cloud computing by providing a "blueprint".

• notebooks will serve as a template to understand how data from different DAAC's can be obtained.

2.) Small-scale processing to enable larger scale processing.

• While our notebooks will not be intended to allow for entire datasets to be downloaded and processed, these notebooks can be copied to a user's local environment and more data can be obtained and analyzed. This can be for researchers or other individuals.

3.) These notebooks can be directly used as a curriculum within workshops or other teaching spaces.

4.) Provide a means for users to do more exploration than what we looked examined in the story.

• e.g., users can select different disaster-related events for derechos, landslides, fires, etc.

[brianna-corremonte]

Thank you for the extra feedback @kyle-lesinger! I have some comments

notebooks will serve as a template to understand how data from different DAAC's can be obtained.

Openscapes has put alot of time and energy in providing notebooks for this same purpose. So I would encourage to either point to those or understand how to integrate.

While our notebooks will not be intended to allow for entire datasets to be downloaded and processed, these notebooks can be copied to a user's local environment and more data can be obtained and analyzed. This can be for researchers or other individuals.

I'm assuming these notebooks are leveraging the fact that they are being run in the hubs (already in the cloud) so it would not be the same experience to run them locally unless by local you mean a users own cloud environment.

Speaking to Sanjay he reminded me of nbgitpuller - @kyle-lesinger can you pass me an example notebook URL and we have a tag up to see if nbgitpuller gets us partially to the solution we're after?

[wildintellect]

@briannapagan @kyle-lesinger

• Binder is for people without a VEDA or Hub account
• I think nbgitpuller is for people with an account so that the notebook opens in a Hub workspace

We should make links in docs for both uses cases.

This is what the Binder urls look like on existing docs:
https://binder.openveda.cloud/v2/gh/NASA-IMPACT/veda-docs/HEAD?labpath=user-guide/notebooks/datasets/ocean-npp-timeseries-analysis.ipynb for https://docs.openveda.cloud/user-guide/notebooks/datasets/ocean-npp-timeseries-analysis.html

So in this case I think it would be ... waiting for build to start (it worked, see note below)...
https://binder.openveda.cloud/v2/gh/NASA-IMPACT/veda-docs/addStory?labpath=user-guide/notebooks/stories/derechos.ipynb because it's in a branch

Minor note - Binder, not sure about nbgitpuller, uses repo2docker on this repo so it doesn't currently use the Pangeo image that VEDA normally uses. Which means all the dependencies need to be listed in https://github.com/NASA-IMPACT/veda-docs/blob/main/requirements.txt

[wildintellect]

Openscapes has put alot of time and energy in providing notebooks for this same purpose. So I would encourage to either point to those or understand how to integrate.

@briannapagan I think the difference here, is that the corresponding VEDA story exists https://www.earthdata.nasa.gov/dashboard/stories/derecho-impacts , it's a not solely a notebook, nor is the focus of the notebook on methods of accessing EarthData like openscapes, but an actual science example. So it could belong in https://nasa-openscapes.github.io/earthdata-cloud-cookbook/tutorials/

@smk0033 can you link our Documentation decision tree for everyone so we can assess where it should belong?

[smk0033]

Documentation decision tree