Datafinder Easy Updates

[genericdata]

In #141 , we said regarding the spreadsheet

the linked Google Sheet doesn't seem to contain many data that the storage finder will need, e.g., questions and options. Converting existing fields like eligibility and auto matching them to options would also require custom parsers to detect keywords and map them to structured choice IDs. We would also need to generate relationships (questions with choices, services with matching choice ID arrays). So it will be somewhat difficult (though not impossible).

Can you think of any way that would make updating the information, questions, services, and their matching easier? Spreadsheet, web form, multiple JSONs?

[kapuic]

I’ll prototype and consider the best way to make data updates easy. Spreadsheets still seem the most user-friendly and reliable option if we don’t want a backend or database, though parsing CSVs and creating the relations might require some careful thoughts. I also think AI might simplify things, if we let it read a spreadsheet diff and determine what JSON to update.

[kapuic]

Status Update for the Prototype

Implementation Plan

• Questions and choices are defined in TypeScript, and each choice is paired with keyword/regex matchers that scan spreadsheet columns for signals.
• It already maps four questions to existing columns (Storable Files, Eligibility, Permission Settings, Backup), generating matching facet IDs while populating the service detail fields (Links, Use Case, Limitations, Permission Settings, Eligibility, Synchronous Access, Alumni Access, Backup).
• The importer enforces single-choice questions by throwing if multiple options match; multi-select facets can use either comma-separated keywords or multiple descriptive phrases, as long as the configured regex patterns can detect them reliably.

Required Actions

Covered questions (already tied to existing columns):

• “What is the risk classification of your data?” ← column Storable Files
• “What is your University affiliation?” ← column Eligibility
• “Who needs access to your data?” ← column Permission Settings
• “Do you need backups, snapshots or replication of your data?” ← column Backup

To keep parity with the current JSON data, we need to add six more questions; each one requires its own column, so the importer can map responses deterministically. Populate each column with the corresponding choice labels (multiple keywords are fine if they apply and wording does not need to be exact):

• Add column Purpose → “For what purpose will you be using this storage?”
  • Examples: Backups, Data storage for a project or team, Hosting an application or database, Long-term preservation, Sharing data externally with a web link.
• Add column Access Locations → “From where will the data be accessed?”
  • Examples: On a NYU campus network or remotely through a VPN, Public Cloud Provider, Off campus without a VPN, From a web browser, From workstations or laptops.
• Add column Max File Size → “How large is any individual file in this collection likely to be?”
  • Examples: Some individual files will be larger than 5 GB, Some individual files will be larger than 10 GB, Some individual files will be larger than 15 GB, Some individual files will be larger than 5 TB (terabytes).
• Add column Additional Capabilities → “What additional capabilities do you require for data storage?”
  • Examples: Access level controls (admin, read, write, etc.), Automated workflows, Code sharing / code management, Data replication (copy of data exists in other location(s)), Data transfer from instruments, Generate DOIs for datasets (Digital Object Identifier), Large data transfers, Version control / snapshots.
• Add column Growth Projection → “How much data do you have and how fast will it grow?”
  • Examples: Unlikely to exceed 1TB in 2 years, Greater than 1TB or likely to exceed in 2 years.
• Add column Usage Pattern → “How are you currently using the data?”
  • Examples: Actively processing data (creating, editing, calculating), Private archive: Access controlled, not published, Publishing / sharing your data: Worldwide, read-only access to a completed data set.

In addition, we need to adjust some existing columns:

• Edit column Use Case: it would be helpful to include more exact keywords (e.g., existing purpose choices from the JSON: Backups, Data storage for a project or team, Hosting an application or database, Long-term preservation, Sharing data externally with a web link) so the regex matchers can map to that question.

Informational-only columns:

Links, Use Case, Limitations, Synchronous Access, Alumni Access currently just populate service details. If we want any of them to become new questions (for example, “Do you need storage to remain accessible for alumni?”), let me know.

[genericdata]

Any updates on this or anything I can assist with?