-
Notifications
You must be signed in to change notification settings - Fork 3
Add Storage Finder CSV importer and weekly sync #228
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
4f8f693
8c425ce
03c6ea5
7ed02be
1443ae7
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||||
|---|---|---|---|---|---|---|---|---|
| @@ -0,0 +1,34 @@ | ||||||||
| name: Sync Storage Finder Data | ||||||||
|
|
||||||||
| on: | ||||||||
| schedule: | ||||||||
| - cron: "0 0 * * 6" | ||||||||
| workflow_dispatch: | ||||||||
|
|
||||||||
| jobs: | ||||||||
| sync: | ||||||||
| runs-on: ubuntu-latest | ||||||||
| steps: | ||||||||
| - name: Checkout | ||||||||
| uses: actions/checkout@v4 | ||||||||
| with: | ||||||||
| fetch-depth: 0 | ||||||||
|
|
||||||||
| - name: Set up Bun | ||||||||
| uses: oven-sh/setup-bun@v2 | ||||||||
|
|
||||||||
| - name: Install dependencies | ||||||||
| run: bun install --frozen-lockfile | ||||||||
|
|
||||||||
| - name: Generate storage finder data from sheet | ||||||||
| run: bun scripts/storage-finder-data-generator/generate.ts --output src/data/storage-finder | ||||||||
|
|
||||||||
| - name: Lint and format | ||||||||
| run: | | ||||||||
| bun lint --fix | ||||||||
|
||||||||
| bun lint --fix | |
| bun lint:js --fix | |
| bun lint:css --fix |
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,45 @@ | ||
| # Storage Finder CSV Importer: Maintainer Guide | ||
|
|
||
| This approach aims to balance user-friendliness (content editors update the Google Sheet) with functionality (developers tune `config.ts` so the JSON stays in sync). | ||
|
|
||
| ## What the CLI does | ||
|
|
||
| - Reads a CSV export of the shared Google Sheet (either via `STORAGE_FINDER_SHEET_URL` or `--csv <path>`). | ||
| - Applies the questions/choices/matchers defined in `scripts/storage-finder-data-generator/config.ts`. | ||
| - Generates `facet-tree.json` and `service-list.json` with slug-based IDs. | ||
|
|
||
| ## How to regenerate data | ||
|
|
||
| ```sh | ||
| bun scripts/storage-finder-data-generator/generate.ts \ | ||
| --csv "Datafinder Data - Sheet1.csv" \ | ||
| --output src/data/storage-finder | ||
| ``` | ||
|
|
||
| - Omit `--csv` to download from `STORAGE_FINDER_SHEET_URL`. | ||
| - Use `--output` to write elsewhere (defaults to `src/data/storage-finder/generated` in the code). | ||
| - If you do not have Bun, install it from [https://bun.sh/](https://bun.sh/) or run with `pnpm dlx tsx scripts/storage-finder-data-generator/generate.ts ...`. Node does not run TypeScript by default; `tsx` provides the TypeScript loader. | ||
|
|
||
| ## How to add or change a question | ||
|
|
||
| 1. Add a column to the sheet that contains the signals you want to match. | ||
| 2. In `scripts/storage-finder-data-generator/config.ts`, add a new facet entry to `FACET_CONFIGS`: | ||
| - Set `id` (slug), `name`, `controlType` (`radio` or `checkbox`), `column` (sheet column name), and `choices` (labels the app should show). | ||
| - Add `matchers`: regex patterns that map cell text to choice IDs. Include `allowMultipleMatches: true` if a radio question legitimately matches more than one choice. | ||
| - If no regex matches, `fallback: "all"` keeps the service visible; otherwise supply an explicit array of choice IDs. | ||
| 3. Regenerate with the CLI and verify in the app. | ||
|
|
||
| ## Service fields | ||
|
|
||
| Field definitions live in `scripts/storage-finder-data-generator/config.ts` (`FIELD_DEFINITIONS`). They map sheet columns to service detail rows (Links, Use Case, Limitations, Permission Settings, Eligibility, Synchronous Access, Alumni Access, Backup). Adjust labels or formatters there if the sheet schema changes. | ||
|
|
||
| ## Naming and IDs | ||
|
|
||
| - Services are slugged from the `Title` column; duplicates get `-2`, `-3`, etc. | ||
| - Facet and choice IDs are slugs defined in `config.ts`; keep them stable to avoid breaking references. | ||
|
|
||
| ## Validation rules | ||
|
|
||
| - Radio facets throw if more than one choice matches unless `allowMultipleMatches` is set. | ||
| - Blank cells render as “Not Available” in service fields. | ||
| - Regexes match against raw cell text; use clear keywords in the sheet for deterministic mapping. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The PR title and description mention "hourly sync", but the cron schedule
"0 0 * * 6"runs weekly on Saturdays at midnight UTC, not hourly.If hourly sync is intended, the cron should be
"0 * * * *". If weekly is correct, the PR description should be updated to reflect this.