Skip to content

Commit caab820

Browse files
Sam-BollingSam-Bolling
committed
docs(usgs-nims): add total bootstrap data model pack
1 parent 0ee06a6 commit caab820

30 files changed

Lines changed: 1709 additions & 0 deletions

‎docs/research/USGS_NIMS_Total_Bootstrap_Data_Model_Enrichment_Pack_2026-03-11.md‎

Lines changed: 160 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,160 @@
1+
# USGS NIMS Total Bootstrap, Data Model, and Enrichment Pack
2+
3+
**Date:** 2026-03-11
4+
**Author:** Codex
5+
**Status:** Created, packaged, and intended for repository handoff
6+
**Scope:** `publishers/usgs_nims` total package including bootstrap guidance, data-model review, metadata enrichment, and zip artifact
7+
8+
---
9+
10+
## 1. Executive Summary
11+
12+
A new comprehensive USGS NIMS package was created under:
13+
14+
- `publishers/usgs_nims/total_bootstrap_data_model_enrichment_pack`
15+
16+
This package is broader than a metadata-only pack. It includes:
17+
18+
- a reviewed resource-model description
19+
- live-source verification notes against the current NIMS API and S3 URL behavior
20+
- metadata sidecars and worked examples
21+
- bootstrap patch candidates for richer procedure, datastream, deployment, and camera sidecar metadata
22+
- a shareable zip artifact of the full package
23+
24+
The package is designed to preserve the current Pattern A shared-system model
25+
while materially improving provenance, semantic clarity, and future maintainability.
26+
27+
---
28+
29+
## 2. Why a Larger Package Was Justified
30+
31+
The USGS NIMS publisher is already stronger than a raw prototype. It already has:
32+
33+
- one shared NIMS imagery procedure
34+
- one imagery datastream per curated selected camera
35+
- reuse of the existing USGS water station systems
36+
- a coherent NIMS-specific deployment tree
37+
- a working runtime that publishes image-reference observations
38+
39+
So the right move was not to redesign it blindly. The right move was to package
40+
the current implementation properly:
41+
42+
- make the Pattern A shared-system model explicit
43+
- anchor the bootstrap to current live NIMS semantics
44+
- identify which metadata enrichments are safe now
45+
- record where the current one-camera-per-station assumption becomes a real constraint
46+
47+
---
48+
49+
## 3. Live Research Findings That Matter
50+
51+
The package was informed by live verification on 2026-03-11 against the current
52+
USGS NIMS API.
53+
54+
The most important findings were:
55+
56+
1. `cameras?camId=...` returns a single camera object with rich directory and cadence metadata.
57+
2. `listFiles` supports both plain filename arrays and `rawItem=true` structured objects.
58+
3. `siteId` discovery can return multiple cameras for one NWIS site.
59+
4. Resolution-specific image URLs and timelapse URLs resolve successfully from the NIMS S3 bucket.
60+
5. The active API path is still `v0`, so the package intentionally keeps `v0` URLs.
61+
62+
The most important modeling conclusion is this:
63+
64+
The current publisher is not a general many-camera-per-site publisher. It is a
65+
selected-camera-per-station publisher attached to reused USGS water systems.
66+
67+
That is a sound model for the current curated set, but it should be documented as
68+
an explicit design choice rather than treated as if the upstream API naturally
69+
exposes only one camera per site.
70+
71+
---
72+
73+
## 4. Package Contents
74+
75+
### 4.1 Notes
76+
77+
The `notes/` section contains:
78+
79+
- live source verification
80+
- audit and recommendations
81+
- apply order
82+
- runtime and model follow-on guidance
83+
84+
### 4.2 Data model
85+
86+
The `data_model/` section contains:
87+
88+
- a resource-model walkthrough
89+
- machine-readable inventory
90+
- current observation-contract documentation
91+
- upstream-to-CSAPI field mapping
92+
93+
### 4.3 Metadata and examples
94+
95+
The `metadata/` section contains:
96+
97+
- official source URLs
98+
- live worked examples from the current NIMS API
99+
- a curated-site live camera-count snapshot
100+
- direct URL-resolution verification
101+
- enriched camera templates and worked examples
102+
103+
### 4.4 Assets
104+
105+
The `assets/` section contains:
106+
107+
- a generic local NIMS camera SVG
108+
- a note explaining why no single official dynamic camera image was bundled
109+
110+
### 4.5 Bootstrap patch candidates
111+
112+
The `patches/` section contains:
113+
114+
- constants and helper URLs
115+
- an enriched procedure body
116+
- an enriched imagery datastream schema
117+
- enriched deployment blocks
118+
- an enriched camera JSON example
119+
- a compact candidate snippet summary
120+
121+
---
122+
123+
## 5. Design Position
124+
125+
This package makes a deliberate distinction between:
126+
127+
- what should be changed now
128+
- what should be documented now but implemented later
129+
130+
### Recommended now
131+
132+
- richer procedure provenance
133+
- richer imagery datastream metadata
134+
- clearer shared-system deployment wording
135+
- optional enriched camera-config sidecars
136+
- better documentation of the current image-reference observation contract
137+
138+
### Recommended later
139+
140+
- decide whether the one-camera-per-station rule remains the project policy
141+
- if not, redesign explicitly for multi-camera sites
142+
- evaluate whether `rawItem=true` should become the runtime default
143+
- decide whether file size or upstream timestamp belong in future observation payloads
144+
145+
This keeps the package robust without forcing unnecessary runtime churn.
146+
147+
---
148+
149+
## 6. Bottom Line
150+
151+
The new USGS NIMS package is not just a metadata patch. It is a reviewed handoff
152+
bundle for the current publisher:
153+
154+
- architecture clarified
155+
- live upstream semantics verified
156+
- enrichment candidates prepared
157+
- zip artifact produced for transport and review
158+
159+
That is the right level of packaging for a publisher that is already functional
160+
and now needs to become more explicit, more authoritative, and easier to extend.

‎publishers/usgs_nims/total_bootstrap_data_model_enrichment_pack/README.md‎

Lines changed: 102 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,102 @@
1+
# USGS NIMS Total Bootstrap, Data Model, and Enrichment Pack
2+
3+
This package is a comprehensive handoff bundle for the current
4+
`publishers/usgs_nims` publisher in `OSHConnect-Python`.
5+
6+
It is broader than a metadata-only enrichment pack. It includes:
7+
8+
- a reviewed data-model section
9+
- live-source verification notes
10+
- metadata sidecars and worked examples
11+
- ready-to-apply bootstrap snippet candidates
12+
- a package manifest suitable for zipping and external sharing
13+
14+
The package was assembled after cross-referencing:
15+
16+
- the current local `bootstrap_usgs_nims.py`
17+
- the current local `usgs_nims_publisher.py`
18+
- the current local `cameras.json`
19+
- the current local USGS API reconnaissance notes
20+
- the current USGS water publisher package and shared-system pattern
21+
- live USGS NIMS API responses verified on 2026-03-11
22+
23+
## Scope
24+
25+
This package is designed to improve and document the USGS NIMS imagery publisher
26+
without forcing risky runtime or architecture changes into the current codepath.
27+
28+
It does three things:
29+
30+
1. documents the current Pattern A shared-system model
31+
2. provides a richer metadata-enrichment layer for NIMS procedure, datastream,
32+
deployment, and camera sidecar metadata
33+
3. captures the most important follow-on runtime and model recommendations
34+
35+
## Why this package exists
36+
37+
The current USGS NIMS publisher already has a coherent architecture:
38+
39+
- one shared imagery procedure
40+
- one imagery datastream per curated camera
41+
- datastreams attached to existing USGS water station systems
42+
- one NIMS-specific deployment tree
43+
- one working publisher that emits image-reference observations instead of binary images
44+
45+
That baseline is sound. The main value of this package is to make the current
46+
design more explicit, more authoritative, and easier to extend safely.
47+
48+
The most important findings from live verification are:
49+
50+
- `cameras?camId=...` currently returns a single camera object
51+
- `cameras?siteId=...` can return multiple cameras for one NWIS site
52+
- `listFiles` supports both plain filename arrays and richer `rawItem=true` objects
53+
- resolution-specific image URLs and timelapse URLs resolve successfully from the
54+
NIMS S3 bucket
55+
- the current publisher model assumes one selected camera per shared station system
56+
57+
## Package layout
58+
59+
- `bundle_manifest.json`
60+
- `notes/`
61+
- `data_model/`
62+
- `metadata/`
63+
- `assets/`
64+
- `patches/`
65+
66+
## Recommended reading order
67+
68+
1. `notes/LIVE_SOURCE_VERIFICATION_2026-03-11.md`
69+
2. `notes/AUDIT_AND_RECOMMENDATIONS.md`
70+
3. `data_model/RESOURCE_MODEL.md`
71+
4. `metadata/source_urls.json`
72+
5. `patches/bootstrap_usgs_nims_metadata_enriched_candidate_snippets.py`
73+
74+
## Implementation stance
75+
76+
This pack is conservative where the current runtime is already good and explicit
77+
where the current metadata and data-model story are too thin.
78+
79+
It does not assume every future enhancement should be applied immediately. In particular:
80+
81+
- metadata enrichment is recommended now
82+
- stronger camera-side provenance is recommended now
83+
- multi-camera-per-site generalization is documented but not forced into the
84+
current package
85+
- runtime contract expansion beyond the current image-reference payload is optional
86+
87+
## Important architecture note
88+
89+
This pack keeps the current Pattern A shared-system model:
90+
91+
- no new NIMS-specific systems are created
92+
- imagery datastreams remain companion datastreams on existing USGS water systems
93+
94+
That means the richest enrichment targets are:
95+
96+
- the NIMS procedure
97+
- the NIMS datastream schema and descriptions
98+
- NIMS-specific deployments
99+
- the curated `cameras.json` sidecar
100+
101+
It also means the package explicitly documents the current one-camera-per-station
102+
assumption and the live evidence that some sites now expose multiple cameras.

‎publishers/usgs_nims/total_bootstrap_data_model_enrichment_pack/assets/official_image_reference.txt‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
No single official image is bundled in this package.
2+
3+
Reason:
4+
5+
- the NIMS publisher is itself an image-reference publisher whose payload points
6+
to live upstream imagery
7+
- the most authoritative "official image" is dynamic and camera-specific
8+
- bundling one camera image into the package would overfit the package to one
9+
moment and one site
10+
11+
This package therefore includes a local generic SVG:
12+
13+
- `assets/usgs_nims_camera_generic.svg`
14+
15+
Use that asset for documentation, demos, or placeholder rendering unless you
16+
choose to render live NIMS imagery directly.

‎publishers/usgs_nims/total_bootstrap_data_model_enrichment_pack/assets/usgs_nims_camera_generic.svg‎

Lines changed: 11 additions & 0 deletions
Loading

‎publishers/usgs_nims/total_bootstrap_data_model_enrichment_pack/bundle_manifest.json‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
{
2+
"name": "USGS NIMS Total Bootstrap, Data Model, and Enrichment Pack",
3+
"packageVersion": "2026-03-11",
4+
"publisherPath": "publishers/usgs_nims",
5+
"generatedDate": "2026-03-11",
6+
"purpose": [
7+
"Document the current USGS NIMS imagery publisher architecture",
8+
"Provide live-verified source references and worked examples",
9+
"Deliver a metadata enrichment pack for NIMS bootstrap resources",
10+
"Package the result as a shareable zip artifact"
11+
],
12+
"artifactGroups": [
13+
"notes",
14+
"data_model",
15+
"metadata",
16+
"assets",
17+
"patches"
18+
],
19+
"upstreamVerifiedOn": "2026-03-11",
20+
"currentLocalInputs": [
21+
"publishers/usgs_nims/bootstrap_usgs_nims.py",
22+
"publishers/usgs_nims/usgs_nims_publisher.py",
23+
"publishers/usgs_nims/cameras.json",
24+
"docs/research/USGS_API_Reconnaissance_Notes.md",
25+
"docs/research/USGS_NIMS_Follow_On_Publishers_Plan.md"
26+
],
27+
"notes": [
28+
"This package is Pattern A specific and assumes reuse of existing USGS water systems.",
29+
"Live verification confirms that some curated NWIS sites currently expose more than one NIMS camera.",
30+
"The zip archive for this package is created as a sibling artifact under publishers/usgs_nims."
31+
]
32+
}

0 commit comments

Comments
 (0)