Skip to content

Commit d5f2303

Browse files
Sam-BollingSam-Bolling
committed
Plan next publisher expansion steps
1 parent c6fc2d9 commit d5f2303

3 files changed

Lines changed: 476 additions & 0 deletions

File tree

‎docs/research/new-publisher-source-planning/New_Publisher_Implementation_Checklist_2026-05-26.md‎

Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
# New Publisher Implementation Checklist
2+
3+
Date: 2026-05-26
4+
5+
## Purpose
6+
7+
This checklist captures the repeatable workflow proven during the Environment Agency Hydrology implementation. Use it as the baseline for the next publisher sources so each integration reaches the same standard: researched, scoped, implemented, documented, visible in Explorer, and verified against the live demo path.
8+
9+
## 1. Source Triage
10+
11+
- Confirm the source is real, public, and machine-readable.
12+
- Identify official documentation, API root, sample list endpoint, sample data endpoint, and license terms.
13+
- Probe endpoints directly and record exact working query shapes.
14+
- Capture quirks early, especially auth, rate limits, pagination, timestamp format, units, and non-standard parameters.
15+
- Decide whether the source should be implemented now, deferred, or split into multiple publisher opportunities.
16+
17+
## 2. Existing Pattern Selection
18+
19+
- Choose the closest mature publisher as the primary exemplar.
20+
- Prefer station-network patterns for fixed monitoring sites.
21+
- Prefer event-feed patterns for earthquakes, alerts, or incident streams.
22+
- Prefer image/media patterns only when the source actually exposes useful media.
23+
- Record any server compatibility constraints from prior publishers before coding.
24+
25+
## 3. Curated First Pass
26+
27+
- Start with a small demo-safe sidecar, not a full-network ingestion.
28+
- Include enough variety to prove the data model.
29+
- Choose stations/events that are geographically legible in Explorer.
30+
- Keep runtime polling bounded and predictable.
31+
- Preserve original source IDs in metadata and observations, even if CSAPI UIDs need sanitized tokens.
32+
33+
## 4. CSAPI Model
34+
35+
- Define one procedure for the source ingestion method.
36+
- Define systems around physical stations, platforms, or logical event sources.
37+
- Define one datastream per observed property/product/statistic combination.
38+
- Define deployments that make map placement and hierarchy explicit.
39+
- Include source URLs, license links, units, parameter names, quality flags, and provenance in metadata.
40+
41+
## 5. Bootstrap
42+
43+
- Use shared bootstrap helpers where possible.
44+
- Create minimal GeoJSON stubs first, then richer SensorML through PUT where the server accepts it.
45+
- Support `--dry-run`, `--clean`, `--clean-only`, and `--force-sml`.
46+
- Log and recover from known server compatibility failures without hiding them.
47+
- Compile and run the bootstrap locally before touching the live server.
48+
49+
## 6. Runtime Publisher
50+
51+
- Load curated source definitions from a sidecar file.
52+
- Fetch only latest or bounded recent readings in normal operation.
53+
- Normalize timestamps to UTC.
54+
- Preserve source quality, completeness, revision, and status metadata when available.
55+
- Dedupe unchanged readings during a running process.
56+
- Support `--dry-run`, `--once`, `--interval`, and source-subset flags.
57+
58+
## 7. Explorer Readiness
59+
60+
- Verify the map can classify the publisher with an appropriate STANAG/MIL-STD-2525 symbol.
61+
- Add source-specific symbol rules only when generic rules produce a poor result.
62+
- Make side-card summaries meaningful for the domain.
63+
- Surface latest observations where the current value is useful to a viewer.
64+
- Add image/media metadata only when it is accurate, licensed, and clearly attributed.
65+
- Use explicit representative-image language when exact station imagery is unavailable.
66+
67+
## 8. Validation
68+
69+
- Compile changed Python modules.
70+
- Run dry-run source fetches.
71+
- Run bootstrap against the live OSH endpoint.
72+
- Run one live publish cycle.
73+
- Verify backend observations directly.
74+
- Verify Explorer visibility on the correct preset.
75+
- Verify production bundle content after pushing Explorer changes.
76+
- Record any server warnings separately from publisher failures.
77+
78+
## 9. Documentation
79+
80+
- Add or update the publisher README.
81+
- Add an implementation plan before coding.
82+
- Add a completion report after first working publish.
83+
- Add a live-demo verification report after Explorer validation.
84+
- Record server compatibility issues as issue-ready drafts or remote issues.
85+
- Keep image attribution and license notes close to the implementation and report.
86+
87+
## 10. Commit And Push
88+
89+
- Keep publisher and Explorer commits separate when they live in separate repositories.
90+
- Verify `git status --short` before each commit.
91+
- Push immediately when the user expects live-demo behavior.
92+
- Recheck the deployed production bundle or runtime after push.
93+
94+
## Minimum Done Definition
95+
96+
A new publisher is not done until all of these are true:
97+
98+
- curated bootstrap succeeds or documented server limitations are isolated,
99+
- runtime can publish at least one clean live cycle,
100+
- observations can be read back from CSAPI,
101+
- Explorer can find and explain the resources,
102+
- side-card/popup output is domain-meaningful,
103+
- docs explain source, model, commands, validation, and limitations,
104+
- commits are pushed to the relevant repositories.

‎docs/research/new-publisher-source-planning/OSH_Server_Compatibility_Issue_Draft_2026-05-26.md‎

Lines changed: 110 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,110 @@
1+
# OSH Server Compatibility Issue Draft
2+
3+
Date: 2026-05-26
4+
5+
## Proposed Title
6+
7+
System SensorML PUT returns HTTP 500 and prevents publisher media metadata updates
8+
9+
## Summary
10+
11+
Environment Agency Hydrology bootstrap can create systems, datastreams, deployments, and observations successfully on the configured OSH endpoint, but system-level SensorML replacement currently fails with HTTP 500. Procedure SensorML PUT succeeds. The failure prevents new system metadata such as representative side-card image documents from reaching the live CSAPI resources through the normal SensorML path.
12+
13+
This is now visible as a publisher interoperability issue because station/system publishers rely on system SensorML `documents` metadata for rich Explorer cards, including thumbnails, source links, and attribution.
14+
15+
## Affected Endpoint
16+
17+
```text
18+
https://os4csapi-osh.duckdns.org/sensorhub/api
19+
```
20+
21+
Representative production preset:
22+
23+
```text
24+
OSH (OS4CSAPI)
25+
https://129-80-248-53.sslip.io/sensorhub/api
26+
```
27+
28+
## Reproduction
29+
30+
From `OSHConnect-Python`:
31+
32+
```powershell
33+
py -m publishers.environment_agency_hydrology.bootstrap_environment_agency_hydrology --force-sml
34+
```
35+
36+
Observed output includes one warning per Environment Agency station system:
37+
38+
```text
39+
[WARN] SML PUT skipped for system urn:os4csapi:system:environment-agency-hydrology:48513a18-e485-4317-ae92-93bf4f7f3e54:v1 (id=05j0): HTTP 500 PUT https://os4csapi-osh.duckdns.org/sensorhub/api/systems/05j0: {
40+
"status": 500,
41+
"message": "Internal server error"
42+
}
43+
```
44+
45+
Additional affected systems during the same run:
46+
47+
```text
48+
urn:os4csapi:system:environment-agency-hydrology:d52d0eab-1e64-4d76-a1f2-e81c7948d2c0-435510:v1
49+
urn:os4csapi:system:environment-agency-hydrology:c7e13884-4a02-4df3-b184-09aea28cf8e8-3-020:v1
50+
urn:os4csapi:system:environment-agency-hydrology:959f3e4f-bb6e-4f4a-8082-0157eea99482:v1
51+
```
52+
53+
## Expected Behavior
54+
55+
`PUT /systems/{id}` with `Content-Type: application/sml+json` should accept a valid SensorML JSON document for an existing system, consistent with procedure SensorML replacement behavior.
56+
57+
At minimum, the server should return a diagnostic 4xx response explaining which SensorML field is invalid instead of an opaque HTTP 500.
58+
59+
## Actual Behavior
60+
61+
The server returns HTTP 500 for every Environment Agency Hydrology system SensorML PUT attempted during `--force-sml`.
62+
63+
The bootstrap safely logs the failure and continues, so operational publishing still works:
64+
65+
- systems exist,
66+
- datastreams exist,
67+
- deployments exist,
68+
- live observations publish successfully.
69+
70+
The missing system SensorML update still blocks normal rich metadata propagation.
71+
72+
## Demo Impact
73+
74+
Explorer side-card thumbnails normally come from system SensorML image documents. Because the system PUT fails, the Environment Agency Hydrology representative gauge photo cannot be relied on from live system SensorML metadata.
75+
76+
Temporary mitigation implemented in Explorer:
77+
78+
```text
79+
OS4CSAPI/ogc-csapi-explorer@5323b4d Show hydrology station thumbnail fallback
80+
```
81+
82+
Publisher metadata and docs update:
83+
84+
```text
85+
OS4CSAPI/OSHConnect-Python@87a8f77 Add hydrology station thumbnail metadata
86+
OS4CSAPI/OSHConnect-Python@c6fc2d9 Record hydrology thumbnail live verification
87+
```
88+
89+
## Related Browser Finding
90+
91+
After reloading the production Explorer, the OSH external URL also produced a browser CORS diagnostic indicating duplicate `Access-Control-Allow-Origin` values:
92+
93+
```text
94+
The 'Access-Control-Allow-Origin' header contains multiple values '*, https://ogc-csapi-explorer.pages.dev', but only one is allowed.
95+
```
96+
97+
The app can still work through the configured proxy path, but this should be tracked as adjacent server/proxy header behavior if direct browser access is expected to remain supported.
98+
99+
## Suggested Labels
100+
101+
```text
102+
bug
103+
server-interop
104+
sensorml
105+
publisher-support
106+
```
107+
108+
## Notes
109+
110+
The GitHub CLI was not available in the current environment and no issue-management tool was exposed, so this file is an issue-ready draft rather than a remotely created GitHub issue.

0 commit comments

Comments
 (0)