Skip to content

Commit 789da21

Browse files
Sam-BollingSam-Bolling
committed
Add BGS SensorThings publisher
1 parent 1bad10b commit 789da21

7 files changed

Lines changed: 1335 additions & 0 deletions

File tree

‎docs/research/new-publisher-source-planning/BGS_SensorThings_Publisher_Completion_Report_2026-05-26.md‎

Lines changed: 133 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,133 @@
1+
# BGS SensorThings Publisher Completion Report
2+
3+
Date: 2026-05-26
4+
5+
## Summary
6+
7+
Implemented the third new publisher: BGS SensorThings telemetry. This publisher uses the public British Geological Survey Sensor Data Service OGC SensorThings API and publishes a curated UKGEOS Glasgow groundwater/geothermal telemetry subset into OSH/CSAPI.
8+
9+
The chosen branch was BGS SensorThings telemetry, not BGS earthquake GeoRSS, because it adds a standards-rich SensorThings-to-CSAPI interoperability story and avoids overlapping the existing USGS earthquake publisher.
10+
11+
## Implemented Files
12+
13+
- `publishers/bgs_sensorthings/stations.json`
14+
- `publishers/bgs_sensorthings/bootstrap_bgs_sensorthings.py`
15+
- `publishers/bgs_sensorthings/bgs_sensorthings_publisher.py`
16+
- `publishers/bgs_sensorthings/README.md`
17+
- `docs/research/new-publisher-source-planning/BGS_SensorThings_Publisher_Implementation_Plan_2026-05-26.md`
18+
- `publishers/README.md` updated with fleet entry and bootstrap command
19+
20+
Explorer production polish was implemented separately in the Explorer repository:
21+
22+
- BGS role inference: `Groundwater Telemetry Site`
23+
- BGS symbol rule: fixed Sensor Emplaced style
24+
- latest-reading label fix for `Water Level maOD`
25+
26+
## Source
27+
28+
- BGS Sensor Data Service: https://sensors.bgs.ac.uk/
29+
- SensorThings API root: https://sensors.bgs.ac.uk/FROST-Server/v1.1
30+
- API docs: https://sensors.bgs.ac.uk/api.html
31+
- Interactive docs: https://sensors-docs.bgs.ac.uk/
32+
33+
Curated datastreams were selected from BGS records reporting unrestricted access and Open Government Licence / UKRI acknowledgement data usage language.
34+
35+
## Curated Systems
36+
37+
| Curated site | SensorThings Thing ID | Name | Coordinates |
38+
| --- | --- | --- | --- |
39+
| `gga01-03` | `195` | Downhole hydro logger GGA01 | -4.200163, 55.839415 |
40+
| `gga05-03` | `197` | Downhole hydro logger GGA05 | -4.199619, 55.837690 |
41+
| `gga07-03` | `200` | Downhole hydro logger GGA07 | -4.201172, 55.838337 |
42+
43+
Each system has three datastreams:
44+
45+
- Water Temperature
46+
- Conductivity
47+
- Water Level maOD
48+
49+
## Live Bootstrap
50+
51+
Command run:
52+
53+
```bash
54+
py -m publishers.bgs_sensorthings.bootstrap_bgs_sensorthings --force-sml
55+
```
56+
57+
Created live resources:
58+
59+
- Procedure: `urn:os4csapi:procedure:bgs-sensorthings:v1`, server ID `04e0`
60+
- Deployments:
61+
- root `05ig`
62+
- group `05j0`
63+
- `gga01-03` deployment `05jg`
64+
- `gga05-03` deployment `05k0`
65+
- `gga07-03` deployment `05kg`
66+
- Systems:
67+
- `gga01-03` system `05n0`
68+
- `gga05-03` system `05ng`
69+
- `gga07-03` system `05o0`
70+
- Datastreams:
71+
- `gga01-03`: `05mg`, `05n0`, `05ng`
72+
- `gga05-03`: `05o0`, `05og`, `05p0`
73+
- `gga07-03`: `05pg`, `05q0`, `05qg`
74+
75+
## Server Compatibility Notes
76+
77+
The live OSH server repeated a known behavior seen in earlier publisher work: some system POST requests returned HTTP 500 after the resource had actually been created. The bootstrap recovered those system IDs by UID and continued successfully.
78+
79+
SensorML PUT for the recovered BGS systems returned HTTP 500. The GeoJSON stubs, datastreams, deployments, and observations were still created successfully, and Explorer was able to render BGS deployed-system cards using the available deployment/system/datastream metadata. Rich SensorML system metadata should be treated as a follow-up compatibility item rather than a blocker for the live demo path.
80+
81+
## Live Publish
82+
83+
Command run:
84+
85+
```bash
86+
py -m publishers.bgs_sensorthings.bgs_sensorthings_publisher --once
87+
```
88+
89+
The first live cycle published eight readings before output ended; `gga07-03` was rerun explicitly to confirm the final station end-to-end:
90+
91+
```bash
92+
py -m publishers.bgs_sensorthings.bgs_sensorthings_publisher --once --stations gga07-03
93+
```
94+
95+
The `gga07-03` rerun published 3 observations, 0 skipped, 0 errors.
96+
97+
## CSAPI Read-Back
98+
99+
Latest observation read-back succeeded for all nine BGS datastreams.
100+
101+
Representative values:
102+
103+
| Site | Datastream | Value | Phenomenon time |
104+
| --- | --- | ---: | --- |
105+
| `gga01-03` | Water Temperature | 12.4466 C | 2026-04-13T00:00:00Z |
106+
| `gga01-03` | Conductivity | 1689.2874 uS/cm | 2026-04-13T00:00:00Z |
107+
| `gga01-03` | Water Level maOD | 9.5584026158 m | 2026-04-13T00:00:00Z |
108+
| `gga05-03` | Water Temperature | 11.5819 C | 2026-04-13T00:00:00Z |
109+
| `gga05-03` | Conductivity | 1693.9048 uS/cm | 2026-04-13T00:00:00Z |
110+
| `gga05-03` | Water Level maOD | 10.0791145586 m | 2026-04-13T00:00:00Z |
111+
| `gga07-03` | Water Temperature | 11.454 C | 2026-04-13T00:00:00Z |
112+
| `gga07-03` | Conductivity | 1669.0203 uS/cm | 2026-04-13T00:00:00Z |
113+
| `gga07-03` | Water Level maOD | 9.5070321891 m | 2026-04-13T00:00:00Z |
114+
115+
## Explorer Verification
116+
117+
Production Explorer found the BGS deployments after filtering by `BGS` and narrowing visible layers to deployments/systems.
118+
119+
Verified card before Explorer polish deployment:
120+
121+
- `BGS GGA07 UKGEOS Deployment`
122+
- system subtitle: `BGS GGA07 Downhole Hydro Logger`
123+
- outputs: Conductivity, Water Level maOD, Water Temperature
124+
- latest readings: Conductivity, water level, water temperature
125+
- source links: BGS SensorThings Thing, BGS SensorThings API Docs, Open Government Licence v3.0
126+
127+
Explorer polish was pushed to improve the card from generic `Deployed System` to `Groundwater Telemetry Site` and to keep `Water Level maOD` as the latest-reading label.
128+
129+
## Remaining Follow-Up
130+
131+
- Recheck Cloudflare after Explorer deploy completes and confirm the production bundle includes `Groundwater Telemetry Site`.
132+
- Investigate the live OSH SensorML PUT 500 for BGS systems if richer system metadata becomes necessary.
133+
- Consider representative BGS/UKGEOS imagery only after separate licensing research; no image was added in this pass.

‎docs/research/new-publisher-source-planning/BGS_SensorThings_Publisher_Implementation_Plan_2026-05-26.md‎

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
# BGS SensorThings Publisher Implementation Plan
2+
3+
Date: 2026-05-26
4+
5+
## Status
6+
7+
Selected for publisher #3 after Environment Agency Hydrology and UK-AIR. This plan chooses the BGS SensorThings telemetry branch rather than the BGS earthquake GeoRSS branch because it adds a new standards-rich interoperability story and avoids duplicating the existing USGS earthquake event-feed pattern.
8+
9+
## Source
10+
11+
Primary source: BGS Sensor Data Service
12+
13+
- Landing page: https://sensors.bgs.ac.uk/
14+
- API documentation: https://sensors.bgs.ac.uk/api.html
15+
- SensorThings root: https://sensors.bgs.ac.uk/FROST-Server/v1.1
16+
- OpenAPI endpoint: https://sensors.bgs.ac.uk/FROST-Server/v1.1/api
17+
- Interactive docs: https://sensors-docs.bgs.ac.uk/
18+
19+
The service is an OGC SensorThings API v1.1 endpoint backed by FROST Server. It exposes Things, Locations, Datastreams, ObservedProperties, Sensors, FeaturesOfInterest, Observations, and MultiDatastreams.
20+
21+
## Access And Usage
22+
23+
The probed SensorThings endpoints responded publicly without authentication. Curated datastreams are restricted to records whose source metadata reports:
24+
25+
- `access_restriction`: `Unrestricted`
26+
- `data_usage`: available under the Open Government Licence with BGS/UKRI acknowledgement language
27+
28+
The publisher will preserve the source `data_usage` text and include OGL/BGS documentation links in SensorML metadata. The curated sidecar normalizes the source acknowledgement text to ASCII as `Contains UKRI materials (c) UKRI [year]`.
29+
30+
## Live Probe Results
31+
32+
Verified endpoints:
33+
34+
- `https://sensors.bgs.ac.uk/FROST-Server/v1.1` returned SensorThings collection links.
35+
- `https://sensors.bgs.ac.uk/FROST-Server/v1.1/Things?$top=5` returned BGS groundwater logger Things with rich properties.
36+
- `https://sensors.bgs.ac.uk/FROST-Server/v1.1/Datastreams?$top=5&$expand=Thing,ObservedProperty,Sensor` returned expanded datastream metadata, units, observed areas, phenomenon time ranges, access restriction, and data usage.
37+
- `https://sensors.bgs.ac.uk/FROST-Server/v1.1/Observations?$top=80&$orderby=phenomenonTime desc&$expand=Datastream($expand=Thing,ObservedProperty)` returned recent numeric observations from downhole hydro loggers.
38+
39+
Recent curated observation candidates included April 2026 measurements from:
40+
41+
- Downhole hydro logger GGA01
42+
- Downhole hydro logger GGA05
43+
- Downhole hydro logger GGA07
44+
45+
## Curated First Pass
46+
47+
Use three UKGEOS Glasgow downhole hydro loggers:
48+
49+
| Thing ID | Thing | Coordinates | Selected datastreams |
50+
| --- | --- | --- | --- |
51+
| 195 | Downhole hydro logger GGA01 | -4.200163, 55.839415 | water temperature, conductivity, water level maOD |
52+
| 197 | Downhole hydro logger GGA05 | -4.199619, 55.837690 | water temperature, conductivity, water level maOD |
53+
| 200 | Downhole hydro logger GGA07 | -4.201172, 55.838337 | water temperature, conductivity, water level maOD |
54+
55+
The selected set keeps the demo small, uses consistent observed properties across stations, and shows geothermal/groundwater telemetry rather than another generic environmental station network.
56+
57+
## CSAPI Model
58+
59+
Pattern: station-network model based on `publishers/environment_agency_hydrology` and `publishers/usgs_water`, with strict bootstrap practices from `publishers/aviation_wx` and `publishers/bootstrap_helpers.py`.
60+
61+
- Procedure: one BGS SensorThings telemetry ingestion procedure.
62+
- Systems: one CSAPI system per BGS SensorThings Thing.
63+
- Datastreams: one CSAPI datastream per selected BGS SensorThings Datastream.
64+
- Deployments: one top-level BGS SensorThings demo deployment, one UKGEOS Glasgow grouping deployment, and one station deployment per logger.
65+
- Observations: one CSAPI observation per latest changed SensorThings observation.
66+
67+
Observation result fields:
68+
69+
- `thingId`
70+
- `sourceThingId`
71+
- `sourceDatastreamId`
72+
- `observedProperty`
73+
- source-specific numeric field, such as `waterTemperatureC`, `conductivityUsCm`, or `waterLevelMaodM`
74+
- `unit`
75+
- `sourceObservationId`
76+
- `sourceUrl`
77+
78+
## Runtime Strategy
79+
80+
- Load curated Things and Datastreams from `publishers/bgs_sensorthings/stations.json`.
81+
- Fetch latest observation via `/Datastreams({id})/Observations?$top=1&$orderby=phenomenonTime desc`.
82+
- Normalize SensorThings phenomenon/result timestamps to UTC ISO strings.
83+
- Preserve source observation ID and `parameters.publish_yn`/`parameters.sen_id` when present.
84+
- Dedupe by `datastreamId|sourceObservationId|phenomenonTime|value` during a running process.
85+
- Support `--dry-run`, `--once`, `--interval`, and `--stations`.
86+
87+
## Explorer Readiness
88+
89+
Expected first-pass Explorer behavior should show three BGS deployed systems near Glasgow. If generic classification is weak, add Explorer symbol/role rules for BGS, SensorThings, UKGEOS, groundwater, and geothermal telemetry after the first live publish.
90+
91+
No station-specific image source was found during initial probing. If the card needs a visual anchor, use only accurately attributed representative UKGEOS/BGS media after separate image research, or leave the card without an image rather than implying a station-specific photograph.
92+
93+
## Validation Checklist
94+
95+
1. Compile the new Python modules.
96+
2. Run dry-run publisher fetches.
97+
3. Bootstrap to live OSH with `--force-sml`.
98+
4. Run one live publish cycle.
99+
5. Read back systems, datastreams, and observations from CSAPI.
100+
6. Verify Explorer map/card visibility on the production OSH preset.
101+
7. Document completion and any Explorer polish changes.

‎publishers/README.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@ server (e.g. [OpenSensorHub](https://opensensorhub.org/)).
2020
| **USGS EQ** | USGS Earthquake Hazards | 60 s |
2121
| **Environment Agency Hydrology** | EA river level, flow, rainfall, groundwater | 15 min |
2222
| **UK-AIR** | Defra UK-AIR NO2, O3, PM10, PM2.5 | 1 h |
23+
| **BGS SensorThings** | BGS UKGEOS Glasgow groundwater/geothermal telemetry | 15 min |
2324

2425
## Quick Start
2526

@@ -57,6 +58,7 @@ python -m publishers.usgs_nims.bootstrap_usgs_nims
5758
python -m publishers.usgs_eq.bootstrap_usgs_eq
5859
python -m publishers.environment_agency_hydrology.bootstrap_environment_agency_hydrology
5960
python -m publishers.uk_air.bootstrap_uk_air
61+
python -m publishers.bgs_sensorthings.bootstrap_bgs_sensorthings
6062
python -m publishers.iss.bootstrap_iss
6163
```
6264

@@ -126,4 +128,6 @@ python -m publishers.nws.nws_publisher --interval 3600
126128
and polls only the curated measures in `stations.json`.
127129
- **UK-AIR** uses public OGL SOS / 52 North Timeseries REST endpoints and polls
128130
only the curated pollutant timeseries in `stations.json`.
131+
- **BGS SensorThings** uses the public BGS Sensor Data Service SensorThings API
132+
and polls only curated unrestricted UKGEOS Glasgow datastreams in `stations.json`.
129133
- All publishers use `--interval <seconds>` and `--dry-run` CLI flags.

‎publishers/bgs_sensorthings/README.md‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
# BGS SensorThings Publisher
2+
3+
Publishes a curated subset of British Geological Survey Sensor Data Service observations into a CSAPI/OSH server.
4+
5+
The source is the public BGS SensorThings API v1.1 endpoint at:
6+
7+
- https://sensors.bgs.ac.uk/FROST-Server/v1.1
8+
9+
## Initial Scope
10+
11+
The first pass uses three UKGEOS Glasgow downhole hydro loggers:
12+
13+
| Site | SensorThings Thing | Selected datastreams |
14+
| --- | --- | --- |
15+
| `gga01-03` | `195` | water temperature, conductivity, water level maOD |
16+
| `gga05-03` | `197` | water temperature, conductivity, water level maOD |
17+
| `gga07-03` | `200` | water temperature, conductivity, water level maOD |
18+
19+
Curated source records are restricted to BGS datastreams that reported unrestricted access and Open Government Licence / UKRI acknowledgement usage text during probing.
20+
21+
## Bootstrap
22+
23+
```bash
24+
py -m publishers.bgs_sensorthings.bootstrap_bgs_sensorthings --dry-run
25+
py -m publishers.bgs_sensorthings.bootstrap_bgs_sensorthings --force-sml
26+
```
27+
28+
Use `--clean`, `--clean-only`, and `--force-sml` with the same semantics as the other publisher bootstraps.
29+
30+
## Publish
31+
32+
```bash
33+
py -m publishers.bgs_sensorthings.bgs_sensorthings_publisher --dry-run --once
34+
py -m publishers.bgs_sensorthings.bgs_sensorthings_publisher --once
35+
```
36+
37+
Options:
38+
39+
- `--interval <seconds>`: polling interval, default `900`.
40+
- `--stations gga01-03,gga05-03`: restrict to selected curated site IDs or source Thing IDs.
41+
- `--dry-run`: fetch and normalize source observations without posting.
42+
- `--once`: run one cycle and exit.
43+
44+
## CSAPI Model
45+
46+
- Procedure: `urn:os4csapi:procedure:bgs-sensorthings:v1`
47+
- Systems: one per curated BGS SensorThings Thing.
48+
- Datastreams: one per selected BGS SensorThings Datastream.
49+
- Deployments: one BGS SensorThings demo root, one UKGEOS Glasgow group, and one deployment per hydro logger.
50+
51+
Observation results preserve:
52+
53+
- curated Thing/site ID
54+
- source SensorThings Thing ID
55+
- source SensorThings Datastream ID
56+
- observed property
57+
- numeric value and unit
58+
- source Observation ID
59+
- source publish flag when present
60+
- latest-observation source URL
61+
62+
## Source Notes
63+
64+
BGS Sensor Data Service exposes SensorThings collections including Things, Locations, Datastreams, ObservedProperties, Sensors, FeaturesOfInterest, Observations, and MultiDatastreams. The curated publisher deliberately avoids BGS event datastreams and records with restrictive access metadata.
65+
66+
License and attribution should follow the source `data_usage` text. The curated records used here state Open Government Licence availability with UKRI acknowledgement language: `Contains UKRI materials (c) UKRI [year]`.

0 commit comments

Comments
 (0)