Skip to content

Commit 8332972

Browse files
Sam-BollingSam-Bolling
committed
Add UK-AIR publisher
1 parent d5f2303 commit 8332972

8 files changed

Lines changed: 1172 additions & 0 deletions

File tree

‎docs/research/new-publisher-source-planning/UK_AIR_Publisher_Completion_Report_2026-05-26.md‎

Lines changed: 133 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,133 @@
1+
# UK-AIR Publisher Completion Report - 2026-05-26
2+
3+
## Summary
4+
5+
Implemented the first-pass UK-AIR publisher package for OSHConnect-Python and bootstrapped it into the live OSH SensorHub backend used by the CSAPI Explorer `OSH (OS4CSAPI)` preset.
6+
7+
The new publisher models three curated UK-AIR monitoring sites and four pollutant timeseries:
8+
9+
| Site | Pollutant | UK-AIR timeseries | Datastream |
10+
| --- | --- | ---: | --- |
11+
| Camden Kerbside | NO2 | 3 | `ukAirNO2` |
12+
| Auchencorth Moss | O3 | 24 | `ukAirO3` |
13+
| Toft Newton | PM10 | 5125 | `ukAirPM10` |
14+
| Toft Newton | PM2.5 | 5130 | `ukAirPM25` |
15+
16+
## Files Added
17+
18+
- `publishers/uk_air/__init__.py`
19+
- `publishers/uk_air/stations.json`
20+
- `publishers/uk_air/README.md`
21+
- `publishers/uk_air/bootstrap_uk_air.py`
22+
- `publishers/uk_air/uk_air_publisher.py`
23+
24+
## Files Updated
25+
26+
- `publishers/README.md`
27+
- Added UK-AIR to the publisher fleet table.
28+
- Added the UK-AIR bootstrap command.
29+
- Added source/API notes for the UK-AIR SOS / 52 North Timeseries REST API.
30+
31+
## Implementation Notes
32+
33+
- Source API: `https://uk-air.defra.gov.uk/sos-ukair/api/v1/`
34+
- Timeseries list endpoint: `https://uk-air.defra.gov.uk/sos-ukair/api/v1/timeseries`
35+
- Recent data endpoint pattern: `timeseries/{id}/getData?timespan=PT72H/{utcEnd}`
36+
- UK-AIR exposes point coordinates as `[lat, lon, alt]`; `stations.json` preserves normalized `lat` and `lon`, and the bootstrap writes standard GeoJSON ordering as `[lon, lat]`.
37+
- UK-AIR timestamps are millisecond Unix epoch values; the runtime normalizes them to CSAPI UTC phenomenon times.
38+
- UK-AIR source units are exposed as `ug.m-3`; the CSAPI observation payload uses display text `ug/m3` while preserving pollutant-specific result fields such as `no2_ugm3` and `pm25_ugm3`.
39+
- Runtime filtering ignores missing, `NaN`, and sentinel `<= -99` values.
40+
41+
## Live Bootstrap Result
42+
43+
Command:
44+
45+
```powershell
46+
py -m publishers.uk_air.bootstrap_uk_air
47+
```
48+
49+
Result:
50+
51+
- Created procedure `urn:os4csapi:procedure:uk-air:v1` with server id `04dg`.
52+
- Created systems:
53+
- `urn:os4csapi:system:uk-air:camden-kerbside:v1` -> `05lg`
54+
- `urn:os4csapi:system:uk-air:auchencorth-moss:v1` -> `05m0`
55+
- `urn:os4csapi:system:uk-air:toft-newton:v1` -> `05mg`
56+
- Created datastreams:
57+
- `ukAirNO2` -> `05kg`
58+
- `ukAirO3` -> `05l0`
59+
- `ukAirPM10` -> `05lg`
60+
- `ukAirPM25` -> `05m0`
61+
- Created deployments:
62+
- `urn:os4csapi:deployment:uk-air-demo:v1` -> `05g0`
63+
- `urn:os4csapi:deployment:uk-air-stations:v1` -> `05gg`
64+
- `urn:os4csapi:deployment:uk-air-camden-kerbside:v1` -> `05h0`
65+
- `urn:os4csapi:deployment:uk-air-auchencorth-moss:v1` -> `05hg`
66+
- `urn:os4csapi:deployment:uk-air-toft-newton:v1` -> `05i0`
67+
68+
The bootstrap helper stripped unsupported datastream `documentation` and `uid` fields before POST. This matches current server behavior and kept the operational datastream creation path clean.
69+
70+
## Live Publish Result
71+
72+
Initial live publish found that the server-accepted datastream schema did not include the runtime `timestamp` field in the observation result. The runtime payload was adjusted so `stationId` is the first result field, matching the schema accepted by OSH.
73+
74+
Command after the adjustment:
75+
76+
```powershell
77+
py -m publishers.uk_air.uk_air_publisher --once
78+
```
79+
80+
Result:
81+
82+
- Published: 4
83+
- Skipped: 0
84+
- Errors: 0
85+
86+
Published readings:
87+
88+
| Site / stream | Phenomenon time | Value |
89+
| --- | --- | --- |
90+
| Camden Kerbside `ukAirNO2` | `2026-05-26T09:00:00Z` | `NO2=27.731 ug/m3` |
91+
| Auchencorth Moss `ukAirO3` | `2026-05-26T09:00:00Z` | `O3=67.854 ug/m3` |
92+
| Toft Newton `ukAirPM10` | `2026-05-26T09:00:00Z` | `PM10=45.1 ug/m3` |
93+
| Toft Newton `ukAirPM25` | `2026-05-26T09:00:00Z` | `PM2.5=18.396 ug/m3` |
94+
95+
## Server Read-Back Verification
96+
97+
Latest-observation read-back from live OSH confirmed the stored results:
98+
99+
| Datastream id | Result field | Value | Phenomenon time |
100+
| --- | --- | ---: | --- |
101+
| `05kg` | `no2_ugm3` | `27.731` | `2026-05-26T09:00:00Z` |
102+
| `05l0` | `o3_ugm3` | `67.854` | `2026-05-26T09:00:00Z` |
103+
| `05lg` | `pm10_ugm3` | `45.1` | `2026-05-26T09:00:00Z` |
104+
| `05m0` | `pm25_ugm3` | `18.396` | `2026-05-26T09:00:00Z` |
105+
106+
## Explorer-Facing Visibility Check
107+
108+
The Explorer-facing OSH endpoint at `https://129-80-248-53.sslip.io/sensorhub/api` returned the new UK-AIR resources by UID.
109+
110+
Verified examples:
111+
112+
- `systems?uid=urn:os4csapi:system:uk-air:camden-kerbside:v1&limit=1000` returned system id `05lg`.
113+
- `deployments?uid=urn:os4csapi:deployment:uk-air-demo:v1&limit=1000` returned one item.
114+
- `deployments?uid=urn:os4csapi:deployment:uk-air-stations:v1&limit=1000` returned one item.
115+
116+
## Validation Commands
117+
118+
```powershell
119+
py -m py_compile publishers\uk_air\bootstrap_uk_air.py publishers\uk_air\uk_air_publisher.py
120+
py -m publishers.uk_air.uk_air_publisher --dry-run --once
121+
py -m publishers.uk_air.bootstrap_uk_air --dry-run
122+
py -m publishers.uk_air.bootstrap_uk_air
123+
py -m publishers.uk_air.uk_air_publisher --once
124+
```
125+
126+
All final validation commands completed successfully.
127+
128+
## Follow-Up Items
129+
130+
1. Open the production Explorer map with the `OSH (OS4CSAPI)` preset and visually confirm marker styling, side-card labels, and popup latest-reading display for UK-AIR deployments.
131+
2. If the marker falls back to a blank/generic NATO symbol, add a targeted UK-AIR / air-quality keyword mapping in the Explorer symbol mapper.
132+
3. Consider adding a representative monitoring-station image/fallback after the visual pass, following the Environment Agency Hydrology pattern.
133+
4. Consider expanding the curated sidecar after demo validation, especially for additional urban NO2/PM sites and rural background stations.

‎docs/research/new-publisher-source-planning/UK_AIR_Publisher_Implementation_Plan_2026-05-26.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,8 @@
22

33
Date: 2026-05-26
44

5+
Status update: first-pass implementation, live bootstrap, live publish, and server read-back were completed on 2026-05-26. See `UK_AIR_Publisher_Completion_Report_2026-05-26.md` for validation details. Remaining follow-up is production Explorer visual review for marker styling, side-card polish, and any representative image decision.
6+
57
## Purpose
68

79
This plan defines the second new publisher activity from the candidate-source triage: a UK-AIR air pollution publisher for OSHConnect-Python. The goal is to add a curated, public, standards-aligned air-quality station network to the OS4CSAPI demo, using the same disciplined workflow proven by Environment Agency Hydrology.

‎publishers/README.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@ server (e.g. [OpenSensorHub](https://opensensorhub.org/)).
1919
| **USGS NIMS** | USGS NIMS Camera Imagery | 15 min |
2020
| **USGS EQ** | USGS Earthquake Hazards | 60 s |
2121
| **Environment Agency Hydrology** | EA river level, flow, rainfall, groundwater | 15 min |
22+
| **UK-AIR** | Defra UK-AIR NO2, O3, PM10, PM2.5 | 1 h |
2223

2324
## Quick Start
2425

@@ -55,6 +56,7 @@ python -m publishers.usgs_water.bootstrap_usgs_water
5556
python -m publishers.usgs_nims.bootstrap_usgs_nims
5657
python -m publishers.usgs_eq.bootstrap_usgs_eq
5758
python -m publishers.environment_agency_hydrology.bootstrap_environment_agency_hydrology
59+
python -m publishers.uk_air.bootstrap_uk_air
5860
python -m publishers.iss.bootstrap_iss
5961
```
6062

@@ -122,4 +124,6 @@ python -m publishers.nws.nws_publisher --interval 3600
122124
- **USGS Water / NIMS** benefit from an optional `USGS_API_KEY`.
123125
- **Environment Agency Hydrology** uses public OGL Hydrology API JSON endpoints
124126
and polls only the curated measures in `stations.json`.
127+
- **UK-AIR** uses public OGL SOS / 52 North Timeseries REST endpoints and polls
128+
only the curated pollutant timeseries in `stations.json`.
125129
- All publishers use `--interval <seconds>` and `--dry-run` CLI flags.

‎publishers/uk_air/README.md‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
# UK-AIR Publisher
2+
3+
Publishes a curated set of recent Defra UK-AIR air pollution readings into a CSAPI/OSH server.
4+
5+
The first-pass sidecar intentionally covers a small demo-safe set of pollutant products:
6+
7+
- nitrogen dioxide (`NO2`),
8+
- ozone (`O3`),
9+
- particulate matter (`PM10`),
10+
- fine particulate matter (`PM2.5`).
11+
12+
Source data comes from the UK-AIR SOS / 52 North Timeseries REST API:
13+
14+
- https://uk-air.defra.gov.uk/data/about_sos
15+
- https://uk-air.defra.gov.uk/data/sos/static/doc/api-doc/
16+
- https://uk-air.defra.gov.uk/sos-ukair/api/v1/
17+
18+
## Bootstrap
19+
20+
```bash
21+
python -m publishers.uk_air.bootstrap_uk_air --dry-run
22+
python -m publishers.uk_air.bootstrap_uk_air
23+
```
24+
25+
The bootstrap is idempotent and creates one procedure, one system per curated monitoring site, one datastream per curated pollutant timeseries, and a deployment hierarchy.
26+
27+
## Run
28+
29+
```bash
30+
python -m publishers.uk_air.uk_air_publisher --dry-run --once
31+
python -m publishers.uk_air.uk_air_publisher --once
32+
python -m publishers.uk_air.uk_air_publisher --interval 3600
33+
```
34+
35+
Use `--stations` with comma-separated curated site IDs, such as `camden-kerbside,toft-newton`, to publish a subset.
36+
37+
## Notes
38+
39+
- UK-AIR timeseries coordinates are exposed as `[lat, lon, alt]`; the bootstrap normalizes them to standard GeoJSON `[lon, lat]`.
40+
- Normal polling reads a bounded recent window from `timeseries/{id}/getData` and publishes the latest valid value.
41+
- Source timestamps are millisecond Unix epoch values and are normalized to UTC CSAPI phenomenon times.
42+
- Source units such as `ug.m-3` are preserved in observation results as `unit`, while SWE schemas use the display-friendly `ug/m3` code.
43+
- The first-pass model consolidates co-located pollutant streams only where the source metadata clearly supports it, such as Toft Newton PM10 and PM2.5.
44+
- Explorer cards should show latest pollutant concentrations in the same latest-reading section used by other station-style publishers.

‎publishers/uk_air/__init__.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
"""UK-AIR publisher package."""

0 commit comments

Comments
 (0)