Skip to content

Commit bf5866e

Browse files
Sam-BollingSam-Bolling
committed
Add Met Office Land Observations publisher
1 parent 61c241a commit bf5866e

8 files changed

Lines changed: 1471 additions & 17 deletions

File tree

‎docs/research/new-publisher-source-planning/Met_Office_Atmospheric_Models_API_Notes_2026-05-26.md‎

Lines changed: 186 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,186 @@
1+
# Met Office Weather DataHub API Notes
2+
3+
Date: 2026-05-26
4+
5+
## Purpose
6+
7+
Capture Met Office Weather DataHub access information and API documentation links for follow-up publisher planning.
8+
9+
This is a staging document. More source details are expected before implementation begins.
10+
11+
## Credential Status
12+
13+
An API key for the Met Office Weather DataHub Atmospheric Models free plan was provided during the working session.
14+
15+
An API key for the Met Office Weather DataHub Global Spot / Site-Specific Forecast free plan was also provided during the working session.
16+
17+
A Met Office Weather DataHub Land Observations free-plan subscription key was provided during the working session.
18+
19+
The raw key is intentionally not recorded in this repository. Store it only in a local secret location such as `.env`, shell profile, or deployment secret storage.
20+
21+
For this workspace, the preferred local file is `publishers/.env`. The repository `.gitignore` excludes `.env`, so `publishers/.env` is ignored by git.
22+
23+
Recommended local environment variable names:
24+
25+
```text
26+
MET_OFFICE_ATMOSPHERIC_API_KEY
27+
MET_OFFICE_GLOBAL_SPOT_API_KEY
28+
MET_OFFICE_LAND_OBSERVATIONS_API_KEY
29+
```
30+
31+
Do not commit the key to git, documentation, test fixtures, screenshots, logs, or issue bodies.
32+
33+
Known Atmospheric Models key context from the provided credential:
34+
35+
- Product family: Met Office Weather DataHub Atmospheric Models
36+
- Subscribed API context: `/atmospheric-models/1.0.0`
37+
- Plan/tier: free atmospheric models plan
38+
- Token type: API key
39+
40+
Known Global Spot / Site-Specific Forecast key context from the provided credential:
41+
42+
- Product family: Met Office Weather DataHub Site-Specific Forecast / Global Spot
43+
- Subscribed API context: `/sitespecific/v0`
44+
- Subscribed API name: `SiteSpecificForecast`
45+
- Plan/tier: free site-specific plan
46+
- Stated call allowance: 360 calls per day
47+
- Token type: API key
48+
49+
Known Land Observations key context:
50+
51+
- Product family: Met Office Weather DataHub Observations / Land Observations
52+
- API family: Observation Location service
53+
- Subscribed API context: `/observation-land/1`
54+
- Subscribed API name: `CDP_Observation_Land`
55+
- Plan/tier: free Land Observations plan
56+
- Stated call allowance: 360 calls per day
57+
- Token type: API key
58+
59+
## Documentation Links
60+
61+
- Atmospheric overview: https://datahub.metoffice.gov.uk/docs/f/category/atmospheric/overview
62+
- Weather DataHub docs root: https://datahub.metoffice.gov.uk/docs
63+
- Weather DataHub glossary: https://datahub.metoffice.gov.uk/docs/glossary
64+
- Land Observations overview: https://datahub.metoffice.gov.uk/docs/g/category/observations/overview
65+
- Land Observations API documentation: https://datahub.metoffice.gov.uk/docs/g/category/observations/type/land-observations/api-documentation
66+
- Land Observations pricing: https://datahub.metoffice.gov.uk/pricing/observations
67+
- Weather DataHub landing page: https://datahub.metoffice.gov.uk/
68+
- Met Office support: https://datahub.metoffice.gov.uk/support/faqs
69+
- Met Office legal: https://www.metoffice.gov.uk/about-us/legal
70+
71+
## Initial Source Understanding
72+
73+
There are now three distinct Met Office Weather DataHub products under consideration.
74+
75+
### Atmospheric Models
76+
77+
Atmospheric Models are not the same product as Land Observations.
78+
79+
Atmospheric Models provide gridded model data in GRIB2 format. GRIB2 is a WMO-sponsored binary format used in meteorology for historical and forecast weather data.
80+
81+
Available model families listed in the public overview include:
82+
83+
- Global deterministic 10 km
84+
- UK deterministic 2 km standard projection
85+
- UK deterministic 2 km latitude-longitude projection
86+
- Met Office Global and Regional Ensemble Prediction System, Global 20 km
87+
- Met Office Global and Regional Ensemble Prediction System, UK 2 km
88+
89+
The Weather DataHub documentation describes an order-based workflow. An order defines the requested model, region of interest, parameters, time steps, model runs, and file delivery preferences. The system then provides retrieval URLs for generated files.
90+
91+
### Global Spot / Site-Specific Forecast
92+
93+
Global Spot is part of the Site-Specific Forecast product family. The Weather DataHub landing page describes Global Spot as a deterministic site-specific forecast product for a single specified location defined by latitude and longitude.
94+
95+
Public product text lists three GeoJSON APIs:
96+
97+
- Global hourly spot data 1.0.2
98+
- Global three-hourly spot data 1.0.2
99+
- Global daily spot data 1.0.2
100+
101+
This product looks more immediately compatible with a CSAPI scalar-observation publisher than Atmospheric Models, because it is point/location forecast data rather than gridded GRIB2 files.
102+
103+
### Land Observations
104+
105+
Land Observations is the closest fit to the original Met Office publisher plan.
106+
107+
The public documentation describes recent observational data from ground-based instruments across roughly 150 UK station locations. The API provides hourly JSON observations for the past 48 hours and currently advertises 9 parameters. Stations report 24 observations daily at 0000 through 2300 UTC.
108+
109+
The documented API flow is:
110+
111+
1. Call `GET /observation-land/1/nearest` with latitude/longitude or geohash.
112+
2. Cache the nearest land observation location returned by that call.
113+
3. Call `GET /observation-land/1/{geohash}` to retrieve observations for that land location.
114+
115+
The public docs explicitly recommend caching `nearest` endpoint results to reduce unnecessary lookups.
116+
117+
## Publisher Implications
118+
119+
Atmospheric Models are a gridded forecast/model file workflow, not a station-network observation workflow.
120+
121+
Likely CSAPI modeling options need more research:
122+
123+
1. Model-run feed pattern
124+
- One procedure for Met Office Atmospheric model retrieval.
125+
- One system representing the model/order adapter.
126+
- One datastream per configured model/order/parameter bundle.
127+
- One observation per retrieved model file, carrying file URL, model run time, forecast steps, region, parameters, and metadata.
128+
129+
2. Derived point extraction pattern
130+
- Retrieve GRIB2 files.
131+
- Extract selected variables at curated demo points or regions.
132+
- Publish scalar point observations into CSAPI.
133+
- Requires a GRIB2 parser and clearer licensing/storage review.
134+
135+
3. Raster/file product pattern
136+
- Treat model files as media/data artifacts rather than scalar observations.
137+
- Publish metadata and links to retrieved GRIB2 files.
138+
- Potentially closer to an imagery/media publisher than to `usgs_water`.
139+
140+
Global Spot likely fits a curated point-forecast pattern:
141+
142+
- One procedure for Met Office Global Spot forecast retrieval.
143+
- One system per curated forecast point or named demo location.
144+
- One datastream per selected forecast parameter/time resolution, or one structured forecast datastream per point if the API returns compact multi-parameter GeoJSON.
145+
- Observations should preserve forecast generation time, valid time, lead time, latitude/longitude, parameter names, units, and source response metadata.
146+
147+
Because Global Spot is forecast data, not observed telemetry, SensorML and card labels should avoid implying a physical deployed sensor at the forecast point.
148+
149+
Land Observations fits the existing station-network publisher model:
150+
151+
- One procedure for Met Office Land Observations ingestion.
152+
- One system per curated Met Office land observation station/location.
153+
- One datastream per selected observed parameter per station, or one structured multi-parameter datastream per station if the live response shape favors compact records.
154+
- One deployment group for the curated Met Office demo set.
155+
- Observations should preserve observation time, source geohash/location identifier, parameter names, units, values, and source response metadata allowed by the terms.
156+
157+
This is the best match for `publishers/usgs_water` and the existing EA/UK-AIR/BGS station-network pattern.
158+
159+
## Open Questions For Next Input
160+
161+
- Which Atmospheric model product/order has been selected in Weather DataHub?
162+
- What order name or retrieval URL should be used?
163+
- Which parameters, region, time steps, and model runs are included in the free-plan order?
164+
- Does the API key authorize order discovery, file listing, direct file retrieval, or all three?
165+
- Are we allowed to persist retrieved GRIB2 files or only metadata/links?
166+
- Should the publisher expose model files as CSAPI observations, extract point values, or both?
167+
- Which Python GRIB2 stack should be used if point extraction is required?
168+
- Which Global Spot endpoint shape should be used: hourly, three-hourly, daily, or a combination?
169+
- What are the exact required query parameters and authentication header for `/sitespecific/v0`?
170+
- Are Global Spot responses GeoJSON FeatureCollections, single Features, or another JSON structure?
171+
- Which curated demo locations should be used for Global Spot forecasts?
172+
- What exact auth header or query parameter does the Land Observations subscription require?
173+
- What exact JSON structure does `GET /observation-land/1/{geohash}` return for live data?
174+
- Which 3-4 curated UK locations best complement the existing demo map while staying well below 360 calls/day?
175+
176+
## Current Recommendation
177+
178+
Stand by for the additional Met Office details before implementing code.
179+
180+
Do not reuse the Land Observations station-network plan for Atmospheric Models without revision. Atmospheric Models need a model-file/order-based design, likely closer to an event/feed or media artifact publisher than a fixed physical station network.
181+
182+
Updated assessment: Land Observations is now the best immediate Met Office publisher candidate because it is real observed telemetry from maintained physical stations, has a station-network model that matches our CSAPI pattern, uses JSON, and has a known free-plan call limit of 360 calls per day.
183+
184+
Global Spot should remain the second Met Office target. It is useful and likely easier than Atmospheric Models, but it is forecast data rather than observed telemetry, so it needs different labeling and modeling.
185+
186+
Atmospheric Models should remain deferred until we explicitly want a GRIB2/model-file publisher.

‎docs/research/new-publisher-source-planning/Met_Office_DataHub_Publisher_Implementation_Plan_2026-05-26.md‎

Lines changed: 34 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ Date: 2026-05-26
66

77
Start the fourth new publisher from the original candidate set using Met Office Weather DataHub Land Observations.
88

9-
This is an access-gated implementation: the publisher should not claim live readiness until a valid Weather DataHub subscription credential is available and tested.
9+
This is an access-gated implementation. A Land Observations free-plan subscription key is now available for local use, but the raw key must remain outside the repository. The publisher should not claim live readiness until that credential is stored locally and tested against the live API.
1010

1111
## Source Summary
1212

@@ -23,6 +23,14 @@ Research findings:
2323
- Access: account/subscription/API credential required
2424
- Free plan: up to 360 calls per day
2525

26+
Recommended local environment variable:
27+
28+
```text
29+
MET_OFFICE_LAND_OBSERVATIONS_API_KEY
30+
```
31+
32+
Preferred local storage for this workspace is `publishers/.env`, which is ignored by git via the repository `.gitignore`.
33+
2634
Documented API flow:
2735

2836
1. Call `GET /observation-land/1/nearest` with latitude/longitude or geohash.
@@ -59,16 +67,25 @@ Choose three locations after credentials are available and nearest-location look
5967

6068
Keep polling comfortably below the free-plan limit. For a three-station demo, hourly polling with one cached nearest lookup per station and one observations call per station per hour should be safe.
6169

70+
## Updated Plan
71+
72+
Land Observations is now the primary Met Office publisher #4 target. It is a better fit than Atmospheric Models for immediate implementation because it publishes observed weather telemetry from physical UK stations in JSON, and it matches the station-network pattern already proven by EA Hydrology, UK-AIR, and BGS SensorThings.
73+
74+
Global Spot should be treated as a follow-on forecast publisher. Atmospheric Models should remain deferred until a GRIB2/model-file product is explicitly desired.
75+
6276
## Implementation Steps
6377

64-
1. Verify credentials and exact request shape.
65-
2. Probe nearest-location lookup for candidate demo coordinates.
66-
3. Save resolved geohash/location metadata in a sidecar file.
67-
4. Implement `bootstrap_met_office_datahub.py` with minimal GeoJSON create stubs and rich SensorML updates.
68-
5. Implement `met_office_datahub_publisher.py` with `--dry-run`, `--once`, and interval controls.
69-
6. Add source docs, pricing/access notes, and terms links to SensorML documents.
70-
7. Run dry-run probes, then bootstrap, then one live publish cycle.
71-
8. Add Explorer role/symbol/thumbnail polish only after confirming source metadata and licensing for any representative image.
78+
1. Store the key locally as `MET_OFFICE_LAND_OBSERVATIONS_API_KEY`.
79+
2. Verify the exact auth header/query parameter and live base URL.
80+
3. Probe `GET /observation-land/1/nearest` for candidate demo coordinates.
81+
4. Cache nearest-location/geohash results; do not call nearest every publish cycle.
82+
5. Probe `GET /observation-land/1/{geohash}` for the resolved locations.
83+
6. Save resolved station/location metadata in a sidecar file.
84+
7. Implement `bootstrap_met_office_datahub.py` with minimal GeoJSON create stubs and rich SensorML updates.
85+
8. Implement `met_office_datahub_publisher.py` with `--dry-run`, `--once`, and interval controls.
86+
9. Add source docs, pricing/access notes, and terms links to SensorML documents.
87+
10. Run dry-run probes, then bootstrap, then one live publish cycle.
88+
11. Add Explorer role/symbol/thumbnail polish only after confirming source metadata and licensing for any representative image.
7289

7390
## Open Questions
7491

@@ -83,8 +100,15 @@ Created:
83100

84101
- `publishers/met_office_datahub/README.md`
85102
- `publishers/met_office_datahub/__init__.py`
103+
- `publishers/met_office_datahub/stations.json`
104+
- `publishers/met_office_datahub/bootstrap_met_office_datahub.py`
105+
- `publishers/met_office_datahub/met_office_datahub_publisher.py`
86106

87107
Updated:
88108

89109
- `publishers/README.md`
90-
- `Publisher_Expansion_Status_Report_2026-05-26.md`
110+
- `Publisher_Expansion_Status_Report_2026-05-26.md`
111+
112+
## Implementation Progress
113+
114+
Initial code now exists for the Land Observations publisher. It reads `MET_OFFICE_LAND_OBSERVATIONS_API_KEY` from the environment, supports the documented nearest/geohash API flow, caches resolved geohashes in ignored runtime state, and includes a `--probe` mode for safely inspecting the live subscribed response shape before bootstrap/live publishing.

‎publishers/.env.example‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,16 @@ OSH_ROOT=sensorhub
1616
# USGS API key — raises rate limits for usgs-water and usgs-nims
1717
# USGS_API_KEY=
1818

19+
# Met Office Weather DataHub Land Observations API key.
20+
# Required for publishers.met_office_datahub.
21+
# MET_OFFICE_LAND_OBSERVATIONS_API_KEY=
22+
23+
# Optional override if Met Office changes/publishes a different gateway URL.
24+
# MET_OFFICE_LAND_OBSERVATIONS_BASE_URL=https://data.hub.api.metoffice.gov.uk/observation-land/1
25+
26+
# Optional API-key header name. Default is apikey.
27+
# MET_OFFICE_DATAHUB_API_KEY_HEADER=apikey
28+
1929
# ── NDBC BuoyCAM ──────────────────────────────────────────────────────
2030
# Base URL where cached BuoyCAM JPEGs are served.
2131
# You must host a static file server (e.g. Nginx) at this URL.

‎publishers/README.md‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -61,6 +61,7 @@ python -m publishers.environment_agency_hydrology.bootstrap_environment_agency_h
6161
python -m publishers.uk_air.bootstrap_uk_air
6262
python -m publishers.bgs_sensorthings.bootstrap_bgs_sensorthings
6363
# Met Office DataHub is access-gated; see publishers/met_office_datahub/README.md
64+
python -m publishers.met_office_datahub.bootstrap_met_office_datahub
6465
python -m publishers.iss.bootstrap_iss
6566
```
6667

@@ -116,7 +117,7 @@ python -m publishers.nws.nws_publisher --interval 3600
116117
| `BOOTSTRAP_URL` | No | Override the full bootstrap API URL |
117118
| `OSH_FORCE_IP` | No | Force DNS resolution to a specific IP |
118119
| `USGS_API_KEY` | No | USGS API key for higher rate limits |
119-
| `MET_OFFICE_DATAHUB_API_KEY` | Planned | Met Office Weather DataHub subscription key for Land Observations |
120+
| `MET_OFFICE_LAND_OBSERVATIONS_API_KEY` | For Met Office | Met Office Weather DataHub Land Observations subscription key |
120121
| `BUOYCAM_CACHE_ROOT` | No | Local directory for BuoyCAM image cache |
121122
| `BUOYCAM_CACHE_BASE_URL` | No | Public URL serving the cached images |
122123

@@ -133,7 +134,7 @@ python -m publishers.nws.nws_publisher --interval 3600
133134
only the curated pollutant timeseries in `stations.json`.
134135
- **BGS SensorThings** uses the public BGS Sensor Data Service SensorThings API
135136
and polls only curated unrestricted UKGEOS Glasgow datastreams in `stations.json`.
136-
- **Met Office DataHub** is started as an access-gated publisher target for Land
137-
Observations. It requires account/subscription credentials before live runtime
138-
implementation can be completed.
137+
- **Met Office DataHub** uses the access-gated Land Observations API and reads
138+
`MET_OFFICE_LAND_OBSERVATIONS_API_KEY` from the environment. It caches nearest
139+
geohash lookups in ignored runtime state to stay within the free-plan call budget.
139140
- All publishers use `--interval <seconds>` and `--dry-run` CLI flags.

0 commit comments

Comments
 (0)