Skip to content

Commit 6842db8

Browse files
Sam-BollingSam-Bolling
committed
Harden publisher secret deployment
1 parent bf5866e commit 6842db8

12 files changed

Lines changed: 223 additions & 21 deletions

‎deploy_go_publishers.sh‎

Lines changed: 16 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,7 @@ declare -A GO_PUBS=(
5151
[usgs-water-publisher-go]=usgs_water
5252
[usgs-nims-publisher-go]=usgs_nims
5353
[iss-publisher-go]=iss
54+
[met-office-datahub-publisher-go]=met_office_datahub
5455
)
5556

5657
for dir_name in "${!GO_PUBS[@]}"; do
@@ -73,12 +74,14 @@ fi
7374
echo ""
7475
echo "=== Running bootstraps against Go server ==="
7576

76-
# Run each bootstrap against the Go server
77-
export OSH_ADDRESS=129-80-248-53.sslip.io
78-
export OSH_PORT=443
79-
export OSH_USER=os4csapi
80-
export OSH_PASS=ogc134mm
81-
export OSH_BASE_URL=$GO_URL
77+
# Run each bootstrap against the Go server. OSH_PASS is intentionally required
78+
# from the caller's environment so server credentials are not stored in git.
79+
export OSH_ADDRESS=${OSH_ADDRESS:-129-80-248-53.sslip.io}
80+
export OSH_PORT=${OSH_PORT:-443}
81+
export OSH_USER=${OSH_USER:-os4csapi}
82+
: "${OSH_PASS:?Set OSH_PASS in the shell or a host-local environment file before running this script}"
83+
export OSH_PASS
84+
export OSH_BASE_URL=${OSH_BASE_URL:-$GO_URL}
8285

8386
# NWS bootstrap
8487
echo "-- NWS Bootstrap --"
@@ -122,4 +125,11 @@ cd $HOME_DIR/iss-publisher-go
122125
python3 -m publishers.iss.bootstrap_iss 2>&1 | tail -20
123126
echo ""
124127

128+
# Met Office DataHub bootstrap (requires MET_OFFICE_LAND_OBSERVATIONS_API_KEY
129+
# only for publisher runtime/probe, not for resource bootstrap)
130+
echo "-- Met Office DataHub Bootstrap --"
131+
cd $HOME_DIR/met-office-datahub-publisher-go
132+
python3 -m publishers.met_office_datahub.bootstrap_met_office_datahub 2>&1 | tail -20
133+
echo ""
134+
125135
echo "=== Bootstrap complete ==="

‎docs/research/CSAPI_Go_Server_Integration_Report_2026-04-17.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -430,7 +430,7 @@ The established pattern for adding Go server support to any publisher:
430430
1. Run bootstrap against Go server:
431431
```bash
432432
OSH_BASE_URL="https://129-80-248-53.sslip.io/csapi-go" \
433-
OSH_USER=dummy OSH_PASS=dummy \
433+
OSH_USER=dummy OSH_PASS=<osh-admin-password> \
434434
python -m publishers.<name>.bootstrap_<name>
435435
```
436436

@@ -440,7 +440,7 @@ The established pattern for adding Go server support to any publisher:
440440
Environment="OSH_ADDRESS=129-80-248-53.sslip.io"
441441
Environment="OSH_BASE_URL=https://129-80-248-53.sslip.io/csapi-go"
442442
Environment="OSH_USER=dummy"
443-
Environment="OSH_PASS=dummy"
443+
Environment="OSH_PASS=<osh-admin-password>"
444444
ExecStart=/usr/bin/python3 -m publishers.<name>.<name>_publisher
445445
```
446446

‎docs/research/ISS_Publisher_Refactor_Plan.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -220,7 +220,7 @@ This is perhaps the most significant long-term benefit. The refactor transforms
220220
export OSH_ADDRESS=my-osh-server.example.com
221221
export OSH_PORT=443
222222
export OSH_USER=admin
223-
export OSH_PASS=secret
223+
export OSH_PASS=<osh-admin-password>
224224
export SYSTEM_UID=urn:osh:sensor:iss-tracker
225225
export DATASTREAM_NAME="ISS Position"
226226
export NORAD_ID=25544

‎docs/research/Publisher_Fleet_Portability_Plan.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -156,7 +156,7 @@ socket.getaddrinfo = _patched_getaddrinfo
156156
OSH_ADDRESS=your-server.example.com
157157
OSH_PORT=443
158158
OSH_USER=admin
159-
OSH_PASS=changeme
159+
OSH_PASS=<osh-admin-password>
160160
OSH_ROOT=sensorhub
161161
162162
# ── Optional ──

‎docs/research/USGS_Water_Publisher_Phase1_Report.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -252,7 +252,7 @@ cd ~/OSHConnect-Python && git pull origin main
252252
python -m publishers.usgs_water.usgs_water_publisher --once
253253
254254
# Optional: set API key for higher rate limits
255-
export USGS_API_KEY=55Xjsea8288I7fnXCCGFIQMICM0ddmcvVHFT6G76
255+
export USGS_API_KEY=<usgs-api-key>
256256
257257
# Create systemd service following existing publisher pattern:
258258
# ExecStart=/path/to/venv/bin/python -m publishers.usgs_water.usgs_water_publisher --interval 900

‎docs/research/new-publisher-source-planning/Oracle_Publisher_Secret_Deployment_2026-05-26.md‎

Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
# Oracle Publisher Secret Deployment
2+
3+
This is the acceptable production pattern for API keys used by live publishers on the Oracle host.
4+
5+
## Principle
6+
7+
Publisher code must stay configurable and secret-free. The repository may contain variable names, `.env.example` placeholders, Docker Compose interpolation, and systemd instructions, but not raw provider keys.
8+
9+
The live Oracle host should inject keys through one of these host-local mechanisms:
10+
11+
- systemd `EnvironmentFile=` with mode `0600`, owned by `root`;
12+
- service-specific root-owned key files referenced by `*_API_KEY_FILE` environment variables;
13+
- Docker Compose `.env` on the host, also mode `0600`, if the publisher is run via Compose.
14+
15+
## Current Key-Backed Publishers
16+
17+
| Publisher | Required? | Variable |
18+
| --- | --- | --- |
19+
| USGS Water | Optional, improves rate limits | `USGS_API_KEY` |
20+
| USGS NIMS | Optional, improves rate limits | `USGS_API_KEY` |
21+
| Met Office Land Observations | Required | `MET_OFFICE_LAND_OBSERVATIONS_API_KEY` |
22+
23+
The Oracle deploy/bootstrap script also expects `OSH_PASS` from the caller's environment or a host-local environment file. Server credentials should follow the same rule as provider API keys: never commit the raw value.
24+
25+
Met Office also supports `MET_OFFICE_LAND_OBSERVATIONS_API_KEY_FILE`, which should point at a host-local file containing the raw key or an assignment line.
26+
27+
## systemd Pattern
28+
29+
Create a shared environment file on Oracle:
30+
31+
```bash
32+
sudo install -d -m 700 -o root -g root /etc/os4csapi
33+
sudo install -m 600 -o root -g root /dev/null /etc/os4csapi/publisher-secrets.env
34+
sudoedit /etc/os4csapi/publisher-secrets.env
35+
```
36+
37+
Example contents, with placeholders only:
38+
39+
```text
40+
USGS_API_KEY=<usgs-key-if-used>
41+
MET_OFFICE_LAND_OBSERVATIONS_API_KEY=<met-office-land-observations-key>
42+
OSH_PASS=<osh-admin-password>
43+
```
44+
45+
Then add a drop-in to each service that needs the keys:
46+
47+
```bash
48+
sudo systemctl edit met-office-datahub-publisher-go
49+
```
50+
51+
Drop-in contents:
52+
53+
```ini
54+
[Service]
55+
EnvironmentFile=/etc/os4csapi/publisher-secrets.env
56+
```
57+
58+
Reload and restart:
59+
60+
```bash
61+
sudo systemctl daemon-reload
62+
sudo systemctl restart met-office-datahub-publisher-go
63+
sudo journalctl -u met-office-datahub-publisher-go -n 80 --no-pager
64+
```
65+
66+
For USGS Water and USGS NIMS, use the same `EnvironmentFile=` drop-in on `usgs-water-publisher-go` and `usgs-nims-publisher-go` when the API key is available.
67+
68+
## Secret File Pattern
69+
70+
For a single-service Met Office secret file:
71+
72+
```bash
73+
sudo install -d -m 700 -o root -g root /etc/os4csapi/secrets
74+
sudo install -m 600 -o root -g root /dev/null /etc/os4csapi/secrets/met-office-land-observations.key
75+
sudoedit /etc/os4csapi/secrets/met-office-land-observations.key
76+
```
77+
78+
The file should contain only the key or this assignment:
79+
80+
```text
81+
MET_OFFICE_LAND_OBSERVATIONS_API_KEY=<met-office-land-observations-key>
82+
```
83+
84+
The service environment then uses:
85+
86+
```ini
87+
[Service]
88+
Environment=MET_OFFICE_LAND_OBSERVATIONS_API_KEY_FILE=/etc/os4csapi/secrets/met-office-land-observations.key
89+
```
90+
91+
Standalone publisher runs may also point at a shared host-local env file:
92+
93+
```ini
94+
[Service]
95+
Environment=PUBLISHERS_ENV_FILE=/etc/os4csapi/publisher-secrets.env
96+
```
97+
98+
## Docker Compose Pattern
99+
100+
When running `publishers/docker-compose.yml` on Oracle, store keys in the host-local `publishers/.env` file. That file is ignored by git and must not be copied into commits or support bundles.
101+
102+
```text
103+
USGS_API_KEY=<usgs-key-if-used>
104+
MET_OFFICE_LAND_OBSERVATIONS_API_KEY=<met-office-land-observations-key>
105+
```
106+
107+
Met Office is an opt-in access-gated Compose service:
108+
109+
```bash
110+
docker compose --profile access-gated up -d met-office-datahub
111+
```

‎publishers/.env.example‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,9 @@ OSH_ROOT=sensorhub
1010
# Full bootstrap URL (derived from OSH_ADDRESS/PORT/ROOT if unset)
1111
# BOOTSTRAP_URL=https://your-server.example.com/sensorhub/api
1212

13+
# Optional local env-file override for standalone publisher runs.
14+
# PUBLISHERS_ENV_FILE=/etc/os4csapi/publisher-secrets.env
15+
1316
# Force DNS resolution to a specific IP (useful behind NAT / split DNS)
1417
# OSH_FORCE_IP=10.0.0.5
1518

@@ -20,6 +23,10 @@ OSH_ROOT=sensorhub
2023
# Required for publishers.met_office_datahub.
2124
# MET_OFFICE_LAND_OBSERVATIONS_API_KEY=
2225

26+
# Optional alternative for systemd/Oracle deployments: point to a root-owned
27+
# file containing either the raw key or MET_OFFICE_LAND_OBSERVATIONS_API_KEY=...
28+
# MET_OFFICE_LAND_OBSERVATIONS_API_KEY_FILE=/etc/os4csapi/secrets/met-office-land-observations.key
29+
2330
# Optional override if Met Office changes/publishes a different gateway URL.
2431
# MET_OFFICE_LAND_OBSERVATIONS_BASE_URL=https://data.hub.api.metoffice.gov.uk/observation-land/1
2532

‎publishers/README.md‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -71,6 +71,7 @@ python -m publishers.iss.bootstrap_iss
7171
cd publishers
7272
docker compose up -d # start all
7373
docker compose up -d nws # start one
74+
docker compose --profile access-gated up -d met-office-datahub
7475
docker compose logs -f nws # follow logs
7576
docker compose ps # status
7677
docker compose down # stop all
@@ -115,9 +116,11 @@ python -m publishers.nws.nws_publisher --interval 3600
115116
| `OSH_PASS` | **Yes** | Auth password |
116117
| `OSH_ROOT` | **Yes** | Server root path (usually `sensorhub`) |
117118
| `BOOTSTRAP_URL` | No | Override the full bootstrap API URL |
119+
| `PUBLISHERS_ENV_FILE` | No | Standalone runtime override for the env file loaded by publishers that support local env loading |
118120
| `OSH_FORCE_IP` | No | Force DNS resolution to a specific IP |
119121
| `USGS_API_KEY` | No | USGS API key for higher rate limits |
120122
| `MET_OFFICE_LAND_OBSERVATIONS_API_KEY` | For Met Office | Met Office Weather DataHub Land Observations subscription key |
123+
| `MET_OFFICE_LAND_OBSERVATIONS_API_KEY_FILE` | For Met Office | Host-local file containing the Land Observations key; useful for Oracle/systemd secrets |
121124
| `BUOYCAM_CACHE_ROOT` | No | Local directory for BuoyCAM image cache |
122125
| `BUOYCAM_CACHE_BASE_URL` | No | Public URL serving the cached images |
123126

@@ -135,6 +138,9 @@ python -m publishers.nws.nws_publisher --interval 3600
135138
- **BGS SensorThings** uses the public BGS Sensor Data Service SensorThings API
136139
and polls only curated unrestricted UKGEOS Glasgow datastreams in `stations.json`.
137140
- **Met Office DataHub** uses the access-gated Land Observations API and reads
138-
`MET_OFFICE_LAND_OBSERVATIONS_API_KEY` from the environment. It caches nearest
139-
geohash lookups in ignored runtime state to stay within the free-plan call budget.
141+
`MET_OFFICE_LAND_OBSERVATIONS_API_KEY` from the environment, or
142+
`MET_OFFICE_LAND_OBSERVATIONS_API_KEY_FILE` from a host-local secret file. It
143+
caches nearest geohash lookups in ignored runtime state to stay within the
144+
free-plan call budget. In Docker Compose it is behind the `access-gated`
145+
profile, so public publishers can still start without a Met Office key.
140146
- All publishers use `--interval <seconds>` and `--dry-run` CLI flags.

‎publishers/docker-compose.yml‎

Lines changed: 18 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -101,7 +101,7 @@ services:
101101
restart: always
102102
environment:
103103
<<: *osh-env
104-
# USGS_API_KEY: "${USGS_API_KEY:-}"
104+
USGS_API_KEY: "${USGS_API_KEY:-}"
105105
command: ["--interval", "900"]
106106

107107
# ── USGS NIMS Imagery (15min cadence) ──
@@ -112,9 +112,25 @@ services:
112112
restart: always
113113
environment:
114114
<<: *osh-env
115-
# USGS_API_KEY: "${USGS_API_KEY:-}"
115+
USGS_API_KEY: "${USGS_API_KEY:-}"
116116
command: ["--interval", "900"]
117117

118+
# ── Met Office Weather DataHub Land Observations (1h cadence) ──
119+
met-office-datahub:
120+
profiles: ["access-gated"]
121+
build:
122+
context: ..
123+
dockerfile: publishers/met_office_datahub/Dockerfile
124+
restart: always
125+
environment:
126+
<<: *osh-env
127+
MET_OFFICE_LAND_OBSERVATIONS_API_KEY: ${MET_OFFICE_LAND_OBSERVATIONS_API_KEY:-}
128+
MET_OFFICE_LAND_OBSERVATIONS_BASE_URL: ${MET_OFFICE_LAND_OBSERVATIONS_BASE_URL:-https://data.hub.api.metoffice.gov.uk/observation-land/1}
129+
MET_OFFICE_DATAHUB_API_KEY_HEADER: ${MET_OFFICE_DATAHUB_API_KEY_HEADER:-apikey}
130+
MET_OFFICE_DATAHUB_REQUEST_DELAY: ${MET_OFFICE_DATAHUB_REQUEST_DELAY:-1.0}
131+
MET_OFFICE_DATAHUB_429_BACKOFF: ${MET_OFFICE_DATAHUB_429_BACKOFF:-3600}
132+
command: ["--interval", "3600"]
133+
118134
# ── USGS Earthquake Feed (60s cadence) ──
119135
usgs-eq:
120136
build:

‎publishers/met_office_datahub/Dockerfile‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
FROM python:3.12-slim
2+
3+
WORKDIR /app
4+
5+
RUN pip install --no-cache-dir git+https://github.com/OS4CSAPI/OSHConnect-Python.git
6+
7+
COPY publishers/ /app/publishers/
8+
9+
ENV OSH_ADDRESS=
10+
ENV OSH_PORT=443
11+
ENV OSH_USER=
12+
ENV OSH_PASS=
13+
ENV OSH_ROOT=sensorhub
14+
ENV MET_OFFICE_LAND_OBSERVATIONS_API_KEY=
15+
ENV MET_OFFICE_LAND_OBSERVATIONS_API_KEY_FILE=
16+
ENV MET_OFFICE_LAND_OBSERVATIONS_BASE_URL=https://data.hub.api.metoffice.gov.uk/observation-land/1
17+
ENV MET_OFFICE_DATAHUB_API_KEY_HEADER=apikey
18+
19+
ENTRYPOINT ["python", "-m", "publishers.met_office_datahub.met_office_datahub_publisher"]
20+
CMD ["--interval", "3600"]

0 commit comments

Comments
 (0)