Skip to content

Commit 1da66f3

Browse files
shuuji3shuuji3
committed
test(internal/dev-infra): copy container scripts from bluesky-social/atproto repository
Source: https://github.com/bluesky-social/atproto/tree/6e382f67aa73532efadfea80ff96a27b526cb178/packages/dev-infra
1 parent 464a908 commit 1da66f3

File tree

5 files changed

+332
-0
lines changed

5 files changed

+332
-0
lines changed

‎packages/internal/dev-infra/README.md‎

Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
# dev-infra
2+
3+
Helpers for working with postgres and redis locally. Previously known as `pg`.
4+
5+
## Usage
6+
7+
### `with-test-db.sh`
8+
9+
This script allows you to run any command with a fresh, ephemeral/single-use postgres database available. When the script starts a Dockerized postgres container starts-up, and when the script completes that container is removed.
10+
11+
The environment variable `DB_POSTGRES_URL` will be set with a connection string that can be used to connect to the database. The [`PG*` environment variables](https://www.postgresql.org/docs/current/libpq-envars.html) that are recognized by libpq (i.e. used by the `psql` client) are also set.
12+
13+
**Example**
14+
15+
```
16+
$ ./with-test-db.sh psql -c 'select 1;'
17+
[+] Running 1/1
18+
⠿ Container pg-db_test-1 Healthy 1.8s
19+
20+
?column?
21+
----------
22+
1
23+
(1 row)
24+
25+
26+
[+] Running 1/1
27+
⠿ Container pg-db_test-1 Stopped 0.1s
28+
Going to remove pg-db_test-1
29+
[+] Running 1/0
30+
⠿ Container pg-db_test-1 Removed
31+
```
32+
33+
### `with-redis-and-test-db.sh`
34+
35+
This script is similar to `with-test-db.sh`, but in addition to an ephemeral/single-use postgres database it also provides a single-use redis instance. When the script starts, Dockerized postgres and redis containers start-up, and when the script completes the containers are removed.
36+
37+
The environment variables `DB_POSTGRES_URL` and `REDIS_HOST` will be set with a connection strings that can be used to connect to postgres and redis respectively.
38+
39+
### `docker-compose.yaml`
40+
41+
The Docker compose file can be used to run containerized versions of postgres either for single use (as is used by `with-test-db.sh`), or for longer-term use. These are setup as separate services named `db_test` and `db` respectively. In both cases the database is available on the host machine's `localhost` and credentials are:
42+
43+
- Username: pg
44+
- Password: password
45+
46+
However, each service uses a different port, documented below, to avoid conflicts.
47+
48+
#### `db_test` service for single use
49+
50+
The single-use `db_test` service does not have any persistent storage. When the container is removed, data in the database disappears with it.
51+
52+
This service runs on port `5433`.
53+
54+
```
55+
$ docker compose up db_test # start container
56+
$ docker compose stop db_test # stop container
57+
$ docker compose rm db_test # remove container
58+
```
59+
60+
#### `db` service for persistent use
61+
62+
The `db` service has persistent storage on the host machine managed by Docker under a volume named `pg_atp_db`. When the container is removed, data in the database will remain on the host machine. In order to start fresh, you would need to remove the volume.
63+
64+
This service runs on port `5432`.
65+
66+
```
67+
$ docker compose up db -d # start container
68+
$ docker compose stop db # stop container
69+
$ docker compose rm db # remove container
70+
$ docker volume rm pg_atp_db # remove volume
71+
```
72+
73+
#### `redis_test` service for single use
74+
75+
The single-use `redis_test` service does not have any persistent storage. When the container is removed, the data in redis disappears with it.
76+
77+
This service runs on port `6380`.
78+
79+
#### `redis` service for persistent use
80+
81+
The `redis` service has persistent storage on the host machine managed by Docker under a volume named `atp_redis`. When the container is removed, the data in redis will remain on the host machine. In order to start fresh, you would need to remove the volume.
82+
83+
This service runs on port `6379`.

‎packages/internal/dev-infra/_common.sh‎

Lines changed: 181 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,181 @@
1+
#!/usr/bin/env sh
2+
3+
# Exit if any command fails
4+
set -e
5+
6+
get_container_id() {
7+
local compose_file=$1
8+
local service=$2
9+
if [ -z "${compose_file}" ] || [ -z "${service}" ]; then
10+
echo "usage: get_container_id <compose_file> <service>"
11+
exit 1
12+
fi
13+
14+
# first line of jq normalizes for docker compose breaking change, see docker/compose#10958
15+
docker compose --file $compose_file ps --format json --status running \
16+
| jq -sc '.[] | if type=="array" then .[] else . end' | jq -s \
17+
| jq -r '.[]? | select(.Service == "'${service}'") | .ID'
18+
}
19+
20+
# Exports all environment variables
21+
export_env() {
22+
export_pg_env
23+
export_redis_env
24+
}
25+
26+
# Exports postgres environment variables
27+
export_pg_env() {
28+
# Based on creds in compose.yaml
29+
export PGPORT=5433
30+
export PGHOST=localhost
31+
export PGUSER=pg
32+
export PGPASSWORD=password
33+
export PGDATABASE=postgres
34+
export DB_POSTGRES_URL="postgresql://pg:password@127.0.0.1:5433/postgres"
35+
}
36+
37+
# Exports redis environment variables
38+
export_redis_env() {
39+
export REDIS_HOST="127.0.0.1:6380"
40+
}
41+
42+
pg_clear() {
43+
local pg_uri=$1
44+
45+
for schema_name in `psql "${pg_uri}" -c "SELECT schema_name FROM information_schema.schemata WHERE schema_name NOT LIKE 'pg_%' AND schema_name NOT LIKE 'information_schema';" -t`; do
46+
psql "${pg_uri}" -c "DROP SCHEMA \"${schema_name}\" CASCADE;"
47+
done
48+
}
49+
50+
pg_init() {
51+
local pg_uri=$1
52+
53+
psql "${pg_uri}" -c "CREATE SCHEMA IF NOT EXISTS \"public\";"
54+
}
55+
56+
redis_clear() {
57+
local redis_uri=$1
58+
redis-cli -u "${redis_uri}" flushall
59+
}
60+
61+
main_native() {
62+
local services=${SERVICES}
63+
local postgres_url_env_var=`[[ $services == *"db_test"* ]] && echo "DB_TEST_POSTGRES_URL" || echo "DB_POSTGRES_URL"`
64+
local redis_host_env_var=`[[ $services == *"redis_test"* ]] && echo "REDIS_TEST_HOST" || echo "REDIS_HOST"`
65+
66+
postgres_url="${!postgres_url_env_var}"
67+
redis_host="${!redis_host_env_var}"
68+
69+
if [ -n "${postgres_url}" ]; then
70+
echo "Using ${postgres_url_env_var} (${postgres_url}) to connect to postgres."
71+
pg_init "${postgres_url}"
72+
else
73+
echo "Postgres connection string missing did you set ${postgres_url_env_var}?"
74+
exit 1
75+
fi
76+
77+
if [ -n "${redis_host}" ]; then
78+
echo "Using ${redis_host_env_var} (${redis_host}) to connect to Redis."
79+
else
80+
echo "Redis connection string missing did you set ${redis_host_env_var}?"
81+
echo "Continuing without Redis..."
82+
fi
83+
84+
cleanup() {
85+
local services=$@
86+
87+
if [ -n "${redis_host}" ] && [[ $services == *"redis_test"* ]]; then
88+
redis_clear "redis://${redis_host}" &> /dev/null
89+
fi
90+
91+
if [ -n "${postgres_url}" ] && [[ $services == *"db_test"* ]]; then
92+
pg_clear "${postgres_url}" &> /dev/null
93+
fi
94+
}
95+
96+
# trap SIGINT and performs cleanup
97+
trap "on_sigint ${services}" INT
98+
on_sigint() {
99+
cleanup $@
100+
exit $?
101+
}
102+
103+
# Run the arguments as a command
104+
DB_POSTGRES_URL="${postgres_url}" \
105+
REDIS_HOST="${redis_host}" \
106+
"$@"
107+
code=$?
108+
109+
cleanup ${services}
110+
111+
exit ${code}
112+
}
113+
114+
main_docker() {
115+
# Expect a SERVICES env var to be set with the docker service names
116+
local services=${SERVICES}
117+
118+
dir=$(dirname $0)
119+
compose_file="${dir}/docker-compose.yaml"
120+
121+
# whether this particular script started the container(s)
122+
started_container=false
123+
124+
# performs cleanup as necessary, i.e. taking down containers
125+
# if this script started them
126+
cleanup() {
127+
local services=$@
128+
echo # newline
129+
if $started_container; then
130+
docker compose --file $compose_file rm --force --stop --volumes ${services}
131+
fi
132+
}
133+
134+
# trap SIGINT and performs cleanup
135+
trap "on_sigint ${services}" INT
136+
on_sigint() {
137+
cleanup $@
138+
exit $?
139+
}
140+
141+
# check if all services are running already
142+
not_running=false
143+
for service in $services; do
144+
container_id=$(get_container_id $compose_file $service)
145+
if [ -z $container_id ]; then
146+
not_running=true
147+
break
148+
fi
149+
done
150+
151+
# if any are missing, recreate all services
152+
if $not_running; then
153+
started_container=true
154+
docker compose --file $compose_file up --wait --force-recreate ${services}
155+
else
156+
echo "all services ${services} are already running"
157+
fi
158+
159+
# do not exit when following commands fail, so we can intercept exit code & tear down docker
160+
set +e
161+
162+
# setup environment variables and run args
163+
export_env
164+
"$@"
165+
# save return code for later
166+
code=$?
167+
168+
# performs cleanup as necessary
169+
cleanup ${services}
170+
exit ${code}
171+
}
172+
173+
# Main entry point
174+
main() {
175+
if ! docker ps >/dev/null 2>&1; then
176+
echo "Docker unavailable. Running on host."
177+
main_native $@
178+
else
179+
main_docker $@
180+
fi
181+
}

‎packages/internal/dev-infra/docker-compose.yaml‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
version: '3.8'
2+
services:
3+
# An ephermerally-stored postgres database for single-use test runs
4+
db_test: &db_test
5+
image: postgres:14.4-alpine
6+
environment:
7+
- POSTGRES_USER=pg
8+
- POSTGRES_PASSWORD=password
9+
ports:
10+
- '5433:5432'
11+
# Healthcheck ensures db is queryable when `docker-compose up --wait` completes
12+
healthcheck:
13+
test: 'pg_isready -U pg'
14+
interval: 500ms
15+
timeout: 10s
16+
retries: 20
17+
# A persistently-stored postgres database
18+
db:
19+
<<: *db_test
20+
ports:
21+
- '5432:5432'
22+
healthcheck:
23+
disable: true
24+
volumes:
25+
- atp_db:/var/lib/postgresql/data
26+
# An ephermerally-stored redis cache for single-use test runs
27+
redis_test: &redis_test
28+
image: redis:7.0-alpine
29+
ports:
30+
- '6380:6379'
31+
# Healthcheck ensures redis is queryable when `docker-compose up --wait` completes
32+
healthcheck:
33+
test: ['CMD-SHELL', '[ "$$(redis-cli ping)" = "PONG" ]']
34+
interval: 500ms
35+
timeout: 10s
36+
retries: 20
37+
# A persistently-stored redis cache
38+
redis:
39+
<<: *redis_test
40+
command: redis-server --save 60 1 --loglevel warning
41+
ports:
42+
- '6379:6379'
43+
healthcheck:
44+
disable: true
45+
volumes:
46+
- atp_redis:/data
47+
volumes:
48+
atp_db:
49+
atp_redis:

‎packages/internal/dev-infra/with-test-db.sh‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
#!/usr/bin/env sh
2+
3+
# Example usage:
4+
# ./with-test-db.sh psql postgresql://pg:password@localhost:5433/postgres -c 'select 1;'
5+
6+
dir=$(dirname $0)
7+
. ${dir}/_common.sh
8+
9+
SERVICES="db_test" main "$@"

‎packages/internal/dev-infra/with-test-redis-and-db.sh‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
#!/usr/bin/env sh
2+
3+
# Example usage:
4+
# ./with-test-redis-and-db.sh psql postgresql://pg:password@localhost:5433/postgres -c 'select 1;'
5+
# ./with-test-redis-and-db.sh redis-cli -h localhost -p 6380 ping
6+
7+
dir=$(dirname $0)
8+
. ${dir}/_common.sh
9+
10+
SERVICES="db_test redis_test" main "$@"

0 commit comments

Comments
 (0)