Skip to content

feat(db): add PostgreSQL backend#755

Merged
gabriel-samfira merged 3 commits into
cloudbase:mainfrom
cbartz:feat/postgres-backend
Jun 3, 2026
Merged

feat(db): add PostgreSQL backend#755
gabriel-samfira merged 3 commits into
cloudbase:mainfrom
cbartz:feat/postgres-backend

Conversation

@cbartz
Copy link
Copy Markdown
Contributor

@cbartz cbartz commented Jun 1, 2026

Closes #753

What

Add PostgreSQL as a third SQL backend alongside SQLite and MySQL, selectable via
backend = "postgresql":

  • config — PostgreSQL backend type, connection settings, validation, and DSN
    construction (sslmode, port-range checks, special-character quoting).
  • driver wiring — open the pgx-backed GORM driver, apply connection-pool
    tuning (max open/idle, lifetimes), and reuse the main connection for blob
    storage (PostgreSQL needs no separate objects database like SQLite uses).
  • file store — store file-object blobs in PostgreSQL Large Objects
    (keyed by lo_oid), not a bytea column. Large Objects support chunked,
    seekable streaming via the lo API, so blobs are read/written incrementally
    and never fully held in memory — required because GARM's file-object interface
    is streaming (CreateFileObject(..., io.Reader) /
    OpenFileObjectContent(...) io.ReadCloser), and a bytea value would have to
    be materialized in full. Writes io.Copy the reader into the Large Object
    (hashing inline); reads return a ReadCloser over lo.Read. SQLite continues
    to use the Content BLOB column.
  • migrations — a dedicated file_object_migrations tracking table with a
    rename shim so existing SQLite installs preserve their file-object migration
    history, plus a functional LOWER(tag) index and the lo_oid column for
    file_object blobs.
  • uuid guardgetUserByID pre-validates UUIDs so PostgreSQL's uuid
    column returns ErrNotFound on malformed input instead of a driver error.
  • tests + CI — parametrize the SQL suite to run against PostgreSQL when
    GARM_TEST_POSTGRES_DSN is set, and add a PostgreSQL job to the test workflow.

Notes for reviewers

  • The bulk of the diff is vendored dependencies (pgx, gorm.io/driver/postgres
    and transitive deps). The hand-written change is ~970 lines; the rest is
    vendor/.
  • Case folding differs subtly between backends: SQLite's LOWER() folds
    ASCII only, PostgreSQL's is locale-aware (folds Unicode). For GARM's
    effectively-ASCII identifiers this is a non-issue, but noting it explicitly.

Testing

  • Unit/store suite (local, Docker PostgreSQL): ran the SQL store and config
    suites against postgres:16 (Docker, fsync=off) using the same image,
    credentials, and DSN as the CI job — database/sql and config pass with
    -race -tags testing -count=1.
    ok github.com/cloudbase/garm/database/sql (PostgreSQL backend)
    ok github.com/cloudbase/garm/config
  • SQLite (unchanged backend): same suite passes in SQLite mode, so both
    backends are covered.
  • Lint: make lint (golangci-lint v2.10.1) is clean — 0 issues.
  • CI: adds a "Go Tests (PostgreSQL)" job that runs make go-test against a
    postgres:16 service via GARM_TEST_POSTGRES_DSN, exercising
    migrations/AutoMigrate and the Large Object file store on a real PostgreSQL.
  • Manual end-to-end: validated a PostgreSQL-backed GARM end-to-end on
    cbartz/garm — created credentials, repo, and a scale set, and confirmed
    runners register with GitHub and execute jobs.

Future work

The nightly integration suite (test/integration, scheduled in
integration-tests.yml) still runs only against SQLite — its config hardcodes
backend = "sqlite3". The PostgreSQL path was validated manually end-to-end (see
Testing); automating that in the nightly suite (add a PG service + template the
[database] config) would be a natural follow-up.

Add PostgreSQL as a third SQL backend alongside SQLite and MySQL:

- config: PostgreSQL backend type, connection settings, validation and DSN
  construction (sslmode, port range, special-character quoting).
- driver wiring: open the pgx-backed gorm driver, apply connection-pool
  tuning, and reuse the main connection for blob storage (no separate
  objects database as SQLite uses).
- file store: store file-object blobs in PostgreSQL Large Objects, keyed by
  lo_oid; SQLite continues to use the Content BLOB column.
- migrations: dedicated file_object_migrations tracking table with a
  rename shim so existing SQLite installs preserve their history, plus a
  functional LOWER(tag) index and lo_oid column for file_object blobs.
- getUserByID pre-validates UUIDs so PostgreSQL's uuid column returns
  ErrNotFound on malformed input instead of a driver error.
- tests: parametrize the SQL suite to run against PostgreSQL when
  GARM_TEST_POSTGRES_DSN is set, and add a PostgreSQL CI job.
@cbartz cbartz force-pushed the feat/postgres-backend branch from 742b264 to 1bad028 Compare June 1, 2026 10:07
@cbartz cbartz marked this pull request as ready for review June 1, 2026 11:01
The self-hosted ubuntu-noble-garm runners have no Docker, so the
go-tests-postgres job failed during job setup with 'docker: command not
found' when initializing the postgres service container. Install and run
PostgreSQL from the distro package instead, matching the apt-based
dependency pattern used by the other jobs on these runners.
@gabriel-samfira
Copy link
Copy Markdown
Member

I added docker to the CI runners. You should be able to use the service: stanza. If you run into any missing packages, just ping me.

@gabriel-samfira gabriel-samfira merged commit 9f16ce3 into cloudbase:main Jun 3, 2026
5 checks passed
@gabriel-samfira
Copy link
Copy Markdown
Member

Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support PostgreSQL as a database backend

2 participants