-
Notifications
You must be signed in to change notification settings - Fork 32
feat: add wal2json container image
#197
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
solidDoWant
wants to merge
8
commits into
cloudnative-pg:main
Choose a base branch
from
solidDoWant:feat/wal2json
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
8 commits
Select commit
Hold shift + click to select a range
1e2f640
feat: add `wal2json` container image
solidDoWant 9a8a969
chore: set `wal2json` maintainer info
solidDoWant 65f2311
chore: add CODEOWNER
gbartolini bfb50ef
chore: add wal2json to the main README
gbartolini 3ac9348
docs: cosmetic change
gbartolini cb70570
Merge branch 'main' into feat/wal2json
gbartolini ed4f736
chore: fix CODEOWNERS
gbartolini ed4448b
chore: switch `wal2json` tests to replication user instead of superuser
solidDoWant File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,26 @@ | ||
| # SPDX-FileCopyrightText: Copyright © contributors to CloudNativePG, established as CloudNativePG a Series of LF Projects, LLC. | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| ARG BASE=ghcr.io/cloudnative-pg/postgresql:18-minimal-trixie | ||
| FROM $BASE AS builder | ||
|
|
||
| ARG PG_MAJOR | ||
| ARG EXT_VERSION | ||
|
|
||
| USER 0 | ||
|
|
||
| # Install extension via `apt-get` | ||
| RUN apt-get update && apt-get install -y --no-install-recommends \ | ||
| "postgresql-${PG_MAJOR}-wal2json=${EXT_VERSION}" | ||
|
|
||
| FROM scratch | ||
| ARG PG_MAJOR | ||
|
|
||
| # Licenses | ||
| COPY --from=builder /usr/share/doc/postgresql-${PG_MAJOR}-wal2json/copyright /licenses/postgresql-${PG_MAJOR}-wal2json/ | ||
|
|
||
| # Libraries | ||
| COPY --from=builder /usr/lib/postgresql/${PG_MAJOR}/lib/wal2json* /lib/ | ||
| COPY --from=builder /usr/lib/postgresql/${PG_MAJOR}/lib/bitcode/ /lib/bitcode/ | ||
|
|
||
| USER 65532:65532 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,117 @@ | ||
| # wal2json | ||
| <!-- | ||
| SPDX-FileCopyrightText: Copyright © contributors to CloudNativePG, established as CloudNativePG a Series of LF Projects, LLC. | ||
| SPDX-License-Identifier: Apache-2.0 | ||
| --> | ||
|
|
||
| [wal2json](https://github.com/eulerto/wal2json) is a PostgreSQL **logical | ||
| decoding output plugin**. It reads the write-ahead log (WAL) of a database via | ||
| a logical replication slot and emits the changes as JSON, which makes it a | ||
| common building block for change data capture (CDC) pipelines (e.g. Debezium, | ||
| custom replication tooling, or audit streams). | ||
|
|
||
| Because `wal2json` is a logical decoding output plugin rather than a regular | ||
| extension, it does **not** expose a `CREATE EXTENSION` object. The plugin is | ||
| loaded on demand by PostgreSQL when a replication slot is created with the | ||
| `wal2json` output plugin. | ||
|
|
||
| This image provides a convenient way to deploy and manage `wal2json` with | ||
| [CloudNativePG](https://cloudnative-pg.io/). | ||
|
|
||
| ## Usage | ||
|
|
||
| ### 1. Add the wal2json extension image to your Cluster | ||
|
|
||
| Define the `wal2json` extension under the `postgresql.extensions` section of | ||
| your `Cluster` resource. Logical decoding requires `wal_level` to be set to | ||
| `logical`: | ||
|
|
||
| ```yaml | ||
| apiVersion: postgresql.cnpg.io/v1 | ||
| kind: Cluster | ||
| metadata: | ||
| name: cluster-wal2json | ||
| spec: | ||
| imageName: ghcr.io/cloudnative-pg/postgresql:18-minimal-trixie | ||
| instances: 1 | ||
|
|
||
| storage: | ||
| size: 1Gi | ||
|
|
||
| postgresql: | ||
| parameters: | ||
| wal_level: logical | ||
| extensions: | ||
| - name: wal2json | ||
| image: | ||
| # renovate: suite=trixie-pgdg depName=postgresql-18-wal2json | ||
| reference: ghcr.io/cloudnative-pg/wal2json:2.6-18-trixie | ||
| ``` | ||
|
|
||
| > [!NOTE] | ||
| > Unlike most extensions, `wal2json` is a logical decoding output plugin and | ||
| > does not require a `Database` resource with `CREATE EXTENSION`. The plugin | ||
| > is referenced directly when a logical replication slot is created. | ||
|
|
||
| ### 2. Create a logical replication slot using `wal2json` | ||
|
|
||
| Once the cluster is ready, connect to the database with `psql` and create a | ||
| logical replication slot that uses the `wal2json` output plugin: | ||
|
|
||
| ```sql | ||
| SELECT * FROM pg_create_logical_replication_slot('wal2json_slot', 'wal2json'); | ||
| ``` | ||
|
|
||
| Some applications (for example, [Teleport](https://goteleport.com/docs/reference/deployment/backends/#postgresql)), | ||
| may do this automatically. | ||
|
|
||
| ### 3. Verify installation | ||
|
|
||
| Consume changes from the slot to confirm the plugin is loaded correctly. After | ||
| producing some activity (e.g. `CREATE TABLE t (id int); INSERT INTO t VALUES (1);`), | ||
| peek at the slot: | ||
|
|
||
| ```sql | ||
| SELECT data | ||
| FROM pg_logical_slot_peek_changes('wal2json_slot', NULL, NULL); | ||
| ``` | ||
|
|
||
| You should see the changes emitted as JSON documents. | ||
|
|
||
| When you are done, drop the slot to release resources: | ||
|
|
||
| ```sql | ||
| SELECT pg_drop_replication_slot('wal2json_slot'); | ||
| ``` | ||
|
|
||
| ## Contributors | ||
|
|
||
| This extension is maintained by: | ||
|
|
||
| - Fred Heinecke (@solidDoWant) | ||
|
|
||
| The maintainers are responsible for: | ||
|
|
||
| - Monitoring upstream releases and security vulnerabilities. | ||
| - Ensuring compatibility with supported PostgreSQL versions. | ||
| - Reviewing and merging contributions specific to this extension's container | ||
| image and lifecycle. | ||
|
|
||
| --- | ||
|
|
||
| ## Licenses and Copyright | ||
|
|
||
| `wal2json`: | ||
|
|
||
| - **Copyright:** (c) 2013-2018 Euler Taveira de Oliveira | ||
| - **License:** BSD 3-Clause License | ||
|
|
||
| All relevant license and copyright information for the `wal2json` extension | ||
| and its dependencies are bundled within the image at: | ||
|
|
||
| ```text | ||
| /licenses/ | ||
| ``` | ||
|
|
||
| By using this image, you agree to comply with the terms of the licenses | ||
| contained therein. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,33 @@ | ||
| # SPDX-FileCopyrightText: Copyright © contributors to CloudNativePG, established as CloudNativePG a Series of LF Projects, LLC. | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
| metadata = { | ||
| name = "wal2json" | ||
| sql_name = "wal2json" | ||
| image_name = "wal2json" | ||
| licenses = ["BSD-3-Clause"] | ||
| shared_preload_libraries = [] | ||
| postgresql_parameters = {} | ||
| extension_control_path = [] | ||
| dynamic_library_path = [] | ||
| ld_library_path = [] | ||
| bin_path = [] | ||
| env = {} | ||
| auto_update_os_libs = false | ||
| required_extensions = [] | ||
| create_extension = false | ||
|
|
||
| versions = { | ||
| bookworm = { | ||
| "18" = { | ||
| // renovate: suite=bookworm-pgdg depName=postgresql-18-wal2json | ||
| package = "2.6-3.pgdg12+1" | ||
| } | ||
| } | ||
| trixie = { | ||
| "18" = { | ||
| // renovate: suite=trixie-pgdg depName=postgresql-18-wal2json | ||
| package = "2.6-3.pgdg13+1" | ||
| } | ||
| } | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,36 @@ | ||
| apiVersion: chainsaw.kyverno.io/v1alpha1 | ||
| kind: Test | ||
| metadata: | ||
| name: verify-wal2json-output-plugin | ||
| spec: | ||
| timeouts: | ||
| apply: 5s | ||
| assert: 3m | ||
| delete: 30s | ||
| description: >- | ||
| Verify that the wal2json shared library is loadable by PostgreSQL by | ||
| creating and dropping a logical replication slot using the wal2json output | ||
| plugin. Slot creation forces PostgreSQL to dlopen wal2json.so, so this is | ||
| the actual end-to-end install check for an output plugin (which has no | ||
| CREATE EXTENSION counterpart). | ||
| steps: | ||
| - name: Provision PostgreSQL Cluster | ||
| description: >- | ||
| Deploy a Cluster resource using the built wal2json image with replication | ||
| access enabled so the slot-creation Job can call | ||
| pg_create_logical_replication_slot. | ||
| try: | ||
| - apply: | ||
| file: cluster.yaml | ||
| - assert: | ||
| file: cluster-assert.yaml | ||
|
|
||
| - name: Create logical replication slot using wal2json | ||
| description: >- | ||
| Run a Job that creates and drops a logical replication slot with the | ||
| 'wal2json' output plugin, exercising dlopen of wal2json.so. | ||
| try: | ||
| - apply: | ||
| file: check-slot.yaml | ||
| - assert: | ||
| file: check-slot-assert.yaml |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| apiVersion: batch/v1 | ||
| kind: Job | ||
| metadata: | ||
| name: wal2json-slot-test | ||
| status: | ||
| succeeded: 1 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,28 @@ | ||
| apiVersion: batch/v1 | ||
| kind: Job | ||
| metadata: | ||
| name: wal2json-slot-test | ||
| spec: | ||
| template: | ||
| spec: | ||
| restartPolicy: OnFailure | ||
| containers: | ||
| - name: slot-test | ||
| env: | ||
| - name: SU_URI | ||
| valueFrom: | ||
| secretKeyRef: | ||
| name: (join('-', [$values.name, 'app'])) | ||
| key: uri | ||
| image: alpine/psql:latest | ||
| command: ['sh', '-c'] | ||
| args: | ||
| - | | ||
| set -e | ||
| SU_URI=$(echo $SU_URI | sed "s|/\*|/|") | ||
| SLOT="wal2json_chainsaw_test" | ||
| psql "$SU_URI" -v ON_ERROR_STOP=1 -tAc \ | ||
| "SELECT slot_name FROM pg_create_logical_replication_slot('${SLOT}', 'wal2json')" | ||
| psql "$SU_URI" -v ON_ERROR_STOP=1 -tAc \ | ||
| "SELECT pg_drop_replication_slot('${SLOT}')" | ||
| echo "wal2json output plugin loaded successfully" | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| apiVersion: postgresql.cnpg.io/v1 | ||
| kind: Cluster | ||
| metadata: | ||
| name: ($values.name) | ||
| status: | ||
| readyInstances: 1 | ||
| phase: Cluster in healthy state | ||
| image: ($values.pg_image) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| apiVersion: postgresql.cnpg.io/v1 | ||
| kind: Cluster | ||
| metadata: | ||
| name: ($values.name) | ||
| spec: | ||
| imageName: ($values.pg_image) | ||
| instances: 1 | ||
| managed: | ||
| roles: | ||
| - name: app | ||
| ensure: present | ||
| login: true | ||
| replication: true | ||
|
|
||
| storage: | ||
| size: 1Gi | ||
|
|
||
| postgresql: | ||
| parameters: ($values.postgresql_parameters) | ||
| shared_preload_libraries: ($values.shared_preload_libraries) | ||
| extensions: ($values.extensions) |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This needs to be changed now as per #219
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This has been fixed and tested