Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .idea/misc.xml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

93 changes: 70 additions & 23 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -14,75 +14,122 @@
# Pull base image
# ---------------
ARG MAVEN_VERSION=3.9.9
ARG NODE_VERSION=20
ARG BASE_REGISTRY=registry.access.redhat.com/ubi8
ARG BASE_IMAGE=ubi-minimal
ARG JAVA_OPT="-XX:UseSVE=0"

FROM docker.io/library/maven:${MAVEN_VERSION} AS builder
#############################################
# Stage 1: Build Java Application
#############################################
FROM docker.io/library/maven:${MAVEN_VERSION} AS java-builder
LABEL stage=pgcomparebuilder
ARG JAVA_OPT

ENV _JAVA_OPTIONS=${JAVA_OPT}

WORKDIR /app
COPY . ./
COPY pom.xml ./
COPY src ./src

RUN mvn clean install -DskipTests

#############################################
# Stage 2: Build Next.js UI
#############################################
FROM docker.io/library/node:${NODE_VERSION}-alpine AS ui-builder

RUN mvn clean install
WORKDIR /app/ui
COPY ui/package*.json ./
RUN npm ci

COPY ui/ ./
RUN npm run build

FROM ${BASE_REGISTRY}/${BASE_IMAGE} as multi-stage
#############################################
# Stage 3: Multi-stage Production Image
#############################################
FROM ${BASE_REGISTRY}/${BASE_IMAGE} AS multi-stage
ARG JAVA_OPT
ARG NODE_VERSION
ARG TARGETARCH

RUN microdnf install java-21-openjdk -y
RUN microdnf install java-21-openjdk tar xz -y && microdnf clean all

RUN NODE_ARCH=$([ "$TARGETARCH" = "arm64" ] && echo "arm64" || echo "x64") && \
curl -fsSL https://nodejs.org/dist/v${NODE_VERSION}.9.0/node-v${NODE_VERSION}.9.0-linux-${NODE_ARCH}.tar.xz | tar -xJ -C /usr/local --strip-components=1

USER 0

RUN mkdir /opt/pgcompare \
RUN mkdir -p /opt/pgcompare/ui /opt/pgcompare/lib \
&& chown -R 1001:1001 /opt/pgcompare

COPY --from=builder /app/docker/start.sh /opt/pgcompare

COPY --from=builder /app/docker/pgcompare.properties /etc/pgcompare/
COPY docker/start.sh /opt/pgcompare/
COPY docker/pgcompare.properties /etc/pgcompare/
COPY --from=java-builder /app/target/*.jar /opt/pgcompare/
COPY --from=java-builder /app/target/lib/ /opt/pgcompare/lib/

COPY --from=builder /app/target/* /opt/pgcompare/
COPY --from=ui-builder /app/ui/.next/standalone/ /opt/pgcompare/ui/
COPY --from=ui-builder /app/ui/.next/static /opt/pgcompare/ui/.next/static
COPY --from=ui-builder /app/ui/public /opt/pgcompare/ui/public

RUN chmod 770 /opt/pgcompare/start.sh
RUN chmod 770 /opt/pgcompare/start.sh \
&& chown -R 1001:1001 /opt/pgcompare

USER 1001

ENV PGCOMPARE_HOME=/opt/pgcompare \
PGCOMPARE_CONFIG=/etc/pgcompare/pgcompare.properties \
PGCOMPARE_MODE=standard \
PATH=/opt/pgcompare:$PATH \
_JAVA_OPTIONS=${JAVA_OPT}
_JAVA_OPTIONS=${JAVA_OPT} \
PORT=3000 \
HOSTNAME=0.0.0.0

EXPOSE 3000

CMD ["start.sh"]

WORKDIR "/opt/pgcompare"

## Local Platform Build
FROM ${BASE_REGISTRY}/${BASE_IMAGE} as local
#############################################
# Stage 4: Local Platform Build
#############################################
FROM ${BASE_REGISTRY}/${BASE_IMAGE} AS local
ARG JAVA_OPT
ARG NODE_VERSION
ARG TARGETARCH

RUN microdnf install java-21-openjdk tar xz -y && microdnf clean all

RUN microdnf install java-21-openjdk -y
RUN NODE_ARCH=$([ "$TARGETARCH" = "arm64" ] && echo "arm64" || echo "x64") && \
curl -fsSL https://nodejs.org/dist/v${NODE_VERSION}.9.0/node-v${NODE_VERSION}.9.0-linux-${NODE_ARCH}.tar.xz | tar -xJ -C /usr/local --strip-components=1

USER 0

RUN mkdir /opt/pgcompare \
RUN mkdir -p /opt/pgcompare/ui /opt/pgcompare/lib \
&& chown -R 1001:1001 /opt/pgcompare

COPY docker/start.sh /opt/pgcompare/

COPY docker/pgcompare.properties /etc/pgcompare/
COPY target/*.jar /opt/pgcompare/
COPY target/lib/ /opt/pgcompare/lib/

COPY target/* /opt/pgcompare/
COPY ui/.next/standalone/ /opt/pgcompare/ui/
COPY ui/.next/static /opt/pgcompare/ui/.next/static
COPY ui/public /opt/pgcompare/ui/public

RUN chmod 770 /opt/pgcompare/start.sh
RUN chmod 770 /opt/pgcompare/start.sh \
&& chown -R 1001:1001 /opt/pgcompare

USER 1001

ENV PGCOMPARE_HOME=/opt/pgcompare \
PGCOMPARE_CONFIG=/etc/pgcompare/pgcompare.properties \
PGCOMPARE_MODE=standard \
PATH=/opt/pgcompare:$PATH \
_JAVA_OPTIONS=${JAVA_OPT}
_JAVA_OPTIONS=${JAVA_OPT} \
PORT=3000 \
HOSTNAME=0.0.0.0

EXPOSE 3000

CMD ["start.sh"]

Expand Down
52 changes: 48 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,13 +51,16 @@ Before initiating the build and installation process, ensure the following prere

# Getting Started

> **Note:** The `main` branch contains active development and may be unstable. For production use, we recommend checking out a stable release tag (e.g., `git checkout v0.6.0`). See [Releases](https://github.com/CrunchyData/pgCompare/releases) for available versions.

## 1. Fork the repository

## 2. Clone and Build

```shell
git clone --depth 1 git@github.com:<your-github-username>/pgCompare.git
cd pgCompare
git checkout v0.6.0 # Optional: checkout a stable release
mvn clean install
```

Expand All @@ -68,6 +71,15 @@ By default, the application looks for the properties file in the execution direc

At a minimal the `repo-xxxxx` parameters are required in the properties file (or specified by environment parameters). Besides the properties file and environment variables, another alternative is to store the property settings in the `dc_project` table. Settings can be stored in the `project_config` column in JSON format ({"parameter": "value"}). Certain system parameters like log-destination can only be specified via the properties file or environment variables.

You can also import/export configuration using the CLI:
```shell
# Export project configuration to properties file
java -jar pgcompare.jar export-config -p 1 --file project-config.properties

# Import properties file to project configuration
java -jar pgcompare.jar import-config -p 1 --file my-config.properties
```

## 4. Initialize Repository

Run the script or use the command below to set up the PostgreSQL repository:
Expand Down Expand Up @@ -96,18 +108,30 @@ Actions:
- **check**: Recompare the out of sync rows from previous compare
- **compare**: Perform database compare
- **copy-table**: Copy pgCompare metadata for table. Must specify table alias to copy using --table option
- **discover**: Disocver tables and columns
- **discover**: Discover tables and columns
- **export-config**: Export project configuration to properties file
- **export-mapping**: Export table/column mappings to YAML file
- **import-config**: Import properties file to project configuration
- **import-mapping**: Import table/column mappings from YAML file
- **init**: Initialize the repository database
- **server**: Run in server mode (daemon that polls work queue for jobs)
- **test-connection**: Test database connections and report status

Options:

-b|--batch {batch nbr}

-o|--file {path} File for export/import operations

--overwrite Overwrite existing mappings during import

-p|--project Project ID

-r|--report {file} Create html report of compare

-t|--table {target table}
-t|--table {target table} (supports wildcards for export/import)

-n|--name {server name} Server name for server mode (default: pgcompare-server)

--help

Expand Down Expand Up @@ -143,7 +167,19 @@ java -jar pgcompare.jar check --batch 0

# Upgrading

## Version 0.5.0 Enhacements
## Version 0.6.0 Enhancements

- **Server Mode** - Run pgCompare as a daemon that polls a work queue for jobs
- **Web UI Enhancements** - Job scheduling, real-time progress tracking, job control (pause/resume/stop)
- **Signal Handling** - Graceful shutdown (SIGINT), immediate termination (SIGTERM), config reload (SIGHUP)
- **Fix SQL Improvements** - Enhanced data type handling, proper column mapping for UPDATE statements
- **Database Schema** - New tables for server mode (dc_server, dc_job, dc_job_control, dc_job_progress, dc_job_log)

**Note:** Drop and recreate the repository to upgrade to 0.6.0.

For more details review the [v0.6.0 Release Notes](RELEASE_NOTES_v0.6.0.md)

## Version 0.5.0 Enhancements

- Snowflake Support - Full integration for Snowflake as source/target
- SQL Fix Generation - Automatic generation of INSERT/UPDATE/DELETE statements (Preview, limited ability)
Expand All @@ -153,7 +189,7 @@ java -jar pgcompare.jar check --batch 0

**Note:** Drop and recreate the repository to upgrade to 0.5.0.

For more details review the [v0.5.0 Release Noes](RELEASE_NOTES_v0.5.0.md)
For more details review the [v0.5.0 Release Notes](RELEASE_NOTES_v0.5.0.md)

## Version 0.4.0 Enhancements

Expand Down Expand Up @@ -201,6 +237,8 @@ Examples:

Projects allow for the repository to maintain different mappings for different compare objectives. This allows a central pgCompare repository to be used for multiple compare projects. Each table has a `pid` column which is the project id. If no project is specified, the default project (pid = 1) is used.

> **Note:** Project 1 (pid=1) is reserved as the default project and is created automatically during repository initialization. Do not delete or reassign this project.

# Viewing Results

## Summary from Last Run
Expand Down Expand Up @@ -272,6 +310,12 @@ Properties are categorized into four sections: system, repository, source, and t

Default: 3

#### job-logging-enabled

When set to true, log messages are written to the `dc_job_log` table in addition to the standard log destination. This enables viewing job logs through the pgCompare UI. The job must have an associated job ID (either from server mode or standalone job tracking).

Default: false

#### loader-threads

Sets the number of threads to load data into the temporary tables. Set to 0 to disable loader threads.
Expand Down
118 changes: 118 additions & 0 deletions RELEASE_NOTES_v0.6.0.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,118 @@
# pgCompare v0.6.0 Release Notes

## New Features

### Server Mode
pgCompare can now run as a daemon server that polls a work queue for jobs. This enables:

- **Multi-server deployment**: Run multiple pgCompare instances that automatically pick up work
- **Work queue scheduling**: Submit compare, check, or discover jobs via the UI or external tools
- **Job control signals**: Pause, resume, stop, or terminate running jobs gracefully
- **Real-time progress tracking**: Monitor job progress at the table level

#### Usage
```shell
# Start pgCompare in server mode
java -jar pgcompare.jar server --name my-server-01

# Server mode only requires repository connection info in properties file
# Source/target database connections are loaded from project configuration
```

#### Command Line Options
- `-n|--name <server name>`: Set the server name (default: pgcompare-server)

### UI Enhancements

#### Dashboard Overview
- Real-time server status monitoring with heartbeat tracking
- Running, pending, and completed job overview
- Quick access to job details and progress

#### Job Scheduling & Management
- Schedule compare, check, or discover jobs from the UI
- Target specific servers or let any available server pick up work
- Set job priority (1-10) for queue ordering
- Optional scheduled start time for deferred execution
- Real-time job progress with per-table status

#### Job Control
- **Pause**: Temporarily halt a running job (preserves progress)
- **Resume**: Continue a paused job
- **Stop**: Gracefully stop a job (completes current table)
- **Terminate**: Immediately stop a job

#### Navigation & Search
- Search/filter projects and tables in the navigation tree
- Breadcrumb navigation component
- Connection status indicator with auto-refresh

#### Data Management
- Bulk operations for enabling/disabling multiple tables
- Export data to CSV or JSON format
- Pagination for large result sets

#### User Experience
- Toast notifications replace browser alerts
- Loading skeletons for better perceived performance
- Keyboard shortcuts support

### Signal Handling & Graceful Shutdown
pgCompare now properly handles OS signals for clean shutdown and query cancellation:

- **SIGINT (Ctrl+C)**: Graceful shutdown - completes the current table comparison before exiting
- **SIGTERM**: Immediate termination - cancels all running database queries and exits
- **SIGHUP**: Reload configuration from properties file without restart

Active database statements are tracked and can be cancelled on demand, preventing orphaned queries on the source/target databases during forced shutdowns.

## Database Schema Changes

**Note:** Drop and recreate the repository to upgrade to 0.6.0.

New tables for server mode:
- `dc_server`: Server registration and heartbeat tracking
- `dc_work_queue`: Job queue with priority scheduling
- `dc_job_control`: Control signals for running jobs
- `dc_job_progress`: Per-table progress tracking

## API Endpoints

New REST API endpoints:
- `GET /api/servers`: List registered servers
- `GET /api/jobs`: List jobs with filtering
- `POST /api/jobs`: Submit a new job
- `GET /api/jobs/{id}`: Get job details
- `DELETE /api/jobs/{id}`: Delete a completed/failed job
- `POST /api/jobs/{id}/control`: Send control signal (pause/resume/stop/terminate)
- `GET /api/jobs/{id}/progress`: Get job progress with per-table status
- `GET /api/health`: Check database connection status

## Configuration

### Server Mode Properties
Server mode only requires repository connection properties:
```properties
repo-host=localhost
repo-port=5432
repo-dbname=pgcompare
repo-schema=pgcompare
repo-user=postgres
repo-password=secret
```

Project-specific source/target database settings are loaded from the `dc_project.project_config` column.

## Upgrade Guide

1. Stop all running pgCompare instances
2. Backup your current repository database
3. Drop the existing pgCompare schema: `DROP SCHEMA pgcompare CASCADE;`
4. Run `java -jar pgcompare.jar init` to create the new schema
5. Reconfigure your projects and table mappings

## Known Limitations

- Server mode requires PostgreSQL 14+ for `SKIP LOCKED` support
- Job control signals may have up to 5 second delay before processing
- Servers are marked offline after 2 minutes without heartbeat
Loading