Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
a026cc1
feat(mcp): hygiene issues — list, get, search, update + resource + pr…
rboni-dk May 1, 2026
e8516e0
Merge branch 'main' into 'enterprise'
aarthy-dk May 4, 2026
a4c557e
fix(ui): visual glitches in connections and test results
luis-dk May 5, 2026
ad79b73
Merge branch 'qa-fixes-5526' into 'enterprise'
May 5, 2026
a111764
fix(bigquery): error on freshness monitor when tolerance is null
aarthy-dk May 5, 2026
61f8d0e
fix(logs): suppress streamlit warnings
aarthy-dk May 5, 2026
e91277e
fix(standalone): command argument precedence - add log path to config…
aarthy-dk May 5, 2026
c1baa0e
Merge branch 'misc-fixes' into 'enterprise'
May 5, 2026
e28d5a8
fix(standalone): honor config.env for ports, hosts, and base URLs
aarthy-dk May 5, 2026
27a43d3
Merge branch 'fix/TG-1064-config-env-port-precedence' into 'enterprise'
May 5, 2026
3c6d005
fix(mcp): hygiene issues review feedback (TG-1029)
rboni-dk May 5, 2026
ead8b6c
Merge remote-tracking branch 'origin/enterprise' into feat/TG-1029-mc…
rboni-dk May 6, 2026
01ad413
Merge branch 'feat/TG-1029-mcp-hygiene-issues' into 'enterprise'
May 6, 2026
fde7321
fix(standalone): graceful shutdown of embedded postgres on signal
aarthy-dk May 6, 2026
043da3c
docs: simplify README install section to point at docs site
aarthy-dk May 6, 2026
07419e2
Merge branch 'install-doc-updates' into 'enterprise'
May 6, 2026
cd0c53b
Merge remote-tracking branch 'origin/enterprise' into graceful-shutdown
May 6, 2026
ab6697e
Merge branch 'graceful-shutdown' into 'enterprise'
May 6, 2026
4e7cc45
fix(standalone): resolve embedded host/port at connection-build time
aarthy-dk May 15, 2026
c002b53
release: 5.32.2 -> 5.33.2
aarthy-dk May 15, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
195 changes: 4 additions & 191 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,199 +29,12 @@ A <b>single place to manage Data Quality</b> across data sets, locations, and te
<img alt="DataKitchen Open Source Data Quality TestGen Features - Single Place" src="https://datakitchen.io/wp-content/uploads/2024/07/Screenshot-dataops-testgen-centralize.png" width="70%">
</p>

## Installation with dk-installer (recommended)
## Installation

The [dk-installer](https://github.com/DataKitchen/data-observability-installer/?tab=readme-ov-file#install-the-testgen-application) program installs DataOps Data Quality TestGen as a [Docker Compose](https://docs.docker.com/compose/) application. This is the recommended mode of installation as Docker encapsulates and isolates the application from other software on your machine and does not require you to manage Python dependencies.
The [dk-installer](https://github.com/DataKitchen/data-observability-installer/?tab=readme-ov-file#install-the-testgen-application) program installs TestGen in either Docker or pip mode. For complete instructions, see the documentation:

### Install the prerequisite software

| Software | Tested Versions | Command to check version |
|-------------------------|-------------------------|-------------------------------|
| [Python](https://www.python.org/downloads/) <br/>- Most Linux and macOS systems have Python pre-installed. <br/>- On Windows machines, you will need to download and install it. <br/> Why Python? To run the installer. | 3.9, 3.10, 3.11, 3.12 | `python3 --version` |
| [Docker](https://docs.docker.com/get-docker/) <br/>[Docker Compose](https://docs.docker.com/compose/install/) <br/> Why Docker? Docker lets you try TestGen without affecting your local software environment. All the dependencies TestGen needs are isolated in its own container, so installation is easy and insulated. | 26.1, 27.5, 28.0 <br/> 2.32, 2.33, 2.34 | `docker -v` <br/> `docker compose version` |

### Download the installer

On Unix-based operating systems, use the following command to download it to the current directory. We recommend creating a new, empty directory.

```shell
curl -o dk-installer.py 'https://raw.githubusercontent.com/DataKitchen/data-observability-installer/main/dk-installer.py'
```

* Alternatively, you can manually download the [`dk-installer.py`](https://github.com/DataKitchen/data-observability-installer/blob/main/dk-installer.py) file from the [data-observability-installer](https://github.com/DataKitchen/data-observability-installer) repository.
* All commands listed below should be run from the folder containing this file.
* For usage help and command options, run `python3 dk-installer.py --help` or `python3 dk-installer.py <command> --help`.

### Install the TestGen application

The installation downloads the latest Docker images for TestGen and deploys a new Docker Compose application. The process may take 5~10 minutes depending on your machine and network connection.

```shell
python3 dk-installer.py tg install
```

The `--port` option may be used to set a custom localhost port for the application (default: 8501).

To enable SSL for HTTPS support, use the `--ssl-cert-file` and `--ssl-key-file` options to specify local file paths to your SSL certificate and key files.

Once the installation completes, verify that you can login to the UI with the URL and credentials provided in the output.

### Optional: Run the TestGen demo setup

The [Data Observability quickstart](https://docs.datakitchen.io/tutorials/quickstart-demo/) walks you through DataOps Data Quality TestGen capabilities to demonstrate how it covers critical use cases for data and analytic teams.

```shell
python3 dk-installer.py tg run-demo
```

In the TestGen UI, you will see that new data profiling and test results have been generated.

## Installation with pip

As an alternative to the Docker Compose [installation with dk-installer (recommended)](#installation-with-dk-installer-recommended), DataOps Data Quality TestGen can also be installed as a Python package via [pip](https://pip.pypa.io/en/stable/). This mode of installation uses the [dataops-testgen](https://pypi.org/project/dataops-testgen/) package published to PyPI, and it requires a PostgreSQL instance to be provisioned for the application database.

### Install the prerequisite software

| Software | Tested Versions | Command to check version |
|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------|------------------------------|
| [Python](https://www.python.org/downloads/) <br/>- Most Linux and macOS systems have Python pre-installed. <br/>- On Windows machines, you will need to download and install it. | 3.11, 3.12, 3.13 | `python3 --version` |
| [PostgreSQL](https://www.postgresql.org/download/) | 14.1, 15.8, 16.4 | `psql --version`|

### Install the TestGen package

We recommend using a Python virtual environment to avoid any dependency conflicts with other applications installed on your machine. The [venv](https://docs.python.org/3/library/venv.html#creating-virtual-environments) module, which is part of the Python standard library, or other third-party tools, like [virtualenv](https://virtualenv.pypa.io/en/latest/) or [conda](https://docs.conda.io/en/latest/), can be used.

Create and activate a virtual environment with a TestGen-compatible version of Python (`>=3.11`). The steps may vary based on your operating system and Python installation - the [Python packaging user guide](https://packaging.python.org/en/latest/tutorials/installing-packages/) is a useful reference.

_On Linux/Mac_
```shell
python3 -m venv venv
source venv/bin/activate
```

_On Windows_
```powershell
py -3.13 -m venv venv
venv\Scripts\activate
```

Within the virtual environment, install the TestGen package with pip.
```shell
pip install dataops-testgen
```

Verify that the [_testgen_ command line](https://docs.datakitchen.io/testgen/cli-reference/) works.
```shell
testgen --help
```

### Set up the application database in PostgresSQL

Create a `local.env` file with the following environment variables, replacing the `<value>` placeholders with appropriate values. Refer to the [TestGen Configuration](docs/configuration.md) document for more details, defaults, and other supported configuration.
```shell
# Connection parameters for the PostgreSQL server
export TG_METADATA_DB_HOST=<postgres_hostname>
export TG_METADATA_DB_PORT=<postgres_port>

# Connection credentials for the PostgreSQL server
# This role must have privileges to create roles, users, database and schema so that the application database can be initialized
export TG_METADATA_DB_USER=<postgres_username>
export TG_METADATA_DB_PASSWORD=<postgres_password>

# Set a password and arbitrary string (the "salt") to be used for encrypting secrets in the application database
export TG_DECRYPT_PASSWORD=<encryption_password>
export TG_DECRYPT_SALT=<encryption_salt>

# Set credentials for the default admin user to be created for TestGen
export TESTGEN_USERNAME=<username>
export TESTGEN_PASSWORD=<password>

# Set an arbitrary base64-encoded string to be used for signing authentication tokens
export TG_JWT_HASHING_KEY=<base64_key>

# Set an accessible path for storing application logs
export TESTGEN_LOG_FILE_PATH=<path_for_logs>
```

Source the file to apply the environment variables. For the Windows equivalent, refer to [this guide](https://bennett4.medium.com/windows-alternative-to-source-env-for-setting-environment-variables-606be2a6d3e1).
```shell
source local.env
```

Make sure the PostgreSQL database server is up and running. Initialize the application database for TestGen.
```shell
testgen setup-system-db --yes
```

### Run the application modules

Run the following command to start TestGen. It will open the browser at [http://localhost:8501](http://localhost:8501).

```shell
testgen run-app
```

Verify that you can login to the UI with the `TESTGEN_USERNAME` and `TESTGEN_PASSWORD` values that you configured in the environment variables.

### Optional: Run the TestGen demo setup

The [Data Observability quickstart](https://docs.datakitchen.io/tutorials/quickstart-demo/) walks you through DataOps Data Quality TestGen capabilities to demonstrate how it covers critical use cases for data and analytic teams.

```shell
testgen quick-start
```

In the TestGen UI, you will see that new data profiling and test results have been generated.

## Useful Commands

The [dk-installer](https://github.com/DataKitchen/data-observability-installer/?tab=readme-ov-file#install-the-testgen-application) and [docker compose CLI](https://docs.docker.com/compose/reference/) can be used to operate the TestGen application installed using dk-installer. All commands must be run in the same folder that contains the `dk-installer.py` and `docker-compose.yml` files used by the installation.

### Remove demo data

After completing the quickstart, you can remove the demo data from the application with the following command.

```shell
python3 dk-installer.py tg delete-demo
```

### Upgrade to latest version

New releases of TestGen are announced on the `#releases` channel on [Data Observability Slack](https://data-observability-slack.datakitchen.io/join), and release notes can be found on the [DataKitchen documentation portal](https://docs.datakitchen.io/testgen/release-notes/). Use the following command to upgrade to the latest released version.

```shell
python3 dk-installer.py tg upgrade
```

### Uninstall the application

The following command uninstalls the Docker Compose application and removes all data, containers, and images related to TestGen from your machine.

```shell
python3 dk-installer.py tg delete
```

### Access the _testgen_ CLI

The [_testgen_ command line](https://docs.datakitchen.io/testgen/cli-reference/) can be accessed within the running container.

```shell
docker compose exec engine bash
```

Use `exit` to return to the regular terminal.

### Stop the application

```shell
docker compose down
```

### Restart the application

```shell
docker compose up -d
```
* [Install on Mac/Linux](https://docs.datakitchen.io/testgen/get-started/install-on-mac-linux/)
* [Install on Windows](https://docs.datakitchen.io/testgen/get-started/install-on-windows/)

## What Next?

Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ build-backend = "setuptools.build_meta"

[project]
name = "dataops-testgen"
version = "5.32.2"
version = "5.33.2"
description = "DataKitchen's Data Quality DataOps TestGen"
authors = [
{ "name" = "DataKitchen, Inc.", "email" = "info@datakitchen.io" },
Expand Down
Loading
Loading