Skip to content

Fix test/install, add test-build, verify-results, docs#18

Open
jnasbyupgrade wants to merge 18 commits intoPostgres-Extensions:masterfrom
jnasbyupgrade:reorg-test
Open

Fix test/install, add test-build, verify-results, docs#18
jnasbyupgrade wants to merge 18 commits intoPostgres-Extensions:masterfrom
jnasbyupgrade:reorg-test

Conversation

@jnasbyupgrade
Copy link
Contributor

@jnasbyupgrade jnasbyupgrade commented Feb 27, 2026

test-build: Useful error messages for SQL syntax errors

When CREATE EXTENSION fails due to a syntax error, PostgreSQL reports a
cryptic error with limited context. test-build runs your extension SQL directly
through pg_regress first, so errors show the exact file, line, and position —
cutting debugging time significantly.

Place SQL files in test/build/ to enable. Auto-detects based on file presence.

test/install: Run expensive setup once instead of per-test

Extensions that install dependencies or load fixtures in every test file pay
that cost once per test. test/install runs setup SQL once before the entire
suite, and all regular tests share the resulting database state. This can
dramatically speed up test suites.

Place SQL files in test/install/ to enable. Auto-detects based on file presence.

  • Uses a pg_regress schedule file so install and test files run in a single
    invocation (database state persists across phases)
  • Expected output lives alongside .sql files as test/install/*.out

verify-results: Prevent blessing incorrect test output

make results now refuses to run when tests are failing. Prevents accidentally
updating expected files with wrong output. Disable with
PGXNTOOL_ENABLE_VERIFY_RESULTS=no.

Other changes

  • Add .asc as a recognized asciidoc extension
  • Document all new features in README

Companion PR: Postgres-Extensions/pgxntool-test#7

jnasbyupgrade and others added 15 commits October 7, 2025 15:27
- Add .claude/settings.json with references to test repos
- Add CLAUDE.md documenting the meta-framework architecture
- Include git commit guidelines

Co-Authored-By: Claude <noreply@anthropic.com>
This ensures all projects using pgxntool will ignore Claude Code local
configuration files

Co-Authored-By: Claude <noreply@anthropic.com>
Add *.md to export-ignore to prevent markdown files (including CLAUDE.md)
from being included in extension distributions

Co-Authored-By: Claude <noreply@anthropic.com>
Add `.claude/commands/commit.md` with comprehensive commit workflow that will be
shared with pgxntool-test via symlink. This ensures consistent commit standards
across both repos.

Document META.json generation process in `build_meta.sh` to explain why we
generate from template (PGXN.org doesn't like empty optional fields) and
future possibilities (could generate control files from template).

Co-Authored-By: Claude <noreply@anthropic.com>
Add testing section to `CLAUDE.md` with critical rules: never use
`make installcheck` directly, never run `make results` without
verification, database connection requirements.

Enhance `README.asc` to recommend `make test`, document `make
results` verification workflow, and emphasize pgTap benefits.

Co-Authored-By: Claude <noreply@anthropic.com>
… validation

- Add `make_results.sh` script to handle copying results while respecting
  `output/*.source` files as source of truth
- Update `base.mk` to properly handle ephemeral files from `.source` files:
  - Track `TEST_OUT_SOURCE_FILES` and `TEST_EXPECTED_FROM_SOURCE`
  - Add ephemeral files to `EXTRA_CLEAN` so `make clean` removes them
  - Create `test/results/` directory automatically
  - Remove rule that created `test/output/` directory (it's an optional input)
- Add validation in `dist-only` target to ensure `.gitattributes` is committed
  before creating distribution (git archive only respects export-ignore for
  committed files)
- Update `README.asc` and `CLAUDE.md` to document that development should be
  done from pgxntool-test repository

Co-Authored-By: Claude <noreply@anthropic.com>
- Add `pgtle` make target to generate pg_tle registration SQL for extensions
- Support pg_tle version ranges (1.0.0-1.5.0 and 1.5.0+) with appropriate API usage
- Add `pgtle.sh` script to generate pg_tle SQL from extension control files and versioned SQL
- Update `base.mk` to include pg_tle generation with proper dependencies on SQL files
- Add pg_tle/ directory to EXTRA_CLEAN for `make clean`

- Enhance `.claude/commands/commit.md` to require checking all 3 repos before committing
- Add explicit guidance to commit all repos with changes together (no empty commits)
- Document multi-repo commit workflow in steps

- Update `CLAUDE.md` with pg_tle development context and documentation
- Update `README.asc` and `README.html` with pg_tle usage documentation

Co-Authored-By: Claude <noreply@anthropic.com>
Add "OPTIONAL TEST FEATURES" section documenting `test-build`, `test/install`, and `verify-results` configuration. Also clarify that `verify-results` works with all `pg_regress`-based tests, not just pgTap, and fix assignment to use `:=` to avoid recursion.

Co-Authored-By: Claude <noreply@anthropic.com>
Replace local commit.md with reference to canonical version in pgxntool-test. All three repos (pgxntool, pgxntool-test, pgxntool-test-template) now share the same commit workflow.

Co-Authored-By: Claude <noreply@anthropic.com>
Add new pg_tle version range to handle API changes in 1.4.0:
- Split 1.0.0-1.5.0 range into 1.0.0-1.4.0 and 1.4.0-1.5.0
- Version 1.4.0 added uninstall function (backward-incompatible)
- Version 1.5.0 added schema parameter (another boundary)
- Update `base.mk` pattern rules for three version ranges
- Update `pgtle.sh` documentation and logic for new range

Extract shared utility functions into `lib.sh`:
- Move `error()`, `die()`, and `debug()` functions from `pgtle.sh`
- Update `pgtle.sh` and `setup.sh` to source `lib.sh`
- Reduces code duplication and improves maintainability

Refine `.gitattributes` export rules:
- Add specific excludes: `CLAUDE.md`, `PLAN-*.md`, `.DS_Store`
- Remove blanket `*.md` exclude (too broad)
- Allows README.md and other docs to be included in distributions

Add documentation about directory purity in `CLAUDE.md`:
- Emphasize that pgxntool directory contains ONLY embedded files
- Warn against adding temporary files or planning documents
- Clarify that such files belong in pgxntool-test instead

Co-Authored-By: Claude <noreply@anthropic.com>
Remove references to pgxntool-test-template (now consolidated):
- Template files moved to pgxntool-test/template/
- Remove template directory from `.claude/settings.json`
- Update `CLAUDE.md` for two-repository pattern

Convert `.claude/commands/commit.md` to symlink:
- Development happens in pgxntool-test, avoid duplication
- Symlink: `commit.md -> ../../../pgxntool-test/.claude/commands/commit.md`

Related changes in pgxntool-test:
- Consolidate template files into pgxntool-test/template/
- Simplify commit workflow to two-phase (remove amend step)

Co-Authored-By: Claude <noreply@anthropic.com>
Merges master's control.mk system and update-setup-files.sh while
preserving test-build, test-install, and verify-results features
from the build-test branch.

Key changes:
- Added control.mk.sh for parsing .control files
- Added update-setup-files.sh for 3-way merging after pgxntool-sync
- Added SETUP_FILES/SETUP_SYMLINKS config to lib.sh
- Simplified meta.mk.sh (defers extension handling to control.mk.sh)
- Used master's variable naming (TEST__SOURCE__*)
- Preserved PGXNTOOL_ENABLE_TEST_BUILD feature
- Preserved PGXNTOOL_ENABLE_TEST_INSTALL feature
- Preserved PGXNTOOL_ENABLE_VERIFY_RESULTS feature

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@jnasbyupgrade
Copy link
Contributor Author

Enhancement request: Override mechanism for install/build file lists

I'm planning to use the test/install support for pg_tle integration in my extensions monorepo. The auto-detection is great for the common case, but I need to control which files run and in what order. Two requests:

1. Documented override for test/install and test-build file lists

TEST_INSTALL_SQL_FILES is auto-populated via wildcard, and the schedule is generated directly from it. There's no clean way to:

  • Control ordering: Wildcard gives filesystem order. I need pgtle-setup to run before extensions.
  • Conditionally include files: I have a pgtle-setup.sql that should only run when PGTLE_MODE is set (Docker/RDS uses pg_tle; local PGXS doesn't).

Suggested approach: Expose an intermediate variable that the schedule generation uses, documented as the public API for overriding:

# Default: auto-detected from test/install/*.sql (unordered)
# Projects can override for ordering or conditional inclusion.
PGXNTOOL_INSTALL_TESTS ?= $(notdir $(basename $(TEST_INSTALL_SQL_FILES)))

$(PGXNTOOL_INSTALL_SCHEDULE):
	@echo "# Auto-generated - DO NOT EDIT" > $@
	@for f in $(PGXNTOOL_INSTALL_TESTS); do \
		echo "test: ../install/$$f" >> $@; \
	done

Then projects can do:

include pgxntool/base.mk

# Explicit ordering + conditional pg_tle setup
PGXNTOOL_INSTALL_TESTS = extensions
ifdef PGTLE_MODE
PGXNTOOL_INSTALL_TESTS := pgtle-setup $(PGXNTOOL_INSTALL_TESTS)
endif

Same pattern would be useful for test-build (PGXNTOOL_BUILD_TESTS). The auto-detect remains the default — this just gives projects an escape hatch with a stable API.

2. Document new make targets to help with conflicts

Projects that already hand-coded test-build or verify-results (like my archive extension) will hit recipe conflicts on upgrade. The PGXNTOOL_ENABLE_* variables handle this, but only if people know about them.

Suggestions:

  • README should have a "New Targets" or "Upgrading" section listing all new target names so upgraders can check for conflicts
  • Ideally upgrade.sh could grep the project Makefile for potential conflicts with newly-introduced targets (e.g., grep -w 'test-build\|verify-results' Makefile). This is tricky since you'd be upgrading across potentially many pgxntool versions, but even a best-effort warning would help. Not a blocker for this PR though.

…s, docs

Replace broken separate-invocation test-install with schedule-based approach:
install files now run in the same pg_regress invocation as regular tests via
an auto-generated schedule file with ../install/ relative paths. State created
by install files persists into the main test suite.

- Add `test-build` feature for pre-test SQL validation via `test/build/`
- Add `verify-results` safeguard preventing `make results` when tests fail
- Add `.asc` to recognized asciidoc extensions
- Document test-build, test/install, and verify-results in README
- Update `_.gitignore` for auto-generated `test/install/schedule`
- Remove `.claude/` directory (moved to pgxntool-test)

Related changes in pgxntool-test:
- Add install persistence test validating state survives across test phases
- Rework `test-test-install.bats` for schedule-based approach
- Add template marker files for install persistence validation

Co-Authored-By: Claude <noreply@anthropic.com>
README.asc Outdated
----
test/build/
├── *.sql # SQL test files (checked in)
├── input/ # Optional: .source files for pg_regress processing
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

.source files would also be checked in

# resolves install files from their original location without copying.
#
TEST_INSTALL_SQL_FILES = $(wildcard $(TESTDIR)/install/*.sql)
ifdef PGXNTOOL_ENABLE_TEST_INSTALL
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is almost the same code as for test/build; propose ways (in a comment) it could be refactored. We certainly want the code to remain clear, but I'm hoping we can remove some of the duplication.

See verify-results as well; it's pretty similar as well.

base.mk Outdated
ifeq ($(PGXNTOOL_ENABLE_TEST_BUILD),yes)
TEST_DEPS += test-build
endif
TEST_DEPS += install
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can't these two lines be combined with 267?

#
# Variable: PGXNTOOL_ENABLE_VERIFY_RESULTS
# - Can be set manually in Makefile or command line
# - Allowed values: "yes" or "no" (case-insensitive)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm... lets also add pgtap as an option (and make it the default). This mode would assume that all test output is from pgTap, and look for anything that indicates a test failure. This should be in addition the checks we already have around regression.diff.

Here's the way I'm handling this in another project:

verify-results:
        @failed=0; \
        if grep -Hn '^not ok' test/results/*.out 2>/dev/null | grep -v '# TODO'; then \
                failed=1; \
        fi; \
        if grep -Hn 'Looks like you planned' test/results/*.out 2>/dev/null; then \
                failed=1; \
        fi; \
        if [ $$failed -eq 1 ]; then \
                echo "ERROR: Failed tests found in results files (see above)"; \
                exit 1; \
        fi
        @echo "test/results/ files are clean (no 'not ok' lines or plan count mismatches found, TODO tests ignored)"

base.mk Outdated
@for file in $(TEST_BUILD_SQL_FILES); do \
cp "$$file" "$(TEST_BUILD_SQL_DIR)/$$(basename $$file)"; \
done
@# Create empty expected files if needed
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need a comment on why we even need to do this

README.asc Outdated
├── expected/ # Expected output files (checked in)
│ └── *.out
└── sql/ # GENERATED - do not edit or check in
└── *.sql # Copied from *.sql above; generated from input/*.source
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/;/ and/

SET client_min_messages = WARNING;

-- Install dependencies your extension requires
CREATE EXTENSION IF NOT EXISTS pgtap CASCADE;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

drop this line


----
\set ECHO none
\i test/pgxntool/psql.sql
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a comment that this sets onerrorfail (or whatever the variable is called)

----
\set ECHO none
\i test/pgxntool/psql.sql
\t
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add comment on why this is desirable

-- Run the actual extension SQL (not CREATE EXTENSION)
\i sql/myext.sql

\echo # BUILD TEST SUCCEEDED
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add comment above this line that if the build actually failed psql would stop immediately because of error handling set by psql.sql.

jnasbyupgrade and others added 2 commits March 9, 2026 17:38
- Combine TEST_DEPS lines: merge separate install/installcheck additions
  into a single line alongside testdeps in the initial assignment

- Add pgtap mode to verify-results: new PGXNTOOL_VERIFY_RESULTS_MODE variable
  (default: pgtap) scans test/results/*.out for "not ok" and plan mismatch
  lines; also retains regression.diffs check as fallback. "diffs" mode
  preserves legacy behavior for non-pgtap projects.

- Add comment explaining why empty expected files are created: pg_regress
  aborts immediately if expected/*.out is missing rather than running the
  test and showing a diff.

- Extract test-build shell setup into pgxntool/run-test-build.sh: directory
  creation, SQL file copying, and empty expected file seeding are now in a
  dedicated script; base.mk only contains the make installcheck invocation
  and diffs check.

- Replace PGXNTOOL_distclean with EXTRA_CLEAN: META.json, meta.mk, and
  control.mk are now listed in EXTRA_CLEAN (cleaned by PGXS make clean)
  instead of a custom distclean target. Removes the distclean: target and
  the clean: distclean prerequisite.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
For all three feature-enable variables (PGXNTOOL_ENABLE_TEST_BUILD,
PGXNTOOL_ENABLE_TEST_INSTALL, PGXNTOOL_ENABLE_VERIFY_RESULTS), add a
note in the doc comment that setting the variable to empty on the command
line (e.g. PGXNTOOL_ENABLE_TEST_BUILD=) also disables the feature, the
same as =no.

This works because Make's ifdef sees an empty value as "not defined",
so the validation block is skipped. But the Makefile's default assignment
cannot override a command-line variable (even an empty one), so the
feature-enable ifeq check sees "" and stays disabled.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant