From e67cbf21b252fca52a3c0fd2e0fa636faf9d7538 Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Tue, 14 Apr 2026 08:54:32 -0600 Subject: [PATCH 1/4] docs: document CI test suite registration requirement New test suites are not automatically discovered by CI. They must be explicitly added to the suite matrix in both pr_build_linux.yml and pr_build_macos.yml workflow files. Add a section to the contributor guide explaining this process, including which groups exist and how to choose the right one. --- docs/source/contributor-guide/development.md | 39 ++++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/docs/source/contributor-guide/development.md b/docs/source/contributor-guide/development.md index 47f034c973..be96fcf43d 100644 --- a/docs/source/contributor-guide/development.md +++ b/docs/source/contributor-guide/development.md @@ -438,6 +438,45 @@ make test-rust make test-jvm ``` +### 5. Register New Test Suites in CI + +Comet's CI does not automatically discover test suites. Instead, test suites are explicitly listed +in the GitHub Actions workflow files so they can be grouped by category and run as separate parallel +jobs. This reduces overall CI time. + +If you add a new Scala test suite, you must add it to the `suite` matrix in **both** workflow files: + +- `.github/workflows/pr_build_linux.yml` +- `.github/workflows/pr_build_macos.yml` + +Each file contains a `suite` matrix with named groups such as `fuzz`, `shuffle`, `parquet`, `csv`, +`exec`, `expressions`, and `sql`. Add your new suite's fully qualified class name to the +appropriate group. For example, if you add a new expression test suite, add it to the +`expressions` group: + +```yaml +- name: "expressions" + value: | + org.apache.comet.CometExpressionSuite + # ... existing suites ... + org.apache.comet.YourNewExpressionSuite # <-- add here +``` + +Choose the group that best matches the area your test covers: + +| Group | Covers | +| ------------- | --------------------------------------------------------- | +| `fuzz` | Fuzz testing and data generation | +| `shuffle` | Shuffle operators and related exchange behavior | +| `parquet` | Parquet read/write and native reader tests | +| `csv` | CSV native read tests | +| `exec` | Execution operators, joins, aggregates, plan rules, TPC-* | +| `expressions` | Expression evaluation, casts, and SQL file tests | +| `sql` | SQL-level behavior tests | + +**Important:** The suite lists in both workflow files must stay in sync. If a suite is missing from +the CI matrix, it will not run in CI, even though it passes locally. + ### Pre-PR Summary ```sh From eeaf3565d059093a4fd866e1df20d7e966c5219d Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Tue, 14 Apr 2026 09:01:04 -0600 Subject: [PATCH 2/4] docs: mention pr_missing_suites CI check that detects missing suites --- docs/source/contributor-guide/development.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/source/contributor-guide/development.md b/docs/source/contributor-guide/development.md index be96fcf43d..83c4dc83e5 100644 --- a/docs/source/contributor-guide/development.md +++ b/docs/source/contributor-guide/development.md @@ -474,8 +474,10 @@ Choose the group that best matches the area your test covers: | `expressions` | Expression evaluation, casts, and SQL file tests | | `sql` | SQL-level behavior tests | -**Important:** The suite lists in both workflow files must stay in sync. If a suite is missing from -the CI matrix, it will not run in CI, even though it passes locally. +**Important:** The suite lists in both workflow files must stay in sync. A separate CI check +(`.github/workflows/pr_missing_suites.yml`) runs `dev/ci/check-suites.py` on every pull request. +It scans for all `*Suite.scala` files in the repository and verifies that each one appears in both +workflow files. If any suite is missing, this check will fail and block the PR. ### Pre-PR Summary From 3fc653629e803b0b64ad1f4387e7a6d55fae88d1 Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Tue, 14 Apr 2026 09:02:12 -0600 Subject: [PATCH 3/4] fix: use synchronize instead of edited trigger for missing suites check The edited event fires when the PR title/body changes, not on new pushes. Replace with synchronize so the check re-runs when new commits are pushed to the PR branch. --- .github/workflows/pr_missing_suites.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/pr_missing_suites.yml b/.github/workflows/pr_missing_suites.yml index 89589c26cc..8efc644003 100644 --- a/.github/workflows/pr_missing_suites.yml +++ b/.github/workflows/pr_missing_suites.yml @@ -19,7 +19,7 @@ name: Check that all test suites are added to PR workflows on: pull_request: - types: [opened, edited, reopened] + types: [opened, synchronize, reopened] jobs: check-missing-suites: From 46d69414a9322ed00ee27741a598bf911f9a57b4 Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Tue, 14 Apr 2026 09:02:31 -0600 Subject: [PATCH 4/4] ci: run missing suites check on pushes to main --- .github/workflows/pr_missing_suites.yml | 3 +++ 1 file changed, 3 insertions(+) diff --git a/.github/workflows/pr_missing_suites.yml b/.github/workflows/pr_missing_suites.yml index 8efc644003..be944dbdb4 100644 --- a/.github/workflows/pr_missing_suites.yml +++ b/.github/workflows/pr_missing_suites.yml @@ -18,6 +18,9 @@ name: Check that all test suites are added to PR workflows on: + push: + branches: + - main pull_request: types: [opened, synchronize, reopened]