PDX-486: feat(mcp) — testrun zero-tests guard (Thread B, PR 1/3) by mrdailey99 · Pull Request #185 · ProvarTesting/provardx-cli

mrdailey99 · 2026-05-19T21:20:00Z

Context

When provar_automation_testrun exits 0 but no tests actually executed — most often because the provardx-properties.json has testCase instead of testCases (or vice versa, or any unknown selector key) — the response previously contained an empty steps[] array and no other signal. Agents (and humans) had no way to distinguish "intentionally zero tests ran" from "typo silently dropped my entire selector," leading to wasted retry loops and downstream confusion when later tooling tried to read results.

This PR is Thread B, PR 1 of 3 in the VALIDATE-TYPO / propagation track. It introduces the RUN-001 zero-tests guard on provar_automation_testrun.

Why

Closes the silent-failure gap on the most common Provar properties typo (testCase ↔ testCases).
Lands the first consumer of the shared WARNING_CODES enum from PDX-485 (PDX-485: chore(mcp) — shared warningCodes.ts enum #182) in production tooling.
Sets up a single JUnit introspection point (introspectJUnit) so the future JUNIT-001 mismatch warning (PDX-491, Thread B PR 3) can extend it cleanly without re-parsing the XML.

What changed

src/mcp/tools/automationTools.ts
- New introspectJUnit(config) helper returning { steps, stepCount, parseWarning, resultsPathResolved } — single read-point for downstream warning emitters.
- Success-path branch now appends a formatWarning(WARNING_CODES.RUN_001, …) entry to a warnings[] array when resultsPathResolved && stepCount === 0. Strictly additive — exitCode and isError are untouched.
- Tool description now mentions the RUN-001 surface and the testCase/testCases spelling pitfall.
docs/mcp.md — documents the new warnings[] field on the testrun output and references RUN-001.
test/unit/mcp/automationTools.test.ts — four new tests covering: fires on empty <testsuite>, does not fire when steps ≥ 1, does not fire on non-zero exit, does not fire when results dir is unresolved. The expected warning string is built via formatWarning (no inline duplicates).

Out of scope (sibling PRs)

Validator unknown-key detection — VALIDATE-TYPO-A (Thread A PR 1).
provar_automation_config_load propagating SCHEMA-001 — scheduled for Thread B PR 3 stitch commit after both halves merge.
properties_file inline param (PDX-487) — Thread B PR 2.
JUNIT-001 mismatch warning (PDX-491) — Thread B PR 3 (will extend introspectJUnit).

Test plan

yarn compile clean.
yarn lint clean.
Full unit suite (node_modules/.bin/nyc node_modules/.bin/mocha "test/**/*.test.ts") — 1163 passing, 1 pending, 0 failing.
automationTools.test.ts — 80/80 passing, includes the four new RUN-001 scenarios.
MCP smoke test — could not run locally on this machine due to "No activated Provar license found" (environmental, unrelated to this change). CI smoke run will validate end-to-end.

Dependencies

Depends on PDX-485: chore(mcp) — shared warningCodes.ts enum #182 (PDX-485 chore/PDX-485-warning-codes-enum) — uses WARNING_CODES.RUN_001 and formatWarning from src/mcp/utils/warningCodes.ts. PDX-485 is already merged into this branch.
Sibling of Thread A PR-1 (VALIDATE-TYPO-A) — that PR adds the validator unknown-key detection and a testcases-typo.json fixture. This PR uses its own inline JUnit XML fixtures to avoid coupling.

…-thread feedback codes RCA: Six sibling Provar MCP thread PRs (validation, properties, automation, RCA, JUnit, parallel-mode tuning) each independently emit warnings with ad-hoc string prefixes — `WARNING:`, `[provarHome]`, `WARN —` — producing inconsistent surface area for AI agents downstream and making cross-tool typo guidance ("Did you mean...?") harder to standardise. PDX-485 wants a single canonical enum the threads can import, so warning codes are coined once, formatted once, and documented once. Fix: Adds src/mcp/utils/warningCodes.ts exporting WARNING_CODES (PROVARHOME-001, DATA-001, PARALLEL-001, SCHEMA-001, RUN-001, JUNIT-001), a WarningCode type derived from the enum, and a formatWarning(code, message, suggestion?) helper that emits `WARNING [<CODE>]: <message>` and appends ` Did you mean '<suggestion>'?` when a suggestion is provided. No call sites are touched in this PR — surface area is intentionally minimal so the six sibling thread PRs can import and adopt without merge conflicts. docs/mcp.md gains a new "Warning codes" reference table linked from the table of contents; per-row meanings are placeholders that subsequent thread PRs will refine. Tests: New test/unit/mcp/utils/warningCodes.test.ts covers (1) each WARNING_CODES key maps to its expected wire string, (2) formatWarning without a suggestion returns the prefixed message exactly, (3) formatWarning with a suggestion appends the "Did you mean" suffix exactly, (4) an empty-string suggestion is treated as no suggestion. Validation: yarn compile clean, yarn lint clean, full mocha 1159 passing / 0 failing. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

RCA: provar_automation_testrun returned exitCode 0 with an empty steps[] array when a misspelled testCase/testCases key in provardx-properties.json caused the run to select nothing. Agents (and humans) had no signal to disambiguate "intentionally zero tests ran" from "typo silently dropped my entire selector", leading to wasted retry loops and downstream confusion when later tooling complained about missing results. Fix: Introduce a single JUnit introspection helper (introspectJUnit) that returns stepCount, parseWarning, and a resultsPathResolved flag. When sf exits 0 and the JUnit XML for the resolved results dir contains zero test cases, the response gains a warnings[] entry built via formatWarning(WARNING_CODES.RUN_001, ...) pointing the agent at the testCase/testCases spelling pitfall. The warning is additive — exitCode and isError remain unchanged. The helper is structured so future PDX-491 JUNIT-001 (expected-vs-actual mismatch) can read stepCount from the same struct without re-parsing. Updated the tool description, docs/mcp.md (RUN-001 output documentation), and added four targeted tests (positive, success-with-steps negative, non-zero-exit negative, unresolved-results-dir silent case).

github-actions · 2026-05-19T21:20:24Z

Quality Orchestrator

🟢 LOW · 14 / 100 · Touches: utils. All changed files have mapped tests.

🧪 Tests to Run · Running 2 of 52 tests

unit/mcp/automationTools.test.ts
unit/mcp/utils/warningCodes.test.ts

▶ Run command

npx vitest run \
  unit/mcp/automationTools.test.ts \
  unit/mcp/utils/warningCodes.test.ts

_{⚡ quality-orchestrator · /qo stub <file> · qo analyze-local}

Copilot

Pull request overview

This PR adds a standardized warning-code surface to MCP and introduces a RUN-001 “zero tests executed” guard on provar_automation_testrun, so consumers can distinguish “success but nothing ran” from a normal successful run with results.

Changes:

Added WARNING_CODES + formatWarning() utility and corresponding unit tests.
Added JUnit introspection helper + success-path warnings[] emission for RUN-001 when step count is zero.
Updated MCP docs and automationTools tests to document/cover the new warnings[] output shape.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
`src/mcp/utils/warningCodes.ts`	Introduces shared warning code constants and canonical formatter.
`src/mcp/tools/automationTools.ts`	Adds `introspectJUnit()` and emits `RUN-001` warning on zero-step successful testruns.
`test/unit/mcp/utils/warningCodes.test.ts`	Adds unit coverage for code mapping and formatter behavior.
`test/unit/mcp/automationTools.test.ts`	Adds unit scenarios for the `RUN-001` warning behavior.
`docs/mcp.md`	Documents warning codes and adds `warnings[]`/`RUN-001` to testrun tool docs.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+        if (junit.resultsPathResolved && junit.stepCount === 0) {
+          const warningStr = formatWarning(WARNING_CODES.RUN_001, ZERO_TESTS_MESSAGE);
+          // Append rather than overwrite so future warning emitters (e.g. JUNIT-001 mismatch
+          // in PDX-491) can coexist on the same response without stepping on each other.
+          const existing = response['warnings'] as string[] | undefined;
+          response['warnings'] = existing ? existing.concat(warningStr) : [warningStr];


 Each entry represents one test case. `status` is `"pass"`, `"fail"`, or `"skip"`. If the results directory cannot be located or contains no JUnit XML, `details.warning` explains why and `steps` is absent.

+**Zero-tests guard (RUN-001):** when the sf command exits 0 but the JUnit report contains zero executed test cases, the response includes a `warnings[]` array containing a [`RUN-001`](#warning-codes) message. This is almost always a typo such as `testCase` vs `testCases` (or some other unknown key) in `provardx-properties.json` — the run silently selected nothing. The warning is additive and never flips `exitCode` or sets `isError`; the failure surface remains driven by the underlying `sf` exit code.
+
 **Error codes:** `AUTOMATION_TESTRUN_FAILED`, `SF_NOT_FOUND`
+**Warning codes:** `RUN-001` (zero tests executed despite success)


+    describe('RUN-001 zero-tests-executed warning', () => {
+      const expectedWarning = formatWarning(
+        WARNING_CODES.RUN_001,
+        'Test run exited successfully but zero tests were executed. ' +
+          'Check the testCase / testCases (note spelling) field in provardx-properties.json.'
+      );
+
+      it('fires when exit is 0 and JUnit step count is 0 (empty <testsuite>)', () => {


mrdailey99 and others added 2 commits May 19, 2026 15:33

Copilot AI review requested due to automatic review settings May 19, 2026 21:20

Copilot started reviewing on behalf of mrdailey99 May 19, 2026 21:20 View session

Copilot AI reviewed May 19, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PDX-486: feat(mcp) — testrun zero-tests guard (Thread B, PR 1/3)#185

PDX-486: feat(mcp) — testrun zero-tests guard (Thread B, PR 1/3)#185
mrdailey99 wants to merge 2 commits into
developfrom
fix/PDX-486-validate-typo-b-zero-tests-guard

mrdailey99 commented May 19, 2026

Uh oh!

github-actions Bot commented May 19, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

mrdailey99 commented May 19, 2026

Context

Why

What changed

Out of scope (sibling PRs)

Test plan

Dependencies

Uh oh!

github-actions Bot commented May 19, 2026

Quality Orchestrator

🧪 Tests to Run · Running 2 of 52 tests

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants