fix: skip PlaywrightCheck constructs during pw-test [RED-155]

[miliberlin]

Background

Reported internally by Stefan: when a project has a PlaywrightCheck construct defined in a .check.ts file, pw-test incorrectly includes it alongside the vanilla Playwright tests. This causes pw-test to run two checks — the construct-based check and the ad-hoc one — when it should only run the ad-hoc one:

__checks__/critical-suite.check.ts
  ✔ Critical User Flows (8s)
playwright.config.ts
  ✖ Playwright Test:  (11s)

The construct-based check passes (self-contained config/deps), but the ad-hoc playwright.config.ts check fails because the construct's bundling context clobbers its dependency resolution.

Root cause

parseProject() was not scoped to the pw-test command:

1. loadAllCheckFiles() ran for all commands including pw-test, picking up .check.ts files that contain new PlaywrightCheck(...) constructs
2. checklyConfigConstructs (from config file evaluation) were added to the project unfiltered, even for pw-test

This meant pw-test would end up with extra PlaywrightCheck instances in the project. Since the checkFilter keeps all PlaywrightCheck instances, both the construct-based check and the ad-hoc check made it into the bundle. The construct's dependency tree interfered with the ad-hoc check's bundling.

Changes

Two changes in parseProject() when currentCommand === 'pw-test':

1. Filter PlaywrightCheck constructs from checklyConfigConstructs before adding to the project
2. Skip loadAllCheckFiles() entirely

Before/after (released CLI v7.0.1 vs this fix)

Before:

Running 2 checks in eu-central-1.

playwright.config.ts
  ✔ Playwright Test (15s)
src/__checks__/synthetics/07-pw-check-construct.check.ts
  ✔ File Based PW Check (should be ignored by pw-test) (13s)

2 passed, 2 total

After:

Running 1 checks in eu-central-1.

playwright.config.ts
  ✔ Playwright Test:  (15s)

1 passed, 1 total

Test results

Each test project includes a PlaywrightCheck construct in a .check.ts file, a local project dependency (src/utils/selectors.ts), and an npm devDependency (lodash). The test files import both to verify dependency resolution works correctly.

Issue	Scenario	Before (v7.0.1)	After (fix)
RED-122	PlaywrightCheck in .check.ts file	2 checks run	1 check run
RED-119	PW construct causes config to be ignored	2 checks run	1 check run
RED-118	PW construct blocks dependency resolution	2 checks run	1 check run

All three child issues verified fixed. The released CLI (v7.0.1) consistently runs 2 checks (the bug), while the fix correctly runs only 1 ad-hoc check. Both local and npm dependencies resolve correctly with the fix.

Before sessions: RED-122 · RED-119 · RED-118
After sessions: RED-122 · RED-119 · RED-118

🤖 Generated with Claude Code

[miliberlin]

Summary of why we need to tweak the config for this test below. I think this is correct since this is how a user would set up the project, but please double-check.

Before our fix, the webServer test worked like this:

1. parseProject() runs with --emulate-pw-test
2. loadAllCheckFiles() picks up test.check.ts, which calls new PlaywrightCheck(...) with playwrightConfigPath: './playwright.config.ts'
3. The PlaywrightCheck constructor runs validateWebServerConfig(), sees webServer in the playwright config, and emits the warning

Our fix adds if (currentCommand !== 'pw-test') around loadAllCheckFiles(), so step 2 never happens — test.check.ts is never loaded, no PlaywrightCheck is created, no warning fires.

But there's a second code path that creates PlaywrightCheck instances: loadPlaywrightChecks(). This function still runs for pw-test. It creates an "ad-hoc" PlaywrightCheck when playwrightConfigPath is set in the checkly config — this is the same path that real users hit when they have checks: { playwrightConfigPath: './playwright.config.ts' } in their checkly.config.ts.

By adding playwrightConfigPath to the fixture's config, the ad-hoc check gets created via loadPlaywrightChecks() instead of via loadAllCheckFiles(). The webServer validation fires on this ad-hoc check, and the test passes.

This also makes the test more realistic — in practice, pw-test users always have playwrightConfigPath in their config (that's how pw-test knows which playwright config to use). The old fixture relied on a .check.ts file to create the PlaywrightCheck, which is exactly the scenario our fix is designed to skip.

[miliberlin]

@ferrandiaz would you have time to check this today? We'd like to close the Polish pw-test project soonish.

[sorccu]

I don't like this currentCommand thing, especially now that it's being used more. It feels like we're just adding a whole bunch of special cases based on the identity of the caller, which is IMO unpredictable and just ass in general. Instead of the caller identifying itself, it should request specific behavior via a relevant flag.

Maybe we can keep this as is if you need this out soon but I do NOT want to see this become a standard.

[miliberlin]

@sorccu I gave replacing currentCommand a shot. Wdyt? The name of the flags can probably be improved, but I think as long as currentCommand exists it will be picked up again and again, better to get it out of the codebase if you don't like it.