introduce-realm-enabled-tests by tintinthong · Pull Request #4191 · cardstack/boxel

tintinthong · 2026-03-16T13:16:35Z

Realm needs to be running to run the test bcos they serve the test code

How to run locally

# Manual browser (services must be running)
open http://localhost:4200/tests?liveTest=true&realmURL=http://localhost:4201/catalog/

# Testem (services must be running, dist must be built)
cd packages/host
pnpm build
ember test --config-file testem-live.js --path ./dist

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7c22d07243

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

packages/host/ember-cli-build.js

github-actions · 2026-03-16T13:33:07Z

Preview deployments

github-actions · 2026-03-16T13:46:48Z

Host Test Results

1 files ± 0 1 suites ±0 2h 6m 42s ⏱️ + 2m 43s
2 030 tests +184 2 015 ✅ +186 15 💤 ±0 0 ❌ - 1
2 045 runs +185 2 030 ✅ +188 15 💤 ±0 0 ❌ - 2

Results for commit 8ef968d. ± Comparison against base commit 86cb8e6.

This pull request removes 1 and adds 185 tests. Note that renamed tests count towards both.

Chrome ‑ Global error: Uncaught TypeError: Failed to fetch at http://localhost:7357/assets/chunk.ba748e4cc236bfaae5b8.js, line 156670  While executing test: Acceptance | code-submode | field playground > single realm: can create new field instance (has preexisting Spec)

Chrome ‑ Acceptance | code-submode | field playground > multiple realms: can autogenerate new Spec and field instance (no preexisting Spec)
Chrome ‑ Acceptance | code-submode | field playground > multiple realms: can autogenerate new field instance when spec exists but has no examples
Chrome ‑ Acceptance | code-submode | field playground > multiple realms: can create new field instance (has preexisting Spec)
Chrome ‑ Acceptance | code-submode | field playground > single realm: can autogenerate new field instance when spec exists but has no examples
Chrome ‑ Acceptance | code-submode | field playground > single realm: can create new field instance (has preexisting Spec)
Chrome ‑ Acceptance | code-submode | field playground > single realm: can request AI assistant to bulk generate samples
Chrome ‑ Acceptance | code-submode | field playground > single realm: can request AI assistant to fill in sample data
Chrome ‑ Acceptance | code-submode | field playground > single realm: does not persist the wrong card for field
Chrome ‑ Acceptance | code-submode | field playground > single realm: editing compound field instance live updates the preview
Chrome ‑ Acceptance | code-submode | field playground > single realm: has default templates when a format template is not provided
…

♻️ This comment has been updated with latest results.

Copilot

Pull request overview

This PR introduces a “live test” runner page intended to execute realm-hosted (card) tests in a host-like Ember/QUnit environment, and refactors an experiments realm sample card test to use a consolidated host test setup helper.

Changes:

Add a dedicated live-test.html + live-test.js runner to load realm modules and execute their runTests() exports under QUnit.
Update the standard host tests/test-helper.js boot to skip normal initialization when running on the live-test page.
Add a setupCardTest(hooks) helper and update sample-command-card.gts tests (including a new search assertion).

Reviewed changes

Copilot reviewed 7 out of 8 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
packages/host/tests/test-helper.js	Skips normal test initialization on `live-test.html`; imports live-test runner module.
packages/host/tests/live-test.js	New runtime for loading realm test modules and starting QUnit manually.
packages/host/tests/live-test.html	New test runner HTML page with QUnit UI and a “Start QUnit” button.
packages/host/tests/helpers/index.gts	Adds `setupCardTest()` convenience helper used by realm-hosted tests.
packages/host/ember-cli-build.js	Formatting-only change.
packages/host/app/lib/externals.ts	Adds shims for QUnit/test helpers/test modules to support realm-loaded tests.
packages/experiments-realm/sample-command-card.gts	Refactors test setup to `setupCardTest()` and adds a search-based assertion.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

packages/host/app/lib/externals.ts

packages/host/tests/test-helper.js

+import './live-test';

-QUnit.dump.maxDepth = 20;
+const isLiveTest = new URL(window.location.href).pathname.endsWith(
+  '/live-test.html',
+);

-useTestWaiters(TestWaiters);
-setApplication(Application.create(config.APP));
+if (!isLiveTest) {
+  QUnit.dump.maxDepth = 20;


Copilot

Pull request overview

This PR adds a “live” test runner mode that can load and execute test modules served from a running realm (instead of only running the host’s compiled test bundle), and wires it into CI.

Changes:

Add a live-test bootstrap (tests/live-test.js + optional tests/live-test.html) and gate the normal tests/test-helper.js startup when ?liveTest=true.
Add a Testem config + wait-for-services script + CI job to run these realm-backed tests headlessly.
Add a convenience helper (setupCardTest) and a sample realm card test that validates indexing/search against a live realm.

Reviewed changes

Copilot reviewed 13 out of 14 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
packages/host/tests/test-helper.js	Skips standard Ember Exam/QUnit startup when running in live-test mode; imports live-test bootstrap.
packages/host/tests/live-test.js	New live-test runner: boot app, discover realm modules, shim helpers into loader, run `runTests()` exports.
packages/host/tests/live-test.html	Optional standalone HTML runner with UI controls/status for live tests.
packages/host/tests/helpers/index.gts	Adds `setupCardTest()` helper for realm card tests.
packages/host/testem-live.js	New Testem config to run `tests/index.html?liveTest=true...` and emit junit in CI.
packages/host/scripts/live-test-wait-for-servers.sh	Waits for dependent services/readiness endpoints then runs live tests.
packages/host/package.json	Adds `test:live` script.
packages/host/ember-cli-build.js	No functional change (whitespace).
packages/host/app/lib/externals.ts	Adds QUnit shim + test-module shims intended to let cards with colocated tests load.
packages/host/.eslintrc.js	Allows linting `testem-live.js`.
packages/catalog-realm/sample-command-card.gts	Updates sample card’s colocated tests; adds an indexing/search assertion.
mise-tasks/test-services-host	Keeps `sample-command-card.gts` when preparing the catalog realm for tests.
.github/workflows/ci-host.yaml	Adds a new “Live Tests (realm)” job to CI.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-03-24T03:28:36Z

packages/host/app/lib/externals.ts

+  const testOnlyStub = (moduleId: string) =>
+    new Proxy(
+      {},
+      {
+        get: () => {
+          throw new Error(
+            `${moduleId} is only available in a test environment.`,
+          );
+        },
+      },
+    );


testOnlyStub throws on any property access via Proxy.get. That will break module evaluation for realm cards that have static imports like import { click } from '@ember/test-helpers' or import { setupCardTest } from '@cardstack/host/tests/helpers', because the export is accessed during module instantiation (before any test code runs). To allow cards to load outside tests, the stub should return callable functions (or undefined) and only throw when those functions are invoked, not when the export is imported.

Suggested change

const testOnlyStub = (moduleId: string) =>

new Proxy(

{},

{

get: () => {

throw new Error(

`${moduleId} is only available in a test environment.`,

);

},

},

);

const testOnlyStub = (moduleId: string) => {

const throwingFunction = function () {

throw new Error(`${moduleId} is only available in a test environment.`);

};

return new Proxy(

{},

{

get() {

return throwingFunction;

},

},

);

};

Copilot · 2026-03-24T03:28:36Z

packages/host/tests/live-test.js

+
+if (!isLiveTest) {
+  console.warn('[live-test] Skipping initialization outside live-test.html');


live-test.js is imported unconditionally from tests/test-helper.js, so this console.warn will execute for every normal (non-live) test run and add noise to CI logs. Consider removing the warning (or gating it behind an explicit debug flag) so standard test runs stay quiet.

Suggested change

if (!isLiveTest) {

console.warn('[live-test] Skipping initialization outside live-test.html');

const isLiveTestDebug = url.searchParams.has('liveTestDebug');

if (!isLiveTest) {

if (isLiveTestDebug) {

console.warn('[live-test] Skipping initialization outside live-test.html');

}

Copilot · 2026-03-24T03:28:37Z

packages/host/tests/test-helper.js

+// eslint-disable-next-line ember/no-test-import-export
+import './live-test';

-QUnit.dump.maxDepth = 20;
+const url = new URL(window.location.href);
+const isLiveTest =
+  url.pathname.endsWith('/live-test.html') || url.searchParams.has('liveTest');


Static-importing ./live-test means its side effects run on every test entrypoint load. Combined with live-test.js's non-live console.warn, this will spam warnings in regular test runs. Prefer conditionally loading the live-test bootstrap only when liveTest is present (e.g., via import('./live-test') inside an if (isLiveTest) block) or ensure the imported module is silent when inactive.

Copilot · 2026-03-24T03:28:37Z

packages/host/tests/live-test.js

+      } catch {
+        testModuleNames = [];


The fetch(realmURL) error is swallowed and replaced with testModuleNames = [], which can lead to a confusing "0 tests" outcome without indicating why test discovery failed. It would be more diagnosable to log the exception (or surface a failing test) when realm discovery fails, especially in CI.

Suggested change

} catch {

testModuleNames = [];

} catch (error) {

console.error('Failed to discover tests from realm', realmURL, error);

throw error;

tintinthong changed the title ~~introduce is live test~~ introduce-realm-enabled-tests Mar 16, 2026

tintinthong marked this pull request as draft March 16, 2026 13:17

chatgpt-codex-connector bot reviewed Mar 16, 2026

View reviewed changes

packages/host/ember-cli-build.js Outdated Show resolved Hide resolved

test working live-test html

891ea2c

tintinthong force-pushed the is-live-test branch from 518f28f to 891ea2c Compare March 19, 2026 10:37

tintinthong requested a review from Copilot March 19, 2026 10:57

Copilot AI reviewed Mar 19, 2026

View reviewed changes

tintinthong added 9 commits March 19, 2026 19:09

fix lint

a486ac3

allow non -test files

34fd893

Merge branch 'main' into is-live-test

e519d04

noop shim for test helpers

76adf2f

cleanup

5eaa4ed

run ci test from publicly readable realm

9818cf1

lint and prettify

8b04074

add eslint

d5bb4a7

Merge branch 'main' into is-live-test

8ef968d

tintinthong requested a review from Copilot March 24, 2026 03:23

Copilot started reviewing on behalf of tintinthong March 24, 2026 03:23 View session

Copilot AI reviewed Mar 24, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

introduce-realm-enabled-tests#4191

introduce-realm-enabled-tests#4191
tintinthong wants to merge 10 commits intomainfrom
is-live-test

tintinthong commented Mar 16, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector bot left a comment

Uh oh!

Uh oh!

github-actions bot commented Mar 16, 2026

Uh oh!

github-actions bot commented Mar 16, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Mar 24, 2026

Uh oh!

Copilot AI Mar 24, 2026

Uh oh!

Copilot AI Mar 24, 2026

Uh oh!

Copilot AI Mar 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants


		if (!isLiveTest) {
		console.warn('[live-test] Skipping initialization outside live-test.html');

-      } catch {
-        testModuleNames = [];
+      } catch (error) {
+        console.error('Failed to discover tests from realm', realmURL, error);
+        throw error;

Conversation

tintinthong commented Mar 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

How to run locally

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

github-actions bot commented Mar 16, 2026

Preview deployments

Uh oh!

github-actions bot commented Mar 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Host Test Results

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Mar 24, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 24, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 24, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 24, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

tintinthong commented Mar 16, 2026 •

edited

Loading

github-actions bot commented Mar 16, 2026 •

edited

Loading