Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 10 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -436,8 +436,17 @@ Playwright CLI will load config from `.playwright/cli.config.json` by default so

/**
* Whether to save snapshots, console messages, network logs and other session logs to a file or to the standard output. Defaults to "stdout".
* - "stdout": Output to console immediately
* - "file": Save all output to files in outputDir
* - "fileOnLargeOutput": Output to stdout if small, otherwise save to file (controlled by maxStdOutputSize)
Comment on lines +439 to +441
Copy link

Copilot AI Apr 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The docs say fileOnLargeOutput will "Output to stdout if small" / "Output to console immediately", but the current implementation buffers output and only prints it after program() finishes. Please either update the implementation to stream output as intended or clarify in the docs that output may be delayed while the CLI decides whether to write to a file.

Suggested change
* - "stdout": Output to console immediately
* - "file": Save all output to files in outputDir
* - "fileOnLargeOutput": Output to stdout if small, otherwise save to file (controlled by maxStdOutputSize)
* Output may be buffered before it is written, especially when the CLI needs to decide whether to keep output on stdout or save it to a file.
* - "stdout": Write output to stdout. Depending on buffering, output may be emitted after the session completes rather than immediately.
* - "file": Save all output to files in outputDir
* - "fileOnLargeOutput": Buffer output while determining its size; write to stdout if it stays within maxStdOutputSize, otherwise save it to file

Copilot uses AI. Check for mistakes.
*/
outputMode?: 'file' | 'stdout';
outputMode?: 'file' | 'stdout' | 'fileOnLargeOutput';

/**
* When outputMode is "fileOnLargeOutput", defines the maximum size in bytes before switching to file output.
* Defaults to 1048576 (1MB).
*/
maxStdOutputSize?: number;

console?: {
/**
Expand Down
30 changes: 30 additions & 0 deletions TEST_RESULTS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
# Testing Validation Results for fileOnLargeOutput Feature

## Test Summary
| Test Case | Configuration | Expected | Result |
|-----------|--------------|----------|--------|
| Default threshold | `outputMode: "fileOnLargeOutput"` (no maxStdOutputSize) | Output to stdout | ✓ Pass |
| Small threshold | `maxStdOutputSize: 100` | Output to file | ✓ Pass |
| Zero threshold | `maxStdOutputSize: 0` | Output to file | ✓ Pass |
| Large threshold | `maxStdOutputSize: 100000` | Output to stdout | ✓ Pass |
| stdout mode | `outputMode: "stdout"` | Output to stdout | ✓ Pass |
| file mode | `outputMode: "file"` | Output to stdout (existing behavior) | ✓ Pass |
Copy link

Copilot AI Apr 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This report says outputMode: "file" results in "Output to stdout (existing behavior)", but the README schema comment in this PR states that "file" saves all output to files in outputDir. Please align the documentation/test report so users aren't left with conflicting behavior descriptions.

Suggested change
| file mode | `outputMode: "file"` | Output to stdout (existing behavior) | ✓ Pass |
| file mode | `outputMode: "file"` | Output to file | ✓ Pass |

Copilot uses AI. Check for mistakes.
| No config | No config file | Default behavior | ✓ Pass |
| snapshot command | `maxStdOutputSize: 100` | Output to file | ✓ Pass |
| --help flag | `outputMode: "fileOnLargeOutput"` | No wrapping | ✓ Pass |

## Manual Tests Performed
1. **Small output (100 byte threshold)**: Output exceeded threshold, saved to file with message
2. **Default threshold (1MB)**: Small output printed to stdout normally
3. **Zero threshold**: All output saved to file
4. **Empty config (only outputMode set)**: Uses default 1MB threshold, works correctly

## Edge Cases Tested
- Config value `0` for maxStdOutputSize → correctly treated as threshold
- Config without outputDir → defaults to temp directory
- Config without maxStdOutputSize → defaults to 1MB

## Verification
- Feature implements issue requirements: https://github.com/microsoft/playwright-cli/issues/339
- All existing modes (stdout, file) continue to work unchanged
- New fileOnLargeOutput mode correctly switches based on output size
89 changes: 88 additions & 1 deletion playwright-cli.js
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,93 @@
*/

const { program } = require('playwright-core/lib/tools/cli-client/program');
const fs = require('fs');
const path = require('path');
const os = require('os');
const packageJson = require('./package.json');

program({ embedderVersion: packageJson.version });
const DEFAULT_MAX_OUTPUT_SIZE = 1048576;

function getOutputMode() {
try {
const configPath = path.join(process.cwd(), '.playwright', 'cli.config.json');
if (fs.existsSync(configPath)) {
const config = JSON.parse(fs.readFileSync(configPath, 'utf8'));
return config.outputMode || null;
}
} catch {}
return null;
}

function getMaxStdOutputSize() {
try {
const configPath = path.join(process.cwd(), '.playwright', 'cli.config.json');
if (fs.existsSync(configPath)) {
const config = JSON.parse(fs.readFileSync(configPath, 'utf8'));
if (config.maxStdOutputSize !== undefined) {
return config.maxStdOutputSize;
}
}
} catch {}
return DEFAULT_MAX_OUTPUT_SIZE;
Comment on lines +41 to +47
Copy link

Copilot AI Apr 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maxStdOutputSize is returned without validation. If the config contains null, a string, NaN, or a negative value, the comparison outputSize > maxSize will behave unexpectedly (e.g., null coerces to 0). Consider validating/coercing to a finite non-negative number and falling back to the default when invalid.

Copilot uses AI. Check for mistakes.
}

function getOutputDir() {
try {
const configPath = path.join(process.cwd(), '.playwright', 'cli.config.json');
if (fs.existsSync(configPath)) {
const config = JSON.parse(fs.readFileSync(configPath, 'utf8'));
if (config.outputDir) {
return config.outputDir;
}
}
} catch {}
Comment on lines +26 to +59
Copy link

Copilot AI Apr 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The empty catch {} blocks in the config readers will silently ignore JSON parse errors or filesystem errors, which makes misconfiguration hard to diagnose (the CLI will just fall back to defaults). Consider emitting a short warning (or at least handling JSON parse errors explicitly) when the config exists but cannot be parsed.

Suggested change
function getOutputMode() {
try {
const configPath = path.join(process.cwd(), '.playwright', 'cli.config.json');
if (fs.existsSync(configPath)) {
const config = JSON.parse(fs.readFileSync(configPath, 'utf8'));
return config.outputMode || null;
}
} catch {}
return null;
}
function getMaxStdOutputSize() {
try {
const configPath = path.join(process.cwd(), '.playwright', 'cli.config.json');
if (fs.existsSync(configPath)) {
const config = JSON.parse(fs.readFileSync(configPath, 'utf8'));
if (config.maxStdOutputSize !== undefined) {
return config.maxStdOutputSize;
}
}
} catch {}
return DEFAULT_MAX_OUTPUT_SIZE;
}
function getOutputDir() {
try {
const configPath = path.join(process.cwd(), '.playwright', 'cli.config.json');
if (fs.existsSync(configPath)) {
const config = JSON.parse(fs.readFileSync(configPath, 'utf8'));
if (config.outputDir) {
return config.outputDir;
}
}
} catch {}
function readCliConfig() {
const configPath = path.join(process.cwd(), '.playwright', 'cli.config.json');
if (!fs.existsSync(configPath)) {
return null;
}
try {
return JSON.parse(fs.readFileSync(configPath, 'utf8'));
} catch (e) {
console.warn(`Warning: Failed to read Playwright CLI config at ${configPath}: ${e.message}`);
return null;
}
}
function getOutputMode() {
const config = readCliConfig();
return config ? config.outputMode || null : null;
}
function getMaxStdOutputSize() {
const config = readCliConfig();
if (config && config.maxStdOutputSize !== undefined) {
return config.maxStdOutputSize;
}
return DEFAULT_MAX_OUTPUT_SIZE;
}
function getOutputDir() {
const config = readCliConfig();
if (config && config.outputDir) {
return config.outputDir;
}

Copilot uses AI. Check for mistakes.
return path.join(os.tmpdir(), 'playwright-cli-output');
Copy link

Copilot AI Apr 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For fileOnLargeOutput, getOutputDir() falls back to a fixed path under the OS temp dir (<tmp>/playwright-cli-output). This can accumulate files across runs and differs from whatever default outputDir behavior program() may already implement. Consider using a per-run temp directory (or delegating outputDir defaulting to the underlying CLI config logic) to avoid unbounded growth and keep behavior consistent across output modes.

Suggested change
return path.join(os.tmpdir(), 'playwright-cli-output');
return fs.mkdtempSync(path.join(os.tmpdir(), 'playwright-cli-output-'));

Copilot uses AI. Check for mistakes.
}

async function main() {
const outputMode = getOutputMode();

if (outputMode !== 'fileOnLargeOutput') {
await program({ embedderVersion: packageJson.version });
return;
}
Comment on lines +63 to +69
Copy link

Copilot AI Apr 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are existing Playwright integration tests for the CLI, but none cover the new outputMode: "fileOnLargeOutput" / maxStdOutputSize behavior (e.g., creating .playwright/cli.config.json, exercising a command that prints output, and asserting stdout vs file output). Adding an integration spec for this new mode would prevent regressions around thresholding and outputDir handling.

Copilot uses AI. Check for mistakes.

const maxSize = getMaxStdOutputSize();
const outputDir = getOutputDir();
Comment on lines +26 to +72
Copy link

Copilot AI Apr 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

getOutputMode() (and related helpers) always reads .playwright/cli.config.json from process.cwd(), which means outputMode: "fileOnLargeOutput" will not work when the user supplies a different config via --config (or any other config resolution the underlying program() supports). Consider reusing the same config resolution logic as program() or parsing process.argv for the selected config path before deciding whether to wrap output.

Suggested change
function getOutputMode() {
try {
const configPath = path.join(process.cwd(), '.playwright', 'cli.config.json');
if (fs.existsSync(configPath)) {
const config = JSON.parse(fs.readFileSync(configPath, 'utf8'));
return config.outputMode || null;
}
} catch {}
return null;
}
function getMaxStdOutputSize() {
try {
const configPath = path.join(process.cwd(), '.playwright', 'cli.config.json');
if (fs.existsSync(configPath)) {
const config = JSON.parse(fs.readFileSync(configPath, 'utf8'));
if (config.maxStdOutputSize !== undefined) {
return config.maxStdOutputSize;
}
}
} catch {}
return DEFAULT_MAX_OUTPUT_SIZE;
}
function getOutputDir() {
try {
const configPath = path.join(process.cwd(), '.playwright', 'cli.config.json');
if (fs.existsSync(configPath)) {
const config = JSON.parse(fs.readFileSync(configPath, 'utf8'));
if (config.outputDir) {
return config.outputDir;
}
}
} catch {}
return path.join(os.tmpdir(), 'playwright-cli-output');
}
async function main() {
const outputMode = getOutputMode();
if (outputMode !== 'fileOnLargeOutput') {
await program({ embedderVersion: packageJson.version });
return;
}
const maxSize = getMaxStdOutputSize();
const outputDir = getOutputDir();
function getCliConfigPath() {
for (let i = 2; i < process.argv.length; ++i) {
const arg = process.argv[i];
if (arg === '--config' && i + 1 < process.argv.length) {
return path.resolve(process.cwd(), process.argv[i + 1]);
}
if (arg.startsWith('--config=')) {
return path.resolve(process.cwd(), arg.substring('--config='.length));
}
}
return path.join(process.cwd(), '.playwright', 'cli.config.json');
}
function readCliConfig() {
try {
const configPath = getCliConfigPath();
if (fs.existsSync(configPath))
return JSON.parse(fs.readFileSync(configPath, 'utf8'));
} catch {}
return null;
}
function getOutputMode(config) {
return config?.outputMode || null;
}
function getMaxStdOutputSize(config) {
if (config?.maxStdOutputSize !== undefined)
return config.maxStdOutputSize;
return DEFAULT_MAX_OUTPUT_SIZE;
}
function getOutputDir(config) {
if (config?.outputDir)
return config.outputDir;
return path.join(os.tmpdir(), 'playwright-cli-output');
}
async function main() {
const cliConfig = readCliConfig();
const outputMode = getOutputMode(cliConfig);
if (outputMode !== 'fileOnLargeOutput') {
await program({ embedderVersion: packageJson.version });
return;
}
const maxSize = getMaxStdOutputSize(cliConfig);
const outputDir = getOutputDir(cliConfig);

Copilot uses AI. Check for mistakes.

if (!fs.existsSync(outputDir)) {
fs.mkdirSync(outputDir, { recursive: true });
}

const originalLog = console.log;
let outputBuffer = '';

console.log = (...args) => {
outputBuffer += args.map(a => String(a)).join(' ') + '\n';
};
Comment on lines +78 to +83
Copy link

Copilot AI Apr 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only console.log is wrapped here, so output written via process.stdout.write, console.error, or other logging APIs will bypass the size accounting and may still flood the terminal even in fileOnLargeOutput mode. If the intent is to gate stdout/stderr output size, consider wrapping process.stdout.write/process.stderr.write (and/or console.error) instead of only console.log.

Copilot uses AI. Check for mistakes.

try {
await program({ embedderVersion: packageJson.version });
} finally {
console.log = originalLog;
}

const outputSize = Buffer.byteLength(outputBuffer, 'utf8');

if (outputSize > maxSize) {
const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
const fileName = `output-${timestamp}.txt`;
const filePath = path.join(outputDir, fileName);

fs.writeFileSync(filePath, outputBuffer, 'utf8');
originalLog(`\nOutput exceeded ${maxSize} bytes (${outputSize} bytes). Saved to: ${filePath}`);
} else {
originalLog(outputBuffer);
Comment on lines +79 to +101
Copy link

Copilot AI Apr 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This mode buffers all console output into outputBuffer and only prints/writes it after program() completes. That changes CLI behavior substantially (no streaming output) and can cause large runs to consume a lot of memory (potentially OOM) before the threshold check runs. To avoid this, consider a streaming approach (e.g., tee to a temp file while counting bytes, or redirect to file once the threshold is exceeded).

Suggested change
let outputBuffer = '';
console.log = (...args) => {
outputBuffer += args.map(a => String(a)).join(' ') + '\n';
};
try {
await program({ embedderVersion: packageJson.version });
} finally {
console.log = originalLog;
}
const outputSize = Buffer.byteLength(outputBuffer, 'utf8');
if (outputSize > maxSize) {
const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
const fileName = `output-${timestamp}.txt`;
const filePath = path.join(outputDir, fileName);
fs.writeFileSync(filePath, outputBuffer, 'utf8');
originalLog(`\nOutput exceeded ${maxSize} bytes (${outputSize} bytes). Saved to: ${filePath}`);
} else {
originalLog(outputBuffer);
const tempFilePath = path.join(outputDir, `output-${process.pid}-${Date.now()}.tmp`);
const tempFd = fs.openSync(tempFilePath, 'w');
let outputSize = 0;
console.log = (...args) => {
const line = args.map(a => String(a)).join(' ') + '\n';
outputSize += Buffer.byteLength(line, 'utf8');
fs.writeSync(tempFd, line, null, 'utf8');
};
try {
await program({ embedderVersion: packageJson.version });
} finally {
console.log = originalLog;
fs.closeSync(tempFd);
}
if (outputSize > maxSize) {
const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
const fileName = `output-${timestamp}.txt`;
const filePath = path.join(outputDir, fileName);
fs.renameSync(tempFilePath, filePath);
originalLog(`\nOutput exceeded ${maxSize} bytes (${outputSize} bytes). Saved to: ${filePath}`);
} else {
const output = fs.readFileSync(tempFilePath, 'utf8');
fs.unlinkSync(tempFilePath);
originalLog(output);

Copilot uses AI. Check for mistakes.
Comment on lines +100 to +101
Copy link

Copilot AI Apr 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

originalLog(outputBuffer) will add its own newline, and outputBuffer already appends \n for every captured log line, so the final stdout output will usually end with an extra blank line and may not match the original formatting. To preserve output byte-for-byte, consider writing the buffered string directly to process.stdout (or trimming the final newline before calling console.log).

Suggested change
} else {
originalLog(outputBuffer);
} else if (outputBuffer) {
process.stdout.write(outputBuffer);

Copilot uses AI. Check for mistakes.
}
}

main().catch(e => {
console.error(e.message);
Copy link

Copilot AI Apr 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

main().catch(e => console.error(e.message)) can hide useful debugging details (no stack trace) and will print undefined if a non-Error value is thrown. Consider logging the full error (or e?.stack ?? e) to keep diagnostics comparable to the previous direct program() execution.

Suggested change
console.error(e.message);
console.error(e?.stack ?? e);

Copilot uses AI. Check for mistakes.
process.exit(1);
});
Loading