fix: sanitize malformed Unicode in MCP responses

[furkankoykiran]

Summary

Fixes "invalid high surrogate in string" JSON serialization errors in MCP responses when page content contains malformed Unicode (lone surrogates).

Changes

1. Add sanitizeUnicode() function (response.ts)

   • Uses String.prototype.toWellFormed() on Node 20+ for native Unicode sanitization
   • Falls back to manual surrogate replacement for Node 18 compatibility
   • Replaces lone surrogates with U+FFFD (replacement character)

2. Integrate into response serialization

   • Applied alongside existing redactText() function in serialize() method
   • Sanitizes all outgoing MCP response text before JSON serialization

3. Add comprehensive tests (unicode-serialization.spec.ts)

   • Lone high/low surrogates
   • Valid surrogate pairs (emoji, CJK)
   • Mixed content with malformed Unicode
   • Console messages with lone surrogates

Context

Closes microsoft/playwright-mcp#1447

Previous attempt (PR #1448 in playwright-mcp) was closed because it fixed the issue at the transport layer in cli.js. The proper fix location is in response.ts where text processing already happens via redactText().

Verification

npm run ctest-mcp unicode-serialization
# 6 passed (35.3s)

All tests pass, confirming that:

• MCP responses don't fail with JSON serialization errors
• Lone surrogates are replaced with U+FFFD
• Valid surrogate pairs (emoji, CJK) are preserved
• Normal text remains unchanged

[pavelfeldman]

Also looks like agent-generated blob that fails lint. Better place to handle it though. Assume Node 20 and hand-craft the tests please.

[furkankoykiran]

Thanks for the feedback. I've simplified the changes based on your suggestions:

Changes made:

1. Removed Node 18 fallback - The sanitizeUnicode() function now only uses String.prototype.toWellFormed() as requested.

2. Rewrote tests by hand - Reduced from 6 tests to 3 focused tests that match the style of existing MCP tests:

   • Uses server.setContent() for simpler setup
   • Cleaner, more idiomatic test structure
   • Removed excessive assertions

3. All tests pass - Confirmed with npm run ctest-mcp unicode-serialization

   • handles lone surrogates in page content
   • preserves valid emoji and surrogate pairs
   • handles console messages with lone surrogates

The branch has also been rebased onto latest main and the PR is now up to date.