feat: add multimodal UIMessage support

[jakobhoeg]

🎯 Changes

When calling append() with a ModelMessage containing multimodal content (images, audio, files), the content was stripped during the ModelMessage → UIMessage conversion because modelMessageToUIMessage() only extracted text via getTextContent(). Along this, the parts of a message doesn't include multimodal parts, making it impossible to build chat UIs that preserve and display multimodal content.

Added new message part types and updated the conversion functions to preserve multimodal content during round-trips:
New Types (@tanstack/ai and @tanstack/ai-client):

• ImageMessagePart - preserves image data with source and optional metadata
• AudioMessagePart - preserves audio data
• VideoMessagePart - preserves video data - (NOT TESTED)
• DocumentMessagePart - preserves document data (e.g., PDFs) - (NOT TESTED)

Updated Conversion Functions:

• modelMessageToUIMessage() - now converts ContentPart[] to corresponding MessagePart[] instead of discarding non-text parts
• uiMessageToModelMessages() - now builds ContentPart[] when multimodal parts are present, preserving part ordering

Example:

// Input ModelMessage with multimodal content
const message: ModelMessage = {
  role: 'user',
  content: [
    { type: 'text', text: 'What is in this image?' },
    { type: 'image', source: { type: 'url', value: '' } }
  ]
}

// UIMessage now preserves all content
const uiMessage = modelMessageToUIMessage(message)
// uiMessage.parts = [
//   { type: 'text', content: 'What is in this image?' },
//   { type: 'image', source: { type: 'url', value: '' } }
// ]

// UI
if (part.type === 'image') { // 'audio' etc.
  ...<Render UI />
}

Demo

Images:
https://github.com/user-attachments/assets/5f62ab32-9f11-44f7-bfc0-87d00678e265

Audio:
https://github.com/user-attachments/assets/bbbdc2f9-f8d7-4d74-99c2-23d15a3278a3

Closes #200

Note

I have not tested this with other adapters than my own community adapter that I'm currently working on.

This contribution touches core message handling. Let me know if the approach doesn't align with the project's vision, I am happy to iterate on it :)

This PR is not ready to be merged because:

• Video and document parts are implemented but not yet tested
• Only tested with my community adapter - needs verification with official adapters (OpenAI, Anthropic, etc.)

✅ Checklist

• I have followed the steps in the Contributing guide.
  • I followed CLAUDE.md, since the link is broken.
• I have tested this code locally with pnpm run test:pr.

🚀 Release Impact

• This change affects published code, and I have generated a changeset.
• This change is docs/CI/dev-only (no release).

Summary by CodeRabbit

Release Notes

• New Features

  • Added multimodal support for UI messages, enabling handling of images, audio, video, and documents alongside text content.

• Tests

  • Added comprehensive test coverage for multimodal message conversion logic and content preservation.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

[coderabbitai]

📝 Walkthrough

Walkthrough

This pull request extends UIMessage to support multimodal content types (images, audio, video, documents) by adding new ContentPart interfaces and updating message conversion logic to preserve multimodal content structure through ModelMessage and UIMessage transformations.

Changes

Cohort / File(s)	Summary
Changeset .changeset/brave-nights-shout.md	Patch release for multimodal UIMessage support feature flag
Type Definitions packages/typescript/ai/src/types.ts, packages/typescript/ai-client/src/types.ts	Added ImagePart, AudioPart, VideoPart, and DocumentPart interfaces with type discriminators, content sources, and metadata; extended MessagePart union to include new multimodal types
Message Conversion Logic packages/typescript/ai/src/activities/chat/messages.ts	Updated uiMessageToModelMessages to build ContentPart[] arrays for multimodal content and modelMessageToUIMessage to preserve multimodal structures instead of converting to text; enhanced to handle all four new part types and their sources
Test Coverage packages/typescript/ai/tests/messages.test.ts	Added 190 lines of comprehensive tests validating text-only and multimodal content preservation, metadata handling, part ordering, and round-trip conversions across all part types

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Suggested reviewers

• AlemTuzlak
• harry-whorlow

Poem

🐰 Hops with glee, through images bright,
Audio and video in perfect flight,
Documents bundled, metadata's care,
Multimodal messages float through the air!

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The PR title 'feat: add multimodal UIMessage support' is concise, clear, and directly summarizes the main change in the changeset.
Description check	✅ Passed	The PR description provides comprehensive context about the changes, includes examples, demo links, testing status, and completed the required checklist items.
Linked Issues check	✅ Passed	The PR addresses all core objectives from issue #200: preserves multimodal ContentPart entries during append(), extends UIMessage.parts to support multimodal types, maintains part ordering, and enables chat UIs to render multimodal content [#200].
Out of Scope Changes check	✅ Passed	All changes are directly aligned with issue #200 objectives. Changes to types.ts, messages.ts, and tests.ts focus on adding multimodal support; the changeset entry documents the feature addition.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

• 📝 Generate docstrings

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[jakobhoeg]

@coderabbitai review

[coderabbitai]

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[AlemTuzlak]

I'm a bit confused, don't these two types match identically? from what I see what you're doing is just coping the old data into the new one?

[jakobhoeg]

I might've gotten carried away here and overcomplicated things.
I initially thought ContentPart (used in ModelMessage.content) and MessagePart (used in UIMessage.parts) were separate type systems for model and ui that needed their own definitions.
Pushed changes to simplify and resolve this.

[ilbertt]

I would also like to send media messages from the client, I need this feature

[AlemTuzlak]

@jherr mind reviewing this one? It looks good to me but as you were in charge of this piece of code I'd feel much more comfortable if you approved it

[nx-cloud]

🤖 Nx Cloud AI Fix Eligible

An automatically generated fix could have helped fix failing tasks for this run, but Self-healing CI is disabled for this workspace. Visit workspace settings to enable it and get automatic fixes in future runs.

_{To disable these notifications, a workspace admin can disable them in workspace settings.}

---

View your CI Pipeline Execution ↗ for commit 273bdc0

Command	Status	Duration	Result
nx affected --targets=test:sherif,test:knip,tes...	❌ Failed	2m 34s	View ↗
nx run-many --targets=build --exclude=examples/**	❌ Failed	1m 4s	View ↗

---

☁️ Nx Cloud last updated this comment at 2026-01-26 10:38:51 UTC