-
Notifications
You must be signed in to change notification settings - Fork 2.8k
feat: add MCP image preview thumbnails and save_image tool #10879
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
roomote
wants to merge
4
commits into
fix/mcp-tool-image-support-10872
Choose a base branch
from
feature/mcp-image-preview-and-save-10877
base: fix/mcp-tool-image-support-10872
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+632
−41
Draft
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
1632659
feat: add MCP image preview thumbnails and save_image tool
roomote 7f1cab9
fix: add i18n translation key and fix security validation path in Sav…
roomote 953f037
fix: MCP image preview thumbnails and save_image tool functionality
roomote f3eaa46
fix: implement intermediate file persistence for MCP images
roomote File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,53 @@ | ||
| import type OpenAI from "openai" | ||
|
|
||
| const SAVE_IMAGE_DESCRIPTION = `Request to save an image to a file. This tool supports two methods: | ||
|
|
||
| 1. **Using source_path (PREFERRED for MCP tools)**: When you receive images from MCP tools like Figma, the images are automatically saved to temporary storage and you receive file paths. Use the source_path parameter to copy the image to your desired location. This is efficient and avoids data corruption. | ||
|
|
||
| 2. **Using data (for base64 data URLs)**: For images provided as base64 data URLs from other sources. | ||
|
|
||
| Parameters: | ||
| - path: (required) The destination file path where the image should be saved (relative to the current workspace directory). The tool will automatically add the appropriate image extension based on the source image format if not provided. | ||
| - source_path: (optional) The absolute path to a source image file (typically from MCP tool temporary storage). Use this for images received from MCP tools - the path is provided in the tool response. PREFERRED over data. | ||
| - data: (optional) Base64-encoded image data URL (e.g., 'data:image/png;base64,...'). Supported formats: PNG, JPG, JPEG, GIF, WEBP, SVG. Only use if source_path is not available. | ||
|
|
||
| NOTE: Either source_path OR data must be provided. | ||
|
|
||
| Example: Saving an image from MCP tool (PREFERRED) | ||
| { "path": "images/figma-screenshot.png", "source_path": "/path/to/temp/figma_get_screenshot_123.png" } | ||
|
|
||
| Example: Saving a base64 image (fallback) | ||
| { "path": "images/screenshot.png", "data": "data:image/png;base64,iVBORw0KGgoAAAANSUhEU..." }` | ||
|
|
||
| const PATH_PARAMETER_DESCRIPTION = `Destination filesystem path (relative to the workspace) where the image should be saved` | ||
|
|
||
| const SOURCE_PATH_PARAMETER_DESCRIPTION = `Absolute path to a source image file (from MCP tool temporary storage). PREFERRED method for saving images from MCP tools.` | ||
|
|
||
| const DATA_PARAMETER_DESCRIPTION = `Base64-encoded image data URL (e.g., 'data:image/png;base64,...'). Only use if source_path is not available.` | ||
|
|
||
| export default { | ||
| type: "function", | ||
| function: { | ||
| name: "save_image", | ||
| description: SAVE_IMAGE_DESCRIPTION, | ||
| strict: false, // Changed to non-strict to allow optional parameters | ||
| parameters: { | ||
| type: "object", | ||
| properties: { | ||
| path: { | ||
| type: "string", | ||
| description: PATH_PARAMETER_DESCRIPTION, | ||
| }, | ||
| source_path: { | ||
| type: "string", | ||
| description: SOURCE_PATH_PARAMETER_DESCRIPTION, | ||
| }, | ||
| data: { | ||
| type: "string", | ||
| description: DATA_PARAMETER_DESCRIPTION, | ||
| }, | ||
| }, | ||
| required: ["path"], | ||
| }, | ||
| }, | ||
| } satisfies OpenAI.Chat.ChatCompletionTool |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,278 @@ | ||
| import path from "path" | ||
| import fs from "fs/promises" | ||
| import * as vscode from "vscode" | ||
| import { Task } from "../task/Task" | ||
| import { formatResponse } from "../prompts/responses" | ||
| import { getReadablePath } from "../../utils/path" | ||
| import { isPathOutsideWorkspace } from "../../utils/pathUtils" | ||
| import { fileExistsAtPath } from "../../utils/fs" | ||
| import { BaseTool, ToolCallbacks } from "./BaseTool" | ||
| import type { ToolUse } from "../../shared/tools" | ||
| import { t } from "../../i18n" | ||
|
|
||
| interface SaveImageParams { | ||
| path: string | ||
| data?: string | ||
| source_path?: string | ||
| } | ||
|
|
||
| export class SaveImageTool extends BaseTool<"save_image"> { | ||
| readonly name = "save_image" as const | ||
|
|
||
| async execute(params: SaveImageParams, task: Task, callbacks: ToolCallbacks): Promise<void> { | ||
| const { path: relPath, data, source_path: sourcePath } = params | ||
| const { handleError, pushToolResult, askApproval } = callbacks | ||
|
|
||
| // Validate required parameters | ||
| if (!relPath) { | ||
| task.consecutiveMistakeCount++ | ||
| task.recordToolError("save_image") | ||
| pushToolResult(await task.sayAndCreateMissingParamError("save_image", "path")) | ||
| return | ||
| } | ||
|
|
||
| // Need either source_path or data | ||
| if (!sourcePath && !data) { | ||
| task.consecutiveMistakeCount++ | ||
| task.recordToolError("save_image") | ||
| await task.say( | ||
| "error", | ||
| t("tools:saveImage.missingSourceOrData", { | ||
| defaultValue: | ||
| "Either 'source_path' or 'data' parameter is required. Use 'source_path' for images from MCP tools, or 'data' for base64 data URLs.", | ||
| }), | ||
| ) | ||
| task.didToolFailInCurrentTurn = true | ||
| pushToolResult( | ||
| formatResponse.toolError( | ||
| "Either 'source_path' or 'data' parameter is required. Use 'source_path' for images from MCP tools, or 'data' for base64 data URLs.", | ||
| ), | ||
| ) | ||
| return | ||
| } | ||
|
|
||
| // If source_path is provided, use it to copy the file | ||
| if (sourcePath) { | ||
| await this.copyFromSourcePath(task, sourcePath, relPath, callbacks) | ||
| return | ||
| } | ||
|
|
||
| // Otherwise, use the data parameter (base64 data URL) | ||
| // Validate the image data format first (to determine finalPath) | ||
| const base64Match = data!.match(/^data:image\/(png|jpeg|jpg|gif|webp|svg\+xml);base64,(.+)$/) | ||
| if (!base64Match) { | ||
| await task.say("error", t("tools:saveImage.invalidDataFormat")) | ||
| task.didToolFailInCurrentTurn = true | ||
| pushToolResult( | ||
| formatResponse.toolError( | ||
| "Invalid image data format. Expected a base64 data URL (e.g., 'data:image/png;base64,...').", | ||
| ), | ||
| ) | ||
| return | ||
| } | ||
|
|
||
| const imageFormat = base64Match[1] | ||
| const base64Data = base64Match[2] | ||
|
|
||
| // Ensure the path has a valid image extension | ||
| let finalPath = relPath | ||
| if (!finalPath.match(/\.(png|jpg|jpeg|gif|webp|svg)$/i)) { | ||
| // Add extension based on the data format | ||
| const ext = imageFormat === "jpeg" ? "jpg" : imageFormat === "svg+xml" ? "svg" : imageFormat | ||
| finalPath = `${finalPath}.${ext}` | ||
| } | ||
|
|
||
| // Validate access via .rooignore (using finalPath after extension is added) | ||
| const accessAllowed = task.rooIgnoreController?.validateAccess(finalPath) | ||
| if (!accessAllowed) { | ||
| await task.say("rooignore_error", finalPath) | ||
| pushToolResult(formatResponse.rooIgnoreError(finalPath)) | ||
| return | ||
| } | ||
|
|
||
| // Check write protection (using finalPath after extension is added) | ||
| const isWriteProtected = task.rooProtectedController?.isWriteProtected(finalPath) || false | ||
|
|
||
| const fullPath = path.resolve(task.cwd, finalPath) | ||
| const isOutsideWorkspace = isPathOutsideWorkspace(fullPath) | ||
|
|
||
| const sharedMessageProps = { | ||
| tool: "saveImage" as const, | ||
| path: getReadablePath(task.cwd, finalPath), | ||
| isOutsideWorkspace, | ||
| isProtected: isWriteProtected, | ||
| } | ||
|
|
||
| try { | ||
| task.consecutiveMistakeCount = 0 | ||
|
|
||
| const approvalMessage = JSON.stringify({ | ||
| ...sharedMessageProps, | ||
| content: `Save image to ${getReadablePath(task.cwd, finalPath)}`, | ||
| }) | ||
|
|
||
| const didApprove = await askApproval("tool", approvalMessage, undefined, isWriteProtected) | ||
|
|
||
| if (!didApprove) { | ||
| return | ||
| } | ||
|
|
||
| // Convert base64 to buffer and save | ||
| const imageBuffer = Buffer.from(base64Data, "base64") | ||
|
|
||
| const absolutePath = path.resolve(task.cwd, finalPath) | ||
| const directory = path.dirname(absolutePath) | ||
| await fs.mkdir(directory, { recursive: true }) | ||
|
|
||
| await fs.writeFile(absolutePath, imageBuffer) | ||
|
|
||
| // Track the file context | ||
| if (finalPath) { | ||
| await task.fileContextTracker.trackFileContext(finalPath, "roo_edited") | ||
| } | ||
|
|
||
| task.didEditFile = true | ||
|
|
||
| task.recordToolUsage("save_image") | ||
|
|
||
| const provider = task.providerRef.deref() | ||
| const fullImagePath = path.join(task.cwd, finalPath) | ||
|
|
||
| let imageUri = provider?.convertToWebviewUri?.(fullImagePath) ?? vscode.Uri.file(fullImagePath).toString() | ||
|
|
||
| // Add cache buster to force refresh | ||
| const cacheBuster = Date.now() | ||
| imageUri = imageUri.includes("?") ? `${imageUri}&t=${cacheBuster}` : `${imageUri}?t=${cacheBuster}` | ||
|
|
||
| await task.say("image", JSON.stringify({ imageUri, imagePath: fullImagePath })) | ||
| pushToolResult(formatResponse.toolResult(`Image saved to ${getReadablePath(task.cwd, finalPath)}`)) | ||
| } catch (error) { | ||
| await handleError("saving image", error as Error) | ||
| } | ||
| } | ||
|
|
||
| /** | ||
| * Copy an image from a source path (typically from MCP temp storage) to the destination path. | ||
| * This is the preferred method for saving images from MCP tools as it avoids passing | ||
| * raw base64 through LLM context. | ||
| */ | ||
| private async copyFromSourcePath( | ||
| task: Task, | ||
| sourcePath: string, | ||
| destRelPath: string, | ||
| callbacks: ToolCallbacks, | ||
| ): Promise<void> { | ||
| const { handleError, pushToolResult, askApproval } = callbacks | ||
|
|
||
| try { | ||
| // Check if source file exists | ||
| const sourceExists = await fileExistsAtPath(sourcePath) | ||
| if (!sourceExists) { | ||
| task.consecutiveMistakeCount++ | ||
| task.recordToolError("save_image") | ||
| await task.say( | ||
| "error", | ||
| t("tools:saveImage.sourceNotFound", { | ||
| defaultValue: `Source image not found at path: ${sourcePath}`, | ||
| path: sourcePath, | ||
| }), | ||
| ) | ||
| task.didToolFailInCurrentTurn = true | ||
| pushToolResult(formatResponse.toolError(`Source image not found at path: ${sourcePath}`)) | ||
| return | ||
| } | ||
|
|
||
| // Get extension from source file | ||
| const sourceExt = path.extname(sourcePath).toLowerCase() | ||
| const validExtensions = [".png", ".jpg", ".jpeg", ".gif", ".webp", ".svg"] | ||
|
|
||
| if (!validExtensions.includes(sourceExt)) { | ||
| task.consecutiveMistakeCount++ | ||
| task.recordToolError("save_image") | ||
| await task.say("error", t("tools:saveImage.invalidSourceFormat")) | ||
| task.didToolFailInCurrentTurn = true | ||
| pushToolResult( | ||
| formatResponse.toolError( | ||
| `Invalid source image format. Supported formats: ${validExtensions.join(", ")}`, | ||
| ), | ||
| ) | ||
| return | ||
| } | ||
|
|
||
| // Ensure the destination path has the correct extension | ||
| let finalPath = destRelPath | ||
| if (!finalPath.match(/\.(png|jpg|jpeg|gif|webp|svg)$/i)) { | ||
| finalPath = `${finalPath}${sourceExt}` | ||
| } | ||
|
|
||
| // Validate access via .rooignore | ||
| const accessAllowed = task.rooIgnoreController?.validateAccess(finalPath) | ||
| if (!accessAllowed) { | ||
| await task.say("rooignore_error", finalPath) | ||
| pushToolResult(formatResponse.rooIgnoreError(finalPath)) | ||
| return | ||
| } | ||
|
|
||
| // Check write protection | ||
| const isWriteProtected = task.rooProtectedController?.isWriteProtected(finalPath) || false | ||
|
|
||
| const fullPath = path.resolve(task.cwd, finalPath) | ||
| const isOutsideWorkspace = isPathOutsideWorkspace(fullPath) | ||
|
|
||
| const sharedMessageProps = { | ||
| tool: "saveImage" as const, | ||
| path: getReadablePath(task.cwd, finalPath), | ||
| isOutsideWorkspace, | ||
| isProtected: isWriteProtected, | ||
| } | ||
|
|
||
| task.consecutiveMistakeCount = 0 | ||
|
|
||
| const approvalMessage = JSON.stringify({ | ||
| ...sharedMessageProps, | ||
| content: `Save image from ${sourcePath} to ${getReadablePath(task.cwd, finalPath)}`, | ||
| }) | ||
|
|
||
| const didApprove = await askApproval("tool", approvalMessage, undefined, isWriteProtected) | ||
|
|
||
| if (!didApprove) { | ||
| return | ||
| } | ||
|
|
||
| // Create destination directory and copy file | ||
| const absolutePath = path.resolve(task.cwd, finalPath) | ||
| const directory = path.dirname(absolutePath) | ||
| await fs.mkdir(directory, { recursive: true }) | ||
|
|
||
| await fs.copyFile(sourcePath, absolutePath) | ||
|
|
||
| // Track the file context | ||
| if (finalPath) { | ||
| await task.fileContextTracker.trackFileContext(finalPath, "roo_edited") | ||
| } | ||
|
|
||
| task.didEditFile = true | ||
| task.recordToolUsage("save_image") | ||
|
|
||
| const provider = task.providerRef.deref() | ||
| const fullImagePath = path.join(task.cwd, finalPath) | ||
|
|
||
| let imageUri = provider?.convertToWebviewUri?.(fullImagePath) ?? vscode.Uri.file(fullImagePath).toString() | ||
|
|
||
| // Add cache buster to force refresh | ||
| const cacheBuster = Date.now() | ||
| imageUri = imageUri.includes("?") ? `${imageUri}&t=${cacheBuster}` : `${imageUri}?t=${cacheBuster}` | ||
|
|
||
| await task.say("image", JSON.stringify({ imageUri, imagePath: fullImagePath })) | ||
| pushToolResult(formatResponse.toolResult(`Image saved to ${getReadablePath(task.cwd, finalPath)}`)) | ||
| } catch (error) { | ||
| await handleError("saving image", error as Error) | ||
| } | ||
| } | ||
|
|
||
| override async handlePartial(task: Task, block: ToolUse<"save_image">): Promise<void> { | ||
| return | ||
| } | ||
| } | ||
|
|
||
| export const saveImageTool = new SaveImageTool() | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The translation key
tools:saveImage.invalidDataFormatis used here but doesn't exist insrc/i18n/locales/en/tools.json. The i18n function will return the raw key string. You should add asaveImagesection to the tools.json translation file with this key, similar to howgenerateImagetranslations are defined.Fix it with Roo Code or mention @roomote and request a fix.