Single Agent: Computer Use abstraction - content types and handler

[eavanvalkenburg]

This issue is to capture demand and question for scenario's around computer use, we currently do not plan on supporting this because the complexity of the second bullet below is vast, but that might change.

Roughly the work to enable this would be:

• defining and adding content types for computer use input and output
• integrating with or creating interfaces between those content types and computers, which should include at least, windows, linux and mac, and a set of defined versions of each.

[TaoChenOSU]

Closed PR attempted to support computer use: #2210

[mikeholczer]

Maybe to reduce initial scope, computer use abstractions could be limited to specific use cases. One I'd be interested in leveraging computer use for tool execution, as discussed in https://www.anthropic.com/engineering/code-execution-with-mcp. The benefits discussed in that post in terms of reducing context usage by tool definitions and intermediate results feels like it would make a big difference.

Edit: Or am I conflating computer use and code execution?