Support image input/output in tools

[khushalsagar]

The API proposal doesn't include support for image input to the tool and for the tool output to include images. This seems like a basic use-case we should support. Worth looking at how MCP is doing it but we'll need something web specific, allowing devs to use image elements for this as an example.

[MiguelsPizza]

Providing some reference material for the tool result types in the MCP spec:

Included in the ToolResult object is an array of Content Blocks

export type ContentBlock =
  | TextContent
  | ImageContent
  | AudioContent
  | ResourceLink
  | EmbeddedResource;

export interface CallToolResult extends Result {
  /**
   * A list of content objects that represent the unstructured result of the tool call.
   */
  content: ContentBlock[];
...

Images can be returned as an ImageContent or an EmbeddedResource. Both of which request the image be represented as a Base64-encoded string.

/**
 * An image provided to or from an LLM.
 */
export interface ImageContent {
  type: "image";

  /**
   * The base64-encoded image data.
   *
   * @format byte
   */
  data: string;

  /**
   * The MIME type of the image. Different providers may support different image types.
   */
  mimeType: string;

  /**
   * Optional annotations for the client.
   */
  annotations?: Annotations;

  /**
   * See [General fields: `_meta`](/specification/2025-06-18/basic/index#meta) for notes on `_meta` usage.
   */
  _meta?: { [key: string]: unknown };
}

I believe most inference providers accept base64. It would nice if the browser did the conversion so the server code could be as simple as this

window.navigator.modelContext.registerTool({
    name: "get_product_image",
    description: "Get the product image from the page",
    inputSchema: {
      type: "object",
      properties: {
        productId: { type: "string" }
      }
    },
    handler: async ({ productId }) => {
      const img = document.querySelector(`#product-${productId} img`);
      
      return { content: [{ type: "image", element: img }] };
    }
});

[khushalsagar]

Thanks for the pointer @MiguelsPizza! +1 to this: "It would nice if the browser did the conversion so the server code could be as simple as this". There are a few other data types to reference pixel content in Web APIs, canvas2d is a good reference for that. Documentation here.

Other than data and mimeType which will come from the image source above, MCP data structure has annotations and _meta. Those are shared types that we should discuss separately. Doesn't need to block the addition of a return type for image resources, we can always define a base structure and add to it going forward.

Related, there should likely be a way to reference the image content in a json response. If you're returning a list of products, you likely have a json with text info and placeholders for the image of each product. Does MCP already support that concept? Some overlap with #9 too.

[tomayac]

We should likely do what the Prompt API does here:

• For image input, the Web platform has the concept of ImageBitmapSource and BufferSource.
• For audio input, there's AudioBuffer, again BufferSource, or plain Blob.