Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 22:23:50 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[PR #5273] [CLOSED] feat: Display images from MCP tool call results in chat #5385

New Issue
Closed
opened 2026-06-05 15:21:11 -04:00 by yindo · 0 comments
yindo commented 2026-06-05 15:21:11 -04:00
Owner
Copy Link
Copy Source

📋 Pull Request Information

Original PR: https://github.com/Mintplex-Labs/anything-llm/pull/5273
Author: @angelplusultra
Created: 3/26/2026
Status: ❌ Closed

Base: master ← Head: feat-render-images-from-mcp-tool-results

📝 Commits (4)

  • 0f1af75 implement image rendering from mcp tool result
  • 58946bb add support for resource links and generic resource
  • 3f40b75 simplify
  • 2252c37 improve readability of extractImageContent

📊 Changes

4 files changed (+202 additions, -0 deletions)

View changed files

➕ frontend/src/components/WorkspaceChat/ChatContainer/ChatHistory/MCPImageContent/index.jsx (+82 -0)
📝 frontend/src/components/WorkspaceChat/ChatContainer/ChatHistory/index.jsx (+3 -0)
📝 frontend/src/utils/chat/agent.js (+21 -0)
📝 server/utils/MCP/index.js (+96 -0)

📄 Description

Pull Request Type

  • ✨ feat (New feature)
  • 🐛 fix (Bug fix)
  • ♻️ refactor (Code refactoring without changing behavior)
  • 💄 style (UI style changes)
  • 🔨 chore (Build, CI, maintenance)
  • 📝 docs (Documentation updates)

Relevant Issues

resolves #4990

Description

MCP tools (Grafana, Tableau, etc.) can return images in their tool call results following the MCP spec. Previously, these images were either lost, displayed as raw base64 text, or shown as broken image icons because returnMCPResult() would JSON-stringify the entire result including image data.

This PR adds support for rendering images from MCP tool results inline in the chat. It handles all three MCP content types that can carry images:

  • type: "image" — inline base64-encoded image with data + mimeType
  • type: "resource_link" — URI pointing to a remotely hosted image
  • type: "resource" — embedded resource with a blob (base64) or uri

How it works:

  1. When an MCP tool returns a result, extractImageContent() checks the content array for image items
  2. Images are sent directly to the frontend via websocket (bypassing the LLM to avoid wasting tokens on large base64 strings)
  3. The LLM receives a lightweight placeholder: [1 image(s) returned by this tool and displayed to the user]
  4. Images are persisted to the database using the same _replySpecialAttributes pattern as rechartVisualize, so they survive page reloads
  5. A new MCPImageContent component renders the images inline with a download button

Files changed:

File Change
server/utils/MCP/index.js Extract images from MCP results, send via websocket, set persistence attrs
frontend/src/utils/chat/agent.js Handle mcpImageContent websocket event
frontend/src/components/.../ChatHistory/index.jsx Route mcpImageContent type to new component
frontend/src/components/.../ChatHistory/MCPImageContent/index.jsx New component — renders images inline with download

Visuals (if applicable)

image image image

Testing

To test this feature, create a test MCP server file (e.g. server/mcp-test.js) with three tools covering all image content types:

mcp-test.js (click to expand) 
const { McpServer } = require("@modelcontextprotocol/sdk/server/mcp.js");
const {
  StdioServerTransport,
} = require("@modelcontextprotocol/sdk/server/stdio.js");
const zlib = require("zlib");

/**
 * Generate a simple PNG image with colored bars (no external dependencies).
 * @param {number[][]} colors - Array of [r, g, b] color values for each stripe
 */
function generateTestPng(colors) {
  const width = 200;
  const height = 100;
  const stripeHeight = Math.floor(height / colors.length);

  const rawData = Buffer.alloc(height * (1 + width * 3));
  for (let y = 0; y < height; y++) {
    const rowOffset = y * (1 + width * 3);
    rawData[rowOffset] = 0;
    const color =
      colors[Math.min(Math.floor(y / stripeHeight), colors.length - 1)];
    for (let x = 0; x < width; x++) {
      const px = rowOffset + 1 + x * 3;
      rawData[px] = color[0];
      rawData[px + 1] = color[1];
      rawData[px + 2] = color[2];
    }
  }

  const compressed = zlib.deflateSync(rawData);
  const signature = Buffer.from([137, 80, 78, 71, 13, 10, 26, 10]);

  const crcTable = new Uint32Array(256);
  for (let n = 0; n < 256; n++) {
    let c = n;
    for (let k = 0; k < 8; k++) c = c & 1 ? 0xedb88320 ^ (c >>> 1) : c >>> 1;
    crcTable[n] = c;
  }
  function crc32(buf) {
    let crc = 0xffffffff;
    for (let i = 0; i < buf.length; i++)
      crc = crcTable[(crc ^ buf[i]) & 0xff] ^ (crc >>> 8);
    return (crc ^ 0xffffffff) | 0;
  }

  function makeChunk(type, data) {
    const len = Buffer.alloc(4);
    len.writeUInt32BE(data.length);
    const typeAndData = Buffer.concat([Buffer.from(type), data]);
    const crc = Buffer.alloc(4);
    crc.writeInt32BE(crc32(typeAndData));
    return Buffer.concat([len, typeAndData, crc]);
  }

  const ihdr = Buffer.alloc(13);
  ihdr.writeUInt32BE(width, 0);
  ihdr.writeUInt32BE(height, 4);
  ihdr[8] = 8;
  ihdr[9] = 2;

  return Buffer.concat([
    signature,
    makeChunk("IHDR", ihdr),
    makeChunk("IDAT", compressed),
    makeChunk("IEND", Buffer.alloc(0)),
  ]).toString("base64");
}

const server = new McpServer({ name: "test-image", version: "1.0.0" });

// Tool 1: Inline base64 image (type: "image")
server.registerTool(
  "test_inline_image",
  {
    description:
      "Returns a base64-encoded image inline (red/green/blue/yellow stripes)",
  },
  async () => ({
    content: [
      { type: "text", text: "Inline base64 image via type: image" },
      {
        type: "image",
        data: generateTestPng([
          [220, 50, 50],
          [50, 180, 50],
          [50, 80, 220],
          [230, 200, 40],
        ]),
        mimeType: "image/png",
      },
    ],
  })
);

// Tool 2: Resource link to a remote image (type: "resource_link")
server.registerTool(
  "test_resource_link_image",
  {
    description: "Returns an image as a resource link (remote URL)",
  },
  async () => ({
    content: [
      { type: "text", text: "Remote image via type: resource_link" },
      {
        type: "resource_link",
        uri: "https://placehold.co/400x200/orange/white?text=MCP+Resource+Link",
        name: "test-placeholder.png",
        mimeType: "image/png",
      },
    ],
  })
);

// Tool 3: Embedded resource with blob (type: "resource")
server.registerTool(
  "test_embedded_resource_image",
  {
    description: "Returns an image as an embedded resource with blob data",
  },
  async () => ({
    content: [
      { type: "text", text: "Embedded resource image via type: resource" },
      {
        type: "resource",
        resource: {
          uri: "resource://test/embedded-image.png",
          mimeType: "image/png",
          blob: generateTestPng([
            [140, 50, 200],
            [50, 190, 180],
          ]),
        },
      },
    ],
  })
);

const transport = new StdioServerTransport();
server.connect(transport);

Then add it to server/storage/plugins/anythingllm_mcp_servers.json:

{
  "mcpServers": {
    "test-image": {
      "command": "node",
      "args": ["/absolute/path/to/mcp-test.js"]
    }
  }
}

Open an agent-enabled workspace and test each tool:

Prompt Expected
"Use the test inline image tool" Red/green/blue/yellow striped image renders inline
"Use the test resource link image tool" Orange placeholder image from placehold.co renders inline
"Use the test embedded resource image tool" Purple/teal striped image renders inline

Also verify:

  • Images persist after page reload
  • Download button saves the image correctly
  • No base64 data appears in server logs being sent to the LLM

Additional Information

This follows the same architectural pattern as rechartVisualize — non-text content is sent directly to the frontend via websocket and persisted using _replySpecialAttributes, keeping large payloads out of the LLM context window.

Developer Validations

  • I ran yarn lint from the root of the repo & committed changes
  • Relevant documentation has been updated (if applicable)
  • I have tested my code functionality
  • Docker build succeeds locally

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/Mintplex-Labs/anything-llm/pull/5273 **Author:** [@angelplusultra](https://github.com/angelplusultra) **Created:** 3/26/2026 **Status:** ❌ Closed **Base:** `master` ← **Head:** `feat-render-images-from-mcp-tool-results` --- ### 📝 Commits (4) - [`0f1af75`](https://github.com/Mintplex-Labs/anything-llm/commit/0f1af75fe89bf5c90da9019f4b4273ad2ab8d63a) implement image rendering from mcp tool result - [`58946bb`](https://github.com/Mintplex-Labs/anything-llm/commit/58946bbe9d233b1244bc5fbf50b0a33a0fcc2c9d) add support for resource links and generic resource - [`3f40b75`](https://github.com/Mintplex-Labs/anything-llm/commit/3f40b75860559b8cab42420f6e96e91e30a46bf1) simplify - [`2252c37`](https://github.com/Mintplex-Labs/anything-llm/commit/2252c377d1ce76ef0074cef1b696759f285d7ed5) improve readability of extractImageContent ### 📊 Changes **4 files changed** (+202 additions, -0 deletions) <details> <summary>View changed files</summary> ➕ `frontend/src/components/WorkspaceChat/ChatContainer/ChatHistory/MCPImageContent/index.jsx` (+82 -0) 📝 `frontend/src/components/WorkspaceChat/ChatContainer/ChatHistory/index.jsx` (+3 -0) 📝 `frontend/src/utils/chat/agent.js` (+21 -0) 📝 `server/utils/MCP/index.js` (+96 -0) </details> ### 📄 Description ### Pull Request Type - [x] ✨ feat (New feature) - [ ] 🐛 fix (Bug fix) - [ ] ♻️ refactor (Code refactoring without changing behavior) - [ ] 💄 style (UI style changes) - [ ] 🔨 chore (Build, CI, maintenance) - [ ] 📝 docs (Documentation updates) ### Relevant Issues resolves #4990 ### Description MCP tools (Grafana, Tableau, etc.) can return images in their tool call results following the [MCP spec](https://modelcontextprotocol.io/specification/2025-11-25/server/tools#image-content). Previously, these images were either lost, displayed as raw base64 text, or shown as broken image icons because `returnMCPResult()` would JSON-stringify the entire result including image data. This PR adds support for rendering images from MCP tool results inline in the chat. It handles all three MCP content types that can carry images: - **`type: "image"`** — inline base64-encoded image with `data` + `mimeType` - **`type: "resource_link"`** — URI pointing to a remotely hosted image - **`type: "resource"`** — embedded resource with a `blob` (base64) or `uri` **How it works:** 1. When an MCP tool returns a result, `extractImageContent()` checks the `content` array for image items 2. Images are sent directly to the frontend via websocket (bypassing the LLM to avoid wasting tokens on large base64 strings) 3. The LLM receives a lightweight placeholder: `[1 image(s) returned by this tool and displayed to the user]` 4. Images are persisted to the database using the same `_replySpecialAttributes` pattern as `rechartVisualize`, so they survive page reloads 5. A new `MCPImageContent` component renders the images inline with a download button **Files changed:** | File | Change | |------|--------| | `server/utils/MCP/index.js` | Extract images from MCP results, send via websocket, set persistence attrs | | `frontend/src/utils/chat/agent.js` | Handle `mcpImageContent` websocket event | | `frontend/src/components/.../ChatHistory/index.jsx` | Route `mcpImageContent` type to new component | | `frontend/src/components/.../ChatHistory/MCPImageContent/index.jsx` | New component — renders images inline with download | ### Visuals (if applicable) <img width="1090" height="501" alt="image" src="https://github.com/user-attachments/assets/6ba8dfe7-e16d-4f26-b1b7-6d7b0ff522cb" /> <img width="1090" height="501" alt="image" src="https://github.com/user-attachments/assets/fd0f1540-cabf-4a0a-8f56-9b66372191f4" /> <img width="1090" height="501" alt="image" src="https://github.com/user-attachments/assets/cc18a17d-f787-4f2a-a5c7-22a76f714651" /> ### Testing To test this feature, create a test MCP server file (e.g. `server/mcp-test.js`) with three tools covering all image content types: <details> <summary><strong>mcp-test.js</strong> (click to expand)</summary> ```js const { McpServer } = require("@modelcontextprotocol/sdk/server/mcp.js"); const { StdioServerTransport, } = require("@modelcontextprotocol/sdk/server/stdio.js"); const zlib = require("zlib"); /** * Generate a simple PNG image with colored bars (no external dependencies). * @param {number[][]} colors - Array of [r, g, b] color values for each stripe */ function generateTestPng(colors) { const width = 200; const height = 100; const stripeHeight = Math.floor(height / colors.length); const rawData = Buffer.alloc(height * (1 + width * 3)); for (let y = 0; y < height; y++) { const rowOffset = y * (1 + width * 3); rawData[rowOffset] = 0; const color = colors[Math.min(Math.floor(y / stripeHeight), colors.length - 1)]; for (let x = 0; x < width; x++) { const px = rowOffset + 1 + x * 3; rawData[px] = color[0]; rawData[px + 1] = color[1]; rawData[px + 2] = color[2]; } } const compressed = zlib.deflateSync(rawData); const signature = Buffer.from([137, 80, 78, 71, 13, 10, 26, 10]); const crcTable = new Uint32Array(256); for (let n = 0; n < 256; n++) { let c = n; for (let k = 0; k < 8; k++) c = c & 1 ? 0xedb88320 ^ (c >>> 1) : c >>> 1; crcTable[n] = c; } function crc32(buf) { let crc = 0xffffffff; for (let i = 0; i < buf.length; i++) crc = crcTable[(crc ^ buf[i]) & 0xff] ^ (crc >>> 8); return (crc ^ 0xffffffff) | 0; } function makeChunk(type, data) { const len = Buffer.alloc(4); len.writeUInt32BE(data.length); const typeAndData = Buffer.concat([Buffer.from(type), data]); const crc = Buffer.alloc(4); crc.writeInt32BE(crc32(typeAndData)); return Buffer.concat([len, typeAndData, crc]); } const ihdr = Buffer.alloc(13); ihdr.writeUInt32BE(width, 0); ihdr.writeUInt32BE(height, 4); ihdr[8] = 8; ihdr[9] = 2; return Buffer.concat([ signature, makeChunk("IHDR", ihdr), makeChunk("IDAT", compressed), makeChunk("IEND", Buffer.alloc(0)), ]).toString("base64"); } const server = new McpServer({ name: "test-image", version: "1.0.0" }); // Tool 1: Inline base64 image (type: "image") server.registerTool( "test_inline_image", { description: "Returns a base64-encoded image inline (red/green/blue/yellow stripes)", }, async () => ({ content: [ { type: "text", text: "Inline base64 image via type: image" }, { type: "image", data: generateTestPng([ [220, 50, 50], [50, 180, 50], [50, 80, 220], [230, 200, 40], ]), mimeType: "image/png", }, ], }) ); // Tool 2: Resource link to a remote image (type: "resource_link") server.registerTool( "test_resource_link_image", { description: "Returns an image as a resource link (remote URL)", }, async () => ({ content: [ { type: "text", text: "Remote image via type: resource_link" }, { type: "resource_link", uri: "https://placehold.co/400x200/orange/white?text=MCP+Resource+Link", name: "test-placeholder.png", mimeType: "image/png", }, ], }) ); // Tool 3: Embedded resource with blob (type: "resource") server.registerTool( "test_embedded_resource_image", { description: "Returns an image as an embedded resource with blob data", }, async () => ({ content: [ { type: "text", text: "Embedded resource image via type: resource" }, { type: "resource", resource: { uri: "resource://test/embedded-image.png", mimeType: "image/png", blob: generateTestPng([ [140, 50, 200], [50, 190, 180], ]), }, }, ], }) ); const transport = new StdioServerTransport(); server.connect(transport); ``` </details> Then add it to `server/storage/plugins/anythingllm_mcp_servers.json`: ```json { "mcpServers": { "test-image": { "command": "node", "args": ["/absolute/path/to/mcp-test.js"] } } } ``` Open an agent-enabled workspace and test each tool: | Prompt | Expected | |--------|----------| | "Use the test inline image tool" | Red/green/blue/yellow striped image renders inline | | "Use the test resource link image tool" | Orange placeholder image from placehold.co renders inline | | "Use the test embedded resource image tool" | Purple/teal striped image renders inline | Also verify: - Images persist after page reload - Download button saves the image correctly - No base64 data appears in server logs being sent to the LLM ### Additional Information This follows the same architectural pattern as `rechartVisualize` — non-text content is sent directly to the frontend via websocket and persisted using `_replySpecialAttributes`, keeping large payloads out of the LLM context window. ### Developer Validations - [x] I ran `yarn lint` from the root of the repo & committed changes - [ ] Relevant documentation has been updated (if applicable) - [x] I have tested my code functionality - [ ] Docker build succeeds locally --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
yindo added the pull-request label 2026-06-05 15:21:11 -04:00
yindo closed this issue 2026-06-05 15:21:11 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#5385
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?