langchain-ai/deepagentsjs
1
0
Fork 0
mirror of https://github.com/langchain-ai/deepagentsjs.git synced 2026-07-19 08:26:01 -04:00
Code Issues 42 Packages Projects Releases 10 Wiki Activity

[PR #298] [MERGED] feat(deepagents): support multimodal files for backends #337

New Issue
Closed
opened 2026-06-05 17:22:43 -04:00 by yindo · 0 comments
yindo commented 2026-06-05 17:22:43 -04:00
Owner
Copy Link
Copy Source

📋 Pull Request Information

Original PR: https://github.com/langchain-ai/deepagentsjs/pull/298
Author: @colifran
Created: 3/11/2026
Status: ✅ Merged
Merged: 3/17/2026
Merged by: @colifran

Base: main ← Head: colifran/multimodal-refactor

📝 Commits (10+)

📊 Changes

61 files changed (+3845 additions, -1096 deletions)

View changed files

➕ .changeset/dry-insects-occur.md (+10 -0)
📝 libs/acp/src/acp-filesystem-backend.test.ts (+10 -6)
📝 libs/acp/src/acp-filesystem-backend.ts (+7 -3)
📝 libs/deepagents/package.json (+1 -1)
📝 libs/deepagents/src/backends/composite.test.ts (+152 -83)
📝 libs/deepagents/src/backends/composite.ts (+90 -53)
📝 libs/deepagents/src/backends/filesystem.test.ts (+174 -44)
📝 libs/deepagents/src/backends/filesystem.ts (+103 -35)
📝 libs/deepagents/src/backends/index.ts (+16 -2)
📝 libs/deepagents/src/backends/local-shell.int.test.ts (+1 -1)
📝 libs/deepagents/src/backends/local-shell.test.ts (+46 -16)
📝 libs/deepagents/src/backends/local-shell.ts (+22 -17)
📝 libs/deepagents/src/backends/protocol.test.ts (+46 -0)
📝 libs/deepagents/src/backends/protocol.ts (+167 -137)
📝 libs/deepagents/src/backends/sandbox.test.ts (+139 -46)
📝 libs/deepagents/src/backends/sandbox.ts (+75 -29)
📝 libs/deepagents/src/backends/state.test.ts (+359 -42)
📝 libs/deepagents/src/backends/state.ts (+97 -32)
📝 libs/deepagents/src/backends/store.test.ts (+281 -46)
📝 libs/deepagents/src/backends/store.ts (+113 -35)

...and 41 more files

📄 Description

Description

Adds support for binary and multimodal files (images, PDFs, audio, video, etc.) across all backend implementations, with a versioned protocol layer that returns structured results instead of plain values or arrays.

Changes

Protocol

  • Introduced BackendProtocolV2 with structured Result return types:
    • ReadResult — for read() operations
    • ReadRawResult — for readRaw() operations
    • GrepResult — for grepRaw() operations
    • LsResult — for lsInfo() operations
    • GlobResult — for globInfo() operations
  • Introduced SandboxBackendProtocolV2 extending BackendProtocolV2; deprecated SandboxBackendProtocol
  • Deprecated v1 interfaces (BackendProtocol, SandboxBackendProtocol) moved to v1/; current interfaces in v2/ for clear separation
  • Added AnyBackendProtocol union type and adaptBackendProtocol() for backward compatibility — public APIs accept either version and adapt internally
  • FileData split into FileDataV1 (legacy line array) and FileDataV2 (single string + mimeType, supports base64)
  • All Result types follow consistent pattern: { error?: string, [data]?: T } for explicit error propagation

Binary File Support

  • All backends (State, Store, Filesystem, Sandbox, Composite) store and retrieve binary files as base64
  • read_file tool returns typed multimodal content blocks (image, audio, video, file) for binary files
  • Binary reads capped at 10MB to stay within provider inline limits
  • MIME type detection via file extension, stored with v2 FileData

Middleware

  • Updated skills.ts middleware to handle LsResult from backend operations
  • Updated fs.ts middleware to handle LsResult and GlobResult with proper error checking

Provider Updates

  • QuickJS REPL: public API accepts AnyBackendProtocol, adapts to v2 internally via adaptBackendProtocol() — fully backward compatible
  • Node VFS: VfsSandbox updated to return LsResult and GlobResult instead of bare arrays

Backward Compatibility

  • v1 backends continue to work everywhere via adaptBackendProtocol() runtime detection
  • v2 backends correctly read and operate on existing v1 FileData (string[] content)
  • No breaking changes to public APIs — all accept AnyBackendProtocol

Tests

  • All backend tests updated to assert on Result types: composite.test.ts, filesystem.test.ts, state.test.ts, store.test.ts, sandbox.test.ts, local-shell.test.ts
  • Added readRaw() tests returning ReadRawResult across State, LocalShell, Composite, and Node VFS backends
  • utils.test.ts updated to test adaptBackendProtocol() with all Result types, including v1→v2 wrapping
  • Middleware tests (fs.test.ts, skills.test.ts) updated to mock backends returning Result types
  • New binary tests for State and Store backends covering upload, download, round-trip, read, grep, and pagination
  • New multimodal content block tests for read_file (image, audio, video, PDF, size limit enforcement)
  • Sandbox test: readRaw() now returns error in Result object instead of throwing
  • Added v1→v2 backward compatibility tests for State and Store backends (mixed v1/v2 data)
  • Manual e2e testing scripts for all 5 backend configurations plus v1→v2 migration simulation

Example:

multimodal

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/langchain-ai/deepagentsjs/pull/298 **Author:** [@colifran](https://github.com/colifran) **Created:** 3/11/2026 **Status:** ✅ Merged **Merged:** 3/17/2026 **Merged by:** [@colifran](https://github.com/colifran) **Base:** `main` ← **Head:** `colifran/multimodal-refactor` --- ### 📝 Commits (10+) - [`24d5544`](https://github.com/langchain-ai/deepagentsjs/commit/24d5544bec0dacc2007736dd6fc67849c646e3ad) update backend protocol interface, types, and utils - [`32575f1`](https://github.com/langchain-ai/deepagentsjs/commit/32575f191d94f3d50c670afe8ab0419985c4f58a) refactor state backend - [`7b69afe`](https://github.com/langchain-ai/deepagentsjs/commit/7b69afe2729d1089fc5611dceb34bc29feacf129) refactor store backend - [`189681e`](https://github.com/langchain-ai/deepagentsjs/commit/189681e0f6d275c4465fc1b27b8d7f1e42f5d4fc) clean up - [`962b7e5`](https://github.com/langchain-ai/deepagentsjs/commit/962b7e5ed3f1269235b731942fe2479484c2cfd1) unit tests - [`04e3514`](https://github.com/langchain-ai/deepagentsjs/commit/04e35145ccb75cafcf351ce2bc33fa5f447a5c87) refactor filesystem - [`f8287a6`](https://github.com/langchain-ai/deepagentsjs/commit/f8287a6c01ba717adf7488465a1fee42a7097966) unit tests - [`0064855`](https://github.com/langchain-ai/deepagentsjs/commit/0064855d9f9c230bb5d2320385e5438f8399caf0) skip binary files in literal search for filesystem backend - [`3d2c312`](https://github.com/langchain-ai/deepagentsjs/commit/3d2c312100cd7732bf3b2ae3a2e48545cdd86d5e) refactor base sandbox - [`75cf96f`](https://github.com/langchain-ai/deepagentsjs/commit/75cf96f4d6badf4ae37ae283c86dbd59806da8a0) base sandbox unit tests ### 📊 Changes **61 files changed** (+3845 additions, -1096 deletions) <details> <summary>View changed files</summary> ➕ `.changeset/dry-insects-occur.md` (+10 -0) 📝 `libs/acp/src/acp-filesystem-backend.test.ts` (+10 -6) 📝 `libs/acp/src/acp-filesystem-backend.ts` (+7 -3) 📝 `libs/deepagents/package.json` (+1 -1) 📝 `libs/deepagents/src/backends/composite.test.ts` (+152 -83) 📝 `libs/deepagents/src/backends/composite.ts` (+90 -53) 📝 `libs/deepagents/src/backends/filesystem.test.ts` (+174 -44) 📝 `libs/deepagents/src/backends/filesystem.ts` (+103 -35) 📝 `libs/deepagents/src/backends/index.ts` (+16 -2) 📝 `libs/deepagents/src/backends/local-shell.int.test.ts` (+1 -1) 📝 `libs/deepagents/src/backends/local-shell.test.ts` (+46 -16) 📝 `libs/deepagents/src/backends/local-shell.ts` (+22 -17) 📝 `libs/deepagents/src/backends/protocol.test.ts` (+46 -0) 📝 `libs/deepagents/src/backends/protocol.ts` (+167 -137) 📝 `libs/deepagents/src/backends/sandbox.test.ts` (+139 -46) 📝 `libs/deepagents/src/backends/sandbox.ts` (+75 -29) 📝 `libs/deepagents/src/backends/state.test.ts` (+359 -42) 📝 `libs/deepagents/src/backends/state.ts` (+97 -32) 📝 `libs/deepagents/src/backends/store.test.ts` (+281 -46) 📝 `libs/deepagents/src/backends/store.ts` (+113 -35) _...and 41 more files_ </details> ### 📄 Description ### Description Adds support for binary and multimodal files (images, PDFs, audio, video, etc.) across all backend implementations, with a versioned protocol layer that returns structured results instead of plain values or arrays. ### Changes #### Protocol - Introduced BackendProtocolV2 with structured Result return types: - ReadResult — for read() operations - ReadRawResult — for readRaw() operations - GrepResult — for grepRaw() operations - LsResult — for lsInfo() operations - GlobResult — for globInfo() operations - Introduced SandboxBackendProtocolV2 extending BackendProtocolV2; deprecated SandboxBackendProtocol - Deprecated v1 interfaces (BackendProtocol, SandboxBackendProtocol) moved to v1/; current interfaces in v2/ for clear separation - Added AnyBackendProtocol union type and adaptBackendProtocol() for backward compatibility — public APIs accept either version and adapt internally - FileData split into FileDataV1 (legacy line array) and FileDataV2 (single string + mimeType, supports base64) - All Result types follow consistent pattern: { error?: string, [data]?: T } for explicit error propagation #### Binary File Support - All backends (State, Store, Filesystem, Sandbox, Composite) store and retrieve binary files as base64 - read_file tool returns typed multimodal content blocks (image, audio, video, file) for binary files - Binary reads capped at 10MB to stay within provider inline limits - MIME type detection via file extension, stored with v2 FileData #### Middleware - Updated skills.ts middleware to handle LsResult from backend operations - Updated fs.ts middleware to handle LsResult and GlobResult with proper error checking #### Provider Updates - QuickJS REPL: public API accepts AnyBackendProtocol, adapts to v2 internally via adaptBackendProtocol() — fully backward compatible - Node VFS: VfsSandbox updated to return LsResult and GlobResult instead of bare arrays #### Backward Compatibility - v1 backends continue to work everywhere via adaptBackendProtocol() runtime detection - v2 backends correctly read and operate on existing v1 FileData (string[] content) - No breaking changes to public APIs — all accept AnyBackendProtocol ### Tests - All backend tests updated to assert on Result types: composite.test.ts, filesystem.test.ts, state.test.ts, store.test.ts, sandbox.test.ts, local-shell.test.ts - Added readRaw() tests returning ReadRawResult across State, LocalShell, Composite, and Node VFS backends - utils.test.ts updated to test adaptBackendProtocol() with all Result types, including v1→v2 wrapping - Middleware tests (fs.test.ts, skills.test.ts) updated to mock backends returning Result types - New binary tests for State and Store backends covering upload, download, round-trip, read, grep, and pagination - New multimodal content block tests for read_file (image, audio, video, PDF, size limit enforcement) - Sandbox test: readRaw() now returns error in Result object instead of throwing - Added v1→v2 backward compatibility tests for State and Store backends (mixed v1/v2 data) - Manual e2e testing scripts for all 5 backend configurations plus v1→v2 migration simulation ### Example: ![multimodal](https://github.com/user-attachments/assets/990de67d-f71d-46d6-87cf-6a50ca0cd707) --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
yindo added the pull-request label 2026-06-05 17:22:43 -04:00
yindo closed this issue 2026-06-05 17:22:43 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
enhancement
help wanted
needs-response
pull-request
Mirrored from GitHub Pull Request
 stale
triage: high-impact
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langchain-ai/deepagentsjs#337
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?