langchain-ai/deepagentsjs
1
0
Fork 0
mirror of https://github.com/langchain-ai/deepagentsjs.git synced 2026-07-18 16:04:44 -04:00
Code Issues 42 Packages Projects Releases 10 Wiki Activity

[PR #215] feat(evals): add harbor infra for terminal-bench evals #218

New Issue
Open
opened 2026-02-16 06:17:28 -05:00 by yindo · 0 comments
yindo commented 2026-02-16 06:17:28 -05:00
Owner
Copy Link
Copy Source

📋 Pull Request Information

Original PR: https://github.com/langchain-ai/deepagentsjs/pull/215
Author: @maahir30
Created: 2/10/2026
Status: 🔄 Open

Base: main ← Head: harbor-evals

📝 Commits (5)

📊 Changes

23 files changed (+3659 additions, -2 deletions)

View changed files

📝 .gitignore (+8 -1)
📝 eslint.config.ts (+1 -1)
➕ libs/harbor/Makefile (+94 -0)
➕ libs/harbor/README.md (+256 -0)
➕ libs/harbor/package.json (+78 -0)
➕ libs/harbor/python/deepagents_js_harbor/__init__.py (+4 -0)
➕ libs/harbor/python/deepagents_js_harbor/langsmith_environment.py (+337 -0)
➕ libs/harbor/python/deepagents_js_harbor/upload_to_langsmith.py (+565 -0)
➕ libs/harbor/python/deepagents_js_harbor/wrapper.py (+487 -0)
➕ libs/harbor/python/langsmith-env-config.yaml (+7 -0)
➕ libs/harbor/python/pyproject.toml (+46 -0)
➕ libs/harbor/scripts/harbor_langsmith.py (+359 -0)
➕ libs/harbor/src/index.ts (+29 -0)
➕ libs/harbor/src/rpc-protocol.test.ts (+97 -0)
➕ libs/harbor/src/rpc-protocol.ts (+189 -0)
➕ libs/harbor/src/rpc-sandbox.test.ts (+300 -0)
➕ libs/harbor/src/rpc-sandbox.ts (+266 -0)
➕ libs/harbor/src/runner.int.test.ts (+187 -0)
➕ libs/harbor/src/runner.ts (+221 -0)
➕ libs/harbor/tsconfig.json (+8 -0)

...and 3 more files

📄 Description

Summary

  • Adds the ability to run deepagents-js against terminal-bench benchmarks via Harbor.

  • Harbor is a Python-only benchmark framework -- it can only load Python agent classes. Since our agent runs in Node.js, we need a thin Python wrapper that acts as a bridge between the two runtimes. This PR adds that bridge.

  • Also adds support for running benchmarks on LangSmith hosted sandboxes as a custom Harbor environment, alongside Docker and Daytona.

Architecture

Harbor requires agents to be Python classes extending BaseAgent. Our wrapper (DeepAgentsJSWrapper) satisfies that contract but delegates all actual agent work to Node.js:

  1. Harbor calls run(instruction, environment) on our Python wrapper
  2. Python spawns a Node.js subprocess running runner.ts, which creates a deepagents agent
  3. The two processes communicate via JSON-RPC over stdin/stdout:
    • When the JS agent needs to execute a shell command (e.g., ls -la), the Node process sends an exec_request to Python
    • Python calls Harbor's environment.exec() (which runs the command in the sandboxed container) and sends the result back as an exec_response
    • All file operations (read, write, edit, grep, glob) are handled inside Node by the existing BaseSandbox class, which builds shell commands and routes them through execute()
  4. When the agent finishes, Node sends a done message with the full message history, and Python saves the trajectory in Harbor's ATIF format
Harbor (Python)  →  wrapper.py (Python)  →  runner.ts (Node.js)  →  createDeepAgent (JS)
                         ↑                        ↓
                         ↑    exec_request/exec_response
                         ↑    (JSON-RPC over stdin/stdout)
                         ↑                        ↓
                    environment.exec()    ←   RpcSandbox.execute()
                    (runs in sandbox)

LangSmith Sandbox Support

Adds LangSmithEnvironment, a custom Harbor BaseEnvironment implementation backed by the langsmith.sandbox SDK. Harbor supports custom environments via --environment-import-path (or environment.import_path in YAML config), which dynamically imports any class that extends BaseEnvironment.

The environment automatically resolves the correct container image from the task's docker_image field in task.toml (the same pre-built images used by Daytona/E2B), so tasks run with the full environment they expect (gcc, weights, test harness, etc.).

Key files:

  • langsmith_environment.py -- implements all BaseEnvironment abstract methods (exec, start, stop, upload_file, download_file, etc.) using AsyncSandboxClient/AsyncSandbox
  • langsmith-env-config.yaml -- minimal YAML config for use with harbor run -c

How to run

# One-time setup
cd libs/harbor/python && uv sync
cd libs/harbor && pnpm build

# Run a single benchmark task locally (Docker)
cd libs/harbor && make bench-docker

# Run a specific task
make bench-docker TASK=gpt2-codegolf

# Run at scale on Daytona
make bench-daytona

# Run on LangSmith sandbox (requires LANGSMITH_API_KEY)
make bench-langsmith
make bench-langsmith TASK=gpt2-codegolf

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/langchain-ai/deepagentsjs/pull/215 **Author:** [@maahir30](https://github.com/maahir30) **Created:** 2/10/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `harbor-evals` --- ### 📝 Commits (5) - [`5262de7`](https://github.com/langchain-ai/deepagentsjs/commit/5262de712bd44c584b8590523f677e99eca56c94) rpc harbor infra - [`891c239`](https://github.com/langchain-ai/deepagentsjs/commit/891c239b06ac98812b8478e9ac8401a4d641d6c7) move python infra into to repo - [`f0bc121`](https://github.com/langchain-ai/deepagentsjs/commit/f0bc12136ce33750ea5b321eb983a57e1af93dfa) add LC sandboxes - [`e285807`](https://github.com/langchain-ai/deepagentsjs/commit/e285807092c1a87d27470f7ad73588aa27d58406) langsmith experiment integration - [`957c47e`](https://github.com/langchain-ai/deepagentsjs/commit/957c47e633d1b38919ef7f6693540ad5bbf63822) fix ### 📊 Changes **23 files changed** (+3659 additions, -2 deletions) <details> <summary>View changed files</summary> 📝 `.gitignore` (+8 -1) 📝 `eslint.config.ts` (+1 -1) ➕ `libs/harbor/Makefile` (+94 -0) ➕ `libs/harbor/README.md` (+256 -0) ➕ `libs/harbor/package.json` (+78 -0) ➕ `libs/harbor/python/deepagents_js_harbor/__init__.py` (+4 -0) ➕ `libs/harbor/python/deepagents_js_harbor/langsmith_environment.py` (+337 -0) ➕ `libs/harbor/python/deepagents_js_harbor/upload_to_langsmith.py` (+565 -0) ➕ `libs/harbor/python/deepagents_js_harbor/wrapper.py` (+487 -0) ➕ `libs/harbor/python/langsmith-env-config.yaml` (+7 -0) ➕ `libs/harbor/python/pyproject.toml` (+46 -0) ➕ `libs/harbor/scripts/harbor_langsmith.py` (+359 -0) ➕ `libs/harbor/src/index.ts` (+29 -0) ➕ `libs/harbor/src/rpc-protocol.test.ts` (+97 -0) ➕ `libs/harbor/src/rpc-protocol.ts` (+189 -0) ➕ `libs/harbor/src/rpc-sandbox.test.ts` (+300 -0) ➕ `libs/harbor/src/rpc-sandbox.ts` (+266 -0) ➕ `libs/harbor/src/runner.int.test.ts` (+187 -0) ➕ `libs/harbor/src/runner.ts` (+221 -0) ➕ `libs/harbor/tsconfig.json` (+8 -0) _...and 3 more files_ </details> ### 📄 Description ## Summary - Adds the ability to run deepagents-js against terminal-bench benchmarks via Harbor. - Harbor is a Python-only benchmark framework -- it can only load Python agent classes. Since our agent runs in Node.js, we need a thin Python wrapper that acts as a bridge between the two runtimes. This PR adds that bridge. - Also adds support for running benchmarks on **LangSmith hosted sandboxes** as a custom Harbor environment, alongside Docker and Daytona. ## Architecture Harbor requires agents to be Python classes extending `BaseAgent`. Our wrapper (`DeepAgentsJSWrapper`) satisfies that contract but delegates all actual agent work to Node.js: 1. **Harbor calls `run(instruction, environment)`** on our Python wrapper 2. **Python spawns a Node.js subprocess** running `runner.ts`, which creates a `deepagents` agent 3. **The two processes communicate via JSON-RPC over stdin/stdout:** - When the JS agent needs to execute a shell command (e.g., `ls -la`), the Node process sends an `exec_request` to Python - Python calls Harbor's `environment.exec()` (which runs the command in the sandboxed container) and sends the result back as an `exec_response` - All file operations (read, write, edit, grep, glob) are handled inside Node by the existing `BaseSandbox` class, which builds shell commands and routes them through `execute()` 4. **When the agent finishes**, Node sends a `done` message with the full message history, and Python saves the trajectory in Harbor's ATIF format ``` Harbor (Python) → wrapper.py (Python) → runner.ts (Node.js) → createDeepAgent (JS) ↑ ↓ ↑ exec_request/exec_response ↑ (JSON-RPC over stdin/stdout) ↑ ↓ environment.exec() ← RpcSandbox.execute() (runs in sandbox) ``` ## LangSmith Sandbox Support Adds `LangSmithEnvironment`, a custom Harbor `BaseEnvironment` implementation backed by the `langsmith.sandbox` SDK. Harbor supports custom environments via `--environment-import-path` (or `environment.import_path` in YAML config), which dynamically imports any class that extends `BaseEnvironment`. The environment automatically resolves the correct container image from the task's `docker_image` field in `task.toml` (the same pre-built images used by Daytona/E2B), so tasks run with the full environment they expect (gcc, weights, test harness, etc.). Key files: - `langsmith_environment.py` -- implements all `BaseEnvironment` abstract methods (`exec`, `start`, `stop`, `upload_file`, `download_file`, etc.) using `AsyncSandboxClient`/`AsyncSandbox` - `langsmith-env-config.yaml` -- minimal YAML config for use with `harbor run -c` ## How to run ```bash # One-time setup cd libs/harbor/python && uv sync cd libs/harbor && pnpm build # Run a single benchmark task locally (Docker) cd libs/harbor && make bench-docker # Run a specific task make bench-docker TASK=gpt2-codegolf # Run at scale on Daytona make bench-daytona # Run on LangSmith sandbox (requires LANGSMITH_API_KEY) make bench-langsmith make bench-langsmith TASK=gpt2-codegolf ``` --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
yindo added the pull-request label 2026-02-16 06:17:28 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
enhancement
help wanted
needs-response
pull-request
Mirrored from GitHub Pull Request
 stale
triage: high-impact
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langchain-ai/deepagentsjs#218
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?