langchain-ai/deepagentsjs
1
0
Fork 0
mirror of https://github.com/langchain-ai/deepagentsjs.git synced 2026-07-21 03:45:22 -04:00
Code Issues 42 Packages Projects Releases 10 Wiki Activity

[PR #238] [MERGED] feat(ci): add initial eval suites #288

New Issue
Closed
opened 2026-06-05 17:22:31 -04:00 by yindo · 0 comments
yindo commented 2026-06-05 17:22:31 -04:00
Owner
Copy Link
Copy Source

📋 Pull Request Information

Original PR: https://github.com/langchain-ai/deepagentsjs/pull/238
Author: @hntrl
Created: 2/18/2026
Status: ✅ Merged
Merged: 2/18/2026
Merged by: @hntrl

Base: main ← Head: hunter/evals

📝 Commits (4)

📊 Changes

6 files changed (+907 additions, -1 deletions)

View changed files

📝 eslint.config.ts (+7 -0)
📝 libs/deepagents/package.json (+3 -0)
➕ libs/deepagents/src/evals/basic.eval.test.ts (+349 -0)
➕ libs/deepagents/src/evals/index.ts (+523 -0)
📝 libs/deepagents/vitest.config.ts (+19 -1)
📝 pnpm-lock.yaml (+6 -0)

📄 Description

Summary

Add a LangSmith-backed eval framework for deepagents using the langsmith/vitest integration. This provides a reusable harness (runAgent, custom matchers, AgentTrajectory types) and an initial suite of evals covering file operations, subagent delegation, and system prompt customization.

Port of https://github.com/langchain-ai/deepagents/pull/1400/changes

Changes

New: Eval harness (libs/deepagents/src/evals/index.ts)

  • runAgent() helper that invokes a deepagent with a user query + optional pre-seeded files and returns a structured AgentTrajectory
  • Custom vitest matchers: toHaveAgentSteps, toHaveToolCallRequests, toHaveToolCallInStep, toHaveFinalTextContaining
  • All matchers log feedback to LangSmith automatically for experiment tracking
  • Shared default agent instance using anthropic:claude-sonnet-4-5-20250929

New: Eval suites

  • File operations (11 tests) — read, write, parallel read/write, edit/replace, ls directory, grep, glob, derived output, and avoiding unnecessary tool calls
  • Subagents (2 tests) — delegating to a named custom subagent and to the general-purpose subagent
  • System prompt (1 test) — verifying custom system prompt behavior

Config changes

  • vitest.config.ts — new eval mode with 120s timeout, langsmith/vitest/reporter, and *.eval.test.ts include pattern; default mode now excludes *.eval.test.ts
  • package.json — new test:eval script; added @langchain/anthropic and langsmith dev dependencies
  • eslint.config.ts — disabled @typescript-eslint/no-empty-object-type globally and for **/evals/** files

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/langchain-ai/deepagentsjs/pull/238 **Author:** [@hntrl](https://github.com/hntrl) **Created:** 2/18/2026 **Status:** ✅ Merged **Merged:** 2/18/2026 **Merged by:** [@hntrl](https://github.com/hntrl) **Base:** `main` ← **Head:** `hunter/evals` --- ### 📝 Commits (4) - [`b024f29`](https://github.com/langchain-ai/deepagentsjs/commit/b024f290c8db1e3cb7bd266c2f5459a7c5c1c339) feat(ci): add initial eval suites - [`9cc9992`](https://github.com/langchain-ai/deepagentsjs/commit/9cc9992a0b69615c10cd9b6dd00883ebc7143e73) cr - [`b5d7e3a`](https://github.com/langchain-ai/deepagentsjs/commit/b5d7e3af485449a756c536763064240130966737) cr - [`6f2c5c9`](https://github.com/langchain-ai/deepagentsjs/commit/6f2c5c9a4b79f914055afd34daf18566503cfe61) cr ### 📊 Changes **6 files changed** (+907 additions, -1 deletions) <details> <summary>View changed files</summary> 📝 `eslint.config.ts` (+7 -0) 📝 `libs/deepagents/package.json` (+3 -0) ➕ `libs/deepagents/src/evals/basic.eval.test.ts` (+349 -0) ➕ `libs/deepagents/src/evals/index.ts` (+523 -0) 📝 `libs/deepagents/vitest.config.ts` (+19 -1) 📝 `pnpm-lock.yaml` (+6 -0) </details> ### 📄 Description ## Summary Add a LangSmith-backed eval framework for deepagents using the `langsmith/vitest` integration. This provides a reusable harness (`runAgent`, custom matchers, `AgentTrajectory` types) and an initial suite of evals covering file operations, subagent delegation, and system prompt customization. Port of https://github.com/langchain-ai/deepagents/pull/1400/changes ## Changes ### New: Eval harness (`libs/deepagents/src/evals/index.ts`) - `runAgent()` helper that invokes a deepagent with a user query + optional pre-seeded files and returns a structured `AgentTrajectory` - Custom vitest matchers: `toHaveAgentSteps`, `toHaveToolCallRequests`, `toHaveToolCallInStep`, `toHaveFinalTextContaining` - All matchers log feedback to LangSmith automatically for experiment tracking - Shared default agent instance using `anthropic:claude-sonnet-4-5-20250929` ### New: Eval suites - **File operations** (11 tests) — read, write, parallel read/write, edit/replace, ls directory, grep, glob, derived output, and avoiding unnecessary tool calls - **Subagents** (2 tests) — delegating to a named custom subagent and to the general-purpose subagent - **System prompt** (1 test) — verifying custom system prompt behavior ### Config changes - `vitest.config.ts` — new `eval` mode with 120s timeout, `langsmith/vitest/reporter`, and `*.eval.test.ts` include pattern; default mode now excludes `*.eval.test.ts` - `package.json` — new `test:eval` script; added `@langchain/anthropic` and `langsmith` dev dependencies - `eslint.config.ts` — disabled `@typescript-eslint/no-empty-object-type` globally and for `**/evals/**` files --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
yindo added the pull-request label 2026-06-05 17:22:31 -04:00
yindo closed this issue 2026-06-05 17:22:31 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
enhancement
help wanted
needs-response
pull-request
Mirrored from GitHub Pull Request
 stale
triage: high-impact
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langchain-ai/deepagentsjs#288
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?