langchain-ai/deepagentsjs
1
0
Fork 0
mirror of https://github.com/langchain-ai/deepagentsjs.git synced 2026-08-24 22:11:26 -04:00
Code Issues 42 Packages Projects Releases 10 Wiki Activity

[PR #455] [MERGED] test(evals): expand parity coverage with python eval suites #477

New Issue
Closed
opened 2026-06-05 17:23:19 -04:00 by yindo · 0 comments
yindo commented 2026-06-05 17:23:19 -04:00
Owner
Copy Link
Copy Source

📋 Pull Request Information

Original PR: https://github.com/langchain-ai/deepagentsjs/pull/455
Author: @hntrl
Created: 4/14/2026
Status: ✅ Merged
Merged: 4/14/2026
Merged by: @hntrl

Base: main ← Head: hunter/sync-eval-suite

📝 Commits (8)

  • 26bb43a test(evals): expand parity coverage with python eval suites
  • 190385f test(evals): use eval name in describe and runner projectName
  • 6bc6f75 test(evals): use eval runner semantics in memory bench suites
  • 89e21d4 test(evals): use eval runner semantics in summarization suite
  • 860a241 cr
  • 247d86a ci: pin pnpm to 10.29.2 in workflows
  • b9d817e ci: remove explicit pnpm version pins
  • 7264c7c fix(lockfile): resolve langsmith peer entry mismatch

📊 Changes

38 files changed (+1916 additions, -4 deletions)

View changed files

📝 .github/workflows/cli-release.yml (+0 -4)
📝 evals/README.md (+8 -0)
➕ evals/external-benchmarks/README.md (+3 -0)
➕ evals/external-benchmarks/data/curated_cases.json (+246 -0)
➕ evals/external-benchmarks/index.test.ts (+68 -0)
➕ evals/external-benchmarks/package.json (+14 -0)
➕ evals/external-benchmarks/vitest.config.ts (+14 -0)
📝 evals/files/index.test.ts (+49 -0)
➕ evals/followup-quality/README.md (+3 -0)
➕ evals/followup-quality/index.test.ts (+131 -0)
➕ evals/followup-quality/package.json (+14 -0)
➕ evals/followup-quality/vitest.config.ts (+14 -0)
➕ evals/memory-agent-bench/README.md (+3 -0)
➕ evals/memory-agent-bench/index.test.ts (+83 -0)
➕ evals/memory-agent-bench/package.json (+15 -0)
➕ evals/memory-agent-bench/vitest.config.ts (+14 -0)
➕ evals/memory-multiturn/README.md (+3 -0)
➕ evals/memory-multiturn/index.test.ts (+111 -0)
➕ evals/memory-multiturn/package.json (+15 -0)
➕ evals/memory-multiturn/vitest.config.ts (+14 -0)

...and 18 more files

📄 Description

Summary

The change set adds new JS eval packages for summarization, external benchmark hard-sets, memory-agent-bench style coverage, tau2-airline style policy tasks, tool selection, follow-up quality, todos, and multi-turn memory. It also extends existing files and memory suites with parity cases previously only present in Python.

Changes

Eval suite parity expansion

Added new eval workspaces under evals/:

  • external-benchmarks/
    • Adds a curated 15-case hard set spanning FRAMES, Nexus, and BFCL-v3-style samples.
    • Includes vendored case data at data/curated_cases.json.
  • summarization/
    • Adds summarization middleware behavior coverage for long-conversation continuation and history offloading.
  • memory-agent-bench/
    • Adds MemoryAgentBench-style scenarios for long-context memorization, conflict updates, and file-seeded retrieval.
  • tau2-airline/
    • Adds a 15-task policy-grounded airline support eval with explicit tool-behavior assertions.
  • tool-selection/
    • Adds direct/indirect tool routing and chained tool invocation coverage.
  • followup-quality/
    • Adds underspecified-request follow-up quality coverage.
  • todos/
    • Adds sequential write_todos behavior coverage.
  • memory-multiturn/
    • Adds multi-turn memory persistence and transient-info filtering coverage.

Each new suite includes package.json, vitest.config.ts, README.md, and index.test.ts and is wired to LangSmith reporting.

Existing suite parity fixes

Updated existing suites:

  • evals/files/index.test.ts
    • Adds pagination tail-recovery and empty-file handling cases.
  • evals/memory/index.test.ts
    • Adds transient memory exclusion, formatting preference persistence, graceful missing-memory behavior, and memory path routing checks.

Suite metadata and docs alignment

  • Updated evals/README.md to include all newly added suites.
  • Updated eval test structure to use ls.describe(runner.name, ...) with stable per-suite projectName values, matching established local pattern.
  • Added/updated evals/EVAL_SUITE_DIVERGENCE_ASSESSMENT.md to reflect parity progress.

Dependency/workspace updates

  • Updated pnpm-lock.yaml to include added eval workspace dependencies.

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/langchain-ai/deepagentsjs/pull/455 **Author:** [@hntrl](https://github.com/hntrl) **Created:** 4/14/2026 **Status:** ✅ Merged **Merged:** 4/14/2026 **Merged by:** [@hntrl](https://github.com/hntrl) **Base:** `main` ← **Head:** `hunter/sync-eval-suite` --- ### 📝 Commits (8) - [`26bb43a`](https://github.com/langchain-ai/deepagentsjs/commit/26bb43a46d4717a0e97f6a62bc4897a34c6d91d2) test(evals): expand parity coverage with python eval suites - [`190385f`](https://github.com/langchain-ai/deepagentsjs/commit/190385fddc0a72b17f0bd961465e8708f0b3ce1b) test(evals): use eval name in describe and runner projectName - [`6bc6f75`](https://github.com/langchain-ai/deepagentsjs/commit/6bc6f75d5ae8186cfb8d4df914d54027da5ba50c) test(evals): use eval runner semantics in memory bench suites - [`89e21d4`](https://github.com/langchain-ai/deepagentsjs/commit/89e21d44294b810a3ef5c38f49eff5ad5c576401) test(evals): use eval runner semantics in summarization suite - [`860a241`](https://github.com/langchain-ai/deepagentsjs/commit/860a2410799191022beab3ecb8b095bc2492a42a) cr - [`247d86a`](https://github.com/langchain-ai/deepagentsjs/commit/247d86ae1909d8c080f415189c3afcee7c05f653) ci: pin pnpm to 10.29.2 in workflows - [`b9d817e`](https://github.com/langchain-ai/deepagentsjs/commit/b9d817e3c039e1cac681727bca9e22f19d51bf3d) ci: remove explicit pnpm version pins - [`7264c7c`](https://github.com/langchain-ai/deepagentsjs/commit/7264c7cb38a01fec924b741b9b25e5b7ba7aab28) fix(lockfile): resolve langsmith peer entry mismatch ### 📊 Changes **38 files changed** (+1916 additions, -4 deletions) <details> <summary>View changed files</summary> 📝 `.github/workflows/cli-release.yml` (+0 -4) 📝 `evals/README.md` (+8 -0) ➕ `evals/external-benchmarks/README.md` (+3 -0) ➕ `evals/external-benchmarks/data/curated_cases.json` (+246 -0) ➕ `evals/external-benchmarks/index.test.ts` (+68 -0) ➕ `evals/external-benchmarks/package.json` (+14 -0) ➕ `evals/external-benchmarks/vitest.config.ts` (+14 -0) 📝 `evals/files/index.test.ts` (+49 -0) ➕ `evals/followup-quality/README.md` (+3 -0) ➕ `evals/followup-quality/index.test.ts` (+131 -0) ➕ `evals/followup-quality/package.json` (+14 -0) ➕ `evals/followup-quality/vitest.config.ts` (+14 -0) ➕ `evals/memory-agent-bench/README.md` (+3 -0) ➕ `evals/memory-agent-bench/index.test.ts` (+83 -0) ➕ `evals/memory-agent-bench/package.json` (+15 -0) ➕ `evals/memory-agent-bench/vitest.config.ts` (+14 -0) ➕ `evals/memory-multiturn/README.md` (+3 -0) ➕ `evals/memory-multiturn/index.test.ts` (+111 -0) ➕ `evals/memory-multiturn/package.json` (+15 -0) ➕ `evals/memory-multiturn/vitest.config.ts` (+14 -0) _...and 18 more files_ </details> ### 📄 Description ## Summary The change set adds new JS eval packages for summarization, external benchmark hard-sets, memory-agent-bench style coverage, tau2-airline style policy tasks, tool selection, follow-up quality, todos, and multi-turn memory. It also extends existing `files` and `memory` suites with parity cases previously only present in Python. ## Changes ### Eval suite parity expansion Added new eval workspaces under `evals/`: - `external-benchmarks/` - Adds a curated 15-case hard set spanning FRAMES, Nexus, and BFCL-v3-style samples. - Includes vendored case data at `data/curated_cases.json`. - `summarization/` - Adds summarization middleware behavior coverage for long-conversation continuation and history offloading. - `memory-agent-bench/` - Adds MemoryAgentBench-style scenarios for long-context memorization, conflict updates, and file-seeded retrieval. - `tau2-airline/` - Adds a 15-task policy-grounded airline support eval with explicit tool-behavior assertions. - `tool-selection/` - Adds direct/indirect tool routing and chained tool invocation coverage. - `followup-quality/` - Adds underspecified-request follow-up quality coverage. - `todos/` - Adds sequential `write_todos` behavior coverage. - `memory-multiturn/` - Adds multi-turn memory persistence and transient-info filtering coverage. Each new suite includes `package.json`, `vitest.config.ts`, `README.md`, and `index.test.ts` and is wired to LangSmith reporting. ### Existing suite parity fixes Updated existing suites: - `evals/files/index.test.ts` - Adds pagination tail-recovery and empty-file handling cases. - `evals/memory/index.test.ts` - Adds transient memory exclusion, formatting preference persistence, graceful missing-memory behavior, and memory path routing checks. ### Suite metadata and docs alignment - Updated `evals/README.md` to include all newly added suites. - Updated eval test structure to use `ls.describe(runner.name, ...)` with stable per-suite `projectName` values, matching established local pattern. - Added/updated `evals/EVAL_SUITE_DIVERGENCE_ASSESSMENT.md` to reflect parity progress. ### Dependency/workspace updates - Updated `pnpm-lock.yaml` to include added eval workspace dependencies. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
yindo added the pull-request label 2026-06-05 17:23:19 -04:00
yindo closed this issue 2026-06-05 17:23:20 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
enhancement
help wanted
needs-response
pull-request
Mirrored from GitHub Pull Request
 stale
triage: high-impact
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langchain-ai/deepagentsjs#477
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?