langchain-ai/deepagentsjs
1
0
Fork 0
mirror of https://github.com/langchain-ai/deepagentsjs.git synced 2026-07-18 16:04:44 -04:00
Code Issues 42 Packages Projects Releases 10 Wiki Activity

[PR #457] [MERGED] feat(evals): add aggregate all-evals entrypoint and suite-builder structure #478

New Issue
Closed
opened 2026-06-05 17:23:20 -04:00 by yindo · 0 comments
yindo commented 2026-06-05 17:23:20 -04:00
Owner
Copy Link
Copy Source

📋 Pull Request Information

Original PR: https://github.com/langchain-ai/deepagentsjs/pull/457
Author: @hntrl
Created: 4/14/2026
Status: ✅ Merged
Merged: 4/14/2026
Merged by: @hntrl

Base: main ← Head: hunter/all-evals-experiment

📝 Commits (7)

  • ab03794 feat(evals): add aggregate all-suite eval package
  • 678a443 feat(evals): allow global LangSmith project and dataset override
  • 7fed4c7 refactor(evals): extract suite builders and add single all-suite entrypoint
  • ba6572f refactor(evals): rename suite files and move describe metadata into eval tests
  • c41cc43 refactor(evals): rename suite registration functions to xSuite
  • 76d1264 cr
  • efe7d31 cr

📊 Changes

79 files changed (+3800 additions, -3509 deletions)

View changed files

📝 evals/README.md (+8 -3)
➕ evals/all/README.md (+8 -0)
➕ evals/all/eval.test.ts (+61 -0)
➕ evals/all/package.json (+18 -0)
➕ evals/all/vitest.config.ts (+14 -0)
➕ evals/basic/eval.test.ts (+13 -0)
➖ evals/basic/index.test.ts (+0 -46)
➕ evals/basic/index.ts (+40 -0)
➕ evals/external-benchmarks/eval.test.ts (+13 -0)
➖ evals/external-benchmarks/index.test.ts (+0 -68)
➕ evals/external-benchmarks/index.ts (+64 -0)
➕ evals/files/eval.test.ts (+13 -0)
➖ evals/files/index.test.ts (+0 -454)
➕ evals/files/index.ts (+448 -0)
➕ evals/followup-quality/eval.test.ts (+13 -0)
📝 evals/followup-quality/index.ts (+29 -38)
➕ evals/hitl/eval.test.ts (+13 -0)
➖ evals/hitl/index.test.ts (+0 -287)
➕ evals/hitl/index.ts (+278 -0)
➕ evals/memory-agent-bench/eval.test.ts (+13 -0)

...and 59 more files

📄 Description

Summary

This PR introduces a first-class aggregate eval entrypoint (evals/all) that runs every existing eval suite through one top-level LangSmith/Vitest describe block.

To support that cleanly, eval suites were refactored into a reusable two-file shape: index.ts defines suite registration logic (define...Suite(runner)), and eval.test.ts owns LangSmith attribution (ls.describe(...)). This removes reliance on ad-hoc global override wiring and gives the all-evals package a single explicit integration surface.

Changes

Add aggregate all-evals package

  • Added @deepagents/eval-all package:
    • evals/all/eval.test.ts imports and invokes every suite registration function.
    • evals/all/vitest.config.ts runs only the aggregate eval.test.ts entrypoint.
    • Added package README and package metadata.

Refactor all eval suites to reusable registration modules

  • Standardized suite implementation files to index.ts.
  • Standardized eval wrappers to eval.test.ts.
  • For Oolong datasets, standardized to <dataset>.ts + <dataset>.eval.test.ts.
  • Moved ls.describe dataset/project attribution into wrapper test files so registration modules focus on test definitions only.
  • Ensured suite registration functions accept runner as an argument across suites.

Ensure all existing evals are included in aggregate execution

  • evals/all/eval.test.ts now imports and calls all suite registration functions for:
    • core eval suites under evals/*
    • oolong dataset suites under evals/oolong/datasets/*

Documentation updates

  • Updated evals/README.md examples and structure references for index.ts + eval.test.ts layout.
  • Updated evals/all/README.md to document aggregate-run architecture.

Lockfile updates

  • Updated pnpm-lock.yaml to include the new evals/all workspace package and dependencies.

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/langchain-ai/deepagentsjs/pull/457 **Author:** [@hntrl](https://github.com/hntrl) **Created:** 4/14/2026 **Status:** ✅ Merged **Merged:** 4/14/2026 **Merged by:** [@hntrl](https://github.com/hntrl) **Base:** `main` ← **Head:** `hunter/all-evals-experiment` --- ### 📝 Commits (7) - [`ab03794`](https://github.com/langchain-ai/deepagentsjs/commit/ab03794757cd130191037691d469d2f0b0c45f95) feat(evals): add aggregate all-suite eval package - [`678a443`](https://github.com/langchain-ai/deepagentsjs/commit/678a4438b1321face68a07f6669cea55b108fc54) feat(evals): allow global LangSmith project and dataset override - [`7fed4c7`](https://github.com/langchain-ai/deepagentsjs/commit/7fed4c72910a20e2dfe39ec9ff312892b0d16998) refactor(evals): extract suite builders and add single all-suite entrypoint - [`ba6572f`](https://github.com/langchain-ai/deepagentsjs/commit/ba6572f41af1b1fed106121c9f01d1f6920b1504) refactor(evals): rename suite files and move describe metadata into eval tests - [`c41cc43`](https://github.com/langchain-ai/deepagentsjs/commit/c41cc4381f2e6bddf4712125ecdc5631f353292b) refactor(evals): rename suite registration functions to xSuite - [`76d1264`](https://github.com/langchain-ai/deepagentsjs/commit/76d126417862aac2ce679764cc461af33797ebcd) cr - [`efe7d31`](https://github.com/langchain-ai/deepagentsjs/commit/efe7d31e4fe29d58dae07fbef48288927cf8f517) cr ### 📊 Changes **79 files changed** (+3800 additions, -3509 deletions) <details> <summary>View changed files</summary> 📝 `evals/README.md` (+8 -3) ➕ `evals/all/README.md` (+8 -0) ➕ `evals/all/eval.test.ts` (+61 -0) ➕ `evals/all/package.json` (+18 -0) ➕ `evals/all/vitest.config.ts` (+14 -0) ➕ `evals/basic/eval.test.ts` (+13 -0) ➖ `evals/basic/index.test.ts` (+0 -46) ➕ `evals/basic/index.ts` (+40 -0) ➕ `evals/external-benchmarks/eval.test.ts` (+13 -0) ➖ `evals/external-benchmarks/index.test.ts` (+0 -68) ➕ `evals/external-benchmarks/index.ts` (+64 -0) ➕ `evals/files/eval.test.ts` (+13 -0) ➖ `evals/files/index.test.ts` (+0 -454) ➕ `evals/files/index.ts` (+448 -0) ➕ `evals/followup-quality/eval.test.ts` (+13 -0) 📝 `evals/followup-quality/index.ts` (+29 -38) ➕ `evals/hitl/eval.test.ts` (+13 -0) ➖ `evals/hitl/index.test.ts` (+0 -287) ➕ `evals/hitl/index.ts` (+278 -0) ➕ `evals/memory-agent-bench/eval.test.ts` (+13 -0) _...and 59 more files_ </details> ### 📄 Description ## Summary This PR introduces a first-class aggregate eval entrypoint (`evals/all`) that runs every existing eval suite through one top-level LangSmith/Vitest describe block. To support that cleanly, eval suites were refactored into a reusable two-file shape: `index.ts` defines suite registration logic (`define...Suite(runner)`), and `eval.test.ts` owns LangSmith attribution (`ls.describe(...)`). This removes reliance on ad-hoc global override wiring and gives the all-evals package a single explicit integration surface. ## Changes ### Add aggregate all-evals package - Added `@deepagents/eval-all` package: - `evals/all/eval.test.ts` imports and invokes every suite registration function. - `evals/all/vitest.config.ts` runs only the aggregate `eval.test.ts` entrypoint. - Added package README and package metadata. ### Refactor all eval suites to reusable registration modules - Standardized suite implementation files to `index.ts`. - Standardized eval wrappers to `eval.test.ts`. - For Oolong datasets, standardized to `<dataset>.ts` + `<dataset>.eval.test.ts`. - Moved `ls.describe` dataset/project attribution into wrapper test files so registration modules focus on test definitions only. - Ensured suite registration functions accept `runner` as an argument across suites. ### Ensure all existing evals are included in aggregate execution - `evals/all/eval.test.ts` now imports and calls all suite registration functions for: - core eval suites under `evals/*` - oolong dataset suites under `evals/oolong/datasets/*` ### Documentation updates - Updated `evals/README.md` examples and structure references for `index.ts` + `eval.test.ts` layout. - Updated `evals/all/README.md` to document aggregate-run architecture. ### Lockfile updates - Updated `pnpm-lock.yaml` to include the new `evals/all` workspace package and dependencies. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
yindo added the pull-request label 2026-06-05 17:23:20 -04:00
yindo closed this issue 2026-06-05 17:23:20 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
enhancement
help wanted
needs-response
pull-request
Mirrored from GitHub Pull Request
 stale
triage: high-impact
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langchain-ai/deepagentsjs#478
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?