mirror of https://github.com/langchain-ai/deepagentsjs.git synced 2026-07-20 19:40:28 -04:00

Files

T

Hunter Lovell d49b175dbd feat(evals): add aggregate all-evals entrypoint and suite-builder structure (#457 )

## Summary

This PR introduces a first-class aggregate eval entrypoint (`evals/all`)
that runs every existing eval suite through one top-level
LangSmith/Vitest describe block.

To support that cleanly, eval suites were refactored into a reusable
two-file shape: `index.ts` defines suite registration logic
(`define...Suite(runner)`), and `eval.test.ts` owns LangSmith
attribution (`ls.describe(...)`). This removes reliance on ad-hoc global
override wiring and gives the all-evals package a single explicit
integration surface.

## Changes

### Add aggregate all-evals package
- Added `@deepagents/eval-all` package:
- `evals/all/eval.test.ts` imports and invokes every suite registration
function.
- `evals/all/vitest.config.ts` runs only the aggregate `eval.test.ts`
entrypoint.
  - Added package README and package metadata.

### Refactor all eval suites to reusable registration modules
- Standardized suite implementation files to `index.ts`.
- Standardized eval wrappers to `eval.test.ts`.
- For Oolong datasets, standardized to `<dataset>.ts` +
`<dataset>.eval.test.ts`.
- Moved `ls.describe` dataset/project attribution into wrapper test
files so registration modules focus on test definitions only.
- Ensured suite registration functions accept `runner` as an argument
across suites.

### Ensure all existing evals are included in aggregate execution
- `evals/all/eval.test.ts` now imports and calls all suite registration
functions for:
  - core eval suites under `evals/*`
  - oolong dataset suites under `evals/oolong/datasets/*`

### Documentation updates
- Updated `evals/README.md` examples and structure references for
`index.ts` + `eval.test.ts` layout.
- Updated `evals/all/README.md` to document aggregate-run architecture.

### Lockfile updates
- Updated `pnpm-lock.yaml` to include the new `evals/all` workspace
package and dependencies.

2026-04-14 00:28:06 -07:00

eval.test.ts

feat(evals): add aggregate all-evals entrypoint and suite-builder structure (#457 )

2026-04-14 00:28:06 -07:00

index.ts

feat(evals): add aggregate all-evals entrypoint and suite-builder structure (#457 )

2026-04-14 00:28:06 -07:00

package.json

chore: upgrade langgraph + langchain (#434 )

2026-04-07 08:41:32 -07:00

README.md

feat: add eval harness and initial eval suites (#309 )