mirror of
https://github.com/langchain-ai/deepagentsjs.git
synced 2026-07-20 19:40:28 -04:00
Hunter Lovell
d49b175dbd
## Summary This PR introduces a first-class aggregate eval entrypoint (`evals/all`) that runs every existing eval suite through one top-level LangSmith/Vitest describe block. To support that cleanly, eval suites were refactored into a reusable two-file shape: `index.ts` defines suite registration logic (`define...Suite(runner)`), and `eval.test.ts` owns LangSmith attribution (`ls.describe(...)`). This removes reliance on ad-hoc global override wiring and gives the all-evals package a single explicit integration surface. ## Changes ### Add aggregate all-evals package - Added `@deepagents/eval-all` package: - `evals/all/eval.test.ts` imports and invokes every suite registration function. - `evals/all/vitest.config.ts` runs only the aggregate `eval.test.ts` entrypoint. - Added package README and package metadata. ### Refactor all eval suites to reusable registration modules - Standardized suite implementation files to `index.ts`. - Standardized eval wrappers to `eval.test.ts`. - For Oolong datasets, standardized to `<dataset>.ts` + `<dataset>.eval.test.ts`. - Moved `ls.describe` dataset/project attribution into wrapper test files so registration modules focus on test definitions only. - Ensured suite registration functions accept `runner` as an argument across suites. ### Ensure all existing evals are included in aggregate execution - `evals/all/eval.test.ts` now imports and calls all suite registration functions for: - core eval suites under `evals/*` - oolong dataset suites under `evals/oolong/datasets/*` ### Documentation updates - Updated `evals/README.md` examples and structure references for `index.ts` + `eval.test.ts` layout. - Updated `evals/all/README.md` to document aggregate-run architecture. ### Lockfile updates - Updated `pnpm-lock.yaml` to include the new `evals/all` workspace package and dependencies.
tool-usage-relational
Evaluates multi-step tool chaining over relational data (users, locations, foods connected by IDs). Ported from the Python deepagents eval test_tool_usage_relational.py. Tests sequential lookups (1→4 hops) and parallel fan-out where independent tool calls should fire in the same step.