## Summary
This PR introduces a first-class aggregate eval entrypoint (`evals/all`)
that runs every existing eval suite through one top-level
LangSmith/Vitest describe block.
To support that cleanly, eval suites were refactored into a reusable
two-file shape: `index.ts` defines suite registration logic
(`define...Suite(runner)`), and `eval.test.ts` owns LangSmith
attribution (`ls.describe(...)`). This removes reliance on ad-hoc global
override wiring and gives the all-evals package a single explicit
integration surface.
## Changes
### Add aggregate all-evals package
- Added `@deepagents/eval-all` package:
- `evals/all/eval.test.ts` imports and invokes every suite registration
function.
- `evals/all/vitest.config.ts` runs only the aggregate `eval.test.ts`
entrypoint.
- Added package README and package metadata.
### Refactor all eval suites to reusable registration modules
- Standardized suite implementation files to `index.ts`.
- Standardized eval wrappers to `eval.test.ts`.
- For Oolong datasets, standardized to `<dataset>.ts` +
`<dataset>.eval.test.ts`.
- Moved `ls.describe` dataset/project attribution into wrapper test
files so registration modules focus on test definitions only.
- Ensured suite registration functions accept `runner` as an argument
across suites.
### Ensure all existing evals are included in aggregate execution
- `evals/all/eval.test.ts` now imports and calls all suite registration
functions for:
- core eval suites under `evals/*`
- oolong dataset suites under `evals/oolong/datasets/*`
### Documentation updates
- Updated `evals/README.md` examples and structure references for
`index.ts` + `eval.test.ts` layout.
- Updated `evals/all/README.md` to document aggregate-run architecture.
### Lockfile updates
- Updated `pnpm-lock.yaml` to include the new `evals/all` workspace
package and dependencies.
## Summary
The change set adds new JS eval packages for summarization, external
benchmark hard-sets, memory-agent-bench style coverage, tau2-airline
style policy tasks, tool selection, follow-up quality, todos, and
multi-turn memory. It also extends existing `files` and `memory` suites
with parity cases previously only present in Python.
## Changes
### Eval suite parity expansion
Added new eval workspaces under `evals/`:
- `external-benchmarks/`
- Adds a curated 15-case hard set spanning FRAMES, Nexus, and
BFCL-v3-style samples.
- Includes vendored case data at `data/curated_cases.json`.
- `summarization/`
- Adds summarization middleware behavior coverage for long-conversation
continuation and history offloading.
- `memory-agent-bench/`
- Adds MemoryAgentBench-style scenarios for long-context memorization,
conflict updates, and file-seeded retrieval.
- `tau2-airline/`
- Adds a 15-task policy-grounded airline support eval with explicit
tool-behavior assertions.
- `tool-selection/`
- Adds direct/indirect tool routing and chained tool invocation
coverage.
- `followup-quality/`
- Adds underspecified-request follow-up quality coverage.
- `todos/`
- Adds sequential `write_todos` behavior coverage.
- `memory-multiturn/`
- Adds multi-turn memory persistence and transient-info filtering
coverage.
Each new suite includes `package.json`, `vitest.config.ts`, `README.md`,
and `index.test.ts` and is wired to LangSmith reporting.
### Existing suite parity fixes
Updated existing suites:
- `evals/files/index.test.ts`
- Adds pagination tail-recovery and empty-file handling cases.
- `evals/memory/index.test.ts`
- Adds transient memory exclusion, formatting preference persistence,
graceful missing-memory behavior, and memory path routing checks.
### Suite metadata and docs alignment
- Updated `evals/README.md` to include all newly added suites.
- Updated eval test structure to use `ls.describe(runner.name, ...)`
with stable per-suite `projectName` values, matching established local
pattern.
- Added/updated `evals/EVAL_SUITE_DIVERGENCE_ASSESSMENT.md` to reflect
parity progress.
### Dependency/workspace updates
- Updated `pnpm-lock.yaml` to include added eval workspace dependencies.
Jacob Lee