Hunter Lovell e3aed6f46f test(evals): expand parity coverage with python eval suites (#455) ...

## Summary

The change set adds new JS eval packages for summarization, external
benchmark hard-sets, memory-agent-bench style coverage, tau2-airline
style policy tasks, tool selection, follow-up quality, todos, and
multi-turn memory. It also extends existing `files` and `memory` suites
with parity cases previously only present in Python.

## Changes

### Eval suite parity expansion

Added new eval workspaces under `evals/`:

- `external-benchmarks/`
- Adds a curated 15-case hard set spanning FRAMES, Nexus, and
BFCL-v3-style samples.
  - Includes vendored case data at `data/curated_cases.json`.
- `summarization/`
- Adds summarization middleware behavior coverage for long-conversation
continuation and history offloading.
- `memory-agent-bench/`
- Adds MemoryAgentBench-style scenarios for long-context memorization,
conflict updates, and file-seeded retrieval.
- `tau2-airline/`
- Adds a 15-task policy-grounded airline support eval with explicit
tool-behavior assertions.
- `tool-selection/`
- Adds direct/indirect tool routing and chained tool invocation
coverage.
- `followup-quality/`
  - Adds underspecified-request follow-up quality coverage.
- `todos/`
  - Adds sequential `write_todos` behavior coverage.
- `memory-multiturn/`
- Adds multi-turn memory persistence and transient-info filtering
coverage.

Each new suite includes `package.json`, `vitest.config.ts`, `README.md`,
and `index.test.ts` and is wired to LangSmith reporting.

### Existing suite parity fixes

Updated existing suites:

- `evals/files/index.test.ts`
  - Adds pagination tail-recovery and empty-file handling cases.
- `evals/memory/index.test.ts`
- Adds transient memory exclusion, formatting preference persistence,
graceful missing-memory behavior, and memory path routing checks.

### Suite metadata and docs alignment

- Updated `evals/README.md` to include all newly added suites.
- Updated eval test structure to use `ls.describe(runner.name, ...)`
with stable per-suite `projectName` values, matching established local
pattern.
- Added/updated `evals/EVAL_SUITE_DIVERGENCE_ASSESSMENT.md` to reflect
parity progress.

### Dependency/workspace updates

- Updated `pnpm-lock.yaml` to include added eval workspace dependencies.

2026-04-13 23:26:17 -07:00

..

curated_cases.json

test(evals): expand parity coverage with python eval suites (#455)

2026-04-13 23:26:17 -07:00