Implementation of Oolong: Evaluating Long Context Reasoning and Aggregation Capabilities by Amanda Bertsch, Adithya Pratapa, Teruko Mitamura, Graham Neubig, and Matthew R. Gormley (2025).

As model context lengths continue to grow, concerns about whether models effectively use the full context length have persisted. Oolong is a benchmark of long-context reasoning tasks that require analyzing individual chunks of text on an atomic level, and then aggregating these analyses to answer distributional questions.

Uses the oolong-synth dataset from HuggingFace. The agent receives a large context_window_text as a seeded file and must answer aggregation questions (counting, frequency, temporal, user-based).

Structure

Each source dataset has its own test file under datasets/:

File	Source dataset
`datasets/spam.test.ts`	SMS spam classification
`datasets/trec_coarse.test.ts`	TREC question type classification
`datasets/agnews.test.ts`	AG News topic classification
`datasets/imdb.test.ts`	IMDB sentiment
`datasets/negation.test.ts`	HiTZ negation detection
`datasets/yahoo.test.ts`	Yahoo Answers topics
`datasets/formality.test.ts`	Pavlick formality
`datasets/multinli.test.ts`	MultiNLI entailment
`datasets/metaphors.test.ts`	BigBench metaphor interpretation
`datasets/app_reviews.test.ts`	App review sentiment

Data loading is handled by loadOolongTasksByDataset() in load-oolong.ts which caches at the module level, so multiple test files share the same data. The shared test logic lives in make-tests.ts.

Running

# Run all datasets
EVAL_RUNNER=sonnet-4-5 pnpm --filter @deepagents/eval-oolong test:eval

# Run a single dataset
EVAL_RUNNER=sonnet-4-5 pnpm --filter @deepagents/eval-oolong test:eval -- datasets/spam.test.ts

# Run all validation tasks (~1300)
OOLONG_MAX_PER_DATASET=0 EVAL_RUNNER=sonnet-4-5 pnpm --filter @deepagents/eval-oolong test:eval

# Custom subset size
OOLONG_MAX_PER_DATASET=5 EVAL_RUNNER=sonnet-4-5 pnpm --filter @deepagents/eval-oolong test:eval

Environment variables

Variable	Default	Description
`EVAL_RUNNER`	(required)	Model runner to use (e.g. `sonnet-4-5`, `opus-4-6`)
`OOLONG_MAX_PER_DATASET`	`10`	Max tasks per source dataset. Set to `0` for all.
`OOLONG_CONTEXT_LEN`	(all)	Filter to a specific `context_len` (e.g. `1024`, `131072`)

Scoring

Scoring is ported from the official Oolong eval harness (synth_process_response) to ensure results are directly comparable to the paper.

Answer parsing -- split on : and take the last segment; strip markdown/bracket artifacts
Exact match -- str(parsed) == str(gold) after parsing
Comparison answers (ANSWER_TYPE.COMPARISON) -- substring containment for "more common than" / "less common than" / "same frequency as"
Numeric answers (ANSWER_TYPE.NUMERIC) -- partial credit via 0.75^|gold - pred|
Date answers (ANSWER_TYPE.DATE) -- flexible date parsing comparison

A prediction is scored 1.0 for exact/comparison/date matches, partial credit for near numeric answers, and 0 otherwise. The test assertion requires a perfect score (1.0).

Citation

@article{bertsch2025oolong,
  title={Oolong: Evaluating Long Context Reasoning and Aggregation Capabilities},
  author={Bertsch, Amanda and Pratapa, Adithya and Mitamura, Teruko and Neubig, Graham and Gormley, Matthew R.},
  journal={arXiv preprint arXiv:2511.02817},
  year={2025}
}

Adaptations

Uses the oolong-synth validation split (all source datasets, not just trec_coarse)
Defaults to a 10-task-per-dataset subset for cost efficiency
Context is seeded as an initialFile rather than a REPL VFS
Agent uses the standard getDefaultRunner() harness rather than a custom RLM agent