langchain-ai/deepagentsjs
1
0
Fork 0
mirror of https://github.com/langchain-ai/deepagentsjs.git synced 2026-07-18 16:04:44 -04:00
Code Issues 42 Packages Projects Releases 10 Wiki Activity

[PR #271] [CLOSED] feat(deepagents): add RLM OOLONG evaluation harness with KVBackend #317

New Issue
Closed
opened 2026-06-05 17:22:38 -04:00 by yindo · 0 comments
yindo commented 2026-06-05 17:22:38 -04:00
Owner
Copy Link
Copy Source

📋 Pull Request Information

Original PR: https://github.com/langchain-ai/deepagentsjs/pull/271
Author: @hntrl
Created: 3/3/2026
Status: ❌ Closed

Base: main ← Head: hunter/repl-rlm

📝 Commits (4)

  • c5d2095 feat(quickjs): add @langchain/quickjs — sandboxed JS/TS REPL with PTC, env isolation, and RLM support
  • 3b7c3df chore: split changesets into quickjs minor + deepagents patch
  • 1c16f74 feat(deepagents): add RLM OOLONG evaluation harness with KVBackend
  • a9cd41b cr

📊 Changes

15 files changed (+1660 additions, -894 deletions)

View changed files

➕ .changeset/deepagents-ptc-task.md (+7 -0)
➕ .changeset/quickjs-repl.md (+10 -0)
📝 .gitignore (+2 -1)
➕ examples/repl/secret-env-agent.ts (+85 -0)
📝 libs/deepagents/package.json (+1 -0)
📝 libs/deepagents/src/backends/index.ts (+1 -0)
➕ libs/deepagents/src/backends/kv.ts (+171 -0)
➕ libs/deepagents/src/evals/rlm-oolong.eval.test.ts (+101 -0)
➕ libs/deepagents/src/evals/rlm/agent.ts (+79 -0)
➕ libs/deepagents/src/evals/rlm/load-oolong.ts (+179 -0)
➕ libs/deepagents/src/evals/rlm/prompts.ts (+25 -0)
➕ libs/deepagents/src/evals/rlm/scoring.ts (+130 -0)
📝 libs/deepagents/src/index.ts (+3 -0)
📝 libs/deepagents/vitest.config.ts (+1 -1)
📝 pnpm-lock.yaml (+865 -892)

📄 Description

Summary

Adds the Recursive Language Model (RLM) evaluation harness for the OOLONG benchmark, along with supporting infrastructure (KVBackend, QuickJS middleware fixes) needed to run the eval.

Changes

RLM OOLONG Evaluation

A vitest-based eval harness that tests the RLM pattern against the OOLONG trec_coarse benchmark — 50 long-context aggregation tasks at 131K tokens each. The agent uses the QuickJS REPL to chunk context, fan out classification to sub-LLMs via tools.task() + Promise.all, and aggregate results programmatically.

  • rlm-oolong.eval.test.ts — test runner with LangSmith experiment tracking. Configurable via OOLONG_MAX_TASKS and OOLONG_EXPERIMENT_NAME env vars.
  • rlm/agent.ts — agent factory: Claude Opus coordinator + Haiku subagent, QuickJS REPL with PTC, shared KVBackend for VFS.
  • rlm/prompts.ts — minimal coordinator prompt based on RLM paper (Appendix D.1). Strategy discovery is left to the model.
  • rlm/load-oolong.ts — dataset loader that fetches trec_coarse 131K tasks from HuggingFace and caches locally as JSONL.
  • rlm/scoring.ts — four-strategy scoring (exact/normalized/contains/numeric match) ported from the Python rlming harness for comparable results.

Paper baselines (GPT-5): Base 44% → RLM 56.5%.

KVBackend

New in-memory key-value BackendProtocol implementation for use inside the QuickJS REPL. All operations mutate a Map directly — no LangGraph Command indirection needed — which is required when the backend is shared between the REPL and subagents within a single tool call.

QuickJS middleware fix

Fixed session lookup to fall back to "__default__" when thread_id isn't available in configurable, and reordered PTC tool filtering to happen before backend resolution.

Misc

  • Exported KVBackend, createFileData, and StateAndStore from deepagents.
  • Added @langchain/quickjs as a dev dependency of deepagents (for the eval).
  • Increased eval test timeout to 20 minutes for long-context tasks.
  • Added oolong-data/ to .gitignore.

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/langchain-ai/deepagentsjs/pull/271 **Author:** [@hntrl](https://github.com/hntrl) **Created:** 3/3/2026 **Status:** ❌ Closed **Base:** `main` ← **Head:** `hunter/repl-rlm` --- ### 📝 Commits (4) - [`c5d2095`](https://github.com/langchain-ai/deepagentsjs/commit/c5d20954efcb27511500182f9fd7c334d67ee6bf) feat(quickjs): add @langchain/quickjs — sandboxed JS/TS REPL with PTC, env isolation, and RLM support - [`3b7c3df`](https://github.com/langchain-ai/deepagentsjs/commit/3b7c3df605de1026d9859b7edb2ace7b5e0bac6a) chore: split changesets into quickjs minor + deepagents patch - [`1c16f74`](https://github.com/langchain-ai/deepagentsjs/commit/1c16f7411367df95a6ca4de2dc884bfb7a69ddd5) feat(deepagents): add RLM OOLONG evaluation harness with KVBackend - [`a9cd41b`](https://github.com/langchain-ai/deepagentsjs/commit/a9cd41bd7d58920a181aa79ea7f94542faa66881) cr ### 📊 Changes **15 files changed** (+1660 additions, -894 deletions) <details> <summary>View changed files</summary> ➕ `.changeset/deepagents-ptc-task.md` (+7 -0) ➕ `.changeset/quickjs-repl.md` (+10 -0) 📝 `.gitignore` (+2 -1) ➕ `examples/repl/secret-env-agent.ts` (+85 -0) 📝 `libs/deepagents/package.json` (+1 -0) 📝 `libs/deepagents/src/backends/index.ts` (+1 -0) ➕ `libs/deepagents/src/backends/kv.ts` (+171 -0) ➕ `libs/deepagents/src/evals/rlm-oolong.eval.test.ts` (+101 -0) ➕ `libs/deepagents/src/evals/rlm/agent.ts` (+79 -0) ➕ `libs/deepagents/src/evals/rlm/load-oolong.ts` (+179 -0) ➕ `libs/deepagents/src/evals/rlm/prompts.ts` (+25 -0) ➕ `libs/deepagents/src/evals/rlm/scoring.ts` (+130 -0) 📝 `libs/deepagents/src/index.ts` (+3 -0) 📝 `libs/deepagents/vitest.config.ts` (+1 -1) 📝 `pnpm-lock.yaml` (+865 -892) </details> ### 📄 Description ## Summary Adds the Recursive Language Model (RLM) evaluation harness for the OOLONG benchmark, along with supporting infrastructure (`KVBackend`, QuickJS middleware fixes) needed to run the eval. ## Changes ### RLM OOLONG Evaluation A vitest-based eval harness that tests the RLM pattern against the OOLONG trec_coarse benchmark — 50 long-context aggregation tasks at 131K tokens each. The agent uses the QuickJS REPL to chunk context, fan out classification to sub-LLMs via `tools.task()` + `Promise.all`, and aggregate results programmatically. - **`rlm-oolong.eval.test.ts`** — test runner with LangSmith experiment tracking. Configurable via `OOLONG_MAX_TASKS` and `OOLONG_EXPERIMENT_NAME` env vars. - **`rlm/agent.ts`** — agent factory: Claude Opus coordinator + Haiku subagent, QuickJS REPL with PTC, shared `KVBackend` for VFS. - **`rlm/prompts.ts`** — minimal coordinator prompt based on RLM paper (Appendix D.1). Strategy discovery is left to the model. - **`rlm/load-oolong.ts`** — dataset loader that fetches trec_coarse 131K tasks from HuggingFace and caches locally as JSONL. - **`rlm/scoring.ts`** — four-strategy scoring (exact/normalized/contains/numeric match) ported from the Python rlming harness for comparable results. Paper baselines (GPT-5): Base 44% → RLM 56.5%. ### `KVBackend` New in-memory key-value `BackendProtocol` implementation for use inside the QuickJS REPL. All operations mutate a `Map` directly — no LangGraph Command indirection needed — which is required when the backend is shared between the REPL and subagents within a single tool call. ### QuickJS middleware fix Fixed session lookup to fall back to `"__default__"` when `thread_id` isn't available in `configurable`, and reordered PTC tool filtering to happen before backend resolution. ### Misc - Exported `KVBackend`, `createFileData`, and `StateAndStore` from `deepagents`. - Added `@langchain/quickjs` as a dev dependency of `deepagents` (for the eval). - Increased eval test timeout to 20 minutes for long-context tasks. - Added `oolong-data/` to `.gitignore`. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
yindo added the pull-request label 2026-06-05 17:22:38 -04:00
yindo closed this issue 2026-06-05 17:22:38 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
enhancement
help wanted
needs-response
pull-request
Mirrored from GitHub Pull Request
 stale
triage: high-impact
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langchain-ai/deepagentsjs#317
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?