mirror of
https://github.com/langchain-ai/deepagents.git
synced 2026-08-27 20:50:04 -04:00
Shrikar Seshadri
ddb69b609a
> [!WARNING] > This PR intentionally makes `GraderResponse.criteria` required. Existing callers that omit it must now pass `criteria=[]` explicitly when an empty list is valid, such as for a `failed` response. Fixes #4450 ## TL;DR `RubricMiddleware` now requires the grader to account for every rubric criterion before a `satisfied` verdict can end the self-improvement loop. It freezes the criterion list after the first pass, retries incomplete grading once, and downgrades still-unverified `satisfied` responses so partial or empty results cannot silently pass. --- ## The problem: The grader can return a terminal `satisfied` verdict with an empty or partial `criteria` list, silently ending the self-improve loop with most of the rubric ungraded. User who surfaced issue saw two issues: - runs where a 17-line rubric came back graded on 9 or 11 criteria - a run that reported a `satisfied` verdict contained an empty criterion list This happens because: 1. The emitted JSON schema told the grader that `criteria` was optional and never said what a criterion is. `criteria` carried `default_factory=list`, which keeps it out of the schema's `required` array, so returning `{"result": "satisfied", "explanation": "..."}` with no criteria at all was a *valid* response. The middleware saw `satisfied`, and terminated. **2. Nothing tied one grading pass to the next.** The rubric is free-form prose. At every invocation of the `RubricMiddleware`, the grading LLM starts from scratch in terms of turning the rubric prose into a list of criteria. There was no stored record of how many criteria the rubric decomposes into, so a second pass that graded 3 of 5 criteria was indistinguishable from a rubric that only ever had 3 criteria to begin with. Thusm there was no way to detect under-coverage. The existing `_check_result_consistency` validator only caught `needs_revision` with all-passing criteria. It could not catch a shrinking criterion set. --- ## How We Fixed This: ### 1. Strengthen the Schema to Make Criteria Descriptive and Required `criteria` is now required, and `name`/`gap` subfields for each criterion carry `Field(description=...)` text via `Annotated`. ``` required: ['result', 'explanation'] # before required: ['result', 'explanation', 'criteria'] # after ``` The `name` subfield instructs the grader to write a functional statement of exactly what is being checked, and to reuse the same wording whenever that criterion is graded again. ### 2. Freeze the criterion list after the first pass The `RubricMiddleware`'s first invocation is the only pass where the grading LLM parses the user-supplied `rubric` text into a criteria list. After this initial pass, the middleware stores the individual criterion names - now descriptive of **exactly** what's being assessed - into private state (`_rubric_criteria`). Pass/fail history isn't replayed, so each new instance of the grading LLM evaluates the criteria without bias from prior completions. The grader payload now two modes, and the rubric is sent in both: | | Payload | |---|---| | **Pass 0** (no frozen list) | rubric + transcript + *"Break the rubric into its individual criteria and return one entry per criterion."* | | **Pass 1+** (frozen list) | rubric + **numbered `<criteria-{nonce}>` checklist** + transcript + *"Return exactly N entries, one per listed criterion, in that order, reusing each name verbatim."* | When the middleware feeds criterion names to the grading LLM on iteration 1 and beyond, it sanitizes them first - since they were written by a grader that had just read untrusted transcript content, they pass through the same nonce-bracketed delimiter scrub used for the rubric and transcript. ### 3. Count validation, one retry, then a verdict gate A response is **unusable** when it verifies nothing (empty criterion list), or when a frozen list exists and the grading LLM doesn't verify all the listed criteria. When the middleware determines that the response is unusable, we retry the grader call (throw away the previous, incorrect one) with a prepended correction line telling the grader "You previously outputted X criterion, you need to output Y". `_grade` was split so the retry is a fresh grader call in the same mode, with a correction line prepended: - _invoke_grader(state, iteration, correction=None) —> performs exactly one grader call: builds the payload, sends it to the grading LLM, parses the structured response back into a `GraderResponse`. When the `correction` variable is set to `True`, it prepends a clarifying sentence to the payload. - _grade(state, iteration) —> owns the retry decision. Calls _invoke_grader once, checks whether the returned criterion count matches the frozen list's count, and if it doesn't, calls _invoke_grader a second time with the correction filled in. Returns whichever response came back last. Exactly one retry. If the second response is still unusable, it goes to the gate (see next paragraph). ### 4. The gate: `satisfied` can't end the loop on unverified evidence | Verdict | Result still unusable after retrying grading LLM call | Result | |---|---|---| | `satisfied` | yes | **downgraded to `needs_revision`**, `unverified=True`, warning logged | | `needs_revision` | yes | left alone - it claims nothing that needs blocking | | `failed` | n/a | left alone - a contradictory rubric has nothing to enumerate | The `max_iterations` check now reads the **downgraded** verdict, so a permanently broken grader still terminates as `max_iterations_reached` instead of looping forever. ### 5. No-regression instruction in the revision prompt The `HumanMessage` back to the main agent now lists the passing criteria under *"Criteria already satisfied -- do not regress these:"* alongside the failing ones and their gaps. When the downgrade fired, the wording changes to say this is a **verification gap, not a defect list** - the agent is told to re-verify and state its evidence, and to *not* change things that are already correct. --- ## Worst-Case End-to-End Workflow: ``` iteration 0 grader → 3 criteria (2 pass, 1 fail) → needs_revision freeze _rubric_criteria = [3 names] inject revision prompt: failing criterion + gap + "Criteria already satisfied -- do not regress these:" + 2 names iteration 1 payload: rubric + numbered checklist of 3 + "Return exactly 3 entries" grader → 1 criterion ← undercounts frozen list of criteria ↓ RETRY (same mode, fresh call) payload: "This is grader iteration 1, regrading after an unusable response. A previous attempt returned 1 criteria; the rubric has exactly 3." + rubric + numbered checklist of 3 grader → "satisfied" with 1 criterion ← still undercounts frozen list of criteria ↓ GATE: satisfied + unusable → needs_revision, unverified=True explanation rewritten, original grader summary preserved warning logged: "downgrading 'satisfied' to 'needs_revision'" inject revision prompt: "A grader reviewed your work but could not verify every criterion in the rubric ... This is a gap in verification, not a list of confirmed defects." iteration 2 grader → 3 criteria, all passing → satisfied → terminate ``` Previously, this workflow would have been one iteration with a 'satsified' rubric, even though 2 out of the three criterion had not been verified. --- ## Subclass seam (added for `deepagents-code`) - The single grader call now sits behind `_invoke_grader`/`_grader_input`, with `context` threaded through, so a subclass that wraps one call (dcode's `ReliableRubricMiddleware` retries transient transport failures) inherits the coverage retry in `_grade` instead of replacing it. `after_agent` also re-raises `GraphBubbleUp`, so a grader that interrupts is no longer recorded as `grader_error`. - `rubric_evaluation_end` now carries `unverified`, so a stream consumer can render a verification gap as such rather than as an empty list of confirmed defects. Consumed by the stacked dcode PR. --------- Co-authored-by: Mason Daugherty <61371264+mdrxy@users.noreply.github.com> Co-authored-by: open-swe[bot] <open-swe@users.noreply.github.com> Co-authored-by: Mason Daugherty <github@mdrxy.com>
4.9 KiB
4.9 KiB
Examples
Real agents and patterns built on Deep Agents.
Featured
Deep Agents CodeA pre-built coding Deep Agent in your terminal — similar to Claude Code or Codex — powered by any LLM. Includes an interactive TUI, web search, remote sandboxes, persistent memory, custom skills, and human-in-the-loop approval. |
Open SWEAn open-source, async coding agent for your org's internal workflows. Runs each task in an isolated cloud sandbox, integrates with Slack, Linear, and GitHub, and ships PRs end-to-end. |
In the wild
Production agents powered by the LangChain stack:
| Project | Description |
|---|---|
| LangSmith Fleet | No-code platform for building AI agents from templates; connect your accounts and let the agent handle routine work |
| Chat LangChain | Documentation assistant that answers questions about LangChain, LangGraph, and LangSmith (source) |
All examples
Research
| Example | Description |
|---|---|
| Deep Research | Multi-step web research with Tavily, parallel sub-agents, and strategic reflection |
| MCP Docs Agent | Docs research agent using MCP tools over LangChain documentation |
Coding
| Example | Description |
|---|---|
| Coding Agent | Autonomous coding agent in a LangSmith sandbox |
| Nemotron Research Agent | NVIDIA Nemotron Super for research + GPU-accelerated execution via RAPIDS |
Content
| Example | Description |
|---|---|
| Content Builder | Blog posts, LinkedIn posts, and tweets with memory (AGENTS.md), skills, and subagents |
| Text-to-SQL | Natural language to SQL with planning and skill-based workflows on the Chinook demo database |
| LLM Wiki | Script-first LLM wiki synced via langsmith hub init/pull/push |
Deployable services
| Example | Description |
|---|---|
| Content Writer | Content writer with per-user memory and Supabase auth |
| GTM Strategist | GTM strategy agent coordinating sync and async subagents |
| Async Subagent Server | Self-hosted Agent Protocol server exposing a researcher as an async subagent |
Advanced patterns
| Example | Description |
|---|---|
| Ralph Loop | Autonomous looping with fresh context each iteration, using the filesystem for persistence |
| Agents as Folders | Download a zip, unzip, and run |
| Better Harness | Eval-driven outer-loop optimization of a Deep Agents harness |
| Rubric Middleware | Grader-model rubric feedback loop that revises output until all criteria pass |
Each example has its own README with setup instructions.
Contributing an example
See the Contributing Guide for general contribution guidelines.
When adding a new example:
- Use uv for dependency management with a
pyproject.tomlanduv.lock(commit the lock file) - Pin to deepagents version — use a version range (e.g.,
>=0.3.5,<0.4.0) in dependencies - Include a
READMEwith clear setup and usage instructions - Add tests for reusable utilities or non-trivial helper logic
- Keep it focused — each example should demonstrate one use-case or workflow
- Follow the structure of existing examples (see
deep_research/ortext-to-sql-agent/as references)
Resources
- LangChain Academy — Comprehensive, free courses on LangChain libraries and products, made by the LangChain team.
- Code of Conduct — community guidelines and standards