langchain-ai/deepagents
1
0
Fork 0
mirror of https://github.com/langchain-ai/deepagents.git synced 2026-07-22 01:35:28 -04:00
Code Issues 169 Packages Projects Releases 16 Wiki Activity

fix(code): make /goal completion and grading reliable (#4691)

Browse Source 
Fixed `/goal` so completed objectives no longer grade later prompts,
while unfinished goals and completion requests that could not be saved
remain recoverable with clear guidance about what happens next.

---

`/goal` turns a plain-language objective into acceptance criteria and
keeps checking each follow-up turn until the work is done. This PR fixes
what happens when that lifecycle ends—or when grading or checkpoint
persistence fails.

Previously, a goal could remain active even after its completion was
approved. The next prompt would then still be graded against work that
was already finished.

Grader connection failures also looked like unmet criteria, and reaching
the grading iteration limit did not explain what was left to do.

Now, an approved completion clears the goal and its rubric after that
transition is saved, so later prompts start clean and `/goal show`
reports that no goal is set. The completion message and evidence remain
in the thread transcript. Blocked goals stay active because they still
need user input and should remain resumable.

If the completion cannot be saved, dcode restores the active goal and
its pending completion request instead of letting the UI diverge from
the thread checkpoint. The request is graded again on the next turn.

Grading is also more reliable and actionable:

- A transient mid-response transport failure retries the grader once
without rerunning the agent turn, so edits and tool calls are not
repeated.
- A persistent grader or infrastructure failure is identified as such
and leaves the unfinished goal intact for a safe retry.
- If grading reaches its iteration limit, dcode shows the unmet criteria
and how to resume, amend, or clear the goal. Headless runs show the same
criterion-level gaps.

## User stories and before/after examples

<details>
<summary>Completed goals stop grading unrelated follow-up
prompts</summary>

**User story:** As a user who has finished a goal, I want it cleared so
later prompts start clean and `/goal show` reflects that there is no
current goal.

**Before**

1. I complete “Add refresh-token support,” and dcode approves the
completion.
2. I ask, “Now explain how cache eviction works.”
3. dcode still grades that explanation against the refresh-token
acceptance criteria.

**After**

1. Once completion is approved and saved, dcode clears the goal, rubric,
status, and pending completion state.
2. “Now explain how cache eviction works” runs without the completed
goal attached.
3. `/goal show` reports “No goal set.” The earlier completion message
and evidence remain in the thread transcript.

</details>

<details>
<summary>A completion that cannot be saved remains safe to
retry</summary>

**User story:** As a user whose thread checkpoint is temporarily
unavailable, I want dcode’s displayed state to match the saved thread so
resuming cannot silently lose or misreport my goal.

**Before**

1. The grader approves completion, but saving the completed state fails.
2. dcode can look complete in the current session even though the thread
still contains an active goal and pending completion.

**After**

1. dcode restores the active goal, rubric, and pending completion
request in the UI.
2. It reports: “Goal completion could not be saved, so the goal remains
active and its completion request is still pending for retry.”
3. The completion request is graded again on the next turn.

</details>

<details>
<summary>A dropped grader response retries grading without repeating the
work</summary>

**User story:** As a user whose grader connection drops while reading a
response, I want dcode to retry the check without rerunning edits or
tool calls that already succeeded.

**Before**

1. The agent finishes the requested work.
2. The grader connection closes partway through its response.
3. The grading attempt ends even though the agent turn itself succeeded.

**After**

1. dcode retries the grader once for recognized mid-response transport
failures.
2. Only the grading subagent runs again; the task agent’s edits and tool
calls are not repeated.
3. If the retry succeeds, the normal rubric result is used.

</details>

<details>
<summary>Persistent grader failures are not presented as missing user
work</summary>

**User story:** As a user facing a grader or infrastructure outage, I
want dcode to identify the outage instead of telling me that my
implementation failed its criteria.

**Before**

A grader connection failure can look like an unmet rubric, leaving it
unclear whether the implementation or the grading service failed.

**After**

- The transcript identifies a persistent transport problem as
“Grader/infrastructure failure.”
- If the grader cannot evaluate the rubric, dcode says so explicitly
rather than reporting ordinary unmet criteria.
- The goal remains active. For an infrastructure failure, its completion
request also remains pending and is re-graded on the next turn.

</details>

<details>
<summary>Iteration limits show the remaining gaps and recovery
choices</summary>

**User story:** As a user whose goal reaches the grading limit, I want
to know what is still missing and how to continue.

**Before**

> ⚠ Acceptance criteria not satisfied (iteration limit reached)

The message does not identify the failing criteria or explain what
happens to the goal.

**After**

> ⚠ Iteration limit reached with unmet acceptance criteria  
> ✗ Refresh-token tests — the expiration case is still failing  
> The goal remains active. Continue with another prompt to resume or
retry, use `/goal <objective>` to amend it, or `/goal clear` to clear
it.

Interactive and headless runs both show criterion-level gaps, and the
unfinished goal remains available for the next action.

</details>
This commit is contained in:
Mason Daugherty
2026-07-14 00:29:26 -04:00
committed by GitHub
parent 65e1ee802e
commit 9da63c64a5
10 changed files with 601 additions and 57 deletions
Download Patch File Download Diff File Expand all files Collapse all files
libs/code/deepagents_code/agent.py
+2 -2
View File
@@ -20,7 +20,6 @@ from deepagents.middleware import (
GRADER_SYSTEM_PROMPT,
FilesystemMiddleware,
MemoryMiddleware,
RubricMiddleware,
SkillsMiddleware,
)
@@ -88,6 +87,7 @@ from deepagents_code.local_context import (
_ExecutableBackend,
)
from deepagents_code.project_utils import ProjectContext, get_server_project_context
from deepagents_code.reliable_rubric import ReliableRubricMiddleware
from deepagents_code.subagents import list_subagents
from deepagents_code.unicode_security import (
check_url_safety,
@@ -1954,7 +1954,7 @@ def create_cli_agent(
}
if rubric_max_iterations is not None:
rubric_kwargs["max_iterations"] = rubric_max_iterations
agent_middleware.append(RubricMiddleware(**rubric_kwargs))
agent_middleware.append(ReliableRubricMiddleware(**rubric_kwargs))
# Create the agent
all_subagents: list[SubAgent | CompiledSubAgent | AsyncSubAgent] = [
libs/code/deepagents_code/app.py
+76 -20
View File
@@ -8808,8 +8808,8 @@ class DeepAgentsApp(App):
def _clear_all_goal_rubric_state(self) -> None:
"""Clear every goal and rubric field (sticky, one-shot, goal, pending).
Single reset point so the clear paths cannot drift out of sync over the
ten correlated fields. Grader settings (`_rubric_model` and
Single reset point so the clear paths cannot drift across the correlated
fields. Grader settings (`_rubric_model` and
`_rubric_max_iterations`) are intentionally left untouched — they are
configured separately via `/rubric model` and `/rubric max-iterations`
and survive `/rubric clear` and `/clear`.
@@ -8920,21 +8920,56 @@ class DeepAgentsApp(App):
note: str,
*,
previous_status: str | None,
) -> None:
"""Mark a rubric-approved completion request complete."""
self._goal_status = "complete"
self._goal_status_note = note
self._pending_goal_completion_note = None
) -> bool:
"""Clear a rubric-approved goal after its completion is saved.
Returns:
`True` when the cleared state was persisted, or `False` when the active
goal was restored for retry.
"""
active_goal = self._active_goal
goal_status = self._goal_status
goal_status_note = self._goal_status_note
active_rubric = self._active_rubric
pending_goal_completion_note = self._pending_goal_completion_note
pending_goal_objective = self._pending_goal_objective
pending_goal_rubric = self._pending_goal_rubric
next_rubric = self._next_rubric
last_consumed_next_rubric = self._last_consumed_next_rubric
last_consumed_next_previous_rubric = self._last_consumed_next_previous_rubric
self._clear_all_goal_rubric_state()
self._sync_status_rubric()
persisted = await self._persist_goal_rubric_state()
await self._announce_goal_status_transition(previous_status)
if not persisted:
await self._mount_message(
ErrorMessage(
"Goal marked complete for this session, but it could not "
"be saved to the thread."
)
if persisted:
if previous_status != "complete":
text = "Goal marked complete by the agent."
if note:
text = f"{text}\n\n{note}"
await self._mount_message(AppMessage(text))
return True
# The checkpoint still contains the active goal and pending completion
# request. Restore that exact local state so the UI and next grading
# turn cannot diverge from it, and so a later sync can safely retry.
self._active_goal = active_goal
self._goal_status = goal_status
self._goal_status_note = goal_status_note
self._active_rubric = active_rubric
self._pending_goal_completion_note = pending_goal_completion_note or note
self._pending_goal_objective = pending_goal_objective
self._pending_goal_rubric = pending_goal_rubric
self._next_rubric = next_rubric
self._last_consumed_next_rubric = last_consumed_next_rubric
self._last_consumed_next_previous_rubric = last_consumed_next_previous_rubric
self._sync_status_rubric()
await self._mount_message(
ErrorMessage(
"Goal completion could not be saved, so the goal remains active "
"and its completion request is still pending for retry."
)
)
return False
async def _clear_pending_goal_completion(self, message: str) -> None:
"""Clear a completion request that did not become complete."""
@@ -8957,6 +8992,28 @@ class DeepAgentsApp(App):
if not self._active_goal or not note or self._goal_status == "complete":
return False
if rubric_status == "grader_error":
await self._mount_message(
ErrorMessage(
"Acceptance-criteria grading failed because of a grader or "
"infrastructure error. The goal remains active, and its completion "
"request is still pending; it will be re-graded on your next turn."
)
)
return False
if rubric_status == "max_iterations_reached":
await self._clear_pending_goal_completion(
"Goal completion was not recorded: the iteration limit was reached "
"with unmet criteria. The goal remains active for resume, amendment, "
"retry, or clearing."
)
return False
if rubric_status == "failed":
await self._clear_pending_goal_completion(
"Goal completion was not recorded because the grader could not "
"evaluate the rubric. The goal remains active."
)
return False
if rubric_status != "satisfied":
await self._clear_pending_goal_completion(
"Goal completion was not recorded because the rubric was not satisfied."
@@ -8964,11 +9021,10 @@ class DeepAgentsApp(App):
return False
if self._session_state is not None and self._session_state.auto_approve:
await self._commit_pending_goal_completion(
return await self._commit_pending_goal_completion(
note,
previous_status=previous_status,
)
return True
action_requests = [
{
@@ -8995,17 +9051,15 @@ class DeepAgentsApp(App):
decision_type = decision.get("type") if isinstance(decision, dict) else None
if decision_type == "auto_approve_all":
await self._on_auto_approve_enabled()
await self._commit_pending_goal_completion(
return await self._commit_pending_goal_completion(
note,
previous_status=previous_status,
)
return True
if decision_type == "approve":
await self._commit_pending_goal_completion(
return await self._commit_pending_goal_completion(
note,
previous_status=previous_status,
)
return True
reject_message = decision.get("message") if isinstance(decision, dict) else None
if isinstance(reject_message, str) and reject_message.strip():
@@ -9497,6 +9551,7 @@ class DeepAgentsApp(App):
self._active_goal = objective
self._goal_status = "active"
self._goal_status_note = None
self._pending_goal_completion_note = None
self._active_rubric = rubric
self._next_rubric = None
self._pending_goal_objective = None
@@ -11335,6 +11390,7 @@ class DeepAgentsApp(App):
sandbox_type=self._sandbox_type,
message_kwargs=message_kwargs,
rubric=rubric,
goal_active=bool(self._active_goal),
blocked_goal_retry_context=blocked_goal_retry_context,
# `auto_approve` is intentionally omitted here: execute_task_textual
# writes it into this context from `session_state.auto_approve` at
libs/code/deepagents_code/client/non_interactive.py
+19 -8
View File
@@ -745,15 +745,26 @@ def _process_rubric_event(
detail = f" — {gap}" if gap else ""
console.print(f"[yellow] ✗ {name}{detail}[/yellow]", highlight=False)
elif result == "max_iterations_reached":
console.print(
"[yellow]⚠ Acceptance criteria not satisfied "
"(iteration limit reached)[/yellow]",
highlight=False,
)
elif result in {"failed", "grader_error"}:
label = "grader failed" if result == "failed" else "grader error"
suffix = f": {escape_markup(explanation)}" if explanation else ""
console.print(f"[red]⚠ Rubric {label}{suffix}[/red]", highlight=False)
message = (
"[yellow]⚠ Iteration limit reached with unmet acceptance criteria"
f"{suffix}[/yellow]"
)
console.print(message, highlight=False)
for criterion in data.get("criteria", []):
if isinstance(criterion, dict) and not criterion.get("passed", True):
name = escape_markup(str(criterion.get("name", "criterion")))
gap = escape_markup(str(criterion.get("gap", "")).strip())
detail = f" — {gap}" if gap else ""
console.print(f"[yellow] ✗ {name}{detail}[/yellow]", highlight=False)
elif result == "failed":
suffix = f": {escape_markup(explanation)}" if explanation else ""
console.print(f"[red]⚠ Rubric grader failed{suffix}[/red]", highlight=False)
elif result == "grader_error":
suffix = f": {escape_markup(explanation)}" if explanation else ""
console.print(
f"[red]⚠ Grader/infrastructure failure{suffix}[/red]", highlight=False
)
elif result is not None:
# A `rubric_evaluation_end` with an unrecognized result is still a
# terminal grading event; surface it rather than letting the run go
libs/code/deepagents_code/reliable_rubric.py
+97
View File
@@ -0,0 +1,97 @@
"""Rubric middleware retries for transient grader transport failures."""
from __future__ import annotations
import logging
from typing import TYPE_CHECKING
import httpx
from deepagents.middleware.rubric import GraderResponse, RubricMiddleware, RubricState
if TYPE_CHECKING:
from collections.abc import Iterator
logger = logging.getLogger(__name__)
def _exception_chain(exc: BaseException) -> Iterator[BaseException]:
"""Yield an exception, its explicit/implicit causes, and group members once.
Descends into `BaseExceptionGroup` members as well as `__cause__` and
`__context__`, so a transient transport error wrapped in an async task group
is still discovered. Each exception is yielded at most once.
"""
pending = [exc]
seen: set[int] = set()
while pending:
current = pending.pop()
if id(current) in seen:
continue
seen.add(id(current))
yield current
if isinstance(current, BaseExceptionGroup):
pending.extend(current.exceptions)
if current.__cause__ is not None:
pending.append(current.__cause__)
elif current.__context__ is not None:
pending.append(current.__context__)
def _is_transient_grader_transport_error(exc: BaseException) -> bool:
"""Return whether a grader failure is a retryable transport/read error.
Matches response-read faults (`httpx`/`httpcore` `ReadError`) and
response-framing faults (`RemoteProtocolError`, aiohttp
`TransferEncodingError`). Connect/timeout errors are intentionally excluded
so only mid-response transport failures trigger the retry.
"""
for current in _exception_chain(exc):
if isinstance(current, (httpx.ReadError, httpx.RemoteProtocolError)):
return True
error_type = type(current)
if error_type.__module__.startswith("httpcore") and error_type.__name__ in {
"ReadError",
"RemoteProtocolError",
}:
return True
if (
error_type.__module__ == "aiohttp.http_exceptions"
and error_type.__name__ == "TransferEncodingError"
and "Not enough data to satisfy transfer length header" in str(current)
):
return True
return False
class ReliableRubricMiddleware(RubricMiddleware):
"""Retry one transient grader transport failure without rerunning agent work.
The retry re-invokes only the grader sub-agent, never the task agent, so it
relies on the grader's own tools being read-only/idempotent. A second
failure — transient or not — propagates to the base middleware, which
surfaces it as a `grader_error` result rather than a silent success.
"""
def _grade(self, state: RubricState, iteration: int) -> GraderResponse:
try:
return super()._grade(state, iteration)
except Exception as exc:
if not _is_transient_grader_transport_error(exc):
raise
logger.warning(
"Rubric grader transport failed; retrying grading once",
exc_info=True,
)
return super()._grade(state, iteration)
async def _agrade(self, state: RubricState, iteration: int) -> GraderResponse:
try:
return await super()._agrade(state, iteration)
except Exception as exc:
if not _is_transient_grader_transport_error(exc):
raise
logger.warning(
"Rubric grader transport failed; retrying grading once",
exc_info=True,
)
return await super()._agrade(state, iteration)
libs/code/deepagents_code/tui/textual_adapter.py
+37 -11
View File
@@ -223,9 +223,15 @@ def _is_summarization_chunk(metadata: dict | None) -> bool:
return metadata.get("lc_source") == "summarization"
def _format_rubric_event(data: dict[str, Any]) -> str | None:
def _format_rubric_event(
data: dict[str, Any], *, goal_active: bool = False
) -> str | None:
"""Format a rubric custom-stream event for the chat transcript.
Args:
data: Custom-stream rubric event payload.
goal_active: Whether the rubric belongs to an unfinished `/goal`.
Returns:
A user-visible message for rubric events, or `None` for custom-stream
events that are not rubric events.
@@ -264,16 +270,29 @@ def _format_rubric_event(data: dict[str, Any]) -> str | None:
lines.append(f" {glyphs.error} {name}" + (f" — {gap}" if gap else ""))
return "\n".join(lines)
if result == "max_iterations_reached":
return (
f"{glyphs.warning} Acceptance criteria not satisfied "
"(iteration limit reached)"
)
if result in {"failed", "grader_error"}:
label = "grader failed" if result == "failed" else "grader error"
return (
f"{glyphs.warning} Rubric "
+ label
lines = [
f"{glyphs.warning} Iteration limit reached with unmet acceptance criteria"
+ (f": {explanation}" if explanation else "")
]
for criterion in data.get("criteria", []):
if isinstance(criterion, dict) and criterion.get("passed") is False:
name = str(criterion.get("name", "criterion"))
gap = str(criterion.get("gap", "")).strip()
lines.append(f" {glyphs.error} {name}" + (f" — {gap}" if gap else ""))
if goal_active:
lines.append(
"The goal remains active. Continue with another prompt to resume or "
"retry, use `/goal <objective>` to amend it, or `/goal clear` to "
"clear it."
)
return "\n".join(lines)
if result == "failed":
return f"{glyphs.warning} Rubric grader failed" + (
f": {explanation}" if explanation else ""
)
if result == "grader_error":
return f"{glyphs.warning} Grader/infrastructure failure" + (
f": {explanation}" if explanation else ""
)
# A `rubric_evaluation_end` with an unrecognized result is still a terminal
# grading event; surface it rather than silently dropping it (e.g. if the
@@ -513,6 +532,7 @@ async def execute_task_textual(
sandbox_type: str | None = None,
message_kwargs: dict[str, Any] | None = None,
rubric: str | None = None,
goal_active: bool = False,
blocked_goal_retry_context: str | None = None,
turn_stats: SessionStats | None = None,
) -> SessionStats:
@@ -541,6 +561,7 @@ async def execute_task_textual(
in the checkpoint).
rubric: Acceptance criteria supplied to `RubricMiddleware` via graph
input state.
goal_active: Whether the rubric belongs to an unfinished `/goal`.
blocked_goal_retry_context: One-turn model context for retrying a
previously blocked goal. This is carried via runtime context so it
is not parsed for file mentions or checkpointed as human input.
@@ -802,7 +823,12 @@ async def execute_task_textual(
if current_stream_mode == "custom":
rubric_message = data if isinstance(data, dict) else None
formatted_rubric_event = (
_format_rubric_event(rubric_message) if rubric_message else None
_format_rubric_event(
rubric_message,
goal_active=goal_active,
)
if rubric_message
else None
)
if formatted_rubric_event is not None and is_main_agent:
await adapter._mount_message(AppMessage(formatted_rubric_event))
libs/code/tests/unit_tests/test_agent.py
+1 -1
View File
@@ -3554,7 +3554,7 @@ class TestCreateCliAgentInterpreterWiring:
patch("deepagents_code.agent.settings", mock_settings),
patch("deepagents_code.agent.SkillsMiddleware"),
patch("deepagents_code.agent.MemoryMiddleware"),
patch("deepagents_code.agent.RubricMiddleware") as mock_rubric,
patch("deepagents_code.agent.ReliableRubricMiddleware") as mock_rubric,
patch(
"deepagents_code.agent.create_deep_agent",
return_value=mock_agent,
libs/code/tests/unit_tests/test_app.py
+179 -9
View File
@@ -6062,6 +6062,7 @@ class TestGoalCommand:
app = DeepAgentsApp(agent=MagicMock())
async with app.run_test() as pilot:
await pilot.pause()
app._pending_goal_completion_note = "evidence for the previous goal"
app._pending_goal_objective = "add refresh tokens"
app._pending_goal_rubric = "- tests pass"
request = AsyncMock(
@@ -6080,6 +6081,7 @@ class TestGoalCommand:
assert app._active_goal == "add refresh tokens"
assert app._goal_status == "active"
assert app._active_rubric == "- tests pass"
assert app._pending_goal_completion_note is None
assert app._pending_goal_objective is None
assert app._pending_goal_rubric is None
assert app._status_bar is not None
@@ -6470,9 +6472,9 @@ class TestGoalCommand:
fetch = AsyncMock(
return_value={
"_goal_objective": "add refresh tokens",
"_goal_status": "complete",
"_goal_status": "blocked",
"_goal_rubric": "- tests pass",
"_goal_status_note": "tests pass",
"_goal_status_note": "waiting for credentials",
}
)
with patch.object(app, "_get_thread_state_values", fetch):
@@ -6480,8 +6482,8 @@ class TestGoalCommand:
fetch.assert_awaited_once_with("thread-1")
assert app._active_goal == "add refresh tokens"
assert app._goal_status == "complete"
assert app._goal_status_note == "tests pass"
assert app._goal_status == "blocked"
assert app._goal_status_note == "waiting for credentials"
assert app._active_rubric == "- tests pass"
async def test_sync_goal_completion_auto_commits_after_rubric_satisfied(
@@ -6512,13 +6514,177 @@ class TestGoalCommand:
with patch.object(app, "_get_thread_state_values", fetch):
await app._sync_goal_rubric_state_from_thread()
assert app._goal_status == "complete"
assert app._goal_status_note == "tests pass"
assert app._active_goal is None
assert app._goal_status is None
assert app._goal_status_note is None
assert app._active_rubric is None
assert app._pending_goal_completion_note is None
assert updater.aupdate_state.await_args is not None
state_update = updater.aupdate_state.await_args.args[1]
assert state_update["_goal_status"] == "complete"
assert state_update["_pending_goal_completion_note"] is None
assert state_update == {
"rubric": None,
"_sticky_rubric": None,
"_goal_objective": None,
"_goal_status": None,
"_goal_rubric": None,
"_goal_status_note": None,
"_pending_goal_completion_note": None,
"_pending_goal_objective": None,
"_pending_goal_rubric": None,
}
async def test_completed_goal_is_cleared_before_next_turn(self) -> None:
app = DeepAgentsApp(agent=MagicMock(), auto_approve=True)
async with app.run_test() as pilot:
await pilot.pause()
app._active_goal = "add refresh tokens"
app._goal_status = "active"
app._active_rubric = "- tests pass"
await app._commit_pending_goal_completion(
"all acceptance tests pass",
previous_status="active",
)
assert app._active_goal is None
assert app._active_rubric is None
assert app._status_bar is not None
assert app._status_bar.rubric_label == ""
with (
patch(
"deepagents_code.tui.textual_adapter.execute_task_textual",
new=AsyncMock(return_value=SessionStats()),
) as execute,
patch.object(
app,
"_sync_goal_rubric_state_from_thread",
new=AsyncMock(),
),
):
await app._run_agent_task("start something else")
assert execute.await_args is not None
assert execute.await_args.kwargs["rubric"] is None
assert execute.await_args.kwargs["goal_active"] is False
await app._show_goal_state()
await pilot.pause()
rendered = "\n".join(str(w._content) for w in app.query(AppMessage))
assert "No goal set." in rendered
assert "Last completed goal:" not in rendered
async def test_failed_completion_persistence_restores_retryable_goal(self) -> None:
"""A failed clear write must leave local state aligned for retry."""
app = DeepAgentsApp(agent=MagicMock(), auto_approve=True)
async with app.run_test() as pilot:
await pilot.pause()
app._agent = SimpleNamespace(
aupdate_state=AsyncMock(side_effect=RuntimeError("down"))
)
app._lc_thread_id = "thread-1"
app._active_goal = "add refresh tokens"
app._goal_status = "active"
app._active_rubric = "- tests pass"
app._pending_goal_completion_note = "all acceptance tests pass"
committed = await app._resolve_pending_goal_completion(
rubric_status="satisfied",
previous_status="active",
)
await pilot.pause()
assert committed is False
assert app._active_goal == "add refresh tokens"
assert app._goal_status == "active"
assert app._active_rubric == "- tests pass"
assert app._pending_goal_completion_note == "all acceptance tests pass"
rendered = "\n".join(str(w._content) for w in app.query(AppMessage))
assert "Goal marked complete by the agent." not in rendered
errors = "\n".join(str(w._content) for w in app.query(ErrorMessage))
assert "goal remains active" in errors
assert "completion request is still pending" in errors
async def test_grader_error_keeps_goal_and_completion_request_active(self) -> None:
app = DeepAgentsApp(agent=MagicMock(), auto_approve=True)
async with app.run_test() as pilot:
await pilot.pause()
app._active_goal = "add refresh tokens"
app._goal_status = "active"
app._active_rubric = "- tests pass"
app._pending_goal_completion_note = "tests pass"
committed = await app._resolve_pending_goal_completion(
rubric_status="grader_error",
previous_status="active",
)
await pilot.pause()
assert committed is False
assert app._active_goal == "add refresh tokens"
assert app._goal_status == "active"
assert app._active_rubric == "- tests pass"
assert app._pending_goal_completion_note == "tests pass"
rendered = "\n".join(str(w._content) for w in app.query(ErrorMessage))
assert "grader or infrastructure error" in rendered
assert "completion request is still pending" in rendered
assert "rubric was not satisfied" not in rendered
async def test_iteration_cap_keeps_goal_active_and_does_not_complete(self) -> None:
app = DeepAgentsApp(agent=MagicMock(), auto_approve=True)
async with app.run_test() as pilot:
await pilot.pause()
app._active_goal = "add refresh tokens"
app._goal_status = "active"
app._active_rubric = "- tests pass"
app._pending_goal_completion_note = "tests pass"
committed = await app._resolve_pending_goal_completion(
rubric_status="max_iterations_reached",
previous_status="active",
)
await app._show_goal_state()
await pilot.pause()
assert committed is False
assert app._active_goal == "add refresh tokens"
assert app._goal_status == "active"
assert app._active_rubric == "- tests pass"
assert app._pending_goal_completion_note is None
rendered = "\n".join(str(w._content) for w in app.query(AppMessage))
assert "iteration limit was reached with unmet criteria" in rendered
assert (
"remains active for resume, amendment, retry, or clearing" in rendered
)
assert "Goal marked complete" not in rendered
assert "Commands:\n/goal clear\n/goal show" in rendered
async def test_grader_failed_keeps_goal_active_with_evaluation_error(self) -> None:
"""A `failed` grade must keep the goal active and blame the grader."""
app = DeepAgentsApp(agent=MagicMock(), auto_approve=True)
async with app.run_test() as pilot:
await pilot.pause()
app._active_goal = "add refresh tokens"
app._goal_status = "active"
app._active_rubric = "- tests pass"
app._pending_goal_completion_note = "tests pass"
committed = await app._resolve_pending_goal_completion(
rubric_status="failed",
previous_status="active",
)
await pilot.pause()
assert committed is False
assert app._active_goal == "add refresh tokens"
assert app._goal_status == "active"
assert app._active_rubric == "- tests pass"
assert app._pending_goal_completion_note is None
rendered = "\n".join(str(w._content) for w in app.query(AppMessage))
assert "grader could not evaluate the rubric" in rendered
assert "The goal remains active." in rendered
assert "Goal marked complete" not in rendered
assert "rubric was not satisfied" not in rendered
async def test_sync_goal_completion_rejects_when_rubric_not_satisfied(
self,
@@ -6593,7 +6759,9 @@ class TestGoalCommand:
action_requests = request_approval.await_args.args[0]
assert action_requests[0]["name"] == "update_goal"
assert action_requests[0]["args"]["status"] == "complete"
assert app._goal_status == "complete"
assert app._active_goal is None
assert app._goal_status is None
assert app._active_rubric is None
assert app._pending_goal_completion_note is None
async def test_sync_goal_rubric_state_drops_unknown_status(self) -> None:
@@ -6798,6 +6966,8 @@ class TestGoalCommand:
rendered = "\n".join(str(w._content) for w in app.query(AppMessage))
assert "Goal marked blocked by the agent." in rendered
assert "missing staging credentials" in rendered
assert app._active_goal == "ship the feature"
assert app._goal_status == "blocked"
async def test_announce_goal_status_no_message_when_unchanged(self) -> None:
"""A status equal to the previous one must not re-announce."""
libs/code/tests/unit_tests/test_reliable_rubric.py
+167
View File
@@ -0,0 +1,167 @@
"""Tests for transient rubric grader transport retries."""
from typing import Any, cast
from unittest.mock import AsyncMock, MagicMock
import httpx
import pytest
from deepagents.middleware.rubric import GraderResponse, RubricState
from langchain_core.messages import AIMessage, HumanMessage
from deepagents_code.reliable_rubric import (
ReliableRubricMiddleware,
_is_transient_grader_transport_error,
)
def _read_error() -> httpx.ReadError:
return httpx.ReadError(
"connection closed while reading",
request=httpx.Request("POST", "https://grader.test"),
)
def _typed_error(module: str, name: str, message: str = "boom") -> Exception:
"""Build an exception whose type mimics an external library's error class."""
error_type = type(name, (Exception,), {"__module__": module})
return error_type(message)
def _state() -> RubricState:
return cast(
"RubricState",
{
"rubric": "tests pass",
"messages": [
HumanMessage(content="implement it"),
AIMessage(content="implementation complete"),
],
},
)
def _satisfied_result() -> dict[str, Any]:
return {
"structured_response": GraderResponse(
result="satisfied",
explanation="all checks pass",
criteria=[],
)
}
class TestTransientGraderTransportClassification:
def test_read_error_is_transient(self) -> None:
assert _is_transient_grader_transport_error(_read_error()) is True
def test_remote_protocol_error_is_transient(self) -> None:
assert (
_is_transient_grader_transport_error(httpx.RemoteProtocolError("boom"))
is True
)
def test_httpcore_read_error_is_transient(self) -> None:
# httpcore errors cannot be caught via a stable isinstance, so the
# classifier matches them by module/name; exercise that path directly.
error = _typed_error("httpcore", "ReadError")
assert _is_transient_grader_transport_error(error) is True
def test_httpcore_remote_protocol_error_is_transient(self) -> None:
error = _typed_error("httpcore._exceptions", "RemoteProtocolError")
assert _is_transient_grader_transport_error(error) is True
def test_read_error_in_exception_group_is_transient(self) -> None:
group = ExceptionGroup(
"grading failed", [ValueError("unrelated"), _read_error()]
)
assert _is_transient_grader_transport_error(group) is True
def test_read_error_in_context_chain_is_transient(self) -> None:
wrapper = RuntimeError("grader request failed")
wrapper.__context__ = _read_error()
assert _is_transient_grader_transport_error(wrapper) is True
def test_transfer_encoding_error_in_cause_chain_is_transient(self) -> None:
error_type = type(
"TransferEncodingError",
(Exception,),
{"__module__": "aiohttp.http_exceptions"},
)
cause = error_type("Not enough data to satisfy transfer length header")
wrapper = RuntimeError("grader request failed")
wrapper.__cause__ = cause
assert _is_transient_grader_transport_error(wrapper) is True
def test_unrelated_exception_is_not_transient(self) -> None:
assert _is_transient_grader_transport_error(RuntimeError("bug")) is False
class TestReliableRubricMiddleware:
async def test_retries_only_grading_without_mutating_agent_transcript(self) -> None:
middleware = ReliableRubricMiddleware(model="fake-model")
error = httpx.ReadError(
"connection closed while reading",
request=httpx.Request("POST", "https://grader.test"),
)
grader = AsyncMock()
grader.ainvoke.side_effect = [error, _satisfied_result()]
middleware._grader = grader
state = _state()
messages_before = list(state["messages"])
result = await middleware._agrade(state, 0)
assert result.result == "satisfied"
assert grader.ainvoke.await_count == 2
assert state["messages"] == messages_before
async def test_does_not_retry_unrelated_exception(self) -> None:
middleware = ReliableRubricMiddleware(model="fake-model")
grader = AsyncMock()
grader.ainvoke.side_effect = RuntimeError("programming error")
middleware._grader = grader
with pytest.raises(RuntimeError, match="programming error"):
await middleware._agrade(_state(), 0)
grader.ainvoke.assert_awaited_once()
def test_sync_grade_retries_transient_transport_failure(self) -> None:
middleware = ReliableRubricMiddleware(model="fake-model")
grader = MagicMock()
grader.invoke.side_effect = [_read_error(), _satisfied_result()]
middleware._grader = grader
result = middleware._grade(_state(), 0)
assert result.result == "satisfied"
assert grader.invoke.call_count == 2
async def test_second_transient_failure_propagates_async(self) -> None:
# The retry is bounded to one attempt: a second transient failure must
# surface so the base middleware can report it as a grader_error.
middleware = ReliableRubricMiddleware(model="fake-model")
grader = AsyncMock()
grader.ainvoke.side_effect = [_read_error(), _read_error()]
middleware._grader = grader
with pytest.raises(httpx.ReadError):
await middleware._agrade(_state(), 0)
assert grader.ainvoke.await_count == 2
def test_second_transient_failure_propagates_sync(self) -> None:
middleware = ReliableRubricMiddleware(model="fake-model")
grader = MagicMock()
grader.invoke.side_effect = [_read_error(), _read_error()]
middleware._grader = grader
with pytest.raises(httpx.ReadError):
middleware._grade(_state(), 0)
assert grader.invoke.call_count == 2
libs/code/tests/unit_tests/test_rubric_cli.py
+2 -2
View File
@@ -308,7 +308,7 @@ class TestProcessRubricEvent:
out = _render_event(
{"type": "rubric_evaluation_end", "result": "max_iterations_reached"}
)
assert "iteration limit reached" in out
assert "Iteration limit reached with unmet acceptance criteria" in out
def test_failed(self) -> None:
out = _render_event(
@@ -331,7 +331,7 @@ class TestProcessRubricEvent:
"explanation": "provider 500",
}
)
assert "grader error" in out
assert "Grader/infrastructure failure" in out
assert "provider 500" in out
def test_unrecognized_terminal_result_surfaced(self) -> None:
libs/code/tests/unit_tests/tui/test_textual_adapter.py
+21 -4
View File
@@ -1298,12 +1298,29 @@ class TestFormatRubricEvent:
)
def test_max_iterations_reached_event(self) -> None:
"""Hitting the iteration cap should warn the user it is unsatisfied."""
"""Hitting the iteration cap should show gaps and preserve the goal."""
assert (
_format_rubric_event(
{"type": "rubric_evaluation_end", "result": "max_iterations_reached"},
{
"type": "rubric_evaluation_end",
"result": "max_iterations_reached",
"explanation": "coverage is still missing",
"criteria": [
{
"name": "tests pass",
"passed": False,
"gap": "integration test failed",
},
{"name": "docs updated", "passed": True},
],
},
goal_active=True,
)
== "⚠ Acceptance criteria not satisfied (iteration limit reached)"
== "⚠ Iteration limit reached with unmet acceptance criteria: "
"coverage is still missing\n"
" ✗ tests pass — integration test failed\n"
"The goal remains active. Continue with another prompt to resume or "
"retry, use `/goal <objective>` to amend it, or `/goal clear` to clear it."
)
def test_grader_failure_results_render_warning(self) -> None:
@@ -1322,7 +1339,7 @@ class TestFormatRubricEvent:
_format_rubric_event(
{"type": "rubric_evaluation_end", "result": "grader_error"},
)
== "⚠ Rubric grader error"
== "⚠ Grader/infrastructure failure"
)
def test_unknown_terminal_result_renders_fallback(self) -> None: