mirror of
https://github.com/esengine/DeepSeek-Reasonix.git
synced 2026-11-07 10:45:59 -10:00
refactor(tools,plan): complete_step takes no evidence; drop conclude_blocked and plan criteria
complete_step now only says a step is done: it names the step and what is now true, and the host advances the task list as before. It no longer asks for commands, files or criteria to cite and no longer checks them against the session's receipts. conclude_blocked existed to let the readiness gate through and goes with it; the prompts that pointed at it now ask for a sentence in the reply. Plan steps lose their acceptance and verification fields: the submit_plan schema, the planner and executor prompts, the rendering and the re-approval rule all stop mentioning them, and plans saved with them still load. The executor's no-action nudge now keys on a plan naming files only.
This commit is contained in:
1 parent
3d0fe85e7e
commit
5766a8ebeb
57 files changed
+96
-3368
No files matched your search
@@ -44,7 +44,7 @@ Boot 效果测试覆盖关闭时缺席、启用时精确的 schema、读取拒
|
||||
| `browser_open` | false | Load a page and return its snapshot, one element per line with [eN] refs. url: http(s), about:blank or a workspace file; no url switches to tab; close closes. The user sees tabs: web_fetch reads static text, close tabs when done. |
|
||||
| `browser_read` | true | Read a browser page: snapshot (default; page with offset/limit, or ref for one subtree), find (lines carrying text, with their refs), screenshot (browser_act x/y use its pixels, and the person is shown it too), logs (errors since you last read them), network (what it requested and what came back; text narrows by URL), tabs. |
|
||||
| `code_index` | true | Lightweight built-in code symbol index. Prefer lsp_* for language semantics and installed code graph MCP tools for call graph, impact, and architecture relationships; use this as the local fallback for file outlines and symbol definition candidates, then verify with read_file or grep. |
|
||||
| `complete_step` | true | Record the evidence-backed completion of ONE step of an approved plan. Call it as you finish each step instead of silently moving on: a completion with no evidence is REJECTED, so don't claim a step is done until you can show why. The host advances the list for you — it marks this step completed and moves the next to in_progress, so a todo_write that marks such an item completed on its own is REJECTED too. Sign off one step per tool-call ROUND, not one per call: the sign-off promotes the next item, and a second sign-off sent beside it signs against the list as it stood before. |
|
||||
| `complete_step` | true | Mark ONE step of an approved plan as done. Call it as you finish each step. The host advances the list for you: it marks this step completed and moves the next to in_progress, so a todo_write that marks such an item completed on its own is REJECTED. Sign off one step per tool-call ROUND, not one per call: the sign-off promotes the next item, and a second sign-off sent beside it signs against the list as it stood before. |
|
||||
| `compress` | true | Compress a selected part of the current model-visible conversation without deleting visible history. Use only when the user explicitly asks for context compression. Choose `before` to summarize everything before the uniquely matched user turn while keeping that turn and later context, or `after` to summarize from that turn through the last completed turn while keeping the active turn. The anchor must be an exact, unique excerpt from a real user message; use a longer excerpt if the tool reports multiple matches. |
|
||||
| `computer_act` | false | Operate an approved application through CUA Driver. launch must be one separate step using an exact installed identity from computer_read what=apps; no window selector or extra arguments. Select window when the app has multiple windows. Use refs from its latest snapshot or coordinates from its latest screenshot. Background clicks, set_value, type/paste, key and scroll never silently escalate. pointer_click explicitly takes foreground control and requires separate approval. wait is bounded. Inspect fresh state after an unverified effect; never assume an action landed. Returns a snapshot afterwards except launch, which returns the verified running identity for a separately approved read. |
|
||||
| `computer_read` | false | See an approved application listed by what=apps. Installed entries without a targetable window provide an exact launch identity for a separate computer_act launch step. Select window by id when it has multiple windows. snapshot returns session-bound element refs in brackets for computer_act. screenshot captures that exact window; computer_act x/y use pixels of this latest displayed image. Old tokens and captures expire on refresh or reconnect. |
|
||||
@@ -156,8 +156,6 @@ The host-control tools ride the same surface at every setting: `todo_write` and
|
||||
|
||||
`setting` finds the user's settings and asks the user to confirm a change. A refusal that a setting would lift ends with `(fix: setting ...)`.
|
||||
|
||||
`conclude_blocked` is registered but not yet on this surface, so it is reachable only through `use_capability` even where the host names it as a way out of an unfinished turn.
|
||||
|
||||
Optional tools (`glob`, `grep`, `ls`, `web_fetch`, MCP, skills, subagents, docs, session history, memory mutation, workflow, and so on) remain in the host registry for dispatch. The model lists, inspects, calls, or declines them via `use_capability` without changing the provider tool list.
|
||||
|
||||
Execution settings change host planning / verification / review policy, not which tools appear on the provider-visible surface. The retired `connect_tool_source` path is no longer registered.
|
||||
@@ -97,7 +97,7 @@ func budgetFixtureAgent(t *testing.T, kind, models, overrides string, sink event
|
||||
isolateConfigHome(t)
|
||||
dir := robustTempDir(t)
|
||||
t.Chdir(dir)
|
||||
rec := &budgetEffectProvider{bulk: strings.Repeat("work output line with detail. ", 400)}
|
||||
rec := &budgetEffectProvider{bulk: strings.Repeat("work output line with detail. ", 380)}
|
||||
provider.Register(kind, func(provider.Config) (provider.Provider, error) { return rec, nil })
|
||||
writeFile(t, dir, "reasonix.toml", `
|
||||
default_model = "conn/`+strings.Split(models, ",")[0]+`"
|
||||
|
||||
@@ -135,13 +135,13 @@ func TestEffectDisabledToolTheHostNeedsStaysAndSaysSo(t *testing.T) {
|
||||
// The tools the loop itself reads stay on when a project file names them, and
|
||||
// the user is told which feature needs each.
|
||||
func TestEffectDisabledToolsTheLoopReadsStayAndSaySo(t *testing.T) {
|
||||
kept := []string{"complete_step", "todo_write", "await_user", "conclude_blocked"}
|
||||
rec, sink := disabledToolsBuild(t, "[tools]\ndisabled = [\"complete_step\", \"todo_write\", \"await_user\", \"conclude_blocked\", \"recall\"]\n", "bash")
|
||||
kept := []string{"complete_step", "todo_write", "await_user"}
|
||||
rec, sink := disabledToolsBuild(t, "[tools]\ndisabled = [\"complete_step\", \"todo_write\", \"await_user\", \"recall\"]\n", "bash")
|
||||
rec.mu.Lock()
|
||||
defer rec.mu.Unlock()
|
||||
names := requestToolNames(rec.reqs[0])
|
||||
for _, name := range kept {
|
||||
if name != "conclude_blocked" && !slices.Contains(names, name) {
|
||||
if !slices.Contains(names, name) {
|
||||
t.Fatalf("request tools = %v, want %s kept", names, name)
|
||||
}
|
||||
if !sink.has(func(ev event.Event) bool {
|
||||
|
||||
@@ -30,7 +30,7 @@ import (
|
||||
// observeCeiling is the whole tool surface of the read-only posture. A tool
|
||||
// added to the tree is absent from it until it declares a reach, and one that
|
||||
// declares a reach shows up here as a failing diff to be reviewed.
|
||||
var observeCeiling = []string{"ask", "code_index", "conclude_blocked", "glob", "grep", "ls", "read_file"}
|
||||
var observeCeiling = []string{"ask", "code_index", "glob", "grep", "ls", "read_file"}
|
||||
|
||||
func observeProject(t *testing.T) string {
|
||||
t.Helper()
|
||||
@@ -632,7 +632,7 @@ func TestEffectObserveGrantNarrowsReadToolsButKeepsHostControl(t *testing.T) {
|
||||
if err := ctrl.Run(context.Background(), "look"); err != nil {
|
||||
t.Fatalf("Run: %v", err)
|
||||
}
|
||||
want := []string{"ask", "conclude_blocked", "read_file"}
|
||||
want := []string{"ask", "read_file"}
|
||||
if got := toolNameList(agentRequests(prov.Requests())[0]); !slices.Equal(got, want) {
|
||||
t.Fatalf("offered %v, want %v: a name the grant lists but the ceiling refuses (bash) must not appear", got, want)
|
||||
}
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"SystemHash": "bc08dbc97c837bba",
|
||||
"ToolsHash": "114e6a978d799611",
|
||||
"PrefixHash": "dc48c1adc826bea4",
|
||||
"ToolsHash": "91b5dbe7076f799c",
|
||||
"PrefixHash": "3a17832abcbd93b5",
|
||||
"LogRewriteVersion": 0,
|
||||
"ToolSchemaTokens": 5964
|
||||
"ToolSchemaTokens": 5538
|
||||
}
|
||||
@@ -101,7 +101,7 @@
|
||||
},
|
||||
{
|
||||
"name": "await_user",
|
||||
"description": "Hand control back to the user with the task list still open, when what the current item needs next is something only they can give: their view on what you just showed them, the material they said they would add, the judgement the list exists to collect. The list is kept as it stands and the turn ends; their next message resumes it, and the host will not continue the list on its own in the meantime. This is not `ask`, which offers options you wrote and answers inside the same turn, and not `conclude_blocked`, which says the work cannot be done at all. When what you wait for is a pick among choices you can name — which way to proceed, which option to take — call `ask` instead: the user answers it with one click and you carry on in this turn. Name the waiting item with `step_id`, and put in `need` one short line saying what you are waiting for, addressed to them. Say the substance — the item you are presenting, the question you want answered — in your own reply; this call records the wait, it does not speak for you.",
|
||||
"description": "Hand control back to the user with the task list still open, when what the current item needs next is something only they can give: their view on what you just showed them, the material they said they would add, the judgement the list exists to collect. The list is kept as it stands and the turn ends; their next message resumes it, and the host will not continue the list on its own in the meantime. This is not `ask`, which offers options you wrote and answers inside the same turn. When what you wait for is a pick among choices you can name — which way to proceed, which option to take — call `ask` instead: the user answers it with one click and you carry on in this turn. Name the waiting item with `step_id`, and put in `need` one short line saying what you are waiting for, addressed to them. Say the substance — the item you are presenting, the question you want answered — in your own reply; this call records the wait, it does not speak for you.",
|
||||
"parameters": {
|
||||
"properties": {
|
||||
"need": {
|
||||
@@ -341,53 +341,9 @@
|
||||
},
|
||||
{
|
||||
"name": "complete_step",
|
||||
"description": "Record the evidence-backed completion of ONE step of an approved plan. Call it as you finish each step instead of silently moving on: a completion with no evidence is REJECTED, so don't claim a step is done until you can show why. The host advances the list for you — it marks this step completed and moves the next to in_progress, so a todo_write that marks such an item completed on its own is REJECTED too. Sign off one step per tool-call ROUND, not one per call: the sign-off promotes the next item, and a second sign-off sent beside it signs against the list as it stood before.",
|
||||
"description": "Mark ONE step of an approved plan as done. Call it as you finish each step. The host advances the list for you: it marks this step completed and moves the next to in_progress, so a todo_write that marks such an item completed on its own is REJECTED. Sign off one step per tool-call ROUND, not one per call: the sign-off promotes the next item, and a second sign-off sent beside it signs against the list as it stood before.",
|
||||
"parameters": {
|
||||
"properties": {
|
||||
"evidence": {
|
||||
"description": "Proof the step is done. At least one item is required.",
|
||||
"items": {
|
||||
"properties": {
|
||||
"command": {
|
||||
"description": "REQUIRED for verification evidence: the command as it actually ran (e.g. \"go test ./...\") — it is checked against this session's real command history.",
|
||||
"type": "string"
|
||||
},
|
||||
"criterion_id": {
|
||||
"description": "The acceptance criterion this proof satisfies, as the plan renders it (e.g. \"c2\" from \"accept [c2]: ...\"). Cite it whenever the step has criteria: a command succeeding is not the same as a criterion being met, and the host records the proof against the criterion you name.",
|
||||
"type": "string"
|
||||
},
|
||||
"kind": {
|
||||
"description": "verification = a command/test was run (command REQUIRED); review = a built-in review run completed and, after changes, inspected the latest changed result (the verdict/findings still apply separately); diff = a concrete code change (paths REQUIRED); files = files created/edited/inspected (paths REQUIRED); manual = a manual check.",
|
||||
"enum": [
|
||||
"verification",
|
||||
"review",
|
||||
"diff",
|
||||
"files",
|
||||
"manual"
|
||||
],
|
||||
"type": "string"
|
||||
},
|
||||
"paths": {
|
||||
"description": "REQUIRED for diff/files evidence: the files this evidence refers to, as the paths were passed to the tools that touched them.",
|
||||
"items": {
|
||||
"type": "string"
|
||||
},
|
||||
"type": "array"
|
||||
},
|
||||
"summary": {
|
||||
"description": "The evidence itself: the test result, what the diff does, or what was confirmed.",
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"kind",
|
||||
"summary"
|
||||
],
|
||||
"type": "object"
|
||||
},
|
||||
"minItems": 1,
|
||||
"type": "array"
|
||||
},
|
||||
"notes": {
|
||||
"description": "Optional caveats, follow-ups, or anything deferred.",
|
||||
"type": "string"
|
||||
@@ -411,7 +367,6 @@
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"evidence",
|
||||
"result"
|
||||
],
|
||||
"type": "object"
|
||||
|
||||
+2
-47
@@ -72,7 +72,7 @@
|
||||
},
|
||||
{
|
||||
"Name": "await_user",
|
||||
"Description": "Hand control back to the user with the task list still open, when what the current item needs next is something only they can give: their view on what you just showed them, the material they said they would add, the judgement the list exists to collect. The list is kept as it stands and the turn ends; their next message resumes it, and the host will not continue the list on its own in the meantime. This is not `ask`, which offers options you wrote and answers inside the same turn, and not `conclude_blocked`, which says the work cannot be done at all. When what you wait for is a pick among choices you can name — which way to proceed, which option to take — call `ask` instead: the user answers it with one click and you carry on in this turn. Name the waiting item with `step_id`, and put in `need` one short line saying what you are waiting for, addressed to them. Say the substance — the item you are presenting, the question you want answered — in your own reply; this call records the wait, it does not speak for you.",
|
||||
"Description": "Hand control back to the user with the task list still open, when what the current item needs next is something only they can give: their view on what you just showed them, the material they said they would add, the judgement the list exists to collect. The list is kept as it stands and the turn ends; their next message resumes it, and the host will not continue the list on its own in the meantime. This is not `ask`, which offers options you wrote and answers inside the same turn. When what you wait for is a pick among choices you can name — which way to proceed, which option to take — call `ask` instead: the user answers it with one click and you carry on in this turn. Name the waiting item with `step_id`, and put in `need` one short line saying what you are waiting for, addressed to them. Say the substance — the item you are presenting, the question you want answered — in your own reply; this call records the wait, it does not speak for you.",
|
||||
"ReadOnly": true,
|
||||
"Schema": {
|
||||
"properties": {
|
||||
@@ -318,54 +318,10 @@
|
||||
},
|
||||
{
|
||||
"Name": "complete_step",
|
||||
"Description": "Record the evidence-backed completion of ONE step of an approved plan. Call it as you finish each step instead of silently moving on: a completion with no evidence is REJECTED, so don't claim a step is done until you can show why. The host advances the list for you — it marks this step completed and moves the next to in_progress, so a todo_write that marks such an item completed on its own is REJECTED too. Sign off one step per tool-call ROUND, not one per call: the sign-off promotes the next item, and a second sign-off sent beside it signs against the list as it stood before.",
|
||||
"Description": "Mark ONE step of an approved plan as done. Call it as you finish each step. The host advances the list for you: it marks this step completed and moves the next to in_progress, so a todo_write that marks such an item completed on its own is REJECTED. Sign off one step per tool-call ROUND, not one per call: the sign-off promotes the next item, and a second sign-off sent beside it signs against the list as it stood before.",
|
||||
"ReadOnly": true,
|
||||
"Schema": {
|
||||
"properties": {
|
||||
"evidence": {
|
||||
"description": "Proof the step is done. At least one item is required.",
|
||||
"items": {
|
||||
"properties": {
|
||||
"command": {
|
||||
"description": "REQUIRED for verification evidence: the command as it actually ran (e.g. \"go test ./...\") — it is checked against this session's real command history.",
|
||||
"type": "string"
|
||||
},
|
||||
"criterion_id": {
|
||||
"description": "The acceptance criterion this proof satisfies, as the plan renders it (e.g. \"c2\" from \"accept [c2]: ...\"). Cite it whenever the step has criteria: a command succeeding is not the same as a criterion being met, and the host records the proof against the criterion you name.",
|
||||
"type": "string"
|
||||
},
|
||||
"kind": {
|
||||
"description": "verification = a command/test was run (command REQUIRED); review = a built-in review run completed and, after changes, inspected the latest changed result (the verdict/findings still apply separately); diff = a concrete code change (paths REQUIRED); files = files created/edited/inspected (paths REQUIRED); manual = a manual check.",
|
||||
"enum": [
|
||||
"verification",
|
||||
"review",
|
||||
"diff",
|
||||
"files",
|
||||
"manual"
|
||||
],
|
||||
"type": "string"
|
||||
},
|
||||
"paths": {
|
||||
"description": "REQUIRED for diff/files evidence: the files this evidence refers to, as the paths were passed to the tools that touched them.",
|
||||
"items": {
|
||||
"type": "string"
|
||||
},
|
||||
"type": "array"
|
||||
},
|
||||
"summary": {
|
||||
"description": "The evidence itself: the test result, what the diff does, or what was confirmed.",
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"kind",
|
||||
"summary"
|
||||
],
|
||||
"type": "object"
|
||||
},
|
||||
"minItems": 1,
|
||||
"type": "array"
|
||||
},
|
||||
"notes": {
|
||||
"description": "Optional caveats, follow-ups, or anything deferred.",
|
||||
"type": "string"
|
||||
@@ -389,7 +345,6 @@
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"evidence",
|
||||
"result"
|
||||
],
|
||||
"type": "object"
|
||||
|
||||
@@ -15,5 +15,4 @@ func addTurnExitTools(reg *tool.Registry) {
|
||||
// ask stays unresolved, await_user refuses.
|
||||
reg.Add(agent.NewAskTool())
|
||||
reg.Add(agent.NewAwaitUserTool())
|
||||
reg.Add(agent.NewConcludeBlockedTool())
|
||||
}
|
||||
@@ -19,7 +19,7 @@ import (
|
||||
"reasonix/internal/state/sessionstore"
|
||||
)
|
||||
|
||||
var observeCeilingTools = []string{"ask", "code_index", "conclude_blocked", "glob", "grep", "ls", "read_file"}
|
||||
var observeCeilingTools = []string{"ask", "code_index", "glob", "grep", "ls", "read_file"}
|
||||
|
||||
func TestScheduleExecWithoutASupervisorRefuses(t *testing.T) {
|
||||
isolateCLIConfigHome(t)
|
||||
|
||||
@@ -1023,11 +1023,11 @@ func (a *Agent) shouldNudgeExecutorHandoff() bool {
|
||||
}
|
||||
|
||||
// planContractExpectsAction reports whether carrying out the plan requires
|
||||
// touching the workspace: a step that names files or a verification command
|
||||
// cannot be delivered as prose alone.
|
||||
// touching the workspace: a step that names files cannot be
|
||||
// delivered as prose alone.
|
||||
func planContractExpectsAction(plan plancontract.Plan) bool {
|
||||
for _, step := range plan.Steps {
|
||||
if len(step.VerifiedFiles) > 0 || len(step.CandidateFiles) > 0 || len(step.Verification) > 0 {
|
||||
if len(step.VerifiedFiles) > 0 || len(step.CandidateFiles) > 0 {
|
||||
return true
|
||||
}
|
||||
}
|
||||
|
||||
@@ -131,7 +131,7 @@ func (*AskTool) Execute(ctx context.Context, args json.RawMessage) (string, erro
|
||||
// rule that puts a call here, so the decision stays unresolved.
|
||||
return "unresolved: this run has no interactive user, and the answer is the user's to give — " +
|
||||
"nothing here can supply it, including you. Do not choose on their behalf. Do whatever the task " +
|
||||
"allows without this decision, then call conclude_blocked naming the decision that is missing.", nil
|
||||
"allows without this decision, then say in your reply which decision is missing.", nil
|
||||
}
|
||||
|
||||
answers, err := asker.Ask(ctx, qs)
|
||||
@@ -203,5 +203,5 @@ func (e *AskTimedOutError) result() string {
|
||||
return fmt.Sprintf("skipped: the user did not answer within %d seconds, so the host skipped this question under their ask-timeout setting. "+
|
||||
"That is a timeout, not an answer: no option was chosen for them and none counts as their preference. "+
|
||||
"Carry on with whatever the task allows without this decision and state the assumption you proceed under in your reply, "+
|
||||
"or call conclude_blocked naming the decision that is missing.", int(e.After.Seconds()))
|
||||
"and say in your reply which decision is missing.", int(e.After.Seconds()))
|
||||
}
|
||||
@@ -185,7 +185,7 @@ func TestAskToolWithNoUserLeavesTheDecisionUnresolved(t *testing.T) {
|
||||
if err != nil {
|
||||
t.Fatalf("Execute: %v", err)
|
||||
}
|
||||
for _, want := range []string{"unresolved", "the user's to give", "Do not choose on their behalf", "conclude_blocked"} {
|
||||
for _, want := range []string{"unresolved", "the user's to give", "Do not choose on their behalf", "which decision is missing"} {
|
||||
if !strings.Contains(out, want) {
|
||||
t.Fatalf("headless answer = %q, want it to contain %q", out, want)
|
||||
}
|
||||
@@ -257,7 +257,7 @@ func TestAskToolTellsTheModelTheQuestionTimedOut(t *testing.T) {
|
||||
if err != nil {
|
||||
t.Fatalf("a timeout surfaced as a tool failure: %v", err)
|
||||
}
|
||||
for _, want := range []string{"skipped:", "90 seconds", "timeout, not an answer", "no option was chosen", "conclude_blocked"} {
|
||||
for _, want := range []string{"skipped:", "90 seconds", "timeout, not an answer", "no option was chosen", "which decision is missing"} {
|
||||
if !strings.Contains(out, want) {
|
||||
t.Fatalf("result lacks %q:\n%s", want, out)
|
||||
}
|
||||
|
||||
@@ -26,7 +26,7 @@ func NewAwaitUserTool() *AwaitUserTool { return &AwaitUserTool{} }
|
||||
func (*AwaitUserTool) Name() string { return "await_user" }
|
||||
|
||||
func (*AwaitUserTool) Description() string {
|
||||
return "Hand control back to the user with the task list still open, when what the current item needs next is something only they can give: their view on what you just showed them, the material they said they would add, the judgement the list exists to collect. The list is kept as it stands and the turn ends; their next message resumes it, and the host will not continue the list on its own in the meantime. This is not `ask`, which offers options you wrote and answers inside the same turn, and not `conclude_blocked`, which says the work cannot be done at all. When what you wait for is a pick among choices you can name — which way to proceed, which option to take — call `ask` instead: the user answers it with one click and you carry on in this turn. Name the waiting item with `step_id`, and put in `need` one short line saying what you are waiting for, addressed to them. Say the substance — the item you are presenting, the question you want answered — in your own reply; this call records the wait, it does not speak for you."
|
||||
return "Hand control back to the user with the task list still open, when what the current item needs next is something only they can give: their view on what you just showed them, the material they said they would add, the judgement the list exists to collect. The list is kept as it stands and the turn ends; their next message resumes it, and the host will not continue the list on its own in the meantime. This is not `ask`, which offers options you wrote and answers inside the same turn. When what you wait for is a pick among choices you can name — which way to proceed, which option to take — call `ask` instead: the user answers it with one click and you carry on in this turn. Name the waiting item with `step_id`, and put in `need` one short line saying what you are waiting for, addressed to them. Say the substance — the item you are presenting, the question you want answered — in your own reply; this call records the wait, it does not speak for you."
|
||||
}
|
||||
|
||||
func (*AwaitUserTool) Schema() json.RawMessage {
|
||||
@@ -68,7 +68,7 @@ func (*AwaitUserTool) Execute(ctx context.Context, args json.RawMessage) (string
|
||||
// No asker means no answer can arrive. The error records no gate, so the
|
||||
// turn's readiness is unchanged.
|
||||
if _, _, asker, ok := CallContext(ctx); !ok || asker == nil {
|
||||
return "", fmt.Errorf("await_user needs an interactive user and this run has none, so there is nobody whose answer would arrive; do whatever the list allows without them, then call conclude_blocked naming the input that is missing")
|
||||
return "", fmt.Errorf("await_user needs an interactive user and this run has none, so there is nobody whose answer would arrive; do whatever the list allows without them, then say in your reply which input is missing")
|
||||
}
|
||||
todos := awaitUserTodos(ctx)
|
||||
if len(todos) == 0 {
|
||||
|
||||
@@ -82,7 +82,7 @@ func TestAwaitUserRejectsWhatItCannotPark(t *testing.T) {
|
||||
// No asker, so no gate is recorded and readiness is unchanged.
|
||||
"headless run": {
|
||||
todos: newsList(), interactive: false,
|
||||
args: `{"need":"your view"}`, want: "conclude_blocked",
|
||||
args: `{"need":"your view"}`, want: "which input is missing",
|
||||
},
|
||||
} {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
|
||||
@@ -112,25 +112,6 @@ func TestE2ESerialPlanHostAdvancesAndAllowsFinalAnswer(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// A command cited with a different string than it ran under (#2917: the model
|
||||
// drops the cd-prefix) is still accepted via segment matching, in-turn.
|
||||
func TestE2ECommandDriftAcceptedInTurn(t *testing.T) {
|
||||
mp := testutil.NewMock("m",
|
||||
testutil.Turn{ToolCalls: []provider.ToolCall{{ID: "b1", Name: "bash",
|
||||
Arguments: `{"command":"cd /Users/x/repo && git merge upstream/main --ff-only"}`}}},
|
||||
testutil.Turn{ToolCalls: []provider.ToolCall{{ID: "c1", Name: "complete_step",
|
||||
Arguments: `{"step":"sync","result":"synced","evidence":[{"kind":"verification","summary":"fast-forwarded","command":"git merge upstream/main --ff-only"}]}`}}},
|
||||
testutil.Turn{Text: "synced"},
|
||||
)
|
||||
a := New(mp, evidenceRegistry(), sessionstore.NewSession("sys"), Options{}, event.Discard)
|
||||
if err := a.Run(context.Background(), "sync the branch"); err != nil {
|
||||
t.Fatalf("Run: %v", err)
|
||||
}
|
||||
if !sessionContains(a, "signed off") {
|
||||
t.Fatal("cd-prefixed command drift rejected a real verification")
|
||||
}
|
||||
}
|
||||
|
||||
func TestE2ECrossTurnPendingSignoffIsRejectedUntilCurrentAdvances(t *testing.T) {
|
||||
sess := sessionstore.NewSession("sys")
|
||||
sess.Add(provider.Message{Role: provider.RoleAssistant, ToolCalls: []provider.ToolCall{{
|
||||
@@ -190,27 +171,6 @@ func TestE2ECrossTurnDiffEvidenceViaSessionFallback(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// A diff citation for a file no turn ever wrote stays rejected — the session
|
||||
// fallback widens what counts as proof, it does not wave through fabrication.
|
||||
func TestE2EUnbackedDiffEvidenceStillRejected(t *testing.T) {
|
||||
mp := testutil.NewMock("m",
|
||||
testutil.Turn{ToolCalls: []provider.ToolCall{{ID: "c1", Name: "complete_step",
|
||||
Arguments: `{"step":"x","result":"y","evidence":[{"kind":"diff","summary":"claimed","paths":["never/written.go"]}]}`}}},
|
||||
testutil.Turn{Text: "done"},
|
||||
)
|
||||
a := New(mp, evidenceRegistry(), sessionstore.NewSession("sys"), Options{}, event.Discard)
|
||||
|
||||
if err := a.Run(context.Background(), "sign off without doing the work"); err != nil {
|
||||
t.Fatalf("Run: %v", err)
|
||||
}
|
||||
if !sessionContains(a, "no matching successful writer") {
|
||||
t.Fatal("a diff citation for a never-written file was accepted")
|
||||
}
|
||||
if sessionContains(a, "signed off") {
|
||||
t.Fatal("an unbacked diff citation was signed off")
|
||||
}
|
||||
}
|
||||
|
||||
// What the model acts on is the tool result on the transcript, not the host's
|
||||
// internal task state. #8816 looped precisely there: every step was completed,
|
||||
// the host knew it, and the last sign-off still said to continue — so the model
|
||||
|
||||
@@ -1,115 +0,0 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"reasonix/internal/contract/tool"
|
||||
"reasonix/internal/safety/evidence"
|
||||
)
|
||||
|
||||
// ConcludeBlockedTool is the execution turn's honest exit. The host otherwise
|
||||
// recognizes one way to finish — everything verified — so saying a task is
|
||||
// impossible was reported as a failed run while quietly faking the check was
|
||||
// not. The claim costs what success costs: every reference must match a receipt
|
||||
// from this turn, so "I cannot" is as investigated as "I did".
|
||||
type ConcludeBlockedTool struct{}
|
||||
|
||||
func NewConcludeBlockedTool() *ConcludeBlockedTool { return &ConcludeBlockedTool{} }
|
||||
|
||||
func (*ConcludeBlockedTool) Name() string { return "conclude_blocked" }
|
||||
|
||||
func (*ConcludeBlockedTool) Description() string {
|
||||
return "Declare that part of the task cannot be completed as specified, after you have established why. Use it when the work is genuinely impossible or needs a decision only the user can make — mutually exclusive requirements, a missing credential, an external service that is down — never as an exit from work that is merely hard, and never before you have done the part that is possible. Cite `evidence` the host can check against what you ran or read this turn; a claim with nothing behind it is rejected. Name the `criterion_id` when a specific acceptance criterion is the one that cannot be met. The turn ends reported as blocked, and `blocker` is what the user is told."
|
||||
}
|
||||
|
||||
func (*ConcludeBlockedTool) Schema() json.RawMessage {
|
||||
return json.RawMessage(`{
|
||||
"type":"object",
|
||||
"properties":{
|
||||
"blocker":{"type":"string","description":"What cannot be done and why it cannot. Shown to the user."},
|
||||
"criterion_id":{"type":"string","description":"Optional id of the acceptance criterion that cannot be met."},
|
||||
"evidence":{
|
||||
"type":"array",
|
||||
"minItems":1,
|
||||
"description":"What you ran or read that establishes the blocker. At least one must match a receipt from this turn.",
|
||||
"items":{
|
||||
"type":"object",
|
||||
"properties":{
|
||||
"command":{"type":"string","description":"A command you ran this turn, exactly as it ran. A command that failed is evidence too."},
|
||||
"paths":{"type":"array","items":{"type":"string"},"description":"Files you read or wrote this turn."}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"required":["blocker","evidence"]
|
||||
}`)
|
||||
}
|
||||
|
||||
// ReadOnly is true: the call records a conclusion and changes nothing. The
|
||||
// evidence it cites was produced by the work that came before it.
|
||||
func (*ConcludeBlockedTool) ReadOnly() bool { return true }
|
||||
|
||||
func (*ConcludeBlockedTool) Reach() tool.Reach { return tool.ReachHostControl }
|
||||
|
||||
type blockedEvidence struct {
|
||||
Command string `json:"command"`
|
||||
Paths []string `json:"paths"`
|
||||
}
|
||||
|
||||
func (*ConcludeBlockedTool) Execute(ctx context.Context, args json.RawMessage) (string, error) {
|
||||
var payload struct {
|
||||
Blocker string `json:"blocker"`
|
||||
CriterionID string `json:"criterion_id"`
|
||||
Evidence []blockedEvidence `json:"evidence"`
|
||||
}
|
||||
if err := json.Unmarshal(args, &payload); err != nil {
|
||||
return "", fmt.Errorf("invalid args: %w", err)
|
||||
}
|
||||
blocker := strings.TrimSpace(payload.Blocker)
|
||||
if blocker == "" {
|
||||
return "", fmt.Errorf("conclude_blocked needs a blocker: state what cannot be done and why")
|
||||
}
|
||||
if _, planning := planSubmissionFromContext(ctx); planning {
|
||||
return "", fmt.Errorf("conclude_blocked ends an execution turn, not a planning one; submit a plan that names the unknown, or call conclude_no_changes")
|
||||
}
|
||||
ledger, ok := evidence.FromContext(ctx)
|
||||
if !ok {
|
||||
return "", fmt.Errorf("conclude_blocked is unavailable without an evidence ledger for this turn")
|
||||
}
|
||||
if err := checkBlockedEvidence(ledger, payload.Evidence); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return "Recorded: blocked. The turn ends here and is reported as blocked — state the same thing in your reply, including what you did complete.", nil
|
||||
}
|
||||
|
||||
// checkBlockedEvidence requires one reference the host observed. Unlike a
|
||||
// sign-off, a failed command counts: the run that proves a task impossible is
|
||||
// usually the run that failed, and demanding a successful one would leave the
|
||||
// honest conclusion unprovable.
|
||||
func checkBlockedEvidence(ledger *evidence.Ledger, items []blockedEvidence) error {
|
||||
if len(items) == 0 {
|
||||
return fmt.Errorf("conclude_blocked needs evidence: cite a command you ran or files you read that establish the blocker")
|
||||
}
|
||||
for i, item := range items {
|
||||
command := strings.TrimSpace(item.Command)
|
||||
if command != "" {
|
||||
if ledger.HasSuccessfulCommand(command) || ledger.HasFailedCommand(command) {
|
||||
return nil
|
||||
}
|
||||
continue
|
||||
}
|
||||
if len(item.Paths) > 0 && ledger.HasSuccessfulReadOrWrite(item.Paths) {
|
||||
return nil
|
||||
}
|
||||
if command == "" && len(item.Paths) == 0 {
|
||||
return fmt.Errorf("evidence %d: give a command or paths", i+1)
|
||||
}
|
||||
}
|
||||
return fmt.Errorf("none of the cited evidence matches a receipt from this turn — cite a command exactly as it ran, or files you actually read")
|
||||
}
|
||||
|
||||
// HostRequiredBy says why the tool cannot be removed by name: the host names it as the way out of a task that cannot pass its check.
|
||||
func (*ConcludeBlockedTool) HostRequiredBy() string { return tool.FeatureTurnExit }
|
||||
@@ -1,89 +0,0 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"reasonix/internal/safety/evidence"
|
||||
)
|
||||
|
||||
func blockedCtx(ledger *evidence.Ledger) context.Context {
|
||||
return evidence.WithLedger(context.Background(), ledger)
|
||||
}
|
||||
|
||||
func TestConcludeBlockedRequiresCheckableEvidence(t *testing.T) {
|
||||
ledger := evidence.NewLedger()
|
||||
ledger.Record(evidence.Receipt{ToolName: "bash", Success: false, Command: "go test ./..."})
|
||||
tool := NewConcludeBlockedTool()
|
||||
ctx := blockedCtx(ledger)
|
||||
|
||||
// A failing run is the usual proof that a task cannot be done, so it counts.
|
||||
out, err := tool.Execute(ctx, json.RawMessage(`{
|
||||
"blocker":"two tests demand different values from one pure function",
|
||||
"evidence":[{"command":"go test ./..."}]
|
||||
}`))
|
||||
if err != nil {
|
||||
t.Fatalf("failed command should count as evidence: %v", err)
|
||||
}
|
||||
if !strings.Contains(out, "blocked") {
|
||||
t.Fatalf("output = %q", out)
|
||||
}
|
||||
|
||||
if _, err := tool.Execute(ctx, json.RawMessage(`{
|
||||
"blocker":"cannot be done",
|
||||
"evidence":[{"command":"go test ./somewhere-else"}]
|
||||
}`)); err == nil {
|
||||
t.Fatal("a command with no receipt must not establish a blocker")
|
||||
}
|
||||
if _, err := tool.Execute(ctx, json.RawMessage(`{"blocker":"cannot be done","evidence":[]}`)); err == nil {
|
||||
t.Fatal("a claim with no evidence must be rejected")
|
||||
}
|
||||
if _, err := tool.Execute(ctx, json.RawMessage(`{"blocker":" ","evidence":[{"command":"go test ./..."}]}`)); err == nil {
|
||||
t.Fatal("an empty blocker must be rejected")
|
||||
}
|
||||
}
|
||||
|
||||
func TestConcludeBlockedAcceptsReadPathsAsEvidence(t *testing.T) {
|
||||
ledger := evidence.NewLedger()
|
||||
ledger.Record(evidence.Receipt{ToolName: "read_file", Success: true, Read: true, Paths: []string{"gateway_test.go"}})
|
||||
if _, err := NewConcludeBlockedTool().Execute(blockedCtx(ledger), json.RawMessage(`{
|
||||
"blocker":"the two suites disagree",
|
||||
"evidence":[{"paths":["gateway_test.go"]}]
|
||||
}`)); err != nil {
|
||||
t.Fatalf("read paths should establish a blocker: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// The declaration waives one thing: that the check passed. A turn that never
|
||||
// ran the check has established nothing and stays unfinished.
|
||||
func TestReadinessAcceptsBlockedOnlyWithAnActualCheck(t *testing.T) {
|
||||
newAgent := func(records ...evidence.Receipt) *Agent {
|
||||
ledger := evidence.NewLedger()
|
||||
for _, r := range records {
|
||||
ledger.Record(r)
|
||||
}
|
||||
a := &Agent{task: taskRuntime{ledger: ledger}}
|
||||
return a
|
||||
}
|
||||
write := evidence.Receipt{ToolName: "edit_file", Success: true, Write: true, Mutation: true, MutationEvidence: evidence.MutationProven, Paths: []string{"quota.go"}}
|
||||
failedCheck := evidence.Receipt{ToolName: "bash", Success: false, Command: "go test ./...", Verification: evidence.VerificationFailed}
|
||||
declaration := evidence.Receipt{ToolName: "conclude_blocked", Success: true, Args: json.RawMessage(`{"blocker":"x"}`)}
|
||||
|
||||
a := newAgent(write, failedCheck, declaration)
|
||||
idx, _ := a.task.ledger.LatestProvenMutationIndex()
|
||||
if !a.task.ledger.HasBlockedConclusionAfter(idx) || !a.task.ledger.HasVerificationCommandAfter(idx) {
|
||||
t.Fatal("a declared blocker backed by a check that ran must be recognized")
|
||||
}
|
||||
|
||||
b := newAgent(write, declaration)
|
||||
idxB, _ := b.task.ledger.LatestProvenMutationIndex()
|
||||
if b.task.ledger.HasVerificationCommandAfter(idxB) {
|
||||
t.Fatal("a turn that never ran a check must not read as having run one")
|
||||
}
|
||||
}
|
||||
|
||||
// A rejected call proves nothing, so it must leave the contract alone. A call
|
||||
// the host accepted is a fact about the task even when it names no single
|
||||
// criterion — the turn is stuck either way.
|
||||
@@ -1,34 +0,0 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"reasonix/internal/state/sessionstore"
|
||||
"testing"
|
||||
|
||||
"reasonix/internal/runtime/plancontract"
|
||||
)
|
||||
|
||||
func criterionPlan() plancontract.Plan {
|
||||
return plancontract.Plan{
|
||||
Objective: "fix the retry race",
|
||||
Steps: []plancontract.Step{{
|
||||
Title: "fix it",
|
||||
Acceptance: []plancontract.Criterion{
|
||||
{Text: "retries no longer double-charge"},
|
||||
{Text: "existing payment tests keep passing", Regression: true},
|
||||
},
|
||||
}},
|
||||
}.Normalize()
|
||||
}
|
||||
|
||||
func TestAcceptanceCriterionIDsReachTheToolCall(t *testing.T) {
|
||||
a := New(nil, nil, sessionstore.NewSession(""), Options{}, nil)
|
||||
if ids := a.acceptanceCriterionIDs(); len(ids) != 0 {
|
||||
t.Fatalf("no plan must yield no ids, got %v", ids)
|
||||
}
|
||||
plan := criterionPlan()
|
||||
a.SetPlanContract(&plan)
|
||||
ids := a.acceptanceCriterionIDs()
|
||||
if len(ids) != 2 {
|
||||
t.Fatalf("ids = %v, want both criteria of the approved plan", ids)
|
||||
}
|
||||
}
|
||||
@@ -3,7 +3,6 @@ package agent
|
||||
import (
|
||||
"context"
|
||||
"reasonix/internal/state/sessionstore"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
@@ -98,122 +97,6 @@ func sessionHasUserMessageContaining(s *sessionstore.Session, needle string) boo
|
||||
return false
|
||||
}
|
||||
|
||||
// TestEvidenceFlowEndToEnd drives a full Run(): turn 1 runs bash then signs the
|
||||
// step off citing that exact command; complete_step must see the host receipt
|
||||
// recorded earlier in the same batch and report it host-verified.
|
||||
func TestEvidenceFlowEndToEnd(t *testing.T) {
|
||||
completeStep, ok := tool.LookupBuiltin("complete_step")
|
||||
if !ok {
|
||||
t.Fatal("complete_step builtin not registered")
|
||||
}
|
||||
reg := tool.NewRegistry()
|
||||
reg.Add(fakeTool{name: "bash", readOnly: false})
|
||||
reg.Add(completeStep)
|
||||
|
||||
prov := &scriptedProvider{name: "p", turns: [][]provider.Chunk{
|
||||
{
|
||||
toolCallChunk("c1", "bash", `{"command":"go test ./..."}`),
|
||||
toolCallChunk("c2", "complete_step", `{
|
||||
"step":"Run the suite",
|
||||
"result":"tests pass",
|
||||
"evidence":[{"kind":"verification","summary":"go test ./... passed","command":"go test ./..."}]
|
||||
}`),
|
||||
{Type: provider.ChunkDone},
|
||||
},
|
||||
{{Type: provider.ChunkText, Text: "done"}, {Type: provider.ChunkDone}},
|
||||
}}
|
||||
|
||||
a := New(prov, reg, sessionstore.NewSession(""), Options{}, event.Discard)
|
||||
if err := a.Run(context.Background(), "run the suite and sign the step off"); err != nil {
|
||||
t.Fatalf("Run: %v", err)
|
||||
}
|
||||
|
||||
if got := toolResult(a.sess.conversation, "complete_step"); !strings.Contains(got, "host-verified 1") {
|
||||
t.Fatalf("complete_step result = %q, want it host-verified from the bash receipt", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDeliveryProfileEnforcesAcceptanceReviewVerificationAndSignoff(t *testing.T) {
|
||||
reg := evidenceRegistry()
|
||||
reg.Add(fakeTool{name: "read_file", readOnly: true})
|
||||
// Keep review available so this ordinary production change exercises the
|
||||
// Medium-risk host-proof alternative instead of the minimal-registry bypass.
|
||||
reg.Add(fakeTool{name: "review", readOnly: true})
|
||||
|
||||
prov := &scriptedProvider{name: "delivery", turns: [][]provider.Chunk{
|
||||
{toolCallChunk("blocked-write", "write_file", `{"path":"main.go","content":"package main"}`), {Type: provider.ChunkDone}},
|
||||
{toolCallChunk("criteria", "todo_write", `{"todos":[{"content":"Ship main","status":"in_progress","activeForm":"Shipping main"}]}`), {Type: provider.ChunkDone}},
|
||||
{toolCallChunk("write", "write_file", `{"path":"main.go","content":"package main"}`), {Type: provider.ChunkDone}},
|
||||
{toolCallChunk("review", "read_file", `{"path":"main.go"}`), {Type: provider.ChunkDone}},
|
||||
{toolCallChunk("verify", "bash", `{"command":"go test ./..."}`), {Type: provider.ChunkDone}},
|
||||
{toolCallChunk("signoff", "complete_step", `{
|
||||
"step":"Ship main",
|
||||
"result":"main is implemented and verified",
|
||||
"evidence":[
|
||||
{"kind":"diff","summary":"main implementation added","paths":["main.go"]},
|
||||
{"kind":"verification","summary":"tests pass","command":"go test ./..."}
|
||||
]
|
||||
}`), {Type: provider.ChunkDone}},
|
||||
{{Type: provider.ChunkText, Text: "delivered"}, {Type: provider.ChunkDone}},
|
||||
}}
|
||||
|
||||
a := New(prov, reg, sessionstore.NewSession(""), Options{DeliveryProfile: true,
|
||||
CheckTargetAccess: allowTestFileAccess,
|
||||
}, event.Discard)
|
||||
if err := a.Run(context.Background(), "implement main"); err != nil {
|
||||
t.Fatalf("Run: %v", err)
|
||||
}
|
||||
if got := toolResult(a.sess.conversation, "write_file"); !strings.Contains(got, "delivery-first mode requires acceptance criteria") {
|
||||
t.Fatalf("first write result = %q, want delivery acceptance gate", got)
|
||||
}
|
||||
if !sessionHasUserMessageContaining(a.sess.conversation, "<execution-policy") {
|
||||
t.Fatal("execution-policy marker was not injected into the turn tail")
|
||||
}
|
||||
if got := lastToolResult(a.sess.conversation, "complete_step"); !strings.Contains(got, "signed off") {
|
||||
t.Fatalf("complete_step result = %q, want successful sign-off", got)
|
||||
}
|
||||
firstSystem := systemMessageContent(prov.requests[0])
|
||||
firstTools := prov.requests[0].Tools
|
||||
for i, req := range prov.requests[1:] {
|
||||
if got := systemMessageContent(req); got != firstSystem {
|
||||
t.Fatalf("delivery request %d changed the cache-stable system prompt", i+2)
|
||||
}
|
||||
if !reflect.DeepEqual(req.Tools, firstTools) {
|
||||
t.Fatalf("delivery request %d changed provider-visible tool schemas", i+2)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func systemMessageContent(req provider.Request) string {
|
||||
for _, msg := range req.Messages {
|
||||
if msg.Role == provider.RoleSystem {
|
||||
return msg.Content
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func TestDeliveryProfileCommandOnlyActionRequiresCriteriaAndSignoff(t *testing.T) {
|
||||
reg := evidenceRegistry()
|
||||
prov := &scriptedProvider{name: "delivery", turns: [][]provider.Chunk{
|
||||
{toolCallChunk("blocked-test", "bash", `{"command":"go test ./..."}`), {Type: provider.ChunkDone}},
|
||||
{toolCallChunk("criteria", "todo_write", `{"todos":[{"content":"Run tests","status":"in_progress"}]}`), {Type: provider.ChunkDone}},
|
||||
{toolCallChunk("verify", "bash", `{"command":"go test ./..."}`), {Type: provider.ChunkDone}},
|
||||
{toolCallChunk("signoff", "complete_step", `{"step":"Run tests","result":"tests pass","evidence":[{"kind":"verification","summary":"tests pass","command":"go test ./..."}]}`), {Type: provider.ChunkDone}},
|
||||
{{Type: provider.ChunkText, Text: "tests pass"}, {Type: provider.ChunkDone}},
|
||||
}}
|
||||
a := New(prov, reg, sessionstore.NewSession(""), Options{DeliveryProfile: true}, event.Discard)
|
||||
if err := a.Run(context.Background(), "run tests"); err != nil {
|
||||
t.Fatalf("Run: %v", err)
|
||||
}
|
||||
if got := toolResult(a.sess.conversation, "bash"); !strings.Contains(got, "delivery-first mode requires acceptance criteria") {
|
||||
t.Fatalf("first bash result = %q, want acceptance gate", got)
|
||||
}
|
||||
if got := lastToolResult(a.sess.conversation, "complete_step"); !strings.Contains(got, "signed off") {
|
||||
t.Fatalf("complete_step result = %q, want successful command-only sign-off", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDeliveryProfileBlocksMixedVerificationBeforeItBecomesMutation(t *testing.T) {
|
||||
reg := evidenceRegistry()
|
||||
prov := &scriptedProvider{name: "delivery", turns: [][]provider.Chunk{
|
||||
@@ -309,45 +192,6 @@ func TestDeliveryProfileAllowsEvidenceBackedReadOnlyAnalysis(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestEvidenceFlowEnforcesProjectChecksAfterWrite(t *testing.T) {
|
||||
completeStep, ok := tool.LookupBuiltin("complete_step")
|
||||
if !ok {
|
||||
t.Fatal("complete_step builtin not registered")
|
||||
}
|
||||
reg := tool.NewRegistry()
|
||||
reg.Add(fakeTool{name: "write_file", readOnly: false, writesPaths: true})
|
||||
reg.Add(fakeTool{name: "bash", readOnly: false})
|
||||
reg.Add(completeStep)
|
||||
|
||||
prov := &scriptedProvider{name: "p", turns: [][]provider.Chunk{
|
||||
{
|
||||
toolCallChunk("c1", "write_file", `{"path":"changed.go","content":"package main"}`),
|
||||
toolCallChunk("c2", "bash", `{"command":"go test ./..."}`),
|
||||
toolCallChunk("c3", "complete_step", `{
|
||||
"step":"Edit code",
|
||||
"result":"changed.go updated",
|
||||
"evidence":[{"kind":"diff","summary":"updated code","paths":["changed.go"]}]
|
||||
}`),
|
||||
{Type: provider.ChunkDone},
|
||||
},
|
||||
{{Type: provider.ChunkText, Text: "done"}, {Type: provider.ChunkDone}},
|
||||
}}
|
||||
|
||||
a := New(prov, reg, sessionstore.NewSession(""), Options{
|
||||
ProjectChecks: []instruction.VerifyCheck{{Command: "go test ./...", SourcePath: "AGENTS.md", Line: 3}},
|
||||
|
||||
CheckTargetAccess: allowTestFileAccess,
|
||||
}, event.Discard)
|
||||
if err := a.Run(context.Background(), "edit and verify"); err != nil {
|
||||
t.Fatalf("Run: %v", err)
|
||||
}
|
||||
|
||||
got := toolResult(a.sess.conversation, "complete_step")
|
||||
if !strings.Contains(got, "project checks 1") {
|
||||
t.Fatalf("complete_step result = %q, want project check verified from same batch", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFinalReadinessAllowsFinalAnswerWithoutWriter(t *testing.T) {
|
||||
prov := &scriptedProvider{name: "p", turns: [][]provider.Chunk{
|
||||
{{Type: provider.ChunkText, Text: "done"}, {Type: provider.ChunkDone}},
|
||||
@@ -440,44 +284,6 @@ func TestReadOnlyContextAndTodoTurnMayEndWithIncompleteTodos(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestEvidenceFlowRejectsUncitedCommand proves the loop rejects a sign-off whose
|
||||
// cited command was never run: bash ran "go test", complete_step cites "go vet".
|
||||
func TestEvidenceFlowRejectsUncitedCommand(t *testing.T) {
|
||||
completeStep, ok := tool.LookupBuiltin("complete_step")
|
||||
if !ok {
|
||||
t.Fatal("complete_step builtin not registered")
|
||||
}
|
||||
reg := tool.NewRegistry()
|
||||
reg.Add(fakeTool{name: "bash", readOnly: false})
|
||||
reg.Add(completeStep)
|
||||
|
||||
prov := &scriptedProvider{name: "p", turns: [][]provider.Chunk{
|
||||
{
|
||||
toolCallChunk("c1", "bash", `{"command":"go test ./..."}`),
|
||||
toolCallChunk("c2", "complete_step", `{
|
||||
"step":"Vet the tree",
|
||||
"result":"vet is clean",
|
||||
"evidence":[{"kind":"verification","summary":"go vet passed","command":"go vet ./..."}]
|
||||
}`),
|
||||
{Type: provider.ChunkDone},
|
||||
},
|
||||
{{Type: provider.ChunkText, Text: "done"}, {Type: provider.ChunkDone}},
|
||||
}}
|
||||
|
||||
a := New(prov, reg, sessionstore.NewSession(""), Options{}, event.Discard)
|
||||
if err := a.Run(context.Background(), "vet the tree and sign off"); err != nil {
|
||||
t.Fatalf("Run: %v", err)
|
||||
}
|
||||
|
||||
got := toolResult(a.sess.conversation, "complete_step")
|
||||
if !strings.Contains(got, "has no matching successful receipt") {
|
||||
t.Fatalf("complete_step result = %q, want the uncited command rejected", got)
|
||||
}
|
||||
if strings.Contains(got, "host-verified") {
|
||||
t.Fatalf("uncited command should not verify, got %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestEvidenceFlowRejectsStepMissingFromTodoWrite(t *testing.T) {
|
||||
todoWrite, ok := tool.LookupBuiltin("todo_write")
|
||||
if !ok {
|
||||
@@ -605,126 +411,6 @@ func TestEvidenceFlowRejectsTodoCompletionWithoutCompleteStep(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestEvidenceFlowRecoversTodoCompletionAfterFailedCompleteStepWithProgress(t *testing.T) {
|
||||
todoWrite, ok := tool.LookupBuiltin("todo_write")
|
||||
if !ok {
|
||||
t.Fatal("todo_write builtin not registered")
|
||||
}
|
||||
completeStep, ok := tool.LookupBuiltin("complete_step")
|
||||
if !ok {
|
||||
t.Fatal("complete_step builtin not registered")
|
||||
}
|
||||
reg := tool.NewRegistry()
|
||||
reg.Add(todoWrite)
|
||||
reg.Add(fakeTool{name: "bash", readOnly: false})
|
||||
reg.Add(completeStep)
|
||||
|
||||
prov := &scriptedProvider{name: "p", turns: [][]provider.Chunk{
|
||||
{
|
||||
toolCallChunk("c1", "todo_write", `{"todos":[{"content":"Run project script","status":"in_progress"}]}`),
|
||||
toolCallChunk("c2", "bash", `{"command":"python \"script.py\""}`),
|
||||
toolCallChunk("c3", "complete_step", `{
|
||||
"step":"Run project script",
|
||||
"result":"script ran",
|
||||
"evidence":[{"kind":"verification","summary":"script completed","command":"python other.py"}]
|
||||
}`),
|
||||
toolCallChunk("c4", "todo_write", `{"todos":[{"content":"Run project script","status":"completed"}]}`),
|
||||
{Type: provider.ChunkDone},
|
||||
},
|
||||
{{Type: provider.ChunkText, Text: "done"}, {Type: provider.ChunkDone}},
|
||||
}}
|
||||
|
||||
a := New(prov, reg, sessionstore.NewSession(""), Options{}, event.Discard)
|
||||
if err := a.Run(context.Background(), "recover after a failed complete_step"); err != nil {
|
||||
t.Fatalf("Run: %v", err)
|
||||
}
|
||||
|
||||
stepResult := lastToolResult(a.sess.conversation, "complete_step")
|
||||
if !strings.Contains(stepResult, "no matching successful receipt") {
|
||||
t.Fatalf("complete_step result = %q, want the sign-off attempt to fail first", stepResult)
|
||||
}
|
||||
if !strings.Contains(stepResult, `python \"script.py\"`) {
|
||||
t.Fatalf("complete_step result = %q, want the self-correction hint to include the real command", stepResult)
|
||||
}
|
||||
if strings.Contains(stepResult, "todo_write") {
|
||||
t.Fatalf("complete_step result = %q, want command hints without todo tool noise", stepResult)
|
||||
}
|
||||
if got := lastToolResult(a.sess.conversation, "todo_write"); !strings.Contains(got, "1 completed") {
|
||||
t.Fatalf("todo_write result = %q, want completion recovery accepted", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestEvidenceFlowRecoversAfterBatchTodoCompletionRejection(t *testing.T) {
|
||||
todoWrite, ok := tool.LookupBuiltin("todo_write")
|
||||
if !ok {
|
||||
t.Fatal("todo_write builtin not registered")
|
||||
}
|
||||
completeStep, ok := tool.LookupBuiltin("complete_step")
|
||||
if !ok {
|
||||
t.Fatal("complete_step builtin not registered")
|
||||
}
|
||||
reg := tool.NewRegistry()
|
||||
reg.Add(todoWrite)
|
||||
reg.Add(completeStep)
|
||||
|
||||
prov := &scriptedProvider{name: "p", turns: [][]provider.Chunk{
|
||||
{
|
||||
toolCallChunk("c1", "todo_write", `{"todos":[
|
||||
{"content":"Port entity imports","status":"in_progress"},
|
||||
{"content":"Run build and tests","status":"pending"}
|
||||
]}`),
|
||||
toolCallChunk("c2", "todo_write", `{"todos":[
|
||||
{"content":"Port entity imports","status":"completed"},
|
||||
{"content":"Run build and tests","status":"completed"}
|
||||
]}`),
|
||||
{Type: provider.ChunkDone},
|
||||
},
|
||||
{
|
||||
toolCallChunk("c3", "complete_step", `{
|
||||
"step":"Port entity imports",
|
||||
"result":"entity imports ported",
|
||||
"evidence":[{"kind":"manual","summary":"checked manually"}]
|
||||
}`),
|
||||
toolCallChunk("c4", "complete_step", `{
|
||||
"step":"Run build and tests",
|
||||
"result":"build and tests ran",
|
||||
"evidence":[{"kind":"manual","summary":"checked manually"}]
|
||||
}`),
|
||||
{Type: provider.ChunkDone},
|
||||
},
|
||||
{
|
||||
toolCallChunk("c5", "complete_step", `{
|
||||
"step":"Run build and tests",
|
||||
"result":"build and tests ran",
|
||||
"evidence":[{"kind":"manual","summary":"checked manually"}]
|
||||
}`),
|
||||
{Type: provider.ChunkDone},
|
||||
},
|
||||
{{Type: provider.ChunkText, Text: "done"}, {Type: provider.ChunkDone}},
|
||||
}}
|
||||
|
||||
a := New(prov, reg, sessionstore.NewSession(""), Options{}, event.Discard)
|
||||
if err := a.Run(context.Background(), "recover from a rejected batch todo update"); err != nil {
|
||||
t.Fatalf("Run: %v", err)
|
||||
}
|
||||
|
||||
stepResults := toolResults(a.sess.conversation, "complete_step")
|
||||
if len(stepResults) < 3 {
|
||||
t.Fatalf("complete_step results = %v, want blocked batch sign-off and a retry", stepResults)
|
||||
}
|
||||
if got := stepResults[1]; !strings.Contains(got, "only one successful complete_step") {
|
||||
t.Fatalf("second batched complete_step result = %q, want serial-signoff block", got)
|
||||
}
|
||||
if got := stepResults[2]; !strings.Contains(got, "signed off") {
|
||||
t.Fatalf("next-round complete_step result = %q, want successful sign-off", got)
|
||||
}
|
||||
for i, todo := range a.CanonicalTodoState() {
|
||||
if todo.Status != "completed" {
|
||||
t.Fatalf("canonical todo %d = %+v, want completed", i+1, todo)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestEvidenceFlowFailedCompleteStepDoesNotAuthorizeTodoCompletion(t *testing.T) {
|
||||
todoWrite, ok := tool.LookupBuiltin("todo_write")
|
||||
if !ok {
|
||||
|
||||
@@ -215,8 +215,6 @@ func TestAnnotateFailureDiagnosticsFallsBackWhenTheModelIsNoHelp(t *testing.T) {
|
||||
func TestHandoffNudgeReadsThePlanContract(t *testing.T) {
|
||||
action := plancontract.Plan{Objective: "fix it", Steps: []plancontract.Step{
|
||||
{Title: "edit the parser", CandidateFiles: []string{"parser.go"}}}}
|
||||
verify := plancontract.Plan{Objective: "check it", Steps: []plancontract.Step{
|
||||
{Title: "run the suite", Verification: []plancontract.Verification{{Command: "go test ./..."}}}}}
|
||||
prose := plancontract.Plan{Objective: "explain it", Steps: []plancontract.Step{
|
||||
{Title: "describe how the retry budget works"}}}
|
||||
|
||||
@@ -227,7 +225,6 @@ func TestHandoffNudgeReadsThePlanContract(t *testing.T) {
|
||||
}{
|
||||
{"no contract trusts the answer", nil, false},
|
||||
{"plan naming files expects action", &action, true},
|
||||
{"plan naming a command expects action", &verify, true},
|
||||
{"prose-only plan does not", &prose, false},
|
||||
} {
|
||||
a := New(nil, tool.NewRegistry(), sessionstore.NewSession("sys"), Options{}, event.Discard)
|
||||
|
||||
@@ -602,34 +602,6 @@ func TestExecuteBatchSegmentsAroundWrites(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteBatchFeedsReceiptsToCompleteStep(t *testing.T) {
|
||||
completeStep, ok := tool.LookupBuiltin("complete_step")
|
||||
if !ok {
|
||||
t.Fatal("complete_step builtin not registered")
|
||||
}
|
||||
reg := tool.NewRegistry()
|
||||
reg.Add(fakeTool{name: "bash", readOnly: false})
|
||||
reg.Add(completeStep)
|
||||
a := New(nil, reg, sessionstore.NewSession(""), Options{}, event.Discard)
|
||||
|
||||
batch := a.executeBatch(context.Background(), &a.turn, []provider.ToolCall{
|
||||
{Name: "bash", Arguments: `{"command":"go test ./internal/..."}`},
|
||||
{Name: "complete_step", Arguments: `{
|
||||
"step":"Run checks",
|
||||
"result":"checks passed",
|
||||
"evidence":[{"kind":"verification","summary":"tests passed","command":"go test ./internal/..."}]
|
||||
}`},
|
||||
})
|
||||
results := batch.results
|
||||
|
||||
if len(results) != 2 {
|
||||
t.Fatalf("got %d results, want 2", len(results))
|
||||
}
|
||||
if !strings.Contains(results[1], "host-verified 1") {
|
||||
t.Fatalf("complete_step did not see bash receipt: %q", results[1])
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteOneFailedReceiptDoesNotVerify(t *testing.T) {
|
||||
reg := tool.NewRegistry()
|
||||
reg.Add(fakeTool{name: "bash", readOnly: false, err: errors.New("boom")})
|
||||
|
||||
@@ -6,7 +6,6 @@ import (
|
||||
"path/filepath"
|
||||
|
||||
"reasonix/internal/runtime/plancontract"
|
||||
"reasonix/internal/runtime/taskcontract"
|
||||
"reasonix/internal/safety/evidence"
|
||||
)
|
||||
|
||||
@@ -41,78 +40,12 @@ func (a *Agent) PlanContract() *plancontract.Plan {
|
||||
return &copied
|
||||
}
|
||||
|
||||
// planFacts projects a plan onto the contract-relevant facts. The projection
|
||||
// lives here rather than in either package so taskcontract keeps never seeing a
|
||||
// plan and plancontract keeps depending on nothing above it.
|
||||
func planFacts(plan plancontract.Plan) taskcontract.PlanFacts {
|
||||
facts := taskcontract.PlanFacts{}
|
||||
seen := map[string]bool{}
|
||||
for _, step := range plan.Steps {
|
||||
for _, c := range step.Acceptance {
|
||||
criterion := taskcontract.PlanCriterion{ID: c.ID, Text: c.Text}
|
||||
switch {
|
||||
case c.Optional:
|
||||
facts.Optional = append(facts.Optional, criterion)
|
||||
case c.Regression:
|
||||
facts.Regressions = append(facts.Regressions, criterion)
|
||||
default:
|
||||
facts.AcceptanceCriteria = append(facts.AcceptanceCriteria, criterion)
|
||||
}
|
||||
}
|
||||
for _, v := range step.Verification {
|
||||
if seen[v.Command] {
|
||||
continue
|
||||
}
|
||||
seen[v.Command] = true
|
||||
facts.Verifications = append(facts.Verifications, v.Command)
|
||||
}
|
||||
if len(step.Risks) > 0 {
|
||||
facts.Risky = true
|
||||
}
|
||||
// Scope is where work is expected, not a claim of fact, so a candidate
|
||||
// path belongs in it even though it was never read.
|
||||
facts.Touchpoints = appendUnseen(facts.Touchpoints, seen, step.VerifiedFiles)
|
||||
facts.Touchpoints = appendUnseen(facts.Touchpoints, seen, step.CandidateFiles)
|
||||
}
|
||||
return facts
|
||||
}
|
||||
|
||||
func appendUnseen(dst []string, seen map[string]bool, add []string) []string {
|
||||
for _, s := range add {
|
||||
if s == "" || seen["path:"+s] {
|
||||
continue
|
||||
}
|
||||
seen["path:"+s] = true
|
||||
dst = append(dst, s)
|
||||
}
|
||||
return dst
|
||||
}
|
||||
|
||||
// acceptanceCriterionIDs lists the approved plan's criterion ids so a tool call
|
||||
// can check a citation against the plan the user approved.
|
||||
func (a *Agent) acceptanceCriterionIDs() []string {
|
||||
plan := a.PlanContract()
|
||||
if plan == nil {
|
||||
return nil
|
||||
}
|
||||
var ids []string
|
||||
for _, step := range plan.Steps {
|
||||
for _, c := range step.Acceptance {
|
||||
if c.ID != "" {
|
||||
ids = append(ids, c.ID)
|
||||
}
|
||||
}
|
||||
}
|
||||
return ids
|
||||
}
|
||||
|
||||
// withContractState attaches what a tool call needs to check a claim against the
|
||||
// approved work: the canonical task list, and the criterion ids a proof may
|
||||
// cite. They travel together because both answer "does this claim name
|
||||
// something real".
|
||||
func (a *Agent) withContractState(ctx context.Context) context.Context {
|
||||
ctx = evidence.WithTodoState(ctx, a.CanonicalTodoState())
|
||||
return evidence.WithAcceptanceCriteria(ctx, a.acceptanceCriterionIDs())
|
||||
return evidence.WithTodoState(ctx, a.CanonicalTodoState())
|
||||
}
|
||||
|
||||
// mutationEscapesPlan reports whether a pending write touches a path the
|
||||
|
||||
@@ -1,67 +1,3 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"slices"
|
||||
"testing"
|
||||
|
||||
"reasonix/internal/runtime/plancontract"
|
||||
"reasonix/internal/runtime/taskcontract"
|
||||
)
|
||||
|
||||
func contractPlan() plancontract.Plan {
|
||||
return plancontract.Plan{
|
||||
Objective: "make the cache key model-aware",
|
||||
Steps: []plancontract.Step{
|
||||
{
|
||||
ID: "p1", Title: "thread the model ref through",
|
||||
VerifiedFiles: []string{"internal/contract/provider/cache.go"},
|
||||
CandidateFiles: []string{"internal/assembly/boot/boot.go"},
|
||||
Risks: []string{"warm caches invalidate once"},
|
||||
Acceptance: []plancontract.Criterion{
|
||||
{Text: "two model refs never share an entry"},
|
||||
{Text: "existing hits keep hitting", Regression: true},
|
||||
{Text: "the hit rate is logged", Optional: true},
|
||||
},
|
||||
Verification: []plancontract.Verification{{Command: "go test ./internal/contract/provider/"}},
|
||||
},
|
||||
},
|
||||
}.Normalize()
|
||||
}
|
||||
|
||||
func TestPlanFactsSeparatesCriteriaByKind(t *testing.T) {
|
||||
plan := contractPlan()
|
||||
facts := planFacts(plan)
|
||||
texts := func(cs []taskcontract.PlanCriterion) []string {
|
||||
out := make([]string, 0, len(cs))
|
||||
for _, c := range cs {
|
||||
if c.ID == "" {
|
||||
t.Errorf("criterion %q lost the identity a proof must cite", c.Text)
|
||||
}
|
||||
out = append(out, c.Text)
|
||||
}
|
||||
return out
|
||||
}
|
||||
if !slices.Equal(texts(facts.AcceptanceCriteria), []string{"two model refs never share an entry"}) {
|
||||
t.Errorf("acceptance = %v", facts.AcceptanceCriteria)
|
||||
}
|
||||
if !slices.Equal(texts(facts.Regressions), []string{"existing hits keep hitting"}) {
|
||||
t.Errorf("regressions = %v", facts.Regressions)
|
||||
}
|
||||
if !slices.Equal(texts(facts.Optional), []string{"the hit rate is logged"}) {
|
||||
t.Errorf("optional = %v", facts.Optional)
|
||||
}
|
||||
// The id the plan assigned is the id the contract must carry.
|
||||
if got, want := facts.AcceptanceCriteria[0].ID, plan.Steps[0].Acceptance[0].ID; got != want {
|
||||
t.Errorf("criterion id = %q, want the plan's %q", got, want)
|
||||
}
|
||||
if !slices.Equal(facts.Verifications, []string{"go test ./internal/contract/provider/"}) {
|
||||
t.Errorf("verifications = %v", facts.Verifications)
|
||||
}
|
||||
if !facts.Risky {
|
||||
t.Error("a step carrying risks must mark the plan risky")
|
||||
}
|
||||
// Scope is where work is expected, so an inferred path belongs in it.
|
||||
if !slices.Equal(facts.Touchpoints, []string{"internal/contract/provider/cache.go", "internal/assembly/boot/boot.go"}) {
|
||||
t.Errorf("touchpoints = %v", facts.Touchpoints)
|
||||
}
|
||||
}
|
||||
import ()
|
||||
@@ -22,7 +22,7 @@ func NewSubmitPlanTool() *SubmitPlanTool { return &SubmitPlanTool{} }
|
||||
func (*SubmitPlanTool) Name() string { return "submit_plan" }
|
||||
|
||||
func (*SubmitPlanTool) Description() string {
|
||||
return "Submit your finished plan as structured data. This is how a plan reaches the host — the host renders it for the user and hands it to the executor, so do NOT also restate the plan in prose. Every step needs a `title`; a step with a `parent_id` is a sub-step of that phase (two levels, keep phases few). Record what you actually READ as `verified_files` and what you only INFERRED as `candidate_files` — never present a guess as a verified path. Attach `acceptance` criteria and command-level `verification` to the steps they belong to, mark must-keep-passing behavior with `regression`, and label anything unproven in `assumptions`. Set `requires_approval` when execution should stop for the user first; the host decides whether it actually gates."
|
||||
return "Submit your finished plan as structured data. This is how a plan reaches the host — the host renders it for the user and hands it to the executor, so do NOT also restate the plan in prose. Every step needs a `title`; a step with a `parent_id` is a sub-step of that phase (two levels, keep phases few). Record what you actually READ as `verified_files` and what you only INFERRED as `candidate_files` — never present a guess as a verified path. Label anything unproven in `assumptions`. Set `requires_approval` when execution should stop for the user first; the host decides whether it actually gates."
|
||||
}
|
||||
|
||||
func (*SubmitPlanTool) Schema() json.RawMessage {
|
||||
@@ -56,30 +56,6 @@ func (*SubmitPlanTool) Schema() json.RawMessage {
|
||||
"depends_on":{"type":"array","items":{"type":"string"},"description":"Ids of sibling steps that must happen first."},
|
||||
"verified_files":{"type":"array","items":{"type":"string"},"description":"Paths you actually opened and read."},
|
||||
"candidate_files":{"type":"array","items":{"type":"string"},"description":"Paths you inferred but did NOT read. Never list an unread path as verified."},
|
||||
"acceptance":{
|
||||
"type":"array",
|
||||
"description":"Checkable conditions this step must meet.",
|
||||
"items":{
|
||||
"type":"object",
|
||||
"properties":{
|
||||
"text":{"type":"string","description":"The condition, stated so it can be checked."},
|
||||
"regression":{"type":"boolean","description":"True when this is existing behavior that must keep working."},
|
||||
"optional":{"type":"boolean","description":"True for a nice-to-have that must never block completion."}
|
||||
},
|
||||
"required":["text"]
|
||||
}
|
||||
},
|
||||
"verification":{
|
||||
"type":"array",
|
||||
"description":"Commands that prove the step.",
|
||||
"items":{
|
||||
"type":"object",
|
||||
"properties":{
|
||||
"command":{"type":"string","description":"The command as the executor should run it."},
|
||||
"expect":{"type":"string","description":"What a pass looks like."}
|
||||
}
|
||||
}
|
||||
},
|
||||
"risks":{"type":"array","items":{"type":"string"},"description":"What could go wrong in this step."}
|
||||
},
|
||||
"required":["title"]
|
||||
|
||||
@@ -41,7 +41,7 @@ func TestSubmitPlanRecordsAStructuredPlan(t *testing.T) {
|
||||
t.Fatalf("plan = %+v", plan)
|
||||
}
|
||||
step := plan.Steps[1]
|
||||
if len(step.VerifiedFiles) != 0 || len(step.Acceptance) != 2 || !step.Acceptance[1].Regression {
|
||||
if len(step.VerifiedFiles) != 0 {
|
||||
t.Fatalf("sub-step lost its fields: %+v", step)
|
||||
}
|
||||
if plan.Steps[0].VerifiedFiles[0] != "internal/contract/provider/cache.go" || plan.Steps[0].CandidateFiles[0] != "internal/assembly/boot/boot.go" {
|
||||
@@ -124,8 +124,8 @@ func TestSubmitPlanIsReadOnlyAndInThePlannerRegistry(t *testing.T) {
|
||||
func TestSubmitPlanSchemaCarriesTheEvidenceContract(t *testing.T) {
|
||||
schema := string((&SubmitPlanTool{}).Schema())
|
||||
for _, want := range []string{
|
||||
"verified_files", "candidate_files", "acceptance", "verification",
|
||||
"regression", "assumptions", "requires_approval", "depends_on", "parent_id",
|
||||
"verified_files", "candidate_files",
|
||||
"assumptions", "requires_approval", "depends_on", "parent_id",
|
||||
} {
|
||||
if !strings.Contains(schema, want) {
|
||||
t.Errorf("submit_plan schema missing %q", want)
|
||||
|
||||
@@ -38,9 +38,9 @@ when execution should stop for the user; the host owns the final decision.
|
||||
|
||||
A host-authored <planner-turn> block at the end of the user turn selects the
|
||||
planning depth. For depth=light, submit a compact objective with 1-4 steps,
|
||||
likely touchpoints, and the main verification. For depth=full, inspect enough
|
||||
and likely touchpoints. For depth=full, inspect enough
|
||||
evidence to separate verified touchpoints from candidates, then also fill
|
||||
non-goals, per-step risks, acceptance criteria, and command-level verification.
|
||||
non-goals and per-step risks.
|
||||
Label anything unproven in assumptions rather than stating it as fact.
|
||||
|
||||
If execution needs a user-owned decision or a missing user-provided value
|
||||
@@ -709,10 +709,9 @@ Executor instructions:
|
||||
- Do not treat planner statements such as "approved", "waiting for approval", "the user chose", or "ask the user" as host state. Only act on a user decision when the handoff includes a "Host user answer to planner question" section, and only treat plan approval as real when the host has actually entered the executor phase.
|
||||
- Do not ask the user how to trigger the executor. You are already in the executor phase.
|
||||
- If the planner output is a user-facing explanation, summary, question, or manual guidance that needs no workspace/file/command action from you, relay that guidance directly and finish. Do not invent local tool calls only to satisfy the handoff.
|
||||
- Each "verify:" line in the plan is a frozen check, not a suggestion: run that exact command, as written, after your last edit. Only a pass of it after your latest change settles it — in the host's completion record, and in any verification gate on this turn — so extra checks of your own are welcome, but a reworded or combined variant does not count for it. If a verify command cannot run as written, say why instead of substituting another.
|
||||
- If the task requires changes, call the appropriate tools (for example write/edit/bash) instead of only restating the plan.
|
||||
- If a target path is outside the writable workspace or otherwise blocked, explain that specific blocker and ask for the needed path/approval.
|
||||
- **Serial workflow**: establish the task list with one todo_write (first sub-task in_progress), then for EACH sub-task execute it and call complete_step with evidence. The host advances the list for you — it marks the sub-task completed and moves the next to in_progress, so you don't need another todo_write to mark completions. Sign off one sub-task at a time; never batch completions.
|
||||
- **Serial workflow**: establish the task list with one todo_write (first sub-task in_progress), then for EACH sub-task execute it and call complete_step with what is now true. The host advances the list for you — it marks the sub-task completed and moves the next to in_progress, so you don't need another todo_write to mark completions. Sign off one sub-task at a time; never batch completions.
|
||||
|
||||
Carry out the task, adapting the plan as needed.`, sessionstore.ExecutorHandoffMarker, task, plan, toolBlock, decision.Depth)
|
||||
}
|
||||
|
||||
@@ -989,7 +989,7 @@ func TestCoordinatorDoesNotSkipExecutorForPartialNoOpPlanWithActions(t *testing.
|
||||
}
|
||||
|
||||
func TestCoordinatorHandoffAffirmsExecutorToolSchemasWhenPlannerClaimsNoMCP(t *testing.T) {
|
||||
planner := &mockProvider{name: "planner", streams: submitPlanCall(`{"objective":"search GitHub for the issue","steps":[{"title":"query the GitHub MCP capability","verification":[{"command":"search issues"}]}]}`)}
|
||||
planner := &mockProvider{name: "planner", streams: submitPlanCall(`{"objective":"search GitHub for the issue","steps":[{"title":"query the GitHub MCP capability","candidate_files":["github"]}]}`)}
|
||||
exec := &mockProvider{name: "executor", streams: [][]provider.Chunk{
|
||||
{
|
||||
{Type: provider.ChunkText, Text: "GitHub MCP is unavailable."},
|
||||
@@ -1230,7 +1230,6 @@ func TestDefaultPlannerPromptDefinesLightAndFullEvidenceContracts(t *testing.T)
|
||||
"depth=light",
|
||||
"depth=full",
|
||||
"submit_plan",
|
||||
"command-level verification",
|
||||
"assumptions",
|
||||
} {
|
||||
// The verified/candidate split is asserted where it is enforced: the schema.
|
||||
|
||||
@@ -84,7 +84,6 @@ func TestSubmittedPlanReachesTheExecutorHandoff(t *testing.T) {
|
||||
"1. thread the model ref through",
|
||||
" - extend cacheKey",
|
||||
"verified: internal/contract/provider/cache.go",
|
||||
"verify: go test ./internal/contract/provider/ — all green",
|
||||
} {
|
||||
if !strings.Contains(handoff, want) {
|
||||
t.Errorf("executor handoff missing %q:\n%s", want, handoff)
|
||||
|
||||
@@ -75,8 +75,6 @@ func changedFields(before, after Step) []string {
|
||||
add("depends_on", slices.Equal(before.DependsOn, after.DependsOn))
|
||||
add("verified_files", slices.Equal(before.VerifiedFiles, after.VerifiedFiles))
|
||||
add("candidate_files", slices.Equal(before.CandidateFiles, after.CandidateFiles))
|
||||
add("acceptance", slices.Equal(before.Acceptance, after.Acceptance))
|
||||
add("verification", slices.Equal(before.Verification, after.Verification))
|
||||
add("risks", slices.Equal(before.Risks, after.Risks))
|
||||
return fields
|
||||
}
|
||||
@@ -91,7 +89,6 @@ func (d Diff) NeedsApproval() bool {
|
||||
}
|
||||
for _, change := range d.Changed {
|
||||
if grew(change.Before.Risks, change.After.Risks) ||
|
||||
grew(change.Before.Acceptance, change.After.Acceptance) ||
|
||||
grew(change.Before.VerifiedFiles, change.After.VerifiedFiles) ||
|
||||
grew(change.Before.CandidateFiles, change.After.CandidateFiles) {
|
||||
return true
|
||||
|
||||
@@ -93,7 +93,6 @@ func TestNeedsApprovalOnlyOnExpansion(t *testing.T) {
|
||||
"a new step": func(p *Plan) { p.Steps = append(p.Steps, Step{ID: "s4", Title: "extra"}) },
|
||||
"a new objective": func(p *Plan) { p.Objective = "something else entirely" },
|
||||
"a new risk": func(p *Plan) { p.Steps[0].Risks = []string{"data loss"} },
|
||||
"a new criterion": func(p *Plan) { p.Steps[0].Acceptance = []Criterion{{Text: "must hold"}} },
|
||||
"a wider surface": func(p *Plan) { p.Steps[1].CandidateFiles = []string{"api.go", "schema.go"} },
|
||||
}
|
||||
for name, mutate := range expansions {
|
||||
|
||||
@@ -38,32 +38,13 @@ type Assumption struct {
|
||||
// is a sub-step of that phase. DependsOn orders siblings and is advisory: the
|
||||
// todo projection is serial, so a dependency becomes order, not gating.
|
||||
type Step struct {
|
||||
ID string `json:"id,omitempty"`
|
||||
ParentID string `json:"parent_id,omitempty"`
|
||||
Title string `json:"title"`
|
||||
DependsOn []string `json:"depends_on,omitempty"`
|
||||
VerifiedFiles []string `json:"verified_files,omitempty"` // paths the planner read
|
||||
CandidateFiles []string `json:"candidate_files,omitempty"` // paths the planner inferred
|
||||
Acceptance []Criterion `json:"acceptance,omitempty"`
|
||||
Verification []Verification `json:"verification,omitempty"`
|
||||
Risks []string `json:"risks,omitempty"`
|
||||
}
|
||||
|
||||
// Criterion is one acceptance criterion. Regression marks must-keep-passing
|
||||
// behavior; Optional marks a nice-to-have that never blocks completion. ID is
|
||||
// host-assigned so it can key an evidence requirement downstream.
|
||||
type Criterion struct {
|
||||
ID string `json:"-"`
|
||||
Text string `json:"text"`
|
||||
Regression bool `json:"regression,omitempty"`
|
||||
Optional bool `json:"optional,omitempty"`
|
||||
}
|
||||
|
||||
// Verification is a command-level check. An empty Command accepts any
|
||||
// delivery-verification command; Expect describes what a pass looks like.
|
||||
type Verification struct {
|
||||
Command string `json:"command,omitempty"`
|
||||
Expect string `json:"expect,omitempty"`
|
||||
ID string `json:"id,omitempty"`
|
||||
ParentID string `json:"parent_id,omitempty"`
|
||||
Title string `json:"title"`
|
||||
DependsOn []string `json:"depends_on,omitempty"`
|
||||
VerifiedFiles []string `json:"verified_files,omitempty"` // paths the planner read
|
||||
CandidateFiles []string `json:"candidate_files,omitempty"` // paths the planner inferred
|
||||
Risks []string `json:"risks,omitempty"`
|
||||
}
|
||||
|
||||
// Normalize returns a canonical copy: trimmed text, dropped empty entries,
|
||||
@@ -102,13 +83,10 @@ func normalizeSteps(steps []Step) []Step {
|
||||
DependsOn: cleanStrings(s.DependsOn),
|
||||
VerifiedFiles: cleanStrings(s.VerifiedFiles),
|
||||
CandidateFiles: cleanStrings(s.CandidateFiles),
|
||||
Acceptance: cleanCriteria(s.Acceptance),
|
||||
Verification: cleanVerifications(s.Verification),
|
||||
Risks: cleanStrings(s.Risks),
|
||||
})
|
||||
}
|
||||
assignStepIDs(out)
|
||||
assignCriterionIDs(out)
|
||||
repairParents(out)
|
||||
repairDependencies(out)
|
||||
return out
|
||||
@@ -144,37 +122,6 @@ func assignStepIDs(steps []Step) {
|
||||
}
|
||||
}
|
||||
|
||||
func assignCriterionIDs(steps []Step) {
|
||||
used := make(map[string]bool)
|
||||
for i := range steps {
|
||||
for j := range steps[i].Acceptance {
|
||||
id := steps[i].Acceptance[j].ID
|
||||
if id == "" || used[id] {
|
||||
steps[i].Acceptance[j].ID = ""
|
||||
continue
|
||||
}
|
||||
used[id] = true
|
||||
}
|
||||
}
|
||||
next := 1
|
||||
for i := range steps {
|
||||
for j := range steps[i].Acceptance {
|
||||
if steps[i].Acceptance[j].ID != "" {
|
||||
continue
|
||||
}
|
||||
for {
|
||||
id := fmt.Sprintf("c%d", next)
|
||||
next++
|
||||
if !used[id] {
|
||||
used[id] = true
|
||||
steps[i].Acceptance[j].ID = id
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// repairParents rewrites every parent reference to the top-level phase the step
|
||||
// actually belongs to, so a normalized plan is literally two levels deep and
|
||||
// Ordered reads the field instead of re-deriving it.
|
||||
@@ -266,39 +213,3 @@ func cleanStrings(in []string) []string {
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func cleanCriteria(in []Criterion) []Criterion {
|
||||
out := make([]Criterion, 0, len(in))
|
||||
for _, c := range in {
|
||||
text := strings.TrimSpace(c.Text)
|
||||
if text == "" {
|
||||
continue
|
||||
}
|
||||
out = append(out, Criterion{
|
||||
ID: strings.TrimSpace(c.ID),
|
||||
Text: text,
|
||||
Regression: c.Regression,
|
||||
Optional: c.Optional,
|
||||
})
|
||||
}
|
||||
if len(out) == 0 {
|
||||
return nil
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func cleanVerifications(in []Verification) []Verification {
|
||||
out := make([]Verification, 0, len(in))
|
||||
for _, v := range in {
|
||||
command := strings.TrimSpace(v.Command)
|
||||
expect := strings.TrimSpace(v.Expect)
|
||||
if command == "" && expect == "" {
|
||||
continue
|
||||
}
|
||||
out = append(out, Verification{Command: command, Expect: expect})
|
||||
}
|
||||
if len(out) == 0 {
|
||||
return nil
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -38,26 +38,6 @@ func TestNormalizeReassignsDuplicateStepIDs(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestNormalizeAssignsCriterionIDsAcrossSteps(t *testing.T) {
|
||||
p := Plan{Objective: "o", Steps: []Step{
|
||||
{ID: "a", Title: "first", Acceptance: []Criterion{{Text: "one"}}},
|
||||
{ID: "b", Title: "second", Acceptance: []Criterion{{Text: "two"}, {Text: "three"}}},
|
||||
}}.Normalize()
|
||||
|
||||
seen := map[string]bool{}
|
||||
for _, step := range p.Steps {
|
||||
for _, c := range step.Acceptance {
|
||||
if c.ID == "" {
|
||||
t.Fatalf("criterion %q has no id", c.Text)
|
||||
}
|
||||
if seen[c.ID] {
|
||||
t.Fatalf("criterion id %q reused across steps", c.ID)
|
||||
}
|
||||
seen[c.ID] = true
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestNormalizeDropsEmptyEntriesAndDeduplicates(t *testing.T) {
|
||||
p := Plan{
|
||||
Objective: " ship it ",
|
||||
@@ -65,9 +45,7 @@ func TestNormalizeDropsEmptyEntriesAndDeduplicates(t *testing.T) {
|
||||
Assumptions: []Assumption{{Text: " "}, {Text: "cache is warm", Confirm: " check hit rate "}},
|
||||
Steps: []Step{
|
||||
{Title: " "},
|
||||
{Title: " do it ", VerifiedFiles: []string{"a.go", "a.go", " b.go "}, Risks: []string{""},
|
||||
Acceptance: []Criterion{{Text: " "}},
|
||||
Verification: []Verification{{Command: " ", Expect: " "}}},
|
||||
{Title: " do it ", VerifiedFiles: []string{"a.go", "a.go", " b.go "}, Risks: []string{""}},
|
||||
},
|
||||
}.Normalize()
|
||||
|
||||
@@ -87,7 +65,7 @@ func TestNormalizeDropsEmptyEntriesAndDeduplicates(t *testing.T) {
|
||||
if len(step.VerifiedFiles) != 2 || step.VerifiedFiles[1] != "b.go" {
|
||||
t.Fatalf("verified files = %v", step.VerifiedFiles)
|
||||
}
|
||||
if step.Risks != nil || step.Acceptance != nil || step.Verification != nil {
|
||||
if step.Risks != nil {
|
||||
t.Fatalf("empty entries survived: %+v", step)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -88,28 +88,6 @@ func renderDetail(b *strings.Builder, step Step, indent string) {
|
||||
if len(step.CandidateFiles) > 0 {
|
||||
fmt.Fprintf(b, "%scandidate: %s\n", indent, strings.Join(step.CandidateFiles, ", "))
|
||||
}
|
||||
for _, c := range step.Acceptance {
|
||||
label := "accept"
|
||||
if c.Regression {
|
||||
label = "regression"
|
||||
}
|
||||
if c.Optional {
|
||||
label += " (optional)"
|
||||
}
|
||||
// The id is rendered because a proof has to cite it: a criterion the
|
||||
// executor cannot name is one it cannot satisfy.
|
||||
fmt.Fprintf(b, "%s%s [%s]: %s\n", indent, label, c.ID, c.Text)
|
||||
}
|
||||
for _, v := range step.Verification {
|
||||
command := v.Command
|
||||
if command == "" {
|
||||
command = "any verification command"
|
||||
}
|
||||
if v.Expect != "" {
|
||||
command += " — " + v.Expect
|
||||
}
|
||||
fmt.Fprintf(b, "%sverify: %s\n", indent, command)
|
||||
}
|
||||
for _, risk := range step.Risks {
|
||||
fmt.Fprintf(b, "%srisk: %s\n", indent, risk)
|
||||
}
|
||||
|
||||
@@ -49,8 +49,6 @@ func richPlan() Plan {
|
||||
},
|
||||
{
|
||||
ID: "s1", ParentID: "p1", Title: "extend cacheKey",
|
||||
Acceptance: []Criterion{{Text: "two model refs never share an entry"}, {Text: "existing hits keep hitting", Regression: true}},
|
||||
Verification: []Verification{{Command: "go test ./internal/contract/provider/", Expect: "all green"}},
|
||||
},
|
||||
{ID: "p2", Title: "record the hit rate"},
|
||||
},
|
||||
@@ -89,9 +87,6 @@ func TestRenderKeepsStepDetailOffTheList(t *testing.T) {
|
||||
" verified: internal/contract/provider/cache.go",
|
||||
" candidate: internal/assembly/boot/boot.go",
|
||||
" risk: existing warm caches invalidate once",
|
||||
" accept [c1]: two model refs never share an entry",
|
||||
" regression [c2]: existing hits keep hitting",
|
||||
" verify: go test ./internal/contract/provider/ — all green",
|
||||
} {
|
||||
if !strings.Contains(out, want+"\n") {
|
||||
t.Errorf("rendered plan missing detail line %q:\n%s", want, out)
|
||||
|
||||
@@ -1,27 +1,3 @@
|
||||
package evidence
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"reasonix/internal/contract/provider"
|
||||
)
|
||||
|
||||
func TestPathsProvenUseNormalizedArguments(t *testing.T) {
|
||||
const raw = `{"arguments":{"path":"a.go"}}`
|
||||
msgs := []provider.Message{
|
||||
{Role: provider.RoleAssistant, ToolCalls: []provider.ToolCall{{
|
||||
ID: "r", Name: "read_file", Arguments: raw,
|
||||
ArgumentNormalization: &provider.ArgumentNormalization{
|
||||
Reason: provider.ArgumentEnvelopeUnwrapped, Original: raw, Effective: `{"path":"a.go"}`,
|
||||
},
|
||||
}}},
|
||||
{Role: provider.RoleTool, ToolCallID: "r", Content: "contents"},
|
||||
}
|
||||
facts := func(string) ToolFacts { return ToolFacts{ReadOnly: true} }
|
||||
if !PathsProvenInSession(msgs, []string{"a.go"}, false, facts) {
|
||||
t.Fatal("replay lost the normalized read path")
|
||||
}
|
||||
if PathsProvenInSession(msgs, []string{"b.go"}, false, facts) {
|
||||
t.Fatal("normalization invented evidence for an unread path")
|
||||
}
|
||||
}
|
||||
import ()
|
||||
@@ -2,7 +2,6 @@ package evidence
|
||||
|
||||
import (
|
||||
"context"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
)
|
||||
|
||||
@@ -13,43 +12,3 @@ type workspaceRootKey struct{}
|
||||
func WithWorkspaceRoot(ctx context.Context, root string) context.Context {
|
||||
return context.WithValue(ctx, workspaceRootKey{}, strings.TrimSpace(root))
|
||||
}
|
||||
|
||||
// WorkspaceRootFromContext returns the root WithWorkspaceRoot attached, or "".
|
||||
func WorkspaceRootFromContext(ctx context.Context) string {
|
||||
root, _ := ctx.Value(workspaceRootKey{}).(string)
|
||||
return root
|
||||
}
|
||||
|
||||
// CitationForms returns the ledger identities of the one file a citation
|
||||
// names. File tools resolve a relative path against the workspace, so a
|
||||
// relative citation and the absolute path under root are the same file; a
|
||||
// path outside root has only its own form.
|
||||
func CitationForms(root, cited string) []string {
|
||||
own := normalizePath(cited)
|
||||
if own == "" {
|
||||
return nil
|
||||
}
|
||||
forms := []string{own}
|
||||
if root == "" {
|
||||
return forms
|
||||
}
|
||||
cited = filepath.Clean(filepath.FromSlash(strings.ReplaceAll(strings.TrimSpace(cited), `\`, `/`)))
|
||||
root = filepath.Clean(root)
|
||||
if !filepath.IsAbs(cited) {
|
||||
if inside(cited) {
|
||||
forms = append(forms, normalizePath(filepath.Join(root, cited)))
|
||||
}
|
||||
return forms
|
||||
}
|
||||
if rel, err := filepath.Rel(root, cited); err == nil && inside(rel) {
|
||||
forms = append(forms, normalizePath(rel))
|
||||
}
|
||||
return forms
|
||||
}
|
||||
|
||||
// inside reports whether a relative path stays below the directory it is
|
||||
// relative to.
|
||||
func inside(rel string) bool {
|
||||
return rel != "." && rel != ".." && !filepath.IsAbs(rel) &&
|
||||
!strings.HasPrefix(rel, ".."+string(filepath.Separator))
|
||||
}
|
||||
@@ -430,13 +430,6 @@ func WithDeliveryProfile(ctx context.Context) context.Context {
|
||||
return context.WithValue(ctx, deliveryProfileKey{}, true)
|
||||
}
|
||||
|
||||
// DeliveryProfileFromContext reports whether the current tool call must produce
|
||||
// evidence that the delivery final-readiness gate can accept.
|
||||
func DeliveryProfileFromContext(ctx context.Context) bool {
|
||||
enabled, _ := ctx.Value(deliveryProfileKey{}).(bool)
|
||||
return enabled
|
||||
}
|
||||
|
||||
// WithSessionMessages attaches a lazy transcript accessor so verifyStepEvidence
|
||||
// can fall back to scanning the conversation when the per-turn ledger misses a
|
||||
// command (cross-turn references, non-bash tool calls, truncated command
|
||||
@@ -447,16 +440,6 @@ func WithSessionMessages(ctx context.Context, snapshot func() []provider.Message
|
||||
return context.WithValue(ctx, sessionMessagesKey{}, snapshot)
|
||||
}
|
||||
|
||||
// SessionMessagesFromContext resolves the transcript accessor attached by
|
||||
// WithSessionMessages, taking the snapshot at call time.
|
||||
func SessionMessagesFromContext(ctx context.Context) ([]provider.Message, bool) {
|
||||
snapshot, ok := ctx.Value(sessionMessagesKey{}).(func() []provider.Message)
|
||||
if !ok || snapshot == nil {
|
||||
return nil, false
|
||||
}
|
||||
return snapshot(), true
|
||||
}
|
||||
|
||||
// WithTodoState attaches the host's canonical task list to a tool call. The
|
||||
// per-turn ledger resets between user messages, while unfinished tasks remain
|
||||
// active across those turns.
|
||||
@@ -470,53 +453,6 @@ func TodoStateFromContext(ctx context.Context) ([]TodoItem, bool) {
|
||||
return append([]TodoItem(nil), todos...), ok
|
||||
}
|
||||
|
||||
// PathsProvenInSession reports whether every path is covered by a successful
|
||||
// (non-errored) tool call somewhere in msgs — the cross-turn fallback for diff
|
||||
// and files evidence, whose per-turn ledger resets each turn. wantWrite
|
||||
// restricts to writer tools (diff); false accepts either. A replay carries only
|
||||
// names, so facts resolves each to the contracts its tool declares.
|
||||
func PathsProvenInSession(msgs []provider.Message, paths []string, wantWrite bool, facts func(string) ToolFacts) bool {
|
||||
wanted := pathSet(normalizePaths(paths))
|
||||
if len(wanted) == 0 {
|
||||
return false
|
||||
}
|
||||
failed := failedSessionCallIDs(msgs)
|
||||
found := map[string]bool{}
|
||||
for _, msg := range msgs {
|
||||
for _, tc := range msg.ToolCalls {
|
||||
if failed[tc.ID] {
|
||||
continue
|
||||
}
|
||||
r := ReceiptFromToolCall(tc.Name, json.RawMessage(tc.ExecutionArguments()), true, facts(tc.Name))
|
||||
if wantWrite && !r.Write {
|
||||
continue
|
||||
}
|
||||
if !wantWrite && !r.Read && !r.Write {
|
||||
continue
|
||||
}
|
||||
for _, p := range normalizePaths(r.Paths) {
|
||||
if _, ok := wanted[p]; ok {
|
||||
found[p] = true
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return len(found) == len(wanted)
|
||||
}
|
||||
|
||||
func failedSessionCallIDs(msgs []provider.Message) map[string]bool {
|
||||
failed := map[string]bool{}
|
||||
for _, msg := range msgs {
|
||||
if msg.Role != provider.RoleTool || msg.ToolCallID == "" {
|
||||
continue
|
||||
}
|
||||
if strings.HasPrefix(msg.Content, "error:") || strings.HasPrefix(msg.Content, "blocked:") {
|
||||
failed[msg.ToolCallID] = true
|
||||
}
|
||||
}
|
||||
return failed
|
||||
}
|
||||
|
||||
func ReceiptFromToolCall(toolName string, args json.RawMessage, success bool, facts ToolFacts) Receipt {
|
||||
class := ToolCallMutationClassWith(toolName, args, facts)
|
||||
r := Receipt{
|
||||
@@ -539,7 +475,7 @@ func ReceiptFromToolCall(toolName string, args json.RawMessage, success bool, fa
|
||||
}
|
||||
if toolName == "complete_step" {
|
||||
r.Step = completeStepIdentity(fields)
|
||||
r.StepProof = completeStepHasProof(fields)
|
||||
r.StepProof = strings.TrimSpace(stringField(fields, "result")) != ""
|
||||
r.CitedChecks = completeStepCitedChecks(fields)
|
||||
}
|
||||
if toolName == "todo_write" {
|
||||
@@ -611,11 +547,6 @@ func ToolCallRequiresDeliveryCriteria(toolName string, args json.RawMessage, rea
|
||||
return bashCommandIsVerification(stringField(fields, "command"))
|
||||
}
|
||||
|
||||
type verificationCommandRecommendation struct {
|
||||
label string
|
||||
examples []string
|
||||
}
|
||||
|
||||
// writeOutputFlags are test-runner and linter flags that write snapshot,
|
||||
// report, or profile files. Snapshot flags rewrite checked-in fixtures (the
|
||||
// --update/-u class rejected above); the others write explicit output paths.
|
||||
|
||||
@@ -4,8 +4,6 @@ import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"reflect"
|
||||
"runtime"
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
@@ -309,24 +307,6 @@ func TestReceiptFromToolCallExtractsCompleteStep(t *testing.T) {
|
||||
if missingResult.StepProof {
|
||||
t.Fatalf("complete_step without result should not count as proof: %+v", missingResult)
|
||||
}
|
||||
|
||||
missingCommand := ReceiptFromToolCall("complete_step", json.RawMessage(`{
|
||||
"step":"Add parser",
|
||||
"result":"parser added",
|
||||
"evidence":[{"kind":"verification","summary":"checked manually"}]
|
||||
}`), false, ToolFacts{ReadOnly: true})
|
||||
if missingCommand.StepProof {
|
||||
t.Fatalf("verification evidence without command should not count as proof: %+v", missingCommand)
|
||||
}
|
||||
|
||||
emptyProof := ReceiptFromToolCall("complete_step", json.RawMessage(`{
|
||||
"step":"Add parser",
|
||||
"result":"parser added",
|
||||
"evidence":[]
|
||||
}`), false, ToolFacts{ReadOnly: true})
|
||||
if emptyProof.StepProof {
|
||||
t.Fatalf("empty complete_step evidence should not count as proof: %+v", emptyProof)
|
||||
}
|
||||
}
|
||||
|
||||
func TestReceiptFromToolCallExtractsCompleteStepIndex(t *testing.T) {
|
||||
@@ -1321,17 +1301,3 @@ func TestLedgerDeliverySignoffRejectsInspectionCommandMasqueradingAsVerification
|
||||
t.Fatal("inspection-only git status must not count as delivery verification")
|
||||
}
|
||||
}
|
||||
|
||||
func TestCitationFormsAnchorAWindowsCitationToTheWorkspace(t *testing.T) {
|
||||
if runtime.GOOS != "windows" {
|
||||
t.Skip("drive-letter paths are absolute only on Windows")
|
||||
}
|
||||
forms := CitationForms(`C:\ws`, "src/a.go")
|
||||
if !slices.Contains(forms, NormalizePath(`C:\ws\src\a.go`)) {
|
||||
t.Fatalf("forms %q miss the workspace path", forms)
|
||||
}
|
||||
forms = CitationForms(`C:\ws`, `c:\WS\src\a.go`)
|
||||
if !slices.Contains(forms, NormalizePath("src/a.go")) {
|
||||
t.Fatalf("forms %q miss the relative path", forms)
|
||||
}
|
||||
}
|
||||
@@ -1,11 +1,8 @@
|
||||
package evidence
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"slices"
|
||||
"testing"
|
||||
|
||||
"reasonix/internal/contract/provider"
|
||||
)
|
||||
|
||||
func recorded(receipts ...Receipt) *Ledger {
|
||||
@@ -65,35 +62,6 @@ func TestReceiptProgressSummaryCountsOnlySuccessfulWork(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestPathsProvenInSessionNeedsEveryPathAndIgnoresFailedCalls(t *testing.T) {
|
||||
call := func(id, name, path string) provider.ToolCall {
|
||||
args, _ := json.Marshal(map[string]string{"path": path})
|
||||
return provider.ToolCall{ID: id, Name: name, Arguments: string(args)}
|
||||
}
|
||||
facts := func(name string) ToolFacts {
|
||||
return ToolFacts{ReadOnly: name == "read_file", WritesNamedPaths: name == "edit_file"}
|
||||
}
|
||||
msgs := []provider.Message{
|
||||
{Role: provider.RoleAssistant, ToolCalls: []provider.ToolCall{call("1", "edit_file", "a.go"), call("2", "edit_file", "b.go"), call("3", "read_file", "c.go")}},
|
||||
{Role: provider.RoleTool, ToolCallID: "2", Content: "error: permission denied"},
|
||||
}
|
||||
if !PathsProvenInSession(msgs, []string{"a.go"}, true, facts) {
|
||||
t.Fatal("a.go was written by a call that did not fail")
|
||||
}
|
||||
if PathsProvenInSession(msgs, []string{"a.go", "b.go"}, true, facts) {
|
||||
t.Fatal("b.go's write failed, so both paths are not proven")
|
||||
}
|
||||
if PathsProvenInSession(msgs, []string{"c.go"}, true, facts) {
|
||||
t.Fatal("a read does not prove a write")
|
||||
}
|
||||
if !PathsProvenInSession(msgs, []string{"c.go"}, false, facts) {
|
||||
t.Fatal("a read proves the path was looked at when writes are not required")
|
||||
}
|
||||
if PathsProvenInSession(msgs, nil, false, facts) {
|
||||
t.Fatal("no paths proves nothing")
|
||||
}
|
||||
}
|
||||
|
||||
func TestHasSuccessfulBashMentioningPathsNeedsEveryPathInASuccessfulCommand(t *testing.T) {
|
||||
l := recorded(
|
||||
Receipt{ToolName: "bash", Success: true, Command: `seq 1 3 > Out/Data.txt`},
|
||||
|
||||
@@ -2,7 +2,6 @@ package evidence
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"testing"
|
||||
)
|
||||
@@ -43,30 +42,3 @@ func TestTouchedPathsAreRecentFirstDistinctBoundedAndSayWhetherTheyWereWritten(t
|
||||
t.Fatal("a nil ledger and a zero limit touch nothing")
|
||||
}
|
||||
}
|
||||
|
||||
func TestCitationFormsNamesTheSameFileRelativeAndAbsoluteButNeverOneOutsideTheRoot(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
abs := filepath.Join(root, "pkg", "a.go")
|
||||
rel := filepath.Join("pkg", "a.go")
|
||||
for name, tc := range map[string]struct {
|
||||
cited string
|
||||
want []string
|
||||
}{
|
||||
"relative inside": {rel, []string{normalizePath(rel), normalizePath(abs)}},
|
||||
"absolute inside": {abs, []string{normalizePath(abs), normalizePath(rel)}},
|
||||
"relative escaping": {filepath.Join("..", "x.go"), []string{normalizePath(filepath.Join("..", "x.go"))}},
|
||||
"absolute outside": {filepath.Join(filepath.Dir(root), "other", "x.go"), []string{normalizePath(filepath.Join(filepath.Dir(root), "other", "x.go"))}},
|
||||
"the root itself": {root, []string{normalizePath(root)}},
|
||||
"backslash relative": {"pkg\\a.go", []string{normalizePath(rel), normalizePath(abs)}},
|
||||
} {
|
||||
if got := CitationForms(root, tc.cited); !slices.Equal(got, tc.want) {
|
||||
t.Errorf("%s: CitationForms(%q) = %v, want %v", name, tc.cited, got, tc.want)
|
||||
}
|
||||
}
|
||||
if got := CitationForms("", rel); !slices.Equal(got, []string{normalizePath(rel)}) {
|
||||
t.Errorf("no root: %v", got)
|
||||
}
|
||||
if got := CitationForms(root, " "); got != nil {
|
||||
t.Errorf("a blank citation names no file: %v", got)
|
||||
}
|
||||
}
|
||||
@@ -5,7 +5,6 @@ package evidence
|
||||
// never carried one, where wording and position are all there is to go on.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
@@ -106,21 +105,3 @@ func todoContentRelates(todo TodoItem, match TodoStepMatch) bool {
|
||||
func textOverlaps(a, b string) bool {
|
||||
return stepTextContains(normalizeStepText(a), normalizeStepText(b))
|
||||
}
|
||||
|
||||
// WithAcceptanceCriteria carries the ids of the approved plan's criteria into a
|
||||
// tool call, so a proof citing one can be checked against the plan the user
|
||||
// approved instead of resolving into nothing.
|
||||
func WithAcceptanceCriteria(ctx context.Context, ids []string) context.Context {
|
||||
if len(ids) == 0 {
|
||||
return ctx
|
||||
}
|
||||
return context.WithValue(ctx, acceptanceCriteriaKey{}, append([]string(nil), ids...))
|
||||
}
|
||||
|
||||
// AcceptanceCriteriaFromContext returns the approved plan's criterion ids.
|
||||
func AcceptanceCriteriaFromContext(ctx context.Context) ([]string, bool) {
|
||||
ids, ok := ctx.Value(acceptanceCriteriaKey{}).([]string)
|
||||
return ids, ok && len(ids) > 0
|
||||
}
|
||||
|
||||
type acceptanceCriteriaKey struct{}
|
||||
@@ -526,46 +526,6 @@ func completeStepCitedChecks(fields map[string]json.RawMessage) []string {
|
||||
return out
|
||||
}
|
||||
|
||||
func completeStepHasProof(fields map[string]json.RawMessage) bool {
|
||||
if strings.TrimSpace(stringField(fields, "result")) == "" {
|
||||
return false
|
||||
}
|
||||
raw, ok := fields["evidence"]
|
||||
if !ok {
|
||||
return false
|
||||
}
|
||||
var items []struct {
|
||||
Kind string `json:"kind"`
|
||||
Summary string `json:"summary"`
|
||||
Command string `json:"command"`
|
||||
Paths []string `json:"paths"`
|
||||
}
|
||||
if err := json.Unmarshal(raw, &items); err != nil || len(items) == 0 {
|
||||
return false
|
||||
}
|
||||
for _, item := range items {
|
||||
kind := strings.TrimSpace(item.Kind)
|
||||
if kind == "" || strings.TrimSpace(item.Summary) == "" {
|
||||
return false
|
||||
}
|
||||
switch kind {
|
||||
case "verification":
|
||||
if strings.TrimSpace(item.Command) == "" {
|
||||
return false
|
||||
}
|
||||
case "diff", "files":
|
||||
if len(normalizePaths(item.Paths)) == 0 {
|
||||
return false
|
||||
}
|
||||
case "manual":
|
||||
// Summary is enough for manual evidence.
|
||||
default:
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func normalizeTodos(todos []TodoItem) []TodoItem {
|
||||
out := make([]TodoItem, 0, len(todos))
|
||||
for _, t := range todos {
|
||||
|
||||
@@ -213,51 +213,6 @@ func IsDeliveryVerificationCommand(command string) bool {
|
||||
return bashCommandIsVerification(command)
|
||||
}
|
||||
|
||||
// verificationCommandRecommendations is the single source for the concrete
|
||||
// model-readable examples and the family labels used to diagnose test failures.
|
||||
// It is intentionally a safe recommended subset rather than an exhaustive
|
||||
// rendering of bashSegmentIsVerification: accepted commands that may install
|
||||
// dependencies or create workspace outputs should not be suggested as the
|
||||
// first recovery action.
|
||||
func verificationCommandRecommendations() []verificationCommandRecommendation {
|
||||
return []verificationCommandRecommendation{
|
||||
{label: "go test|vet", examples: []string{"go test ./...", "go vet ./..."}},
|
||||
{label: "git diff --check", examples: []string{"git diff --check"}},
|
||||
{label: "pytest/py.test", examples: []string{"pytest tests/", "py.test tests/"}},
|
||||
{label: "gotestsum", examples: []string{"gotestsum"}},
|
||||
{label: "staticcheck", examples: []string{"staticcheck ./..."}},
|
||||
{label: "golangci-lint", examples: []string{"golangci-lint run"}},
|
||||
{label: "tsc", examples: []string{"tsc --noEmit"}},
|
||||
{label: "mypy (no report flag)", examples: []string{"mypy src/"}},
|
||||
{label: "npm|pnpm|yarn|bun test|check|lint", examples: []string{"npm test", "pnpm check", "yarn lint", "bun test"}},
|
||||
{label: "npm run test|check|lint|typecheck", examples: []string{"npm run typecheck"}},
|
||||
{label: "cargo test|check|clippy", examples: []string{"cargo test", "cargo check", "cargo clippy"}},
|
||||
{label: "node --check|--test", examples: []string{"node --check index.js", "node --test"}},
|
||||
{label: "make|just test|check|lint|verify|ci", examples: []string{"make test", "just verify"}},
|
||||
{label: "python -m pytest|unittest", examples: []string{"python -m pytest", "python -m unittest"}},
|
||||
{label: "dotnet test", examples: []string{"dotnet test"}},
|
||||
{label: "swift test", examples: []string{"swift test"}},
|
||||
{label: "mvn|gradle test|check|verify", examples: []string{"mvn test", "gradle check"}},
|
||||
}
|
||||
}
|
||||
|
||||
// VerificationCommandSummary returns compact, model-readable recovery
|
||||
// guidance. It lists only recommended command families that the classifier
|
||||
// accepts, while omitting known self-installing and direct workspace-output
|
||||
// command forms from first-line guidance.
|
||||
func VerificationCommandSummary() string {
|
||||
recommendations := verificationCommandRecommendations()
|
||||
commands := make([]string, 0, len(recommendations))
|
||||
for _, recommendation := range recommendations {
|
||||
commands = append(commands, recommendation.examples...)
|
||||
}
|
||||
return "recommended recognized verification commands: " + strings.Join(commands, ", ") + ". " +
|
||||
"Read-only inspection commands (grep/find/cat/wc/head/tail) are NOT verification; " +
|
||||
"inline interpreters (node -e, python -c) are blocked in delivery mode. " +
|
||||
"A read-only extraction pipeline ending in a recognized verifier " +
|
||||
"(e.g. tail -n +1 file | node --check -) is accepted."
|
||||
}
|
||||
|
||||
func bashSegmentIsVerification(fields []string) bool {
|
||||
if len(fields) == 0 {
|
||||
return false
|
||||
|
||||
@@ -1,306 +1,9 @@
|
||||
package evidence
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// TestVerificationCommandSummaryRecommendationsAreRecognized guards the
|
||||
// single structured source used to render the summary and exercise the
|
||||
// classifier. Every concrete example must render and remain accepted.
|
||||
func TestVerificationCommandSummaryRecommendationsAreRecognized(t *testing.T) {
|
||||
summary := VerificationCommandSummary()
|
||||
for _, recommendation := range verificationCommandRecommendations() {
|
||||
for _, command := range recommendation.examples {
|
||||
if !strings.Contains(summary, command) {
|
||||
t.Errorf("summary family %q missing concrete example %q: %s", recommendation.label, command, summary)
|
||||
}
|
||||
if !IsDeliveryVerificationCommand(command) {
|
||||
t.Errorf("summary family %q advertises %q, but classifier rejects it", recommendation.label, command)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestGoBuildAlwaysCountsAsMutation(t *testing.T) {
|
||||
// Even ./... can expand to one main package and write a root executable.
|
||||
// The static classifier cannot know package expansion or inherited GOFLAGS,
|
||||
// so every go build form must fail closed as a mutation.
|
||||
for _, command := range []string{
|
||||
"go build",
|
||||
"go build .",
|
||||
"go build ./cmd/reasonix",
|
||||
"go build -race ./cmd/reasonix",
|
||||
"go build ./...",
|
||||
"go build -tags integration ./...",
|
||||
"go build -tags=integration ./...",
|
||||
"go build -race -trimpath ./...",
|
||||
"go build -p 2 -mod=readonly ./...",
|
||||
"go build -buildvcs ./...",
|
||||
"go build -buildvcs=auto -- ./...",
|
||||
"go build -tags ./... ./cmd/reasonix",
|
||||
"go build -coverpkg ./... ./cmd/reasonix",
|
||||
"go build -pkgdir ./... ./cmd/reasonix",
|
||||
"go build -pkgdir=./... ./cmd/reasonix",
|
||||
"go build -o reasonix ./...",
|
||||
"go build -o=reasonix ./...",
|
||||
"go build -mod=mod ./...",
|
||||
"go build -n ./...",
|
||||
"go build -work ./...",
|
||||
"go build -future-flag ./...",
|
||||
"go build ./... -o reasonix",
|
||||
} {
|
||||
if IsDeliveryVerificationCommand(command) {
|
||||
t.Errorf("%q must not count as non-mutating verification", command)
|
||||
}
|
||||
args, err := json.Marshal(map[string]string{"command": command})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !ToolCallMutates("bash", args, false) {
|
||||
t.Errorf("%q must be classified as a mutation", command)
|
||||
}
|
||||
}
|
||||
if strings.Contains(VerificationCommandSummary(), "go build") {
|
||||
t.Fatal("recovery summary must not recommend go build as a first-line verifier")
|
||||
}
|
||||
}
|
||||
|
||||
func TestTSCVerificationRequiresExplicitNoEmit(t *testing.T) {
|
||||
for _, command := range []string{
|
||||
"tsc --noEmit",
|
||||
"tsc --noEmit=true",
|
||||
"tsc --project tsconfig.json --noEmit",
|
||||
"tsc --noEmit --listFiles",
|
||||
"tsc --noEmit --pretty false",
|
||||
} {
|
||||
if !IsDeliveryVerificationCommand(command) {
|
||||
t.Errorf("%q should be recognized as an explicit no-emit type check", command)
|
||||
}
|
||||
args, err := json.Marshal(map[string]string{"command": command})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if ToolCallMutates("bash", args, false) {
|
||||
t.Errorf("%q should remain a non-mutating verification", command)
|
||||
}
|
||||
}
|
||||
|
||||
for _, command := range []string{
|
||||
"tsc",
|
||||
"tsc --outDir dist",
|
||||
"tsc --noEmit=false",
|
||||
"tsc --noEmit false",
|
||||
"tsc --noEmit=true --noEmit=false",
|
||||
"tsc --noEmit --incremental --tsBuildInfoFile victim.ts",
|
||||
"tsc --noEmit --incremental --tsBuildInfoFile=victim.ts",
|
||||
"tsc --noEmit --generateTrace trace-dir",
|
||||
"tsc --noEmit --generateTrace=trace-dir",
|
||||
"tsc --noEmit --generateCpuProfile profile.cpuprofile",
|
||||
"tsc --noEmit --generateCpuProfile=profile.cpuprofile",
|
||||
"tsc --noEmit --init",
|
||||
"tsc --noEmit --help",
|
||||
"tsc --noEmit -h",
|
||||
"tsc --noEmit -?",
|
||||
"tsc --noEmit --all",
|
||||
"tsc --noEmit --version",
|
||||
"tsc --noEmit -v",
|
||||
"tsc --noEmit --showConfig",
|
||||
"tsc --noEmit --listFilesOnly",
|
||||
"tsc --noEmit --noCheck",
|
||||
"tsc --noEmit --watch",
|
||||
"tsc --noEmit -w",
|
||||
"tsc --build --noEmit",
|
||||
"tsc -b --noEmit",
|
||||
"tsc --build --clean --noEmit",
|
||||
} {
|
||||
if IsDeliveryVerificationCommand(command) {
|
||||
t.Errorf("%q is not a bounded no-emit type check and must not count as verification", command)
|
||||
}
|
||||
args, err := json.Marshal(map[string]string{"command": command})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !ToolCallMutates("bash", args, false) {
|
||||
t.Errorf("%q must fail closed as a mutation", command)
|
||||
}
|
||||
}
|
||||
|
||||
if !strings.Contains(VerificationCommandSummary(), "tsc --noEmit") {
|
||||
t.Fatal("recovery summary must render the concrete no-emit TypeScript command")
|
||||
}
|
||||
}
|
||||
|
||||
func TestSwiftTestRecognizedAsVerification(t *testing.T) {
|
||||
// swift test runs the SwiftPM test suite; the build cache lands in the
|
||||
// package's own .build directory, mirroring cargo test acceptance. Other
|
||||
// swift subcommands emit binaries, run arbitrary code, or mutate the
|
||||
// package graph and must fail closed as mutations.
|
||||
for _, command := range []string{
|
||||
"swift test",
|
||||
"swift test --parallel",
|
||||
"swift test --filter SomeTests",
|
||||
"swift test --enable-code-coverage",
|
||||
} {
|
||||
if !IsDeliveryVerificationCommand(command) {
|
||||
t.Errorf("%q should be recognized as a read-only Swift test verifier", command)
|
||||
}
|
||||
args, err := json.Marshal(map[string]string{"command": command})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if ToolCallMutates("bash", args, false) {
|
||||
t.Errorf("%q should remain a non-mutating verification", command)
|
||||
}
|
||||
}
|
||||
|
||||
for _, command := range []string{
|
||||
"swift",
|
||||
"swift build",
|
||||
"swift build -c release",
|
||||
"swift run",
|
||||
"swift package resolve",
|
||||
"swift package update",
|
||||
"swift test --xunit-output report.xml",
|
||||
"swift test --xunit-output=report.xml",
|
||||
"swift test --scratch-path /tmp/out",
|
||||
"swift test --build-path /tmp/out",
|
||||
"swift test --build-path=/tmp/out",
|
||||
"swift test --event-stream-output-path /tmp/events.json",
|
||||
"swift test --experimental-event-stream-output /tmp/events.json",
|
||||
"swift test --attachments-path /tmp/attachments",
|
||||
"swift test --experimental-attachments-path /tmp/attachments",
|
||||
"swift test --cache-path /tmp/cache",
|
||||
"swift test --help",
|
||||
"swift test --list-tests",
|
||||
} {
|
||||
if IsDeliveryVerificationCommand(command) {
|
||||
t.Errorf("%q can build, run, or mutate the package and must not count as verification", command)
|
||||
}
|
||||
args, err := json.Marshal(map[string]string{"command": command})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !ToolCallMutates("bash", args, false) {
|
||||
t.Errorf("%q must fail closed as a mutation", command)
|
||||
}
|
||||
}
|
||||
|
||||
if !strings.Contains(VerificationCommandSummary(), "swift test") {
|
||||
t.Fatal("recovery summary must render the concrete Swift test command")
|
||||
}
|
||||
}
|
||||
|
||||
func TestPythonCompileallAlwaysCountsAsMutation(t *testing.T) {
|
||||
for _, command := range []string{
|
||||
"python -m compileall .",
|
||||
"python3 -m compileall -q src/",
|
||||
"python -m compileall -b package.py",
|
||||
} {
|
||||
if IsDeliveryVerificationCommand(command) {
|
||||
t.Errorf("%q writes bytecode and must not count as verification", command)
|
||||
}
|
||||
args, err := json.Marshal(map[string]string{"command": command})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !ToolCallMutates("bash", args, false) {
|
||||
t.Errorf("%q writes bytecode and must be classified as a mutation", command)
|
||||
}
|
||||
}
|
||||
if strings.Contains(VerificationCommandSummary(), "compileall") {
|
||||
t.Fatal("recovery summary must not recommend bytecode-emitting compileall")
|
||||
}
|
||||
}
|
||||
|
||||
func TestNpxVerificationUsesSafeKnownRunners(t *testing.T) {
|
||||
for _, command := range []string{
|
||||
"npx vitest run src/lib/foo.test.ts",
|
||||
"npx [email protected] run src/lib/foo.test.ts",
|
||||
"npx jest src/lib/foo.test.ts",
|
||||
"npx mocha test/",
|
||||
"npx ava",
|
||||
"npx eslint src/",
|
||||
"npx prettier --check .",
|
||||
"npx prettier --list-different src/",
|
||||
"npx tsc --noEmit",
|
||||
"npx tsc --project tsconfig.json --noEmit",
|
||||
"npx mocha test/ 2>&1 | tail -40",
|
||||
} {
|
||||
if !IsDeliveryVerificationCommand(command) {
|
||||
t.Errorf("%q should be recognized as a known read-only npx verifier", command)
|
||||
}
|
||||
args, err := json.Marshal(map[string]string{"command": command})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if ToolCallMutates("bash", args, false) {
|
||||
t.Errorf("%q should remain non-mutating verification", command)
|
||||
}
|
||||
}
|
||||
|
||||
for _, command := range []string{
|
||||
"npx --yes vitest run",
|
||||
"npx eslint@npm:evil src/",
|
||||
"npx vitest@file:../fake run",
|
||||
"npx ./eslint src/",
|
||||
"npx /tmp/eslint src/",
|
||||
"npx playwright test",
|
||||
"npx cypress run",
|
||||
"npx tsx check.ts",
|
||||
"npx ts-node check.ts",
|
||||
"npx eslint src/ --fix",
|
||||
"npx eslint src/ --output-file report.txt",
|
||||
"npx prettier --write .",
|
||||
"npx prettier .",
|
||||
"npx tsc",
|
||||
"npx tsc --noEmit --tsBuildInfoFile victim.ts",
|
||||
"npx tsc --noEmit --generateTrace trace-dir",
|
||||
"npx tsc --noEmit --init",
|
||||
"npx tsc --noEmit --showConfig",
|
||||
"npx tsc --noEmit --listFilesOnly",
|
||||
"npx tsc --noEmit --noCheck",
|
||||
"npx tsc --noEmit --watch",
|
||||
"npx tsc --build --noEmit",
|
||||
} {
|
||||
if IsDeliveryVerificationCommand(command) {
|
||||
t.Errorf("%q can install, execute, or write output and must fail closed", command)
|
||||
}
|
||||
args, err := json.Marshal(map[string]string{"command": command})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !ToolCallMutates("bash", args, false) {
|
||||
t.Errorf("%q must remain a mutation", command)
|
||||
}
|
||||
}
|
||||
|
||||
if strings.Contains(VerificationCommandSummary(), "npx") {
|
||||
t.Fatal("self-installing npx commands must not be first-line recovery recommendations")
|
||||
}
|
||||
}
|
||||
|
||||
// TestVerificationCommandSummaryNamesTheDeadlockTraps ensures the summary
|
||||
// explicitly warns about the two command shapes that used to send models into
|
||||
// the readiness deadlock loop: inline interpreters (blocked before execution)
|
||||
// and read-only inspection commands (executed but never classified as
|
||||
// verification).
|
||||
func TestVerificationCommandSummaryNamesTheDeadlockTraps(t *testing.T) {
|
||||
s := VerificationCommandSummary()
|
||||
for _, want := range []string{
|
||||
"python -c",
|
||||
"node -e",
|
||||
"grep/find/cat/wc",
|
||||
"NOT verification",
|
||||
"blocked in delivery mode",
|
||||
} {
|
||||
if !strings.Contains(s, want) {
|
||||
t.Errorf("summary should warn about trap %q, got: %s", want, s)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestVerificationCommandSummaryAcceptedPipeline ensures the escape hatch
|
||||
// advertised by the summary (a read-only extraction pipeline ending in a
|
||||
// recognized verifier) really is accepted by the classifier.
|
||||
|
||||
@@ -87,7 +87,7 @@ func (f *folderRefusal) explains(tool, subject, policyReason string, args json.R
|
||||
}
|
||||
|
||||
func (f folderRefusal) reason() string {
|
||||
return fmt.Sprintf("this workspace folder %s %s, so file edits and shell commands in it need approval, and this run has nobody to give it. Nobody declined this call and retrying or rewriting it cannot change that. The person can %s. Do the part that needs no approval, then call conclude_blocked naming the untrusted folder.", strconv.Quote(f.root), UntrustedFolderCause(f.declined), UntrustedFolderRemedy(f.root, f.declined))
|
||||
return fmt.Sprintf("this workspace folder %s %s, so file edits and shell commands in it need approval, and this run has nobody to give it. Nobody declined this call and retrying or rewriting it cannot change that. The person can %s. Do the part that needs no approval, then say in your reply that the untrusted folder blocked the rest.", strconv.Quote(f.root), UntrustedFolderCause(f.declined), UntrustedFolderRemedy(f.root, f.declined))
|
||||
}
|
||||
|
||||
// ApproveWithReason says which refusal this is: without a reason the gate reports
|
||||
@@ -104,7 +104,7 @@ func (a denyPermissionApprover) ApproveWithPolicyReason(_ context.Context, tool,
|
||||
if a.folder.explains(tool, subject, policyReason, args) {
|
||||
return false, false, a.folder.reason(), nil
|
||||
}
|
||||
reason := "this session has no interactive approver, so any call that needs approval is refused — nobody declined it, and neither retrying nor rewriting it can change that. If this work is meant to run unattended, it needs a permission mode that does not ask (or an explicit allow rule for this tool). Otherwise do the part that needs no approval and call conclude_blocked naming what was refused."
|
||||
reason := "this session has no interactive approver, so any call that needs approval is refused — nobody declined it, and neither retrying nor rewriting it can change that. If this work is meant to run unattended, it needs a permission mode that does not ask (or an explicit allow rule for this tool). Otherwise do the part that needs no approval and say in your reply what was refused."
|
||||
if why := explicitApprovalReason(tool, subject); why != "" {
|
||||
reason = why + " " + reason
|
||||
}
|
||||
|
||||
@@ -18,7 +18,7 @@ import (
|
||||
)
|
||||
|
||||
func TestPlanApprovedMessageStatesAutoSemantics(t *testing.T) {
|
||||
for _, want := range []string{"ordinary writer fallback", "explicit ask/deny rules", "forced fresh reviews"} {
|
||||
for _, want := range []string{"ordinary writer fallback", "explicit ask/deny rules"} {
|
||||
if !strings.Contains(planApprovedMessage, want) {
|
||||
t.Fatalf("planApprovedMessage missing %q: %s", want, planApprovedMessage)
|
||||
}
|
||||
|
||||
@@ -109,7 +109,7 @@ func TestAskLeavesTheDecisionUnresolvedWhenNoAskerIsWired(t *testing.T) {
|
||||
if err := c.runOneTurn(context.Background(), orchestratedTurn{input: "pick one", raw: "pick one"}); err != nil {
|
||||
t.Fatalf("runOneTurn: %v", err)
|
||||
}
|
||||
if result := lastToolResult(prov); !strings.Contains(result, "unresolved") || !strings.Contains(result, "conclude_blocked") {
|
||||
if result := lastToolResult(prov); !strings.Contains(result, "unresolved") || !strings.Contains(result, "which decision is missing") {
|
||||
t.Fatalf("a headless ask returned %q, want the decision left unresolved", result)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -685,7 +685,7 @@ const ManagedConfigWriteApprovalTool = "config_write"
|
||||
|
||||
// planApprovedMessage is the follow-up turn sent once the user approves a plan —
|
||||
// the in-context nudge to execute and keep the (already-seeded) task list honest.
|
||||
const planApprovedMessage = "Plan approved — plan mode is off. Implement the plan now. The ordinary writer fallback is approved for this execution turn; explicit ask/deny rules and forced fresh reviews still apply. Use this serial workflow: 1) mark the first sub-step in_progress with todo_write (this establishes the task list); 2) execute the sub-step; 3) call complete_step with evidence — the host then marks that sub-step completed and moves the next one to in_progress for you. Repeat 2–3 for each remaining sub-step. You don’t need another todo_write to mark steps completed; each complete_step advances the list. Sign off one sub-step at a time — never batch multiple completions."
|
||||
const planApprovedMessage = "Plan approved — plan mode is off. Implement the plan now. The ordinary writer fallback is approved for this execution turn; explicit ask/deny rules still apply. Use this serial workflow: 1) mark the first sub-step in_progress with todo_write (this establishes the task list); 2) execute the sub-step; 3) call complete_step with what is now true — the host then marks that sub-step completed and moves the next one to in_progress for you. Repeat 2–3 for each remaining sub-step. You don’t need another todo_write to mark steps completed; each complete_step advances the list. Sign off one sub-step at a time — never batch multiple completions."
|
||||
|
||||
type preparedInvocationTurn struct {
|
||||
composed string
|
||||
|
||||
@@ -963,4 +963,4 @@ const completionWaysOut = "Mark each finished item with complete_step — that i
|
||||
"If an item should no longer be done — the request changed, or it was superseded — send a todo_write without it. " +
|
||||
"If one turns on a decision only the user can make, put it to them with ask; not having heard back is not an answer to assume one from. " +
|
||||
"If one cannot go on until the user says something only they can say, call await_user naming it — the list is kept and this message stops. " +
|
||||
"If one cannot be done as specified, call conclude_blocked with the evidence for why."
|
||||
"If one cannot be done as specified, report update_goal blocked with the reason."
|
||||
@@ -227,7 +227,7 @@ func TestHeadlessRefusalDoesNotBlameAUser(t *testing.T) {
|
||||
if strings.Contains(reason, "the user declined") {
|
||||
t.Errorf("refusal blames a user who was never asked: %q", reason)
|
||||
}
|
||||
for _, want := range []string{"no interactive approver", "conclude_blocked"} {
|
||||
for _, want := range []string{"no interactive approver", "what was refused"} {
|
||||
if !strings.Contains(reason, want) {
|
||||
t.Errorf("reason = %q, missing %q", reason, want)
|
||||
}
|
||||
|
||||
@@ -4,50 +4,22 @@ import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"unicode/utf8"
|
||||
|
||||
"reasonix/internal/contract/planmode"
|
||||
"reasonix/internal/contract/provider"
|
||||
"reasonix/internal/contract/tool"
|
||||
"reasonix/internal/safety/evidence"
|
||||
"reasonix/internal/state/instruction"
|
||||
)
|
||||
|
||||
func init() { tool.RegisterBuiltin(completeStep{}) }
|
||||
|
||||
// completeStep records an evidence-backed completion of one step of an approved
|
||||
// plan. It has no host side effects; what it exists for is the enforcement in
|
||||
// Execute, which rejects a completion carrying no evidence — so a step cannot
|
||||
// flip to "done" without showing why. todo_write keeps the list moving;
|
||||
// complete_step is the formal sign-off of a finished step.
|
||||
type completeStep struct{}
|
||||
|
||||
type stepEvidence struct {
|
||||
Kind string `json:"kind"`
|
||||
Summary string `json:"summary"`
|
||||
Command string `json:"command,omitempty"`
|
||||
Paths []string `json:"paths,omitempty"`
|
||||
CriterionID string `json:"criterion_id,omitempty"`
|
||||
}
|
||||
|
||||
// validEvidenceKinds are the evidence forms a completion may cite. "checkpoint"
|
||||
// (main's fourth kind) is omitted — v2 has no checkpoint system.
|
||||
var validEvidenceKinds = map[string]bool{
|
||||
"verification": true, // a command/test was run; cite it and its outcome
|
||||
"review": true, // a completed built-in review run, fresh for any later mutation
|
||||
"diff": true, // a concrete code change; cite what changed
|
||||
"files": true, // files created/edited/inspected; cite the paths
|
||||
"manual": true, // a manual check; cite what was confirmed and how
|
||||
}
|
||||
|
||||
func (completeStep) Name() string { return "complete_step" }
|
||||
|
||||
func (completeStep) Description() string {
|
||||
return "Record the evidence-backed completion of ONE step of an approved plan. Call it as you finish each step instead of silently moving on: a completion with no evidence is REJECTED, so don't claim a step is done until you can show why. The host advances the list for you — it marks this step completed and moves the next to in_progress, so a todo_write that marks such an item completed on its own is REJECTED too. Sign off one step per tool-call ROUND, not one per call: the sign-off promotes the next item, and a second sign-off sent beside it signs against the list as it stood before."
|
||||
return "Mark ONE step of an approved plan as done. Call it as you finish each step. The host advances the list for you: it marks this step completed and moves the next to in_progress, so a todo_write that marks such an item completed on its own is REJECTED. Sign off one step per tool-call ROUND, not one per call: the sign-off promotes the next item, and a second sign-off sent beside it signs against the list as it stood before."
|
||||
}
|
||||
|
||||
func (completeStep) Schema() json.RawMessage {
|
||||
@@ -58,30 +30,12 @@ func (completeStep) Schema() json.RawMessage {
|
||||
"step":{"type":"string","description":"Which plan step this completes — its title or number, matching the task list. Use only when the item has no step_id."},
|
||||
"step_index":{"type":"integer","minimum":1,"description":"Optional 1-based task-list item number. Use only when the item has no step_id; an index goes stale the moment a step is inserted above it."},
|
||||
"result":{"type":"string","description":"What is now true or changed as a result of finishing this step."},
|
||||
"evidence":{
|
||||
"type":"array",
|
||||
"minItems":1,
|
||||
"description":"Proof the step is done. At least one item is required.",
|
||||
"items":{
|
||||
"type":"object",
|
||||
"properties":{
|
||||
"criterion_id":{"type":"string","description":"The acceptance criterion this proof satisfies, as the plan renders it (e.g. \"c2\" from \"accept [c2]: ...\"). Cite it whenever the step has criteria: a command succeeding is not the same as a criterion being met, and the host records the proof against the criterion you name."},
|
||||
"kind":{"type":"string","enum":["verification","review","diff","files","manual"],"description":"verification = a command/test was run (command REQUIRED); review = a built-in review run completed and, after changes, inspected the latest changed result (the verdict/findings still apply separately); diff = a concrete code change (paths REQUIRED); files = files created/edited/inspected (paths REQUIRED); manual = a manual check."},
|
||||
"summary":{"type":"string","description":"The evidence itself: the test result, what the diff does, or what was confirmed."},
|
||||
"command":{"type":"string","description":"REQUIRED for verification evidence: the command as it actually ran (e.g. \"go test ./...\") — it is checked against this session's real command history."},
|
||||
"paths":{"type":"array","items":{"type":"string"},"description":"REQUIRED for diff/files evidence: the files this evidence refers to, as the paths were passed to the tools that touched them."}
|
||||
},
|
||||
"required":["kind","summary"]
|
||||
}
|
||||
},
|
||||
"notes":{"type":"string","description":"Optional caveats, follow-ups, or anything deferred."}
|
||||
},
|
||||
"required":["result","evidence"]
|
||||
"required":["result"]
|
||||
}`)
|
||||
}
|
||||
|
||||
// ReadOnly is true: complete_step only records a claim (no filesystem or process
|
||||
// effect), so it never needs approval and stays available alongside todo_write.
|
||||
func (completeStep) ReadOnly() bool { return true }
|
||||
|
||||
// Sequential is true even though ReadOnly is: signing a step off advances the
|
||||
@@ -112,12 +66,11 @@ func (completeStep) PlanModeSafe() bool { return false }
|
||||
|
||||
func (completeStep) Execute(ctx context.Context, args json.RawMessage) (string, error) {
|
||||
var p struct {
|
||||
StepID string `json:"step_id"`
|
||||
Step string `json:"step"`
|
||||
StepIndex int `json:"step_index"`
|
||||
Result string `json:"result"`
|
||||
Evidence []stepEvidence `json:"evidence"`
|
||||
Notes string `json:"notes"`
|
||||
StepID string `json:"step_id"`
|
||||
Step string `json:"step"`
|
||||
StepIndex int `json:"step_index"`
|
||||
Result string `json:"result"`
|
||||
Notes string `json:"notes"`
|
||||
}
|
||||
if err := json.Unmarshal(args, &p); err != nil {
|
||||
return "", fmt.Errorf("invalid args: %w", err)
|
||||
@@ -132,53 +85,11 @@ func (completeStep) Execute(ctx context.Context, args json.RawMessage) (string,
|
||||
if strings.TrimSpace(p.Result) == "" {
|
||||
return "", fmt.Errorf("result is required — state what is now true after finishing this step")
|
||||
}
|
||||
if len(p.Evidence) == 0 {
|
||||
return "", fmt.Errorf("at least one evidence item is required — don't mark a step complete without showing why it's done (run a check, cite the diff, or confirm manually)")
|
||||
}
|
||||
kinds := make([]string, 0, len(p.Evidence))
|
||||
for i, e := range p.Evidence {
|
||||
if !validEvidenceKinds[e.Kind] {
|
||||
return "", fmt.Errorf("evidence %d: invalid kind %q (want verification|diff|files|manual)", i+1, e.Kind)
|
||||
}
|
||||
if strings.TrimSpace(e.Summary) == "" {
|
||||
return "", fmt.Errorf("evidence %d: summary is required — the evidence is the summary, not just its kind", i+1)
|
||||
}
|
||||
kinds = append(kinds, e.Kind)
|
||||
}
|
||||
if err := verifyCitedCriteria(ctx, p.Evidence); err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
todoMatch, hasTodo, err := verifyTodoStep(ctx, step)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
hostVerified, manualUnverified, err := verifyStepEvidence(ctx, p.Evidence)
|
||||
if err != nil {
|
||||
if hasTodo && todoMatch.Status == "in_progress" {
|
||||
return "", fmt.Errorf("%w; todo %d %q remains in_progress — repair the evidence and retry this step before moving on", err, todoMatch.Index, todoMatch.Content)
|
||||
}
|
||||
return "", err
|
||||
}
|
||||
projectVerified, err := verifyProjectChecks(ctx, p.Evidence)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
hostStatus := ""
|
||||
if _, ok := evidence.FromContext(ctx); ok {
|
||||
hostStatus = fmt.Sprintf(" Host evidence: host-verified %d, manual/unverified %d.", hostVerified, manualUnverified)
|
||||
}
|
||||
todoStatus := ""
|
||||
if hasTodo {
|
||||
todoStatus = fmt.Sprintf(" Todo step: todo-matched %d (%q).", todoMatch.Index, todoMatch.Content)
|
||||
}
|
||||
projectStatus := ""
|
||||
if projectVerified > 0 {
|
||||
projectStatus = fmt.Sprintf(" Project checks: project checks %d.", projectVerified)
|
||||
}
|
||||
advanceStatus := planAdvanceStatus(ctx, todoMatch, hasTodo)
|
||||
return fmt.Sprintf("Step %q signed off with %d evidence item(s) [%s].%s%s",
|
||||
step, len(p.Evidence), strings.Join(kinds, ", "), hostStatus+todoStatus+projectStatus, advanceStatus), nil
|
||||
return fmt.Sprintf("Step %q done.%s", step, planAdvanceStatus(ctx, todoMatch, hasTodo)), nil
|
||||
}
|
||||
|
||||
// planAdvanceStatus says what the sign-off left behind. Whether a step remains
|
||||
@@ -247,124 +158,6 @@ func completeStepIdentity(stepID, step string, stepIndex int) string {
|
||||
return strings.TrimSpace(step)
|
||||
}
|
||||
|
||||
func verifyStepEvidence(ctx context.Context, items []stepEvidence) (hostVerified int, manualUnverified int, err error) {
|
||||
ledger, ok := evidence.FromContext(ctx)
|
||||
if !ok {
|
||||
return 0, 0, nil
|
||||
}
|
||||
for i, e := range items {
|
||||
switch e.Kind {
|
||||
case "verification":
|
||||
command := strings.TrimSpace(e.Command)
|
||||
if command == "" {
|
||||
return 0, 0, fmt.Errorf("evidence %d: verification command is required for host verification — cite the command you ran, or use kind \"manual\"", i+1)
|
||||
}
|
||||
if !ledger.HasSuccessfulCommand(command) && !verifyCommandFromSession(ctx, command) {
|
||||
if ledger.HasFailedCommand(command) {
|
||||
return 0, 0, fmt.Errorf("evidence %d: verification command %q ran but exited non-zero, so it can't prove the step; if the non-zero exit is itself the expected proof (e.g. a file is gone), re-run it so it succeeds (append \"|| true\") and sign off again", i+1, command)
|
||||
}
|
||||
hint := allCommandHints(ctx, ledger)
|
||||
return 0, 0, fmt.Errorf("evidence %d: verification command %q has no matching successful receipt — cite the command exactly as it ran in the session%s", i+1, command, hint)
|
||||
}
|
||||
_, deliveryHasMutation := ledger.LatestSuccessfulMutationIndex()
|
||||
if evidence.DeliveryProfileFromContext(ctx) && deliveryHasMutation && !evidence.IsDeliveryVerificationCommand(command) {
|
||||
// Naming only the category sends the model back with another
|
||||
// unrecognized command — a project's own checker (go run ./tools/x)
|
||||
// reads as "a project lint command" and is refused just the same.
|
||||
// The summary is the accepted set, so cite it here as well.
|
||||
return 0, 0, fmt.Errorf("evidence %d: command %q ran successfully but is not a recognized delivery verification; do not cite an opaque command as verification. If this was only a visible/manual inspection, cite kind manual or files without a command, then rerun and cite a recognized verifier after any opaque mutation. %s", i+1, command, evidence.VerificationCommandSummary())
|
||||
}
|
||||
hostVerified++
|
||||
case "review":
|
||||
if !ledger.HasCompletedReview() {
|
||||
return 0, 0, fmt.Errorf("evidence %d: review evidence requires a completed review run in this turn; after a mutation, the review must be newer and cover the changed result", i+1)
|
||||
}
|
||||
hostVerified++
|
||||
case "diff":
|
||||
if len(e.Paths) == 0 {
|
||||
return 0, 0, fmt.Errorf("evidence %d: diff evidence requires paths for host verification — cite the files you changed", i+1)
|
||||
}
|
||||
if !everyCitationProven(ctx, e.Paths, func(p []string) bool {
|
||||
return ledger.HasSuccessfulWrite(p) || verifyPathsFromSession(ctx, p, true)
|
||||
}) {
|
||||
return 0, 0, fmt.Errorf("evidence %d: diff paths have no matching successful writer receipt in this turn%s%s", i+1, receiptHint("files written this turn", ledger.TouchedPaths(8, true)), changedByCommandHint(ledger, e.Paths))
|
||||
}
|
||||
hostVerified++
|
||||
case "files":
|
||||
if len(e.Paths) == 0 {
|
||||
return 0, 0, fmt.Errorf("evidence %d: files evidence requires paths for host verification — cite the files you touched", i+1)
|
||||
}
|
||||
if !everyCitationProven(ctx, e.Paths, func(p []string) bool {
|
||||
return ledger.HasSuccessfulReadOrWrite(p) || ledger.HasSuccessfulBashMentioningPaths(p) || verifyPathsFromSession(ctx, p, false)
|
||||
}) {
|
||||
return 0, 0, fmt.Errorf("evidence %d: file paths have no matching successful read/write receipt in this turn%s", i+1, receiptHint("files touched this turn", ledger.TouchedPaths(8, false)))
|
||||
}
|
||||
hostVerified++
|
||||
case "manual":
|
||||
manualUnverified++
|
||||
}
|
||||
}
|
||||
// A manual note stands alone only where the turn has nothing else to cite.
|
||||
// Once the ledger holds a write or a command, the proof is in the receipts,
|
||||
// and prose may not stand in for the one the host can actually read.
|
||||
if hostVerified == 0 && manualUnverified > 0 {
|
||||
_, mutated := ledger.LatestSuccessfulMutationIndex()
|
||||
if mutated || ledger.HasSuccessfulVerificationCommand() {
|
||||
return 0, 0, fmt.Errorf("manual evidence cannot stand alone in a turn that already has receipts — cite what the host observed: the command you ran (kind verification), or the files you changed or read (kind diff/files)%s", receiptHint("touched this turn", ledger.TouchedPaths(8, false)))
|
||||
}
|
||||
}
|
||||
return hostVerified, manualUnverified, nil
|
||||
}
|
||||
|
||||
func verifyProjectChecks(ctx context.Context, items []stepEvidence) (int, error) {
|
||||
checks := instruction.FromContext(ctx)
|
||||
if len(checks) == 0 {
|
||||
return 0, nil
|
||||
}
|
||||
ledger, ok := evidence.FromContext(ctx)
|
||||
if !ok {
|
||||
return 0, nil
|
||||
}
|
||||
after, ok := latestWriteBackedEvidenceIndex(ledger, items)
|
||||
if !ok {
|
||||
return 0, nil
|
||||
}
|
||||
for _, check := range checks {
|
||||
command := strings.TrimSpace(check.Command)
|
||||
if command == "" {
|
||||
continue
|
||||
}
|
||||
if !ledger.HasSuccessfulCommandAfter(command, after) {
|
||||
return 0, fmt.Errorf("project check %q from %s has no matching successful bash receipt after the latest matching write in this turn", command, checkSource(check))
|
||||
}
|
||||
}
|
||||
return len(checks), nil
|
||||
}
|
||||
|
||||
func latestWriteBackedEvidenceIndex(ledger *evidence.Ledger, items []stepEvidence) (int, bool) {
|
||||
latest := -1
|
||||
for _, item := range items {
|
||||
switch item.Kind {
|
||||
case "diff", "files":
|
||||
if i, ok := ledger.LatestSuccessfulWriteIndex(item.Paths); ok && i > latest {
|
||||
latest = i
|
||||
}
|
||||
}
|
||||
}
|
||||
return latest, latest >= 0
|
||||
}
|
||||
|
||||
func checkSource(check instruction.VerifyCheck) string {
|
||||
source := strings.TrimSpace(check.SourcePath)
|
||||
if source == "" {
|
||||
source = "project memory"
|
||||
}
|
||||
if check.Line > 0 {
|
||||
return fmt.Sprintf("%s:%d", source, check.Line)
|
||||
}
|
||||
return source
|
||||
}
|
||||
|
||||
func verifyTodoStep(ctx context.Context, step string) (evidence.TodoStepMatch, bool, error) {
|
||||
ledger, ok := evidence.FromContext(ctx)
|
||||
var todos []evidence.TodoItem
|
||||
@@ -388,7 +181,7 @@ func verifyTodoStep(ctx context.Context, step string) (evidence.TodoStepMatch, b
|
||||
}
|
||||
if allCompleted {
|
||||
last := len(todos) - 1
|
||||
return evidence.TodoStepMatch{}, true, fmt.Errorf("step %q has no matching todo_write item and every current todo is already completed; this is a renewal sign-off, so retry complete_step with step_index %d (the final existing todo %q) and the fresh evidence — do not invent a new step or rewrite the completed list", step, last+1, todos[last].Content)
|
||||
return evidence.TodoStepMatch{}, true, fmt.Errorf("step %q has no matching todo_write item and every current todo is already completed; this is a renewal sign-off, so retry complete_step with step_index %d (the final existing todo %q) — do not invent a new step or rewrite the completed list", step, last+1, todos[last].Content)
|
||||
}
|
||||
if ids := evidence.TodoStepIDs(todos); len(ids) > 0 {
|
||||
return evidence.TodoStepMatch{}, true, fmt.Errorf("step %q has no matching todo_write item in the current task list; cite the item's stable step_id — available ids: %s (list: %s)", step, strings.Join(ids, ", "), todoListInventory(todos))
|
||||
@@ -422,14 +215,6 @@ func verifyTodoStep(ctx context.Context, step string) (evidence.TodoStepMatch, b
|
||||
}
|
||||
}
|
||||
|
||||
func todoInventory(ledger *evidence.Ledger) string {
|
||||
todos, ok := ledger.LatestTodos()
|
||||
if !ok || len(todos) == 0 {
|
||||
return "(no todos recorded this turn)"
|
||||
}
|
||||
return todoListInventory(todos)
|
||||
}
|
||||
|
||||
func todoListInventory(todos []evidence.TodoItem) string {
|
||||
parts := make([]string, 0, len(todos))
|
||||
for i, t := range todos {
|
||||
@@ -446,273 +231,5 @@ func todoListInventory(todos []evidence.TodoItem) string {
|
||||
return strings.Join(parts, ", ")
|
||||
}
|
||||
|
||||
// verifyCommandFromSession scans the full conversation history (not just the
|
||||
// per-turn ledger) so a complete_step can cite a command that ran in an
|
||||
// earlier turn (the ledger resets per turn) or via a named tool instead of
|
||||
// bash. Calls whose recorded result is an error or a block are skipped — they
|
||||
// prove the command was attempted, not that it succeeded.
|
||||
func verifyCommandFromSession(ctx context.Context, command string) bool {
|
||||
msgs, ok := evidence.SessionMessagesFromContext(ctx)
|
||||
if !ok {
|
||||
return false
|
||||
}
|
||||
lookup := strings.TrimSuffix(strings.TrimSuffix(strings.TrimSpace(command), "..."), "…")
|
||||
if lookup == "" {
|
||||
return false
|
||||
}
|
||||
toolName := firstWord(lookup)
|
||||
failed := failedCallIDs(msgs)
|
||||
|
||||
for _, msg := range msgs {
|
||||
for _, tc := range msg.ToolCalls {
|
||||
if failed[tc.ID] {
|
||||
continue
|
||||
}
|
||||
cmd := extractCommandFromCall(tc.Name, tc.Arguments)
|
||||
if cmd == "" {
|
||||
continue
|
||||
}
|
||||
if evidence.CommandMatches(lookup, cmd) {
|
||||
return true
|
||||
}
|
||||
if toolName != "" && toolName != "bash" && tc.Name == toolName {
|
||||
return true
|
||||
}
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// builtinToolFacts answers for a name in the transcript out of the set this
|
||||
// binary ships, which is the same reach the writer name table it replaces had.
|
||||
// An unregistered name reads as a read, so a replay invents no writes.
|
||||
var builtinToolFacts = func() func(string) evidence.ToolFacts {
|
||||
byName := sync.OnceValue(func() map[string]evidence.ToolFacts {
|
||||
m := make(map[string]evidence.ToolFacts)
|
||||
for _, t := range tool.Builtins() {
|
||||
m[t.Name()] = evidence.ToolFacts{ReadOnly: t.ReadOnly(), WritesNamedPaths: tool.WritesNamedPaths(t), EffectsUnstated: tool.EffectsUnstated(t)}
|
||||
}
|
||||
return m
|
||||
})
|
||||
return func(name string) evidence.ToolFacts {
|
||||
if f, ok := byName()[name]; ok {
|
||||
return f
|
||||
}
|
||||
return evidence.ToolFacts{ReadOnly: true}
|
||||
}
|
||||
}()
|
||||
|
||||
// everyCitationProven reports whether each cited path has a receipt under one
|
||||
// of the forms that name the same workspace file.
|
||||
func everyCitationProven(ctx context.Context, cited []string, proven func([]string) bool) bool {
|
||||
root := evidence.WorkspaceRootFromContext(ctx)
|
||||
for _, p := range cited {
|
||||
if !slices.ContainsFunc(evidence.CitationForms(root, p), func(form string) bool { return proven([]string{form}) }) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// verifyPathsFromSession is the diff/files analogue of verifyCommandFromSession:
|
||||
// it lets a completion cite a file written or read in an earlier turn (the
|
||||
// per-turn ledger only has this turn). wantWrite restricts to writer tools.
|
||||
func verifyPathsFromSession(ctx context.Context, paths []string, wantWrite bool) bool {
|
||||
msgs, ok := evidence.SessionMessagesFromContext(ctx)
|
||||
if !ok {
|
||||
return false
|
||||
}
|
||||
return evidence.PathsProvenInSession(msgs, paths, wantWrite, builtinToolFacts)
|
||||
}
|
||||
|
||||
func failedCallIDs(msgs []provider.Message) map[string]bool {
|
||||
failed := map[string]bool{}
|
||||
for _, msg := range msgs {
|
||||
if msg.Role != provider.RoleTool || msg.ToolCallID == "" {
|
||||
continue
|
||||
}
|
||||
if strings.HasPrefix(msg.Content, "error:") || strings.HasPrefix(msg.Content, "blocked:") {
|
||||
failed[msg.ToolCallID] = true
|
||||
}
|
||||
}
|
||||
return failed
|
||||
}
|
||||
|
||||
// changedByCommandHint answers a diff citation for paths only a command
|
||||
// touched. A writer receipt comes from the file tools, so a change made by a
|
||||
// command never has one, but the same paths cited as files evidence are proven
|
||||
// by that command; without this the model retries diff until it gives up.
|
||||
func changedByCommandHint(ledger *evidence.Ledger, paths []string) string {
|
||||
if !ledger.HasSuccessfulBashMentioningPaths(paths) {
|
||||
return ""
|
||||
}
|
||||
return "; a command in this turn mentions these paths, and diff evidence counts only file-tool writes: cite them as files evidence instead, or cite the check you ran"
|
||||
}
|
||||
|
||||
func receiptHint(label string, items []string) string {
|
||||
if len(items) == 0 {
|
||||
return ""
|
||||
}
|
||||
return fmt.Sprintf("; %s: %q — cite one as it actually ran, or run the check now", label, distinguish(items))
|
||||
}
|
||||
|
||||
// What tells one of these apart from the next, within a budget. Cutting the
|
||||
// tail is the wrong end for paths: under a deep root every entry becomes the
|
||||
// same eighty characters and an ellipsis — one real session offered three
|
||||
// identical strings as the evidence it should have cited. The shared head
|
||||
// carries nothing once they are read together, so it goes before the middle.
|
||||
func distinguish(items []string) []string {
|
||||
out := make([]string, len(items))
|
||||
copy(out, items)
|
||||
if head := commonDirPrefix(out); head != "" {
|
||||
for i, item := range out {
|
||||
out[i] = "…/" + strings.TrimPrefix(item, head)
|
||||
}
|
||||
}
|
||||
for i, item := range out {
|
||||
if len([]rune(item)) <= receiptHintWidth {
|
||||
continue
|
||||
}
|
||||
r := []rune(item)
|
||||
keep := receiptHintWidth - 1
|
||||
out[i] = string(r[:keep/3]) + "…" + string(r[len(r)-(keep-keep/3):])
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// How wide one cited item may be. Wide enough for a path with a couple of
|
||||
// directories on it, narrow enough that a list of eight stays readable.
|
||||
const receiptHintWidth = 80
|
||||
|
||||
// The longest directory prefix every item shares. Directory, not character: a
|
||||
// prefix that stops mid-name would read as a different file.
|
||||
func commonDirPrefix(items []string) string {
|
||||
if len(items) < 2 {
|
||||
return ""
|
||||
}
|
||||
head := items[0]
|
||||
for _, item := range items[1:] {
|
||||
for !strings.HasPrefix(item, head) {
|
||||
cut := strings.LastIndexByte(strings.TrimSuffix(head, "/"), '/')
|
||||
if cut <= 0 {
|
||||
return ""
|
||||
}
|
||||
head = head[:cut+1]
|
||||
}
|
||||
}
|
||||
if !strings.HasSuffix(head, "/") {
|
||||
cut := strings.LastIndexByte(head, '/')
|
||||
if cut <= 0 {
|
||||
return ""
|
||||
}
|
||||
head = head[:cut+1]
|
||||
}
|
||||
return head
|
||||
}
|
||||
|
||||
// allCommandHints builds a combined hint from both the per-turn ledger and the
|
||||
// full session history, so the model can self-correct a mismatched citation.
|
||||
func allCommandHints(ctx context.Context, ledger *evidence.Ledger) string {
|
||||
seen := map[string]bool{}
|
||||
var cmds []string
|
||||
if ledger != nil {
|
||||
for _, c := range ledger.SuccessfulCommands(8) {
|
||||
if !seen[c] {
|
||||
seen[c] = true
|
||||
cmds = append(cmds, c)
|
||||
}
|
||||
}
|
||||
}
|
||||
if msgs, ok := evidence.SessionMessagesFromContext(ctx); ok {
|
||||
failed := failedCallIDs(msgs)
|
||||
for _, msg := range msgs {
|
||||
for _, tc := range msg.ToolCalls {
|
||||
if failed[tc.ID] {
|
||||
continue
|
||||
}
|
||||
if tc.Name == "todo_write" || tc.Name == "complete_step" {
|
||||
continue
|
||||
}
|
||||
c := extractCommandFromCall(tc.Name, tc.Arguments)
|
||||
if c == "" || seen[c] {
|
||||
continue
|
||||
}
|
||||
seen[c] = true
|
||||
cmds = append(cmds, c)
|
||||
if len(cmds) >= 12 {
|
||||
break
|
||||
}
|
||||
}
|
||||
if len(cmds) >= 12 {
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
if len(cmds) == 0 {
|
||||
return ""
|
||||
}
|
||||
// Truncate long entries for readability.
|
||||
for i, c := range cmds {
|
||||
if len(c) > 80 {
|
||||
cut := 80
|
||||
for cut > 0 && !utf8.RuneStart(c[cut]) {
|
||||
cut--
|
||||
}
|
||||
cmds[i] = c[:cut] + "…"
|
||||
}
|
||||
}
|
||||
return fmt.Sprintf("; commands that ran: %q — pick the matching one and retry complete_step", cmds)
|
||||
}
|
||||
|
||||
func firstWord(s string) string {
|
||||
s = strings.TrimSpace(s)
|
||||
if idx := strings.IndexAny(s, " \t\n"); idx >= 0 {
|
||||
return s[:idx]
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// extractCommandFromCall extracts the bash "command" argument from a tool call
|
||||
// args JSON, or returns the tool name + path for non-bash tools.
|
||||
func extractCommandFromCall(name string, argsJSON string) string {
|
||||
if name == "bash" {
|
||||
var args struct {
|
||||
Command string `json:"command"`
|
||||
}
|
||||
if err := json.Unmarshal([]byte(argsJSON), &args); err != nil {
|
||||
return ""
|
||||
}
|
||||
return strings.TrimSpace(args.Command)
|
||||
}
|
||||
// For non-bash tools, return "name path" so the command "ls ." can match
|
||||
// against a tool call `ls` with path `.`.
|
||||
var args struct {
|
||||
Path string `json:"path"`
|
||||
}
|
||||
if err := json.Unmarshal([]byte(argsJSON), &args); err != nil || args.Path == "" {
|
||||
return name
|
||||
}
|
||||
return name + " " + args.Path
|
||||
}
|
||||
|
||||
// verifyCitedCriteria rejects a proof citing a criterion the approved plan does
|
||||
// not have. Resolving an unknown id into nothing would leave the real criterion
|
||||
// unproven and only surface much later, as a completion the host refuses for a
|
||||
// reason the model never connected to this call.
|
||||
func verifyCitedCriteria(ctx context.Context, items []stepEvidence) error {
|
||||
known, ok := evidence.AcceptanceCriteriaFromContext(ctx)
|
||||
if !ok {
|
||||
return nil
|
||||
}
|
||||
for i, item := range items {
|
||||
id := strings.TrimSpace(item.CriterionID)
|
||||
if id == "" || slices.Contains(known, id) {
|
||||
continue
|
||||
}
|
||||
return fmt.Errorf("evidence %d: criterion_id %q is not in the approved plan; cite one of: %s", i+1, id, strings.Join(known, ", "))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// HostRequiredBy says why the tool cannot be removed by name: the loop advances the plan from its receipts and todo_write refuses a completion without one.
|
||||
func (completeStep) HostRequiredBy() string { return tool.FeaturePlanSteps }
|
||||
@@ -1,34 +0,0 @@
|
||||
package builtin
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"reasonix/internal/contract/provider"
|
||||
"reasonix/internal/safety/evidence"
|
||||
)
|
||||
|
||||
func TestAllCommandHintsCutsLongCommandOnRuneBoundary(t *testing.T) {
|
||||
// 80 bytes into this command falls inside a three-byte character, and
|
||||
// %q would render the split bytes as \x escapes in the hint.
|
||||
cmd := `echo "` + strings.Repeat("测试", 20) + `"`
|
||||
args, err := json.Marshal(map[string]string{"command": cmd})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
msgs := []provider.Message{{
|
||||
Role: provider.RoleAssistant,
|
||||
ToolCalls: []provider.ToolCall{{ID: "c1", Name: "bash", Arguments: string(args)}},
|
||||
}}
|
||||
ctx := evidence.WithSessionMessages(context.Background(), func() []provider.Message { return msgs })
|
||||
|
||||
hint := allCommandHints(ctx, nil)
|
||||
if !strings.Contains(hint, "echo") {
|
||||
t.Fatalf("hint does not list the command: %q", hint)
|
||||
}
|
||||
if strings.Contains(hint, `\x`) {
|
||||
t.Fatalf("hint split a character: %s", hint)
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large.
Load diff
@@ -1,56 +0,0 @@
|
||||
package builtin
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"reasonix/internal/safety/evidence"
|
||||
)
|
||||
|
||||
func criterionCitationArgs(t *testing.T, criterionID string) json.RawMessage {
|
||||
t.Helper()
|
||||
args, err := json.Marshal(map[string]any{
|
||||
"step": "fix it",
|
||||
"result": "the race is gone",
|
||||
"evidence": []map[string]any{
|
||||
{"kind": "manual", "summary": "walked the retry path", "criterion_id": criterionID},
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("marshal: %v", err)
|
||||
}
|
||||
return args
|
||||
}
|
||||
|
||||
// A citation the plan does not contain would resolve into nothing and only
|
||||
// surface much later, as a completion refused for a reason the model never
|
||||
// connected to this call. Reject it here, with the ids it may cite.
|
||||
func TestCompleteStepRejectsAnUnknownCriterionCitation(t *testing.T) {
|
||||
ctx := evidence.WithAcceptanceCriteria(context.Background(), []string{"c1", "c2"})
|
||||
_, err := (completeStep{}).Execute(ctx, criterionCitationArgs(t, "c9"))
|
||||
if err == nil {
|
||||
t.Fatal("an unknown criterion_id must be rejected")
|
||||
}
|
||||
for _, want := range []string{"c9", "c1, c2"} {
|
||||
if !strings.Contains(err.Error(), want) {
|
||||
t.Errorf("error %q should name %q", err, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestCompleteStepAcceptsAKnownCriterionCitation(t *testing.T) {
|
||||
ctx := evidence.WithAcceptanceCriteria(context.Background(), []string{"c1", "c2"})
|
||||
if _, err := (completeStep{}).Execute(ctx, criterionCitationArgs(t, "c2")); err != nil {
|
||||
t.Fatalf("a criterion the plan contains must be accepted: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// An unplanned turn carries no criteria, so citation checking must stand down
|
||||
// rather than reject every proof.
|
||||
func TestCompleteStepSkipsCitationCheckWithoutAPlan(t *testing.T) {
|
||||
if _, err := (completeStep{}).Execute(context.Background(), criterionCitationArgs(t, "c1")); err != nil {
|
||||
t.Fatalf("without an approved plan the citation must not be checked: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -1,50 +0,0 @@
|
||||
package builtin
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// The hint exists so the model can cite something real. Under a deep root every
|
||||
// entry used to arrive as the same eighty characters of shared prefix — three
|
||||
// identical strings, no way to tell which file was which. What survives the
|
||||
// budget has to be the part that differs.
|
||||
func TestDistinguishKeepsWhatDiffers(t *testing.T) {
|
||||
deep := "/private/tmp/claude-501/-Users-yhh-projects-DeepSeek-Reasonix/0cbfc17d-2fbc-4e89-930f-f506537bcc4e/scratchpad/proj/"
|
||||
got := distinguish([]string{deep + "store.go", deep + "store_test.go", deep + "cmd/todo/main.go"})
|
||||
|
||||
seen := map[string]bool{}
|
||||
for _, g := range got {
|
||||
if seen[g] {
|
||||
t.Fatalf("two entries render the same: %q", g)
|
||||
}
|
||||
seen[g] = true
|
||||
if len([]rune(g)) > receiptHintWidth {
|
||||
t.Errorf("over budget (%d runes): %q", len([]rune(g)), g)
|
||||
}
|
||||
}
|
||||
for i, want := range []string{"store.go", "store_test.go", "main.go"} {
|
||||
if !strings.HasSuffix(got[i], want) {
|
||||
t.Errorf("lost the end that names the file: %q does not end in %q", got[i], want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A single item has nothing to share a prefix with, and a short one needs no
|
||||
// help at all — neither should come back decorated.
|
||||
func TestDistinguishLeavesShortListsAlone(t *testing.T) {
|
||||
got := distinguish([]string{"go test ./..."})
|
||||
if len(got) != 1 || got[0] != "go test ./..." {
|
||||
t.Errorf("rewrote something that fit: %q", got)
|
||||
}
|
||||
if got := distinguish([]string{"a/x.go", "b/y.go"}); got[0] != "a/x.go" || got[1] != "b/y.go" {
|
||||
t.Errorf("stripped a prefix that was not shared: %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
// Cutting at a character boundary would invent a filename that never existed.
|
||||
func TestCommonDirPrefixStopsAtSeparator(t *testing.T) {
|
||||
if got := commonDirPrefix([]string{"/w/store.go", "/w/storefront.go"}); got != "/w/" {
|
||||
t.Errorf("commonDirPrefix = %q, want /w/", got)
|
||||
}
|
||||
}
|
||||
@@ -262,32 +262,6 @@ func TestTodoWriteRejectsFailedCompleteStepReceipt(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestTodoWriteRejectsFailedCompleteStepWithoutProof(t *testing.T) {
|
||||
ledger := evidence.NewLedger()
|
||||
ledger.Record(evidence.Receipt{
|
||||
ToolName: "todo_write",
|
||||
Success: true,
|
||||
Todos: []evidence.TodoItem{{Content: "Run project script", Status: "in_progress"}},
|
||||
})
|
||||
ledger.Record(evidence.Receipt{
|
||||
ToolName: "bash",
|
||||
Success: true,
|
||||
Command: `python "script.py"`,
|
||||
})
|
||||
ledger.Record(evidence.ReceiptFromToolCall("complete_step", json.RawMessage(`{
|
||||
"step":"Run project script",
|
||||
"result":"script ran",
|
||||
"evidence":[]
|
||||
}`), false, evidence.ToolFacts{ReadOnly: true}))
|
||||
ctx := evidence.WithLedger(context.Background(), ledger)
|
||||
args := json.RawMessage(`{"todos":[{"content":"Run project script","status":"completed"}]}`)
|
||||
|
||||
_, err := (todoWrite{}).Execute(ctx, args)
|
||||
if err == nil || !strings.Contains(err.Error(), "complete_step") {
|
||||
t.Fatalf("failed complete_step without proof should not authorize completion, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTodoWriteRejectsFailedCompleteStepMissingResult(t *testing.T) {
|
||||
ledger := evidence.NewLedger()
|
||||
ledger.Record(evidence.Receipt{
|
||||
|
||||
@@ -12,7 +12,7 @@
|
||||
"doc-owner": 27,
|
||||
"doc-prose": 2128,
|
||||
"error-text": 9,
|
||||
"essay": 2579,
|
||||
"essay": 2576,
|
||||
"file-size": 18416,
|
||||
"flat-view": 0,
|
||||
"frontend-parity": 13,
|
||||
@@ -1164,10 +1164,7 @@
|
||||
},
|
||||
"internal/safety/evidence/verification_command.go": {
|
||||
"complexity": 6,
|
||||
"essay": 18
|
||||
},
|
||||
"internal/safety/evidence/verification_summary_test.go": {
|
||||
"essay": 1
|
||||
"essay": 17
|
||||
},
|
||||
"internal/safety/permission/bash_decompose.go": {
|
||||
"essay": 23
|
||||
@@ -1475,9 +1472,6 @@
|
||||
"internal/tools/builtin/clientio.go": {
|
||||
"essay": 14
|
||||
},
|
||||
"internal/tools/builtin/completestep.go": {
|
||||
"essay": 1
|
||||
},
|
||||
"internal/tools/builtin/confine.go": {
|
||||
"essay": 16
|
||||
},
|
||||
|
||||
Reference in new issue
Block a user