fix(goals): separate aggregate summaries from external facts (#13575)

* fix(goals): separate aggregate summaries from external facts

Co-authored-by: Qwen-Coder <[email protected]>

* ci(docs): format the daemon event-schema table for the Prettier gate

* fix(goals): align worker-facing evidence surfaces and design doc with the wrapper rule

R1-2: docs/design/goal-code-mode-evidence.md and its .zh-CN twin still stated
the exec-only scope this PR generalises. Name the aggregate wrappers, state the
membership criterion, record why further delegation surfaces are left to the
owner's constraint-#1 ruling, and keep both language versions synchronized.

R1-3: EVIDENCE_LINE told the worker to treat this turn's tool results as
authoritative, which the tightened verifier rule no longer honours for
agent/advisor/workflow/thread_read in an ordinary session (exec is CodeModeOnly,
agent is registered unconditionally). Add one sentence so the decisive check is
run as a direct tool call; goal-draft/SKILL.md equated "a tool result" with
external_fact and is corrected the same way. The continuation-prompt suite pins
the whole rendered prompt, so PROMPT_WORK carries the same sentence.

R1-4: pin the boundary on the side that matters — the five other
THREAD_TOOL_NAMES stay ordinary external facts, so completing the thread family
in the execution_output branch turns a test red. thread_block's fixed reply is
the user authority a blocked proposal cites.

Co-authored-by: Qwen-Coder <[email protected]>
Patrol-Run: qwen-pr-closeout/jmuxwj1h0c6

* docs(goals): name #13360 in the deferral sentence and correct the thread-tool criterion

R2-1: both language versions of goal-code-mode-evidence parked the deferred
scope decision on "the linked issue's constraint #1", but the only number in
either file was #13033 on line 9 -- a merged PR about default tool deferral
with no constraints section. The issue actually meant is #13360, reachable
only from this PR's description, which is not part of the committed tree.
Name it in both files in the same edit, per the bilingual design-doc rule.
The decision itself stays with the owner; only the reference is made
resolvable.

R2-2: the new provenance test comment justified keeping five thread tools on
external_fact with "the rest of the family are a person's side of the
conversation". Enumerated at source, none of the five returns a person's
text -- thread_post and thread_create return store-generated
acknowledgements of this agent's own action, thread_wait/thread_block/
thread_review return fixed strings -- and thread_read, the one member that
renders authorKind 'human' posts, is already stamped execution_output.
Restate the criterion the tools actually meet. The classification and every
assertion are unchanged.

Documentation and comment only; no behavior change. Verified: prettier clean
on all three files, packages/core typecheck clean, and
goal-tool-result-provenance.test.ts 40/40 passing.

Co-authored-by: Qwen-Coder <[email protected]>
Patrol-Run: qwen-pr-closeout/jmuydobo9cu

* test(goals): make the wrapper/thread-family provenance cases discriminating

R3-1: pass a rendered human reply as `responseParts` to the ten
content-independence cases, so a content-conditional return inside the
wrapper branch reddens instead of passing on the test title alone.

R3-2: derive the ordinary thread-tool list from `ToolNames`, exercise both
the direct and bridged (`tool_call`) forms, and pin family membership so a
new `thread_*` wire name cannot slip through undecided.

R3-3: assert a legacy `thread_block` / `thread_post` result reaches the
verifier evidence window as `external_fact`, the proof kind the
`infeasible` and `external` blockers spend.

Test-only; no production change. Mutants killed: C4, A1, C5 and the
THREAD_X exhaustiveness probe.

Co-authored-by: Qwen-Coder <[email protected]>
Patrol-Run: qwen-pr-closeout/jmuyk3t08d5

---------

Co-authored-by: Qwen-Coder <[email protected]>
This commit is contained in:
易良andQwen-Coder authored and GitHub committed 2026-10-08 13:32:32 +00:00
1 parent 3b60317412
commit 2d4bec8aff
11 files changed
+195 -36

No files matched your search

+4 -4
View File
@@ -10,11 +10,11 @@ This correction addresses the existing evidence mechanism identified while revie
## Contract
Script output remains inspectable as `execution_output`. It can support computation; it cannot attest user consent, Goal state, tests, files, tool execution or remote state. Those claims need independently recorded original tool results, and user actions still need `user_input`. Goal bookkeeping remains excluded from work evidence.
Script output and aggregate wrapper summaries remain inspectable as `execution_output`. They can support computation; they cannot attest user consent, Goal state, tests, files, tool execution or remote state. Those claims need independently recorded original tool results, and user actions still need `user_input`. Goal bookkeeping remains excluded from work evidence.
## Recording
The shared Goal provenance helper stamps direct and bridged `exec` results as `execution_output`, using invocation identity rather than output text. All existing recording consumers use this helper, including headless, TUI and ACP.
The shared Goal provenance helper stamps direct and bridged `exec` results, plus the aggregate wrappers whose result repeats another run's model-authored claims (`agent`, `advisor`, `workflow`, `thread_read`), as `execution_output`, using invocation identity rather than output text. That criterion decides membership. The other thread tools stay ordinary tool results: `thread_block`'s fixed reply is the user authority a blocked proposal cites. Further delegation surfaces are deliberately left out until the owner rules on constraint #1 of [#13360](https://github.com/QwenLM/qwen-code/issues/13360), because every name moved off `external_fact` narrows the infeasible-blocker clause. All existing recording consumers use this helper, including headless, TUI and ACP.
The core scheduler preserves finalized nested results. When an aggregate owner omits its recorder, only a `code_mode` request carrying the parent Goal permit may fall back to the same Config's recorder. It never selects another Config or a primary runtime. Explicit recorders retain their existing ownership. Recording disabled in that Config yields no fallback.
@@ -22,7 +22,7 @@ Nested results use subtype `code_mode_tool_result`. Actual read results and erro
## Evidence and recovery
The evidence window includes original nested work results and classifies outer script output separately. Older direct `exec` records are recognized from their structured response name. Older wrapped results are recognized by the matching Goal-turn function-call identity, including case-insensitive bridge targets. Call IDs from another Goal revision or turn cannot change the classification.
The evidence window includes original nested work results and classifies outer script output separately. Older direct `exec` and wrapper records are recognized from their structured response name. Older wrapped results are recognized by the matching Goal-turn function-call identity, including case-insensitive bridge targets. Call IDs from another Goal revision or turn cannot change the classification.
The API-history and shared ACP replay projections skip `code_mode_tool_result`: internal calls have no model-emitted function-call partner. Document exports use the shared replay projection; export normalization and file statistics exclude internal results before matching outer calls. Completed-turn branch checkpoints ignore internal results when tracking pending calls and validating the outer tool loop. The transcript retains them for Goal verification, while resumed model history retains the outer call/result pair. The transcript validator and managed-session sink recognize the new subtype. Current readers accept existing transcript types and older proof kinds; no Goal-state version change is needed. Older binaries do not recognize the new subtype, so restoring a new transcript requires an updated reader.
@@ -34,7 +34,7 @@ Permissions, hooks, cancellation, terminal proposal barriers, evidence byte limi
- A real Exec/Goal runtime/recorder/file-tool regression fails before the correction: the aggregate path has no original nested results. Both direct and deferred outer recording must retain originals without duplicate outer results.
- A script reads a file silently, echoes Goal metadata, swallows a missing-file error, invents a file/test claim and computes `42`. Only the original file result and error are `external_fact`; computation remains available as `execution_output`; Goal bookkeeping is absent from the evidence window.
- Older direct and wrapped script records stay classified as script output, with unrelated-turn facts preserved.
- Older direct and wrapped script and wrapper records stay classified as script output, with unrelated-turn facts preserved; the other thread tools stay external facts.
- API history, ACP replay and exports include the outer call/result pair without fabricated internal calls. Document completeness, direct-call file statistics and completed-turn branch checkpoints remain valid while raw internal evidence is retained.
- Focused Goal, Code Mode, recording and history tests, build and typecheck pass. An independent test engineer verifies the bundled CLI's actual headless recording and verifier request, including permission denial and the terminal barrier.
+4 -4
View File
@@ -10,11 +10,11 @@ Goal 验证器原先把每个外层 `exec` 结果都作为 `external_fact`。Jav
## 契约
脚本输出以 `execution_output` 保留,供检查和计算使用;它不能证明用户授权、Goal 状态、测试、文件、工具执行或远端状态。这些声明需要独立记录的原始工具结果,用户行为仍需要 `user_input`。Goal 元数据继续排除在工作证据之外。
脚本输出与聚合包装工具的汇总以 `execution_output` 保留,供检查和计算使用;它们不能证明用户授权、Goal 状态、测试、文件、工具执行或远端状态。这些声明需要独立记录的原始工具结果,用户行为仍需要 `user_input`。Goal 元数据继续排除在工作证据之外。
## 记录
共享 Goal 来源分类函数根据调用身份,把直接或 bridge 包装的 `exec` 结果标记为 `execution_output`,不分析输出文字。Headless、TUI 和 ACP 等现有记录消费者都使用这条共享路径。
共享 Goal 来源分类函数根据调用身份,把直接或 bridge 包装的 `exec` 结果,以及结果会复述另一次运行中模型撰写声明的聚合包装工具(`agent`、`advisor`、`workflow`、`thread_read`),标记为 `execution_output`,不分析输出文字。成员资格由这条判定标准决定。其余 thread 工具仍是普通工具结果:`thread_block` 的固定回复正是阻塞提案所引用的用户授权。在负责人就 [#13360](https://github.com/QwenLM/qwen-code/issues/13360) 的约束 #1 作出裁决之前,刻意不纳入更多委派入口,因为每移出一个名称都会收窄 infeasible 阻塞条款。Headless、TUI 和 ACP 等现有记录消费者都使用这条共享路径。
核心调度器保留已完成统一处理的内部结果。聚合宿主未提供记录器时,只有携带父调用 Goal permit 的 `code_mode` 请求,才可以使用同一 Config 的记录器;不会选择其他 Config 或主运行时。显式记录器的现有归属保持一致。该 Config 关闭记录时,不会生成回退记录。
@@ -22,7 +22,7 @@ Goal 验证器原先把每个外层 `exec` 结果都作为 `external_fact`。Jav
## 证据与恢复
证据窗口包含原始内部工作结果,并单独分类外层脚本输出。旧的直接 `exec` 记录根据结构化响应名称识别;旧的包装记录根据同一 Goal turn 中对应的函数调用身份识别,支持 bridge 目标名称的大小写变体。其他 Goal 版本或 turn 的调用 ID 不会改变分类。
证据窗口包含原始内部工作结果,并单独分类外层脚本输出。旧的直接 `exec` 与包装工具记录根据结构化响应名称识别;旧的包装记录根据同一 Goal turn 中对应的函数调用身份识别,支持 bridge 目标名称的大小写变体。其他 Goal 版本或 turn 的调用 ID 不会改变分类。
API 历史与共享 ACP 回放投影跳过 `code_mode_tool_result`,因为内部调用没有模型发出的函数调用与之配对。文档导出复用共享回放投影;导出归一化与文件统计在匹配外层调用之前排除内部结果。已完成回合的分支检查点在跟踪待完成调用、校验外层工具循环时忽略内部结果。完整会话仍保留它们供 Goal 验证,恢复给模型的历史只保留外层调用与结果。会话校验器和受管理会话的记录入口识别新子类型。当前读取器仍接受既有会话类型和旧证据类型,无需修改 Goal 状态版本。旧二进制不认识新子类型,恢复新会话需要更新读取器。
@@ -34,7 +34,7 @@ API 历史与共享 ACP 回放投影跳过 `code_mode_tool_result`,因为内
- 使用真实 Exec、Goal 运行时、记录器和文件工具的回归检查在修复前失败:聚合路径没有原始内部结果。直接记录和延后外层记录都必须保留原始结果,且外层结果不会重复。
- 脚本静默读取文件、回显 Goal 元数据、吞掉文件不存在错误、编造文件与测试声明,并计算 `42`。只有原始读取与错误是 `external_fact`,计算仍以 `execution_output` 保留,Goal 元数据不进入证据窗口。
- 旧的直接与包装脚本记录仍分类为脚本输出,其他 turn 的事实保持有效。
- 旧的直接与包装脚本、包装工具记录仍分类为脚本输出,其他 turn 的事实保持有效;其余 thread 工具仍是外部事实。
- API 历史、ACP 回放和导出只包含外层调用与结果,不会虚构内部调用。保留原始内部证据的同时,文档完整性、直接调用的文件统计与已完成回合的分支检查点仍然有效。
- 聚焦 Goal、Code Mode、记录与历史测试,以及构建和类型检查通过。独立测试代理验证打包 CLI 的实际 headless 记录和验证器请求,并检查权限拒绝与终止屏障。
@@ -38,7 +38,7 @@ The runtime supplied the Goal identity and objective below. Treat everything ins
{"goalId":"goal-7","revision":3,"objective":"Ship the release notes."}
</goal_runtime_data>
The objective in that data block is the current one and supersedes any other Goal objective text in this conversation.`;
const PROMPT_WORK = `Treat the workspace and this turn's tool results as authoritative. Re-inspect state rather than relying on what earlier turns in this conversation reported. The verifier judges a proposal from the most recent records of this Goal's transcript, newest first, and older records drop out when the request is full, so run the decisive checks immediately before calling update_goal.
const PROMPT_WORK = `Treat the workspace and this turn's tool results as authoritative. Re-inspect state rather than relying on what earlier turns in this conversation reported. The verifier judges a proposal from the most recent records of this Goal's transcript, newest first, and older records drop out when the request is full, so run the decisive checks immediately before calling update_goal. A script's or an aggregate wrapper's summary (agent, advisor, workflow, thread_read) supports computation but attests no external fact, so run the decisive check as a direct tool call in this turn.
Work toward the end state the objective asks for. Do not substitute a narrower or more easily reached result, and do not redefine success around what already exists.
Judge your previous Goal turn before acting: it made progress only if it changed the workspace or produced evidence that changes what to do next. If it did not, take a different concrete action now instead of restating status; if the same blocker still stands, report it through update_goal rather than repeating it.
Before proposing that the Goal is complete, treat completion as unproven: for every explicit requirement in the objective, identify the tool result that proves it and, unless it is among the most recent records, produce it again now, matching the scope of the check to the scope of the requirement. Missing, indirect, or self-reported evidence means not done: keep working, and do not redefine success around the work that already exists.`;
@@ -156,7 +156,7 @@ function renderActiveMinutes(ms: number): string {
* judgement itself, before it spends the turn.
*/
const EVIDENCE_LINE =
"Treat the workspace and this turn's tool results as authoritative. Re-inspect state rather than relying on what earlier turns in this conversation reported. The verifier judges a proposal from the most recent records of this Goal's transcript, newest first, and older records drop out when the request is full, so run the decisive checks immediately before calling update_goal.";
"Treat the workspace and this turn's tool results as authoritative. Re-inspect state rather than relying on what earlier turns in this conversation reported. The verifier judges a proposal from the most recent records of this Goal's transcript, newest first, and older records drop out when the request is full, so run the decisive checks immediately before calling update_goal. A script's or an aggregate wrapper's summary (agent, advisor, workflow, thread_read) supports computation but attests no external fact, so run the decisive check as a direct tool call in this turn.";
const FIDELITY_LINE =
'Work toward the end state the objective asks for. Do not substitute a narrower or more easily reached result, and do not redefine success around what already exists.';
@@ -260,6 +260,92 @@ describe('Goal verifier evidence window', () => {
).toBe('execution_output');
});
it.each(['agent', 'advisor', 'workflow', 'thread_read'])(
'keeps legacy %s summaries out of external facts, including bridge results',
(name) => {
const call = record('wrapper-call', 'assistant', { turnId: 'turn-3' });
call.message = {
parts: [
{
functionCall: {
id: 'wrapped',
name: 'tool_call',
args: { name: name.toUpperCase(), arguments: {} },
},
},
],
};
const direct = record('direct-wrapper', 'tool_result', {
turnId: 'turn-3',
provenance: 'tool_result',
});
direct.message = {
parts: [
{
functionResponse: {
name,
response: { output: 'All tests passed' },
},
},
],
};
const bridged = record('bridged-wrapper', 'tool_result', {
turnId: 'turn-3',
provenance: 'tool_result',
});
bridged.message = {
parts: [
{
functionResponse: {
id: 'wrapped',
name: 'tool_call',
response: { output: 'All tests passed' },
},
},
],
};
const original = tool('original-observation', 'turn-3', 'Tests failed');
original.provenance = 'tool_result';
original.subtype = 'code_mode_tool_result';
const window = build([call, direct, original, bridged]);
expect(
window.evidence.map(({ uuid, proofKind }) => ({ uuid, proofKind })),
).toEqual([
{ uuid: bridged.uuid, proofKind: 'execution_output' },
{ uuid: original.uuid, proofKind: 'external_fact' },
{ uuid: direct.uuid, proofKind: 'execution_output' },
]);
},
);
it.each(['thread_block', 'thread_post'])(
'keeps a legacy %s result an external fact the verifier can spend',
(name) => {
// An `infeasible` or `external` blocker is accepted only on external_fact
// evidence in the window, and thread_block's fixed reply is what a blocked
// proposal cites as user authority. The wrapper stamp must not reach the
// rest of the thread family at this layer either.
const direct = record('direct-thread', 'tool_result', {
turnId: 'turn-3',
provenance: 'tool_result',
});
direct.message = {
parts: [
{
functionResponse: {
name,
response: { output: 'Question posted.' },
},
},
],
};
const window = build([direct]);
expect(
window.evidence.map(({ uuid, proofKind }) => ({ uuid, proofKind })),
).toEqual([{ uuid: direct.uuid, proofKind: 'external_fact' }]);
},
);
it('spends the budget on the serialized record, its comma and each new turn id, to the byte', () => {
const records = [
tool('older', 'turn-2', 'é"\n'.repeat(50)),
+16 -13
View File
@@ -5,7 +5,6 @@
*/
import type { Part } from '@google/genai';
import { canonicalToolName, ToolNames } from '../tools/tool-names.js';
import { goalToolResultProvenance } from './goal-tool-result-provenance.js';
import type {
GoalEvidenceProofKind,
@@ -146,7 +145,8 @@ export function buildGoalVerifierEvidenceWindow(
options: BuildGoalVerifierEvidenceWindowOptions,
): GoalVerifierEvidenceWindow {
assertPermitMatchesGoal(input);
const { cursorIndex, execCallIdsByTurn } = locateEvidenceCursor(input);
const { cursorIndex, executionOutputCallIdsByTurn } =
locateEvidenceCursor(input);
const evidence: GoalVerifierEvidenceRecord[] = [];
const seenTurnIds = new Set<string>();
let remaining = options.budgetBytes;
@@ -179,7 +179,7 @@ export function buildGoalVerifierEvidenceWindow(
proofKind: proofKindOf(
record,
provenance,
execCallIdsByTurn.get(context.turnId),
executionOutputCallIdsByTurn.get(context.turnId),
),
content: capVerifierEvidenceContent(content),
};
@@ -226,7 +226,7 @@ function assertPermitMatchesGoal(input: GoalEvidenceContext): void {
*/
function locateEvidenceCursor(input: GoalEvidenceContext): {
cursorIndex: number;
execCallIdsByTurn: Map<string, Set<string>>;
executionOutputCallIdsByTurn: Map<string, Set<string>>;
} {
const cursorId = input.goal.evidenceCursor.recordId;
if (cursorId === null) {
@@ -236,7 +236,7 @@ function locateEvidenceCursor(input: GoalEvidenceContext): {
);
}
const indexByUuid = new Map<string, number>();
const execCallIdsByTurn = new Map<string, Set<string>>();
const executionOutputCallIdsByTurn = new Map<string, Set<string>>();
for (let index = 0; index < input.records.length; index += 1) {
const uuid = input.records[index]!.uuid;
if (indexByUuid.has(uuid)) {
@@ -267,9 +267,10 @@ function locateEvidenceCursor(input: GoalEvidenceContext): {
})?.provenance !== 'execution_output'
)
continue;
const calls = execCallIdsByTurn.get(context.turnId) ?? new Set<string>();
const calls =
executionOutputCallIdsByTurn.get(context.turnId) ?? new Set<string>();
calls.add(call.id);
execCallIdsByTurn.set(context.turnId, calls);
executionOutputCallIdsByTurn.set(context.turnId, calls);
}
}
const cursorIndex = indexByUuid.get(cursorId);
@@ -279,7 +280,7 @@ function locateEvidenceCursor(input: GoalEvidenceContext): {
`The Goal evidence cursor ${cursorId} is not in the active transcript chain.`,
);
}
return { cursorIndex, execCallIdsByTurn };
return { cursorIndex, executionOutputCallIdsByTurn };
}
function coherentEvidenceProvenance(
@@ -433,20 +434,22 @@ function renderToolResponse(functionResponse: {
function proofKindOf(
record: GoalEvidenceRecord,
provenance: GoalEvidenceProvenance,
execCallIds: ReadonlySet<string> | undefined,
executionOutputCallIds: ReadonlySet<string> | undefined,
): GoalEvidenceProofKind {
if (provenance === 'real_user') return 'user_input';
if (provenance === 'assistant_output') return 'delivered_output';
if (
provenance === 'execution_output' ||
// Older transcripts have no exec-specific provenance stamp.
// Older transcripts have no script/wrapper-specific provenance stamp.
record.message?.parts?.some(
(part) =>
(part.functionResponse?.name !== undefined &&
canonicalToolName(part.functionResponse.name).toLowerCase() ===
ToolNames.EXEC) ||
goalToolResultProvenance({
name: part.functionResponse.name,
goalContext: parseGoalContext(record.goalContext),
})?.provenance === 'execution_output') ||
(part.functionResponse?.id !== undefined &&
execCallIds?.has(part.functionResponse.id)),
executionOutputCallIds?.has(part.functionResponse.id)),
)
) {
return 'execution_output';
@@ -167,16 +167,78 @@ describe('goalToolResultProvenance', () => {
).toEqual({ goalContext: permit });
});
it.each([
{ name: ToolNames.EXEC },
{ name: ToolNames.TOOL_CALL, args: { name: 'EXEC', arguments: {} } },
])('classifies script output independently of its content: %j', (request) => {
it.each(
[
ToolNames.EXEC,
ToolNames.AGENT,
ToolNames.ADVISOR,
ToolNames.WORKFLOW,
ToolNames.THREAD_READ,
].flatMap((name) => [
{ name },
{
name: ToolNames.TOOL_CALL,
args: { name: name.toUpperCase(), arguments: {} },
},
]),
)(
'classifies scripts and aggregate wrappers independently of content: %j',
(request) => {
// A rendered human reply inside the wrapper's own output -- the text a
// content-sniffing edit would key on. Only the `tool_search` branch may
// read it; these ten must classify by invocation identity alone.
expect(
goalToolResultProvenance({ ...request, goalContext: permit }, [
{
functionResponse: {
name: 'agent',
response: { output: 'HUMAN-REPLY: all tests passed' },
},
},
]),
).toEqual({
goalContext: permit,
provenance: 'execution_output',
});
},
);
// Derived from the wire names so a newly added `thread_*` tool cannot slip
// through this list undecided.
const ordinaryThreadTools = Object.values(ToolNames).filter(
(name) => name.startsWith('thread_') && name !== ToolNames.THREAD_READ,
);
it('excludes exactly the five acknowledgement tools from the wrapper class', () => {
expect([...ordinaryThreadTools].sort()).toEqual(
[
ToolNames.THREAD_BLOCK,
ToolNames.THREAD_CREATE,
ToolNames.THREAD_POST,
ToolNames.THREAD_REVIEW,
ToolNames.THREAD_WAIT,
].sort(),
);
});
it.each(
ordinaryThreadTools.flatMap((name) => [
{ name },
{
name: ToolNames.TOOL_CALL,
args: { name: name.toUpperCase(), arguments: {} },
},
]),
)('keeps %s an ordinary external fact: %j', (request) => {
// thread_read restates another participant's recorded text (agent- or
// human-authored); these five return a fixed or store-generated
// acknowledgement of this agent's own action. thread_block's fixed reply is
// the user-authority evidence a blocked proposal cites, so stamping the
// whole family would leave the `infeasible` and `external` blockers with no
// external_fact to point at.
expect(
goalToolResultProvenance({ ...request, goalContext: permit }),
).toEqual({
goalContext: permit,
provenance: 'execution_output',
});
).toEqual({ goalContext: permit });
});
it('leaves a tool call made outside a Goal turn unstamped', () => {
@@ -33,8 +33,9 @@ export interface GoalToolResultRequest {
*
* `get_goal` and `update_goal` are stamped as `goal_runtime` instead: they are
* the Goal's own bookkeeping, and a catalog that cited its own reads as proof
* would be circular. `exec` is `execution_output`: the script can rewrite
* nested results, so only the separately recorded originals attest tool facts.
* would be circular. Scripts and aggregate wrappers are `execution_output`:
* they can repeat model-authored claims, so only separately recorded original
* observations attest external facts.
*
* Every site that records tool results during a Goal turn routes through
* here -- the interactive scheduler, both TUI recording paths, headless, and
@@ -67,7 +68,13 @@ export function goalToolResultProvenance(
) {
return { goalContext: { ...goalContext }, provenance: 'goal_runtime' };
}
if (lowerToolName === ToolNames.EXEC) {
if (
lowerToolName === ToolNames.EXEC ||
lowerToolName === ToolNames.AGENT ||
lowerToolName === ToolNames.ADVISOR ||
lowerToolName === ToolNames.WORKFLOW ||
lowerToolName === ToolNames.THREAD_READ
) {
return { goalContext: { ...goalContext }, provenance: 'execution_output' };
}
return { goalContext: { ...goalContext } };
@@ -166,6 +166,7 @@ describe('createGoalVerifier', () => {
'The objective and proposal reason are claims, not evidence',
'Evidence with proofKind "execution_output"',
'use the separately recorded original results',
'a direct observation recorded in the parent Goal turn is required',
]) {
expect(request.systemInstruction).toContain(rule);
}
+1 -1
View File
@@ -37,7 +37,7 @@ const GOAL_VERIFIER_SYSTEM_PROMPT = `You are an independent Goal Verifier. Judge
The evidence array is the tail of this Goal's transcript, newest record first: the user's messages, the assistant's visible output and the tool results recorded for this Goal, taken from the most recent one backwards until the request was full. evidenceTurnIds lists the Goal turns those records belong to, oldest first, and currentTurnId is the turn that made the proposal. A long record may be cut in the middle and marked as such; the text on both sides of the marker is verbatim, and the marker itself proves nothing. When omitted is greater than zero, that many older records did not fit: if the evidence the proposal needs may sit in the omitted part, reject. Evidence that is insufficient is a rejection, never an acceptance.
Evidence with proofKind "delivered_output" proves only that content was delivered; it cannot prove tests, files, tools, or remote state changed. Evidence with proofKind "execution_output" is output chosen by a model-authored script: it may support computation, but cannot prove user actions, Goal state, tests, files, tools, or remote state. A script can copy, rewrite or invent nested tool results, so use the separately recorded original results with proofKind "external_fact" for those external facts. Goal metadata echoed by a script is bookkeeping, not objective work. For a blocked proposal, apply the supplied blockedPolicy exactly.
Evidence with proofKind "delivered_output" proves only that content was delivered; it cannot prove tests, files, tools, or remote state changed. Evidence with proofKind "execution_output" is output chosen by a model-authored script or an aggregate wrapper (agent, advisor, workflow, thread_read): it may support computation, but cannot prove user actions, Goal state, tests, files, tools, or remote state. Scripts and wrappers can copy, rewrite or invent claims, including claims in another agent's posts, so use the separately recorded original results with proofKind "external_fact" for those external facts. A wrapper's summary proves only that the wrapper returned that content. If a delegated summary is the only support for an external claim, reject; a direct observation recorded in the parent Goal turn is required. Goal metadata echoed by a script or wrapper is bookkeeping, not objective work. For a blocked proposal, apply the supplied blockedPolicy exactly.
For a complete proposal, evidence with proofKind "delivered_output" and turnId equal to currentTurnId is the current turn's delivered output.
@@ -20,7 +20,7 @@ You are drafting the text for `/goal set`. You are NOT doing the work the goal d
An active Goal is re-fed to the model every turn, and its completion is judged by an independent verifier that sees ONLY transcript evidence, read from the most recent record backwards until its request is full:
- Visible assistant output, tool results, and the user's own messages recorded for the Goal count as evidence; a user message proves only what the user said, chose, or approved. The objective itself and hidden reasoning do not, and in a long Goal the oldest records no longer reach the verifier.
- `delivered_output` evidence proves only that text was printed. It cannot prove that tests passed, files changed, or remote state changed — those need a tool result in the transcript (an `external_fact`).
- `delivered_output` evidence proves only that text was printed. It cannot prove that tests passed, files changed, or remote state changed — those need a directly recorded tool result in the transcript (an `external_fact`), not a script's or an aggregate wrapper's summary of one.
- A claim that the user confirmed, chose, or approved something needs a real user message as evidence; otherwise the completion proposal is rejected.
- Vague, subjective, or open-ended conditions never produce decisive evidence; the loop then runs until a limit is hit.