mirror of
https://github.com/QwenLM/qwen-code.git
synced 2026-11-07 06:19:43 -10:00
test(sdk): read before an approved write in the tool-control E2E tests (#16366)
On the shared agent loop, a tool batch's approvals are asked for before any of its calls runs, and a write that needs approval checks that its file was read when approval is asked for. The two tests that read and write a file in one response therefore failed the write as unread on the nightly unified-loop leg. The fake model now reads in one round and writes in the next, as a real model must, so both loops exercise canUseTool. The design doc lists the consequence with the batch approvals difference.
This commit is contained in:
1 parent
a818643533
commit
0a81a22ad0
3 files changed
+20
-2
No files matched your search
@@ -511,7 +511,7 @@ The suite runs in the integration job. The nightly E2E run, and one started by h
|
||||
- With the setting on, headless differs from today in these intended ways. Most follow from scheduling a batch as one unit, as the TUI already does:
|
||||
- a stream-json message past `--max-session-turns` ends the run at the session turn limit, as `-p` does (the error on stderr, exit code 53), instead of a success that repeats the previous answer (#15325);
|
||||
- a notification turn (cron, monitor and other queued items) counts each of its model rounds toward `--max-session-turns`, as the main turn does, instead of one turn for the whole item, so a run whose notification turns call tools can reach the limit sooner;
|
||||
- a batch's PreToolUse hooks and approvals (stream-json `can_use_tool`) run for all of its calls before any of them is launched, instead of one call at a time; under `--max-tool-calls`, a call past the budget can be approved and then stopped before it runs;
|
||||
- a batch's PreToolUse hooks and approvals (stream-json `can_use_tool`) run for all of its calls before any of them is launched, instead of one call at a time; under `--max-tool-calls`, a call past the budget can be approved and then stopped before it runs; a write that needs approval checks that its file was read when the approval is asked for, so when one batch reads a file and writes it, the write fails as unread, as in the TUI;
|
||||
- a PostToolBatch hook fires once per batch with all of its calls, and the batch output budget applies across the batch; today each call is its own batch;
|
||||
- telemetry follows the batch, as in the TUI: a plan-mode entry's siblings are logged as failed tool calls, also in the JSON `stats`, and a call stopped before its launch still has a tool span; with Mod tool hooks the calls are still scheduled one at a time, so neither applies;
|
||||
- the results of a batch with several sequential calls are printed when the batch completes, instead of after each call;
|
||||
|
||||
@@ -511,7 +511,7 @@ PR 4–6 每一项之后,同样各自跟随默认开启和删除旧 loop 两
|
||||
- 设置开启时,headless 与现在相比有以下有意的差异。其中大多数源于把一批调用作为一个整体调度,这也是 TUI 现在的做法:
|
||||
- 在 stream-json 中,超过 `--max-session-turns` 的消息会在会话轮数上限处结束运行,与 `-p` 相同(错误写到 stderr,退出码 53),而不是返回一个重复上一条回答的成功结果(#15325);
|
||||
- 通知轮次(cron、monitor 等排队条目)的每一轮模型调用都计入 `--max-session-turns`,与主轮次相同,而不是整个条目只计一轮,因此通知轮次中调用工具的运行可能更早到达上限;
|
||||
- 一批调用的 PreToolUse hook 和审批(stream-json 的 `can_use_tool`)会在这一批的任何调用启动之前为所有调用执行,而不是一次一个调用;在 `--max-tool-calls` 下,超出预算的调用可能先被审批通过,然后在执行前被停下;
|
||||
- 一批调用的 PreToolUse hook 和审批(stream-json 的 `can_use_tool`)会在这一批的任何调用启动之前为所有调用执行,而不是一次一个调用;在 `--max-tool-calls` 下,超出预算的调用可能先被审批通过,然后在执行前被停下;需要审批的写操作在请求审批时检查文件是否读过,所以同一批里先读后写同一个文件时,写会因为文件没读过而失败,和 TUI 相同;
|
||||
- PostToolBatch hook 每批触发一次,带上这一批的所有调用,批次输出预算也按整批计算;现在每个调用都是单独的一批;
|
||||
- 遥测按整批记录,与 TUI 一致:进入 plan mode 那个调用的同批调用记为失败的工具调用(JSON 输出的 `stats` 也计入),启动前被拦下的调用仍有工具 span;带 Mod 工具 hook 时调用仍逐个调度,这两点都不适用;
|
||||
- 一批中有多个串行调用时,它们的结果在整批完成时输出,而不是每个调用完成后输出;
|
||||
|
||||
@@ -925,6 +925,9 @@ describe('Tool Control Parameters (E2E)', () => {
|
||||
const canUseToolCalls: string[] = [];
|
||||
|
||||
const fakeServer = await startFakeOpenAIServer(({ requestIndex }) => {
|
||||
// The read comes back before the write: a write that needs
|
||||
// approval checks the read when approval is asked for, and the
|
||||
// shared loop asks for a batch's approvals before any call runs.
|
||||
if (requestIndex === 0) {
|
||||
return {
|
||||
toolCalls: [
|
||||
@@ -933,6 +936,12 @@ describe('Tool Control Parameters (E2E)', () => {
|
||||
{ file_path: helper.getPath('test.txt') },
|
||||
'read-test',
|
||||
),
|
||||
],
|
||||
};
|
||||
}
|
||||
if (requestIndex === 1) {
|
||||
return {
|
||||
toolCalls: [
|
||||
fakeToolCall(
|
||||
'write_file',
|
||||
{
|
||||
@@ -1400,6 +1409,9 @@ describe('Tool Control Parameters (E2E)', () => {
|
||||
const canUseToolCalls: string[] = [];
|
||||
|
||||
const fakeServer = await startFakeOpenAIServer(({ requestIndex }) => {
|
||||
// The read comes back before the write: a write that needs
|
||||
// approval checks the read when approval is asked for, and the
|
||||
// shared loop asks for a batch's approvals before any call runs.
|
||||
if (requestIndex === 0) {
|
||||
return {
|
||||
toolCalls: [
|
||||
@@ -1408,6 +1420,12 @@ describe('Tool Control Parameters (E2E)', () => {
|
||||
{ file_path: helper.getPath('test.txt') },
|
||||
'read-test',
|
||||
),
|
||||
],
|
||||
};
|
||||
}
|
||||
if (requestIndex === 1) {
|
||||
return {
|
||||
toolCalls: [
|
||||
fakeToolCall(
|
||||
'write_file',
|
||||
{
|
||||
|
||||
Reference in new issue
Block a user