mirror of
https://github.com/codewhale-hq/Codewhale.git
synced 2026-11-07 17:09:13 -10:00
feat(tui): cap request images after a provider refuses the image count
A session whose history held more images than the route accepts in one
request became permanently unsendable: the provider 400 became
LlmError::InvalidRequest, the transport did not retry it, and every
later turn, compaction and restart rebuilt the same over-limit history.
When the 400 also mentioned image_url/vision, is_image_input_rejection_
message misread it as "no image input" and stripped every image.
Learn from rejection only (no proactive default, no RouteLimits,
models.dev or custom_models field in this slice): a route that never
refuses sends every image exactly as before.
- llm_client::image_count_rejection sits next to is_context_length_
message: phrase list ("too many images", "maximum number of images",
"max images", "images per request/prompt/message", "image(s) may be
provided", "number of images" + exceed/more than/at most/limit, "at
most N image") plus a stated-limit parser over the matched clause
(at most / max is / maximum of / limit of / up to / allowed / max ...).
It never matches a context-length message.
- model_step classifies a count rejection before recover_child_request
and before the image-input branch. cap = stated limit when it is below
the images sent, else sent/2, 0 when sent <= 1. A cap of 0 reuses the
existing ImageInputRejectedResent strip-and-resend branch (now the
shared resend_without_images helper). A cap >= 1 resends at most 3
times per turn, each strictly smaller; then the turn ends with a typed
ErrorEnvelope::image_count_exceeded (InvalidInput, Error,
recoverable, code image_count_exceeded) carrying the localized
ImageCountRejected hint plus the provider text. The image-input
branch now also requires the rejected request to carry an image, so it
can never resend an image-free request unchanged.
- is_image_input_rejection_message returns false for a count rejection
unless its stated limit is 0.
- is_transient_subagent_provider_error returns false for a count
rejection, so a child never retries one as transient (including the
untyped HTTP/1.1 "stream request" path).
- process_stream: a mid-stream count rejection is terminal for that
request (never resumed or transparently resent) and still teaches the
cap.
- Learned caps live on the Engine, keyed by (provider identity key,
model), in memory only; child engines learn their own. A learned 0
marks image input unsupported for that route.
- image_attach::cap_request_images applies the window to the outbound
copy right after strip_images_when_unsupported: each ImageUrl counts 1,
each tool result carrying an image counts 1, oldest first, with
E = 0 if N <= cap else ceil((N-cap)/K)*K, K = max(1, cap/4). Stored
history is untouched. Placeholders are fixed bytes with neutral
claims; the tool note says to take a fresh observation instead of
repeating an action.
- The same pure policy (route identity + learned map) feeds the turn
(preparation.rs), the compaction summary request (PreparedCompaction-
Envelope.image_cap and config.image_input, threaded into create_
summary) and /preview-request, which now also applies the strip for
its planned route.
- Model-visible means logged: when the window (route, cap, elided)
changes, the engine emits Event::RequestImageWindow {provider, model,
total, elided, cap, cap_source, message} with a protocol twin
(request_image_window in EVENT_KINDS), runtime_threads persists it as
a Status item with metadata code request_image_window, exec stream-json
prints it, the TUI shows it as a toast, and the prefix tracker records
reset:image_window. The compaction window goes through the compaction
notice sink. New MessageIds RequestImageWindow and ImageCountRejected
are translated in all 15 packs.
KV-cache effect: the pinned system+tools prefix is never touched. The
elision point moves only once every K new images, so K-1 of every K
requests stay a pure append of the previous one; a move re-prefills from
the first changed image (Anthropic: from the history breakpoint). With
K = cap/4 the amortized re-prefill is about 3 images' worth per step
whatever the cap, against cap images per step for a naive keep-newest.
Known limits (also in CHANGELOG [Unreleased]): nothing changes until a
provider refuses once; no proactive or documented per-model limits; the
learned cap is not persisted, so a restart or new child pays one refused
request; elision is by age, user attachments before tool screenshots; a
mid-stream refusal only teaches the next send; a gateway 5xx count
refusal is retried unchanged by the transport first; a request-body
size (HTTP 413) refusal on an ordinary turn still has no recovery; a
per-message limit lowers the whole request's cap; a parse that finds no
number falls back to halving; the bounded child-report request is capped
by the window but does not learn from a refusal.
Verification:
- scripts/dev-test.sh tui image_attach llm_client error_taxonomy
compaction preview turn_loop model_step context runtime_threads
protocol_parity subagent exec_agent: 2183 tests run: 2183 passed,
13168 skipped (nextest).
- scripts/dev-test.sh protocol: 75 tests run: 75 passed.
- scripts/dev-cargo.sh nextest run -p codewhale-localization --lib
--locked: 54 tests run: 54 passed.
- CI clippy (workspace, all targets, all features, -D warnings with the
CI allows): exit 0, no warnings; run before a rustfmt-only reformat of
three files. cargo fmt --all -- --check: clean after that reformat.
- npm test: exit 0 (node 101/101, runtime-sdk 19/19, extension-host 413
passed + 7 skipped of 420, web dependency-security 7/7, vitest 772/772
in 74 files). npm run check:web: exit 0.
- sync-changelog.sh and release/check-versions.sh --require-dated-
release: OK. npm --prefix web run prebuild: no generated change
(docs/CACHE.md does not feed agent-docs).
- Real binary (target/debug/codewhale exec --output-format stream-json,
isolated HOME, DEEPSEEK_BASE_URL to a local stub that returns 400
"At most N image(s) may be provided in one prompt." when a request
carries more than N images):
- 6 attachments, N=3: request 1 400 with 6 image_url parts; request 2
200 with 3 image_url parts and 3 "Image not attached" notes; no
consecutive identical bodies; stream-json request_image_window
{total 6, elided 3, cap 3, cap_source provider_rejection}; saved
session still holds all 6 image_url blocks and no notes.
- 2 attachments, N=0: request 1 400; request 2 200 with 0 image_url
parts and 2 strip notes.
- 16 attachments, unstated "Too many images in request." always:
16 -> 8 -> 4 -> 2 images (all distinct bodies), then the turn ends
with "probe-model refused the 2 image(s) in this request; Codewhale
did not resend it unchanged..." after 3 recoveries.
Not verified: the runtime_threads Status item and the TUI toast were not
exercised in a running app-server or TUI (compile and existing tests
only); no real provider was called; the Anthropic and Responses wires
were not probed (Chat Completions only); a mid-stream count rejection
and the child-engine path were not driven end to end; the compaction
summary window was not driven by a live compaction.
Refs CW-7235
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01NBKbhx19BRbZtvWK8rVKjz
This commit is contained in:
1 parent
b095416f55
commit
f0913c3f62
39 files changed
+646
-21
No files matched your search
@@ -7,6 +7,36 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
- A long session whose history holds more images than its model accepts in
|
||||
one request no longer becomes permanently unsendable. When a provider
|
||||
refuses the number of images (for example vLLM's "At most 3 image(s) may
|
||||
be provided in one prompt", "too many images" or "maximum number of
|
||||
images"), Codewhale learns that limit for the provider and model (the
|
||||
number the provider states, otherwise half the images sent) and resends
|
||||
with only the newest images attached. Each older image is replaced by a
|
||||
fixed note telling the model it remains in the session and not to guess
|
||||
what it showed; a tool image's note says to take a fresh observation, such
|
||||
as a new screenshot, rather than repeat the action. Each resend carries
|
||||
strictly fewer images, there are at most three per turn and never an
|
||||
unchanged one; after that the turn ends with an `image_count_exceeded`
|
||||
error and the next message uses the learned limit. A stated limit of 0
|
||||
sends images as text, as for a model without image input. Before, a count
|
||||
rejection either ended every later turn the same way or was misread as
|
||||
"no image input" and stripped every image. The window moves in steps of a
|
||||
quarter of the limit, so the cached history prefix changes only once every
|
||||
few new images. Each change is recorded as a `request_image_window` event
|
||||
(runtime threads store it as a status item with that code, `exec
|
||||
--output-format stream-json` prints it and the TUI shows it), and
|
||||
`/preview-request` and the compaction summary request apply the same
|
||||
window. A sub-agent no longer retries a count rejection as a transient
|
||||
failure. Known limits: nothing changes until a provider has refused once,
|
||||
and no documented per-model limits are applied; the learned limit is kept
|
||||
in memory per engine and per sub-agent, so a restart learns it again with
|
||||
one refused request; the oldest images go first, user attachments
|
||||
included; a refusal that arrives mid-stream ends that request and only the
|
||||
next send uses the limit; a gateway that reports the refusal as a 5xx is
|
||||
retried unchanged by the transport first; and a request-body size (HTTP
|
||||
413) refusal on an ordinary turn still has no recovery (Refs CW-7235).
|
||||
- Switching provider is now refused while a turn is running, as model,
|
||||
thinking and permission changes already were. The refusal covers
|
||||
`/provider <id>`, the provider picker, a custom provider, setup and login
|
||||
|
||||
@@ -370,6 +370,8 @@
|
||||
"ScreenModeUnchanged": "Pantalla sense canvis: {reason}.",
|
||||
"ImageInputRejectedResent": "{model} no accepta imatges — s'han reenviat com a text; usa image_ocr per llegir-les",
|
||||
"ImageInputOmitted": "{model} no pot veure imatges — {count} imatge(s) s'han enviat com a text en aquest torn; usa image_ocr per llegir-les o /model per canviar",
|
||||
"RequestImageWindow": "{model} accepta com a màxim {cap} imatge(s) per sol·licitud: s'envien les {kept} més recents de {total}; {elided} imatge(s) més antigues van com a notes de text",
|
||||
"ImageCountRejected": "{model} ha rebutjat les {count} imatge(s) d'aquesta sol·licitud; Codewhale no l'ha tornat a enviar sense canvis. Torna a enviar per reintentar-ho amb menys imatges, o canvia de model",
|
||||
"ProviderToolCallMissing": "El proveïdor ha acabat amb `{reason}`, però no ha proporcionat cap crida a una eina. Torna a provar aquest torn per continuar.",
|
||||
"ConfigLabelShowThinking": "Raonament del model al xat",
|
||||
"ConfigLabelThinkingHighlight": "Ressaltat de fons del raonament",
|
||||
|
||||
@@ -370,6 +370,8 @@
|
||||
"ScreenModeUnchanged": "Bildschirm unverändert: {reason}.",
|
||||
"ImageInputRejectedResent": "{model} akzeptiert keine Bilder — sie wurden als Text erneut gesendet; nutze image_ocr zum Lesen",
|
||||
"ImageInputOmitted": "{model} kann keine Bilder sehen — {count} Bild(er) wurden in diesem Zug als Text gesendet; nutze image_ocr zum Lesen oder /model zum Wechseln",
|
||||
"RequestImageWindow": "{model} akzeptiert höchstens {cap} Bild(er) pro Anfrage: Die neuesten {kept} von {total} werden gesendet; {elided} ältere Bild(er) gehen als Textnotizen mit",
|
||||
"ImageCountRejected": "{model} hat die {count} Bild(er) dieser Anfrage abgelehnt; Codewhale hat sie nicht unverändert erneut gesendet. Sende erneut, um es mit weniger Bildern zu versuchen, oder wechsle das Modell",
|
||||
"ProviderToolCallMissing": "Der Anbieter hat mit `{reason}` beendet, aber keinen Werkzeugaufruf geliefert. Wiederhole diesen Durchgang, um fortzufahren.",
|
||||
"ConfigLabelShowThinking": "Modell-Reasoning im Chat",
|
||||
"ConfigLabelThinkingHighlight": "Reasoning-Hintergrund hervorheben",
|
||||
|
||||
@@ -371,6 +371,8 @@
|
||||
"ScreenModeUnchanged": "Screen unchanged: {reason}.",
|
||||
"ImageInputRejectedResent": "{model} does not accept images — resent as text; use image_ocr to read them",
|
||||
"ImageInputOmitted": "{model} can't see images — {count} image(s) sent as text this turn; use image_ocr to read them or /model to switch",
|
||||
"RequestImageWindow": "{model} accepts at most {cap} image(s) per request: sending the newest {kept} of {total}; {elided} older image(s) go as text notes",
|
||||
"ImageCountRejected": "{model} refused the {count} image(s) in this request; Codewhale did not resend it unchanged. Send again to retry with fewer images, or switch models",
|
||||
"ProviderToolCallMissing": "Provider ended with `{reason}` but supplied no tool call. Retry the turn to continue.",
|
||||
"ConfigLabelShowThinking": "Model reasoning in chat",
|
||||
"ConfigLabelThinkingHighlight": "Thinking background highlight",
|
||||
|
||||
@@ -370,6 +370,8 @@
|
||||
"ScreenModeUnchanged": "Pantalla sin cambios: {reason}.",
|
||||
"ImageInputRejectedResent": "{model} no acepta imágenes — se reenviaron como texto; usa image_ocr para leerlas",
|
||||
"ImageInputOmitted": "{model} no puede ver imágenes — {count} imagen(es) se enviaron como texto en este turno; usa image_ocr para leerlas o /model para cambiar",
|
||||
"RequestImageWindow": "{model} acepta como máximo {cap} imagen(es) por solicitud: se envían las {kept} más recientes de {total}; {elided} imagen(es) más antiguas van como notas de texto",
|
||||
"ImageCountRejected": "{model} rechazó las {count} imagen(es) de esta solicitud; Codewhale no la reenvió sin cambios. Envía de nuevo para reintentar con menos imágenes, o cambia de modelo",
|
||||
"ProviderToolCallMissing": "El proveedor terminó con `{reason}`, pero no incluyó ninguna llamada a herramientas. Vuelve a intentar este turno para continuar.",
|
||||
"ConfigLabelShowThinking": "Razonamiento del modelo en el chat",
|
||||
"ConfigLabelThinkingHighlight": "Resaltado de fondo del razonamiento",
|
||||
|
||||
@@ -370,6 +370,8 @@
|
||||
"ScreenModeUnchanged": "Écran inchangé : {reason}.",
|
||||
"ImageInputRejectedResent": "{model} n’accepte pas les images — elles ont été renvoyées sous forme de texte ; utilisez image_ocr pour les lire",
|
||||
"ImageInputOmitted": "{model} ne voit pas les images — {count} image(s) envoyée(s) sous forme de texte à ce tour ; utilisez image_ocr pour les lire ou /model pour changer",
|
||||
"RequestImageWindow": "{model} accepte au plus {cap} image(s) par requête : envoi des {kept} plus récentes sur {total} ; {elided} image(s) plus ancienne(s) partent sous forme de notes texte",
|
||||
"ImageCountRejected": "{model} a refusé les {count} image(s) de cette requête ; Codewhale ne l'a pas renvoyée telle quelle. Renvoyez pour réessayer avec moins d'images, ou changez de modèle",
|
||||
"ProviderToolCallMissing": "Le fournisseur a terminé avec `{reason}`, mais sans fournir d’appel d’outil. Réessayez ce tour pour continuer.",
|
||||
"ConfigLabelShowThinking": "Raisonnement du modèle dans le chat",
|
||||
"ConfigLabelThinkingHighlight": "Surlignage du fond du raisonnement",
|
||||
|
||||
@@ -370,6 +370,8 @@
|
||||
"ScreenModeUnchanged": "स्क्रीन अपरिवर्तित: {reason}।",
|
||||
"ImageInputRejectedResent": "{model} इमेज स्वीकार नहीं करता — उन्हें टेक्स्ट के रूप में फिर भेजा गया; पढ़ने के लिए image_ocr का उपयोग करें",
|
||||
"ImageInputOmitted": "{model} इमेज नहीं देख सकता — इस टर्न में {count} इमेज टेक्स्ट के रूप में भेजी गईं; पढ़ने के लिए image_ocr या बदलने के लिए /model का उपयोग करें",
|
||||
"RequestImageWindow": "{model} प्रति अनुरोध अधिकतम {cap} इमेज स्वीकार करता है: {total} में से नवीनतम {kept} भेजी जा रही हैं; {elided} पुरानी इमेज टेक्स्ट नोट के रूप में जाती हैं",
|
||||
"ImageCountRejected": "{model} ने इस अनुरोध की {count} इमेज अस्वीकार कर दीं; Codewhale ने इसे बिना बदलाव के दोबारा नहीं भेजा। कम इमेज के साथ फिर से प्रयास करने के लिए दोबारा भेजें, या मॉडल बदलें",
|
||||
"ProviderToolCallMissing": "प्रदाता ने `{reason}` के साथ उत्तर समाप्त किया, लेकिन कोई टूल कॉल नहीं दिया। जारी रखने के लिए इस टर्न को फिर से आज़माएँ।",
|
||||
"ConfigLabelShowThinking": "चैट में मॉडल रीज़निंग",
|
||||
"ConfigLabelThinkingHighlight": "रीज़निंग पृष्ठभूमि हाइलाइट",
|
||||
|
||||
@@ -370,6 +370,8 @@
|
||||
"ScreenModeUnchanged": "Layar tidak berubah: {reason}.",
|
||||
"ImageInputRejectedResent": "{model} tidak menerima gambar — gambar dikirim ulang sebagai teks; gunakan image_ocr untuk membacanya",
|
||||
"ImageInputOmitted": "{model} tidak dapat melihat gambar — {count} gambar dikirim sebagai teks pada giliran ini; gunakan image_ocr untuk membacanya atau /model untuk beralih",
|
||||
"RequestImageWindow": "{model} menerima paling banyak {cap} gambar per permintaan: mengirim {kept} terbaru dari {total}; {elided} gambar lama dikirim sebagai catatan teks",
|
||||
"ImageCountRejected": "{model} menolak {count} gambar dalam permintaan ini; Codewhale tidak mengirimnya ulang tanpa perubahan. Kirim lagi untuk mencoba dengan lebih sedikit gambar, atau ganti model",
|
||||
"ProviderToolCallMissing": "Penyedia mengakhiri respons dengan `{reason}`, tetapi tidak menyertakan panggilan alat. Coba ulang giliran ini untuk melanjutkan.",
|
||||
"ConfigLabelShowThinking": "Penalaran model di chat",
|
||||
"ConfigLabelThinkingHighlight": "Sorotan latar penalaran",
|
||||
|
||||
@@ -370,6 +370,8 @@
|
||||
"ScreenModeUnchanged": "画面は変更されませんでした:{reason}。",
|
||||
"ImageInputRejectedResent": "{model} は画像を受け付けないため、画像をテキストとして再送しました。画像を読むには image_ocr を使用してください",
|
||||
"ImageInputOmitted": "{model} は画像を認識できないため、このターンの画像 {count} 件をテキストとして送信しました。読むには image_ocr、切り替えるには /model を使用してください",
|
||||
"RequestImageWindow": "{model} は 1 リクエストあたり最大 {cap} 件の画像しか受け付けません。{total} 件中の新しい {kept} 件を送信し、古い {elided} 件はテキストの注記として送ります",
|
||||
"ImageCountRejected": "{model} がこのリクエストの画像 {count} 件を拒否しました。Codewhale は同じ内容のまま再送していません。もう一度送信すると画像を減らして再試行します。またはモデルを切り替えてください",
|
||||
"ProviderToolCallMissing": "プロバイダーは `{reason}` で応答を終了しましたが、ツール呼び出しがありません。続けるにはこのターンを再試行してください。",
|
||||
"ConfigLabelShowThinking": "会話にモデル推論を表示",
|
||||
"ConfigLabelThinkingHighlight": "推論の背景ハイライト",
|
||||
|
||||
@@ -370,6 +370,8 @@
|
||||
"ScreenModeUnchanged": "화면 변경 없음: {reason}.",
|
||||
"ImageInputRejectedResent": "{model}은(는) 이미지를 지원하지 않아 텍스트로 다시 보냈습니다. image_ocr를 사용해 읽으세요",
|
||||
"ImageInputOmitted": "{model}은(는) 이미지를 볼 수 없어 이번 턴의 이미지 {count}개를 텍스트로 보냈습니다. image_ocr로 읽거나 /model로 전환하세요",
|
||||
"RequestImageWindow": "{model}은(는) 요청당 최대 {cap}개의 이미지만 받습니다. {total}개 중 최신 {kept}개를 보내고, 오래된 {elided}개는 텍스트 메모로 보냅니다",
|
||||
"ImageCountRejected": "{model}이(가) 이 요청의 이미지 {count}개를 거부했습니다. Codewhale은 변경 없이 다시 보내지 않았습니다. 다시 보내면 더 적은 이미지로 재시도하며, 모델을 바꿀 수도 있습니다",
|
||||
"ProviderToolCallMissing": "공급자가 `{reason}`으로 응답을 종료했지만 도구 호출을 제공하지 않았습니다. 계속하려면 이 턴을 다시 시도하세요.",
|
||||
"ConfigLabelShowThinking": "채팅에 모델 추론 표시",
|
||||
"ConfigLabelThinkingHighlight": "추론 배경 강조",
|
||||
|
||||
@@ -370,6 +370,8 @@
|
||||
"ScreenModeUnchanged": "Tela inalterada: {reason}.",
|
||||
"ImageInputRejectedResent": "{model} não aceita imagens — reenviado como texto; use image_ocr para lê-las",
|
||||
"ImageInputOmitted": "{model} não consegue ver imagens — {count} imagem(ns) enviada(s) como texto neste turno; use image_ocr para lê-las ou /model para trocar",
|
||||
"RequestImageWindow": "{model} aceita no máximo {cap} imagem(ns) por solicitação: enviando as {kept} mais recentes de {total}; {elided} imagem(ns) mais antiga(s) vão como notas de texto",
|
||||
"ImageCountRejected": "{model} recusou as {count} imagem(ns) desta solicitação; o Codewhale não a reenviou sem alterações. Envie novamente para tentar com menos imagens, ou troque de modelo",
|
||||
"ProviderToolCallMissing": "O provedor encerrou com `{reason}`, mas não forneceu nenhuma chamada de ferramenta. Tente este turno novamente para continuar.",
|
||||
"ConfigLabelShowThinking": "Raciocínio do modelo no chat",
|
||||
"ConfigLabelThinkingHighlight": "Destaque do fundo do raciocínio",
|
||||
|
||||
@@ -370,6 +370,8 @@
|
||||
"ScreenModeUnchanged": "Экран без изменений: {reason}.",
|
||||
"ImageInputRejectedResent": "{model} не принимает изображения — они отправлены повторно как текст; используйте image_ocr для чтения",
|
||||
"ImageInputOmitted": "{model} не видит изображения — {count} изображ. в этом ходе отправлено как текст; используйте image_ocr для чтения или /model для смены модели",
|
||||
"RequestImageWindow": "{model} принимает не более {cap} изображ. за запрос: отправляются {kept} новейших из {total}; {elided} старых изображ. уходят как текстовые заметки",
|
||||
"ImageCountRejected": "{model} отклонил {count} изображ. в этом запросе; Codewhale не отправил его повторно без изменений. Отправьте снова, чтобы повторить с меньшим числом изображений, или смените модель",
|
||||
"ProviderToolCallMissing": "Провайдер завершил ответ с `{reason}`, но не передал вызов инструмента. Повторите этот ход, чтобы продолжить.",
|
||||
"ConfigLabelShowThinking": "Рассуждения модели в чате",
|
||||
"ConfigLabelThinkingHighlight": "Подсветка фона рассуждений",
|
||||
|
||||
@@ -370,6 +370,8 @@
|
||||
"ScreenModeUnchanged": "Екран без змін: {reason}.",
|
||||
"ImageInputRejectedResent": "{model} не приймає зображення — їх повторно надіслано як текст; скористайтеся image_ocr, щоб їх прочитати",
|
||||
"ImageInputOmitted": "{model} не бачить зображень — {count} зображ. у цьому ході надіслано як текст; скористайтеся image_ocr, щоб прочитати, або /model, щоб змінити модель",
|
||||
"RequestImageWindow": "{model} приймає не більше {cap} зображ. за запит: надсилаються {kept} найновіших із {total}; {elided} старіших зображ. йдуть як текстові нотатки",
|
||||
"ImageCountRejected": "{model} відхилив {count} зображ. у цьому запиті; Codewhale не надіслав його повторно без змін. Надішліть ще раз, щоб повторити з меншою кількістю зображень, або змініть модель",
|
||||
"ProviderToolCallMissing": "Провайдер завершив відповідь із `{reason}`, але не передав виклик інструмента. Повторіть цей хід, щоб продовжити.",
|
||||
"ConfigLabelShowThinking": "Міркування моделі в чаті",
|
||||
"ConfigLabelThinkingHighlight": "Підсвічування тла міркувань",
|
||||
|
||||
@@ -370,6 +370,8 @@
|
||||
"ScreenModeUnchanged": "Màn hình không thay đổi: {reason}.",
|
||||
"ImageInputRejectedResent": "{model} không chấp nhận hình ảnh — đã gửi lại dưới dạng văn bản; hãy dùng image_ocr để đọc chúng",
|
||||
"ImageInputOmitted": "{model} không thể xem hình ảnh — {count} hình ảnh đã được gửi dưới dạng văn bản trong lượt này; hãy dùng image_ocr để đọc hoặc /model để chuyển",
|
||||
"RequestImageWindow": "{model} chỉ nhận tối đa {cap} hình ảnh mỗi yêu cầu: gửi {kept} hình mới nhất trong {total}; {elided} hình cũ hơn được gửi dưới dạng ghi chú văn bản",
|
||||
"ImageCountRejected": "{model} đã từ chối {count} hình ảnh trong yêu cầu này; Codewhale không gửi lại nguyên trạng. Gửi lại để thử với ít hình ảnh hơn, hoặc đổi mô hình",
|
||||
"ProviderToolCallMissing": "Nhà cung cấp kết thúc với `{reason}` nhưng không gửi lệnh gọi công cụ nào. Hãy thử lại lượt này để tiếp tục.",
|
||||
"ConfigLabelShowThinking": "Hiển thị suy luận trong trò chuyện",
|
||||
"ConfigLabelThinkingHighlight": "Tô nền phần suy luận",
|
||||
|
||||
@@ -371,6 +371,8 @@
|
||||
"ScreenModeUnchanged": "屏幕未改变:{reason}。",
|
||||
"ImageInputRejectedResent": "{model} 不接受图像,已将图像替换为文本重新发送;请使用 image_ocr 读取图像",
|
||||
"ImageInputOmitted": "{model} 无法查看图像,本回合 {count} 张图像已作为文本发送;请使用 image_ocr 读取,或用 /model 切换模型",
|
||||
"RequestImageWindow": "{model} 每个请求最多接受 {cap} 张图像:发送 {total} 张中最新的 {kept} 张;较早的 {elided} 张以文本说明代替",
|
||||
"ImageCountRejected": "{model} 拒绝了此请求中的 {count} 张图像;Codewhale 没有原样重发。再次发送将以更少的图像重试,或切换模型",
|
||||
"ProviderToolCallMissing": "提供商以 `{reason}` 结束响应,但未提供工具调用。请重试本回合以继续。",
|
||||
"ConfigLabelShowThinking": "对话中显示模型推理",
|
||||
"ConfigLabelThinkingHighlight": "思考背景高亮",
|
||||
|
||||
@@ -369,6 +369,8 @@
|
||||
"ScreenModeUnchanged": "畫面未變更:{reason}。",
|
||||
"ImageInputRejectedResent": "{model} 不接受圖片,已將圖片替換為文字重新傳送;請使用 image_ocr 讀取圖片",
|
||||
"ImageInputOmitted": "{model} 無法檢視圖片,本回合 {count} 張圖片已作為文字傳送;請使用 image_ocr 讀取,或用 /model 切換模型",
|
||||
"RequestImageWindow": "{model} 每個請求最多接受 {cap} 張圖片:傳送 {total} 張中最新的 {kept} 張;較早的 {elided} 張以文字說明代替",
|
||||
"ImageCountRejected": "{model} 拒絕了此請求中的 {count} 張圖片;Codewhale 沒有原樣重送。再次傳送將以更少的圖片重試,或切換模型",
|
||||
"ProviderToolCallMissing": "供應商以 `{reason}` 結束回應,但未提供工具呼叫。請重試本回合以繼續。",
|
||||
"ConfigLabelShowThinking": "對話中顯示模型推理",
|
||||
"ConfigLabelThinkingHighlight": "思考背景醒目提示",
|
||||
|
||||
@@ -190,6 +190,8 @@ pub enum MessageId {
|
||||
ScreenModeUnchanged,
|
||||
ImageInputRejectedResent,
|
||||
ImageInputOmitted,
|
||||
RequestImageWindow,
|
||||
ImageCountRejected,
|
||||
ProviderToolCallMissing,
|
||||
// First-run anonymous usage disclosure.
|
||||
TelemetryNoticeDefaultOn,
|
||||
@@ -2867,6 +2869,8 @@ pub const ALL_MESSAGE_IDS: &[MessageId] = &[
|
||||
MessageId::ScreenModeUnchanged,
|
||||
MessageId::ImageInputRejectedResent,
|
||||
MessageId::ImageInputOmitted,
|
||||
MessageId::RequestImageWindow,
|
||||
MessageId::ImageCountRejected,
|
||||
MessageId::ProviderToolCallMissing,
|
||||
MessageId::TelemetryNoticeDefaultOn,
|
||||
MessageId::TelemetryNoticeHeadline,
|
||||
|
||||
@@ -300,6 +300,18 @@ pub enum EventMsg {
|
||||
omitted_tool_count: u64,
|
||||
},
|
||||
|
||||
RequestImageWindow {
|
||||
thread_id: ThreadId,
|
||||
session_id: SessionId,
|
||||
provider: String,
|
||||
model: String,
|
||||
total: u64,
|
||||
elided: u64,
|
||||
cap: u64,
|
||||
cap_source: String,
|
||||
message: String,
|
||||
},
|
||||
|
||||
/// Workspace snapshots (undo) are off for this workspace. `reason` is one
|
||||
/// rendered, localized line: the consequence, the gate that refused, and
|
||||
/// the recovery that actually lifts *that* gate (the size cap's config key
|
||||
@@ -800,6 +812,7 @@ pub struct EventEnvelope {
|
||||
/// proves the list and `kind_str` agree.
|
||||
pub const EVENT_KINDS: &[&str] = &[
|
||||
"tool_projection_warning",
|
||||
"request_image_window",
|
||||
"message_started",
|
||||
"response_delta",
|
||||
"message_complete",
|
||||
@@ -856,6 +869,7 @@ impl EventMsg {
|
||||
pub fn kind_str(&self) -> &'static str {
|
||||
match self {
|
||||
Self::ToolProjectionWarning { .. } => "tool_projection_warning",
|
||||
Self::RequestImageWindow { .. } => "request_image_window",
|
||||
Self::SnapshotsDisabled { .. } => "snapshots_disabled",
|
||||
Self::MessageStarted { .. } => "message_started",
|
||||
Self::ResponseDelta { .. } => "response_delta",
|
||||
@@ -915,6 +929,7 @@ impl EventMsg {
|
||||
pub fn thread_id(&self) -> &ThreadId {
|
||||
match self {
|
||||
Self::ToolProjectionWarning { thread_id, .. }
|
||||
| Self::RequestImageWindow { thread_id, .. }
|
||||
| Self::SnapshotsDisabled { thread_id, .. }
|
||||
| Self::MessageStarted { thread_id, .. }
|
||||
| Self::ResponseDelta { thread_id, .. }
|
||||
@@ -974,6 +989,7 @@ impl EventMsg {
|
||||
pub fn session_id(&self) -> &SessionId {
|
||||
match self {
|
||||
Self::ToolProjectionWarning { session_id, .. }
|
||||
| Self::RequestImageWindow { session_id, .. }
|
||||
| Self::SnapshotsDisabled { session_id, .. }
|
||||
| Self::MessageStarted { session_id, .. }
|
||||
| Self::ResponseDelta { session_id, .. }
|
||||
@@ -1081,6 +1097,17 @@ mod tests {
|
||||
omitted_tool_names: vec!["a".into()],
|
||||
omitted_tool_count: 3,
|
||||
},
|
||||
EventMsg::RequestImageWindow {
|
||||
thread_id: t.clone(),
|
||||
session_id: s.clone(),
|
||||
provider: "vllm".into(),
|
||||
model: "m".into(),
|
||||
total: 6,
|
||||
elided: 3,
|
||||
cap: 3,
|
||||
cap_source: "provider_rejection".into(),
|
||||
message: "m".into(),
|
||||
},
|
||||
EventMsg::MessageStarted {
|
||||
thread_id: t.clone(),
|
||||
session_id: s.clone(),
|
||||
|
||||
@@ -7,6 +7,36 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
- A long session whose history holds more images than its model accepts in
|
||||
one request no longer becomes permanently unsendable. When a provider
|
||||
refuses the number of images (for example vLLM's "At most 3 image(s) may
|
||||
be provided in one prompt", "too many images" or "maximum number of
|
||||
images"), Codewhale learns that limit for the provider and model (the
|
||||
number the provider states, otherwise half the images sent) and resends
|
||||
with only the newest images attached. Each older image is replaced by a
|
||||
fixed note telling the model it remains in the session and not to guess
|
||||
what it showed; a tool image's note says to take a fresh observation, such
|
||||
as a new screenshot, rather than repeat the action. Each resend carries
|
||||
strictly fewer images, there are at most three per turn and never an
|
||||
unchanged one; after that the turn ends with an `image_count_exceeded`
|
||||
error and the next message uses the learned limit. A stated limit of 0
|
||||
sends images as text, as for a model without image input. Before, a count
|
||||
rejection either ended every later turn the same way or was misread as
|
||||
"no image input" and stripped every image. The window moves in steps of a
|
||||
quarter of the limit, so the cached history prefix changes only once every
|
||||
few new images. Each change is recorded as a `request_image_window` event
|
||||
(runtime threads store it as a status item with that code, `exec
|
||||
--output-format stream-json` prints it and the TUI shows it), and
|
||||
`/preview-request` and the compaction summary request apply the same
|
||||
window. A sub-agent no longer retries a count rejection as a transient
|
||||
failure. Known limits: nothing changes until a provider has refused once,
|
||||
and no documented per-model limits are applied; the learned limit is kept
|
||||
in memory per engine and per sub-agent, so a restart learns it again with
|
||||
one refused request; the oldest images go first, user attachments
|
||||
included; a refusal that arrives mid-stream ends that request and only the
|
||||
next send uses the limit; a gateway that reports the refusal as a 5xx is
|
||||
retried unchanged by the transport first; and a request-body size (HTTP
|
||||
413) refusal on an ordinary turn still has no recovery (Refs CW-7235).
|
||||
- Switching provider is now refused while a turn is running, as model,
|
||||
thinking and permission changes already were. The refusal covers
|
||||
`/provider <id>`, the provider picker, a custom provider, setup and login
|
||||
|
||||
@@ -105,6 +105,7 @@ pub struct PreparedCompactionEnvelope {
|
||||
/// request that omits it shares no cacheable prefix with the turn it
|
||||
/// summarizes and re-bills the whole history uncached.
|
||||
pub reasoning_effort: Option<String>,
|
||||
pub image_cap: Option<usize>,
|
||||
/// User-visible progress sink supplied by the host that owns the run
|
||||
/// (the interactive engine, typically). `None` keeps every notice in the
|
||||
/// log only — the pass itself never depends on it.
|
||||
@@ -118,6 +119,7 @@ impl std::fmt::Debug for PreparedCompactionEnvelope {
|
||||
.field("session_id", &self.session_id)
|
||||
.field("tools", &self.tools)
|
||||
.field("reasoning_effort", &self.reasoning_effort)
|
||||
.field("image_cap", &self.image_cap)
|
||||
.field("notice_sink", &self.notice_sink.as_ref().map(|_| "<sink>"))
|
||||
.finish()
|
||||
}
|
||||
@@ -131,6 +133,7 @@ impl PartialEq for PreparedCompactionEnvelope {
|
||||
&& self.session_id == other.session_id
|
||||
&& self.tools == other.tools
|
||||
&& self.reasoning_effort == other.reasoning_effort
|
||||
&& self.image_cap == other.image_cap
|
||||
}
|
||||
}
|
||||
|
||||
@@ -142,6 +145,7 @@ impl PreparedCompactionEnvelope {
|
||||
session_id: None,
|
||||
tools: None,
|
||||
reasoning_effort: None,
|
||||
image_cap: None,
|
||||
notice_sink: None,
|
||||
}
|
||||
}
|
||||
@@ -1447,6 +1451,7 @@ pub async fn compact_messages_safe(
|
||||
system_prompt,
|
||||
prepared.tools.as_deref(),
|
||||
prepared.reasoning_effort.as_deref(),
|
||||
prepared.image_cap,
|
||||
prepared.notice_sink.as_deref(),
|
||||
&mut quality_retries,
|
||||
invocation_usage,
|
||||
@@ -1613,6 +1618,7 @@ async fn compact_messages(
|
||||
None,
|
||||
None,
|
||||
None,
|
||||
None,
|
||||
&mut quality_retries,
|
||||
&mut invocation_usage,
|
||||
)
|
||||
@@ -1627,6 +1633,7 @@ async fn compact_messages_with_metadata(
|
||||
system_prompt: Option<&SystemPrompt>,
|
||||
tools: Option<&[Tool]>,
|
||||
reasoning_effort: Option<&str>,
|
||||
image_cap: Option<usize>,
|
||||
notice_sink: Option<&dyn CompactionNoticeSink>,
|
||||
quality_retries: &mut u32,
|
||||
invocation_usage: &mut Usage,
|
||||
@@ -1642,6 +1649,7 @@ async fn compact_messages_with_metadata(
|
||||
system_prompt,
|
||||
tools,
|
||||
reasoning_effort,
|
||||
image_cap,
|
||||
notice_sink,
|
||||
quality_retries,
|
||||
invocation_usage,
|
||||
@@ -1877,6 +1885,7 @@ async fn create_summary(
|
||||
system_prompt: Option<&SystemPrompt>,
|
||||
tools: Option<&[Tool]>,
|
||||
reasoning_effort: Option<&str>,
|
||||
image_cap: Option<usize>,
|
||||
notice_sink: Option<&dyn CompactionNoticeSink>,
|
||||
quality_retries: &mut u32,
|
||||
invocation_usage: &mut Usage,
|
||||
@@ -1895,6 +1904,20 @@ async fn create_summary(
|
||||
"Compaction omitted {stripped_images} image block(s) unsupported by its route"
|
||||
));
|
||||
}
|
||||
if let Some(cap) = image_cap {
|
||||
let window = crate::image_attach::cap_request_images(&mut request_messages, cap);
|
||||
if window.elided > 0 {
|
||||
let message = format!(
|
||||
"Making room: the summary request sends the newest {} of {} images; {} older go as text notes (limit {} per request).",
|
||||
window.total - window.elided,
|
||||
window.total,
|
||||
window.elided,
|
||||
window.cap,
|
||||
);
|
||||
logging::warn(&message);
|
||||
deliver_compaction_notice(notice_sink, message);
|
||||
}
|
||||
}
|
||||
request_messages.push(Message {
|
||||
role: Role::User,
|
||||
content: vec![ContentBlock::Text {
|
||||
|
||||
@@ -1053,6 +1053,8 @@ pub struct Engine {
|
||||
/// key, and wire protocol. Part of the route identity the input-bill
|
||||
/// carry-over is keyed on; `None` until a route is installed.
|
||||
active_route_endpoint: Option<codewhale_config::route::ResolvedEndpoint>,
|
||||
learned_image_caps: HashMap<(String, String), usize>,
|
||||
last_image_window: Option<(String, String, usize, usize)>,
|
||||
rx_op: mpsc::Receiver<Op>,
|
||||
live_runtime_authority: Arc<StdMutex<LiveRuntimeAuthorityState>>,
|
||||
compaction_cancellation: Arc<StdMutex<CompactionCancellationState>>,
|
||||
@@ -1748,6 +1750,97 @@ impl Engine {
|
||||
format!("{message}\n\n{hint}")
|
||||
}
|
||||
|
||||
pub(super) fn image_route_provider_key(&self) -> String {
|
||||
self.api_provider_identity.as_ref().map_or_else(
|
||||
|| self.api_provider.as_str().to_string(),
|
||||
|identity| identity.key.to_string(),
|
||||
)
|
||||
}
|
||||
|
||||
pub(super) fn request_image_policy(
|
||||
&self,
|
||||
provider: &str,
|
||||
model: &str,
|
||||
route_image_input: CapabilityState,
|
||||
) -> (CapabilityState, Option<usize>) {
|
||||
match self
|
||||
.learned_image_caps
|
||||
.get(&(provider.to_string(), model.to_string()))
|
||||
{
|
||||
Some(0) => (CapabilityState::Unsupported, None),
|
||||
cap => (route_image_input, cap.copied()),
|
||||
}
|
||||
}
|
||||
|
||||
pub(super) fn learn_image_cap(
|
||||
&mut self,
|
||||
model: &str,
|
||||
sent: usize,
|
||||
rejection: crate::llm_client::ImageCountRejection,
|
||||
) -> Option<usize> {
|
||||
if sent == 0 {
|
||||
return None;
|
||||
}
|
||||
let cap = match rejection.limit {
|
||||
Some(limit) if limit < sent => limit,
|
||||
_ if sent <= 1 => 0,
|
||||
_ => sent / 2,
|
||||
};
|
||||
let key = (self.image_route_provider_key(), model.to_string());
|
||||
let learned = self.learned_image_caps.entry(key).or_insert(cap);
|
||||
*learned = (*learned).min(cap);
|
||||
crate::logging::warn(format!(
|
||||
"request_images: model {model} refused {sent} image(s) in one request (stated limit {:?}); learned cap {}",
|
||||
rejection.limit, *learned
|
||||
));
|
||||
Some(*learned)
|
||||
}
|
||||
|
||||
pub(super) async fn note_request_image_window(
|
||||
&mut self,
|
||||
provider: String,
|
||||
window: crate::image_attach::ImageWindow,
|
||||
) {
|
||||
let model = self.session.model.clone();
|
||||
let current = (window.elided > 0)
|
||||
.then(|| (provider.clone(), model.clone(), window.cap, window.elided));
|
||||
if current == self.last_image_window {
|
||||
return;
|
||||
}
|
||||
self.last_image_window = current;
|
||||
if window.elided == 0 {
|
||||
return;
|
||||
}
|
||||
let cap_source = "provider_rejection";
|
||||
crate::logging::warn(format!(
|
||||
"request_images: provider={provider} model={model} total={} elided={} cap={} cap_source={cap_source}",
|
||||
window.total, window.elided, window.cap
|
||||
));
|
||||
if let Some(pm) = self.session.prefix_stability.as_mut() {
|
||||
pm.note_history_reset("image_window");
|
||||
}
|
||||
let message = codewhale_localization::tr(
|
||||
codewhale_localization::resolve_locale(&self.config.locale_tag),
|
||||
codewhale_localization::MessageId::RequestImageWindow,
|
||||
)
|
||||
.replace("{model}", &model)
|
||||
.replace("{cap}", &window.cap.to_string())
|
||||
.replace("{kept}", &(window.total - window.elided).to_string())
|
||||
.replace("{total}", &window.total.to_string())
|
||||
.replace("{elided}", &window.elided.to_string());
|
||||
let _ = self
|
||||
.send_event(Event::RequestImageWindow {
|
||||
provider,
|
||||
model,
|
||||
total: window.total,
|
||||
elided: window.elided,
|
||||
cap: window.cap,
|
||||
cap_source: cap_source.to_string(),
|
||||
message,
|
||||
})
|
||||
.await;
|
||||
}
|
||||
|
||||
/// A provider's input bill describes one route's tokenization of one
|
||||
/// prompt. The auto-compaction gate and the preflight guard lift the
|
||||
/// honest estimate to the last bill, so a bill carried across a route
|
||||
@@ -2361,6 +2454,8 @@ impl Engine {
|
||||
active_route_limits,
|
||||
active_route_capabilities: codewhale_config::route::RouteCapabilities::default(),
|
||||
active_route_endpoint: None,
|
||||
learned_image_caps: HashMap::new(),
|
||||
last_image_window: None,
|
||||
rx_op,
|
||||
live_runtime_authority: Arc::clone(&live_runtime_authority),
|
||||
compaction_cancellation: Arc::clone(&compaction_cancellation),
|
||||
|
||||
@@ -228,7 +228,14 @@ impl Engine {
|
||||
config
|
||||
.workspace
|
||||
.get_or_insert_with(|| self.config.workspace.clone());
|
||||
let (image_input, image_cap) = self.request_image_policy(
|
||||
&self.image_route_provider_key(),
|
||||
&config.model,
|
||||
config.image_input,
|
||||
);
|
||||
config.image_input = image_input;
|
||||
let mut prepared = PreparedCompactionEnvelope::new(config);
|
||||
prepared.image_cap = image_cap;
|
||||
prepared.session_id = Some(self.session.id.clone());
|
||||
prepared.notice_sink = Some(std::sync::Arc::new(EngineCompactionNoticeSink {
|
||||
tx: self.tx_event.clone(),
|
||||
|
||||
@@ -872,6 +872,11 @@ pub(super) fn context_does_not_fit_message(
|
||||
|
||||
pub(super) fn is_image_input_rejection_message(message: &str) -> bool {
|
||||
let lower = message.to_lowercase();
|
||||
if crate::llm_client::image_count_rejection(&lower)
|
||||
.is_some_and(|rejection| rejection.limit != Some(0))
|
||||
{
|
||||
return false;
|
||||
}
|
||||
let image_signal = lower.contains("image_url")
|
||||
|| lower.contains("content.type")
|
||||
|| lower.contains("content type")
|
||||
|
||||
@@ -465,7 +465,7 @@ impl Engine {
|
||||
system_prompt.as_ref(),
|
||||
);
|
||||
|
||||
let request = prepare_primary_turn_request(PrimaryTurnRequest {
|
||||
let mut request = prepare_primary_turn_request(PrimaryTurnRequest {
|
||||
model: model.clone(),
|
||||
messages: outbound_messages,
|
||||
max_tokens: effective_max_output_tokens_for_route(provider, &model, limits),
|
||||
@@ -474,6 +474,19 @@ impl Engine {
|
||||
tool_choice: tool_choice.clone(),
|
||||
reasoning_effort: effective_reasoning_effort,
|
||||
});
|
||||
let (image_input, image_cap) = self.request_image_policy(
|
||||
route.identity.key.as_str(),
|
||||
&model,
|
||||
route_context.capabilities.image_input,
|
||||
);
|
||||
crate::image_attach::strip_images_when_unsupported(
|
||||
&mut request.messages,
|
||||
image_input,
|
||||
&model,
|
||||
);
|
||||
if let Some(cap) = image_cap {
|
||||
crate::image_attach::cap_request_images(&mut request.messages, cap);
|
||||
}
|
||||
|
||||
let prepared = match route.client.prepare_outbound_request(request, true) {
|
||||
Ok(prepared) => {
|
||||
|
||||
@@ -28,6 +28,7 @@ use codewhale_core::request::{PrimaryTurnRequest, prepare_primary_turn_request};
|
||||
use codewhale_models::Role;
|
||||
|
||||
const MAX_APPROVAL_INTENT_SUMMARY_CHARS: usize = 2_000;
|
||||
const MAX_IMAGE_COUNT_RECOVERIES: u8 = 3;
|
||||
|
||||
// Private bookkeeping for this one outer loop. The existing TurnContext,
|
||||
// Engine session, clocks, event queue, prompt and approval owners remain authoritative.
|
||||
@@ -38,6 +39,7 @@ struct TurnLoopProgress {
|
||||
context_recovery_attempts: u8,
|
||||
auto_compaction_suppressed: bool,
|
||||
image_rejection_recovered: bool,
|
||||
image_count_recoveries: u8,
|
||||
mode: AppMode,
|
||||
tool_catalog: Vec<codewhale_models::Tool>,
|
||||
active_tool_names: std::collections::HashSet<String>,
|
||||
@@ -1564,6 +1566,7 @@ impl Engine {
|
||||
context_recovery_attempts,
|
||||
auto_compaction_suppressed,
|
||||
image_rejection_recovered,
|
||||
image_count_recoveries: 0,
|
||||
mode,
|
||||
tool_catalog,
|
||||
active_tool_names,
|
||||
@@ -4550,6 +4553,27 @@ impl Engine {
|
||||
let message = self.decorate_auth_error_message(e.to_string());
|
||||
let user_message =
|
||||
stream_read_error_user_message(&message, any_content_received);
|
||||
if self.rlm_host.is_none()
|
||||
&& let Some(rejection) = crate::llm_client::image_count_rejection(
|
||||
&format!("{e:#}").to_lowercase(),
|
||||
)
|
||||
{
|
||||
let sent =
|
||||
crate::image_attach::count_request_images(&stream_request.messages);
|
||||
let model = self.session.model.clone();
|
||||
if self.learn_image_cap(&model, sent, rejection).is_some() {
|
||||
terminal_stream_error = true;
|
||||
stream_error.get_or_insert(user_message.clone());
|
||||
let _ = self
|
||||
.send_stream_event(Event::error(
|
||||
crate::error_taxonomy::ErrorEnvelope::image_count_exceeded(
|
||||
user_message,
|
||||
),
|
||||
))
|
||||
.await;
|
||||
break;
|
||||
}
|
||||
}
|
||||
let envelope =
|
||||
crate::error_taxonomy::envelope_for_llm_error(e, user_message.clone());
|
||||
if self.child_host.is_some() {
|
||||
|
||||
@@ -175,6 +175,15 @@ impl Engine {
|
||||
Ok(false) => {}
|
||||
}
|
||||
}
|
||||
if self.rlm_host.is_none()
|
||||
&& let Some(rejection) =
|
||||
crate::llm_client::image_count_rejection(&format!("{e:#}").to_lowercase())
|
||||
&& let Some(result) = self
|
||||
.recover_image_count_rejection(progress, &stream_request, rejection, &e)
|
||||
.await
|
||||
{
|
||||
return result;
|
||||
}
|
||||
if let Some(recovery) = self.recover_child_request(progress, &e).await {
|
||||
return recovery;
|
||||
}
|
||||
@@ -201,23 +210,9 @@ impl Engine {
|
||||
&& is_image_input_rejection_message(&message)
|
||||
&& self.active_route_capabilities.image_input != CapabilityState::Unsupported
|
||||
&& !progress.image_rejection_recovered
|
||||
&& crate::image_attach::count_request_images(&stream_request.messages) > 0
|
||||
{
|
||||
progress.image_rejection_recovered = true;
|
||||
// This path tells the user itself; the resend must
|
||||
// not announce the same omission a second time.
|
||||
progress.image_omission_notified = true;
|
||||
self.active_route_capabilities.image_input = CapabilityState::Unsupported;
|
||||
crate::logging::warn(format!(
|
||||
"model {} rejected image content; resending with images replaced by text",
|
||||
self.session.model
|
||||
));
|
||||
let status = codewhale_localization::tr(
|
||||
codewhale_localization::resolve_locale(&self.config.locale_tag),
|
||||
codewhale_localization::MessageId::ImageInputRejectedResent,
|
||||
)
|
||||
.replace("{model}", &self.session.model);
|
||||
let _ = self.send_event(Event::status(status)).await;
|
||||
return PhaseResult::Retry;
|
||||
return self.resend_without_images(progress).await;
|
||||
}
|
||||
let display_message = self.decorate_auth_error_message(
|
||||
initial_stream_error_user_message(&self.config.locale_tag, &e),
|
||||
@@ -1040,6 +1035,71 @@ impl Engine {
|
||||
prepared_output_tokens: stream_request.max_tokens,
|
||||
})
|
||||
}
|
||||
async fn resend_without_images(
|
||||
&mut self,
|
||||
progress: &mut TurnLoopProgress,
|
||||
) -> PhaseResult<AcceptedModelStep> {
|
||||
progress.image_rejection_recovered = true;
|
||||
// This path tells the user itself; the resend must
|
||||
// not announce the same omission a second time.
|
||||
progress.image_omission_notified = true;
|
||||
self.active_route_capabilities.image_input = CapabilityState::Unsupported;
|
||||
crate::logging::warn(format!(
|
||||
"model {} rejected image content; resending with images replaced by text",
|
||||
self.session.model
|
||||
));
|
||||
let status = codewhale_localization::tr(
|
||||
codewhale_localization::resolve_locale(&self.config.locale_tag),
|
||||
codewhale_localization::MessageId::ImageInputRejectedResent,
|
||||
)
|
||||
.replace("{model}", &self.session.model);
|
||||
let _ = self.send_event(Event::status(status)).await;
|
||||
PhaseResult::Retry
|
||||
}
|
||||
|
||||
async fn recover_image_count_rejection(
|
||||
&mut self,
|
||||
progress: &mut TurnLoopProgress,
|
||||
request: &codewhale_models::MessageRequest,
|
||||
rejection: crate::llm_client::ImageCountRejection,
|
||||
error: &anyhow::Error,
|
||||
) -> Option<PhaseResult<AcceptedModelStep>> {
|
||||
let sent = crate::image_attach::count_request_images(&request.messages);
|
||||
let model = self.session.model.clone();
|
||||
let cap = self.learn_image_cap(&model, sent, rejection)?;
|
||||
if cap == 0 {
|
||||
if self.active_route_capabilities.image_input != CapabilityState::Unsupported
|
||||
&& !progress.image_rejection_recovered
|
||||
{
|
||||
return Some(self.resend_without_images(progress).await);
|
||||
}
|
||||
} else if progress.image_count_recoveries < MAX_IMAGE_COUNT_RECOVERIES {
|
||||
progress.image_count_recoveries = progress.image_count_recoveries.saturating_add(1);
|
||||
return Some(PhaseResult::Retry);
|
||||
}
|
||||
let hint = codewhale_localization::tr(
|
||||
codewhale_localization::resolve_locale(&self.config.locale_tag),
|
||||
codewhale_localization::MessageId::ImageCountRejected,
|
||||
)
|
||||
.replace("{model}", &model)
|
||||
.replace("{count}", &sent.to_string());
|
||||
let detail = self.decorate_auth_error_message(initial_stream_error_user_message(
|
||||
&self.config.locale_tag,
|
||||
error,
|
||||
));
|
||||
let display_message = format!("{hint}\n{detail}");
|
||||
progress.turn_error = Some(display_message.clone());
|
||||
let _ = self
|
||||
.send_event(Event::error(
|
||||
crate::error_taxonomy::ErrorEnvelope::image_count_exceeded(display_message),
|
||||
))
|
||||
.await;
|
||||
Some(PhaseResult::Return((
|
||||
TurnOutcomeStatus::Failed,
|
||||
progress.turn_error.take(),
|
||||
)))
|
||||
}
|
||||
|
||||
async fn recover_child_request(
|
||||
&mut self,
|
||||
progress: &mut TurnLoopProgress,
|
||||
|
||||
@@ -608,9 +608,15 @@ impl Engine {
|
||||
// outbound copy is rewritten, and it is rewritten to text that says
|
||||
// why rather than being dropped.
|
||||
let fresh_images = crate::image_attach::images_since_last_user_prompt(&request.messages);
|
||||
let image_provider = self.image_route_provider_key();
|
||||
let (image_input, image_cap) = self.request_image_policy(
|
||||
&image_provider,
|
||||
&self.session.model,
|
||||
self.active_route_capabilities.image_input,
|
||||
);
|
||||
let stripped_images = crate::image_attach::strip_images_when_unsupported(
|
||||
&mut request.messages,
|
||||
self.active_route_capabilities.image_input,
|
||||
image_input,
|
||||
&self.session.model,
|
||||
);
|
||||
if stripped_images > 0 {
|
||||
@@ -629,6 +635,10 @@ impl Engine {
|
||||
let _ = self.send_event(Event::status(status)).await;
|
||||
}
|
||||
}
|
||||
if let Some(cap) = image_cap {
|
||||
let window = crate::image_attach::cap_request_images(&mut request.messages, cap);
|
||||
self.note_request_image_window(image_provider, window).await;
|
||||
}
|
||||
let tool_request_snapshot =
|
||||
crate::tool_inspection::ToolInspectionSnapshot::from_prepared_request_with_surface(
|
||||
&turn.id,
|
||||
|
||||
@@ -208,6 +208,16 @@ pub enum Event {
|
||||
omitted_tool_count: usize,
|
||||
},
|
||||
|
||||
RequestImageWindow {
|
||||
provider: String,
|
||||
model: String,
|
||||
total: usize,
|
||||
elided: usize,
|
||||
cap: usize,
|
||||
cap_source: String,
|
||||
message: String,
|
||||
},
|
||||
|
||||
/// Workspace snapshots (undo) could not be enabled for this workspace.
|
||||
/// Emitted once per session/workspace so another session cannot consume
|
||||
/// its notice. The disabled state also remains visible in `/status` (#5930).
|
||||
|
||||
@@ -454,6 +454,25 @@ pub fn event_to_protocol(event: &Event, ids: &ProtocolIds) -> wire::EventMsg {
|
||||
omitted_tool_names: omitted_tool_names.clone(),
|
||||
omitted_tool_count: count(*omitted_tool_count),
|
||||
},
|
||||
Event::RequestImageWindow {
|
||||
provider,
|
||||
model,
|
||||
total,
|
||||
elided,
|
||||
cap,
|
||||
cap_source,
|
||||
message,
|
||||
} => wire::EventMsg::RequestImageWindow {
|
||||
thread_id,
|
||||
session_id,
|
||||
provider: provider.clone(),
|
||||
model: model.clone(),
|
||||
total: count(*total),
|
||||
elided: count(*elided),
|
||||
cap: count(*cap),
|
||||
cap_source: cap_source.clone(),
|
||||
message: message.clone(),
|
||||
},
|
||||
Event::SnapshotsDisabled { workspace, reason } => wire::EventMsg::SnapshotsDisabled {
|
||||
thread_id,
|
||||
session_id,
|
||||
|
||||
@@ -156,6 +156,17 @@ impl ErrorEnvelope {
|
||||
)
|
||||
}
|
||||
|
||||
#[must_use]
|
||||
pub fn image_count_exceeded(message: impl Into<String>) -> Self {
|
||||
Self::new(
|
||||
ErrorCategory::InvalidInput,
|
||||
ErrorSeverity::Error,
|
||||
true,
|
||||
"image_count_exceeded",
|
||||
message,
|
||||
)
|
||||
}
|
||||
|
||||
/// A stop Codewhale applied to itself (step ceiling, per-turn wall clock).
|
||||
/// The turn failed, so the card is an error, but the session stays usable:
|
||||
/// sending another message continues, so it is `recoverable` and does not
|
||||
|
||||
@@ -923,6 +923,29 @@ pub(crate) async fn run_exec_agent(
|
||||
)
|
||||
);
|
||||
}
|
||||
Event::RequestImageWindow {
|
||||
provider,
|
||||
model,
|
||||
total,
|
||||
elided,
|
||||
cap,
|
||||
cap_source,
|
||||
message,
|
||||
} => {
|
||||
if output_format == ExecOutputFormat::StreamJson {
|
||||
emit_exec_stream_event(&ExecStreamEvent::RequestImageWindow {
|
||||
provider,
|
||||
model,
|
||||
total,
|
||||
elided,
|
||||
cap,
|
||||
cap_source,
|
||||
message,
|
||||
})?;
|
||||
} else if !json_output {
|
||||
eprintln!("{message}");
|
||||
}
|
||||
}
|
||||
Event::ToolCallStarted {
|
||||
id, name, input, ..
|
||||
} => {
|
||||
|
||||
@@ -1005,6 +1005,85 @@ pub fn strip_images_when_unsupported(
|
||||
stripped
|
||||
}
|
||||
|
||||
pub(crate) const ELIDED_IMAGE_NOTE: &str = "[Image not attached to this request: this model accepts only a limited number of images per request, so the oldest are left out. It remains in the session; do not guess what it showed.]";
|
||||
|
||||
pub(crate) const ELIDED_TOOL_IMAGE_NOTE: &str = "[Tool image not attached to this request: this model accepts only a limited number of images per request, so the oldest are left out; do not guess what it showed. Take a fresh observation (for example a new screenshot) if you need the current state; do not repeat an action to recover its image.]";
|
||||
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
|
||||
pub(crate) struct ImageWindow {
|
||||
pub total: usize,
|
||||
pub elided: usize,
|
||||
pub cap: usize,
|
||||
}
|
||||
|
||||
fn block_carries_image(block: &ContentBlock) -> bool {
|
||||
match block {
|
||||
ContentBlock::ImageUrl { .. } => true,
|
||||
ContentBlock::ToolResult { content_blocks, .. } => {
|
||||
content_blocks.as_ref().is_some_and(|blocks| {
|
||||
blocks.iter().any(|value| {
|
||||
value.get("type").and_then(serde_json::Value::as_str) == Some("image")
|
||||
})
|
||||
})
|
||||
}
|
||||
_ => false,
|
||||
}
|
||||
}
|
||||
|
||||
#[must_use]
|
||||
pub(crate) fn count_request_images(messages: &[codewhale_models::Message]) -> usize {
|
||||
messages
|
||||
.iter()
|
||||
.flat_map(|message| &message.content)
|
||||
.filter(|block| block_carries_image(block))
|
||||
.count()
|
||||
}
|
||||
|
||||
#[must_use]
|
||||
pub(crate) fn images_to_elide(total: usize, cap: usize) -> usize {
|
||||
if total <= cap {
|
||||
return 0;
|
||||
}
|
||||
let chunk = (cap / 4).max(1);
|
||||
((total - cap).div_ceil(chunk) * chunk).min(total)
|
||||
}
|
||||
|
||||
pub(crate) fn cap_request_images(
|
||||
messages: &mut [codewhale_models::Message],
|
||||
cap: usize,
|
||||
) -> ImageWindow {
|
||||
let total = count_request_images(messages);
|
||||
let elided = images_to_elide(total, cap);
|
||||
let mut remaining = elided;
|
||||
for block in messages.iter_mut().flat_map(|message| &mut message.content) {
|
||||
if remaining == 0 {
|
||||
break;
|
||||
}
|
||||
if !block_carries_image(block) {
|
||||
continue;
|
||||
}
|
||||
match block {
|
||||
ContentBlock::ImageUrl { .. } => {
|
||||
*block = ContentBlock::Text {
|
||||
text: ELIDED_IMAGE_NOTE.to_string(),
|
||||
cache_control: None,
|
||||
};
|
||||
}
|
||||
ContentBlock::ToolResult {
|
||||
content,
|
||||
content_blocks,
|
||||
..
|
||||
} => {
|
||||
*content_blocks = None;
|
||||
*content = format!("{content}\n{ELIDED_TOOL_IMAGE_NOTE}");
|
||||
}
|
||||
_ => continue,
|
||||
}
|
||||
remaining -= 1;
|
||||
}
|
||||
ImageWindow { total, elided, cap }
|
||||
}
|
||||
|
||||
/// Total inline-image byte budget for one compaction retry that follows a
|
||||
/// provider body-size rejection.
|
||||
///
|
||||
|
||||
@@ -13887,6 +13887,16 @@ struct ExecStreamInputAnalysis {
|
||||
enum ExecStreamEvent {
|
||||
#[serde(rename = "status")]
|
||||
Status { message: String },
|
||||
#[serde(rename = "request_image_window")]
|
||||
RequestImageWindow {
|
||||
provider: String,
|
||||
model: String,
|
||||
total: usize,
|
||||
elided: usize,
|
||||
cap: usize,
|
||||
cap_source: String,
|
||||
message: String,
|
||||
},
|
||||
#[serde(rename = "content")]
|
||||
Content { content: String },
|
||||
#[serde(rename = "tool_use")]
|
||||
|
||||
@@ -787,6 +787,70 @@ pub(crate) fn is_context_length_message(lower: &str) -> bool {
|
||||
.any(|phrase| lower.contains(phrase))
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub(crate) struct ImageCountRejection {
|
||||
pub limit: Option<usize>,
|
||||
}
|
||||
|
||||
pub(crate) fn image_count_rejection(lower: &str) -> Option<ImageCountRejection> {
|
||||
if is_context_length_message(lower) {
|
||||
return None;
|
||||
}
|
||||
let phrase = [
|
||||
"too many images",
|
||||
"maximum number of images",
|
||||
"max number of images",
|
||||
"max images",
|
||||
"maximum images",
|
||||
"images per request",
|
||||
"images per prompt",
|
||||
"images per message",
|
||||
"image(s) may be provided",
|
||||
"images may be provided",
|
||||
]
|
||||
.iter()
|
||||
.find_map(|phrase| lower.find(*phrase));
|
||||
let number_of_images = lower.find("number of images").filter(|_| {
|
||||
["exceed", "more than", "at most", "limit"]
|
||||
.iter()
|
||||
.any(|word| lower.contains(*word))
|
||||
});
|
||||
let at_most_images = lower.match_indices("at most ").find_map(|(index, marker)| {
|
||||
let rest = &lower[index + marker.len()..];
|
||||
let digits = rest.len() - rest.trim_start_matches(|c: char| c.is_ascii_digit()).len();
|
||||
(digits > 0 && rest[digits..].trim_start().starts_with("image")).then_some(index)
|
||||
});
|
||||
let start = phrase.or(number_of_images).or(at_most_images)?;
|
||||
let clause_start = lower[..start]
|
||||
.rfind(['.', '\n', '{', '"'])
|
||||
.map_or(0, |index| index + 1);
|
||||
let clause = &lower[clause_start..];
|
||||
let limit = [
|
||||
"at most ",
|
||||
"maximum of ",
|
||||
"max of ",
|
||||
"max is ",
|
||||
"maximum is ",
|
||||
"limit is ",
|
||||
"limit of ",
|
||||
"up to ",
|
||||
"allowed: ",
|
||||
"allowed ",
|
||||
"maximum: ",
|
||||
"max: ",
|
||||
"max ",
|
||||
"maximum ",
|
||||
]
|
||||
.iter()
|
||||
.find_map(|marker| {
|
||||
let rest = &clause[clause.find(*marker)? + marker.len()..];
|
||||
let rest = rest.trim_start();
|
||||
let digits = rest.len() - rest.trim_start_matches(|c: char| c.is_ascii_digit()).len();
|
||||
rest[..digits].parse::<usize>().ok()
|
||||
});
|
||||
Some(ImageCountRejection { limit })
|
||||
}
|
||||
|
||||
fn explicit_quota_code(body: &str) -> Option<String> {
|
||||
let value: Value = serde_json::from_str(body).ok()?;
|
||||
[
|
||||
|
||||
@@ -18772,6 +18772,48 @@ impl RuntimeThreadManager {
|
||||
)
|
||||
.await?;
|
||||
}
|
||||
EngineEvent::RequestImageWindow {
|
||||
provider,
|
||||
model,
|
||||
total,
|
||||
elided,
|
||||
cap,
|
||||
cap_source,
|
||||
message,
|
||||
} => {
|
||||
let item = TurnItemRecord {
|
||||
schema_version: CURRENT_RUNTIME_SCHEMA_VERSION,
|
||||
id: runtime_record_id("item"),
|
||||
turn_id: turn_id.clone(),
|
||||
kind: TurnItemKind::Status,
|
||||
status: TurnItemLifecycleStatus::Completed,
|
||||
summary: summarize_text(&message, SUMMARY_LIMIT),
|
||||
detail: Some(message),
|
||||
metadata: Some(json!({
|
||||
"code": "request_image_window",
|
||||
"provider": provider,
|
||||
"model": model,
|
||||
"total": total,
|
||||
"elided": elided,
|
||||
"cap": cap,
|
||||
"cap_source": cap_source,
|
||||
})),
|
||||
artifact_refs: Vec::new(),
|
||||
artifacts: Vec::new(),
|
||||
started_at: Some(Utc::now()),
|
||||
ended_at: Some(Utc::now()),
|
||||
};
|
||||
self.store.save_item(&item)?;
|
||||
self.attach_item_to_turn(&turn_id, &item.id)?;
|
||||
self.emit_event(
|
||||
&thread_id,
|
||||
Some(&turn_id),
|
||||
Some(&item.id),
|
||||
"item.completed",
|
||||
json!({ "item": item }),
|
||||
)
|
||||
.await?;
|
||||
}
|
||||
EngineEvent::Error { envelope, .. } => {
|
||||
turn_status = Some(RuntimeTurnStatus::Failed);
|
||||
turn_error = Some(envelope.message.clone());
|
||||
|
||||
@@ -15137,6 +15137,10 @@ fn is_transient_subagent_provider_error(error: &anyhow::Error) -> bool {
|
||||
// quota error mentioning "429" is not a transient rate limit. Retain the
|
||||
// transport fallback for generic/parse errors and relay HTTP 400s, which
|
||||
// can carry an upstream timeout rather than an invalid request.
|
||||
let message = format!("{error:#}").to_ascii_lowercase();
|
||||
if crate::llm_client::image_count_rejection(&message).is_some() {
|
||||
return false;
|
||||
}
|
||||
match error.downcast_ref::<LlmError>() {
|
||||
Some(
|
||||
LlmError::QuotaExhausted(_)
|
||||
@@ -15150,7 +15154,6 @@ fn is_transient_subagent_provider_error(error: &anyhow::Error) -> bool {
|
||||
_ => {}
|
||||
}
|
||||
|
||||
let message = format!("{error:#}").to_ascii_lowercase();
|
||||
[
|
||||
"did not receive response headers",
|
||||
"response headers",
|
||||
|
||||
@@ -3691,6 +3691,9 @@ pub(crate) async fn run_event_loop(
|
||||
.replace("{tools}", &tools);
|
||||
app.push_status_toast(message, StatusToastLevel::Warning, Some(12_000));
|
||||
}
|
||||
EngineEvent::RequestImageWindow { message, .. } => {
|
||||
app.push_status_toast(message, StatusToastLevel::Warning, Some(12_000));
|
||||
}
|
||||
EngineEvent::SnapshotsDisabled { reason, .. } => {
|
||||
// Undo is silently off otherwise: the engine's stderr
|
||||
// notice never reaches the alternate screen (#5930).
|
||||
|
||||
+3
-1
@@ -49,7 +49,9 @@ one-request miss):
|
||||
| Session construction | `initial` |
|
||||
|
||||
History resets that legitimately invalidate the tail (not the header) are
|
||||
logged as `reset:<what>` — `reset:compaction`, `reset:clear`.
|
||||
logged as `reset:<what>` — `reset:compaction`, `reset:clear`, and
|
||||
`reset:image_window` when a provider's per-request image limit moves the
|
||||
request image window (the oldest images go as fixed text notes).
|
||||
|
||||
Anything else that changes the header bytes with **no** declared reason is
|
||||
**drift**: it is logged as `drift:<component>`, the original pin is **kept**
|
||||
|
||||
@@ -46,7 +46,8 @@
|
||||
| 会话构建 | `initial` |
|
||||
|
||||
历史重置会合法地让尾部(而不是头部)失效,记录为 `reset:<what>`——
|
||||
`reset:compaction`、`reset:clear`。
|
||||
`reset:compaction`、`reset:clear`,以及在提供方的单请求图像上限移动请求图像窗口时记录的
|
||||
`reset:image_window`(最早的图像以固定的文本说明代替)。
|
||||
|
||||
除此之外,任何在**没有**声明原因的情况下改动头部字节的行为都算**漂移**:
|
||||
它会记为 `drift:<component>`,原来的固定点被**保留**下来。同一个未声明的前缀
|
||||
|
||||
Reference in new issue
Block a user