anomalyco/opencode
1
0
Fork 0
mirror of https://github.com/anomalyco/opencode.git synced 2026-07-22 02:05:26 -04:00
Code Issues 5.9k Packages Projects Releases 19 Wiki Activity

applyCaching called too freely, causes LiteLLM to fail #4012

New Issue
Open
opened 2026-02-16 17:42:17 -05:00 by yindo · 10 comments
yindo commented 2026-02-16 17:42:17 -05:00
Owner
Copy Link
Copy Source

Originally created by @RingOfStorms on GitHub (Dec 30, 2025).

Originally assigned to: @rekram1-node on GitHub.

Description

Calls to models with the string claude or anthropic in the id fail when they are not actual anthropic upstream providers.

I have this custom provider in my config:

  "provider": {
    "custom": {
      "models": {
        "claude-opus-4.1": {
          "name": "claude-opus-4.1"
        },
      },
      "npm": "@ai-sdk/openai-compatible",
      "options": {
        "apiKey": "na",
        "baseURL": "https://custom.com"
      }
    }
  },

Chatting with this model fails with

litellm.BadRequestError: Vertex_aiException BadRequestError - b'{"type":"error","error":{"type":"invalid_request_error","message":"invalid beta flag"},"request_id":"req_vrtx_011CWdZLV7UDvRZBkvyfXcKJ"}'. Received Model Group=claude-opus-4.1
Available Model Group Fallbacks=None

Because of this logic https://github.com/sst/opencode/blob/3fe5d91372fdf859e09ed5a2aefe359e0648ed10/packages/opencode/src/provider/transform.ts#L198-L208

Even though my model has claude in the name I am explicitly saying it is an openai provider with "npm": "@ai-sdk/openai-compatible". In this case I am using LiteLLM proxy to front all LLM access and it does not support the added "cache_control" key this injects to every message.

Some possible solutions that could be done:

  • config option to disable the applyCaching functionality, either per model or provider
  • (... existing ors) && model.api.npm !== "@ai-sdk/openai-compatible" be added to the if statement logic there. Though I am not sure if that is the best approach or if it should apply to other providers as well.

OpenCode version

1.0.212

Steps to reproduce

  1. Setup LiteLLM in front of Anthropic models where model id includes claude or anthropic
  2. Setup a custom provider in opencode using LiteLLM as a @ai-sdk/openai-compatible provider
  3. call on a model and it fails due to injected cache keys in messages array.

Screenshot and/or share link

Image

Operating System

NixOS/macos

Terminal

Kitty/Iterm2

Originally created by @RingOfStorms on GitHub (Dec 30, 2025). Originally assigned to: @rekram1-node on GitHub. ### Description Calls to models with the string `claude` or `anthropic` in the id fail when they are not actual anthropic upstream providers. I have this custom provider in my config: ```json "provider": { "custom": { "models": { "claude-opus-4.1": { "name": "claude-opus-4.1" }, }, "npm": "@ai-sdk/openai-compatible", "options": { "apiKey": "na", "baseURL": "https://custom.com" } } }, ``` Chatting with this model fails with > litellm.BadRequestError: Vertex_aiException BadRequestError - b'{"type":"error","error":{"type":"invalid_request_error","message":"invalid beta flag"},"request_id":"req_vrtx_011CWdZLV7UDvRZBkvyfXcKJ"}'. Received Model Group=claude-opus-4.1 Available Model Group Fallbacks=None Because of this logic https://github.com/sst/opencode/blob/3fe5d91372fdf859e09ed5a2aefe359e0648ed10/packages/opencode/src/provider/transform.ts#L198-L208 Even though my model has claude in the name I am explicitly saying it is an openai provider with `"npm": "@ai-sdk/openai-compatible"`. In this case I am using LiteLLM proxy to front all LLM access and it does not support the added "cache_control" key this injects to every message. Some possible solutions that could be done: - config option to disable the applyCaching functionality, either per model or provider - `(... existing ors) && model.api.npm !== "@ai-sdk/openai-compatible"` be added to the if statement logic there. Though I am not sure if that is the best approach or if it should apply to other providers as well. ### OpenCode version 1.0.212 ### Steps to reproduce 1. Setup LiteLLM in front of Anthropic models where model id includes claude or anthropic 2. Setup a custom provider in opencode using LiteLLM as a @ai-sdk/openai-compatible provider 3. call on a model and it fails due to injected cache keys in messages array. ### Screenshot and/or share link <img width="1020" height="304" alt="Image" src="https://github.com/user-attachments/assets/4eed9ba2-184f-4bbc-9e2e-5e9b616beb34" /> ### Operating System NixOS/macos ### Terminal Kitty/Iterm2
yindo added the bug label 2026-02-16 17:42:17 -05:00
yindo commented 2026-02-16 17:42:17 -05:00
Author
Owner
Copy Link
Copy Source

@github-actions[bot] commented on GitHub (Dec 30, 2025):

This issue might be a duplicate of existing issues. Please check:

  • #4386: Extra inputs are not permitted, field: 'promptCacheKey' - Similar issue where cache-related fields are being added to OpenAI-compatible providers that don't support them
  • #5416: [FEATURE]: Anthropic (and others) caching improvement - Related discussion about caching behavior across different providers

Feel free to ignore if none of these address your specific case.

@github-actions[bot] commented on GitHub (Dec 30, 2025): This issue might be a duplicate of existing issues. Please check: - #4386: Extra inputs are not permitted, field: 'promptCacheKey' - Similar issue where cache-related fields are being added to OpenAI-compatible providers that don't support them - #5416: [FEATURE]: Anthropic (and others) caching improvement - Related discussion about caching behavior across different providers Feel free to ignore if none of these address your specific case.
yindo commented 2026-02-16 17:42:17 -05:00
Author
Owner
Copy Link
Copy Source

@rekram1-node commented on GitHub (Dec 30, 2025):

I think litellm has a flag to drop unsupported parameters, dmed person that wrote that code originally

@rekram1-node commented on GitHub (Dec 30, 2025): I think litellm has a flag to drop unsupported parameters, dmed person that wrote that code originally
yindo commented 2026-02-16 17:42:17 -05:00
Author
Owner
Copy Link
Copy Source

@RingOfStorms commented on GitHub (Dec 31, 2025):

For anyone else hitting this, this plugin will work (for now at least) since the model.api.id is used primarily before this event hook is called so if we change it during this stage the applyLogic comes after.

This could break something in the future but it works as a patch for me now.

import type { Plugin } from "@opencode-ai/plugin";

export const PatchCachingBug: Plugin = async () => {
  return {
    async "chat.params"(input) {
      const model = input.model;
      // Only touch the specific bad combo: OpenAI-compatible / gateway model
      // whose api.id contains "claude" but is NOT actually Anthropic.
      if (
        model.providerID !== "anthropic" &&
        model.api.npm !== "@ai-sdk/anthropic" &&
        typeof model.api.id === "string" &&
        (model.api.id.includes("claude") || model.api.id.includes("anthropic"))
      ) {
        // Strip the "claude" substring so ProviderTransform.message
        // does not enter the Anthropic caching branch.
        model.api.id = model.api.id.replace(/claude/gi, "cl4ude");
      }
    },
  };
};

I think litellm has a flag to drop unsupported parameters, dmed person that wrote that code originally

I don't have full control over this particular LiteLLM instance myself so I can't easily get this option added yet.

@RingOfStorms commented on GitHub (Dec 31, 2025): For anyone else hitting this, this plugin will work (for now at least) since the model.api.id is used primarily before this event hook is called so if we change it during this stage the applyLogic comes after. This could break something in the future but it works as a patch for me now. ```plugin.ts import type { Plugin } from "@opencode-ai/plugin"; export const PatchCachingBug: Plugin = async () => { return { async "chat.params"(input) { const model = input.model; // Only touch the specific bad combo: OpenAI-compatible / gateway model // whose api.id contains "claude" but is NOT actually Anthropic. if ( model.providerID !== "anthropic" && model.api.npm !== "@ai-sdk/anthropic" && typeof model.api.id === "string" && (model.api.id.includes("claude") || model.api.id.includes("anthropic")) ) { // Strip the "claude" substring so ProviderTransform.message // does not enter the Anthropic caching branch. model.api.id = model.api.id.replace(/claude/gi, "cl4ude"); } }, }; }; ``` > I think litellm has a flag to drop unsupported parameters, dmed person that wrote that code originally I don't have full control over this particular LiteLLM instance myself so I can't easily get this option added yet.
yindo commented 2026-02-16 17:42:17 -05:00
Author
Owner
Copy Link
Copy Source

@RingOfStorms commented on GitHub (Jan 22, 2026):

Well the above plugin no longer works, either the execution order changed or something. Pretty annoying, don't feel like trying to fix it again.

Upstream litellm has not fixed this yet either with drop params. I've resorted to running a full litellm proxy locally just to alias the names of the models that cause issues. Litellm team said they added this to their backlog so just waiting for it to get fixed there.

@RingOfStorms commented on GitHub (Jan 22, 2026): Well the above plugin no longer works, either the execution order changed or something. Pretty annoying, don't feel like trying to fix it again. Upstream litellm has not fixed this yet either with drop params. I've resorted to running a full litellm proxy locally just to alias the names of the models that cause issues. Litellm team said they added this to their backlog so just waiting for it to get fixed there.
yindo commented 2026-02-16 17:42:17 -05:00
Author
Owner
Copy Link
Copy Source

@rekram1-node commented on GitHub (Jan 22, 2026):

Ah, the plugin still runs it's the code that changed internally that makes it not take affect

@rekram1-node commented on GitHub (Jan 22, 2026): Ah, the plugin still runs it's the code that changed internally that makes it not take affect
yindo commented 2026-02-16 17:42:17 -05:00
Author
Owner
Copy Link
Copy Source

@hendem commented on GitHub (Jan 22, 2026):

I'm seeing the same issue: any custom model whose id contains "claude"/"anthropic" triggers applyCaching and injects cache_control into messages, which LiteLLM/OpenAI-compatible proxies reject (400 invalid_request_error / LiteLLM General Error).\n\nThis also makes a plugin workaround impossible now because the caching injection happens inside ProviderTransform.message() before any accessible plugin hooks run (chat.params runs later and experimental.chat.messages.transform can't affect the model id). So a plugin can't prevent applyCaching or strip cache_control early enough.\n\nNet: LiteLLM/OpenAI-compatible usage is broken for these model ids unless you rename/alias the model or patch core.

@hendem commented on GitHub (Jan 22, 2026): I'm seeing the same issue: any custom model whose id contains "claude"/"anthropic" triggers applyCaching and injects cache_control into messages, which LiteLLM/OpenAI-compatible proxies reject (400 invalid_request_error / LiteLLM General Error).\n\nThis also makes a plugin workaround impossible now because the caching injection happens inside ProviderTransform.message() before any accessible plugin hooks run (chat.params runs later and experimental.chat.messages.transform can't affect the model id). So a plugin can't prevent applyCaching or strip cache_control early enough.\n\nNet: LiteLLM/OpenAI-compatible usage is broken for these model ids unless you rename/alias the model or patch core.
yindo commented 2026-02-16 17:42:17 -05:00
Author
Owner
Copy Link
Copy Source

@rekram1-node commented on GitHub (Jan 22, 2026):

Yeah I think we need some better approach internally, note that a plugin can still strip them out, it's kinda weird but you can override the fetch implementation for any provider and it could strip out the things, example fetch being overriden:
https://github.com/anomalyco/opencode/blob/c3f393bcc10ece40c979fee8861ced19844a8108/packages/opencode/src/plugin/codex.ts#L376

Ofc this isnt ideal solution just providing work around

@rekram1-node commented on GitHub (Jan 22, 2026): Yeah I think we need some better approach internally, note that a plugin can still strip them out, it's kinda weird but you can override the fetch implementation for any provider and it could strip out the things, example fetch being overriden: https://github.com/anomalyco/opencode/blob/c3f393bcc10ece40c979fee8861ced19844a8108/packages/opencode/src/plugin/codex.ts#L376 Ofc this isnt ideal solution just providing work around
yindo commented 2026-02-16 17:42:18 -05:00
Author
Owner
Copy Link
Copy Source

@hendem commented on GitHub (Jan 22, 2026):

Thanks for the suggestion — that makes sense. I'll try the fetch-override approach and report back.

@hendem commented on GitHub (Jan 22, 2026): Thanks for the suggestion — that makes sense. I'll try the fetch-override approach and report back.
yindo commented 2026-02-16 17:42:18 -05:00
Author
Owner
Copy Link
Copy Source

@hendem commented on GitHub (Jan 22, 2026):

Hey @rekram1-node, thanks for the fetch-override suggestion — it worked for me.

I had to hook the provider ID that actually owns the OpenAI-compatible/LiteLLM proxy (redacted below), and make sure auth exists for that provider so the loader runs. After that, stripping cache_control before the request goes out fixed the 400s.

Here’s the working plugin (provider name redacted):

import type { Plugin } from "@opencode-ai/plugin";

const STRIP_KEYS = new Set(["cache_control", "cacheControl", "promptCacheKey"]);

function scrub(value: any): void {
  if (!value || typeof value !== "object") return;
  if (Array.isArray(value)) {
    for (const item of value) scrub(item);
    return;
  }

  for (const key of Object.keys(value)) {
    if (STRIP_KEYS.has(key)) {
      delete value[key];
      continue;
    }
    scrub(value[key]);
  }
}

function decodeBody(body: RequestInit["body"]): string | null {
  if (typeof body === "string") return body;
  if (body instanceof Uint8Array) return new TextDecoder().decode(body);
  if (body instanceof ArrayBuffer) return new TextDecoder().decode(new Uint8Array(body));
  return null;
}

export const LiteLLMFetchScrub: Plugin = async () => {
  return {
    auth: {
      provider: "redacted-classified-company",
      async loader(getAuth) {
        const auth = await getAuth();
        if (!auth) return {};

        return {
          async fetch(requestInput: RequestInfo | URL, init?: RequestInit) {
            if (!init?.body) return fetch(requestInput, init);

            const bodyText = decodeBody(init.body);
            if (!bodyText) return fetch(requestInput, init);

            try {
              const payload = JSON.parse(bodyText);
              scrub(payload);
              const nextInit: RequestInit = {
                ...init,
                body: JSON.stringify(payload),
              };
              return fetch(requestInput, nextInit);
            } catch {
              return fetch(requestInput, init);
            }
          },
        };
      },
      methods: [],
    },
  };
};

export default LiteLLMFetchScrub;
@hendem commented on GitHub (Jan 22, 2026): Hey @rekram1-node, thanks for the fetch-override suggestion — it worked for me. I had to hook the provider ID that actually owns the OpenAI-compatible/LiteLLM proxy (redacted below), and make sure auth exists for that provider so the loader runs. After that, stripping cache_control before the request goes out fixed the 400s. Here’s the working plugin (provider name redacted): ```ts import type { Plugin } from "@opencode-ai/plugin"; const STRIP_KEYS = new Set(["cache_control", "cacheControl", "promptCacheKey"]); function scrub(value: any): void { if (!value || typeof value !== "object") return; if (Array.isArray(value)) { for (const item of value) scrub(item); return; } for (const key of Object.keys(value)) { if (STRIP_KEYS.has(key)) { delete value[key]; continue; } scrub(value[key]); } } function decodeBody(body: RequestInit["body"]): string | null { if (typeof body === "string") return body; if (body instanceof Uint8Array) return new TextDecoder().decode(body); if (body instanceof ArrayBuffer) return new TextDecoder().decode(new Uint8Array(body)); return null; } export const LiteLLMFetchScrub: Plugin = async () => { return { auth: { provider: "redacted-classified-company", async loader(getAuth) { const auth = await getAuth(); if (!auth) return {}; return { async fetch(requestInput: RequestInfo | URL, init?: RequestInit) { if (!init?.body) return fetch(requestInput, init); const bodyText = decodeBody(init.body); if (!bodyText) return fetch(requestInput, init); try { const payload = JSON.parse(bodyText); scrub(payload); const nextInit: RequestInit = { ...init, body: JSON.stringify(payload), }; return fetch(requestInput, nextInit); } catch { return fetch(requestInput, init); } }, }; }, methods: [], }, }; }; export default LiteLLMFetchScrub; ```
yindo commented 2026-02-16 17:42:18 -05:00
Author
Owner
Copy Link
Copy Source

@ririnto commented on GitHub (Feb 3, 2026):

use other model id and re-mapping in config.yaml(litellm).

@ririnto commented on GitHub (Feb 3, 2026): use other model id and re-mapping in config.yaml(litellm).
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
acp
bad
bug
discussion
docs
documentation
enhancement
extension
good first issue
growth
help-wanted
mcp
model-problem
needs:compliance
nix
opentui
perf
pull-request
Mirrored from GitHub Pull Request
 tui
web
windows
zen
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: anomalyco/opencode#4012
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?