anomalyco/opencode
1
0
Fork 0
mirror of https://github.com/anomalyco/opencode.git synced 2026-07-22 02:05:26 -04:00
Code Issues 5.9k Packages Projects Releases 19 Wiki Activity

Anthropic caching not really working via OpenRouter #877

New Issue
Closed
opened 2026-02-16 17:28:39 -05:00 by yindo · 5 comments
yindo commented 2026-02-16 17:28:39 -05:00
Owner
Copy Link
Copy Source

Originally created by @neominik on GitHub (Jul 23, 2025).

Originally assigned to: @thdxr on GitHub.

When using Anthropic models via OpenRouter, the caching doesn't work as well as when using Anthropic directly as a provider.
With the current config, OpenRouter seems to cache the system message and maybe the first message, but then never update what is cached. So as the conversation gets longer, OpenRouter becomes an order of magnitude more expensive than Anthropic. In the OpenRouter activity logs, I can see that for every message some caching is applied, but the cost savings are constant, not increasing with more messages.

Since Anthropic is often overloaded and usually the slowest provider, OpenRouter would be a great alternative. However, without proper caching, this is not viable.

There are some docs on OpenRouter that describe that some extra care needs to be taken for Anthropic models: https://openrouter.ai/docs/features/prompt-caching#anthropic-claude

Originally created by @neominik on GitHub (Jul 23, 2025). Originally assigned to: @thdxr on GitHub. When using Anthropic models via OpenRouter, the caching doesn't work as well as when using Anthropic directly as a provider. With the current config, OpenRouter seems to cache the system message and maybe the first message, but then never update what is cached. So as the conversation gets longer, OpenRouter becomes an order of magnitude more expensive than Anthropic. In the OpenRouter activity logs, I can see that for every message some caching is applied, but the cost savings are constant, not increasing with more messages. Since Anthropic is often overloaded and usually the slowest provider, OpenRouter would be a great alternative. However, without proper caching, this is not viable. There are some docs on OpenRouter that describe that some extra care needs to be taken for Anthropic models: https://openrouter.ai/docs/features/prompt-caching#anthropic-claude
yindo closed this issue 2026-02-16 17:28:39 -05:00
yindo commented 2026-02-16 17:28:39 -05:00
Author
Owner
Copy Link
Copy Source

@thdxr commented on GitHub (Jul 23, 2025):

in my testing i can see it does seem to work

3 messages in convo - you can see first is very expensive and the following 2 are cheaper and if you inspect it provides cache info (ignore haiku req, it's for title gen)

i would need some help debugging this further if you can reproduce different behavior

image.png image.png
@thdxr commented on GitHub (Jul 23, 2025): in my testing i can see it does seem to work<br><br>3 messages in convo - you can see first is very expensive and the following 2 are cheaper and if you inspect it provides cache info (ignore haiku req, it's for title gen)<br><br>i would need some help debugging this further if you can reproduce different behavior <img src="https://uploads.linear.app/a959060b-9262-4398-9650-16bf10f1450e/7586ac7b-0195-4369-9e6e-ec2f5ac93b5a/3dfce809-9f6a-49b8-b07e-6458e5c13c62?signature=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJwYXRoIjoiL2E5NTkwNjBiLTkyNjItNDM5OC05NjUwLTE2YmYxMGYxNDUwZS83NTg2YWM3Yi0wMTk1LTQzNjktOWU2ZS1lYzJmNWFjOTNiNWEvM2RmY2U4MDktOWY2YS00OWI4LWIwN2UtNjQ1OGU1YzEzYzYyIiwiaWF0IjoxNzUzMjg2ODQ3LCJleHAiOjMzMzIzODQ2ODQ3fQ.Y8fV_nxGgZI_5R6kmqxdZlvRRvYoXXGqP95QBAa86KU " alt="image.png" width="2691" data-linear-height="1308" /> <img src="https://uploads.linear.app/a959060b-9262-4398-9650-16bf10f1450e/81dc0b31-2f8e-43eb-acad-87c6c323d8dc/e977194b-3aa2-4c91-9415-7c30c5561c3b?signature=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJwYXRoIjoiL2E5NTkwNjBiLTkyNjItNDM5OC05NjUwLTE2YmYxMGYxNDUwZS84MWRjMGIzMS0yZjhlLTQzZWItYWNhZC04N2M2YzMyM2Q4ZGMvZTk3NzE5NGItM2FhMi00YzkxLTk0MTUtN2MzMGM1NTYxYzNiIiwiaWF0IjoxNzUzMjg2ODQ3LCJleHAiOjMzMzIzODQ2ODQ3fQ.NyYJ18SWtJTrSzFKRprX2jNDWpL_h9FnonsfAP9CJ4I " alt="image.png" width="3432" data-linear-height="828" />
yindo commented 2026-02-16 17:28:39 -05:00
Author
Owner
Copy Link
Copy Source

@neominik commented on GitHub (Jul 23, 2025):

It does seem to cache the first message just fine. However, if you click on the > arrow for more info, you should see that both of the last messages (and all future messages) have the exact same "Cache read cost" discount applied (approx. $ -0.04 depending on the length of AGENTS.md), so the cache is never updated. If the agent starts reading files, those input tokens won't be cached, drastically increasing the cost of future messages.

@neominik commented on GitHub (Jul 23, 2025): It does seem to cache the first message just fine. However, if you click on the > arrow for more info, you should see that both of the last messages (and all future messages) have the exact same "Cache read cost" discount applied (approx. $ -0.04 depending on the length of AGENTS.md), so the cache is never updated. If the agent starts reading files, those input tokens won't be cached, drastically increasing the cost of future messages.
yindo commented 2026-02-16 17:28:39 -05:00
Author
Owner
Copy Link
Copy Source

@thdxr commented on GitHub (Jul 23, 2025):

gotcha - i read their docs and tbh their own understanding of how anthropic caching works seems wrong (not surprising i myself was very confused for a while)

the way we have it implemented in opencode is identical to how it's implemented in claude code (unless we have a bug) - basically the cache breakpoints are set as the 2 system prompts and always the last 2 messages in the user prompt

so it basically moves down as the conversation continues - anthropic caches any message that was previous cached so in theory the whole convo gets cached unless something gets busted

that logic is here https://github.com/sst/opencode/blob/dev/packages/opencode/src/provider/transform.ts#L5

@thdxr commented on GitHub (Jul 23, 2025): gotcha - i read their docs and tbh their own understanding of how anthropic caching works seems wrong (not surprising i myself was very confused for a while) the way we have it implemented in opencode is identical to how it's implemented in claude code (unless we have a bug) - basically the cache breakpoints are set as the 2 system prompts and always the last 2 messages in the user prompt so it basically moves down as the conversation continues - anthropic caches any message that was previous cached so in theory the whole convo gets cached unless something gets busted that logic is here https://github.com/sst/opencode/blob/dev/packages/opencode/src/provider/transform.ts#L5
yindo commented 2026-02-16 17:28:39 -05:00
Author
Owner
Copy Link
Copy Source

@neominik commented on GitHub (Jul 24, 2025):

I'm wondering how these provider options are applied by the ai sdk and how that would lead to different requests. I assume that the anthropic provider uses the anthropic package which actually implements putting the cache_control option into the last message part while openrouter might use the generic openai compatible package which does not?

@neominik commented on GitHub (Jul 24, 2025): I'm wondering how these provider options are applied by the ai sdk and how that would lead to different requests. I assume that the anthropic provider uses the anthropic package which [actually implements](https://github.com/vercel/ai/blob/5a25d576122a7a5cc6d62e55e428bf4ad9a6caa9/packages/anthropic/src/convert-to-anthropic-messages-prompt.ts#L114) putting the `cache_control` option into the last message part while openrouter might use the generic openai compatible package which does not?
yindo commented 2026-02-16 17:28:39 -05:00
Author
Owner
Copy Link
Copy Source

@neominik commented on GitHub (Jul 25, 2025):

The issue is that the cache control needs to be applied to the last content part instead of to the message itself for user messages. The anthropic SDK handles this, but the openaiCompatible doesn't (the @openrouter/ai-sdk-provider has handling, but doesn't do it correctly either, adding cache breakpoints to each part which results in requests failing with > 4 cache breakpoints).
I created https://github.com/sst/opencode/pull/1305 with a workaround. Tested locally and it does correctly move the cache breakpoint forward, same as for the native anthropic provider.

@neominik commented on GitHub (Jul 25, 2025): The issue is that the cache control needs to be applied to the last content part instead of to the message itself for user messages. The anthropic SDK handles this, but the openaiCompatible doesn't (the @openrouter/ai-sdk-provider has handling, but doesn't do it correctly either, adding cache breakpoints to each part which results in requests failing with > 4 cache breakpoints). I created https://github.com/sst/opencode/pull/1305 with a workaround. Tested locally and it does correctly move the cache breakpoint forward, same as for the native anthropic provider.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
acp
bad
bug
discussion
docs
documentation
enhancement
extension
good first issue
growth
help-wanted
mcp
model-problem
needs:compliance
nix
opentui
perf
pull-request
Mirrored from GitHub Pull Request
 tui
web
windows
zen
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: anomalyco/opencode#877
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?