anomalyco/opencode
1
0
Fork 0
mirror of https://github.com/anomalyco/opencode.git synced 2026-07-22 02:05:26 -04:00
Code Issues 5.9k Packages Projects Releases 19 Wiki Activity

Groq Kimi K2 Caching #1446

New Issue
Open
opened 2026-02-16 17:30:59 -05:00 by yindo · 13 comments
yindo commented 2026-02-16 17:30:59 -05:00
Owner
Copy Link
Copy Source

Originally created by @garrett-hopper on GitHub (Aug 22, 2025).

Originally assigned to: @thdxr on GitHub.

See https://github.com/cline/cline/pull/5697

Originally created by @garrett-hopper on GitHub (Aug 22, 2025). Originally assigned to: @thdxr on GitHub. See https://github.com/cline/cline/pull/5697
yindo commented 2026-02-16 17:30:59 -05:00
Author
Owner
Copy Link
Copy Source

@github-actions[bot] commented on GitHub (Aug 22, 2025):

This issue might be a duplicate of existing issues. Please check:

  • #1018: Similar Groq + Kimi K2 combination with function calling failures
  • #222: General Groq model errors with tool execution
  • #1446: Specific Groq Kimi K2 Instruct tool call validation issues

Feel free to ignore if none of these address your specific case.

@github-actions[bot] commented on GitHub (Aug 22, 2025): This issue might be a duplicate of existing issues. Please check: - #1018: Similar Groq + Kimi K2 combination with function calling failures - #222: General Groq model errors with tool execution - #1446: Specific Groq Kimi K2 Instruct tool call validation issues Feel free to ignore if none of these address your specific case.
yindo commented 2026-02-16 17:30:59 -05:00
Author
Owner
Copy Link
Copy Source

@rekram1-node commented on GitHub (Aug 22, 2025):

From groq:

Prompt caching works automatically on all your API requests with no code changes required and no additional fees.

But ig you are saying to track the cached tokens?

@rekram1-node commented on GitHub (Aug 22, 2025): From groq: > Prompt caching works automatically on all your API requests with no code changes required and no additional fees. But ig you are saying to track the cached tokens?
yindo commented 2026-02-16 17:31:00 -05:00
Author
Owner
Copy Link
Copy Source

@garrett-hopper commented on GitHub (Aug 22, 2025):

Ha, I'd not noticed that.
Tracking cached tokens info returned by the API would be ideal of course for accurate pricing estimate.
Lower priority perhaps though, feel free to close.

@garrett-hopper commented on GitHub (Aug 22, 2025): Ha, I'd not noticed that. Tracking cached tokens info returned by the API would be ideal of course for accurate pricing estimate. Lower priority perhaps though, feel free to close.
yindo commented 2026-02-16 17:31:00 -05:00
Author
Owner
Copy Link
Copy Source

@rekram1-node commented on GitHub (Aug 22, 2025):

I will leave open because we may as well track it properly if we aren't

@rekram1-node commented on GitHub (Aug 22, 2025): I will leave open because we may as well track it properly if we aren't
yindo commented 2026-02-16 17:31:00 -05:00
Author
Owner
Copy Link
Copy Source

@djmaze commented on GitHub (Aug 25, 2025):

Does the prompt caching work correctly for others? It seems Groq is still billing me the full cost for the total number of input tokens for each request in a session without considering the cache.

It is unfortunate they don't show this information in their dashboard logs.

@djmaze commented on GitHub (Aug 25, 2025): Does the prompt caching work correctly for others? It seems Groq is still billing me the full cost for the total number of input tokens for each request in a session without considering the cache. It is unfortunate they don't show this information in their dashboard logs.
yindo commented 2026-02-16 17:31:00 -05:00
Author
Owner
Copy Link
Copy Source

@rekram1-node commented on GitHub (Aug 25, 2025):

Groq only has the caching for kimi k2 last I checked, it is rolling out to other models gradually

@djmaze

@rekram1-node commented on GitHub (Aug 25, 2025): Groq only has the caching for kimi k2 last I checked, it is rolling out to other models gradually @djmaze
yindo commented 2026-02-16 17:31:00 -05:00
Author
Owner
Copy Link
Copy Source

@djmaze commented on GitHub (Aug 26, 2025):

Groq only has the caching for kimi k2 last I checked, it is rolling out to other models gradually

@djmaze

Yes, I indeed used Kimi K2 for testing this.

@djmaze commented on GitHub (Aug 26, 2025): > Groq only has the caching for kimi k2 last I checked, it is rolling out to other models gradually > > [@djmaze](https://github.com/djmaze) Yes, I indeed used Kimi K2 for testing this.
yindo commented 2026-02-16 17:31:00 -05:00
Author
Owner
Copy Link
Copy Source

@kajdo commented on GitHub (Nov 7, 2025):

So - this is a bit longer, but maybe helpful

Spoiler for attention the session count from opencode in the python session (see screens below) ended after 7th call with ~23,8k tokens and $0.23 costs BUT the real cost might differ, because of the ~235k tokens which will be invoiced by groq

Update: The cost actually doesn't differ too much in my test - the dashboard shows me $0.16 for today

I recently contacted groq because a session blew up to 10mio tokens (and I'm pretty sure that the session would have been 1-2mio on moonshot directly).

They explained me that there might be a difference because other providers might not charge cached tokens or make it wayyy cheaper.

I still had the impression something fishy is going on ... and since i love the speed of groq - and think kimi is fantastic i invested some time to track down my issue.

TL;DR

  • I think that for some reason either opencode or groq add some "overhead" tokens for each request (between 10k and 30k), which because of the cached token cost of groq sooner or later get a bit expensive.
  • Those 10-30k might not seem a lot, but considering that opencode can work in a multi-step agentic way (f.e. calling 10 times the llm for 1 user input) - those tokens add up quickly (see my intro - 10mio)
  • This overhead seems smaller when calling moonshot.ai (same model)
    • first request for java hello world ~ 10k vs. ~ 20k
    • hello world in python with some text changes - ~10.4k
  • Note the session count from opencode in the python session (see screens below) ended after 7th call with ~23,8k tokens and $0.23 costs BUT the real cost might differ, because of the ~235k tokens which will be invoiced by groq

Here are my notes from my test:

Opencode session starts at ~11k tokens

  • directly after asking for a hello world in python opencode shows 11k
  • i don't understand why - neither the system prompt is that big, nor my request to "write a hello world python script" command
  • for the second request (change the hello world to something else) - the token usage was again ~10k ... which ... i don't understand as well
  • starting from there it kept somewhat reasonably sized

Groq Dashboard shows above input tokens

  • groq counts the 11k from the first call
  • groq counts the 10k from the second call
  • i started with ~8k so from here - i was at +/- 30k tokens
  • those 30k tokens pay dividends

Caching confusion

second call

  • during the second call - groq counts the 11k from the first call as cached
  • plus 10k not cache hit

third call

  • cache when i started 8k plus
  • already first call cache 11k plus
  • prev. non cached 10k becoming addon cache plus
  • 800 new input (no cache hit tokens)

fourth .. n-th call

  • already 30k cached + n-1 call = new cache
  • minimum sized token as input cache
  • Cache after last call in this session: 205,3k
  • Uncached input after last call in this session: 34,3k
  • Total token count: 241,2k

NEW(!) Session

  • started a new session with /new expecting to start with "0" for the session
  • asked for a hello world in java
  • this time the request caused a ~22k token count (see screen of opencode below)
  • groq's uncached input token count reports for today 34,8k (so ~500 tokens new)
  • Cached Input Tokens: 226,8k (so ~21k cached)

new session +1

  • this request changed the Opencode counter from 21,7k to 22,1k ... so ~400 tokens
  • groq's uncached input token count reports: 35,1k (so ~300 new tokens added)
  • groq's cached input token count reports: 248,6k (so ~22k tokens added)

Baseline (I started with some tokens already used)
Image

First python request

Image Image

Second python request

Image Image

3rd python request

Image Image

4th python request

Image Image

5th python request

Image Image

6th python request

Image Image

7th python request

Image Image

Opening new session asking for java

Image Image

2nd java request

Image Image
@kajdo commented on GitHub (Nov 7, 2025): So - this is a bit longer, but maybe helpful **Spoiler for attention** _the session count from opencode in the python session (see screens below) ended after 7th call with ~23,8k tokens and $0.23 costs **BUT** the real cost might differ, because of the ~235k tokens which will be invoiced by groq_ **Update:** The cost actually doesn't differ too much in my test - the dashboard shows me $0.16 for today I recently contacted groq because a session blew up to 10mio tokens (and I'm pretty sure that the session would have been 1-2mio on moonshot directly). They explained me that there might be a difference because other providers might not charge cached tokens or make it wayyy cheaper. I still had the impression something fishy is going on ... and since i love the speed of groq - and think kimi is fantastic i invested some time to track down my issue. ## TL;DR - I think that for some reason either opencode or groq add some "overhead" tokens for each request (between 10k and 30k), which because of the cached token cost of groq sooner or later get a bit expensive. - Those 10-30k might not seem a lot, but considering that opencode can work in a multi-step agentic way (f.e. calling 10 times the llm for 1 user input) - those tokens add up quickly (see my intro - 10mio) - This overhead seems smaller when calling moonshot.ai (same model) - first request for java hello world ~ 10k vs. ~ 20k - hello world in python with some text changes - ~10.4k - **Note** the session count from opencode in the python session (see screens below) ended after 7th call with ~23,8k tokens and $0.23 costs **BUT** the real cost might differ, because of the ~235k tokens which will be invoiced by groq Here are my notes from my test: ## Opencode session starts at ~11k tokens - directly after asking for a hello world in python opencode shows 11k - i don't understand why - neither the system prompt is that big, nor my request to "write a hello world python script" command - i did check anthropic.txt with https://tokencounter.org/ - i did check gemini.txt with https://tokencounter.org/ - i did check copilot-gpt-5.txt with https://tokencounter.org/ - for the second request (change the hello world to something else) - the token usage was again ~10k ... which ... i don't understand as well - starting from there it kept somewhat reasonably sized ## Groq Dashboard shows above input tokens - groq counts the 11k from the first call - groq counts the 10k from the second call - i started with ~8k so from here - i was at +/- 30k tokens - *those 30k tokens pay dividends* ## Caching confusion ### second call - during the second call - groq counts the 11k from the first call as cached - plus 10k not cache hit ### third call - cache when i started 8k plus - already first call cache 11k plus - prev. non cached 10k becoming addon cache plus - **800** new input (no cache hit tokens) ### fourth .. n-th call - already 30k cached + n-1 call = new cache - **minimum** sized token as input cache - Cache after last call in this session: **205,3k** - Uncached input after last call in this session: **34,3k** - Total token count: **241,2k** ## NEW(!) Session - started a new session with `/new` expecting to start with "0" for the session - asked for a hello world in java - this time the request caused a **~22k token count** (see screen of opencode below) - groq's uncached input token count reports for today **34,8k** (so ~500 tokens new) - Cached Input Tokens: **226,8k** (so ~21k cached) ## new session +1 - this request changed the Opencode counter from 21,7k to 22,1k ... so ~400 tokens - groq's uncached input token count reports: 35,1k (so ~300 new tokens added) - groq's cached input token count reports: 248,6k (so ~22k tokens added) Baseline (I started with some tokens already used) <img width="899" height="576" alt="Image" src="https://github.com/user-attachments/assets/612d492e-f98b-4594-b783-f6eabef5cb5f" /> First python request <img width="790" height="861" alt="Image" src="https://github.com/user-attachments/assets/028ba1da-7e0b-425d-ad08-cf59a2ec84e2" /> <img width="742" height="580" alt="Image" src="https://github.com/user-attachments/assets/554b437d-e371-4088-b23b-2ddea5371517" /> Second python request <img width="793" height="867" alt="Image" src="https://github.com/user-attachments/assets/27ea6414-0002-4473-baf6-187bacc89b79" /> <img width="737" height="551" alt="Image" src="https://github.com/user-attachments/assets/54868bdf-96f9-4464-95e3-be32eee99e2d" /> 3rd python request <img width="793" height="863" alt="Image" src="https://github.com/user-attachments/assets/787e694d-a4e2-4768-b6bf-701a173b70fb" /> <img width="747" height="580" alt="Image" src="https://github.com/user-attachments/assets/e56a3bcf-5b1e-4d83-8514-77b7a2094fef" /> 4th python request <img width="789" height="866" alt="Image" src="https://github.com/user-attachments/assets/13a5e2e5-89d8-4a7f-a2c6-16bc6fe5f302" /> <img width="759" height="596" alt="Image" src="https://github.com/user-attachments/assets/68566b14-0fb2-46a2-adbf-95a75f1528bd" /> 5th python request <img width="793" height="862" alt="Image" src="https://github.com/user-attachments/assets/08f082fe-d453-4e70-9645-b3eb57fde50a" /> <img width="759" height="602" alt="Image" src="https://github.com/user-attachments/assets/731940ba-eb76-46e5-8e89-4f442a52036c" /> 6th python request <img width="788" height="862" alt="Image" src="https://github.com/user-attachments/assets/76be03fd-792c-4a1d-a606-d98b91d90a36" /> <img width="760" height="598" alt="Image" src="https://github.com/user-attachments/assets/be3cd65b-4c9b-4a42-a596-e3307cd2c386" /> 7th python request <img width="789" height="865" alt="Image" src="https://github.com/user-attachments/assets/c7366ca8-c824-4365-832e-573e0fd5f07e" /> <img width="757" height="597" alt="Image" src="https://github.com/user-attachments/assets/d4cdb213-f032-4c74-b790-45bcef64acbf" /> Opening new session asking for java <img width="790" height="859" alt="Image" src="https://github.com/user-attachments/assets/5627a3e2-c991-4cda-9528-ae61f6bb1107" /> <img width="750" height="571" alt="Image" src="https://github.com/user-attachments/assets/b536575d-3eab-4261-924b-3992dd78cc4f" /> 2nd java request <img width="790" height="864" alt="Image" src="https://github.com/user-attachments/assets/f71c82b2-e621-42de-add4-1da323b5e7b9" /> <img width="744" height="574" alt="Image" src="https://github.com/user-attachments/assets/3ead218a-d4c3-424c-81b3-db73ad7ae558" />
yindo commented 2026-02-16 17:31:00 -05:00
Author
Owner
Copy Link
Copy Source

@rekram1-node commented on GitHub (Nov 7, 2025):

I thought grok caches automatically? And if so isn't this a problem on their end?

@rekram1-node commented on GitHub (Nov 7, 2025): I thought grok caches automatically? And if so isn't this a problem on their end?
yindo commented 2026-02-16 17:31:00 -05:00
Author
Owner
Copy Link
Copy Source

@kajdo commented on GitHub (Nov 7, 2025):

it does - the big question is "where do those 20k tokens come from with each request" which (yes) gets cached, but still blow up the token count

if i ask "please write hello world in python" - this should not have 22k input token

if i ask "change the string from hello world to hello github" - this should not have 22k input token

it's not a "does groq cache" question - it does and i prove it in the test ... i think the main concern of OP (and me) are why do i still pay more than expected, and my test shows, that with each call there is some overhead constantly transferred (or magically added by groq)

and if my request in openspec triggers 10 sub steps - the overhead is multiplied by 10 - so there are 200k tokens cached (and invoiced with $0.33 per mio) ... spending 2-3 hours on a bigger change might get a bit painful

@kajdo commented on GitHub (Nov 7, 2025): it does - the **big** question is "where do those 20k tokens come from with each request" which (yes) gets cached, but still blow up the token count if i ask "please write hello world in python" - this should not have 22k input token if i ask "change the string from hello world to hello github" - this should not have 22k input token it's not a "does groq cache" question - it does and i prove it in the test ... i think **the main concern** of OP (and me) are why do i still pay more than expected, and my test shows, that with each call there is some overhead constantly transferred (or magically added by groq) and if my request in openspec triggers 10 sub steps - the overhead is multiplied by 10 - so there are 200k tokens cached (and invoiced with $0.33 per mio) ... spending 2-3 hours on a bigger change might get a bit painful
yindo commented 2026-02-16 17:31:00 -05:00
Author
Owner
Copy Link
Copy Source

@rekram1-node commented on GitHub (Nov 7, 2025):

@kajdo well there is the system prompt + tools and their descriptions

in addition to that there are AGENTS.md files or (CLAUDE.md as a fallback)

Also any mcp servers you may have

All those together generally come out to ~11-20k tokens depending on your setup.

The opencode tui sometimes reports higher token counts because some providers return cache read + read and we add them but other providers dont so in some cases the token size is presented as higher than you are actually being billed for

@rekram1-node commented on GitHub (Nov 7, 2025): @kajdo well there is the system prompt + tools and their descriptions in addition to that there are AGENTS.md files or (CLAUDE.md as a fallback) Also any mcp servers you may have All those together generally come out to ~11-20k tokens depending on your setup. The opencode tui sometimes reports higher token counts because some providers return cache read + read and we add them but other providers dont so in some cases the token size is presented as higher than you are actually being billed for
yindo commented 2026-02-16 17:31:00 -05:00
Author
Owner
Copy Link
Copy Source

@kajdo commented on GitHub (Nov 7, 2025):

@kajdo well there is the system prompt + tools and their descriptions

in addition to that there are AGENTS.md files or (CLAUDE.md as a fallback)

Also any mcp servers you may have

All those together generally come out to ~11-20k tokens depending on your setup.

The opencode tui sometimes reports higher token counts because some providers return cache read + read and we add them but other providers dont so in some cases the token size is presented as higher than you are actually being billed for

i just checked the difference between calling with Build mode vs. plan mode (only asking "how would a hello world in python look like")

Build Mode (tools activated but not used): 23.7k
Plan Mode (tools disabled): 11.8k

so you're definitely right, that the tools activation alone makes a big difference

assuming you're right with AGENTS.md as well - I guess that's the answer

not too happy, since it looks like i have to accept to burn more tokens :) ... but at least it makes more sense to me now (drove me crazy tbh) - thank you!

@kajdo commented on GitHub (Nov 7, 2025): > [@kajdo](https://github.com/kajdo) well there is the system prompt + tools and their descriptions > > in addition to that there are AGENTS.md files or (CLAUDE.md as a fallback) > > Also any mcp servers you may have > > All those together generally come out to ~11-20k tokens depending on your setup. > > The opencode tui sometimes reports higher token counts because some providers return cache read + read and we add them but other providers dont so in some cases the token size is presented as higher than you are actually being billed for i just checked the difference between calling with Build mode vs. plan mode (only asking "how would a hello world in python look like") Build Mode (tools activated but not used): 23.7k Plan Mode (tools disabled): 11.8k so you're definitely right, that the tools activation alone makes a big difference assuming you're right with AGENTS.md as well - I guess that's the answer not too happy, since it looks like i have to accept to burn more tokens :) ... but at least it makes more sense to me now (drove me crazy tbh) - thank you!
yindo commented 2026-02-16 17:31:00 -05:00
Author
Owner
Copy Link
Copy Source

@rekram1-node commented on GitHub (Nov 7, 2025):

Yeah ofc!

@rekram1-node commented on GitHub (Nov 7, 2025): Yeah ofc!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
acp
bad
bug
discussion
docs
documentation
enhancement
extension
good first issue
growth
help-wanted
mcp
model-problem
needs:compliance
nix
opentui
perf
pull-request
Mirrored from GitHub Pull Request
 tui
web
windows
zen
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: anomalyco/opencode#1446
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?