Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #2966] Gemini context window lookup #1892

New Issue
Closed
opened 2026-02-22 18:27:06 -05:00 by yindo · 3 comments
yindo commented 2026-02-22 18:27:06 -05:00
Owner
Copy Link
Copy Source

Originally created by @DangerousBerries on GitHub (Jan 12, 2025).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/2966

Originally assigned to: @timothycarambat on GitHub.

How are you running AnythingLLM?

Docker (local)

What happened?

When using some Gemini models (for example gemini-exp-1206, gemini-2.0-flash-thinking-exp, and learnlm-1.5-pro-experimental) with a high "Max Context Snippets" setting, the model appears to truncate or ignore recent chat history. The conversation continues as if new messages are being attached to an older part of the conversation, rather than maintaining the full, recent context.

This issue persists even when the "Document similarity threshold" is set to High, and occurs despite no citations being shown in the UI (no "Show Citations" button visible). The only workaround is reducing the Max Context Snippets value.

gemini-2.0-flash-thinking-exp has a 33k token limit so that may be where the problem starts to appear, gemini-exp-1206 is supposed to have a 2 million token limit and works correctly in Google AI Studio, so that's strange.

Are there known steps to reproduce?

  1. Choose gemini-exp-1206, learnlm-1.5-pro-experimental, or gemini-2.0-flash-thinking-exp as the model
  2. Use a high number for Max Context Snippets in the settings (like 200)
  3. Start a conversation with a lot of back-and-forth messages and plenty of documents
  4. See that the model's responses begin to ignore recent chat history, instead continuing as if responding to older messages
  5. You can verify this by:
    • Reducing Max Context Snippets and see that the problem resolves
    • Setting Document similarity threshold to High so no citations are being shown in the UI
Originally created by @DangerousBerries on GitHub (Jan 12, 2025). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/2966 Originally assigned to: @timothycarambat on GitHub. ### How are you running AnythingLLM? Docker (local) ### What happened? When using some Gemini models (for example gemini-exp-1206, gemini-2.0-flash-thinking-exp, and learnlm-1.5-pro-experimental) with a high "Max Context Snippets" setting, the model appears to truncate or ignore recent chat history. The conversation continues as if new messages are being attached to an older part of the conversation, rather than maintaining the full, recent context. This issue persists even when the "Document similarity threshold" is set to High, and occurs despite no citations being shown in the UI (no "Show Citations" button visible). The only workaround is reducing the Max Context Snippets value. gemini-2.0-flash-thinking-exp has a 33k token limit so that may be where the problem starts to appear, gemini-exp-1206 is supposed to have a 2 million token limit and works correctly in Google AI Studio, so that's strange. ### Are there known steps to reproduce? 1. Choose gemini-exp-1206, learnlm-1.5-pro-experimental, or gemini-2.0-flash-thinking-exp as the model 2. Use a high number for Max Context Snippets in the settings (like 200) 3. Start a conversation with a lot of back-and-forth messages and plenty of documents 4. See that the model's responses begin to ignore recent chat history, instead continuing as if responding to older messages 5. You can verify this by: - Reducing Max Context Snippets and see that the problem resolves - Setting Document similarity threshold to High so no citations are being shown in the UI
yindo added the enhancement label 2026-02-22 18:27:06 -05:00
yindo closed this issue 2026-02-22 18:27:06 -05:00
yindo commented 2026-02-22 18:27:06 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Jan 12, 2025):

Are you using document pinning and/or have more than 1M of context? Message history count is mearly a factor of how many tokens are in the window. If that are more tokens in the window than the model permits - we truncate it.

To see if this is happening, can you repro the loss of history from the memory and pull the backend logs? At the end of the day, the model can only process so many tokens and if you are sending a query beyond that, we have to truncate or else the inference will fail.
https://docs.anythingllm.com/installation-desktop/debug

Outside of that possibility, how are you confirming that a message is sent or pruned from the context?

@timothycarambat commented on GitHub (Jan 12, 2025): Are you using document pinning and/or have more than 1M of context? Message history count is mearly a factor of how many tokens are in the window. If that are more tokens in the window than the model permits - we truncate it. To see if this is happening, can you repro the loss of history from the memory and pull the `backend` logs? At the end of the day, the model can only process so many tokens and if you are sending a query beyond that, we have to truncate or else the inference will fail. https://docs.anythingllm.com/installation-desktop/debug Outside of that possibility, how are you confirming that a message is sent or pruned from the context?
yindo commented 2026-02-22 18:27:06 -05:00
Author
Owner
Copy Link
Copy Source

@DangerousBerries commented on GitHub (Jan 13, 2025):

Are you using document pinning and/or have more than 1M of context? Message history count is mearly a factor of how many tokens are in the window. If that are more tokens in the window than the model permits - we truncate it.

To see if this is happening, can you repro the loss of history from the memory and pull the backend logs? At the end of the day, the model can only process so many tokens and if you are sending a query beyond that, we have to truncate or else the inference will fail. https://docs.anythingllm.com/installation-desktop/debug

Outside of that possibility, how are you confirming that a message is sent or pruned from the context?

This still happens with no documents pinned. I'm definitely not using more than 1M of context because gemini-2.0-flash-exp doesn't truncate even with the max number of Chat History, Max Context Snippets, and no restriction for the document similarity threshold. gemini-exp-1206 also has a 2M token limit so not even 1M should be a problem.

Here is the log:
2025-01-13 11:33:14 [backend] info: [GeminiLLM] Initialized with model: gemini-exp-1206
2025-01-13 11:33:14 [backend] info: [OllamaEmbedder] Embedding 1 chunks of text with bge-m3:latest.
2025-01-13 11:33:14 [backend] info: [NativeEmbeddingReranker] Initialized
2025-01-13 11:33:14 [backend] info: [NativeEmbeddingReranker] Reranker suite already initialized - reusing.
2025-01-13 11:33:14 [backend] info: [NativeEmbeddingReranker] Reranking 50 documents...
2025-01-13 11:33:16 [backend] info: [NativeEmbeddingReranker] Reranking 50 documents to top 200 took 1576ms
2025-01-13 11:33:16 [backend] info: [fillSourceWindow] Need to backfill 150 chunks to fill in the source window for RAG!
2025-01-13 11:33:16 [backend] info: [fillSourceWindow] Citations backfilled to 200 references from 50 original citations.
2025-01-13 11:33:17 [backend] info: Cannonball results 3109 -> 1238 tokens.
2025-01-13 11:33:17 [backend] info: Cannonball results 82500 -> 3695 tokens.

I see that it backfills references in. In one of my earlier issues I asked about that and you said it's supposed to show in the output chat, but the output chat's Show Citations button only shows 50 references from four documents, not 200.

If gemini-exp-1206 has a 2M token limit why does it turn 82500 to 3695 tokens?

@DangerousBerries commented on GitHub (Jan 13, 2025): > Are you using document pinning and/or have more than 1M of context? Message history count is mearly a factor of how many tokens are in the window. If that are more tokens in the window than the model permits - we truncate it. > > To see if this is happening, can you repro the loss of history from the memory and pull the `backend` logs? At the end of the day, the model can only process so many tokens and if you are sending a query beyond that, we have to truncate or else the inference will fail. https://docs.anythingllm.com/installation-desktop/debug > > Outside of that possibility, how are you confirming that a message is sent or pruned from the context? This still happens with no documents pinned. I'm definitely not using more than 1M of context because gemini-2.0-flash-exp doesn't truncate even with the max number of Chat History, Max Context Snippets, and no restriction for the document similarity threshold. gemini-exp-1206 also has a 2M token limit so not even 1M should be a problem. Here is the log: 2025-01-13 11:33:14 [backend] info: [GeminiLLM] Initialized with model: gemini-exp-1206 2025-01-13 11:33:14 [backend] info: [OllamaEmbedder] Embedding 1 chunks of text with bge-m3:latest. 2025-01-13 11:33:14 [backend] info: [NativeEmbeddingReranker] Initialized 2025-01-13 11:33:14 [backend] info: [NativeEmbeddingReranker] Reranker suite already initialized - reusing. 2025-01-13 11:33:14 [backend] info: [NativeEmbeddingReranker] Reranking 50 documents... 2025-01-13 11:33:16 [backend] info: [NativeEmbeddingReranker] Reranking 50 documents to top 200 took 1576ms 2025-01-13 11:33:16 [backend] info: [fillSourceWindow] Need to backfill 150 chunks to fill in the source window for RAG! 2025-01-13 11:33:16 [backend] info: [fillSourceWindow] Citations backfilled to 200 references from 50 original citations. 2025-01-13 11:33:17 [backend] info: Cannonball results 3109 -> 1238 tokens. 2025-01-13 11:33:17 [backend] info: Cannonball results 82500 -> 3695 tokens. I see that it backfills references in. In one of my earlier issues I asked about that and you said it's supposed to show in the output chat, but the output chat's Show Citations button only shows 50 references from four documents, not 200. If gemini-exp-1206 has a 2M token limit why does it turn 82500 to 3695 tokens?
yindo commented 2026-02-22 18:27:06 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Jan 13, 2025):

This is the primary issue then

2025-01-13 11:33:17 [backend] info: Cannonball results 3109 -> 1238 tokens.
2025-01-13 11:33:17 [backend] info: Cannonball results 82500 -> 3695 tokens.

So yeah, we are truncating - this has to be because of this:
https://github.com/Mintplex-Labs/anything-llm/blob/865f7eea296e544b2eb1ab8c1f322208eaf5eb05/server/utils/AiProviders/gemini/index.js#L109-L111

Because we dont have every single model key in Gemini it is probably assuming a context of 30,720 - so that is why since there is also a partition limit for each section of a chat like this

 this.limits = {
      history: this.promptWindowLimit() * 0.15,
      system: this.promptWindowLimit() * 0.15,
      user: this.promptWindowLimit() * 0.7,
    };

15% of 30,720 = 4608, which given the buffer factor would be close to that 3695 value.

The solution is to probably:

  • Pull and cache Gemini models so we can hash table their context windows if they are in the response (not sure if gemini /models reports the max content. Otherwise, we need to just update the keys in the MODEL_MAP
@timothycarambat commented on GitHub (Jan 13, 2025): This is the primary issue then ``` 2025-01-13 11:33:17 [backend] info: Cannonball results 3109 -> 1238 tokens. 2025-01-13 11:33:17 [backend] info: Cannonball results 82500 -> 3695 tokens. ``` So yeah, we are truncating - this has to be because of this: https://github.com/Mintplex-Labs/anything-llm/blob/865f7eea296e544b2eb1ab8c1f322208eaf5eb05/server/utils/AiProviders/gemini/index.js#L109-L111 Because we dont have every single model key in Gemini it is probably assuming a context of 30,720 - so that is why since there is also a partition limit for each section of a chat like this ``` this.limits = { history: this.promptWindowLimit() * 0.15, system: this.promptWindowLimit() * 0.15, user: this.promptWindowLimit() * 0.7, }; ``` 15% of 30,720 = 4608, which given the buffer factor would be close to that 3695 value. The solution is to probably: - Pull and cache Gemini models so we can hash table their context windows if they are in the response (not sure if gemini `/models` reports the max content. Otherwise, we need to just update the keys in the `MODEL_MAP`
yindo changed title from Gemini context window lookup to [GH-ISSUE #2966] Gemini context window lookup 2026-06-05 14:43:18 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label enhancement
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#1892
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?