Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #3384] [DOCS]: messages of history discarded by messageArrayCompressor #2179

New Issue
Closed
opened 2026-02-22 18:28:30 -05:00 by yindo · 2 comments
yindo commented 2026-02-22 18:28:30 -05:00
Owner
Copy Link
Copy Source

Originally created by @jbymy2000 on GitHub (Mar 3, 2025).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/3384

Description

Issue Description

AnythingLLM has a strategy to compress history content based on the model's max_token size output. Specifically, it discards portions of the system_prompt, ai_prompt, and user_prompt at certain ratios according to the overall prompt limit.
This approach doesn't significantly impact models with larger context windows. However, when using LLMs with smaller context windows, important information gets discarded. The external manifestation is that responses in AnythingLLM lose considerable context information.
This problem is particularly pronounced when hardcoding certain models. Many models default to a 4k length, which directly leads to the issue described above.

Example

For example, generic OpenAI models have a limit setting:

javascriptCopythis.limits = {
  history: this.promptWindowLimit() * 0.15,
  system: this.promptWindowLimit() * 0.15,
  user: this.promptWindowLimit() * 0.7,
};

if the length break the limit. mesage will be discarded.

Suggestion

I believe this behavior should be explained in the documentation to make users aware of this limitation.

Originally created by @jbymy2000 on GitHub (Mar 3, 2025). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/3384 ### Description # Issue Description AnythingLLM has a strategy to compress history content based on the model's max_token size output. Specifically, it discards portions of the system_prompt, ai_prompt, and user_prompt at certain ratios according to the overall prompt limit. This approach doesn't significantly impact models with larger context windows. However, when using LLMs with smaller context windows, important information gets discarded. The external manifestation is that responses in AnythingLLM lose considerable context information. This problem is particularly pronounced when hardcoding certain models. Many models default to a 4k length, which directly leads to the issue described above. # Example For example, generic OpenAI models have a limit setting: ``` javascriptCopythis.limits = { history: this.promptWindowLimit() * 0.15, system: this.promptWindowLimit() * 0.15, user: this.promptWindowLimit() * 0.7, }; ``` if the length break the limit. mesage will be discarded. # Suggestion I believe this behavior should be explained in the documentation to make users aware of this limitation.
yindo added the documentation label 2026-02-22 18:28:30 -05:00
yindo closed this issue 2026-02-22 18:28:30 -05:00
yindo commented 2026-02-22 18:28:30 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Mar 3, 2025):

There is nothing to really "fix" here though. A context window is the context window. If we didn't do any message management the response would outright crash. So our choice is either crash the LLM/throw an error - or manage the content by some reasonable management process and lose information.

The generic connector also does allow changing the token limit so that you can set it to the true context limit of the model. This will then apply that math to your true context windows instead of some base value.
https://github.com/Mintplex-Labs/anything-llm/blob/fab74037fae86e5c11b3ad0caecdb66a0dff691e/server/utils/AiProviders/genericOpenAi/index.js#L27-L29

@timothycarambat commented on GitHub (Mar 3, 2025): There is nothing to really "fix" here though. A context window is the context window. If we didn't do any message management the response would outright crash. So our choice is either crash the LLM/throw an error - or manage the content by some reasonable management process and lose information. The generic connector also does allow changing the token limit so that you can set it to the true context limit of the model. This will then apply that math to your true context windows instead of some base value. https://github.com/Mintplex-Labs/anything-llm/blob/fab74037fae86e5c11b3ad0caecdb66a0dff691e/server/utils/AiProviders/genericOpenAi/index.js#L27-L29
yindo commented 2026-02-22 18:28:30 -05:00
Author
Owner
Copy Link
Copy Source

@jbymy2000 commented on GitHub (Mar 3, 2025):

I setup the anythingllm in my company, It was great tool but some users found their dialogs was 'forgotten' sometimes. So i digged into it and found the exsiting comprasssing strategy. I note it here so other people has same doubt would notice.

@jbymy2000 commented on GitHub (Mar 3, 2025): I setup the anythingllm in my company, It was great tool but some users found their dialogs was 'forgotten' sometimes. So i digged into it and found the exsiting comprasssing strategy. I note it here so other people has same doubt would notice.
yindo changed title from [DOCS]: messages of history discarded by messageArrayCompressor to [GH-ISSUE #3384] [DOCS]: messages of history discarded by messageArrayCompressor 2026-06-05 14:45:02 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label documentation
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#2179
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?