Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #973] [FEAT]: Implement LLMLingua-2 #597

New Issue
Closed
opened 2026-02-22 18:20:23 -05:00 by yindo · 1 comment
yindo commented 2026-02-22 18:20:23 -05:00
Owner
Copy Link
Copy Source

Originally created by @Q-efx on GitHub (Mar 26, 2024).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/973

What would you like to see?

Implement: https://github.com/microsoft/LLMLingua to reduce cost and increase speed, especially for pinned documents.

Originally created by @Q-efx on GitHub (Mar 26, 2024). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/973 ### What would you like to see? Implement: https://github.com/microsoft/LLMLingua to reduce cost and increase speed, especially for pinned documents.
yindo added the enhancementfeature request labels 2026-02-22 18:20:23 -05:00
yindo closed this issue 2026-02-22 18:20:23 -05:00
yindo commented 2026-02-22 18:20:24 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Mar 26, 2024):

While I can see a use case for this in specific circumstances, having an LLM pre-summarize the full-text pinned document defeats the purpose of pinning. It also adds some overhead and technical issues that just don't fit for AnythingLLM and its intended use-case.

Adding an additional 25s+ latency for what essentially comes down to a black-box summarization of text pre-input to the actual LLM that can stream is going to be unacceptable for most. Given the only way this would save tokens is if we routed it through a local model - otherwise, we are using a cloud-based LLM to summarize ( $ ) and then send that hopefully ok compressed response back to the same model ( $ + $ ).

Intuitively, this is really just what a vector database does, but likely more accurate at the expense of both speed & cost (if on cloud). Since all this process does is use an LLM to "trim the fat" from context, that is exactly what vector databases do, but faster (albeit at the price of accuracy sometimes). The only overt benefit is the KV-Cache not being large when using this method, but this benefit is insignificant in performance given the above tradeoff.

Lastly, this is in Python, which given the tradeoffs, is not worth the lift to transform into NodeJS code since we could probably get 100x more returns from just implementing knowledge-graphs, which can be built async and are non-blocking.
This is being done on another branch right now and has great returns for very little costs.

There are about 1,000+ ways to improve RAG, only very few of which can be applied to a general use case, and in the interest of keeping AnythingLLM useable and understandable to a person who is not deep into this space, but requires this kind of experience - we can only add so many "RAG improvements".

I think this does open a conversation for a "plugin" style arch for allowing this kind of custom implementation of tooling without anyone having to code it except the user who explicitly requires it.

@timothycarambat commented on GitHub (Mar 26, 2024): While I can see a use case for this in specific circumstances, having an LLM pre-summarize the full-text pinned document defeats the purpose of pinning. It also adds some overhead and technical issues that just don't fit for AnythingLLM and its intended use-case. Adding an additional 25s+ latency for what essentially comes down to a black-box summarization of text pre-input to the actual LLM that can stream is going to be unacceptable for most. Given the only way this would save tokens is if we routed it through a local model - otherwise, we are using a cloud-based LLM to summarize ( \$ ) and then send that hopefully ok compressed response back to the same model ( \$ + \$ ). Intuitively, this is really just what a vector database does, but likely more accurate at the expense of both speed & cost (if on cloud). Since all this process does is use an LLM to "trim the fat" from context, that is exactly what vector databases do, but faster (albeit at the price of accuracy sometimes). The only overt benefit is the KV-Cache not being large when using this method, but this benefit is insignificant in performance given the above tradeoff. Lastly, this is in Python, which given the tradeoffs, is not worth the lift to transform into NodeJS code since we could probably get 100x more returns from just implementing knowledge-graphs, which can be built async and are non-blocking. This is being done on another branch right now and has great returns for very little costs. There are about 1,000+ ways to improve RAG, only very few of which can be applied to a general use case, and in the interest of keeping AnythingLLM useable and understandable to a person who is not deep into this space, but requires this kind of experience - we can only add so many "RAG improvements". I think this does open a conversation for a "plugin" style arch for allowing this kind of custom implementation of tooling without anyone having to code it except the user who explicitly requires it.
yindo changed title from [FEAT]: Implement LLMLingua-2 to [GH-ISSUE #973] [FEAT]: Implement LLMLingua-2 2026-06-05 14:36:12 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label enhancement feature request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#597
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?