Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 22:23:50 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #1418] [BUG]: Chat with doc (RAG) is 100x slower than chat with same base model #904

New Issue
Closed
opened 2026-02-22 18:22:07 -05:00 by yindo · 3 comments
yindo commented 2026-02-22 18:22:07 -05:00
Owner
Copy Link
Copy Source

Originally created by @cuihee on GitHub (May 16, 2024).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/1418

How are you running AnythingLLM?

AnythingLLM Docker (local)
LM Studio (local)
Model: QWen chat 1.5 7B q8 gguf
GPU: NVIDIA RTX4000 SFF Ada 20GB

What happened?

When I am in a workspace without documents, it start streaming after one second.
When I am in a workspace with documents, it takes about 80 seconds to start streaming.

My document is not very complex or large.
My document has two files, one doc and one txt

Are there known steps to reproduce?

No response

Originally created by @cuihee on GitHub (May 16, 2024). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/1418 ### How are you running AnythingLLM? AnythingLLM Docker (local) LM Studio (local) Model: QWen chat 1.5 7B q8 gguf GPU: NVIDIA RTX4000 SFF Ada 20GB ### What happened? When I am in a workspace without documents, it start streaming after one second. When I am in a workspace with documents, it takes about 80 seconds to start streaming. My document is not very complex or large. My document has two files, one doc and one txt ### Are there known steps to reproduce? _No response_
yindo added the questionwontfix labels 2026-02-22 18:22:07 -05:00
yindo closed this issue 2026-02-22 18:22:07 -05:00
yindo commented 2026-02-22 18:22:07 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (May 16, 2024):

How many words are in the documents? You would be surprised that even a bit more context in the window dramatically impacts time to first token. This would include the snippets injected into the prompt.

Also, how are you running AnythingLLM? If this is on CPU and you are not on an M1,M2,M3 Mac then this is not unexpected

@timothycarambat commented on GitHub (May 16, 2024): How many words are in the documents? You would be surprised that even a bit more context in the window _dramatically_ impacts time to first token. This would include the snippets injected into the prompt. Also, how are you running AnythingLLM? If this is on CPU and you are not on an M1,M2,M3 Mac then this is not unexpected
yindo commented 2026-02-22 18:22:07 -05:00
Author
Owner
Copy Link
Copy Source

@frost19k commented on GitHub (May 18, 2024):

I'm using the LM Studio backend, and in the AnythingLLM UI agents don't seem to stream. Their response show up as a full text blob. But in the LM Studio logs I can see that text generation has begun. This makes it appear slower than it actually is...

@frost19k commented on GitHub (May 18, 2024): I'm using the LM Studio backend, and in the AnythingLLM UI agents don't seem to stream. Their response show up as a full text blob. But in the LM Studio logs I can see that text generation has begun. This makes it appear slower than it actually is...
yindo commented 2026-02-22 18:22:07 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (May 18, 2024):

@frost19k we dont stream agent responses (because of tool calling) but we will be resolving that soon. The "latency" from this very well may just be the model generating the full response. We do stream the regular responses obviously

@timothycarambat commented on GitHub (May 18, 2024): @frost19k we dont stream agent responses (because of tool calling) but we will be resolving that soon. The "latency" from this very well may just be the model generating the full response. We do stream the regular responses obviously
yindo changed title from [BUG]: Chat with doc (RAG) is 100x slower than chat with same base model to [GH-ISSUE #1418] [BUG]: Chat with doc (RAG) is 100x slower than chat with same base model 2026-06-05 14:37:52 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label question wontfix
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#904
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?