Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #2231] [BUG]: time delay for the first word show up in the chat window with RAG return #1454

New Issue
Closed
opened 2026-02-22 18:24:53 -05:00 by yindo · 3 comments
yindo commented 2026-02-22 18:24:53 -05:00
Owner
Copy Link
Copy Source

Originally created by @justinzh on GitHub (Sep 6, 2024).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/2231

How are you running AnythingLLM?

Docker (local)

What happened?

I am not sure it is a bug. but when I test my system, its first word response time can be upto 12 seconds

I do notice that the first response time goes up to 12 seconds if the vector db returns context. or else it can drop to about 3 seconds.

here is my setup:
ollama hosted within docker container
Qdran hosted within docker container
anything llm hosted within docker container
host machine: ubuntu / 128 cpu / 2 x L40s

Are there known steps to reproduce?

No response

Originally created by @justinzh on GitHub (Sep 6, 2024). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/2231 ### How are you running AnythingLLM? Docker (local) ### What happened? I am not sure it is a bug. but when I test my system, its first word response time can be upto 12 seconds I do notice that the first response time goes up to 12 seconds if the vector db returns context. or else it can drop to about 3 seconds. here is my setup: ollama hosted within docker container Qdran hosted within docker container anything llm hosted within docker container host machine: ubuntu / 128 cpu / 2 x L40s ### Are there known steps to reproduce? _No response_
yindo added the possible bug label 2026-02-22 18:24:53 -05:00
yindo closed this issue 2026-02-22 18:24:53 -05:00
yindo commented 2026-02-22 18:24:53 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Sep 6, 2024):

More context => more time to the first token. However, you have 2 L40's so inference time should be very fast. Are you running ollama in the docker container with the --gpus all flag? It sounds like you are still using CPU and unless the context is really massive this sounds like the LLM is not using GPU as the latency to do a semantic search should be very fast.

Are you using Ollama for embedding as well? It could be Ollama needing to load the embedder to do the search, then unloading that to load in the LLM for inferencing. Both of which would introduce latency. If using the default embedding this runs on CPU so should be quite fast on this machine.

Something for sure on the config side, not a code related bug at this time, but we can keep chatting on this issue and ill get notifs for debugging further

@timothycarambat commented on GitHub (Sep 6, 2024): More context => more time to the first token. However, you have 2 L40's so inference time should be very fast. Are you running ollama in the docker container with the `--gpus all` flag? It sounds like you are still using CPU and unless the context is _really massive_ this sounds like the LLM is not using GPU as the latency to do a semantic search should be very fast. Are you using Ollama for embedding as well? It could be Ollama needing to load the embedder to do the search, then unloading that to load in the LLM for inferencing. Both of which would introduce latency. If using the default embedding this runs on CPU so should be quite fast on this machine. Something for sure on the config side, not a code related bug at this time, but we can keep chatting on this issue and ill get notifs for debugging further
yindo commented 2026-02-22 18:24:53 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Sep 6, 2024):

You can additionally tail the logs, it would should the steps being taken and where the longest step is occurring by seeing where the latency is introduced

@timothycarambat commented on GitHub (Sep 6, 2024): You can additionally tail the logs, it would should the steps being taken and where the longest step is occurring by seeing where the latency is introduced
yindo commented 2026-02-22 18:24:53 -05:00
Author
Owner
Copy Link
Copy Source

@Justin-KM commented on GitHub (Sep 9, 2024):

thanks for the reply. I am using ollama to host llm. qdrant for vector database. ollama is not the issue. I test it with ollama run, it responses almost instantly.

both ollama, and qdrant are dockered and i user docker logs -f to monitor their response.

the observation is that if qdrant has context returns, the response time will be long 912s. if no context returns (no search hit) the response time is around 35s.

I use docker logs -f to check anythingllm's logs. I don't see much useful information. do we have a debug switch for more detailed logs?

@Justin-KM commented on GitHub (Sep 9, 2024): thanks for the reply. I am using ollama to host llm. qdrant for vector database. ollama is not the issue. I test it with ollama run, it responses almost instantly. both ollama, and qdrant are dockered and i user docker logs -f to monitor their response. the observation is that if qdrant has context returns, the response time will be long 9~12s. if no context returns (no search hit) the response time is around 3~5s. I use docker logs -f to check anythingllm's logs. I don't see much useful information. do we have a debug switch for more detailed logs?
yindo changed title from [BUG]: time delay for the first word show up in the chat window with RAG return to [GH-ISSUE #2231] [BUG]: time delay for the first word show up in the chat window with RAG return 2026-06-05 14:40:52 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label possible bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#1454
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?