Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #3641] Performance issue #2352

New Issue
Closed
opened 2026-02-22 18:29:18 -05:00 by yindo · 1 comment
yindo commented 2026-02-22 18:29:18 -05:00
Owner
Copy Link
Copy Source

Originally created by @shravanveladi on GitHub (Apr 13, 2025).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/3641

We are preparing custom chatbox with stack(Anythingllm+ollama+llama3.2:3b+qdrant+nomic) on c6i.2xlarge instance.
The system taking more than 1 min to get response when we ask any prompt. How can i improve it ?
When i ask a prompt in console like:

ollama run llama3.2
"my prompt(2+2=?)"
I will get quick resonse withing 10 sec. but when ask same prompt in anythingllm workspace it takes morethan 1 minute some time even though it's easy prompt like "2+2=?".

How i fix it and how i make pre-load model to memory and is there any settings need to do in anythingllm to optimize speed/performance ?

Originally created by @shravanveladi on GitHub (Apr 13, 2025). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/3641 We are preparing custom chatbox with stack(Anythingllm+ollama+llama3.2:3b+qdrant+nomic) on c6i.2xlarge instance. The system taking more than 1 min to get response when we ask any prompt. How can i improve it ? When i ask a prompt in console like: >ollama run llama3.2 > "my prompt(2+2=?)" I will get quick resonse withing 10 sec. but when ask same prompt in anythingllm workspace it takes morethan 1 minute some time even though it's easy prompt like "2+2=?". How i fix it and how i make pre-load model to memory and is there any settings need to do in anythingllm to optimize speed/performance ?
yindo closed this issue 2026-02-22 18:29:18 -05:00
yindo commented 2026-02-22 18:29:18 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Apr 14, 2025):

Compute Value
vCPUs 8
Memory (GiB) 16.0
Memory per vCPU (GiB) 2.0
Physical Processor Intel Xeon 8375C (Ice Lake)
Clock Speed (GHz) 3.5
CPU Architecture x86_64
GPU 0
GPU Architecture None
Video Memory (GiB) 0
GPU Compute Capability (?) 0
FPGA 0

With only running a model on CPU you are going to get slow inference, especially as context grows or even when more messages are in the request. You can improve it by getting a bigger resource, this is going to be slow even with the most minimal context. If anything, you should be provisioning a GPU server - that will be dramatically faster.

@timothycarambat commented on GitHub (Apr 14, 2025): Compute | Value -- | -- vCPUs | 8 Memory (GiB) | 16.0 Memory per vCPU (GiB) | 2.0 Physical Processor | Intel Xeon 8375C (Ice Lake) Clock Speed (GHz) | 3.5 CPU Architecture | x86_64 GPU | 0 GPU Architecture | None Video Memory (GiB) | 0 GPU Compute Capability (?) | 0 FPGA | 0 With only running a model on CPU you are going to get slow inference, especially as context grows or even when more messages are in the request. You can improve it by getting a bigger resource, this is going to be slow even with the most minimal context. If anything, you should be provisioning a GPU server - that will be dramatically faster.
yindo changed title from Performance issue to [GH-ISSUE #3641] Performance issue 2026-06-05 14:46:04 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#2352
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?