Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #4972] [BUG]: Generic OpenAI on llama.cpp server incorrect metrics #3119

New Issue
Closed
opened 2026-02-22 18:32:42 -05:00 by yindo · 0 comments
yindo commented 2026-02-22 18:32:42 -05:00
Owner
Copy Link
Copy Source

Originally created by @Vulpes-Stellae on GitHub (Feb 7, 2026).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/4972

Originally assigned to: @angelplusultra on GitHub.

Stack

  • AnythingLLM Desktop Windows v1.10.0.
  • Docker hosted llama.cpp server in router mode.

Issue

The displayed metrics inside AnythingLLM Desktop and the corresponding metrics for the same request inside the llama.cpp server logs.

Image 
[38311] prompt eval time =     131.41 ms /    53 tokens (    2.48 ms per token,   403.33 tokens per second)
[38311]        eval time =    1809.32 ms /   222 tokens (    8.15 ms per token,   122.70 tokens per second)
[38311]       total time =    1940.73 ms /   275 tokens

The evaluation time is correct, 1.809s for 1809.32ms.
The tokens per second are incorrect, they should be 122.70 tok/s and not 4.98 tok/s.

Details

Tested with GLM-4.7-Flash and Qwen3-VL-30B-A3B-Thinking so I suspect the issue happens with any LLM.

I have not tested on a llama.cpp server not in router mode but I don't think it makes any difference.

Originally created by @Vulpes-Stellae on GitHub (Feb 7, 2026). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/4972 Originally assigned to: @angelplusultra on GitHub. ### Stack - AnythingLLM Desktop Windows v1.10.0. - Docker hosted llama.cpp server in router mode. ### Issue The displayed metrics inside AnythingLLM Desktop and the corresponding metrics for the same request inside the llama.cpp server logs. <img width="466" height="55" alt="Image" src="https://github.com/user-attachments/assets/ad100920-8ddf-4b1f-9327-f685ae014662" /> ``` [38311] prompt eval time = 131.41 ms / 53 tokens ( 2.48 ms per token, 403.33 tokens per second) [38311] eval time = 1809.32 ms / 222 tokens ( 8.15 ms per token, 122.70 tokens per second) [38311] total time = 1940.73 ms / 275 tokens ``` The evaluation time is correct, 1.809s for 1809.32ms. The tokens per second are incorrect, they should be 122.70 tok/s and not 4.98 tok/s. ### Details Tested with GLM-4.7-Flash and Qwen3-VL-30B-A3B-Thinking so I suspect the issue happens with any LLM. I have not tested on a llama.cpp server not in router mode but I don't think it makes any difference.
yindo added the possible bugcore-team-onlyinvestigating labels 2026-02-22 18:32:42 -05:00
yindo closed this issue 2026-02-22 18:32:42 -05:00
yindo changed title from [BUG]: Generic OpenAI on llama.cpp server incorrect metrics to [GH-ISSUE #4972] [BUG]: Generic OpenAI on llama.cpp server incorrect metrics 2026-06-05 14:50:24 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label core-team-only investigating possible bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#3119
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?