Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #4567] [BUG]: Inaccurate duration metric from Ollama #2903

New Issue
Closed
opened 2026-02-22 18:31:45 -05:00 by yindo · 1 comment
yindo commented 2026-02-22 18:31:45 -05:00
Owner
Copy Link
Copy Source

Originally created by @jonathanortega2023 on GitHub (Oct 21, 2025).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/4567

How are you running AnythingLLM?

Docker (local)

What happened?

I'm setting up analytics in my fork and noticed tps/duration not lining up with expectations (especially when not streaming). In my tests with Phi4:14b streaming was about 60tps, but non-streaming was about 15tps. In either case, metrics are capturing duration and using that (with network latency + time to first token) to calculate outputTps, which is misleading.

Instead of using duration for the calculation, it should be done with eval_duration, as this is how output speed is generally calculated.

Originally created by @jonathanortega2023 on GitHub (Oct 21, 2025). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/4567 ### How are you running AnythingLLM? Docker (local) ### What happened? I'm setting up analytics in my fork and noticed tps/duration not lining up with expectations (especially when not streaming). In my tests with Phi4:14b streaming was about 60tps, but non-streaming was about 15tps. In either case, metrics are capturing duration and using that (with network latency + time to first token) to calculate outputTps, which is misleading. Instead of using duration for the calculation, it should be done with eval_duration, as this is how output speed is generally calculated.
yindo added the possible bug label 2026-02-22 18:31:45 -05:00
yindo closed this issue 2026-02-22 18:31:45 -05:00
yindo commented 2026-02-22 18:31:45 -05:00
Author
Owner
Copy Link
Copy Source

@danny0094 commented on GitHub (Oct 27, 2025):

You’re absolutely right — using duration for tps mixes in network latency and streaming overhead, which makes models appear slower than they are.
Using eval_duration (pure evaluation time) would provide a more accurate “tokens per second” metric.
This matches how LLM benchmarks (like OpenAI or vLLM) calculate throughput.

uggestion (logic fix):
const outputTps = totalTokens / eval_duration;
instead of
const outputTps = totalTokens / duration;

Why this matters:
Streaming models (e.g. Phi4:14b) appear slower than they are.
Non-streaming models get skewed metrics due to latency.
Aligns AnythingLLM with industry-standard eval metrics.

It feels like a mistake since you're comparing streaming and non-streaming runs. :)

@danny0094 commented on GitHub (Oct 27, 2025): You’re absolutely right — using duration for tps mixes in network latency and streaming overhead, which makes models appear slower than they are. Using eval_duration (pure evaluation time) would provide a more accurate “tokens per second” metric. This matches how LLM benchmarks (like OpenAI or vLLM) calculate throughput. uggestion (logic fix): `const outputTps = totalTokens / eval_duration;` instead of `const outputTps = totalTokens / duration;` Why this matters: Streaming models (e.g. Phi4:14b) appear slower than they are. Non-streaming models get skewed metrics due to latency. Aligns AnythingLLM with industry-standard eval metrics. It feels like a mistake since you're comparing streaming and non-streaming runs. :)
yindo changed title from [BUG]: Inaccurate duration metric from Ollama to [GH-ISSUE #4567] [BUG]: Inaccurate duration metric from Ollama 2026-06-05 14:49:11 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label possible bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#2903
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?