Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[PR #2825] [MERGED] LLM performance metric tracking #4137

New Issue
Closed
opened 2026-02-22 18:35:13 -05:00 by yindo · 0 comments
yindo commented 2026-02-22 18:35:13 -05:00
Owner
Copy Link
Copy Source

📋 Pull Request Information

Original PR: https://github.com/Mintplex-Labs/anything-llm/pull/2825
Author: @timothycarambat
Created: 12/13/2024
Status: ✅ Merged
Merged: 12/16/2024
Merged by: @timothycarambat

Base: master ← Head: performance-metrics

📝 Commits (10+)

  • 5bd5621 WIP performance metric tracking
  • 5a4699a fix: patch UI trying to .toFixed() null metric
  • caadbed Apipie implmentation, not tested
  • ae60d7c Cleanup Anthropic notes, Add support for AzureOpenAI tracking
  • c6223f7 bedrock token metric tracking
  • 43316db Cohere support
  • b8d31d6 feat: improve default stream handler to track for provider who are actually OpenAI compliant in usage reporting
  • ad51ee7 feat: Add FireworksAI tracking reporting
  • 1a4f9d1 Add token reporting for GenericOpenAI
  • adb0f79 token reporting for koboldcpp + lmstudio

📊 Changes

42 files changed (+1770 additions, -566 deletions)

View changed files

➕ frontend/src/components/WorkspaceChat/ChatContainer/ChatHistory/HistoricalMessage/Actions/RenderMetrics/index.jsx (+118 -0)
📝 frontend/src/components/WorkspaceChat/ChatContainer/ChatHistory/HistoricalMessage/Actions/index.jsx (+3 -0)
📝 frontend/src/components/WorkspaceChat/ChatContainer/ChatHistory/HistoricalMessage/index.jsx (+2 -0)
📝 frontend/src/components/WorkspaceChat/ChatContainer/ChatHistory/index.jsx (+1 -0)
📝 frontend/src/components/WorkspaceChat/ChatContainer/ChatTooltips/index.jsx (+6 -0)
📝 frontend/src/components/WorkspaceChat/ChatContainer/index.jsx (+11 -8)
📝 frontend/src/utils/chat/index.js (+38 -11)
📝 server/.env.example (+4 -0)
📝 server/endpoints/api/admin/index.js (+20 -28)
📝 server/utils/AiProviders/anthropic/index.js (+61 -18)
📝 server/utils/AiProviders/apipie/index.js (+53 -20)
📝 server/utils/AiProviders/azureOpenAi/index.js (+50 -12)
📝 server/utils/AiProviders/bedrock/index.js (+57 -18)
📝 server/utils/AiProviders/cohere/index.js (+62 -18)
📝 server/utils/AiProviders/deepseek/index.js (+40 -17)
📝 server/utils/AiProviders/fireworksAi/index.js (+35 -13)
📝 server/utils/AiProviders/gemini/index.js (+48 -9)
📝 server/utils/AiProviders/genericOpenAi/index.js (+44 -20)
📝 server/utils/AiProviders/groq/index.js (+42 -17)
📝 server/utils/AiProviders/huggingface/index.js (+40 -14)

...and 22 more files

📄 Description

Pull Request Type

  • ✨ feat
  • 🐛 fix
  • ♻️ refactor
  • 💄 style
  • 🔨 chore
  • 📝 docs

What is in this change?

Adds token metric/usage tracking to every request for sync and streaming chats.

Additional Information

Adds LLM token calculations to all providers, even those which do not support metrics
Update Ollama client to use native ollama lib and not Langchain

  • Anthropic
  • Apipie**
  • Azure OpenAI
  • AWS Bedrock
  • Cohere
  • DeepSeek
  • FireworksAI
  • Gemini
  • Generic OpenAI
  • Groq
  • Hugging Face**
  • KoboldCPP
  • LiteLLM**
  • LMStudio
  • LocalAI**
  • Mistral
  • Native
  • Novita
  • NVIDIA NIM
  • Ollama
  • OpenAI
  • OpenRouter
  • Perplexity
  • TextWebGenUI
  • Together
  • XAI

Developer Validations

QA

  • Test Sync and Streaming for every provider
  • Test Frontend chat endpoint for every provider
  • Test Embed endpoint for every provider
  • Test API chat endpoint for every provider
  • Test OpenAI compatible API chat endpoint for every provider
  • I ran yarn lint from the root of the repo & committed changes
  • Relevant documentation has been updated
  • I have tested my code functionality
  • Docker build succeeds locally

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/Mintplex-Labs/anything-llm/pull/2825 **Author:** [@timothycarambat](https://github.com/timothycarambat) **Created:** 12/13/2024 **Status:** ✅ Merged **Merged:** 12/16/2024 **Merged by:** [@timothycarambat](https://github.com/timothycarambat) **Base:** `master` ← **Head:** `performance-metrics` --- ### 📝 Commits (10+) - [`5bd5621`](https://github.com/Mintplex-Labs/anything-llm/commit/5bd5621a2eb6c26d4405a9d33b9713d3608f48c7) WIP performance metric tracking - [`5a4699a`](https://github.com/Mintplex-Labs/anything-llm/commit/5a4699a00252a2e6f03e470c015dd6e35bc0e69d) fix: patch UI trying to .toFixed() null metric - [`caadbed`](https://github.com/Mintplex-Labs/anything-llm/commit/caadbed40a4bd2d0bd0e4690f0c4615ae0a39343) Apipie implmentation, not tested - [`ae60d7c`](https://github.com/Mintplex-Labs/anything-llm/commit/ae60d7c171aba641a2cccac878818c19319b4c86) Cleanup Anthropic notes, Add support for AzureOpenAI tracking - [`c6223f7`](https://github.com/Mintplex-Labs/anything-llm/commit/c6223f714ef9647a3430872c32f77c1059b7f0b9) bedrock token metric tracking - [`43316db`](https://github.com/Mintplex-Labs/anything-llm/commit/43316db24522375e4cde4e2993bce133e1841494) Cohere support - [`b8d31d6`](https://github.com/Mintplex-Labs/anything-llm/commit/b8d31d6ced4bb133cb4ed80bd74444313f07ba62) feat: improve default stream handler to track for provider who are actually OpenAI compliant in usage reporting - [`ad51ee7`](https://github.com/Mintplex-Labs/anything-llm/commit/ad51ee756e241d3ad9397276b3eb68acfa8a0d8b) feat: Add FireworksAI tracking reporting - [`1a4f9d1`](https://github.com/Mintplex-Labs/anything-llm/commit/1a4f9d1860b9cd8b285bd8981989d107c64aef61) Add token reporting for GenericOpenAI - [`adb0f79`](https://github.com/Mintplex-Labs/anything-llm/commit/adb0f79c7846f6b468a2bcfc80cc6323658d9d06) token reporting for koboldcpp + lmstudio ### 📊 Changes **42 files changed** (+1770 additions, -566 deletions) <details> <summary>View changed files</summary> ➕ `frontend/src/components/WorkspaceChat/ChatContainer/ChatHistory/HistoricalMessage/Actions/RenderMetrics/index.jsx` (+118 -0) 📝 `frontend/src/components/WorkspaceChat/ChatContainer/ChatHistory/HistoricalMessage/Actions/index.jsx` (+3 -0) 📝 `frontend/src/components/WorkspaceChat/ChatContainer/ChatHistory/HistoricalMessage/index.jsx` (+2 -0) 📝 `frontend/src/components/WorkspaceChat/ChatContainer/ChatHistory/index.jsx` (+1 -0) 📝 `frontend/src/components/WorkspaceChat/ChatContainer/ChatTooltips/index.jsx` (+6 -0) 📝 `frontend/src/components/WorkspaceChat/ChatContainer/index.jsx` (+11 -8) 📝 `frontend/src/utils/chat/index.js` (+38 -11) 📝 `server/.env.example` (+4 -0) 📝 `server/endpoints/api/admin/index.js` (+20 -28) 📝 `server/utils/AiProviders/anthropic/index.js` (+61 -18) 📝 `server/utils/AiProviders/apipie/index.js` (+53 -20) 📝 `server/utils/AiProviders/azureOpenAi/index.js` (+50 -12) 📝 `server/utils/AiProviders/bedrock/index.js` (+57 -18) 📝 `server/utils/AiProviders/cohere/index.js` (+62 -18) 📝 `server/utils/AiProviders/deepseek/index.js` (+40 -17) 📝 `server/utils/AiProviders/fireworksAi/index.js` (+35 -13) 📝 `server/utils/AiProviders/gemini/index.js` (+48 -9) 📝 `server/utils/AiProviders/genericOpenAi/index.js` (+44 -20) 📝 `server/utils/AiProviders/groq/index.js` (+42 -17) 📝 `server/utils/AiProviders/huggingface/index.js` (+40 -14) _...and 22 more files_ </details> ### 📄 Description ### Pull Request Type <!-- For change type, change [ ] to [x]. --> - [x] ✨ feat - [ ] 🐛 fix - [ ] ♻️ refactor - [ ] 💄 style - [ ] 🔨 chore - [ ] 📝 docs ### What is in this change? Adds token metric/usage tracking to every request for sync and streaming chats. ### Additional Information Adds LLM token calculations to all providers, even those which do not support metrics Update `Ollama` client to use native ollama lib and not Langchain - [x] Anthropic - [x] Apipie** - [x] Azure OpenAI - [x] AWS Bedrock - [x] Cohere - [x] DeepSeek - [x] FireworksAI - [x] Gemini - [x] Generic OpenAI - [x] Groq - [x] Hugging Face** - [x] KoboldCPP - [x] LiteLLM** - [x] LMStudio - [x] LocalAI** - [x] Mistral - [x] ~Native~ - [x] Novita - [x] NVIDIA NIM - [x] Ollama - [x] OpenAI - [x] OpenRouter - [x] Perplexity - [x] TextWebGenUI - [x] Together - [x] XAI ### Developer Validations QA - Test Sync and Streaming for every provider - Test Frontend chat endpoint for every provider - Test Embed endpoint for every provider - Test API chat endpoint for every provider - Test OpenAI compatible API chat endpoint for every provider --- - [ ] I ran `yarn lint` from the root of the repo & committed changes - [ ] Relevant documentation has been updated - [ ] I have tested my code functionality - [ ] Docker build succeeds locally --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
yindo added the pull-request label 2026-02-22 18:35:13 -05:00
yindo closed this issue 2026-02-22 18:35:13 -05:00
yindo changed title from [PR #2825] LLM performance metric tracking to [PR #2825] [MERGED] LLM performance metric tracking 2026-06-05 15:17:01 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#4137
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?