Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[PR #4279] [MERGED] Enable real-time agent tool call streaming for all providers #4576

New Issue
Closed
opened 2026-02-22 18:36:06 -05:00 by yindo · 0 comments
yindo commented 2026-02-22 18:36:06 -05:00
Owner
Copy Link
Copy Source

📋 Pull Request Information

Original PR: https://github.com/Mintplex-Labs/anything-llm/pull/4279
Author: @timothycarambat
Created: 8/12/2025
Status: ✅ Merged
Merged: 10/1/2025
Merged by: @timothycarambat

Base: master ← Head: agentic-streaming

📝 Commits (10+)

  • fc003c5 WIP agentic tool call streaming
  • 6ac95b3 WIP rest of providers EXCLUDES Bedrock and GenericOpenAI
  • 40dcf8b patch untooled complete/streaming to use chatCallback provider from provider class and not assume OpenAI client struct
  • 79f2e09 modify ollama to function with its own overrides
  • c710eee dev build
  • c7b427f Merge branch 'master' into agentic-streaming
  • 9a99933 fix message sanization for anthropic agent streaming
  • 155bb7a Merge branch 'master' into agentic-streaming
  • 2d0768b wip fix anthropic agentic streaming sanitization
  • 6256998 patch gemini, webgenui, generic aibitat providers + disable providers unable to test

📊 Changes

36 files changed (+1727 additions, -1275 deletions)

View changed files

📝 .github/workflows/dev-build.yaml (+1 -1)
📝 frontend/src/components/WorkspaceChat/ChatContainer/ChatHistory/HistoricalMessage/index.jsx (+1 -0)
📝 frontend/src/components/WorkspaceChat/ChatContainer/ChatHistory/StatusResponse/index.jsx (+1 -5)
📝 frontend/src/components/WorkspaceChat/ChatContainer/ChatHistory/index.jsx (+0 -4)
📝 frontend/src/components/WorkspaceChat/ChatContainer/index.jsx (+2 -1)
📝 frontend/src/utils/chat/agent.js (+88 -2)
📝 server/utils/agents/aibitat/index.js (+166 -30)
📝 server/utils/agents/aibitat/providers/ai-provider.js (+78 -1)
📝 server/utils/agents/aibitat/providers/anthropic.js (+212 -59)
📝 server/utils/agents/aibitat/providers/apipie.js (+27 -53)
📝 server/utils/agents/aibitat/providers/azure.js (+8 -3)
📝 server/utils/agents/aibitat/providers/cometapi.js (+27 -47)
📝 server/utils/agents/aibitat/providers/deepseek.js (+27 -53)
📝 server/utils/agents/aibitat/providers/dellProAiStudio.js (+27 -52)
📝 server/utils/agents/aibitat/providers/fireworksai.js (+27 -53)
📝 server/utils/agents/aibitat/providers/gemini.js (+107 -2)
📝 server/utils/agents/aibitat/providers/genericOpenAi.js (+29 -52)
📝 server/utils/agents/aibitat/providers/groq.js (+27 -53)
📝 server/utils/agents/aibitat/providers/helpers/untooled.js (+266 -14)
📝 server/utils/agents/aibitat/providers/koboldcpp.js (+27 -53)

...and 16 more files

📄 Description

Pull Request Type

  • ✨ feat
  • 🐛 fix
  • ♻️ refactor
  • 💄 style
  • 🔨 chore
  • 📝 docs

Relevant Issues

resolves #xxx

What is in this change?

Adds support for the following models to support real-time tool call streaming of outputs:

  • OpenAI
  • Anthropic - Patched and thoroughly tested for message/function calling formatting
  • LMStudio
  • Ollama
  • Groq
  • TogetherAI
  • AzureOpenAi - Not able to test & disabled streaming
  • KoboldCPP
  • LocalAI - Not able to test & disabled streaming
  • OpenRouter
  • Mistral
  • GenericOpenAi - Patched & no streaming enabled since this is a generic connector
  • DeepSeek
  • Perplexity
  • TextWebGenUi - Patched and tested
  • AWSBedrock - Not able to test
  • FireworksAI
  • LiteLLM
  • ApiPie
  • XAI
  • Novita
  • NvidiaNim - Not able to test & disabled streaming
  • PPIO - Not able to test & disabled streaming
  • Gemini - Disabled streaming

Gemini does not support streaming with tool calling via OpenAI SDK, disabled streaming and patched complete() function.

Tried rewriting using @google/generative-ai native package but is very specific with tool naming and requires reworking aibitat to sanitize function names before calling funcitons

  • DellProAiStudio - Not able to test & disabled streaming
  • MoonshotAi

Todo:

  • AWS Bedrock (complex support?)
  • Generic OpenAI (do we do tools or functions)?

Additional Information

For Provider only inherited classes:

  • When fully OpenAI Compatible (Azure/OpenAI) the stream function on Provider will apply.
  • All complete endpoints however still lie on their respective classes

For Provider + Untooled inherited classes:

  • Both stream and complete calls are delegated to the Untooled stream and complete
  • Ollama is the exception, which has its own override methods since it uses the ollama SDK.

Developer Validations & Tests

Always to be sure to test these conditions

  • @agent with a for sure tool call
  • @agent that is just a regular message
  • Mix of contionous conversation of tool/chats intermixed
  • Tested both sync and steam agent and non-agent calls on the developer API (with and without tools)
  • Tested frontend agentic flows for every provider.
  • Test agent Flow support (should work all the same)
  • I ran yarn lint from the root of the repo & committed changes
  • Relevant documentation has been updated
  • I have tested my code functionality
  • Docker build succeeds locally

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/Mintplex-Labs/anything-llm/pull/4279 **Author:** [@timothycarambat](https://github.com/timothycarambat) **Created:** 8/12/2025 **Status:** ✅ Merged **Merged:** 10/1/2025 **Merged by:** [@timothycarambat](https://github.com/timothycarambat) **Base:** `master` ← **Head:** `agentic-streaming` --- ### 📝 Commits (10+) - [`fc003c5`](https://github.com/Mintplex-Labs/anything-llm/commit/fc003c5bbf3942e1069289d35f429ecd80e2975b) WIP agentic tool call streaming - [`6ac95b3`](https://github.com/Mintplex-Labs/anything-llm/commit/6ac95b3a240fa010c6596799235ff596fd1d7e63) WIP rest of providers EXCLUDES Bedrock and GenericOpenAI - [`40dcf8b`](https://github.com/Mintplex-Labs/anything-llm/commit/40dcf8b247049ef92e7a73b6f0dd3e3ac34c1cfb) patch untooled complete/streaming to use chatCallback provider from provider class and not assume OpenAI client struct - [`79f2e09`](https://github.com/Mintplex-Labs/anything-llm/commit/79f2e0935af49c053a96343d91655a66289139bc) modify ollama to function with its own overrides - [`c710eee`](https://github.com/Mintplex-Labs/anything-llm/commit/c710eee773f6ac43bda8ead000ee2ac2946a54a4) dev build - [`c7b427f`](https://github.com/Mintplex-Labs/anything-llm/commit/c7b427fb143b2a02c370894d13a66b68e370b535) Merge branch 'master' into agentic-streaming - [`9a99933`](https://github.com/Mintplex-Labs/anything-llm/commit/9a99933a77c04831b4563cdf497a0ab5df247b0f) fix message sanization for anthropic agent streaming - [`155bb7a`](https://github.com/Mintplex-Labs/anything-llm/commit/155bb7a457f080e7ede16006dbd0ec977cbc27c3) Merge branch 'master' into agentic-streaming - [`2d0768b`](https://github.com/Mintplex-Labs/anything-llm/commit/2d0768b150a404e291833c98fbeea3e70d2ee72c) wip fix anthropic agentic streaming sanitization - [`6256998`](https://github.com/Mintplex-Labs/anything-llm/commit/625699887209e149abdc517c5a7a0083c0fa07a1) patch gemini, webgenui, generic aibitat providers + disable providers unable to test ### 📊 Changes **36 files changed** (+1727 additions, -1275 deletions) <details> <summary>View changed files</summary> 📝 `.github/workflows/dev-build.yaml` (+1 -1) 📝 `frontend/src/components/WorkspaceChat/ChatContainer/ChatHistory/HistoricalMessage/index.jsx` (+1 -0) 📝 `frontend/src/components/WorkspaceChat/ChatContainer/ChatHistory/StatusResponse/index.jsx` (+1 -5) 📝 `frontend/src/components/WorkspaceChat/ChatContainer/ChatHistory/index.jsx` (+0 -4) 📝 `frontend/src/components/WorkspaceChat/ChatContainer/index.jsx` (+2 -1) 📝 `frontend/src/utils/chat/agent.js` (+88 -2) 📝 `server/utils/agents/aibitat/index.js` (+166 -30) 📝 `server/utils/agents/aibitat/providers/ai-provider.js` (+78 -1) 📝 `server/utils/agents/aibitat/providers/anthropic.js` (+212 -59) 📝 `server/utils/agents/aibitat/providers/apipie.js` (+27 -53) 📝 `server/utils/agents/aibitat/providers/azure.js` (+8 -3) 📝 `server/utils/agents/aibitat/providers/cometapi.js` (+27 -47) 📝 `server/utils/agents/aibitat/providers/deepseek.js` (+27 -53) 📝 `server/utils/agents/aibitat/providers/dellProAiStudio.js` (+27 -52) 📝 `server/utils/agents/aibitat/providers/fireworksai.js` (+27 -53) 📝 `server/utils/agents/aibitat/providers/gemini.js` (+107 -2) 📝 `server/utils/agents/aibitat/providers/genericOpenAi.js` (+29 -52) 📝 `server/utils/agents/aibitat/providers/groq.js` (+27 -53) 📝 `server/utils/agents/aibitat/providers/helpers/untooled.js` (+266 -14) 📝 `server/utils/agents/aibitat/providers/koboldcpp.js` (+27 -53) _...and 16 more files_ </details> ### 📄 Description ### Pull Request Type <!-- For change type, change [ ] to [x]. --> - [ ] ✨ feat - [ ] 🐛 fix - [ ] ♻️ refactor - [ ] 💄 style - [ ] 🔨 chore - [ ] 📝 docs ### Relevant Issues <!-- Use "resolves #xxx" to auto resolve on merge. Otherwise, please use "connect #xxx" --> resolves #xxx ### What is in this change? Adds support for the following models to support real-time tool call streaming of outputs: - [x] OpenAI - [x] Anthropic - Patched and thoroughly tested for message/function calling formatting - [x] LMStudio - [x] Ollama - [x] Groq - [x] TogetherAI - [ ] AzureOpenAi - Not able to test & disabled streaming - [x] KoboldCPP - [ ] LocalAI - Not able to test & disabled streaming - [x] OpenRouter - [x] Mistral - [x] GenericOpenAi - Patched & no streaming enabled since this is a generic connector - [x] DeepSeek - [x] Perplexity - [x] TextWebGenUi - Patched and tested - [ ] AWSBedrock - Not able to test - [x] FireworksAI - [x] LiteLLM - [x] ApiPie - [x] XAI - [x] Novita - [ ] NvidiaNim - Not able to test & disabled streaming - [ ] PPIO - Not able to test & disabled streaming - [x] Gemini - Disabled streaming > Gemini does not support streaming with tool calling via OpenAI SDK, disabled streaming and patched complete() function. > Tried rewriting using @google/generative-ai native package but is very specific with tool naming and requires reworking aibitat to sanitize function names before calling funcitons - [ ] DellProAiStudio - Not able to test & disabled streaming - [x] MoonshotAi <!-- Describe the changes in this PR that are impactful to the repo. --> ## Todo: - [ ] AWS Bedrock (complex support?) - [ ] Generic OpenAI (do we do tools or functions)? ### Additional Information For `Provider` only inherited classes: - When fully OpenAI Compatible (Azure/OpenAI) the `stream` function on Provider will apply. - All `complete` endpoints however still lie on their respective classes For `Provider + Untooled` inherited classes: - Both `stream` and `complete` calls are delegated to the Untooled `stream` and `complete` - Ollama is the exception, which has its own override methods since it uses the `ollama` SDK. ### Developer Validations & Tests **Always to be sure to test these conditions** - `@agent` with a for sure tool call - `@agent` that is just a regular message - Mix of contionous conversation of tool/chats intermixed ----- - [x] Tested both sync and steam agent and non-agent calls on the developer API (with and without tools) - [x] Tested frontend agentic flows for every provider. - [x] Test agent Flow support (should work all the same) <!-- All of the applicable items should be checked. --> - [ ] I ran `yarn lint` from the root of the repo & committed changes - [ ] Relevant documentation has been updated - [ ] I have tested my code functionality - [ ] Docker build succeeds locally --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
yindo added the pull-request label 2026-02-22 18:36:06 -05:00
yindo closed this issue 2026-02-22 18:36:06 -05:00
yindo changed title from [PR #4279] Enable real-time agent tool call streaming for all providers to [PR #4279] [MERGED] Enable real-time agent tool call streaming for all providers 2026-06-05 15:19:16 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#4576
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?