Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 22:23:50 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[PR #5071] [MERGED] Native Tool calling #5287

New Issue
Closed
opened 2026-06-05 15:20:51 -04:00 by yindo · 0 comments
yindo commented 2026-06-05 15:20:51 -04:00
Owner
Copy Link
Copy Source

📋 Pull Request Information

Original PR: https://github.com/Mintplex-Labs/anything-llm/pull/5071
Author: @timothycarambat
Created: 2/26/2026
Status: ✅ Merged
Merged: 2/26/2026
Merged by: @timothycarambat

Base: master ← Head: tool-calling-native

📝 Commits (10+)

  • 4b1e57c checkpoint
  • 9106bae test MCP and flows
  • 69cb942 add native tool call detection back to LMStudio
  • 21c6497 add native tool call loops for Ollama
  • 184149a Add ablity detection to DMR (regex parse)
  • 8fac20c bedrock and generic openai with ENV flag
  • 65edd14 deepseek native tool calling
  • d5c1670 localAI native function
  • b7c8636 groq support
  • eced291 linting, add litellm and OR native tool calling via flag

📊 Changes

42 files changed (+2003 additions, -410 deletions)

View changed files

📝 docker/.env.example (+6 -1)
📝 server/.env.example (+6 -1)
📝 server/utils/AiProviders/dockerModelRunner/index.js (+42 -0)
📝 server/utils/AiProviders/lmStudio/index.js (+59 -2)
📝 server/utils/AiProviders/ollama/index.js (+26 -0)
📝 server/utils/agentFlows/index.js (+22 -3)
📝 server/utils/agents/aibitat/index.js (+45 -6)
📝 server/utils/agents/aibitat/providers/apipie.js (+9 -0)
📝 server/utils/agents/aibitat/providers/azure.js (+29 -253)
📝 server/utils/agents/aibitat/providers/bedrock.js (+251 -21)
📝 server/utils/agents/aibitat/providers/cohere.js (+9 -0)
📝 server/utils/agents/aibitat/providers/cometapi.js (+9 -0)
📝 server/utils/agents/aibitat/providers/deepseek.js (+92 -12)
📝 server/utils/agents/aibitat/providers/dellProAiStudio.js (+9 -0)
📝 server/utils/agents/aibitat/providers/dockerModelRunner.js (+98 -12)
📝 server/utils/agents/aibitat/providers/fireworksai.js (+9 -0)
📝 server/utils/agents/aibitat/providers/foundry.js (+9 -0)
📝 server/utils/agents/aibitat/providers/genericOpenAi.js (+110 -12)
📝 server/utils/agents/aibitat/providers/giteeai.js (+9 -0)
📝 server/utils/agents/aibitat/providers/groq.js (+107 -15)

...and 22 more files

📄 Description

Pull Request Type

  • ✨ feat (New feature)
  • 🐛 fix (Bug fix)
  • ♻️ refactor (Code refactoring without changing behavior)
  • 💄 style (UI style changes)
  • 🔨 chore (Build, CI, maintenance)
  • 📝 docs (Documentation updates)

Description

Improves tool calling for more native functionality. This was providing a limited experience for any user using a provider which inherited Untooled which was a great patch for early agent days but now the agentic world requires chaining and model intelligence is at a point where we can reliably do this.

This requires us to manually test or implement for every provider, but will DRAMATICALLY improve tool call salience for tool-call capable models to parallelize tool calling as well calling tools to complete a whole query and not just call one tool and exit.

Visuals (if applicable)

Additional Information

  • Total refactor and short-circuit to keep current implementation for Untooled model providers.
  • Adds hard max tool call of 10 calls per invocation - this is to prevent infinite recursion if the model is using a very dumb model who cannot exit the loop. This does not abort the query - it just prevents it from calling more tools and return a real response early.

For some providers, we have added a new ENV you can flag for native tool calling with the PROVIDER_SUPPORTS_NATIVE_TOOL_CALLING env
PROVIDER_SUPPORTS_NATIVE_TOOL_CALLING="generic-openai,bedrock,localai"

Not all providers support this, you need to verify it honors this flag within its aibitat/providers definition file since not all providers will actually honor this kind of native tool calling and it can depend on model. This is a fixed flag since the aforementioned providers do not provide a smart way to detecting if a model can/cant use tools natively.

Developer Validations

  • I ran yarn lint from the root of the repo & committed changes
  • Relevant documentation has been updated (if applicable)
  • I have tested my code functionality
  • Docker build succeeds locally

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/Mintplex-Labs/anything-llm/pull/5071 **Author:** [@timothycarambat](https://github.com/timothycarambat) **Created:** 2/26/2026 **Status:** ✅ Merged **Merged:** 2/26/2026 **Merged by:** [@timothycarambat](https://github.com/timothycarambat) **Base:** `master` ← **Head:** `tool-calling-native` --- ### 📝 Commits (10+) - [`4b1e57c`](https://github.com/Mintplex-Labs/anything-llm/commit/4b1e57c8e383192bff2a8421c66daccb6f22c20d) checkpoint - [`9106bae`](https://github.com/Mintplex-Labs/anything-llm/commit/9106bae4fbed8279bb11d327384dbab30fee4ae0) test MCP and flows - [`69cb942`](https://github.com/Mintplex-Labs/anything-llm/commit/69cb942c2b98268aab2b146c8e5e6147b0fc24e0) add native tool call detection back to LMStudio - [`21c6497`](https://github.com/Mintplex-Labs/anything-llm/commit/21c64972dbd7bd3ecbce51d4e021c8ddad0ca56e) add native tool call loops for Ollama - [`184149a`](https://github.com/Mintplex-Labs/anything-llm/commit/184149ad534bc042b25feb6130fc79fd4331967b) Add ablity detection to DMR (regex parse) - [`8fac20c`](https://github.com/Mintplex-Labs/anything-llm/commit/8fac20c80a6feeb566aa22bf09018918e6a469fd) bedrock and generic openai with ENV flag - [`65edd14`](https://github.com/Mintplex-Labs/anything-llm/commit/65edd14b5adede5f22bf6281c7ae7097a4eacdea) deepseek native tool calling - [`d5c1670`](https://github.com/Mintplex-Labs/anything-llm/commit/d5c16700940738a577c67c41ab319c75975ace42) localAI native function - [`b7c8636`](https://github.com/Mintplex-Labs/anything-llm/commit/b7c86367ad248879d4adc3c502b3b73acc34f934) groq support - [`eced291`](https://github.com/Mintplex-Labs/anything-llm/commit/eced291fea3bc0b279fc7e619f6da978504e5c5a) linting, add litellm and OR native tool calling via flag ### 📊 Changes **42 files changed** (+2003 additions, -410 deletions) <details> <summary>View changed files</summary> 📝 `docker/.env.example` (+6 -1) 📝 `server/.env.example` (+6 -1) 📝 `server/utils/AiProviders/dockerModelRunner/index.js` (+42 -0) 📝 `server/utils/AiProviders/lmStudio/index.js` (+59 -2) 📝 `server/utils/AiProviders/ollama/index.js` (+26 -0) 📝 `server/utils/agentFlows/index.js` (+22 -3) 📝 `server/utils/agents/aibitat/index.js` (+45 -6) 📝 `server/utils/agents/aibitat/providers/apipie.js` (+9 -0) 📝 `server/utils/agents/aibitat/providers/azure.js` (+29 -253) 📝 `server/utils/agents/aibitat/providers/bedrock.js` (+251 -21) 📝 `server/utils/agents/aibitat/providers/cohere.js` (+9 -0) 📝 `server/utils/agents/aibitat/providers/cometapi.js` (+9 -0) 📝 `server/utils/agents/aibitat/providers/deepseek.js` (+92 -12) 📝 `server/utils/agents/aibitat/providers/dellProAiStudio.js` (+9 -0) 📝 `server/utils/agents/aibitat/providers/dockerModelRunner.js` (+98 -12) 📝 `server/utils/agents/aibitat/providers/fireworksai.js` (+9 -0) 📝 `server/utils/agents/aibitat/providers/foundry.js` (+9 -0) 📝 `server/utils/agents/aibitat/providers/genericOpenAi.js` (+110 -12) 📝 `server/utils/agents/aibitat/providers/giteeai.js` (+9 -0) 📝 `server/utils/agents/aibitat/providers/groq.js` (+107 -15) _...and 22 more files_ </details> ### 📄 Description ### Pull Request Type <!-- For change type, change [ ] to [x]. --> - [x] ✨ feat (New feature) - [ ] 🐛 fix (Bug fix) - [x] ♻️ refactor (Code refactoring without changing behavior) - [ ] 💄 style (UI style changes) - [ ] 🔨 chore (Build, CI, maintenance) - [ ] 📝 docs (Documentation updates) ### Description Improves tool calling for more native functionality. This was providing a limited experience for any user using a provider which inherited `Untooled` which was a great patch for early agent days but now the agentic world requires chaining and model intelligence is at a point where we can reliably do this. This requires us to manually test or implement for every provider, but will **DRAMATICALLY** improve tool call salience for tool-call capable models to parallelize tool calling as well calling tools to complete a whole query and not just call one tool and exit. ### Visuals (if applicable) <!-- Add screenshots or screen recordings to demonstrate the changes, especially for UI updates. --> ### Additional Information - Total refactor and short-circuit to keep current implementation for Untooled model providers. - Adds hard max tool call of 10 calls per invocation - this is to prevent infinite recursion if the model is using a very dumb model who cannot exit the loop. This does not abort the query - it just prevents it from calling more tools and return a real response early. For some providers, we have added a new ENV you can flag for native tool calling with the `PROVIDER_SUPPORTS_NATIVE_TOOL_CALLING` env `PROVIDER_SUPPORTS_NATIVE_TOOL_CALLING="generic-openai,bedrock,localai"` Not all providers support this, you need to verify it honors this flag within its `aibitat/providers` definition file since not all providers will actually honor this kind of native tool calling and it can depend on model. This is a fixed flag since the aforementioned providers **do not** provide a smart way to detecting if a model can/cant use tools natively. <!-- Add any other context about the Pull Request here that was not captured above. --> ### Developer Validations <!-- All of the applicable items should be checked. --> - [ ] I ran `yarn lint` from the root of the repo & committed changes - [ ] Relevant documentation has been updated (if applicable) - [ ] I have tested my code functionality - [ ] Docker build succeeds locally --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
yindo added the pull-request label 2026-06-05 15:20:51 -04:00
yindo closed this issue 2026-06-05 15:20:51 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#5287
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?