Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-21 01:25:28 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #3753] [FEAT]: Add tools support for OpenAI Compatible Endpoints #2415

New Issue
Open
opened 2026-02-22 18:29:35 -05:00 by yindo · 2 comments
yindo commented 2026-02-22 18:29:35 -05:00
Owner
Copy Link
Copy Source

Originally created by @AlekseyKapustyanenko on GitHub (May 2, 2025).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/3753

What would you like to see?

Description:
Currently, AnythingLLM's OpenAI Compatible Endpoints do not support the function calling feature (i.e., the tools parameter in the request that describes available tools/functions for the LLM). Please add support for function calling in these endpoints to enable the use of advanced agentic workflows and integrations.

Motivation:

  • In many real-world scenarios, especially in enterprise and regulated environments, it is essential to use private or self-hosted LLMs for reasons of privacy, compliance, or cost.
  • Local LLMs, while private, are often slower and less capable than cloud-hosted alternatives. Providers like AWS, Azure, and Nutanix now offer powerful, scalable, and private LLM endpoints with OpenAI-compatible APIs.
  • AnythingLLM is an excellent tool for bridging local and remote LLMs, providing a unified interface, workspace management, and API compatibility.
  • When developing complex applications-such as custom MCP (Model Control Plane) servers-it is often necessary to debug and iterate on function calling logic, including the structure and registration of functions/tools. Local development and testing are much faster and safer in this context.
  • With function calling support, developers could use AnythingLLM as a local proxy for remote LLMs:
    • Deploy an LLM in AWS (or another provider), run AnythingLLM alongside it, and set up a workspace and API key in AnythingLLM.
    • On a developer’s local machine, run AnythingLLM and configure an OpenAI-compatible provider that points to the remote AnythingLLM instance.
    • Connect the custom MCP server (or any agentic application) to the local AnythingLLM instance and test function calling logic, tool registration, and agent workflows before deploying to production.
  • This setup would streamline development, debugging, and deployment of agent-based systems using private or cloud LLMs, leveraging AnythingLLM as a flexible proxy and integration layer.
Originally created by @AlekseyKapustyanenko on GitHub (May 2, 2025). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/3753 ### What would you like to see? **Description:** Currently, AnythingLLM's OpenAI Compatible Endpoints do not support the function calling feature (i.e., the `tools` parameter in the request that describes available tools/functions for the LLM). Please add support for function calling in these endpoints to enable the use of advanced agentic workflows and integrations. --- **Motivation:** - In many real-world scenarios, especially in enterprise and regulated environments, it is essential to use private or self-hosted LLMs for reasons of privacy, compliance, or cost. - Local LLMs, while private, are often slower and less capable than cloud-hosted alternatives. Providers like AWS, Azure, and Nutanix now offer powerful, scalable, and private LLM endpoints with OpenAI-compatible APIs. - AnythingLLM is an excellent tool for bridging local and remote LLMs, providing a unified interface, workspace management, and API compatibility. - When developing complex applications-such as custom MCP (Model Control Plane) servers-it is often necessary to debug and iterate on function calling logic, including the structure and registration of functions/tools. Local development and testing are much faster and safer in this context. - With function calling support, developers could use AnythingLLM as a local proxy for remote LLMs: - Deploy an LLM in AWS (or another provider), run AnythingLLM alongside it, and set up a workspace and API key in AnythingLLM. - On a developer’s local machine, run AnythingLLM and configure an OpenAI-compatible provider that points to the remote AnythingLLM instance. - Connect the custom MCP server (or any agentic application) to the local AnythingLLM instance and test function calling logic, tool registration, and agent workflows before deploying to production. - This setup would streamline development, debugging, and deployment of agent-based systems using private or cloud LLMs, leveraging AnythingLLM as a flexible proxy and integration layer.
yindo added the enhancementfeature request labels 2026-02-22 18:29:35 -05:00
yindo commented 2026-02-22 18:29:35 -05:00
Author
Owner
Copy Link
Copy Source

@mybr4inhurts commented on GitHub (Jul 7, 2025):

I'd like to add that currently it seems that you cannot invoke agent sessions for Azure models at all.
Besides that I cannot agree enough with @AlekseyKapustyanenko plus not only private models are used but also Azure OpenAI models in O365 enterprise accounts.

Just to add some context and the actual error message you'll get when trying to invoke an agent session with Azure models (using the Azure connector from AnythingLLM).

Image

the corresponding Azure documentation part:

Image

https://learn.microsoft.com/en-us/azure/ai-foundry/openai/how-to/function-calling

@mybr4inhurts commented on GitHub (Jul 7, 2025): I'd like to add that currently it seems that you cannot invoke agent sessions for Azure models at all. Besides that I cannot agree enough with @AlekseyKapustyanenko plus not only private models are used but also Azure OpenAI models in O365 enterprise accounts. Just to add some context and the actual error message you'll get when trying to invoke an agent session with Azure models (using the Azure connector from AnythingLLM). ![Image](https://github.com/user-attachments/assets/92127b04-4043-4b58-a7fd-aca2b5423ba9) the corresponding Azure documentation part: ![Image](https://github.com/user-attachments/assets/168c997b-2bec-465b-970a-e015795e76cd) https://learn.microsoft.com/en-us/azure/ai-foundry/openai/how-to/function-calling
yindo changed title from [FEAT]: Add tools support for OpenAI Compatible Endpoints to [GH-ISSUE #3753] [FEAT]: Add tools support for OpenAI Compatible Endpoints 2026-06-05 14:46:25 -04:00
yindo commented 2026-06-05 14:46:26 -04:00
Author
Owner
Copy Link
Copy Source

@dbudavari commented on GitHub (Apr 22, 2026):

Are there any plans to support this?

<!-- gh-comment-id:4298003404 --> @dbudavari commented on GitHub (Apr 22, 2026): Are there any plans to support this?
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label enhancement feature request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#2415
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?