Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 22:23:50 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #5639] [FEAT]: Document Generic OpenAI native tool-calling flag and warn when automatic agents fall back to UnTooled #5211

New Issue
Closed
opened 2026-06-05 14:52:40 -04:00 by yindo · 0 comments
yindo commented 2026-06-05 14:52:40 -04:00
Owner
Copy Link
Copy Source

Originally created by @thomasmaerz on GitHub (May 15, 2026).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/5639

Summary

Generic OpenAI endpoints that support native tool/function calling can silently fall back to UnTooled unless PROVIDER_SUPPORTS_NATIVE_TOOL_CALLING includes generic-openai.

This is documented on the general configuration page, but it is easy to miss from the Generic OpenAI setup flow. The current UX also makes this hard to diagnose because automatic agent mode may appear enabled while tool use still falls back to the prompt-based UnTooled path.

Observed Behavior

On AnythingLLM 1.12.1 with:

LLM_PROVIDER='generic-openai'
GENERIC_OPEN_AI_MODEL_PREF='cx/gpt-5.4-mini'

and no:

PROVIDER_SUPPORTS_NATIVE_TOOL_CALLING="generic-openai"

logs show:

Generic OpenAI supports native tool calling is DISABLED. Will use UnTooled instead.
Untooled.stream - will process this chat completion.

In this state:

  • Workspace chatMode='automatic' does not invoke tools without @agent.
  • Agent skills are attached and selected.
  • Multi-step tools can behave unreliably because the system is using the prompt-based fallback.
  • In one real case, Gmail repeatedly invoked gmail-search but never invoked gmail-read-thread.

After setting:

PROVIDER_SUPPORTS_NATIVE_TOOL_CALLING="generic-openai"

and restarting, logs changed to:

Generic OpenAI supports native tool calling is ENABLED.
Provider.stream (tooled) - will process this chat completion.

The same Gmail workflow then successfully invoked:

gmail-get-inbox
gmail-read-thread

and automatic mode worked without @agent.

UX Problem

The current behavior is technically documented, but it is not surfaced where users configure Generic OpenAI.

The Generic OpenAI setup page describes OpenAI-compatible endpoints, but does not clearly call out that native tool calling requires:

PROVIDER_SUPPORTS_NATIVE_TOOL_CALLING="generic-openai"

The general configuration page also says:

This setting only applies to local LLM providers.

but the listed providers include Generic OpenAI, Groq, AWS Bedrock, and OpenRouter, which are not necessarily local providers. That wording is confusing.

Suggested Improvements

  • Add a warning/note to the Generic OpenAI setup page:
    • If your endpoint supports native tools/function calling, set PROVIDER_SUPPORTS_NATIVE_TOOL_CALLING="generic-openai".
    • Restart the container/service after changing it.
  • In the UI, warn when:
    • Provider is generic-openai
    • Workspace chat mode is automatic
    • Agent skills are enabled
    • Native tool calling is not enabled
  • Improve log wording to include the exact remediation:
    • Set PROVIDER_SUPPORTS_NATIVE_TOOL_CALLING="generic-openai" to enable native tool calling for this provider.
  • Clarify the docs wording that this applies to provider integrations that cannot auto-detect tool support, not only local providers.

Why This Matters

Without this flag, users can end up debugging model, proxy, connector, or network behavior when the real issue is that AnythingLLM is intentionally using UnTooled.

This particularly affects multi-step tools such as Gmail, where search/list tools may run but follow-up tools like gmail-read-thread may never be invoked.

Originally created by @thomasmaerz on GitHub (May 15, 2026). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/5639 ## Summary Generic OpenAI endpoints that support native tool/function calling can silently fall back to `UnTooled` unless `PROVIDER_SUPPORTS_NATIVE_TOOL_CALLING` includes `generic-openai`. This is documented on the general configuration page, but it is easy to miss from the Generic OpenAI setup flow. The current UX also makes this hard to diagnose because automatic agent mode may appear enabled while tool use still falls back to the prompt-based `UnTooled` path. ## Observed Behavior On AnythingLLM `1.12.1` with: ```env LLM_PROVIDER='generic-openai' GENERIC_OPEN_AI_MODEL_PREF='cx/gpt-5.4-mini' ``` and no: ```env PROVIDER_SUPPORTS_NATIVE_TOOL_CALLING="generic-openai" ``` logs show: ```text Generic OpenAI supports native tool calling is DISABLED. Will use UnTooled instead. Untooled.stream - will process this chat completion. ``` In this state: - Workspace `chatMode='automatic'` does not invoke tools without `@agent`. - Agent skills are attached and selected. - Multi-step tools can behave unreliably because the system is using the prompt-based fallback. - In one real case, Gmail repeatedly invoked `gmail-search` but never invoked `gmail-read-thread`. After setting: ```env PROVIDER_SUPPORTS_NATIVE_TOOL_CALLING="generic-openai" ``` and restarting, logs changed to: ```text Generic OpenAI supports native tool calling is ENABLED. Provider.stream (tooled) - will process this chat completion. ``` The same Gmail workflow then successfully invoked: ```text gmail-get-inbox gmail-read-thread ``` and automatic mode worked without `@agent`. ## UX Problem The current behavior is technically documented, but it is not surfaced where users configure Generic OpenAI. The Generic OpenAI setup page describes OpenAI-compatible endpoints, but does not clearly call out that native tool calling requires: ```env PROVIDER_SUPPORTS_NATIVE_TOOL_CALLING="generic-openai" ``` The general configuration page also says: ```text This setting only applies to local LLM providers. ``` but the listed providers include `Generic OpenAI`, `Groq`, `AWS Bedrock`, and `OpenRouter`, which are not necessarily local providers. That wording is confusing. ## Suggested Improvements - Add a warning/note to the Generic OpenAI setup page: - If your endpoint supports native tools/function calling, set `PROVIDER_SUPPORTS_NATIVE_TOOL_CALLING="generic-openai"`. - Restart the container/service after changing it. - In the UI, warn when: - Provider is `generic-openai` - Workspace chat mode is `automatic` - Agent skills are enabled - Native tool calling is not enabled - Improve log wording to include the exact remediation: - `Set PROVIDER_SUPPORTS_NATIVE_TOOL_CALLING="generic-openai" to enable native tool calling for this provider.` - Clarify the docs wording that this applies to provider integrations that cannot auto-detect tool support, not only local providers. ## Why This Matters Without this flag, users can end up debugging model, proxy, connector, or network behavior when the real issue is that AnythingLLM is intentionally using `UnTooled`. This particularly affects multi-step tools such as Gmail, where search/list tools may run but follow-up tools like `gmail-read-thread` may never be invoked.
yindo closed this issue 2026-06-05 14:52:40 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#5211
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?