Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-21 01:25:28 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #5520] [FEAT]: Auto mode always invokes agent mode regardless of prompt intent - should include lightweight classifier to determine tool necessity #5141

New Issue
Open
opened 2026-06-05 14:52:15 -04:00 by yindo · 5 comments
yindo commented 2026-06-05 14:52:15 -04:00
Owner
Copy Link
Copy Source

Originally created by @elevatingcreativity on GitHub (Apr 24, 2026).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/5520

What would you like to see?

Current Behavior (v1.12.0+)

When “Auto mode” is enabled on a workspace and the provider supports native tool calling, agent mode is always invoked regardless of the user’s actual intent or explicit instructions. The system does not:

  • Analyze whether tools are actually needed for the query
  • Respect system prompt instructions to avoid agent mode
  • Provide any opt-out mechanism within Auto mode

The Problem: Model Separation Is Impossible

Our specific use case highlights why this design is problematic:

What We Want

Purpose Model Reasoning
Main Chat Claude Sonnet/Opus Processing complex text documents requires high-powered AI that can't run locally
Agent Mode Gemma 27B or Qwen 3.5 (local) Smaller, locally-running models that excel at focused agent tasks without being overly chatty or burning paid tokens

What Actually Happens

  1. Auto mode ALWAYS invokes agent mode for any prompt when using Claude (or other providers with native tool calling support)
  2. Claude Sonnet is forced into agent execution for every query, even simple ones that don’t require tools
  3. Token waste is severe: Sonnet quickly burns through tokens iterating to the 10-step limit and erroring out (separate issue)
  4. System prompt instructions are ignored: Even explicit prompts like “Let’s explore this using REGULAR CHAT NOT AGENT MODE!!!” still trigger agent mode
  5. The default chat model is effectively ignored: You can configure different models for chat vs agent, but Auto mode makes it impossible to actually use them that way

What We Thought Auto Mode Would Do

We understood "Auto mode" to mean: "Intelligently decide when agent mode is needed based on the query intent."

What we discovered after investigating the code: "Auto mode" actually means "always on agent mode" for providers that support native tool calling.

This is highly confusing because:

  • The name "Auto" implies intelligent decision-making, not always-on behavior
  • Users can configure different models for chat vs agent mode, but there's no way to actually utilize that separation
  • System prompt instructions about when to use agent mode are bypassed at the routing level

Why This Is a Problem

  1. Token waste: Simple queries trigger full agent execution with multiple tool calls
  2. Poor UX for certain models: Claude performs poorly as an agent (too many iterations, token waste, hitting step limits)
  3. No model separation: Users cannot use a cheaper local model for agent tasks while keeping an expensive cloud model for main chat
  4. Misleading feature name: "Auto mode" suggests the system will decide when tools are needed, but it always uses them for capable providers

Proposed Solution: Intent-Checking Layer Before Agent Invocation

Add a lightweight intent-checking step before committing to agent mode. A small, fast model (configurable by the user) would analyze each incoming prompt to determine if tools are actually needed. The classifier would consider explicit opt-out instructions in the message, respect workspace system prompt guidance, and include a failsafe to default to regular chat if the response is unclear. This would make "Auto mode" actually intelligent rather than simply always-on.

Proposed Implementation (We Will Submit PR)

I plan to implement this fix and submit a PR. The implementation will:

  1. Modify the agent routing logic to add an intent-checking step before agent execution
  2. Add workspace settings for enableIntentClassifier, intentClassifierModel, and classifierSensitivity
  3. Include explicit opt-out pattern matching (e.g., "not agent mode", "regular chat", "no tools")
  4. Provide failsafe behavior on classifier errors (default to regular chat)

Why This Isn't Just a Bug Report

I understand from PR #5143 and Issue #4932 that this behavior was intentional design—specifically to prevent small models from over-calling tools. However, the implementation went too far in the opposite direction: it now always invokes agent mode for capable providers without any intent analysis.

A true "auto" mode should intelligently decide when tools are needed, not always use them. This feature request is for making the "Auto" mode actually live up to its name.

Environment

  • AnythingLLM Version: 1.12.1 (self-hosted docker)
  • Main Chat Model: Claude Sonnet/Opus
  • Agent Mode Models Tested: Gemma 27B-A4B(8 bit), Qwen 3.5-122b-a10b (8 bit MLX)
  • Workspace Mode: Auto

Additional Context

This feature would enable a powerful workflow pattern:

  • Fast, cheap model handles intent checking and simple agent tasks locally
  • Expensive cloud model reserved for complex reasoning that requires its capabilities
  • True separation of concerns: Each model does what it's best at

I believe this is a relatively straightforward fix that would significantly improve the user experience and make "Auto mode" actually work as users expect. I'll be working on a PR and would appreciate any feedback from the maintainers before implementation.

(PR report assisted by qwen3.5-122b-a10b in AnythingLLM)

Originally created by @elevatingcreativity on GitHub (Apr 24, 2026). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/5520 ### What would you like to see? ## Current Behavior (v1.12.0+) When “Auto mode” is enabled on a workspace and the provider supports native tool calling, agent mode is always invoked regardless of the user’s actual intent or explicit instructions. The system does not: - Analyze whether tools are actually needed for the query - Respect system prompt instructions to avoid agent mode - Provide any opt-out mechanism within Auto mode ## The Problem: Model Separation Is Impossible Our specific use case highlights why this design is problematic: ### What We Want | Purpose | Model | Reasoning | |---------|-------|-----------| | **Main Chat** | Claude Sonnet/Opus | Processing complex text documents requires high-powered AI that can't run locally | | **Agent Mode** | Gemma 27B or Qwen 3.5 (local) | Smaller, locally-running models that excel at focused agent tasks without being overly chatty or burning paid tokens | ### What Actually Happens 1. Auto mode ALWAYS invokes agent mode for any prompt when using Claude (or other providers with native tool calling support) 2. Claude Sonnet is forced into agent execution for every query, even simple ones that don’t require tools 3. Token waste is severe: Sonnet quickly burns through tokens iterating to the 10-step limit and erroring out (separate issue) 4. System prompt instructions are ignored: Even explicit prompts like “Let’s explore this using REGULAR CHAT NOT AGENT MODE!!!” still trigger agent mode 5. The default chat model is effectively ignored: You can configure different models for chat vs agent, but Auto mode makes it impossible to actually use them that way ### What We Thought Auto Mode Would Do We understood "Auto mode" to mean: *"Intelligently decide when agent mode is needed based on the query intent."* What we discovered after investigating the code: *"Auto mode" actually means "always on agent mode" for providers that support native tool calling.* This is highly confusing because: - The name "Auto" implies intelligent decision-making, not always-on behavior - Users can configure different models for chat vs agent mode, but there's no way to actually utilize that separation - System prompt instructions about when to use agent mode are bypassed at the routing level ## Why This Is a Problem 1. **Token waste**: Simple queries trigger full agent execution with multiple tool calls 2. **Poor UX for certain models**: Claude performs poorly as an agent (too many iterations, token waste, hitting step limits) 3. **No model separation**: Users cannot use a cheaper local model for agent tasks while keeping an expensive cloud model for main chat 4. **Misleading feature name**: "Auto mode" suggests the system will decide when tools are needed, but it always uses them for capable providers ## Proposed Solution: Intent-Checking Layer Before Agent Invocation Add a lightweight intent-checking step before committing to agent mode. A small, fast model (configurable by the user) would analyze each incoming prompt to determine if tools are actually needed. The classifier would consider explicit opt-out instructions in the message, respect workspace system prompt guidance, and include a failsafe to default to regular chat if the response is unclear. This would make "Auto mode" actually intelligent rather than simply always-on. ## Proposed Implementation (We Will Submit PR) I plan to implement this fix and submit a PR. The implementation will: 1. Modify the agent routing logic to add an intent-checking step before agent execution 2. Add workspace settings for `enableIntentClassifier`, `intentClassifierModel`, and `classifierSensitivity` 3. Include explicit opt-out pattern matching (e.g., "not agent mode", "regular chat", "no tools") 4. Provide failsafe behavior on classifier errors (default to regular chat) ## Why This Isn't Just a Bug Report I understand from PR #5143 and Issue #4932 that this behavior was intentional design—specifically to prevent small models from over-calling tools. However, the implementation went too far in the opposite direction: it now **always** invokes agent mode for capable providers without any intent analysis. A true "auto" mode should intelligently decide when tools are needed, not always use them. This feature request is for making the "Auto" mode actually live up to its name. ## Environment - **AnythingLLM Version**: 1.12.1 (self-hosted docker) - **Main Chat Model**: Claude Sonnet/Opus - **Agent Mode Models Tested**: Gemma 27B-A4B(8 bit), Qwen 3.5-122b-a10b (8 bit MLX) - **Workspace Mode**: Auto ## Additional Context This feature would enable a powerful workflow pattern: - **Fast, cheap model** handles intent checking and simple agent tasks locally - **Expensive cloud model** reserved for complex reasoning that requires its capabilities - **True separation of concerns**: Each model does what it's best at I believe this is a relatively straightforward fix that would significantly improve the user experience and make "Auto mode" actually work as users expect. I'll be working on a PR and would appreciate any feedback from the maintainers before implementation. (PR report assisted by qwen3.5-122b-a10b in AnythingLLM)
yindo added the enhancementfeature request labels 2026-06-05 14:52:15 -04:00
yindo commented 2026-06-05 14:52:16 -04:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Apr 24, 2026):

To clarify our design philosophy here: Auto mode was intentionally built to solve the #1 UX complaint we’ve received: "Why do I have to manually type @agent every time I want a tool used?" By defaulting to "Always-on Agent" for providers that support native tool-calling, we’ve aligned AnythingLLM with the industry-standard UX found in platforms like ChatGPT or Claude. Users generally expect the AI to "just work" with the tools available without manual invocation.

The docs even say that is the intention: https://docs.anythingllm.com/features/chat-modes#available-chat-modes

Why we wont implementing a dynamic router:

We did consider an "intelligent decision" layer, but decided against it for a few reasons:

  • Predictability: Introducing a "router" model to decide if another model should be an agent creates a "black box within a black box" scenario. This leads to indeterminate states where the agent fails to trigger when the user clearly wanted it, or vice versa. It is not a simple classification problem at all.
  • Latency & Complexity: Especially in local-first environments, adding a pre-processing step to "decide" the intent adds unnecessary overhead. The "escape" hatch here to move to chat is just a click or two away to solve the issue totally.
  • Clear Alternatives: If you want a "pure" LLM response without any tool-calling potential, that is exactly what Chat Mode is for! See our documentation on Chat Modes here.

The Path Forward:

We have to have an opinion on the "intended" default experience for the majority of users, and right now, that is a seamless agent experience. However, I hear your regarding workspace setup.

Instead of changing the fundamental logic of Auto mode - which by far is better UX for most, we could support adding a UI Preference option that allows you to set the Default Workspace Mode. This way, if your workflow favors standard Chat, every new workspace you create will respect that preference, and you won’t have to manually toggle it every time which would be annoying? How is that?

In the meantime, switching your current workspaces to Chat mode will resolve the aforementioned issues you're seeing.

<!-- gh-comment-id:4316372316 --> @timothycarambat commented on GitHub (Apr 24, 2026): To clarify our design philosophy here: **Auto** mode was intentionally built to solve the `#1` UX complaint we’ve received: *"Why do I have to manually type `@agent` every time I want a tool used?"* By defaulting to "Always-on Agent" for providers that support native tool-calling, we’ve aligned AnythingLLM with the industry-standard UX found in platforms like ChatGPT or Claude. Users generally expect the AI to "just work" with the tools available without manual invocation. The docs even say that is the intention: https://docs.anythingllm.com/features/chat-modes#available-chat-modes ### Why we wont implementing a dynamic router: We did consider an "intelligent decision" layer, but decided against it for a few reasons: * **Predictability:** Introducing a "router" model to decide if another model should be an agent creates a "black box within a black box" scenario. This leads to indeterminate states where the agent fails to trigger when the user clearly wanted it, or vice versa. It is **not** a simple classification problem at all. * **Latency & Complexity:** Especially in local-first environments, adding a pre-processing step to "decide" the intent adds unnecessary overhead. The "escape" hatch here to move to `chat` is just a click or two away to solve the issue totally. * **Clear Alternatives:** If you want a "pure" LLM response without any tool-calling potential, that is exactly what **Chat Mode** is for! [See our documentation on Chat Modes here](https://docs.anythingllm.com/features/chat-modes#available-chat-modes). ### The Path Forward: We have to have an opinion on the "intended" default experience for the majority of users, and right now, that is a seamless agent experience. However, I hear your regarding workspace setup. Instead of changing the fundamental logic of Auto mode - which by far is better UX for most, we could support adding a UI Preference option that allows you to set the **Default Workspace Mode**. This way, if your workflow favors standard Chat, every new workspace you create will respect that preference, and you won’t have to manually toggle it every time which would be annoying? How is that? In the meantime, switching your current workspaces to `Chat` mode will resolve the aforementioned issues you're seeing.
yindo commented 2026-06-05 14:52:16 -04:00
Author
Owner
Copy Link
Copy Source

@elevatingcreativity commented on GitHub (Apr 24, 2026):

@timothycarambat I understand your considerations.

However, we are using our workspace with naive users.

Having to explicitly invoke "@agent" mode for them is very confusing.

So, we were elated when "auto" mode came in, thinking it would solve the problem.

Instead what it did is broke in three ways:

  1. There's a literal bug where it just crashes on Claude Sonnet (at least it did in V1.12.0) after 10 agent invocations
  2. With models like Claude Sonnet, Agent mode WAY overdoes things, burning through far more tokens than needed
  3. It ignores the "workspace agent setting" altogether, only using the "agent workspace" setting - making it highly confusing for any user about which model is actually being used.

If nothing else, I beg and plead, if you are unable to consider another alternative, please please consider renaming "auto" mode to "agent only" mode or something. Because it is NOT auto. My team and my users (and myself) were all confused by that.

In terms of the "black box" argument - I get it. I get that you don't want it to be some black box of decision making. I was literally working on the prompt that would drive that decision making just a few minutes ago. It could be tricky.

But, what I did is a simple fix: if a user doesn't like the result (e.g. want agent mode) they can include @agent as a word anywhere in the prompt, and invoke agent mode. If they DO NOT want agent mode, they can include @noagent or @chat or @chatonly and it would route to chat.

I know there's not an ideal solution here, but I have been involved in software design for more decades than I'd like to admit - and there is almost always a solution if you are willing to work on it.

It is my humble opinion that this forced invocation of agent mode when the user sets it to "auto" is confusing and problematic.

It may work for personal users working with smaller models. But for more complex and demanding needs, this is not a workable solution.

I am really trying to make AnythingLLM work for our team and users so we can wean away from the other platforms. I'd happily contribute to the project in time and money - but there are just many glitches where assumptions like this one are made based on a single user in a simple situation, that actively impede usage in a more complex, team oriented environment.

I wish I had time to truly maintain my own codebase for this, but I don't. I have submitted multiple PR's that have been ignored to fix other issues.

Perhaps if you have the time, we could discuss it on a call, so you can understand how we are trying to use it and what kinds of issues like this one we are running into.

Thanks,
Morgan Giddings

<!-- gh-comment-id:4316671434 --> @elevatingcreativity commented on GitHub (Apr 24, 2026): @timothycarambat I understand your considerations. However, we are using our workspace with naive users. Having to explicitly invoke "@agent" mode for them is very confusing. So, we were elated when "auto" mode came in, thinking it would solve the problem. Instead what it did is broke in three ways: 1. There's a literal bug where it just crashes on Claude Sonnet (at least it did in V1.12.0) after 10 agent invocations 2. With models like Claude Sonnet, Agent mode WAY overdoes things, burning through far more tokens than needed 3. It ignores the "workspace agent setting" altogether, only using the "agent workspace" setting - making it highly confusing for any user about which model is actually being used. If nothing else, I beg and plead, if you are unable to consider another alternative, please please consider renaming "auto" mode to "agent only" mode or something. Because it is NOT auto. My team and my users (and myself) were all confused by that. In terms of the "black box" argument - I get it. I get that you don't want it to be some black box of decision making. I was literally working on the prompt that would drive that decision making just a few minutes ago. It could be tricky. But, what I did is a simple fix: if a user doesn't like the result (e.g. want agent mode) they can include @agent as a word anywhere in the prompt, and invoke agent mode. If they DO NOT want agent mode, they can include @noagent or @chat or @chatonly and it would route to chat. I know there's not an ideal solution here, but I have been involved in software design for more decades than I'd like to admit - and there is almost always a solution if you are willing to work on it. It is my humble opinion that this forced invocation of agent mode when the user sets it to "auto" is confusing and problematic. It may work for personal users working with smaller models. But for more complex and demanding needs, this is not a workable solution. I am really trying to make AnythingLLM work for our team and users so we can wean away from the other platforms. I'd happily contribute to the project in time and money - but there are just many glitches where assumptions like this one are made based on a single user in a simple situation, that actively impede usage in a more complex, team oriented environment. I wish I had time to truly maintain my own codebase for this, but I don't. I have submitted multiple PR's that have been ignored to fix other issues. Perhaps if you have the time, we could discuss it on a call, so you can understand how we are trying to use it and what kinds of issues like this one we are running into. Thanks, Morgan Giddings
yindo commented 2026-06-05 14:52:16 -04:00
Author
Owner
Copy Link
Copy Source

@elevatingcreativity commented on GitHub (Apr 24, 2026):

One concrete example of why ‘just use Chat mode’ doesn’t work for us:
We store transcripts of calls. When a user later asks a specific question about a past decision, Chat mode’s default RAG retrieval frequently misses it - I’ve tested this repeatedly. Agent mode works because it iterates through the knowledge base until it finds the right answer. This isn’t a tool-use scenario - it’s deep knowledge retrieval. Chat mode simply fails here.

AnythingLLM’s RAG retrieval could use improvement (hybrid search, re-ranking, configurable retrieval depth) so that Chat mode can actually compete with Agent mode for knowledge retrieval tasks. Until then, we’re stuck needing Agent mode for basic knowledge management and accurate retrieval - which leads back to being able to control the switching, preferably automated.

At the end of the day, with the way this is designed currently, there is no such thing as a separate "agent", there's just one mode to its operation, so the whole intent behind "agent" versus "chat" mode becomes moot. If that's the case, what's the intended purpose of that?

I appreciate your time in considering this, and again, am happy to talk if you like.

<!-- gh-comment-id:4316737782 --> @elevatingcreativity commented on GitHub (Apr 24, 2026): One concrete example of why ‘just use Chat mode’ doesn’t work for us: We store transcripts of calls. When a user later asks a specific question about a past decision, Chat mode’s default RAG retrieval frequently misses it - I’ve tested this repeatedly. Agent mode works because it iterates through the knowledge base until it finds the right answer. This isn’t a tool-use scenario - it’s deep knowledge retrieval. Chat mode simply fails here. AnythingLLM’s RAG retrieval could use improvement (hybrid search, re-ranking, configurable retrieval depth) so that Chat mode can actually compete with Agent mode for knowledge retrieval tasks. Until then, we’re stuck needing Agent mode for basic knowledge management and accurate retrieval - which leads back to being able to control the switching, preferably automated. At the end of the day, with the way this is designed currently, there is no such thing as a separate "agent", there's just one mode to its operation, so the whole intent behind "agent" versus "chat" mode becomes moot. If that's the case, what's the intended purpose of that? I appreciate your time in considering this, and again, am happy to talk if you like.
yindo commented 2026-06-05 14:52:16 -04:00
Author
Owner
Copy Link
Copy Source

@elevatingcreativity commented on GitHub (Apr 29, 2026):

Thank you @timothycarambat for taking my feedback and updating the name to agent mode.

<!-- gh-comment-id:4347094938 --> @elevatingcreativity commented on GitHub (Apr 29, 2026): Thank you @timothycarambat for taking my feedback and updating the name to agent mode.
yindo commented 2026-06-05 14:52:16 -04:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Apr 29, 2026):

@elevatingcreativity Working on addressing the other issue as changing the name doesnt solve all your valid points in the above. I am working out a plan to best address this for your (and presumably) others use cases.

Apologies for delay in response!

<!-- gh-comment-id:4348196433 --> @timothycarambat commented on GitHub (Apr 29, 2026): @elevatingcreativity Working on addressing the other issue as changing the name doesnt solve all your valid points in the above. I am working out a plan to best address this for your (and presumably) others use cases. Apologies for delay in response!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label enhancement feature request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#5141
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?