Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #5209] [FEAT]: Model Router #4967

New Issue
Closed
opened 2026-06-05 14:51:11 -04:00 by yindo · 2 comments
yindo commented 2026-06-05 14:51:11 -04:00
Owner
Copy Link
Copy Source

Originally created by @PikkonMG on GitHub (Mar 15, 2026).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/5209

What would you like to see?

Feature Request: Intelligent Model Router

Really liking AnythingLLM.

That said, I’d love to see AnythingLLM go a step further with something like an Intelligent Model Router built directly into the platform.

The basic idea: instead of manually switching workspaces depending on the task, AnythingLLM could automatically send the prompt to the best model based on rules you define. Kind of like a traffic controller sitting in front of your Ollama stack.

What I’m imagining

A routing rules panel in Settings where you can define things like:

If the prompt contains code → route to qwen2.5-coder:7b

If the prompt includes an image → route to qwen3.5:9b

If the prompt is tagged security-related → route to my cybersecurity model

Everything else → fall back to qwen2.5:14b

Rules could match things like:

keywords in the prompt

prompt length

attached file types

tags or workspace metadata

or even a small intent/classifier model that decides what the request is about before routing.

Why this would be huge for local setups

A lot of us running local Ollama stacks on consumer GPUs end up with multiple specialized models, a coder model, a vision model, maybe a domain-specific model.

Right now we have to manually switch workspaces depending on what we’re doing. It works, but it also breaks the flow a bit.

A router would let the right model automatically handle the prompt, so you can just ask the question and keep moving.

How it might work technically

Something like:

A lightweight intent classifier runs first (could even just be keyword matching or a tiny local model).

The router checks the defined rules.

The prompt is sent to the selected model.

Optionally show which model was used for transparency.

Why this would be powerful

This would make AnythingLLM feel less like just a UI for individual models and more like a true AI orchestration layer for local setups.

For power users running multiple specialized models, it would be a huge quality-of-life improvement.

Originally created by @PikkonMG on GitHub (Mar 15, 2026). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/5209 ### What would you like to see? Feature Request: Intelligent Model Router Really liking AnythingLLM. That said, I’d love to see AnythingLLM go a step further with something like an Intelligent Model Router built directly into the platform. The basic idea: instead of manually switching workspaces depending on the task, AnythingLLM could automatically send the prompt to the best model based on rules you define. Kind of like a traffic controller sitting in front of your Ollama stack. What I’m imagining A routing rules panel in Settings where you can define things like: If the prompt contains code → route to qwen2.5-coder:7b If the prompt includes an image → route to qwen3.5:9b If the prompt is tagged security-related → route to my cybersecurity model Everything else → fall back to qwen2.5:14b Rules could match things like: keywords in the prompt prompt length attached file types tags or workspace metadata or even a small intent/classifier model that decides what the request is about before routing. Why this would be huge for local setups A lot of us running local Ollama stacks on consumer GPUs end up with multiple specialized models, a coder model, a vision model, maybe a domain-specific model. Right now we have to manually switch workspaces depending on what we’re doing. It works, but it also breaks the flow a bit. A router would let the right model automatically handle the prompt, so you can just ask the question and keep moving. How it might work technically Something like: A lightweight intent classifier runs first (could even just be keyword matching or a tiny local model). The router checks the defined rules. The prompt is sent to the selected model. Optionally show which model was used for transparency. Why this would be powerful This would make AnythingLLM feel less like just a UI for individual models and more like a true AI orchestration layer for local setups. For power users running multiple specialized models, it would be a huge quality-of-life improvement.
yindo added the enhancementfeature request labels 2026-06-05 14:51:11 -04:00
yindo closed this issue 2026-06-05 14:51:12 -04:00
yindo commented 2026-06-05 14:51:12 -04:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Mar 16, 2026):

I think we have discussed something like this already in Discord, but we do plan to have something exactly like this and is a milestone V2 feature we plan to be working on April 15-24. We plan on having a ruleset generator you can specify with:

Calculated rules:

  • Context length, message count, keyword

Semantic Rules: (may be limited to 3?)

  • Can define a title/description for the semantic match of a ruleset based on a user prompt
  • This will work via model running on the instance via llama.cpp which will require a gpu-all flag for Docker.
  • On desktop this will be using a model powered by the built-in LLM runtime

Then an admin/owner can specify a ruleset to a workspace(s), every matched rule will output to a provider/model combination that will redirect the chat in-session without an issue since we have normalized all interfaces. If no result is found you stay on last selected model until another trigger is reached. Will will probably be debouncing this as well to prevent this from running every single chat and/or may be admin configurable just due to overhead & UX concerns.

So you can actually expect this soon. We already have the internal tooling for it and across our benchmarks and test-cases we are seeing an 96% routing accuracy for chat history+prompt to intended route with ~100-400ms route response when on GPU.

So this is something that we already are working on and is 100% on v2 roadmap👍

<!-- gh-comment-id:4068839506 --> @timothycarambat commented on GitHub (Mar 16, 2026): I think we have discussed something like this already in Discord, but we do plan to have something exactly like this and is a milestone V2 feature we plan to be working on April 15-24. We plan on having a ruleset generator you can specify with: Calculated rules: - Context length, message count, keyword Semantic Rules: (may be limited to 3?) - Can define a title/description for the semantic match of a ruleset based on a user prompt - This will work via model running on the instance via `llama.cpp` which will require a gpu-all flag for Docker. - On desktop this will be using a model powered by the built-in LLM runtime Then an admin/owner can specify a ruleset to a workspace(s), every matched rule will output to a provider/model combination that will redirect the chat in-session without an issue since we have normalized all interfaces. If no result is found you stay on last selected model until another trigger is reached. Will will probably be debouncing this as well to prevent this from running every single chat and/or may be admin configurable just due to overhead & UX concerns. So you can actually expect this soon. We already have the internal tooling for it and across our benchmarks and test-cases we are seeing an 96% routing accuracy for chat history+prompt to intended route with ~100-400ms route response when on GPU. So this is something that we already are working on and is 100% on v2 roadmap👍
yindo commented 2026-06-05 14:51:12 -04:00
Author
Owner
Copy Link
Copy Source

@PikkonMG commented on GitHub (Mar 20, 2026):

Here is something i've been messing with locally, routes cloud and locally depending on what you set it up to do. https://github.com/FaqFirebase/Nexus-Orchestrator/

<!-- gh-comment-id:4099745543 --> @PikkonMG commented on GitHub (Mar 20, 2026): Here is something i've been messing with locally, routes cloud and locally depending on what you set it up to do. https://github.com/FaqFirebase/Nexus-Orchestrator/
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label enhancement feature request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#4967
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?