Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-21 09:35:22 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #5720] Add OrcaRouter as an OpenAI-compatible LLM provider #5255

New Issue
Open
opened 2026-06-05 14:52:57 -04:00 by yindo · 0 comments
yindo commented 2026-06-05 14:52:57 -04:00
Owner
Copy Link
Copy Source

Originally created by @jinhaosong-source on GitHub (May 29, 2026).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/5720

What would you like to see?

Motivation

AnythingLLM users on cloud models configure one provider at a time and pay each provider's list price, with no easy way to compare models or optimize cost across them. (This is about the remote-model path — local stays local.)

Proposal

Add OrcaRouter (https://orcarouter.ai) as a selectable LLM provider. It's an OpenAI-compatible gateway that routes each request across providers:

  • 200+ models behind one endpoint, one API key (BYOK or unified billing) — instead of switching provider configs to try different models
  • Adaptive routing to the cheapest model that still meets the latency + capability bar (~40% less spend at the same answer quality on heavier workloads, per OrcaRouter's own benchmarks)
  • <1ms routing overhead
  • 0% routing markup

Since AnythingLLM already supports OpenAI-compatible / "Generic OpenAI" providers, this might be as small as a preset provider entry pointing at OrcaRouter's base URL — I'd confirm the right shape with a maintainer.

What I can contribute

I'm on the OrcaRouter side and happy to do the work: open the PR, add docs, and maintain the integration.

Open questions

  1. What's the right way to add a provider — a preset entry, or does it need code?
  2. Open to it being a built-in/listed provider, or would you prefer users add it via Generic OpenAI config?
  3. Who owns provider integrations — happy to coordinate.
Originally created by @jinhaosong-source on GitHub (May 29, 2026). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/5720 ### What would you like to see? ## Motivation AnythingLLM users on cloud models configure one provider at a time and pay each provider's list price, with no easy way to compare models or optimize cost across them. (This is about the remote-model path — local stays local.) ## Proposal Add OrcaRouter (https://orcarouter.ai) as a selectable LLM provider. It's an OpenAI-compatible gateway that routes each request across providers: - 200+ models behind one endpoint, one API key (BYOK or unified billing) — instead of switching provider configs to try different models - Adaptive routing to the cheapest model that still meets the latency + capability bar (~40% less spend at the same answer quality on heavier workloads, per OrcaRouter's own benchmarks) - <1ms routing overhead - 0% routing markup Since AnythingLLM already supports OpenAI-compatible / "Generic OpenAI" providers, this might be as small as a preset provider entry pointing at OrcaRouter's base URL — I'd confirm the right shape with a maintainer. ## What I can contribute I'm on the OrcaRouter side and happy to do the work: open the PR, add docs, and maintain the integration. ## Open questions 1. What's the right way to add a provider — a preset entry, or does it need code? 2. Open to it being a built-in/listed provider, or would you prefer users add it via Generic OpenAI config? 3. Who owns provider integrations — happy to coordinate.
yindo added the enhancementIntegration Requestfeature request labels 2026-06-05 14:52:57 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label Integration Request enhancement feature request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#5255
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?