Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #5681] OpenAI and Anthropic provider calls should use explicit timeouts and retry/backoff #5231

New Issue
Closed
opened 2026-06-05 14:52:48 -04:00 by yindo · 2 comments
yindo commented 2026-06-05 14:52:48 -04:00
Owner
Copy Link
Copy Source

Originally created by @michaelmanly on GitHub (May 22, 2026).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/5681

What would you like to see?

Support/document OpenAI-compatible gateway base URLs for production reliability

Issue body

## Question

Would maintainers be open to a small PR that documents or improves support for OpenAI-compatible gateway base URLs?

Some production users route LLM calls through a gateway layer so they can manage:

- retries
- fallback routing
- request tracking
- provider timeout handling
- debugging failed provider calls

For example, an OpenAI-compatible client can usually point at a gateway by changing only the API key and base URL:

```ts
import OpenAI from "openai";

const openai = new OpenAI({
  apiKey: process.env.BADGR_API_KEY,
  baseURL: "https://api.aibadgr.com/v1",
});

Why I’m asking

I noticed a few LLM provider paths where reliability behavior may currently depend on the underlying provider/client defaults:

  • server/utils/AiProviders/openAi/index.js

    • OpenAI requests appear to be issued directly without an explicit timeout or abort budget
    • completion calls appear to rely on direct provider errors without a shared retry/backoff policy
    • error rethrows may drop useful provider metadata such as status code, headers, request IDs, or retry hints

  • server/utils/AiProviders/anthropic/index.js

    • Anthropic requests appear to run without an explicit timeout or abort budget
    • transient 429 / 5xx provider errors may fail immediately
    • one error path appears to return the error object as textResponse, which could be treated like a successful model response upstream

  • server/utils/chats/apiChatHandler.js

    • chat execution appears to use one resolved provider per request without an optional runtime fallback chain if that provider is unavailable

  • server/utils/helpers/chat/LLMPerformanceMonitor.js

    • performance tracking appears to capture duration/token metrics, but not full upstream failure taxonomy such as provider status, retry count, fallback step, timeout reason, or request correlation ID

These may already be handled elsewhere, so I wanted to ask before opening a larger PR.

Possible small PR options

Happy to follow whatever maintainers prefer:

  1. a small docs PR showing how to configure an OpenAI-compatible gateway base URL
  2. a small config improvement for custom OpenAI-compatible base URLs
  3. a small timeout/retry improvement in one provider path
  4. improved structured error metadata for provider failures
  5. no change, if this is outside the project scope

The goal would be optional production reliability support without changing default behavior.

Originally created by @michaelmanly on GitHub (May 22, 2026). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/5681 ### What would you like to see? ```md Support/document OpenAI-compatible gateway base URLs for production reliability ``` **Issue body** ````md ## Question Would maintainers be open to a small PR that documents or improves support for OpenAI-compatible gateway base URLs? Some production users route LLM calls through a gateway layer so they can manage: - retries - fallback routing - request tracking - provider timeout handling - debugging failed provider calls For example, an OpenAI-compatible client can usually point at a gateway by changing only the API key and base URL: ```ts import OpenAI from "openai"; const openai = new OpenAI({ apiKey: process.env.BADGR_API_KEY, baseURL: "https://api.aibadgr.com/v1", }); ```` ## Why I’m asking I noticed a few LLM provider paths where reliability behavior may currently depend on the underlying provider/client defaults: * `server/utils/AiProviders/openAi/index.js` * OpenAI requests appear to be issued directly without an explicit timeout or abort budget * completion calls appear to rely on direct provider errors without a shared retry/backoff policy * error rethrows may drop useful provider metadata such as status code, headers, request IDs, or retry hints * `server/utils/AiProviders/anthropic/index.js` * Anthropic requests appear to run without an explicit timeout or abort budget * transient `429` / `5xx` provider errors may fail immediately * one error path appears to return the error object as `textResponse`, which could be treated like a successful model response upstream * `server/utils/chats/apiChatHandler.js` * chat execution appears to use one resolved provider per request without an optional runtime fallback chain if that provider is unavailable * `server/utils/helpers/chat/LLMPerformanceMonitor.js` * performance tracking appears to capture duration/token metrics, but not full upstream failure taxonomy such as provider status, retry count, fallback step, timeout reason, or request correlation ID These may already be handled elsewhere, so I wanted to ask before opening a larger PR. ## Possible small PR options Happy to follow whatever maintainers prefer: 1. a small docs PR showing how to configure an OpenAI-compatible gateway base URL 2. a small config improvement for custom OpenAI-compatible base URLs 3. a small timeout/retry improvement in one provider path 4. improved structured error metadata for provider failures 5. no change, if this is outside the project scope The goal would be optional production reliability support without changing default behavior.
yindo added the enhancementfeature request labels 2026-06-05 14:52:48 -04:00
yindo closed this issue 2026-06-05 14:52:49 -04:00
yindo commented 2026-06-05 14:52:49 -04:00
Author
Owner
Copy Link
Copy Source

@markseibal-png commented on GitHub (May 22, 2026):

Probably OpenAI timeout handling. Running npx ai-doctor might point out missing timeout guards and unprotected requests.

<!-- gh-comment-id:4516384371 --> @markseibal-png commented on GitHub (May 22, 2026): Probably OpenAI timeout handling. Running `npx ai-doctor` might point out missing timeout guards and unprotected requests.
yindo commented 2026-06-05 14:52:49 -04:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (May 22, 2026):

will be resolved by #5637
duplicate of #5631

<!-- gh-comment-id:4520767788 --> @timothycarambat commented on GitHub (May 22, 2026): will be resolved by #5637 duplicate of #5631
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label enhancement feature request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#5231
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?