Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #3570] [FEAT]: Configurable Delay Between Embedding Requests #2303

New Issue
Closed
opened 2026-02-22 18:29:05 -05:00 by yindo · 4 comments
yindo commented 2026-02-22 18:29:05 -05:00
Owner
Copy Link
Copy Source

Originally created by @RuiU on GitHub (Mar 31, 2025).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/3570

What would you like to see?

I'm currently using the experimental Gemini Embeddings model via the generic OpenAI API in AnythingLLM Desktop. The model has a rate limit of 5/10 RPM. Even after setting the max concurrent chunks to 1, I sometimes still encounter error 429 when uploading some PDF files (sent one by one every few minutes, with each file being less than 1MB). The same files can be embedded smoothly using Gemini API text-embedding-004.

[429 RESOURCE_EXHAUSTED You've exceeded the rate limit. You are sending too many requests per minute with the free tier Gemini API. Ensure you're within the model's rate limit. Request a quota increase if needed]

Would it be possible to add a configuration option that allows users to set a delay between embedding requests?

Originally created by @RuiU on GitHub (Mar 31, 2025). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/3570 ### What would you like to see? I'm currently using the [experimental Gemini Embeddings model](https://developers.googleblog.com/en/gemini-embedding-text-model-now-available-gemini-api/) via the generic OpenAI API in AnythingLLM Desktop. The model has a rate limit of 5/10 RPM. Even after setting the max concurrent chunks to 1, I sometimes still encounter error 429 when uploading some PDF files (sent one by one every few minutes, with each file being less than 1MB). The same files can be embedded smoothly using Gemini API text-embedding-004. [429 RESOURCE_EXHAUSTED You've exceeded the rate limit. You are sending too many requests per minute with the free tier Gemini API. Ensure you're within the model's rate limit. Request a quota increase if needed] Would it be possible to add a configuration option that allows users to set a delay between embedding requests?
yindo added the enhancementfeature request labels 2026-02-22 18:29:05 -05:00
yindo closed this issue 2026-02-22 18:29:05 -05:00
yindo commented 2026-02-22 18:29:05 -05:00
Author
Owner
Copy Link
Copy Source

@chaserhkj commented on GitHub (Aug 20, 2025):

This is a very relevant request. I am trying to run embedding with llama.cpp backend which provides a generic OpenAI API as well. Even with concurrent chunks to 1 I still can not reliably embed anything, AnythingLLM is just making requests way too fast for llama.cpp backend to keep up even if everything is sequential.

@chaserhkj commented on GitHub (Aug 20, 2025): This is a very relevant request. I am trying to run embedding with llama.cpp backend which provides a generic OpenAI API as well. Even with concurrent chunks to 1 I still can not reliably embed anything, AnythingLLM is just making requests way too fast for llama.cpp backend to keep up even if everything is sequential.
yindo commented 2026-02-22 18:29:05 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Aug 20, 2025):

In theory, would some arbitrary waitFor(timeout) if some ENV is manually set help you out here? Could set it in some ms value just to throttle the request throughput?

For gemini for example:
https://github.com/Mintplex-Labs/anything-llm/blob/bb7d65f0eb6bebdc8465d66881a42011d4a3c2e5/server/utils/EmbeddingEngines/gemini/index.js#L52-L113

Just being able to add a conditionally applied wait timer at the end of each batch - this would obviously slow down the embedder process, but it would allow an API cooldown

@timothycarambat commented on GitHub (Aug 20, 2025): In theory, would some arbitrary `waitFor(timeout)` if some ENV is manually set help you out here? Could set it in some `ms` value just to throttle the request throughput? For gemini for example: https://github.com/Mintplex-Labs/anything-llm/blob/bb7d65f0eb6bebdc8465d66881a42011d4a3c2e5/server/utils/EmbeddingEngines/gemini/index.js#L52-L113 Just being able to add a conditionally applied wait timer at the end of each batch - this would obviously slow down the embedder process, but it would allow an API cooldown
yindo commented 2026-02-22 18:29:05 -05:00
Author
Owner
Copy Link
Copy Source

@chaserhkj commented on GitHub (Aug 20, 2025):

Yeah I think a conditional wait would definitely suffice for this issue - I don't really think this warrants any complex strategy to cater to the needs of the API on other end, especially in my use case.

@chaserhkj commented on GitHub (Aug 20, 2025): Yeah I think a conditional wait would definitely suffice for this issue - I don't really think this warrants any complex strategy to cater to the needs of the API on other end, especially in my use case.
yindo commented 2026-02-22 18:29:05 -05:00
Author
Owner
Copy Link
Copy Source

@chaserhkj commented on GitHub (Aug 20, 2025):

Actually I can wrap up a quick PR for the generic OpenAI backend - doesn't look too complicated.

@chaserhkj commented on GitHub (Aug 20, 2025): Actually I can wrap up a quick PR for the generic OpenAI backend - doesn't look too complicated.
yindo changed title from [FEAT]: Configurable Delay Between Embedding Requests to [GH-ISSUE #3570] [FEAT]: Configurable Delay Between Embedding Requests 2026-06-05 14:45:46 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label enhancement feature request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#2303
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?