Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #2709] [BUG]: Token should not be exceeded (Azure OpenAI, GPT-4o, Chat Model Token Limit 128,000) but llm ist generating response forever #1743

New Issue
Closed
opened 2026-02-22 18:26:19 -05:00 by yindo · 1 comment
yindo commented 2026-02-22 18:26:19 -05:00
Owner
Copy Link
Copy Source

Originally created by @DediCATeD88 on GitHub (Nov 24, 2024).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/2709

How are you running AnythingLLM?

Docker (local)

What happened?

We do use Whisper (Whisper-WebUI) for transcription services. We do not use the integrated transcribe feature because users cannot upload mp3, wav, mp4 and the integrated transcribing cannot be configured for Condition On Previous Text disabled and Repetition Penalty 2 or 3.

We are facing the following issue with AnythingLLM, Docker, Azure OpenAI, GPT-4o, Chat Model Token Limit 128,000:

If we use the chat window for long transcriptions (text) and post them directly into the chat, the chat window stays at generating response forever. No error whatsoever. We can stop generating response, yes. But we should not exceed the token limit of 128,000.

Even if we split into multiple parts same thing.

We tried to remove all CR-LF/LF same thing.

Some examples:
Full: Characters 89219 words 16336 sentences 81 paragraphs 7240 Spaces 9096
Split Part 1: Characters 15604 words 2743 sentences 21 paragraphs 886 Spaces 1858

Does anyone have a clue what's going on there?

Are there known steps to reproduce?

Paste long text directly into chat window and send to llm.

Originally created by @DediCATeD88 on GitHub (Nov 24, 2024). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/2709 ### How are you running AnythingLLM? Docker (local) ### What happened? We do use Whisper (Whisper-WebUI) for transcription services. We do not use the integrated transcribe feature because users cannot upload mp3, wav, mp4 and the integrated transcribing cannot be configured for Condition On Previous Text disabled and Repetition Penalty 2 or 3. We are facing the following issue with AnythingLLM, Docker, Azure OpenAI, GPT-4o, Chat Model Token Limit 128,000: If we use the chat window for long transcriptions (text) and post them directly into the chat, the chat window stays at generating response forever. No error whatsoever. We can stop generating response, yes. But we should not exceed the token limit of 128,000. Even if we split into multiple parts same thing. We tried to remove all CR-LF/LF same thing. Some examples: Full: Characters 89219 words 16336 sentences 81 paragraphs 7240 Spaces 9096 Split Part 1: Characters 15604 words 2743 sentences 21 paragraphs 886 Spaces 1858 Does anyone have a clue what's going on there? ### Are there known steps to reproduce? Paste long text directly into chat window and send to llm.
yindo added the possible bug label 2026-02-22 18:26:19 -05:00
yindo closed this issue 2026-02-22 18:26:19 -05:00
yindo commented 2026-02-22 18:26:19 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Nov 24, 2024):

Are you saying the LLM never returns a response when pasting in long text? This is likely because your are hitting your OpenAI Azure rate limits, which have a pretty low tokens/sec rate and you probably need to raise them. Its not weird for more tokens to result in a longer time to first token, but it should return something in a reasonable time.

@timothycarambat commented on GitHub (Nov 24, 2024): Are you saying the LLM never returns a response when pasting in long text? This is likely because your are hitting your OpenAI Azure rate limits, which have a pretty low tokens/sec rate and you probably need to raise them. Its not weird for more tokens to result in a longer time to first token, but it should return something in a reasonable time.
yindo changed title from [BUG]: Token should not be exceeded (Azure OpenAI, GPT-4o, Chat Model Token Limit 128,000) but llm ist generating response forever to [GH-ISSUE #2709] [BUG]: Token should not be exceeded (Azure OpenAI, GPT-4o, Chat Model Token Limit 128,000) but llm ist generating response forever 2026-06-05 14:42:25 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label possible bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#1743
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?