Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #2452] [BUG]: Huggingface response truncated #1596

New Issue
Closed
opened 2026-02-22 18:25:35 -05:00 by yindo · 6 comments
yindo commented 2026-02-22 18:25:35 -05:00
Owner
Copy Link
Copy Source

Originally created by @gcleaves on GitHub (Oct 9, 2024).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/2452

How are you running AnythingLLM?

Docker (remote machine)

What happened?

I've embedded the chat widget in a web page. When asking a question via the chat embed, the response is truncated. When asking the same question within the AnywhereLLM workspace, the response is not truncated.

I got 374 characters in the embed widget.

Are there known steps to reproduce?

No response

Originally created by @gcleaves on GitHub (Oct 9, 2024). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/2452 ### How are you running AnythingLLM? Docker (remote machine) ### What happened? I've embedded the chat widget in a web page. When asking a question via the chat embed, the response is truncated. When asking the same question within the AnywhereLLM workspace, the response is not truncated. I got 374 characters in the embed widget. ### Are there known steps to reproduce? _No response_
yindo added the possible bugneeds info / can't replicate labels 2026-02-22 18:25:35 -05:00
yindo closed this issue 2026-02-22 18:25:35 -05:00
yindo commented 2026-02-22 18:25:35 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Oct 9, 2024):

We will need more information on this.

  • Do you see any client disconnections from the chat embed widget from the network inspector
  • Does the message saved in the chat embed history show the truncated amount as well - or is it fully formed?
  • What provider and model are you using?
  • Do you have associated logs from the Docker container as well (if errors persist)
@timothycarambat commented on GitHub (Oct 9, 2024): We will need more information on this. - Do you see any client disconnections from the chat embed widget from the network inspector - Does the message saved in the chat embed history show the truncated amount as well - or is it fully formed? - What provider and model are you using? - Do you have associated logs from the Docker container as well (if errors persist)
yindo commented 2026-02-22 18:25:35 -05:00
Author
Owner
Copy Link
Copy Source

@gcleaves commented on GitHub (Oct 9, 2024):

My reference to the embedded chat widget was a red herring. (I've change the title of the issue.) Even when I use the standard interface to dockerized AnythingLLM the response is truncated. My desktop install running on my Mac M1, on the other hand, does NOT truncate the data.

  • Docker instance
    • LLM provider is Hugginface with dedicated inference endpoint running mistral-7b-instruct-v0-3-byo.
      • GPU · Nvidia A10G · 1x GPU · 24 GB
      • Quantization: none
      • Max Input Length (per Query): 2048
      • Max Number of Tokens (per Query): 4096 | Doubled to 8192 but no change in truncation.
      • Max Batch Prefill Tokens: 8192
    • Default vector DB
      • Max Context Snippets: 4
    • Default embedder
  • Local instance
    • LLM provider is the included AnythingLLM provider running Mistral7B.
    • Default vector DB
      • Max Context Snippets: 4
    • Default embedder

In both instances I have uploaded 8 word documents which contain the answer to my truncated response.

I don't understand much about LLMs (shocker). How do the document embeddings created by AnythingLLM get to the model? Are they included somehow in the prompt? Might my document embeddings not be reaching the Huggingface model in full? Although it seems more like a token response limit that is hampering my Docker instance.

@gcleaves commented on GitHub (Oct 9, 2024): My reference to the embedded chat widget was a red herring. (I've change the title of the issue.) Even when I use the standard interface to dockerized AnythingLLM the response is truncated. My desktop install running on my Mac M1, on the other hand, does NOT truncate the data. - Docker instance - LLM provider is Hugginface with dedicated inference endpoint running **mistral-7b-instruct-v0-3-byo**. - GPU · Nvidia A10G · 1x GPU · 24 GB - Quantization: none - Max Input Length (per Query): 2048 - Max Number of Tokens (per Query): 4096 | Doubled to 8192 but no change in truncation. - Max Batch Prefill Tokens: 8192 - Default vector DB - Max Context Snippets: 4 - Default embedder - Local instance - LLM provider is the included AnythingLLM provider running **Mistral7B**. - Default vector DB - Max Context Snippets: 4 - Default embedder In both instances I have uploaded 8 word documents which contain the answer to my truncated response. I don't understand much about LLMs (shocker). How do the document embeddings created by AnythingLLM get to the model? Are they included somehow in the prompt? Might my document embeddings not be reaching the Huggingface model in full? Although it seems more like a token response limit that is hampering my Docker instance.
yindo commented 2026-02-22 18:25:35 -05:00
Author
Owner
Copy Link
Copy Source

@gcleaves commented on GitHub (Oct 9, 2024):

This looks to be a Huggingface problem:

https://discuss.huggingface.co/t/text-generation-response-truncation/53155
https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3/discussions/23

There's reference to a max_new_tokens parameter which may or may not solve the problem.

@gcleaves commented on GitHub (Oct 9, 2024): This looks to be a Huggingface problem: https://discuss.huggingface.co/t/text-generation-response-truncation/53155 https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3/discussions/23 There's reference to a max_new_tokens parameter which may or may not solve the problem.
yindo commented 2026-02-22 18:25:35 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Oct 9, 2024):

Ah, that makes more sense. max_tokens or max_new_tokens should resolve the truncation, there is little or no documentation I can find currently that lets us pass like -1 to that property to allow as many tokens out as needed.

We may have to make an input for this so that we can easily have this property controlled. That being said it is very weird the desktop does not truncate.

Does truncation happen on an empty workspace on the first message? There is usually a max_tokens parameter that is the maximum of input+output tokens and that could cause truncation at random points in a conversation since we inject system prompt + context + history - so a seemingly small query can be many more tokens on the backend - thus resulting in truncation.

@timothycarambat commented on GitHub (Oct 9, 2024): Ah, that makes more sense. `max_tokens` or `max_new_tokens` _should_ resolve the truncation, there is little or no documentation I can find currently that lets us pass like `-1` to that property to allow as many tokens out as needed. We may have to make an input for this so that we can easily have this property controlled. That being said it is very weird the desktop does not truncate. Does truncation happen on an empty workspace on the first message? There is usually a `max_tokens` parameter that is the maximum of **input+output** tokens and that could cause truncation at random points in a conversation since we inject system prompt + context + history - so a seemingly small query can be many more tokens on the backend - thus resulting in truncation.
yindo commented 2026-02-22 18:25:35 -05:00
Author
Owner
Copy Link
Copy Source

@gcleaves commented on GitHub (Oct 10, 2024):

Could the reason truncation doesn't occur on desktop be because the desktop uses AnythingLLM/Ollama as the LLM provider, while Docker relies on HF?

@gcleaves commented on GitHub (Oct 10, 2024): Could the reason truncation doesn't occur on desktop be because the desktop uses AnythingLLM/Ollama as the LLM provider, while Docker relies on HF?
yindo commented 2026-02-22 18:25:36 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Oct 11, 2024):

Could the reason truncation doesn't occur on desktop be because the desktop uses AnythingLLM/Ollama as the LLM provider, while Docker relies on HF?

The desktop simply uses Ollama as the default. But you can use whatever you want as usual. The HF providers are the same on desktop/docker so if you used the same HF credentials on Desktop you should still get the same behavior. Its the same code, but its certainly specific to the HF connector for the LLM and the issue should be present in both Desktop and Docker

@timothycarambat commented on GitHub (Oct 11, 2024): > Could the reason truncation doesn't occur on desktop be because the desktop uses AnythingLLM/Ollama as the LLM provider, while Docker relies on HF? The desktop simply _uses_ Ollama as the default. But you can use whatever you want as usual. The HF providers are the same on desktop/docker so if you used the same HF credentials on Desktop you should still get the same behavior. Its the same code, but its certainly specific to the HF connector for the LLM and the issue should be present in both Desktop and Docker
yindo changed title from [BUG]: Huggingface response truncated to [GH-ISSUE #2452] [BUG]: Huggingface response truncated 2026-06-05 14:41:38 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label needs info / can't replicate possible bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#1596
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?