Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #525] [BUG] Handle special LLM tokens from unsanitized model responses #308

New Issue
Closed
opened 2026-02-22 18:18:53 -05:00 by yindo · 3 comments
yindo commented 2026-02-22 18:18:53 -05:00
Owner
Copy Link
Copy Source

Originally created by @Joly0 on GitHub (Jan 4, 2024).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/525

Originally assigned to: @timothycarambat on GitHub.

Hey, i have a working ollama instance on my server with the falcon 7b model. I installed anything-llm and connected it to ollama, which worked without an issue. No i have this problem:
grafik

In the logs of anything-llm i can see this:

Error: The text contains a special token that is not allowed: <|endoftext|>
    at _Tiktoken.encode (/app/server/node_modules/js-tiktoken/dist/index.cjs:120:15)
    at TokenManager.countFromString (/app/server/utils/helpers/tiktoken.js:30:33)
    at /app/server/utils/helpers/tiktoken.js:44:28
    at Array.reduce (<anonymous>)
    at TokenManager.statsFrom (/app/server/utils/helpers/tiktoken.js:43:39)
    at messageArrayCompressor (/app/server/utils/helpers/chat/index.js:54:20)
    at OllamaAILLM.compressMessages (/app/server/utils/AiProviders/ollama/index.js:178:18)
    at OllamaAILLM.streamChat (/app/server/utils/AiProviders/ollama/index.js:129:33)
    at streamEmptyEmbeddingChat (/app/server/utils/chats/stream.js:181:39)
    at async streamChatWithWorkspace (/app/server/utils/chats/stream.js:54:12)

I am not sure, why "<|endoftext|>" is added to the end of the answer of ollama, but that is an issue for anything-llm

Originally created by @Joly0 on GitHub (Jan 4, 2024). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/525 Originally assigned to: @timothycarambat on GitHub. Hey, i have a working ollama instance on my server with the falcon 7b model. I installed anything-llm and connected it to ollama, which worked without an issue. No i have this problem: ![grafik](https://github.com/Mintplex-Labs/anything-llm/assets/13993216/c1717314-eae1-49bc-9e05-7685ff714e15) In the logs of anything-llm i can see this: ``` Error: The text contains a special token that is not allowed: <|endoftext|> at _Tiktoken.encode (/app/server/node_modules/js-tiktoken/dist/index.cjs:120:15) at TokenManager.countFromString (/app/server/utils/helpers/tiktoken.js:30:33) at /app/server/utils/helpers/tiktoken.js:44:28 at Array.reduce (<anonymous>) at TokenManager.statsFrom (/app/server/utils/helpers/tiktoken.js:43:39) at messageArrayCompressor (/app/server/utils/helpers/chat/index.js:54:20) at OllamaAILLM.compressMessages (/app/server/utils/AiProviders/ollama/index.js:178:18) at OllamaAILLM.streamChat (/app/server/utils/AiProviders/ollama/index.js:129:33) at streamEmptyEmbeddingChat (/app/server/utils/chats/stream.js:181:39) at async streamChatWithWorkspace (/app/server/utils/chats/stream.js:54:12) ``` I am not sure, why "<|endoftext|>" is added to the end of the answer of ollama, but that is an issue for anything-llm
yindo added the bugIntegration Request labels 2026-02-22 18:18:53 -05:00
yindo closed this issue 2026-02-22 18:18:53 -05:00
yindo commented 2026-02-22 18:18:53 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Jan 4, 2024):

Are you using a quantized model that is < 5-bit quant? The fact that it includes its <|endtext|> in its response usually indicates a model that is rough around the edges.

This is still a bug and should be handled properly, but if the model is small or heavily quantized these odd behaviours happen more often.

A simple fix is send /reset in chat to reset the thread to remove that special token.

@timothycarambat commented on GitHub (Jan 4, 2024): Are you using a quantized model that is < 5-bit quant? The fact that it includes its `<|endtext|>` in its response usually indicates a model that is rough around the edges. This is still a bug and should be handled properly, but if the model is small or heavily quantized these odd behaviours happen more often. A simple fix is send `/reset` in chat to reset the thread to remove that special token.
yindo commented 2026-02-22 18:18:53 -05:00
Author
Owner
Copy Link
Copy Source

@Joly0 commented on GitHub (Jan 4, 2024):

Thanks for the answer. According to this https://ollama.ai/library/falcon:7b it should be 4-bit quant

@Joly0 commented on GitHub (Jan 4, 2024): Thanks for the answer. According to this https://ollama.ai/library/falcon:7b it should be 4-bit quant
yindo commented 2026-02-22 18:18:53 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Jan 4, 2024):

@Joly0 then getting weird tokens back in the response would be par for the course 😞 . I had this same issue when running a quantized model that is less than 5-bit. if possible I would try one step up, but will be fixing this issue regardless

@timothycarambat commented on GitHub (Jan 4, 2024): @Joly0 then getting weird tokens back in the response would be par for the course 😞 . I had this same issue when running a quantized model that is less than 5-bit. _if_ possible I would try one step up, but will be fixing this issue regardless
yindo changed title from [BUG] Handle special LLM tokens from unsanitized model responses to [GH-ISSUE #525] [BUG] Handle special LLM tokens from unsanitized model responses 2026-06-05 14:34:37 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label Integration Request bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#308
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?