Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #2562] [BUG]: mistral-nemo-minitron-8b-instruct results in endless loop (LMStudio LLM Provider) / How to disable system prompt? #1659

New Issue
Closed
opened 2026-02-22 18:25:55 -05:00 by yindo · 2 comments
yindo commented 2026-02-22 18:25:55 -05:00
Owner
Copy Link
Copy Source

Originally created by @joomgallerytestit on GitHub (Oct 31, 2024).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/2562

How are you running AnythingLLM?

AnythingLLM desktop app

What happened?

Hi,

when I use LMStudio as an LLM provider and load ”
lmstudio-community/mistral-nemo-minitron-8b-instruct” in LMStudio,
I end up in an endless loop in the chat in AnythingLLM.

Because every few sentences there is then ‘<extra_id_1>User’ and a question and then <extra_id_1>Answer...

What is this “extra_id_1” all about? How can I control or disable it in AnythingLLM?

Apparently, at least one problem has nothing to do with AnythingLLM, but with the prompt format of XY:
“ Prompt format <extra_id_0>System
{system_prompt}

<extra_id_1>User
{prompt}
<extra_id_1>Assistant
”
Unfortunately, I am not yet sufficiently familiar with the subject matter to know how to solve the problem I have described.

A similar problem seems to be that some LLMs simply do not support a system prompt and throw an error in the AnythingLLM chat if a system prompt is stored in the AnythingLLM workspace. Even if the field for the system prompt is empty. How can I disable system prompt passing in AnythingLLM?

Thanks in advance and best regards
Joomgallerytestit

Originally created by @joomgallerytestit on GitHub (Oct 31, 2024). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/2562 ### How are you running AnythingLLM? AnythingLLM desktop app ### What happened? Hi, when I use LMStudio as an LLM provider and load ” **lmstudio-community/mistral-nemo-minitron-8b-instruct**” in LMStudio, I end up in an endless loop in the chat in AnythingLLM. Because every few sentences there is then ‘**<extra_id_1>User**’ and a question and then **<extra_id_1>Answer**... What is this “extra_id_1” all about? How can I control or disable it in AnythingLLM? Apparently, at least one problem has nothing to do with AnythingLLM, but with the prompt format of XY: “ Prompt format <extra_id_0>System {system_prompt} <extra_id_1>User {prompt} <extra_id_1>Assistant ” Unfortunately, I am not yet sufficiently familiar with the subject matter to know how to solve the problem I have described. A similar problem seems to be that some LLMs simply do not support a system prompt and throw an error in the AnythingLLM chat if a system prompt is stored in the AnythingLLM workspace. Even if the field for the system prompt is empty. How can I disable system prompt passing in AnythingLLM? Thanks in advance and best regards Joomgallerytestit
yindo added the possible bug label 2026-02-22 18:25:55 -05:00
yindo closed this issue 2026-02-22 18:25:55 -05:00
yindo commented 2026-02-22 18:25:55 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Oct 31, 2024):

This should be fixable by the chat template specification in LMStudio, no? Since this is a model-specific issue. Also, what quantization are you running in LMStudio because behaviors like outputting its own stop/chat tokens are a huge problem when a small model (8B) is over-quantized.

With even removal of a system prompt it would likely still be broken and spitting out pure nonsense

@timothycarambat commented on GitHub (Oct 31, 2024): This should be fixable by the chat template specification in LMStudio, no? Since this is a model-specific issue. Also, what quantization are you running in LMStudio because behaviors like outputting its own stop/chat tokens are a huge problem when a small model (8B) is over-quantized. With even removal of a system prompt it would likely still be broken and spitting out pure nonsense
yindo commented 2026-02-22 18:25:55 -05:00
Author
Owner
Copy Link
Copy Source

@joomgallerytestit commented on GitHub (Nov 1, 2024):

Thanks for your answer!

I want to build a good local system for RAG!

Since “Chat Mode” is set to Query, quantization Q4 should not play a major role.

In my tests I had several LLMs that threw an error because they did not support system prompts. So it would be important to be able to deactivate this in AnythingLLM.

@joomgallerytestit commented on GitHub (Nov 1, 2024): Thanks for your answer! I want to build a good local system for RAG! Since “Chat Mode” is set to Query, quantization **Q4** should not play a major role. In my tests I had several LLMs that threw an error because they did **not support system prompts**. So it would be important to be able to deactivate this in AnythingLLM.
yindo changed title from [BUG]: mistral-nemo-minitron-8b-instruct results in endless loop (LMStudio LLM Provider) / How to disable system prompt? to [GH-ISSUE #2562] [BUG]: mistral-nemo-minitron-8b-instruct results in endless loop (LMStudio LLM Provider) / How to disable system prompt? 2026-06-05 14:41:59 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label possible bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#1659
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?