Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #3682] [BUG]: System prompt fallback message not working for Llama3-ChatQA-1.5-8B model #2376

New Issue
Closed
opened 2026-02-22 18:29:24 -05:00 by yindo · 1 comment
yindo commented 2026-02-22 18:29:24 -05:00
Owner
Copy Link
Copy Source

Originally created by @johnec2 on GitHub (Apr 20, 2025).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/3682

How are you running AnythingLLM?

Local development

What happened?

We have changed the model from Llama3.2:3b to the Llama3-ChatQA-1.5-8B model as we work with QA pairs.

The system prompt defined as:
You are a helpful question-answering assistant. Use the provided QA pairs from the document context to answer the user's query accurately. Only use information from the context to form your response. If the context does not contain enough information to answer the query or the query is unclear, respond with: "I'm sorry, I don't have enough information to answer that question accurately. Please provide more details or clarify your request."
Context:
{retrieved_context}

When queried, if the answer is not found in the uploaded file does not respond with a fallback message displays a numeric value(0 or 1...). Then next if the query or question from the file is asked then also the response is returning as numeric value only.

How to restrict to retrieve fallback message with our custom message.

Are there known steps to reproduce?

No response

Originally created by @johnec2 on GitHub (Apr 20, 2025). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/3682 ### How are you running AnythingLLM? Local development ### What happened? We have changed the model from Llama3.2:3b to the Llama3-ChatQA-1.5-8B model as we work with QA pairs. The system prompt defined as: You are a helpful question-answering assistant. Use the provided QA pairs from the document context to answer the user's query accurately. Only use information from the context to form your response. If the context does not contain enough information to answer the query or the query is unclear, respond with: "I'm sorry, I don't have enough information to answer that question accurately. Please provide more details or clarify your request." Context: {retrieved_context} When queried, if the answer is not found in the uploaded file does not respond with a fallback message displays a numeric value(0 or 1...). Then next if the query or question from the file is asked then also the response is returning as numeric value only. How to restrict to retrieve fallback message with our custom message. ### Are there known steps to reproduce? _No response_
yindo added the possible bug label 2026-02-22 18:29:24 -05:00
yindo closed this issue 2026-02-22 18:29:24 -05:00
yindo commented 2026-02-22 18:29:25 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Apr 20, 2025):

I'm sorry, I don't have enough information to answer that question accurately. Please provide more details or clarify your request

It is not foolproof and amounts to a standard refusal/LLM hallunication. You are expecting a non-deterministic system to be deterministic and when it comes to instruction following the model matters. If you are using a heavily quantized, or small param model this is not unexpected behavior. Since you have provided no description of what model you are using, how it is running or otherwise, it is almost certainly the case.

That being said, what you are doing is actually directly supported by the product. Set the chat mode to query and edit the "Query refusal response" in the workspace settings. Then, when no documents are in the context window to answer a query it will give a static formal response.

Using a system prompt for what you are attempting to do is a well-known behavior of an LLM system. Even if using a model like Anthropic, you might get some compliance but a simple query asking it to ignore that instruction would easily circumvent this kind of rule following.

@timothycarambat commented on GitHub (Apr 20, 2025): > I'm sorry, I don't have enough information to answer that question accurately. Please provide more details or clarify your request It is not foolproof and amounts to a standard refusal/LLM hallunication. You are expecting a non-deterministic system to be deterministic and when it comes to instruction following the model _matters_. If you are using a heavily quantized, or small param model this is not unexpected behavior. Since you have provided no description of what model you are using, how it is running or otherwise, it is almost certainly the case. That being said, what you are doing is actually directly supported by the product. Set the chat mode to query and edit the "Query refusal response" in the workspace settings. Then, when no documents are in the context window to answer a query it will give a static formal response. Using a system prompt for what you are attempting to do is a well-known behavior of an LLM system. Even if using a model like Anthropic, you might get some compliance but a simple query asking it to ignore that instruction would easily circumvent this kind of rule following.
yindo changed title from [BUG]: System prompt fallback message not working for Llama3-ChatQA-1.5-8B model to [GH-ISSUE #3682] [BUG]: System prompt fallback message not working for Llama3-ChatQA-1.5-8B model 2026-06-05 14:46:13 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label possible bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#2376
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?