Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #1550] [BUG]: Chat answer always starts with "According to the context provided" when system prompt says not to use that phrase #1002

New Issue
Closed
opened 2026-02-22 18:22:36 -05:00 by yindo · 2 comments
yindo commented 2026-02-22 18:22:36 -05:00
Owner
Copy Link
Copy Source

Originally created by @thebaldgeek on GitHub (May 27, 2024).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/1550

How are you running AnythingLLM?

Docker (local)

What happened?

Every single chat answer starts with the phrase "According to the context provided..." (or phrases along those lines).
My team is so sick and tired of it its become a mocking point of the project.

image

I have tested many system prompts and am now of the belief that the prompt is totally ignored.

Using local Windows 11 Ollama Llama3.0, Docker Desktop (pulled 18 hours ago - cant find a version in docker). Default and built in database etc.
Answers (beyond the starting phrase) are from the trained docs and are exactly as expected.

Are there known steps to reproduce?

Train the system on some docs and ask questions.
Its a very simple process thanks to the amazing work the anything-llm team has done.

Originally created by @thebaldgeek on GitHub (May 27, 2024). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/1550 ### How are you running AnythingLLM? Docker (local) ### What happened? Every single chat answer starts with the phrase "According to the context provided..." (or phrases along those lines). My team is so sick and tired of it its become a mocking point of the project. ![image](https://github.com/Mintplex-Labs/anything-llm/assets/7385116/a1ee910d-7c6b-4a71-8a41-e303d8382e5e) I have tested many system prompts and am now of the belief that the prompt is totally ignored. Using local Windows 11 Ollama Llama3.0, Docker Desktop (pulled 18 hours ago - cant find a version in docker). Default and built in database etc. Answers (beyond the starting phrase) are from the trained docs and are exactly as expected. ### Are there known steps to reproduce? Train the system on some docs and ask questions. Its a very simple process thanks to the amazing work the anything-llm team has done.
yindo added the possible bug label 2026-02-22 18:22:36 -05:00
yindo closed this issue 2026-02-22 18:22:36 -05:00
yindo commented 2026-02-22 18:22:37 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (May 27, 2024):

Unfortunately, this is not really a "bug" and is simply a facet of LLM response generation. Not all models are equally good at what can be referred to as "instruction following". This obviously makes putting guardrails in a prompt to do something especially problematic with smaller, quantized, open source models - which are particularly impact by this.

Some things that have helped those trying to get a better prompt-following model but are only interested in OSS models:

  • Expanding hardware resource to allow large context window (if limited)
  • Swapping to the same model, but a higher bit quantization. Ollama default is 4-bit, the typical max is 8-bit. This becomes a substantial factor with smaller models.

Llama3 8B__Q4 -> small model, medium compression -> overall pretty high compression, but super portable
Llama3 70B__Q4 -> massive model, medium compression -> overall pretty OK compression, but still resource intensive

I am assuming you are using LLama3 8B and the default quantization of Q4, I would try Q8 first and see if results improve

@timothycarambat commented on GitHub (May 27, 2024): Unfortunately, this is not really a "bug" and is simply a facet of LLM response generation. Not all models are equally good at what can be referred to as "instruction following". This obviously makes putting guardrails in a prompt to do something especially problematic with smaller, quantized, open source models - which are **particularly** impact by this. Some things that have helped those trying to get a better prompt-following model but are only interested in OSS models: - Expanding hardware resource to allow large context window (if limited) - Swapping to the _same_ model, but a higher bit quantization. Ollama default is 4-bit, the typical max is 8-bit. This becomes a **substantial** factor with smaller models. Llama3 8B__Q4 -> small model, medium compression -> overall pretty high compression, but super portable Llama3 70B__Q4 -> massive model, medium compression -> overall pretty OK compression, but still resource intensive I am assuming you are using LLama3 8B and the default quantization of Q4, I would try Q8 first and see if results improve
yindo commented 2026-02-22 18:22:37 -05:00
Author
Owner
Copy Link
Copy Source

@thebaldgeek commented on GitHub (May 27, 2024):

Currently using Llama3 70B.

I have tested dozens of prompts and there simply no impact on the responses.
I am very very happy with all the answers and citations provided... Just having every. Single. Response. Start with the same phrase is watering down the overall project.

Sounds like there is no way to use the system prompt in this application.
I will start looking at using the API and that way I can just remove the start of every message up to the first comma via the API code.

Thanks for all your work on this and the quick reply to this issue.

@thebaldgeek commented on GitHub (May 27, 2024): Currently using Llama3 70B. I have tested dozens of prompts and there simply no impact on the responses. I am very very happy with all the answers and citations provided... Just having every. Single. Response. Start with the same phrase is watering down the overall project. Sounds like there is no way to use the system prompt in this application. I will start looking at using the API and that way I can just remove the start of every message up to the first comma via the API code. Thanks for all your work on this and the quick reply to this issue.
yindo changed title from [BUG]: Chat answer always starts with "According to the context provided" when system prompt says not to use that phrase to [GH-ISSUE #1550] [BUG]: Chat answer always starts with "According to the context provided" when system prompt says not to use that phrase 2026-06-05 14:38:25 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label possible bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#1002
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?