Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #2369] Undesirable Responses with Custom Prompts in AnythingLLM using OpenAI. #1545

New Issue
Closed
opened 2026-02-22 18:25:21 -05:00 by yindo · 2 comments
yindo commented 2026-02-22 18:25:21 -05:00
Owner
Copy Link
Copy Source

Originally created by @KavAndritz on GitHub (Sep 25, 2024).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/2369

How are you running AnythingLLM?

Local development

What happened?

Current Behaviour:
When setting custom prompts in the chat settings of AnythingLLM, I have configured the agent to respond with "Out of scope" for questions that are not relevant to the database. However, the agent is incorrectly responding with "Out of scope" to all questions, regardless of their relevance to the database.

Expected Behavior:
The agent should only respond with "Out of scope" when the question is genuinely unrelated to the database. For relevant questions, the agent should provide appropriate responses based on the database content.

Are there known steps to reproduce?

• Set Up the Database:
Configure the database in AnythingLLM using the OpenAI GPT-4 model. Issue also tested with GPT-4o, GPT-4-mini, and
GPT-4-turbo.
• Customize Chat Settings:
Set custom prompts in the chat settings. Tested with 4-5 different prompt variations.
• Test the Agent:
Ask the agent both relevant and irrelevant questions related to the use case (presses). The agent responds with "Out of
scope" for all questions, regardless of relevance.

Originally created by @KavAndritz on GitHub (Sep 25, 2024). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/2369 ### How are you running AnythingLLM? Local development ### What happened? Current Behaviour: When setting custom prompts in the chat settings of AnythingLLM, I have configured the agent to respond with "Out of scope" for questions that are not relevant to the database. However, the agent is incorrectly responding with "Out of scope" to all questions, regardless of their relevance to the database. Expected Behavior: The agent should only respond with "Out of scope" when the question is genuinely unrelated to the database. For relevant questions, the agent should provide appropriate responses based on the database content. ### Are there known steps to reproduce? • **Set Up the Database:** Configure the database in AnythingLLM using the OpenAI GPT-4 model. Issue also tested with GPT-4o, GPT-4-mini, and GPT-4-turbo. • **Customize Chat Settings:** Set custom prompts in the chat settings. Tested with 4-5 different prompt variations. • **Test the Agent:** Ask the agent both relevant and irrelevant questions related to the use case (presses). The agent responds with "Out of scope" for all questions, regardless of relevance.
yindo added the question label 2026-02-22 18:25:21 -05:00
yindo closed this issue 2026-02-22 18:25:21 -05:00
yindo commented 2026-02-22 18:25:21 -05:00
Author
Owner
Copy Link
Copy Source

@tranmh commented on GitHub (Sep 27, 2024):

Hi!

Why was it labeled from "possible bug" to "question". Did you reproduce the bug already and are saying "work-as-designed"?

Thx.

@tranmh commented on GitHub (Sep 27, 2024): Hi! Why was it labeled from "possible bug" to "question". Did you reproduce the bug already and are saying "work-as-designed"? Thx.
yindo commented 2026-02-22 18:25:21 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Sep 27, 2024):

Because this is not a bug in the software sense. You labeled this as a bug, but it's just how you set up the system and are using it to seemingly do something it probably is not explicitly designed to do but the issue is not really clear to me what the intentions are.

Ask the agent both relevant and irrelevant questions related to the use case (presses). The agent responds with "Out of
scope" for all questions, regardless of relevance.

this is basically a use case that is not working for you. There is not bug or feature request here to action on in the software sense. if a model does not work the way you want it is on the model side and could be a hallucination, refusal, or otherwise and there is not amount of code to write to make a model behave in a strict way.

You are using agent to basically refuse a query based on what is in a database (using the SQL connector?) but how can it know what is in the DB in advance without querying it?

What are some examples of what you want to accomplish and how are you guiding the LLM to understand queries

@timothycarambat commented on GitHub (Sep 27, 2024): Because this is not a bug in the software sense. You labeled this as a bug, but it's just how you set up the system and are using it to seemingly do something it probably is not explicitly designed to do but the issue is not really clear to me what the intentions are. > Ask the agent both relevant and irrelevant questions related to the use case (presses). The agent responds with "Out of scope" for all questions, regardless of relevance. this is basically a use case that is not working for you. There is not bug or feature request here to action on in the software sense. if a model does not work the way you want it is on the _model side_ and could be a hallucination, refusal, or otherwise and there is not amount of code to write to make a model behave in a strict way. You are using agent to basically refuse a query based on what is in a database (using the SQL connector?) but how can it know what is in the DB in advance without querying it? What are some examples of what you want to accomplish and how are you guiding the LLM to understand queries
yindo changed title from Undesirable Responses with Custom Prompts in AnythingLLM using OpenAI. to [GH-ISSUE #2369] Undesirable Responses with Custom Prompts in AnythingLLM using OpenAI. 2026-06-05 14:41:19 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label question
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#1545
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?