Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #1430] Query always provides answers regardless of whether document context is found. #912

New Issue
Closed
opened 2026-02-22 18:22:10 -05:00 by yindo · 5 comments
yindo commented 2026-02-22 18:22:10 -05:00
Owner
Copy Link
Copy Source

Originally created by @Freffles on GitHub (May 17, 2024).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/1430

Using Ollama LL (phi3) and chat mode set to "Query". Query will provide answers only if document context is found.

Using ChromaDB with Ollama embeddings (nomic-embed-text)

Asking a question that is obviously outside the document context (i.e. What is a prime number) rather than getting the specified query mode refusal response, I get an answer to my question with citations that point to document context that are clearly incorrect.

So, this is two issues. 1. I should be getting a query mode refusal and 2. the citations given have no relationship to the answer provided.

image

FWIW, get exactly the same behavior using System default LLM with Ollama embeddings.

Originally created by @Freffles on GitHub (May 17, 2024). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/1430 Using Ollama LL (phi3) and chat mode set to "Query". Query will provide answers only if document context is found. Using ChromaDB with Ollama embeddings (nomic-embed-text) Asking a question that is obviously outside the document context (i.e. What is a prime number) rather than getting the specified query mode refusal response, I get an answer to my question with citations that point to document context that are clearly incorrect. So, this is two issues. 1. I should be getting a query mode refusal and 2. the citations given have no relationship to the answer provided. ![image](https://github.com/Mintplex-Labs/anything-llm/assets/122719748/72b8e712-016a-46cd-8684-866ba2561a87) FWIW, get exactly the same behavior using System default LLM with Ollama embeddings.
yindo closed this issue 2026-02-22 18:22:10 -05:00
yindo commented 2026-02-22 18:22:10 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (May 17, 2024):

You should likely modify the Document Similarity Threshold since that is likely why you are getting results for a search even though the prompt is irrelevant. What is the score reporting for those chunks when you view the citations? The score is below each chunk

@timothycarambat commented on GitHub (May 17, 2024): You should likely modify the `Document Similarity Threshold` since that is likely why you are getting results for a search even though the prompt is irrelevant. What is the score reporting for those chunks when you view the citations? The score is below each chunk
yindo commented 2026-02-22 18:22:10 -05:00
Author
Owner
Copy Link
Copy Source

@Freffles commented on GitHub (May 17, 2024):

I've deleted all of that and going back to basics. Just maiting for llama2 to download and I will give it a try again using "System Default" LLM with Anything LLM embeddings and Lance DB. I have Max Context set to 10 and Document Similarity Threshold set to High for this run. Will post the update when I have run it with this setup.

@Freffles commented on GitHub (May 17, 2024): I've deleted all of that and going back to basics. Just maiting for llama2 to download and I will give it a try again using "System Default" LLM with Anything LLM embeddings and Lance DB. I have Max Context set to 10 and Document Similarity Threshold set to High for this run. Will post the update when I have run it with this setup.
yindo commented 2026-02-22 18:22:10 -05:00
Author
Owner
Copy Link
Copy Source

@Freffles commented on GitHub (May 17, 2024):

I did get it to work properly with all the defaults (LLM, Embeddings and VectorDB) but I had to reload my data (from the git repo) because deleting the vectors wasn't enough.

If I may make a suggestion, it would be nice to be able to enter a similarity threshold in numeric form as well as have the low, medium and high options.

@Freffles commented on GitHub (May 17, 2024): I did get it to work properly with all the defaults (LLM, Embeddings and VectorDB) but I had to reload my data (from the git repo) because deleting the vectors wasn't enough. If I may make a suggestion, it would be nice to be able to enter a similarity threshold in numeric form as well as have the low, medium and high options.
yindo commented 2026-02-22 18:22:10 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (May 17, 2024):

@Freffles I agree, there should be some custom option. The reason that we have those "pre-defined" stops is because it prevents people from playing with it too much since it's a very black-box kind of toggle. It can be useful but I was initially worried about people accidentally foot-gunning themselves due to bad configs

@timothycarambat commented on GitHub (May 17, 2024): @Freffles I agree, there should be some custom option. The reason that we have those "pre-defined" stops is because it prevents people from playing with it too much since it's a very black-box kind of toggle. It _can_ be useful but I was initially worried about people accidentally foot-gunning themselves due to bad configs
yindo commented 2026-02-22 18:22:11 -05:00
Author
Owner
Copy Link
Copy Source

@Freffles commented on GitHub (May 17, 2024):

Should have added this but before. Regarding score, I found that things that should have been a direct hit were returning scores less than 60. I could not get anything returned when I set to high. Maybe I need to play around with the chunk size.

@Freffles commented on GitHub (May 17, 2024): Should have added this but before. Regarding score, I found that things that should have been a direct hit were returning scores less than 60. I could not get anything returned when I set to high. Maybe I need to play around with the chunk size.
yindo changed title from Query always provides answers regardless of whether document context is found. to [GH-ISSUE #1430] Query always provides answers regardless of whether document context is found. 2026-06-05 14:37:53 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#912
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?