Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #2254] Cannot get consistent results from "Chat with document" #1471

New Issue
Closed
opened 2026-02-22 18:24:58 -05:00 by yindo · 6 comments
yindo commented 2026-02-22 18:24:58 -05:00
Owner
Copy Link
Copy Source

Originally created by @mtomas7 on GitHub (Sep 9, 2024).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/2254

Originally assigned to: @timothycarambat on GitHub.

Setup:

  • Windows Desktop App 1.6.4
  • Open new work space, set the default settings with local Qwen2 7b instruct fp16 model for Chat and Agents
  • -Chat Mode: Chat
  • Use Paperclip icon or drag and drop the PDF document into chat window.
  • Icon indicates that Document was uploaded.
  • Document is not listed under main Workspace document upload screen (I assume, this screen is specifically dedicated to RAG)
  • I ask some questions, but answers sometimes would totally miss the mark, it's like parts of the short 3 page document were not included.
  • After first interaction the Document upload icon disappears and it is not clear if that document is still part of the conversation or not.
  • @agent cannot see that document was dropped into conversation (perhaps by design)
  • Even after uploading document via main Workspace upload screen (I assume RAG) (checked that vector DB was updated) and using prompt from the documentation:
  • @agent what documents can you see?
  • Agent would reply:
  • "As an AI language model, I don’t have the capability to “see” or access files directly from your device or any external storage because I operate within a text-based interface without direct file access permissions. My role is to provide information and assistance based on my training data and the questions you ask me. If you need help managing documents or understanding how to organize them, I’d be glad to offer advice or guidance!"

So first, there is no clear distinction if drop-in docs and RAG upload docs are treated in a same manner and if drop-in docs are added to the Workspace vector DB.

Compared the drop-in doc feature with LM Studio, when dropping file in, it evaluates if there is enough memory to load whole document and if yes, it does that. Answer quality is much better. Perhaps it would be possible to implement this memory check in AnythingLLM too?

Originally created by @mtomas7 on GitHub (Sep 9, 2024). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/2254 Originally assigned to: @timothycarambat on GitHub. Setup: - Windows Desktop App 1.6.4 - Open new work space, set the default settings with local Qwen2 7b instruct fp16 model for Chat and Agents - -Chat Mode: Chat - Use Paperclip icon or drag and drop the PDF document into chat window. - Icon indicates that Document was uploaded. - Document is not listed under main Workspace document upload screen (I assume, this screen is specifically dedicated to RAG) - I ask some questions, but answers sometimes would totally miss the mark, it's like parts of the short 3 page document were not included. - After first interaction the Document upload icon disappears and it is not clear if that document is still part of the conversation or not. - @agent cannot see that document was dropped into conversation (perhaps by design) - Even after uploading document via main Workspace upload screen (I assume RAG) (checked that vector DB was updated) and using prompt from the documentation: - `@agent what documents can you see?` - Agent would reply: - "As an AI language model, I don’t have the capability to “see” or access files directly from your device or any external storage because I operate within a text-based interface without direct file access permissions. My role is to provide information and assistance based on my training data and the questions you ask me. If you need help managing documents or understanding how to organize them, I’d be glad to offer advice or guidance!" So first, there is no clear distinction if drop-in docs and RAG upload docs are treated in a same manner and if drop-in docs are added to the Workspace vector DB. Compared the drop-in doc feature with LM Studio, when dropping file in, it evaluates if there is enough memory to load whole document and if yes, it does that. Answer quality is much better. Perhaps it would be possible to implement this memory check in AnythingLLM too?
yindo added the question label 2026-02-22 18:24:58 -05:00
yindo closed this issue 2026-02-22 18:24:58 -05:00
yindo commented 2026-02-22 18:24:59 -05:00
Author
Owner
Copy Link
Copy Source

@ValfarDeveloper commented on GitHub (Sep 12, 2024):

Having the same here, using Milvus, could be related to the model embeddings?

@ValfarDeveloper commented on GitHub (Sep 12, 2024): Having the same here, using Milvus, could be related to the model embeddings?
yindo commented 2026-02-22 18:24:59 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Sep 12, 2024):

Compared the drop-in doc feature with LM Studio, when dropping file in, it evaluates if there is enough memory to load whole document and if yes, it does that. Answer quality is much better. Perhaps it would be possible to implement this memory check in AnythingLLM too?

This is because it injects the entire document into the context window. We have RAG with a vector db built in. LMStudio does not. The reason you get better results is because the whole document is in context!

We preserve context windows by default with RAG - but you can always replicate the same behavior with document pinning. Use that if the entire document needs to be injected. Otherwise, you can mess with the controls in this doc to get better RAG results.

https://docs.anythingllm.com/llm-not-using-my-docs

@timothycarambat commented on GitHub (Sep 12, 2024): > Compared the drop-in doc feature with LM Studio, when dropping file in, it evaluates if there is enough memory to load whole document and if yes, it does that. Answer quality is much better. Perhaps it would be possible to implement this memory check in AnythingLLM too? This is because it injects the _entire_ document into the context window. We have RAG with a vector db built in. LMStudio does not. The reason you get better results is because the whole document is in context! We preserve context windows by default with RAG - but you can always replicate the same behavior with [document pinning](https://docs.anythingllm.com/llm-not-using-my-docs#document-pinning). Use that if the entire document needs to be injected. Otherwise, you can mess with the controls in this doc to get better RAG results. https://docs.anythingllm.com/llm-not-using-my-docs
yindo commented 2026-02-22 18:24:59 -05:00
Author
Owner
Copy Link
Copy Source

@mtomas7 commented on GitHub (Sep 12, 2024):

@timothycarambat Thank you for the clarification! I'm trying to look at this from a practical perspective - an average user will not care so much about the internal process, eg. which embedding method was used, but will really care about the quality of the answer. I'm preparing to present this for new users and I know that some of them already have a habit to drop file into conversation as it is done in ChatGPT... Would it make a sense to do this pre-loading evaluation and suggest user to use document pinning if file is not very large. This way user will act as it is already "standard way" to interact with AI model and will get good results. Or perhaps doc. pinning should be always used if document is dropped into chat window? In case of a large file or multiple files, user would be guided to create RAG instance.

Thank you!

@mtomas7 commented on GitHub (Sep 12, 2024): @timothycarambat Thank you for the clarification! I'm trying to look at this from a practical perspective - an average user will not care so much about the internal process, eg. which embedding method was used, but will really care about the quality of the answer. I'm preparing to present this for new users and I know that some of them already have a habit to drop file into conversation as it is done in ChatGPT... Would it make a sense to do this pre-loading evaluation and suggest user to use document pinning if file is not very large. This way user will act as it is already "standard way" to interact with AI model and will get good results. Or perhaps doc. pinning should be always used if document is dropped into chat window? In case of a large file or multiple files, user would be guided to create RAG instance. Thank you!
yindo commented 2026-02-22 18:24:59 -05:00
Author
Owner
Copy Link
Copy Source

@mtomas7 commented on GitHub (Sep 12, 2024):

And not to forget, looks like these points could be potentially improved in the app:

  • After first interaction the Document upload icon disappears and it is not clear if that document is still part of the conversation or not.
  • agent cannot see that document was dropped into conversation
  • Even after uploading document via main Workspace upload screen (I assume RAG) (checked that vector DB was updated) and using prompt from the documentation:
    @agent what documents can you see?
  • Agent would reply:
    "As an AI language model, I don’t have the capability to “see” or access files directly from your device or any external storage because I operate within a text-based interface without direct file access permissions. My role is to provide information and assistance based on my training data and the questions you ask me. If you need help managing documents or understanding how to organize them, I’d be glad to offer advice or guidance!"
@mtomas7 commented on GitHub (Sep 12, 2024): And not to forget, looks like these points could be potentially improved in the app: - After first interaction the Document upload icon disappears and it is not clear if that document is still part of the conversation or not. - agent cannot see that document was dropped into conversation - Even after uploading document via main Workspace upload screen (I assume RAG) (checked that vector DB was updated) and using prompt from the documentation: @agent what documents can you see? - Agent would reply: "As an AI language model, I don’t have the capability to “see” or access files directly from your device or any external storage because I operate within a text-based interface without direct file access permissions. My role is to provide information and assistance based on my training data and the questions you ask me. If you need help managing documents or understanding how to organize them, I’d be glad to offer advice or guidance!"
yindo commented 2026-02-22 18:24:59 -05:00
Author
Owner
Copy Link
Copy Source

@designfusionjb commented on GitHub (Feb 19, 2025):

I also have this problem. Was it ever solved?

@designfusionjb commented on GitHub (Feb 19, 2025): I also have this problem. Was it ever solved?
yindo commented 2026-02-22 18:24:59 -05:00
Author
Owner
Copy Link
Copy Source

@the-cyber-specialist commented on GitHub (Jan 7, 2026):

I have exactly this problem also. I have documents in the workspace but I want to add new revisions of a file that I am iterating over with the LLM so it does not need to be part of the larger Workspace context. However, the models all say they cannot work with files. I will prompt with something like:

given the attached tab-delimited file (named “20260101 NIST CSF (Mistral-Small-Q5_K_XL_export.csv”), how could I go about deriving a set of security engineering requirements that are traceable back to their NIST CSF categories or subcategories?

I get:

I’m unable to directly access or output rows from attached files. However, I can help you understand the content of the files you’ve provided or assist with any questions you have about them.

The AnythingLLM docs lead me to believe that I am trying to go about this the correct way. I am running Mistral-small in LM Studio with 80k context so i know everything fits in to the context.

@the-cyber-specialist commented on GitHub (Jan 7, 2026): I have exactly this problem also. I have documents in the workspace but I want to add new revisions of a file that I am iterating over with the LLM so it does not need to be part of the larger Workspace context. However, the models all say they cannot work with files. I will prompt with something like: > given the attached tab-delimited file (named “20260101 NIST CSF (Mistral-Small-Q5_K_XL_export.csv”), how could I go about deriving a set of security engineering requirements that are traceable back to their NIST CSF categories or subcategories? I get: > I’m unable to directly access or output rows from attached files. However, I can help you understand the content of the files you’ve provided or assist with any questions you have about them. The AnythingLLM docs lead me to believe that I am trying to go about this the correct way. I am running Mistral-small in LM Studio with 80k context so i know everything fits in to the context.
yindo changed title from Cannot get consistent results from "Chat with document" to [GH-ISSUE #2254] Cannot get consistent results from "Chat with document" 2026-06-05 14:40:57 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label question
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#1471
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?