Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #4030] [QUESTION]: Weird RAG Context Issue from API with 1.8.2 #2563

New Issue
Closed
opened 2026-02-22 18:30:16 -05:00 by yindo · 9 comments
yindo commented 2026-02-22 18:30:16 -05:00
Owner
Copy Link
Copy Source

Originally created by @digitalassassins on GitHub (Jun 22, 2025).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/4030

How are you running AnythingLLM?

AnythingLLM desktop app

What happened?

I have developed a plugin for external software using the AnythingLLM API to upload documents, embed, pin, query the document and then remove the documents one by one. Everything was working great. I noticed there was an update 1.8.2, so I downloaded it and installed it. Tested the plugin with 1.8.2 and was getting an error where no context was being passed to the Chat AI Model. I was testing with a reasoning model, so it explains the context used.

I had logs from the day before the update, where it was running great, the answers were correct, and the responses were perfect every time. So I thought I would try to uninstall 1.8.2 and reinstall it in case there was an error with the installation. I also deleted the anythingllm-desktop folder in %appdata%/roaming just to start fresh.

After the install, the first query from the API retrieves context, but every consecutive request after that first query results in no context being passed to the chat. This is the thinking response I get from the workspace. The document is embedded and the document is pinned.

------------------------------------------------------------
############             Thinking:              ############
------------------------------------------------------------
 Okay, so I need to figure out how to analyze this PDF file named '3828ER3018U_Trio - AIdegae1I.pdf' and extract
 specific information about the author. The user has asked me to look into a field called "Author" and return 
the result in JSON format. If the information isn't found, I should just put "Not Available".  First, I'm thinking 
about how to approach this. Since I can't actually view or open the PDF file, I have to rely on whatever metadata 
or text is available through other means. Maybe the filename itself has some clues? Let's look at it: 
'3828ER3018U_Trio - AIdegae1I.pdf'. Hmm, that seems like a random string of numbers and letters with 
"Trio" and "AIdegae1I". I don't see any obvious author names there.  Next, I should consider the structure of PDFs.
 Typically, PDFs can have embedded metadata such as title, author, subject, keywords, etc., which are stored
 in the document's properties. If this PDF has such metadata, maybe the author is listed there. But without 
access to the file, I can't check that. I'm more than willing to help if they provide the text or any specific details 
from the book. Another thought: sometimes authors' names might appear in the text of the document itself, 
like on a title page or in an introduction. Again, without being able to read through the content, it's hard to 
say if the author is mentioned there.  I could also think about the filename structure. Sometimes filenames
include the author's name, but in this case, "Trio" and "AIdegae1I" don't seem like typical author names. 
Maybe "AIdegae1I" is a username or some identifier rather than an actual name.  
Wait, maybe "3828ER3018U" refers to something else, like a project code or a document reference number.
That doesn't help with the author either.  So, considering all these possibilities and since I can't access 
the PDF's content or metadata, it seems that extracting the author information isn't feasible right now. 
Therefore, the most accurate response would be to indicate that the information is not available.

If I reinstall and start fresh again with 1.8.2. I get one more query where it works, and the AI says:

using context 0 it mentions *** and context 1 mentions ***

And then it breaks again for the next query on the exact same document. I still had the 1.8.1 installer for Windows in my download folder, so I rolled back the install and installed the previous version. Everything is working as expected. So unsure if it is a problem with the API or a problem with the embedding model in the latest version.

Checked the Vector DB after embedding in 1.8.1, and it has 2960 vectors, but in 1.8.2 I'm getting 22 vectors for the same document. So looks like it isn't embedding the full document properly.

Just checked, and if I do it from within Anything LLM in 1.8.2 It doesn't show the citation, but in 1.8.1 it does. Is there any hidden temporary files somewhere I need to clear that might be conflicting with the latest version?

Are there known steps to reproduce?

No response

Originally created by @digitalassassins on GitHub (Jun 22, 2025). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/4030 ### How are you running AnythingLLM? AnythingLLM desktop app ### What happened? I have developed a plugin for external software using the AnythingLLM API to upload documents, embed, pin, query the document and then remove the documents one by one. Everything was working great. I noticed there was an update 1.8.2, so I downloaded it and installed it. Tested the plugin with 1.8.2 and was getting an error where no context was being passed to the Chat AI Model. I was testing with a reasoning model, so it explains the context used. I had logs from the day before the update, where it was running great, the answers were correct, and the responses were perfect every time. So I thought I would try to uninstall 1.8.2 and reinstall it in case there was an error with the installation. I also deleted the `anythingllm-desktop` folder in `%appdata%/roaming` just to start fresh. After the install, the first query from the API retrieves context, but every consecutive request after that first query results in no context being passed to the chat. This is the thinking response I get from the workspace. The document is embedded and the document is pinned. ``` ------------------------------------------------------------ ############ Thinking: ############ ------------------------------------------------------------ Okay, so I need to figure out how to analyze this PDF file named '3828ER3018U_Trio - AIdegae1I.pdf' and extract specific information about the author. The user has asked me to look into a field called "Author" and return the result in JSON format. If the information isn't found, I should just put "Not Available". First, I'm thinking about how to approach this. Since I can't actually view or open the PDF file, I have to rely on whatever metadata or text is available through other means. Maybe the filename itself has some clues? Let's look at it: '3828ER3018U_Trio - AIdegae1I.pdf'. Hmm, that seems like a random string of numbers and letters with "Trio" and "AIdegae1I". I don't see any obvious author names there. Next, I should consider the structure of PDFs. Typically, PDFs can have embedded metadata such as title, author, subject, keywords, etc., which are stored in the document's properties. If this PDF has such metadata, maybe the author is listed there. But without access to the file, I can't check that. I'm more than willing to help if they provide the text or any specific details from the book. Another thought: sometimes authors' names might appear in the text of the document itself, like on a title page or in an introduction. Again, without being able to read through the content, it's hard to say if the author is mentioned there. I could also think about the filename structure. Sometimes filenames include the author's name, but in this case, "Trio" and "AIdegae1I" don't seem like typical author names. Maybe "AIdegae1I" is a username or some identifier rather than an actual name. Wait, maybe "3828ER3018U" refers to something else, like a project code or a document reference number. That doesn't help with the author either. So, considering all these possibilities and since I can't access the PDF's content or metadata, it seems that extracting the author information isn't feasible right now. Therefore, the most accurate response would be to indicate that the information is not available. ``` If I reinstall and start fresh again with 1.8.2. I get one more query where it works, and the AI says: ``` using context 0 it mentions *** and context 1 mentions *** ``` And then it breaks again for the next query on the exact same document. I still had the 1.8.1 installer for Windows in my download folder, so I rolled back the install and installed the previous version. Everything is working as expected. So unsure if it is a problem with the API or a problem with the embedding model in the latest version. Checked the Vector DB after embedding in 1.8.1, and it has 2960 vectors, but in 1.8.2 I'm getting 22 vectors for the same document. So looks like it isn't embedding the full document properly. Just checked, and if I do it from within Anything LLM in 1.8.2 It doesn't show the citation, but in 1.8.1 it does. Is there any hidden temporary files somewhere I need to clear that might be conflicting with the latest version? ### Are there known steps to reproduce? _No response_
yindo added the questionneeds info / can't replicateinvestigating labels 2026-02-22 18:30:16 -05:00
yindo closed this issue 2026-02-22 18:30:16 -05:00
yindo commented 2026-02-22 18:30:16 -05:00
Author
Owner
Copy Link
Copy Source

@reekob commented on GitHub (Jun 23, 2025):

How are you retaining context of the chat for subsequent questions after the first one?

@reekob commented on GitHub (Jun 23, 2025): How are you retaining context of the chat for subsequent questions after the first one?
yindo commented 2026-02-22 18:30:16 -05:00
Author
Owner
Copy Link
Copy Source

@digitalassassins commented on GitHub (Jun 23, 2025):

I'm not retaining the context of the chat for each question. I get better results by either pinning the document. Then, asking individual query questions on that document. Or using Vector DB and letting it return results for that document for the specific question.

There is something going on between V1.8.1 and V1.8.2, as I'm seeing completely different results between the two versions using RAG. In v1.8.1, I could type the filename into the chat to reference an individual document in the workspace. But it seems now that isn't working as it was in v1.8.1. Not sure if that feature was intentionally stripped out of the API.

@digitalassassins commented on GitHub (Jun 23, 2025): I'm not retaining the context of the chat for each question. I get better results by either pinning the document. Then, asking individual query questions on that document. Or using Vector DB and letting it return results for that document for the specific question. There is something going on between V1.8.1 and V1.8.2, as I'm seeing completely different results between the two versions using RAG. In v1.8.1, I could type the filename into the chat to reference an individual document in the workspace. But it seems now that isn't working as it was in v1.8.1. Not sure if that feature was intentionally stripped out of the API.
yindo commented 2026-02-22 18:30:16 -05:00
Author
Owner
Copy Link
Copy Source

@digitalassassins commented on GitHub (Jun 23, 2025):

Ok, after some testing and checking Debug logs, it looks like Anything LLM, when pinning a document, seems to first query the Vector database, then realises the document is pinned. Then doesn't use the vector DB retrieval as it knows the document is pinned. Then sends the message to Ollama without the pinned document context. So the AI doesn't have any context to work with. If I don't pin the document, the AI gets context from the Vector DB and can answer the questions correctly with less accuracy.

Then I have my Workspace settings as follows:
Chat Mode: Query Temperature: 0 (Zero)

The Workspace is empty with 0 vectors stored in the database. I uploaded a user manual for a D-Link DWL-2100ap Wireless Router, pinned the document and said:

Summarise this document, if you don't know the answer or can't provide a summary, simply return "Not Available"

The AI responded,
This is a purchase receipt for a pink motorised Barbie car...

A) This mode is on Query, so the AI should only provide answers from the context in the document or refuse.
B) The Temperature is at zero, so the AI model shouldn't make things up; there has never been an upload of a purchase receipt for a pink Motorised Barbie car ever. The LanceDB has just been installed fresh and is empty.

The Implementation of document pinning in the RAG system in Anything LLM has gone from rock solid to a broken mess in one update.

I will finish the implementation of GPT4All and OpenWebUI in the plugin and come back to AnythingLLM further down the line, see if it has been fixed.

@digitalassassins commented on GitHub (Jun 23, 2025): Ok, after some testing and checking Debug logs, it looks like Anything LLM, when pinning a document, seems to first query the Vector database, then realises the document is pinned. Then doesn't use the vector DB retrieval as it knows the document is pinned. Then sends the message to Ollama without the pinned document context. So the AI doesn't have any context to work with. If I don't pin the document, the AI gets context from the Vector DB and can answer the questions correctly with less accuracy. Then I have my Workspace settings as follows: `Chat Mode: Query Temperature: 0 (Zero)` The Workspace is empty with 0 vectors stored in the database. I uploaded a user manual for a D-Link DWL-2100ap Wireless Router, pinned the document and said: `Summarise this document, if you don't know the answer or can't provide a summary, simply return "Not Available"` The AI responded, `This is a purchase receipt for a pink motorised Barbie car...` A) This mode is on `Query`, so the AI should only provide answers from the context in the document or refuse. B) The `Temperature` is at zero, so the AI model shouldn't make things up; there has never been an upload of a purchase receipt for a pink Motorised Barbie car ever. The LanceDB has just been installed fresh and is empty. The Implementation of document pinning in the RAG system in Anything LLM has gone from rock solid to a broken mess in one update. I will finish the implementation of GPT4All and OpenWebUI in the plugin and come back to AnythingLLM further down the line, see if it has been fixed.
yindo commented 2026-02-22 18:30:17 -05:00
Author
Owner
Copy Link
Copy Source

@digitalassassins commented on GitHub (Jun 23, 2025):

Ive also tested it in the chat, by chatting with a document within AnythingLLM to see if it's a problem with the API.
Ive uploaded a document to the workspace: file is '2602130.pdf': D-Link AirPlus Xtreme G™ DWL-2100AP Manual
Chat Mode: Query
Temperature: 0

Prompt:
using file ‘2602130.pdf’. In a field called Summary, please provide your own summary of the book or document. The summary should be clear, concise, and written in your own words. Do not reference the filename in the summary, as filenames can change and may not reflect the actual content. The Summary must be 4 paragraphs in length. If there is a literary story, do not provide any spoilers. Return only the required field. Return the response in JSON format, only provide the JSON as the Response. If you cannot find the information for the field, simply return Not Available. Use this structure: { “Summary”: “” }

Reply:

  "Summary": "This work explores the complex and often fraught relationship between humans and animals, specifically focusing on the historical and ongoing practices of domestication and animal husbandry. It argues that the seemingly straightforward narrative of human control over animals is, in reality, a deeply intertwined and reciprocal process, shaped by mutual adaptation and unintended consequences. The text examines how human societies have rationalized their use of animals for labor, food, and companionship, often employing philosophical and religious justifications that mask underlying power dynamics and environmental impacts. \n\nThe book delves into the evolution of animal domestication, tracing its origins and charting its impact on both human and animal populations. It highlights the significant ecological and genetic changes that have resulted from this long-term interaction, emphasizing the ways in which domesticated species have been fundamentally altered to serve human needs. The work also investigates the impact on wild animal populations and ecosystems, demonstrating the far-reaching consequences of human interventions in the natural world. \n\nBeyond the practical aspects of animal management, the text investigates the cultural and symbolic significance of animals in various societies throughout history. It explores how animals have been portrayed in art, literature, and mythology, often reflecting anxieties about human identity, control, and the boundaries between the human and animal realms. The book challenges readers to reconsider common assumptions about human dominance and to appreciate the intricate and often overlooked connections between humans and other species.\n\nUltimately, the work calls for a more ethical and sustainable approach to our interactions with animals. It encourages a deeper understanding of the ecological, cultural, and moral implications of animal use, advocating for practices that respect animal welfare and promote biodiversity. By acknowledging the profound and ongoing influence that animals have had on human civilization, the author hopes to foster a greater sense of responsibility for their well-being and the health of the planet."
}

Image

This is with a clean install using the built-in AnythingLLM for LLM provider, Gemma3 12B, Built-in Embedder, everything default apart from workspace, changed temp 0 and chat mode to query..

Original Document Uploaded to workspace:

2602130.pdf

The RAG system used to work perfectly in 1.8.1. Now it just hallucinates with every document you throw at it.
Deepseek was telling me the document was a purchase receipt for a pink motorised Barbie car 😂

No matter what model, embedder or vector DB I choose. It just Hallucinates..

I've tried with different Versions of Ollama, Swapping out LanceDB for Chroma DB etc..

@digitalassassins commented on GitHub (Jun 23, 2025): Ive also tested it in the chat, by chatting with a document within AnythingLLM to see if it's a problem with the API. Ive uploaded a document to the workspace: file is '2602130.pdf': D-Link AirPlus Xtreme G™ DWL-2100AP Manual **Chat Mode:** Query **Temperature:** 0 Prompt: ```using file ‘2602130.pdf’. In a field called Summary, please provide your own summary of the book or document. The summary should be clear, concise, and written in your own words. Do not reference the filename in the summary, as filenames can change and may not reflect the actual content. The Summary must be 4 paragraphs in length. If there is a literary story, do not provide any spoilers. Return only the required field. Return the response in JSON format, only provide the JSON as the Response. If you cannot find the information for the field, simply return Not Available. Use this structure: { “Summary”: “” }``` Reply: ```{ "Summary": "This work explores the complex and often fraught relationship between humans and animals, specifically focusing on the historical and ongoing practices of domestication and animal husbandry. It argues that the seemingly straightforward narrative of human control over animals is, in reality, a deeply intertwined and reciprocal process, shaped by mutual adaptation and unintended consequences. The text examines how human societies have rationalized their use of animals for labor, food, and companionship, often employing philosophical and religious justifications that mask underlying power dynamics and environmental impacts. \n\nThe book delves into the evolution of animal domestication, tracing its origins and charting its impact on both human and animal populations. It highlights the significant ecological and genetic changes that have resulted from this long-term interaction, emphasizing the ways in which domesticated species have been fundamentally altered to serve human needs. The work also investigates the impact on wild animal populations and ecosystems, demonstrating the far-reaching consequences of human interventions in the natural world. \n\nBeyond the practical aspects of animal management, the text investigates the cultural and symbolic significance of animals in various societies throughout history. It explores how animals have been portrayed in art, literature, and mythology, often reflecting anxieties about human identity, control, and the boundaries between the human and animal realms. The book challenges readers to reconsider common assumptions about human dominance and to appreciate the intricate and often overlooked connections between humans and other species.\n\nUltimately, the work calls for a more ethical and sustainable approach to our interactions with animals. It encourages a deeper understanding of the ecological, cultural, and moral implications of animal use, advocating for practices that respect animal welfare and promote biodiversity. By acknowledging the profound and ongoing influence that animals have had on human civilization, the author hopes to foster a greater sense of responsibility for their well-being and the health of the planet." } ``` ![Image](https://github.com/user-attachments/assets/3cc50941-e2ea-42f2-8d15-30f346f587dc) This is with a clean install using the built-in AnythingLLM for LLM provider, Gemma3 12B, Built-in Embedder, everything default apart from workspace, changed temp 0 and chat mode to query.. Original Document Uploaded to workspace: [2602130.pdf](https://github.com/user-attachments/files/20870476/2602130.pdf) The RAG system used to work perfectly in 1.8.1. Now it just hallucinates with every document you throw at it. Deepseek was telling me the document was a purchase receipt for a pink motorised Barbie car 😂 No matter what model, embedder or vector DB I choose. It just Hallucinates.. I've tried with different Versions of Ollama, Swapping out LanceDB for Chroma DB etc..
yindo commented 2026-02-22 18:30:17 -05:00
Author
Owner
Copy Link
Copy Source

@digitalassassins commented on GitHub (Jun 23, 2025):

So I thought O.k., what if i install the docker version to see if this is only a Windows Desktop version problem.

Fired up docker, ran the docker run command..

Left everything default..
Created a New Workspace.
Downloaded Gemma 12B model in Ollama 0.9.0.
Changed workspace:
Temperature: 0
Chat Mode: Query
Uploaded the same document as above for a D-Link Wireless Router, embedded it into workspace.

sent the same prompt by copy and paste, this is the response i got in a fresh docker install:

{
  "Summary": "This document details the development and implementation of a new system for managing and analyzing customer feedback. It outlines the motivations behind creating this system, which stemmed from a desire to improve customer satisfaction and gain deeper insights into customer needs and preferences. The existing feedback collection methods were deemed inefficient and lacked the analytical capabilities required for effective decision-making. The new system aims to centralize feedback data from various sources, automate the analysis process, and provide actionable insights to relevant teams. The core components of the system include a centralized feedback repository, a natural language processing (NLP) engine for sentiment analysis and topic extraction, and a dashboard for visualizing key metrics and trends. The document describes the architecture of the system, highlighting the integration of different modules and the flow of data. It also addresses the technical challenges encountered during development, such as ensuring data security and scalability. The selection of appropriate technologies and tools is justified based on their ability to meet the project’s requirements. Implementation of the system involved a phased approach, starting with a pilot program to test its functionality and gather user feedback. This iterative process allowed for continuous improvement and refinement of the system based on real-world usage. Training programs were also conducted to ensure that employees were proficient in using the new system. The document emphasizes the importance of user adoption and provides recommendations for promoting system usage throughout the organization. Ultimately, the new feedback management system is expected to significantly enhance the organization's ability to understand and respond to customer needs. By providing timely and actionable insights, the system will enable data-driven decision-making and contribute to improved customer satisfaction and loyalty. The document concludes with a discussion of potential future enhancements and expansion of the system to incorporate additional data sources and analytical capabilities."
}

Image

So its not just the windows install that is hallucinating..

@digitalassassins commented on GitHub (Jun 23, 2025): So I thought O.k., what if i install the docker version to see if this is only a Windows Desktop version problem. Fired up docker, ran the docker run command.. Left everything default.. Created a New Workspace. Downloaded Gemma 12B model in Ollama 0.9.0. Changed workspace: **Temperature:** 0 **Chat Mode:** Query Uploaded the same document as above for a D-Link Wireless Router, embedded it into workspace. sent the same prompt by copy and paste, this is the response i got in a fresh docker install: ```json { "Summary": "This document details the development and implementation of a new system for managing and analyzing customer feedback. It outlines the motivations behind creating this system, which stemmed from a desire to improve customer satisfaction and gain deeper insights into customer needs and preferences. The existing feedback collection methods were deemed inefficient and lacked the analytical capabilities required for effective decision-making. The new system aims to centralize feedback data from various sources, automate the analysis process, and provide actionable insights to relevant teams. The core components of the system include a centralized feedback repository, a natural language processing (NLP) engine for sentiment analysis and topic extraction, and a dashboard for visualizing key metrics and trends. The document describes the architecture of the system, highlighting the integration of different modules and the flow of data. It also addresses the technical challenges encountered during development, such as ensuring data security and scalability. The selection of appropriate technologies and tools is justified based on their ability to meet the project’s requirements. Implementation of the system involved a phased approach, starting with a pilot program to test its functionality and gather user feedback. This iterative process allowed for continuous improvement and refinement of the system based on real-world usage. Training programs were also conducted to ensure that employees were proficient in using the new system. The document emphasizes the importance of user adoption and provides recommendations for promoting system usage throughout the organization. Ultimately, the new feedback management system is expected to significantly enhance the organization's ability to understand and respond to customer needs. By providing timely and actionable insights, the system will enable data-driven decision-making and contribute to improved customer satisfaction and loyalty. The document concludes with a discussion of potential future enhancements and expansion of the system to incorporate additional data sources and analytical capabilities." } ``` ![Image](https://github.com/user-attachments/assets/799496fe-b1da-4625-829b-e7b0b6e963e5) So its not just the windows install that is hallucinating..
yindo commented 2026-02-22 18:30:17 -05:00
Author
Owner
Copy Link
Copy Source

@digitalassassins commented on GitHub (Jun 23, 2025):

So i installed the docker container for 1.81 using docker pull mintplexlabs/anythingllm:1.8.1

Ran the exact same prompt in the 1.81 version and it works perfectly, the response is exactly as expected. Explains it is a user manual for a D-Link Wireless Access point.

Image

{
  "Summary": "This document serves as a user manual and warranty information for the D-Link DWL-2100AP wireless access point. It details the contents of the package, which includes the access point itself, a power adapter, a manual and warranty on CD, a quick installation guide, an Ethernet cable, and software requirements like Internet Explorer or Netscape Navigator. The device is designed to provide wireless network connectivity for computers running Windows, Macintosh, or Linux operating systems with an installed Ethernet adapter. \n\nThe manual likely provides instructions on how to install and configure the access point, including connecting it to a network, setting up security protocols, and managing user access. It probably covers troubleshooting common issues and provides technical specifications for the device, such as its wireless standard (802.11g), data transfer rate (108Mbps), and operating frequencies. The warranty information outlines the terms and conditions of the manufacturer's warranty, including the duration of coverage and the process for making a claim. \n\nThe quick installation guide provides a simplified, step-by-step process for setting up the access point, aimed at users who prefer a more streamlined approach. It likely focuses on the essential steps required to establish a basic wireless connection, without delving into the more advanced configuration options. The inclusion of a CD with the manual and warranty suggests that the document is also available in digital format, providing users with an alternative way to access the information. \n\nOverall, this package provides everything needed to set up and maintain a wireless network using the included access point. The combination of a detailed manual, a quick installation guide, and warranty information ensures that users have the resources necessary to install, configure, and troubleshoot the device, as well as understand their rights and responsibilities under the manufacturer’s warranty."
}

and this is the document:

Image

Perfect answer to the question I asked.

Image

With Document Pinning the answer was even more accurate.. It also shows the citation at the bottom, which it doesn't anymore in 1.8.2.

Image

in 1.8.2 it was saying it is a:

  • document on the relationships between humans and animals
  • a purchase receipt for a pink motorised Barbie car
  • a document on analysing and managing customer feedback

Ollama Version Bundled With Windows: 0.5.4:
Works with 1.8.1
Hallucinates 1.8.2

Ollama Version installed in Docker: latest:0.9.0
Works with 1.8.1
Hallucinates 1.8.2

RAG is well and truly F**ked in the latest version...

@digitalassassins commented on GitHub (Jun 23, 2025): So i installed the docker container for 1.81 using `docker pull mintplexlabs/anythingllm:1.8.1` Ran the exact same prompt in the 1.81 version and it works perfectly, the response is exactly as expected. Explains it is a user manual for a D-Link Wireless Access point. ![Image](https://github.com/user-attachments/assets/d856f2b4-a7b7-4795-b5b6-487bc346ecd0) ```python { "Summary": "This document serves as a user manual and warranty information for the D-Link DWL-2100AP wireless access point. It details the contents of the package, which includes the access point itself, a power adapter, a manual and warranty on CD, a quick installation guide, an Ethernet cable, and software requirements like Internet Explorer or Netscape Navigator. The device is designed to provide wireless network connectivity for computers running Windows, Macintosh, or Linux operating systems with an installed Ethernet adapter. \n\nThe manual likely provides instructions on how to install and configure the access point, including connecting it to a network, setting up security protocols, and managing user access. It probably covers troubleshooting common issues and provides technical specifications for the device, such as its wireless standard (802.11g), data transfer rate (108Mbps), and operating frequencies. The warranty information outlines the terms and conditions of the manufacturer's warranty, including the duration of coverage and the process for making a claim. \n\nThe quick installation guide provides a simplified, step-by-step process for setting up the access point, aimed at users who prefer a more streamlined approach. It likely focuses on the essential steps required to establish a basic wireless connection, without delving into the more advanced configuration options. The inclusion of a CD with the manual and warranty suggests that the document is also available in digital format, providing users with an alternative way to access the information. \n\nOverall, this package provides everything needed to set up and maintain a wireless network using the included access point. The combination of a detailed manual, a quick installation guide, and warranty information ensures that users have the resources necessary to install, configure, and troubleshoot the device, as well as understand their rights and responsibilities under the manufacturer’s warranty." } ``` and this is the document: ![Image](https://github.com/user-attachments/assets/9895b060-ef70-48f0-aefb-329994d2560a) Perfect answer to the question I asked. ![Image](https://github.com/user-attachments/assets/5ecd90b0-87a5-468a-8a55-83efb7eb263e) With Document Pinning the answer was even more accurate.. It also shows the citation at the bottom, which it doesn't anymore in 1.8.2. ![Image](https://github.com/user-attachments/assets/afb2d0db-ed48-4243-aed1-0d7b982db9c3) in 1.8.2 it was saying it is a: - document on the relationships between humans and animals - a purchase receipt for a pink motorised Barbie car - a document on analysing and managing customer feedback Ollama Version Bundled With Windows: 0.5.4: Works with 1.8.1 Hallucinates 1.8.2 Ollama Version installed in Docker: latest:0.9.0 Works with 1.8.1 Hallucinates 1.8.2 RAG is well and truly F**ked in the latest version...
yindo commented 2026-02-22 18:30:17 -05:00
Author
Owner
Copy Link
Copy Source

@digitalassassins commented on GitHub (Jun 26, 2025):

To replicate, you would be best to uninstall the containers, images & volumes relating to AnythingLLM from Docker, then run:

docker image prune
docker volume prune
docker builder prune

then pull 1.8.2 image.. It's a container image, so it will have the exact same codebase.

Install it as if you are a new user.

Then:

  • download the PDF document
  • Create a workspace and change the temp to 0
  • Change the chat mode to: Query
  • Upload the document to the workspace
  • Then send the prompt I posted:
using file ‘2602130.pdf’. In a field called Summary, please provide your own summary of the book or document. The summary should be clear, concise, and written in your own words. Do not reference the filename in the summary, as filenames can change and may not reflect the actual content. The Summary must be 4 paragraphs in length. If there is a literary story, do not provide any spoilers. Return only the required field. Return the response in JSON format, only provide the JSON as the Response. If you cannot find the information for the field, simply return Not Available. Use this structure: { “Summary”: “” }

You should get the same outcome I had, well, different because it's hallucinating.

Then pull Docker image 1.8.1 and run the same test again. You will see the differences.

@digitalassassins commented on GitHub (Jun 26, 2025): To replicate, you would be best to uninstall the containers, images & volumes relating to AnythingLLM from Docker, then run: ``` docker image prune docker volume prune docker builder prune ``` then pull 1.8.2 image.. It's a container image, so it will have the exact same codebase. Install it as if you are a new user. Then: - [download the PDF document](https://github.com/user-attachments/files/20870476/2602130.pdf) - Create a workspace and change the temp to 0 - Change the chat mode to: Query - Upload the document to the workspace - Then send the prompt I posted: ``` using file ‘2602130.pdf’. In a field called Summary, please provide your own summary of the book or document. The summary should be clear, concise, and written in your own words. Do not reference the filename in the summary, as filenames can change and may not reflect the actual content. The Summary must be 4 paragraphs in length. If there is a literary story, do not provide any spoilers. Return only the required field. Return the response in JSON format, only provide the JSON as the Response. If you cannot find the information for the field, simply return Not Available. Use this structure: { “Summary”: “” } ``` You should get the same outcome I had, well, different because it's hallucinating. Then pull Docker image 1.8.1 and run the same test again. You will see the differences.
yindo commented 2026-02-22 18:30:17 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Jun 26, 2025):

You should get the same outcome I had, well, different because it's hallucinating.

Your query is the issue. You are asking a meta-question, not a question about the content of the document. Which is what RAG explicitly is for - the content itself.

Try

@agent using file ‘2602130.pdf’. In a field called Summary, please provide your own summary of the book or document. The summary should be clear, concise, and written in your own words. Do not reference the filename in the summary, as filenames can change and may not reflect the actual content. The Summary must be 4 paragraphs in length. If there is a literary story, do not provide any spoilers. Return only the required field. Return the response in JSON format, only provide the JSON as the Response. If you cannot find the information for the field, simply return Not Available. Use this structure: { “Summary”: “” }

Which knows and can "see" a file. The confusion is between RAG =/= agentic chat.
https://docs.anythingllm.com/llm-not-using-my-docs#llms-do-not-introspect

@timothycarambat commented on GitHub (Jun 26, 2025): > You should get the same outcome I had, well, different because it's hallucinating. Your query is the issue. You are asking a meta-question, not a question _about_ the content of the document. Which is what RAG explicitly is for - the content itself. Try ``` @agent using file ‘2602130.pdf’. In a field called Summary, please provide your own summary of the book or document. The summary should be clear, concise, and written in your own words. Do not reference the filename in the summary, as filenames can change and may not reflect the actual content. The Summary must be 4 paragraphs in length. If there is a literary story, do not provide any spoilers. Return only the required field. Return the response in JSON format, only provide the JSON as the Response. If you cannot find the information for the field, simply return Not Available. Use this structure: { “Summary”: “” } ``` Which knows and can "see" a file. The confusion is between RAG =/= agentic chat. https://docs.anythingllm.com/llm-not-using-my-docs#llms-do-not-introspect
yindo commented 2026-02-22 18:30:17 -05:00
Author
Owner
Copy Link
Copy Source

@digitalassassins commented on GitHub (Jun 27, 2025):

How come it worked fine in 1.8.1 via the API and the results were always 100% perfect, but now it doesn't work anymore, and we have to use @agent?

Before, it used to bypass @agent and work perfectly. Whenever I use @agent, the result is always worse than when just chatting with the model directly in chat.

Are you saying it was broken before in 1.8.1, the side effect being it was more accurate and user-friendly, but now that it's fixed in 1.8.2, it's harder to use for the average user and less accurate?

I can see on Discord, some users are saying they are having the exact same issue.

With other systems like GPT4All and OpenWebUI, you do not need to type @agent before every question to use the RAG system?

Surely capturing if the user types " Analyse this file 'myfile.pdf' ", the system should detect its RAG chat and, in the background, switch to agent chat automatically? It would make it more user-friendly. I've seen that other systems use either regex or a model as the screening agent to detect the context of the prompt and switch the mode automatically. Isn't this something that could be implemented?

The workspace is set to Query mode, so that doesn't make sense, because Query mode is supposed to query the documents in the workspace and not to chat with the AI bot to find out the weather.

What you're saying is that Query chat has a provision that you also need to put @agent before every message, as well as having the mode set to Query (Which is only for Querying documents)? That sounds backwards to me and totally counterintuitive..

@digitalassassins commented on GitHub (Jun 27, 2025): How come it worked fine in 1.8.1 via the API and the results were always 100% perfect, but now it doesn't work anymore, and we have to use @agent? Before, it used to bypass @agent and work perfectly. Whenever I use @agent, the result is always worse than when just chatting with the model directly in chat. Are you saying it was broken before in 1.8.1, the side effect being it was more accurate and user-friendly, but now that it's fixed in 1.8.2, it's harder to use for the average user and less accurate? I can see on Discord, some users are saying they are having the exact same issue. With other systems like GPT4All and OpenWebUI, you do not need to type @agent before every question to use the RAG system? Surely capturing if the user types " Analyse this file 'myfile.pdf' ", the system should detect its RAG chat and, in the background, switch to agent chat automatically? It would make it more user-friendly. I've seen that other systems use either regex or a model as the screening agent to detect the context of the prompt and switch the mode automatically. Isn't this something that could be implemented? The workspace is set to Query mode, so that doesn't make sense, because Query mode is supposed to query the documents in the workspace and not to chat with the AI bot to find out the weather. What you're saying is that Query chat has a provision that you also need to put @agent before every message, as well as having the mode set to Query (Which is only for Querying documents)? That sounds backwards to me and totally counterintuitive..
yindo changed title from [QUESTION]: Weird RAG Context Issue from API with 1.8.2 to [GH-ISSUE #4030] [QUESTION]: Weird RAG Context Issue from API with 1.8.2 2026-06-05 14:47:16 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label investigating needs info / can't replicate question
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#2563
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?