Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #4246] [FEAT]: RAG retrieval process misses reference to chat history #2703

New Issue
Closed
opened 2026-02-22 18:30:52 -05:00 by yindo · 2 comments
yindo commented 2026-02-22 18:30:52 -05:00
Owner
Copy Link
Copy Source

Originally created by @astrager on GitHub (Aug 5, 2025).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/4246

What would you like to see?

Hello AnythingLLM team,
This is a feature request that had been open previously, but was closed. This was the previous context of issue #4071:
"Dear AnythingLLM team,

I have a feeling that the RAG pipeline does not reference the chat history the LLM model is aware of.

For example, imagine that I have fed data into my chatbot for an online course adviser. Let's assume that I have multiple sports courses in my repertoire.
When I ask for sports courses, the chatbot lists three courses.

However, if I then ask for more, the sources provided are completely irrelevant to my previous request. The LLM knows the chat history, but the RAG pipeline for retrieving suitable sources does not reference it."

I have also experienced this and believe a feature regarding it would be extremely helpful. I also checked the response to the previous issue and am confident that context window and small model size are not the problem.

The feature I would love to see would be the ability to at least concatenate the previous prompt with the most current one before performing the RAG semantic search. This would allow for much better retrieval for a sequence of questions like this: "What is Python?" followed by "What are the best data management libraries for it?"

With the current implementation the second set of documents pulled might not be relevant (could be for an entirely different coding language) because the semantic search does not recognize that "it" is referring to Python.

Another better, but more time intensive way to go about solving this issue would be allowing the user to select a small llm that sits in front of the RAG pipeline and rephrases the most recent question to not rely on the context of the conversation. Such as "What are the best data management libraries for python?" from the example above.

I may attempt to fork and figure it out, but I think this is a feature that should be prioritized as it will greatly improve dynamic conversation with the models.

Thank you all for the amazing open-source project! Has been so much fun playing around with it!

Originally created by @astrager on GitHub (Aug 5, 2025). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/4246 ### What would you like to see? Hello AnythingLLM team, This is a feature request that had been open previously, but was closed. This was the previous context of issue #4071: "Dear AnythingLLM team, I have a feeling that the RAG pipeline does not reference the chat history the LLM model is aware of. For example, imagine that I have fed data into my chatbot for an online course adviser. Let's assume that I have multiple sports courses in my repertoire. When I ask for sports courses, the chatbot lists three courses. However, if I then ask for more, the sources provided are completely irrelevant to my previous request. The LLM knows the chat history, but the RAG pipeline for retrieving suitable sources does not reference it." I have also experienced this and believe a feature regarding it would be extremely helpful. I also checked the response to the previous issue and am confident that context window and small model size are not the problem. The feature I would love to see would be the ability to at least concatenate the previous prompt with the most current one before performing the RAG semantic search. This would allow for much better retrieval for a sequence of questions like this: "What is Python?" followed by "What are the best data management libraries for it?" With the current implementation the second set of documents pulled might not be relevant (could be for an entirely different coding language) because the semantic search does not recognize that "it" is referring to Python. Another better, but more time intensive way to go about solving this issue would be allowing the user to select a small llm that sits in front of the RAG pipeline and rephrases the most recent question to not rely on the context of the conversation. Such as "What are the best data management libraries for python?" from the example above. I may attempt to fork and figure it out, but I think this is a feature that should be prioritized as it will greatly improve dynamic conversation with the models. Thank you all for the amazing open-source project! Has been so much fun playing around with it!
yindo added the enhancementfeature request labels 2026-02-22 18:30:52 -05:00
yindo closed this issue 2026-02-22 18:30:52 -05:00
yindo commented 2026-02-22 18:30:52 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Aug 8, 2025):

The chat history that has previously been used keeps its citations and new citations can be found from subsequent queries. This means that context answering previous questions stays in the history + new ones + chat history.

Doing a RAG search would very likely not yeild better results since semantically the queries may be quite different and passing in the previous prompts into one aggregate prompt wont benefit results that much.

To solve this, you should leverage reranking for the workspace
https://docs.anythingllm.com/llm-not-using-my-docs#vector-database-settings--search-preference

This helps dramatically

@timothycarambat commented on GitHub (Aug 8, 2025): The chat history that has previously been used _keeps_ its citations and new citations can be found from subsequent queries. This means that context answering previous questions stays in the history + new ones + chat history. Doing a RAG search would very likely not yeild better results since semantically the queries may be quite different and passing in the previous prompts into one aggregate prompt wont benefit results that much. To solve this, you should leverage reranking for the workspace https://docs.anythingllm.com/llm-not-using-my-docs#vector-database-settings--search-preference This helps dramatically
yindo commented 2026-02-22 18:30:52 -05:00
Author
Owner
Copy Link
Copy Source

@astrager commented on GitHub (Aug 8, 2025):

I have enabled reranking, but the problem I was experiencing is with the incoming information from the RAG pipeline after an initial prompt. In my example the documents fetched after the 1st prompt could be too broad to cover the details needed for the second prompt. Then the documents fetched for the second prompt will be more general without specific bearing to Python. The ability to have a small rewriting model before it would be great to allow for better RAG searches.

Thanks for the response!

@astrager commented on GitHub (Aug 8, 2025): I have enabled reranking, but the problem I was experiencing is with the incoming information from the RAG pipeline after an initial prompt. In my example the documents fetched after the 1st prompt could be too broad to cover the details needed for the second prompt. Then the documents fetched for the second prompt will be more general without specific bearing to Python. The ability to have a small rewriting model before it would be great to allow for better RAG searches. Thanks for the response!
yindo changed title from [FEAT]: RAG retrieval process misses reference to chat history to [GH-ISSUE #4246] [FEAT]: RAG retrieval process misses reference to chat history 2026-06-05 14:48:04 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label enhancement feature request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#2703
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?