Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #4072] RAG retrieval process chat history (previous issue) #2597

New Issue
Closed
opened 2026-02-22 18:30:23 -05:00 by yindo · 2 comments
yindo commented 2026-02-22 18:30:23 -05:00
Owner
Copy Link
Copy Source

Originally created by @Alminc91 on GitHub (Jun 30, 2025).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/4072

I am referencing your comment on the closed issue #4071
I will formulate my question better and answer. Thanks for your help:

I tried it with 32k context window and it is only one follow-up question (so maybe about 6-8k tokens).
I use Qwen2.5-72B with AWQ 4-bit quantization.

It still fails at that. It seems this technique is called Conversational RAG.
Maybe another example is helpful:

I ask for three courses with instructors. In the next prompt, I ask for more courses from the second instructor. This works with providers like Chatbase.

I assume: they query the model two times with the whole context for a query rewrite, so this is a bit more complex and not easily integrated into AnythingLLM, I just assume.

So that would be a system issue, but quite complex and more for commercial, advanced systems? Thank you.

P.S.: My mistake previously: The courses were not hallucinated, they exist. They are just not sports courses.

The original comment:

This isn't user error nor an issue with the system, this is a model issue! The hallunications are the model behavior, likely the model is too small or over-quantized to get coherent results. In general, small over-quantized models get worse the longer a chat goes as the context window fills up.

As for RAG results, each query gets its own results. If none are found, previous citations will backfill this. Then, on top of this, the chat history is always included. However this is all assuming your context window is not tiny. To not overflow the context window, crashing the model, we will auto-prune the history. If the context window for your model is set to a low value youll see behavior like this.

You didnt mention what provider or model or system specs so I am making assumptions that usually explain the above for most people

_Originally posted by @timothycarambat in #4071

Originally created by @Alminc91 on GitHub (Jun 30, 2025). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/4072 I am referencing your comment on the closed issue #4071 I will formulate my question better and answer. Thanks for your help: I tried it with 32k context window and it is only one follow-up question (so maybe about 6-8k tokens). I use Qwen2.5-72B with AWQ 4-bit quantization. It still fails at that. It seems this technique is called **Conversational RAG**. Maybe another example is helpful: I ask for three courses with instructors. In the next prompt, I ask for more courses from the second instructor. This works with providers like Chatbase. I assume: they query the model two times with the whole context for a query rewrite, so this is a bit more complex and not easily integrated into AnythingLLM, I just assume. So that would be a system issue, but quite complex and more for commercial, advanced systems? Thank you. P.S.: My mistake previously: The courses were not hallucinated, they exist. They are just not sports courses. The original comment: > This isn't user error nor an issue with the system, this is a model issue! The hallunications are the model behavior, likely the model is too small or over-quantized to get coherent results. In general, small over-quantized models get _worse_ the longer a chat goes as the context window fills up. > > As for RAG results, each query gets its own results. If none are found, previous citations will backfill this. Then, on top of this, the chat history is always included. **However** this is all assuming your context window is not tiny. To not overflow the context window, crashing the model, we will auto-prune the history. If the context window for your model is set to a low value youll see behavior like this. > > You didnt mention what provider or model or system specs so I am making assumptions that usually explain the above for most people _Originally posted by @timothycarambat in [#4071](https://github.com/Mintplex-Labs/anything-llm/issues/4071#issuecomment-3019259082)
yindo closed this issue 2026-02-22 18:30:23 -05:00
yindo commented 2026-02-22 18:30:23 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Jun 30, 2025):

I ask for three courses with instructors. In the next prompt, I ask for more courses from the second instructor. This works with providers like Chatbase.

They are putting the entire context into the chat window. This is a huge burden on the system running the LLM because you are injecting tokens that ultimately are "useless" to have in context. However, you get more coherent responses when requiring full-text comprehension. This can be solved by pinning the document.

We dont do this by default since it blows up your context window needlessly, but it will give you better results. An "inbetween" solution is just to change the chat search preference to "Accuracy optimized" which will then rerank document snippets - this gives you more context than normal rag without injecting the entire corpus of text
https://docs.anythingllm.com/llm-not-using-my-docs#vector-database-settings--search-preference

@timothycarambat commented on GitHub (Jun 30, 2025): > I ask for three courses with instructors. In the next prompt, I ask for more courses from the second instructor. This works with providers like Chatbase. They are putting the entire context into the chat window. This is a huge burden on the system running the LLM because you are injecting tokens that ultimately are "useless" to have in context. However, you get more coherent responses when requiring full-text comprehension. This can be solved by [pinning the document](https://docs.anythingllm.com/llm-not-using-my-docs#document-pinning). We dont do this by default since it blows up your context window needlessly, but it will give you better results. An "inbetween" solution is just to change the chat search preference to "Accuracy optimized" which will then rerank document snippets - this gives you more context than normal rag without injecting the entire corpus of text https://docs.anythingllm.com/llm-not-using-my-docs#vector-database-settings--search-preference
yindo commented 2026-02-22 18:30:23 -05:00
Author
Owner
Copy Link
Copy Source

@Alminc91 commented on GitHub (Jun 30, 2025):

I ask for three courses with instructors. In the next prompt, I ask for more courses from the second instructor. This works with providers like Chatbase.

They are putting the entire context into the chat window. This is a huge burden on the system running the LLM because you are injecting tokens that ultimately are "useless" to have in context. However, you get more coherent responses when requiring full-text comprehension. This can be solved by pinning the document.

We dont do this by default since it blows up your context window needlessly, but it will give you better results. An "inbetween" solution is just to change the chat search preference to "Accuracy optimized" which will then rerank document snippets - this gives you more context than normal rag without injecting the entire corpus of text https://docs.anythingllm.com/llm-not-using-my-docs#vector-database-settings--search-preference

@timothycarambat Sorry for bothering so much with this...and thank you for helping.
Can you still provide one more answer? I think it is still not perfectly clear. I feed the system small seperate sources for every course and set retrieval to 4 as recommended. I do not have one big document with thousands of courses for one instructor. But, for example, 10 small documents with ten courses from that instructor.

The chatbot first answers like this:

  1. course:
    bla bla
    instructor: Jason

  2. course
    bla bla

  3. course:
    bla bla

my second prompt:
"does the instructor of the first course have more courses?"

Without writing the name "Jason". The request then does not understand I am asking about courses for the insctructor "Jason" but any random instructor.

Does this really have to do with just more context?? And this is not "conversational RAG"?? I don't believe that pinning or reranking helps in any way with this use case? Please correct me, if I am wrong.

@Alminc91 commented on GitHub (Jun 30, 2025): > > I ask for three courses with instructors. In the next prompt, I ask for more courses from the second instructor. This works with providers like Chatbase. > > They are putting the entire context into the chat window. This is a huge burden on the system running the LLM because you are injecting tokens that ultimately are "useless" to have in context. However, you get more coherent responses when requiring full-text comprehension. This can be solved by [pinning the document](https://docs.anythingllm.com/llm-not-using-my-docs#document-pinning). > > We dont do this by default since it blows up your context window needlessly, but it will give you better results. An "inbetween" solution is just to change the chat search preference to "Accuracy optimized" which will then rerank document snippets - this gives you more context than normal rag without injecting the entire corpus of text https://docs.anythingllm.com/llm-not-using-my-docs#vector-database-settings--search-preference @timothycarambat Sorry for bothering so much with this...and thank you for helping. Can you still provide one more answer? I think it is still not perfectly clear. I feed the system small seperate sources for every course and set retrieval to 4 as recommended. I do not have one big document with thousands of courses for one instructor. But, for example, 10 small documents with ten courses from that instructor. The chatbot first answers like this: 1. course: bla bla instructor: Jason 2. course bla bla 3. course: bla bla my second prompt: "does the instructor of the first course have more courses?" Without writing the name "Jason". The request then does not understand I am asking about courses for the insctructor "Jason" but any random instructor. Does this really have to do with just more context?? And this is not "conversational RAG"?? I don't believe that pinning or reranking helps in any way with this use case? Please correct me, if I am wrong.
yindo changed title from RAG retrieval process chat history (previous issue) to [GH-ISSUE #4072] RAG retrieval process chat history (previous issue) 2026-06-05 14:47:28 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#2597
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?