Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #4033] Conversation context lost when using RAG #2565

New Issue
Closed
opened 2026-02-22 18:30:16 -05:00 by yindo · 11 comments
yindo commented 2026-02-22 18:30:16 -05:00
Owner
Copy Link
Copy Source

Originally created by @reekob on GitHub (Jun 23, 2025).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/4033

Chat mode. Chunking, overlap make no difference with a larger source set.

I have fine tuned the source documents with markdown title, descriptions and keywords which contains multiple models of cars and service related entries. For initial queries with car model included it is always accurate. The system seems to struggle with retrieval on subsequent queries after the initial query because it doesn’t appear to keep the context consistent, example:



Question 1) How much is a Tesla Model 3 tyre change?
Notes: 90%+ match in most instances with correct price.

Question 2) What about a service?
Notes: Very low level %, misses often, is not consistent.

Question 3: What about a windscreen replacement?
Notes: Very low level %, misses often, is not consistent.

It appears this is a basic RAG flaw, where context for subsequent queries is lost.

Without creating an external rewrite wrapper for the users query is this a problem that anythingllm has an out the box solution for?

Originally created by @reekob on GitHub (Jun 23, 2025). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/4033 Chat mode. Chunking, overlap make no difference with a larger source set. I have fine tuned the source documents with markdown title, descriptions and keywords which contains multiple models of cars and service related entries. For initial queries with car model included it is always accurate. The system seems to struggle with retrieval on subsequent queries after the initial query because it doesn’t appear to keep the context consistent, example:

 Question 1) How much is a Tesla Model 3 tyre change? Notes: 90%+ match in most instances with correct price. Question 2) What about a service? Notes: Very low level %, misses often, is not consistent. Question 3: What about a windscreen replacement? Notes: Very low level %, misses often, is not consistent. It appears this is a basic RAG flaw, where context for subsequent queries is lost. Without creating an external rewrite wrapper for the users query is this a problem that anythingllm has an out the box solution for?
yindo added the needs info / can't replicate label 2026-02-22 18:30:16 -05:00
yindo closed this issue 2026-02-22 18:30:16 -05:00
yindo commented 2026-02-22 18:30:16 -05:00
Author
Owner
Copy Link
Copy Source

@digitalassassins commented on GitHub (Jun 23, 2025):

This has only started happening in 1.8.2 I've noticed. It used to work perfectly. Now the RAG seems useless as it doesn't seem to use the pinned document in each consecutive question. The AI Model says it has no context.. There also seems like a problem with the Embedding model built into AnythingLLM now. don't know if it is a coding issue with the chats, will test today with other chat providers and embedding providers.

Before I upgraded, embedding with 1.8.1 via the API used to take about 30 seconds to 1 minute to complete. Now it is done almost instantly with barely any vectors stored in the database. Not sure if it is timing out or breaking the embedding or something.
It could be a configuration problem on my end. I will also test on a new machine. I'm still trying to work it out.

@digitalassassins commented on GitHub (Jun 23, 2025): This has only started happening in 1.8.2 I've noticed. It used to work perfectly. Now the RAG seems useless as it doesn't seem to use the pinned document in each consecutive question. The AI Model says it has no context.. There also seems like a problem with the Embedding model built into AnythingLLM now. don't know if it is a coding issue with the chats, will test today with other chat providers and embedding providers. Before I upgraded, embedding with 1.8.1 via the API used to take about 30 seconds to 1 minute to complete. Now it is done almost instantly with barely any vectors stored in the database. Not sure if it is timing out or breaking the embedding or something. It could be a configuration problem on my end. I will also test on a new machine. I'm still trying to work it out.
yindo commented 2026-02-22 18:30:16 -05:00
Author
Owner
Copy Link
Copy Source

@reekob commented on GitHub (Jun 23, 2025):

Does it work with your first initial query or does that not work either?

Have you tried rewriting your query to suit the llm and sources?

@reekob commented on GitHub (Jun 23, 2025): Does it work with your first initial query or does that not work either? Have you tried rewriting your query to suit the llm and sources?
yindo commented 2026-02-22 18:30:16 -05:00
Author
Owner
Copy Link
Copy Source

@digitalassassins commented on GitHub (Jun 23, 2025):

The query is automatically written for each question to get the perfect answer and is sent programmatically to the API.

When I do a fresh install, it seems to work with the first query from the API, but then it breaks.
So I'm heading down the path of it could be something to do with the built-in embedding model or LanceDB hitting a snag, then breaking subsequent requests after that. because the embedding takes like 1 second to do a massive document, whereas before the API would be waiting for a response back for minutes for large documents in v1.8.1.

I couldn't tell users of the Plugin, ask a question on a document, then totally reinstall AnythingLLM, then ask another question. Wouldn't be feasible 😂

@digitalassassins commented on GitHub (Jun 23, 2025): The query is automatically written for each question to get the perfect answer and is sent programmatically to the API. When I do a fresh install, it seems to work with the first query from the API, but then it breaks. So I'm heading down the path of it could be something to do with the built-in embedding model or LanceDB hitting a snag, then breaking subsequent requests after that. because the embedding takes like 1 second to do a massive document, whereas before the API would be waiting for a response back for minutes for large documents in v1.8.1. I couldn't tell users of the Plugin, ask a question on a document, then totally reinstall AnythingLLM, then ask another question. Wouldn't be feasible 😂
yindo commented 2026-02-22 18:30:17 -05:00
Author
Owner
Copy Link
Copy Source

@reekob commented on GitHub (Jun 23, 2025):

How do you automatically rewrite the initial query?

How do you automatically rewrite the subsequent queries?

@reekob commented on GitHub (Jun 23, 2025): How do you automatically rewrite the initial query? How do you automatically rewrite the subsequent queries?
yindo commented 2026-02-22 18:30:17 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Jun 24, 2025):

Any logs at all?

  • Also what model/context size are you using - since this greatly determines the actual content that is passed to the LLM provider
  • Using another embedding model besides default?
  • Language of documents you are embedding (default works best with english)
@timothycarambat commented on GitHub (Jun 24, 2025): Any logs at all? - Also what model/context size are you using - since this greatly determines the actual content that is passed to the LLM provider - Using another embedding model besides default? - Language of documents you are embedding (default works best with english)
yindo commented 2026-02-22 18:30:17 -05:00
Author
Owner
Copy Link
Copy Source

@reekob commented on GitHub (Jun 24, 2025):

model is 4.1-mini, o3-mini, although have tried most OpenAI and deepseek with the same results. I have set the snippets to 200 and chunks to 8191, 30 overlap. I have played with these settings and pinning documents but with a larger source set (200+) the results are the same.

text-embedding-3-large - although the same results with small and ada-002.

all English documents in various format, .txt and html (I think) from url scraping.

For individual queries with the correct information (see question 1 above) the answer is accurate. The issue seems to be in the subsequent questions, the context of previous questions is not maintained for that new unique query. For instance - if question 2 was "What about a service on my Tesla 3" then the answer would be accurate.

This seems to be a RAG limitation, not an anythingllm thing. Just wondered if anythingllm had an out the box solution for this? What I think it needs is a middle wrapper (an llm with system prompt) to re-write the users query using the recent conversation history. This would be extra tokens on the rewrite request but would be less tokens than having larger max snippets.

I believe that if the query was rewritten to be in context of the conversation then the answer would be accurate. It would also allow lower max snippet to account for more accurate searching. RAG without query rewriting doesn't really work for chat mode without the user giving clear queries.

@reekob commented on GitHub (Jun 24, 2025): > model is 4.1-mini, o3-mini, although have tried most OpenAI and deepseek with the same results. I have set the snippets to 200 and chunks to 8191, 30 overlap. I have played with these settings and pinning documents but with a larger source set (200+) the results are the same. > text-embedding-3-large - although the same results with small and ada-002. > all English documents in various format, .txt and html (I think) from url scraping. For individual queries with the correct information (see question 1 above) the answer is accurate. The issue seems to be in the subsequent questions, the context of previous questions is not maintained for that new unique query. For instance - if question 2 was "What about a service on my Tesla 3" then the answer would be accurate. This seems to be a RAG limitation, not an anythingllm thing. Just wondered if anythingllm had an out the box solution for this? What I think it needs is a middle wrapper (an llm with system prompt) to re-write the users query using the recent conversation history. This would be extra tokens on the rewrite request but would be less tokens than having larger max snippets. I believe that if the query was rewritten to be in context of the conversation then the answer would be accurate. It would also allow lower max snippet to account for more accurate searching. RAG without query rewriting doesn't really work for chat mode without the user giving clear queries.
yindo commented 2026-02-22 18:30:17 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Jun 24, 2025):

Assuming you are in chat mode and not query, have you tried reranking in the workspace?

This is enabled when using the default vector db and really helps a ton, especially with those followup questions which usually have lower semantic meaning than an original query.

@timothycarambat commented on GitHub (Jun 24, 2025): Assuming you are in `chat` mode and not query, have you tried [reranking](https://docs.anythingllm.com/llm-not-using-my-docs#vector-database-settings--search-preference) in the workspace? This is enabled when using the default vector db and really helps a ton, especially with those followup questions which usually have lower semantic meaning than an original query.
yindo commented 2026-02-22 18:30:17 -05:00
Author
Owner
Copy Link
Copy Source

@reekob commented on GitHub (Jun 24, 2025):

Yes, chat mode not query.

Yes, tried accuracy mode. Interestingly with accuracy mode on and max snippets set to 200 the max snippets the OpenAI logs show is 50. Not sure if that's deliberate or a bug but either way makes no difference to the problem.

Unless you can advise otherwise I'm quite confident that each query is unique and not in context of the conversation with regards to the rag query.

If a previously matched source from a prior question matches the new query, then it is used for answer, accurately.

If the previous questions results had nothing to match the new query then the rag query for the new question is unique to that question - for my example the lack of car model means that the hits are more broad and random (lower %). It's more of a lottery if matched or not.

Ideally we could set our chunks to 300 to match our source content and then keep max snippets at 10 ish. If the query was rewritten to be more accurate then this would hit every time (like the first question does with the model).

@reekob commented on GitHub (Jun 24, 2025): Yes, chat mode not query. Yes, tried accuracy mode. Interestingly with accuracy mode on and max snippets set to 200 the max snippets the OpenAI logs show is 50. Not sure if that's deliberate or a bug but either way makes no difference to the problem. Unless you can advise otherwise I'm quite confident that each query is unique and not in context of the conversation with regards to the rag query. If a previously matched source from a prior question matches the new query, then it is used for answer, accurately. If the previous questions results had nothing to match the new query then the rag query for the new question is unique to that question - for my example the lack of car model means that the hits are more broad and random (lower %). It's more of a lottery if matched or not. Ideally we could set our chunks to 300 to match our source content and then keep max snippets at 10 ish. If the query was rewritten to be more accurate then this would hit every time (like the first question does with the model).
yindo commented 2026-02-22 18:30:17 -05:00
Author
Owner
Copy Link
Copy Source

@reekob commented on GitHub (Jun 24, 2025):

If I create a separate workspace with the below prompt, and write question 2 and 3 into it, then use that for my main workspace, the answers and recall are accurate.

SYSTEM
You are an INLINE rewriter for a Tesla–focused auto service.
Your task on every turn:

STEP 1 CLASSIFY SERVICE
• Examine the NEW_USER message (and, if needed, the last five turns in HISTORY).
• Return exactly one of:
“repair” – vehicle fixes (battery, tire, body damage, software update…)
“tradein” – user wants to trade in or sell their car
“sales” – user wants to buy a new or used Tesla
“general” – anything else (charging station locations, pricing info, thanks, etc.)

STEP 2 EXTRACT or INHERIT MODEL & ISSUE
• If NEW_USER already names a car model (e.g. “Model 3”, “Model Y Long Range”), use it.
• Else, if HISTORY contains a model mentioned by the user in the last five turns, inherit that model.
• Otherwise set model = "" (blank).
• For repair: extract an issue keyword (e.g. “battery,” “tire,” “autopilot,” “door sensor”).
• For tradein / sales: issue = "".

STEP 3 REWRITE
• If service = “general” → rewritten = NEW_USER unchanged.
• If service ≠ “general” and model ≠ "" →
– produce a single English sentence that explicitly includes the model and the user’s intent/issue.
Examples:
NEW_USER: “what about the battery?”
➜ rewritten: “How much is a Tesla Model 3 battery replacement?”
NEW_USER: “looking to trade in” + model inherited = “Model Y”
➜ rewritten: “What trade-in value can I get for my Tesla Model Y?”
• If service ≠ “general” but model = "" →
rewritten = NEW_USER unchanged (assistant should ask for clarification).

RESPONSE FORMAT
Return strict JSON exactly as below, with no extra commentary:

{
"service": "<repair|tradein|sales|general>",
"model": "",
"issue": "",
"rewritten":""
}

================ INPUTS ================
HISTORY (up to five prior user+assistant turns):
◆〈history〉◆

NEW_USER:
◆〈current_user〉◆

@reekob commented on GitHub (Jun 24, 2025): If I create a separate workspace with the below prompt, and write question 2 and 3 into it, then use that for my main workspace, the answers and recall are accurate. SYSTEM You are an INLINE rewriter for a Tesla–focused auto service. Your task on every turn: STEP 1 CLASSIFY SERVICE • Examine the NEW_USER message (and, if needed, the last five turns in HISTORY). • Return **exactly one** of: “repair” – vehicle fixes (battery, tire, body damage, software update…) “tradein” – user wants to trade in or sell their car “sales” – user wants to buy a new or used Tesla “general” – anything else (charging station locations, pricing info, thanks, etc.) STEP 2 EXTRACT or INHERIT MODEL & ISSUE • If NEW_USER already names a car model (e.g. “Model 3”, “Model Y Long Range”), use it. • Else, if HISTORY contains a model mentioned by the user in the last five turns, inherit that model. • Otherwise set model = "" (blank). • For repair: extract an issue keyword (e.g. “battery,” “tire,” “autopilot,” “door sensor”). • For tradein / sales: issue = "". STEP 3 REWRITE • If service = “general” → rewritten = NEW_USER unchanged. • If service ≠ “general” and model ≠ "" → – produce a *single English sentence* that explicitly includes the model **and** the user’s intent/issue. Examples: NEW_USER: “what about the battery?” ➜ rewritten: “How much is a Tesla Model 3 battery replacement?” NEW_USER: “looking to trade in” + model inherited = “Model Y” ➜ rewritten: “What trade-in value can I get for my Tesla Model Y?” • If service ≠ “general” but model = "" → rewritten = NEW_USER unchanged (assistant should ask for clarification). RESPONSE FORMAT Return **strict JSON** exactly as below, with no extra commentary: { "service": "<repair|tradein|sales|general>", "model": "<car model or empty string>", "issue": "<issue keyword or empty string>", "rewritten":"<single-sentence question or original message>" } ================ INPUTS ================ HISTORY (up to five prior user+assistant turns): ◆〈history〉◆ NEW_USER: ◆〈current_user〉◆ =======================================
yindo commented 2026-02-22 18:30:17 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Jun 24, 2025):

Yes, tried accuracy mode. Interestingly with accuracy mode on and max snippets set to 200 the max snippets the OpenAI logs show is 50. Not sure if that's deliberate or a bug but either way makes no difference to the problem.

It is deliberate since reranking n documents can be very intensive, esp on CPU.

I'm quite confident that each query is unique and not in context of the conversation with regards to the rag query.

This is true, each query is embedded and sent to search for similarity (results filtered base on relevance threshold). However, we also will dedupe and backfill the context from previous query sources to at least fill the window since often followup questions are within semantically similar chunks of content.

https://github.com/Mintplex-Labs/anything-llm/blob/c4f49ff6b445f8f1b15d3bd42f8a2d24317b745c/server/utils/helpers/chat/index.js#L348-L381

@timothycarambat commented on GitHub (Jun 24, 2025): > Yes, tried accuracy mode. Interestingly with accuracy mode on and max snippets set to 200 the max snippets the OpenAI logs show is 50. Not sure if that's deliberate or a bug but either way makes no difference to the problem. It is deliberate since reranking `n` documents [can be very intensive](https://github.com/Mintplex-Labs/anything-llm/blob/c4f49ff6b445f8f1b15d3bd42f8a2d24317b745c/server/utils/vectorDbProviders/lance/index.js#L104-L107), esp on CPU. > I'm quite confident that each query is unique and not in context of the conversation with regards to the rag query. This is true, each query is embedded and sent to search for similarity (results filtered base on relevance threshold). However, we also will dedupe and backfill the context from previous query sources to at least fill the window since often followup questions are within semantically similar chunks of content. https://github.com/Mintplex-Labs/anything-llm/blob/c4f49ff6b445f8f1b15d3bd42f8a2d24317b745c/server/utils/helpers/chat/index.js#L348-L381
yindo commented 2026-02-22 18:30:17 -05:00
Author
Owner
Copy Link
Copy Source

@reekob commented on GitHub (Jun 24, 2025):

This explains why larger chunking is more likely to give an accurate answer, but rag hit % is low, to subsequent questions. This seems like extra tokens but also a blind spot.

It seems that rewriting would give much better output and cost a lot less tokens. Does anythingllm have any out of the box solution for rewriting user queries? If no is it possible to do this via agent flow or as an external call in the natural process?

@reekob commented on GitHub (Jun 24, 2025): This explains why larger chunking is more likely to give an accurate answer, but rag hit % is low, to subsequent questions. This seems like extra tokens but also a blind spot. It seems that rewriting would give much better output and cost a lot less tokens. Does anythingllm have any out of the box solution for rewriting user queries? If no is it possible to do this via agent flow or as an external call in the natural process?
yindo changed title from Conversation context lost when using RAG to [GH-ISSUE #4033] Conversation context lost when using RAG 2026-06-05 14:47:17 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label needs info / can't replicate
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#2565
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?