Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #4352] [FEAT]: Contextual RAG & Document Structuring Improvements šŸš€ #2771

New Issue
Closed
opened 2026-02-22 18:31:10 -05:00 by yindo Ā· 7 comments
yindo commented 2026-02-22 18:31:10 -05:00
Owner
Copy Link
Copy Source

Originally created by @trade-magnet on GitHub (Sep 1, 2025).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/4352

What would you like to see?

First off ā€“ bloody amazing product šŸ™Œ. Absolutely love AnythingLLM. A huge kudos to the team.

I genuinely think that for the success of this product to keep growing, its biggest strength (and main differentiator) is in the RAG component. This is essentially why youā€™d choose AnythingLLM over something like an OpenWebUI. Strengthening how retrieval works is what will really set this project apart from other products in the space.

That said, Iā€™d love to suggest a couple of improvements around contextual RAG and semantic search:

1. Document Structuring Before Embedding

At the moment, when docs are uploaded, it looks like theyā€™re just scraped top-to-bottom without much structure. This can hurt retrieval quality.

Suggestion:

  • Allow the ability to plug in an external API endpoint (e.g. Mistralā€™s OCR/structuring endpoint) that can transform the document into clean Markdown or structured text before embeddings are generated.
  • This extra step would massively improve consistency and reliability of the RAG pipeline.

2. Contextual RAG / Chunk Enrichment

One of the biggest wins Iā€™ve found in my own testing is contextualizing each chunk. For example:

  • Appending a contextual sentence at the beginning of each chunk (summarizing what the chunk relates to in the broader doc).
  • Selecting how many pages/sections of a doc should be used for contextual grounding (e.g. first 10 pages only, or user-defined).

This approach has boosted context awareness for me dramatically ā€” in some cases, itā€™s the difference between a vague answer and a highly relevant one.

3. Future Ideas (Optional)

  • Knowledge graph integration to map relationships between entities across docs.
  • Advanced page/chunk selection options for power users.

Even if AnythingLLM doesnā€™t go all-in on contextual RAG, I think doubling down here would be a huge win. Itā€™s the RAG component that makes this tool shine ā€” and a few targeted improvements could push retrieval performance through the roof.

Thanks again for such an awesome project šŸš€

Originally created by @trade-magnet on GitHub (Sep 1, 2025). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/4352 ### What would you like to see? First off ā€“ bloody amazing product šŸ™Œ. Absolutely love AnythingLLM. A huge kudos to the team. I genuinely think that for the success of this product to keep growing, its biggest strength (and main differentiator) is in the **RAG component**. This is essentially why youā€™d choose AnythingLLM over something like an OpenWebUI. Strengthening how retrieval works is what will really set this project apart from other products in the space. That said, Iā€™d love to suggest a couple of improvements around **contextual RAG and semantic search**: ### 1. Document Structuring Before Embedding At the moment, when docs are uploaded, it looks like theyā€™re just scraped top-to-bottom without much structure. This can hurt retrieval quality. **Suggestion:** - Allow the ability to plug in an external API endpoint (e.g. Mistralā€™s OCR/structuring endpoint) that can transform the document into clean Markdown or structured text *before* embeddings are generated. - This extra step would massively improve consistency and reliability of the RAG pipeline. ### 2. Contextual RAG / Chunk Enrichment One of the biggest wins Iā€™ve found in my own testing is **contextualizing each chunk**. For example: - Appending a contextual sentence at the beginning of each chunk (summarizing what the chunk relates to in the broader doc). - Selecting how many pages/sections of a doc should be used for contextual grounding (e.g. first 10 pages only, or user-defined). This approach has boosted context awareness for me dramatically ā€” in some cases, itā€™s the difference between a vague answer and a highly relevant one. ### 3. Future Ideas (Optional) - Knowledge graph integration to map relationships between entities across docs. - Advanced page/chunk selection options for power users. --- Even if AnythingLLM doesnā€™t go all-in on contextual RAG, I think doubling down here would be a huge win. Itā€™s the RAG component that makes this tool shine ā€” and a few targeted improvements could push retrieval performance through the roof. Thanks again for such an awesome project šŸš€
yindo added the enhancementfeature request labels 2026-02-22 18:31:10 -05:00
yindo commented 2026-02-22 18:31:10 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Sep 3, 2025):

Allow the ability to plug in an external API endpoint (e.g. Mistralā€™s OCR/structuring endpoint) that can transform the document into clean Markdown or structured text before embeddings are generated.
This extra step would massively improve consistency and reliability of the RAG pipeline.

Agreed.

One of the biggest wins Iā€™ve found in my own testing is contextualizing each chunk. For example:

Appending a contextual sentence at the beginning of each chunk (summarizing what the chunk relates to in the broader doc).
Selecting how many pages/sections of a doc should be used for contextual grounding (e.g. first 10 pages only, or user-defined).
This approach has boosted context awareness for me dramatically ā€” in some cases, itā€™s the difference between a vague answer and a highly relevant one.

This can be quite complex for the local use case, where the rolling summarization would lockup inference/compute while doing this. The page splitting though. We do some basic metadata chunk appending though to help a bit with retrieval.

Also, have you tried Reranking? This usually helps massively for minimal work.

Knowledge graph integration to map relationships between entities across docs.

#1008

@timothycarambat commented on GitHub (Sep 3, 2025): > Allow the ability to plug in an external API endpoint (e.g. Mistralā€™s OCR/structuring endpoint) that can transform the document into clean Markdown or structured text before embeddings are generated. This extra step would massively improve consistency and reliability of the RAG pipeline. Agreed. > One of the biggest wins Iā€™ve found in my own testing is contextualizing each chunk. For example: > > Appending a contextual sentence at the beginning of each chunk (summarizing what the chunk relates to in the broader doc). Selecting how many pages/sections of a doc should be used for contextual grounding (e.g. first 10 pages only, or user-defined). This approach has boosted context awareness for me dramatically ā€” in some cases, itā€™s the difference between a vague answer and a highly relevant one. This can be quite complex for the local use case, where the rolling summarization would lockup inference/compute while doing this. The page splitting though. We do some basic metadata chunk appending though to help a bit with retrieval. Also, have you tried [Reranking](https://docs.anythingllm.com/chatting-with-documents/introduction#vector-database-settings--search-preference-reranking)? This usually helps massively for minimal work. > Knowledge graph integration to map relationships between entities across docs. #1008
yindo commented 2026-02-22 18:31:11 -05:00
Author
Owner
Copy Link
Copy Source

@alainlizotte commented on GitHub (Sep 19, 2025):

+1 for better embedding

@alainlizotte commented on GitHub (Sep 19, 2025): +1 for better embedding
yindo commented 2026-02-22 18:31:11 -05:00
Author
Owner
Copy Link
Copy Source

@Alminc91 commented on GitHub (Sep 24, 2025):

@timothycarambat @trade-magnet
I have put the topic of contextual RAG up two times as a ticket and it was just closed without a real look into it.

My problem is: I let the Chatbot suggest a follow-up question, based on the retrieved data and the user query. E.g. the user asks about art, the chatbot helps by recommending some educational arts courses and asks the user if he would be maybe interested in a specific style, like abstract or figurative.

If the user now answers: abstract. The RAG pipeline retrieves completely out-of-context data, not at all related to abstract art, but just the keyword abstract.

Thank you for putting this up! I believe that this is highly important for an actual conversational RAG chatbot. Although I am not sure on how to solve this best...one idea was to use a smaller LLM for query rewriting, that gets the whole context and the query and rewrites the RAG query to something like: "abstract art" instead of "abstract".

But I am not sure if there are easier and more efficient ways to integrate this. Would love to hear opinions about that. It seemed to me that in Openweb UI that worked in some way, that RAG is accurate in this case. But I would need to test it again.

I am not sure anymore if I used hybrid search + reranking, maybe that helped. I don't know about the implementation.
Aside from that: would it be possible to enable hybrid search with LanceDB in order to also incorporate keyword-based search, e.g. searching for an exact identifier number, e.g. a course number "135-SD-12", or an exact location or full name. Semantical search can be quite bad at this in my tests.

Thanks! Would lobe to hear your feedback on this.

@Alminc91 commented on GitHub (Sep 24, 2025): @timothycarambat @trade-magnet I have put the topic of contextual RAG up two times as a ticket and it was just closed without a real look into it. My problem is: I let the Chatbot suggest a follow-up question, based on the retrieved data and the user query. E.g. the user asks about art, the chatbot helps by recommending some educational arts courses and asks the user if he would be maybe interested in a specific style, like abstract or figurative. If the user now answers: abstract. The RAG pipeline retrieves completely out-of-context data, not at all related to abstract art, but just the keyword abstract. Thank you for putting this up! I believe that this is highly important for an actual conversational RAG chatbot. Although I am not sure on how to solve this best...one idea was to use a smaller LLM for query rewriting, that gets the whole context and the query and rewrites the RAG query to something like: "abstract art" instead of "abstract". But I am not sure if there are easier and more efficient ways to integrate this. Would love to hear opinions about that. It seemed to me that in Openweb UI that worked in some way, that RAG is accurate in this case. But I would need to test it again. I am not sure anymore if I used hybrid search + reranking, maybe that helped. I don't know about the implementation. Aside from that: would it be possible to enable hybrid search with LanceDB in order to also incorporate keyword-based search, e.g. searching for an exact identifier number, e.g. a course number "135-SD-12", or an exact location or full name. Semantical search can be quite bad at this in my tests. Thanks! Would lobe to hear your feedback on this.
yindo commented 2026-02-22 18:31:11 -05:00
Author
Owner
Copy Link
Copy Source

@alainlizotte commented on GitHub (Sep 24, 2025):

Tried it with my setup.
The LLM looked back to the conversation history and returned a correct answer.

Maybe you could try other settings ?

Qwen3 thinking:

Okay, the user asked ā€œContinue lourdā€ which translates to ā€œHeavy continuousā€ in English. Looking back at the conversation history, the previous discussion was about handling high temperatures in the workplace, specifically using the WBGT index and adjusting work schedules based on the intensity of the work. The user is likely referring to the ā€œTravail lourdā€ (heavy work) category mentioned in the table from CONTEXT 3.......

@alainlizotte commented on GitHub (Sep 24, 2025): Tried it with my setup. The LLM looked back to the conversation history and returned a correct answer. Maybe you could try other settings ? Qwen3 thinking: Okay, the user asked ā€œContinue lourdā€ which translates to ā€œHeavy continuousā€ in English. Looking back at the conversation history, the previous discussion was about handling high temperatures in the workplace, specifically using the WBGT index and adjusting work schedules based on the intensity of the work. The user is likely referring to the ā€œTravail lourdā€ (heavy work) category mentioned in the table from CONTEXT 3.......
yindo commented 2026-02-22 18:31:11 -05:00
Author
Owner
Copy Link
Copy Source

@Alminc91 commented on GitHub (Oct 2, 2025):

@alainlizotte what I mean is: the LLM knows the chat history, but the RAG pipeline does not. In my example: if I write "abstract", it will try to retrieve anything that matches the term abstract the best.

It will not understand it needs to search for the term "abstract art", for example. From the context. And retrieve best matches for that.

I could imagine the rolling summary could be nice for solving this and appending it to metadata. But I don't know how this would work. If a smaller internal LLM would be needed, that could be chosen also by selection in the GUI or if an LLM call is not necessary. I am still not sure about the best and most efficient implementation.

@Alminc91 commented on GitHub (Oct 2, 2025): @alainlizotte what I mean is: the LLM knows the chat history, but the RAG pipeline does not. In my example: if I write "abstract", it will try to retrieve anything that matches the term abstract the best. It will not understand it needs to search for the term "abstract art", for example. From the context. And retrieve best matches for that. I could imagine the rolling summary could be nice for solving this and appending it to metadata. But I don't know how this would work. If a smaller internal LLM would be needed, that could be chosen also by selection in the GUI or if an LLM call is not necessary. I am still not sure about the best and most efficient implementation.
yindo commented 2026-02-22 18:31:11 -05:00
Author
Owner
Copy Link
Copy Source

@alainlizotte commented on GitHub (Oct 2, 2025):

Ok. Thanks for the explanations.
Hoping it gets improved.

@alainlizotte commented on GitHub (Oct 2, 2025): Ok. Thanks for the explanations. Hoping it gets improved.
yindo commented 2026-02-22 18:31:11 -05:00
Author
Owner
Copy Link
Copy Source

@Alminc91 commented on GitHub (Oct 2, 2025):

This is different with live AI agents, there is no RAG component and the AI has the whole relevant context at that moment, retrieving it live.

But with RAG, you always need to think about both components. E.g. I want to make a time-based request. For example: some German A1 courses this month.

With the new system variables, my LLM in AnythingLLM knows the date. But the RAG pipeline just searches for similar requests as the term: "some German A1 courses this month.". It does not really know which month is this month and will retrieve whatever results, but usualy not correct ones. It's unreliable.

So to get a RAG chatbot to a productive and (commercially) usable state is not that easy..

@Alminc91 commented on GitHub (Oct 2, 2025): This is different with live AI agents, there is no RAG component and the AI has the whole relevant context at that moment, retrieving it live. But with RAG, you always need to think about both components. E.g. I want to make a time-based request. For example: some German A1 courses this month. With the new system variables, my LLM in AnythingLLM knows the date. But the RAG pipeline just searches for similar requests as the term: "some German A1 courses this month.". It does not really know which month is this month and will retrieve whatever results, but usualy not correct ones. It's unreliable. So to get a RAG chatbot to a productive and (commercially) usable state is not that easy..
yindo changed title from [FEAT]: Contextual RAG & Document Structuring Improvements šŸš€ to [GH-ISSUE #4352] [FEAT]: Contextual RAG & Document Structuring Improvements šŸš€ 2026-06-05 14:48:28 -04:00
yindo closed this issue 2026-06-05 14:48:28 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label enhancement feature request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#2771
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?