Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-22 18:15:21 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #490] Chunking and Text Splitter customization #285

New Issue
Closed
opened 2026-02-22 18:18:44 -05:00 by yindo · 10 comments
yindo commented 2026-02-22 18:18:44 -05:00
Owner
Copy Link
Copy Source

Originally created by @timothycarambat on GitHub (Dec 26, 2023).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/490

Originally assigned to: @timothycarambat on GitHub.

We should build out the text-splitting and chunking strategy to be more configurable per workspace. This should be delineated as an advanced setting since in general most should not need to change it. For those who understand the implications should be allowed to modify it.

More on splitters

It would make sense for the UI to have a way to set file extensions => splitter settings with a way to select a document to "preview" how a document would be split in real-time.

Originally created by @timothycarambat on GitHub (Dec 26, 2023). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/490 Originally assigned to: @timothycarambat on GitHub. We should build out the text-splitting and chunking strategy to be more configurable per workspace. This should be delineated as an advanced setting since in general most should not need to change it. For those who understand the implications should be allowed to modify it. [More on splitters](https://js.langchain.com/docs/modules/data_connection/document_transformers/text_splitters/character_text_splitter) It would make sense for the UI to have a way to set file extensions => splitter settings with a way to select a document to "preview" how a document would be split in real-time.
yindo added the enhancementcore-team-only labels 2026-02-22 18:18:45 -05:00
yindo closed this issue 2026-02-22 18:18:45 -05:00
yindo commented 2026-02-22 18:18:45 -05:00
Author
Owner
Copy Link
Copy Source

@j-loquat commented on GitHub (Dec 26, 2023):

Consider using intelligent chunking approach like Unstructured:
https://medium.com/@unstructured-io

@j-loquat commented on GitHub (Dec 26, 2023): Consider using intelligent chunking approach like Unstructured: https://medium.com/@unstructured-io
yindo commented 2026-02-22 18:18:45 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Dec 26, 2023):

It is likely not wise for us to spin up an embedded container running Unstructured inside of AnythingLLM to rebuild the document parser and/or rely on a third-party API for this. Their tool appears robust and powerful but I am not sure the dependency is worth overhead at this time currently. Will investigate further though

@timothycarambat commented on GitHub (Dec 26, 2023): It is likely not wise for us to spin up an embedded container running Unstructured inside of AnythingLLM to rebuild the document parser and/or rely on a third-party API for this. Their tool appears robust and powerful but I am not sure the dependency is worth overhead at this time currently. Will investigate further though
yindo commented 2026-02-22 18:18:45 -05:00
Author
Owner
Copy Link
Copy Source

@j-loquat commented on GitHub (Dec 26, 2023):

Agree it would be a challenge. No question that more intelligent "chunking" of content during ingestion will really help the quality of the future search results, however.
I wonder if there would be a way to offer a choice to send the documents to a second optional container that could run a different parser and then return the chunks (JSON format?) back to your main container for embedding? Sort of like an effects loop for a guitar amp.

@j-loquat commented on GitHub (Dec 26, 2023): Agree it would be a challenge. No question that more intelligent "chunking" of content during ingestion will really help the quality of the future search results, however. I wonder if there would be a way to offer a choice to send the documents to a second optional container that could run a different parser and then return the chunks (JSON format?) back to your main container for embedding? Sort of like an effects loop for a guitar amp.
yindo commented 2026-02-22 18:18:45 -05:00
Author
Owner
Copy Link
Copy Source

@j-loquat commented on GitHub (Jan 4, 2024):

Txtai looks very good as a one stop shop way of doing embeddings and different types of search:

https://neuml.github.io/txtai

@j-loquat commented on GitHub (Jan 4, 2024): Txtai looks very good as a one stop shop way of doing embeddings and different types of search: https://neuml.github.io/txtai
yindo commented 2026-02-22 18:18:45 -05:00
Author
Owner
Copy Link
Copy Source

@junxu-ai commented on GitHub (Jan 5, 2024):

We should build out the text-splitting and chunking strategy to be more configurable per workspace. This should be delineated as an advanced setting since in general most should not need to change it. For those who understand the implications should be allowed to modify it.

More on splitters

It would make sense for the UI to have a way to set file extensions => splitter settings with a way to select a document to "preview" how a document would be split in real-time.

strongly agreed with this point, as i noticed that the default embedding is a nightmare to get the right input for LLM generation.

@junxu-ai commented on GitHub (Jan 5, 2024): > We should build out the text-splitting and chunking strategy to be more configurable per workspace. This should be delineated as an advanced setting since in general most should not need to change it. For those who understand the implications should be allowed to modify it. > > [More on splitters](https://js.langchain.com/docs/modules/data_connection/document_transformers/text_splitters/character_text_splitter) > > It would make sense for the UI to have a way to set file extensions => splitter settings with a way to select a document to "preview" how a document would be split in real-time. strongly agreed with this point, as i noticed that the default embedding is a nightmare to get the right input for LLM generation.
yindo commented 2026-02-22 18:18:45 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Jan 5, 2024):

@junxu-ai do you mean the splitting or the embedding?

@timothycarambat commented on GitHub (Jan 5, 2024): @junxu-ai do you mean the splitting or the embedding?
yindo commented 2026-02-22 18:18:45 -05:00
Author
Owner
Copy Link
Copy Source

@Stihotvor commented on GitHub (Jun 6, 2024):

I would like to re-open the issue. There is still no way to choose the splitting/chunking. I would like to switch to a code-specific chunking, sentence based, semantic, etc.

As of now the only way to do it - chunk outside to separate files and push them into the UI 😄

@Stihotvor commented on GitHub (Jun 6, 2024): I would like to re-open the issue. There is still no way to choose the splitting/chunking. I would like to switch to a code-specific chunking, sentence based, semantic, etc. As of now the only way to do it - chunk outside to separate files and push them into the UI 😄
yindo commented 2026-02-22 18:18:45 -05:00
Author
Owner
Copy Link
Copy Source

@mstfldmr commented on GitHub (Feb 6, 2025):

are there any updates about this?

I need to be able to select different chunking strategies (per page, semantic, hierarchical...) per workspace

@mstfldmr commented on GitHub (Feb 6, 2025): are there any updates about this? I need to be able to select different chunking strategies (per page, semantic, hierarchical...) per workspace
yindo commented 2026-02-22 18:18:45 -05:00
Author
Owner
Copy Link
Copy Source

@arty-hlr commented on GitHub (Feb 13, 2025):

I tried anythingLLM last week and I also agree, being able to customize how a document is split would go a long way towards better RAG.

@arty-hlr commented on GitHub (Feb 13, 2025): I tried anythingLLM last week and I also agree, being able to customize how a document is split would go a long way towards better RAG.
yindo commented 2026-02-22 18:18:45 -05:00
Author
Owner
Copy Link
Copy Source

@Zeddddd commented on GitHub (Apr 29, 2025):

The current issue is that when the chunk size is too small, the returned answer only includes content from the chunk, and subsequent content (even within the same paragraph) is discarded. To address this, I increased the chunk size to 1024 and re-embedded the data by deleting the cache files and re-uploading & re-embedding the content. However, the recall performance worsened. It seems that contextual coherence measures are needed to ensure more complete information presentation.

@Zeddddd commented on GitHub (Apr 29, 2025): The current issue is that when the chunk size is too small, the returned answer only includes content from the chunk, and subsequent content (even within the same paragraph) is discarded. To address this, I increased the chunk size to 1024 and re-embedded the data by deleting the cache files and re-uploading & re-embedding the content. However, the recall performance worsened. It seems that contextual coherence measures are needed to ensure more complete information presentation.
yindo changed title from Chunking and Text Splitter customization to [GH-ISSUE #490] Chunking and Text Splitter customization 2026-06-05 14:34:29 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label core-team-only enhancement
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#285
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?