Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-21 01:25:28 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #4528] [FEAT]: Add Option to Disable Chunking for Pre-Chunked or Preprocessed Files #2882

New Issue
Open
opened 2026-02-22 18:31:39 -05:00 by yindo · 3 comments
yindo commented 2026-02-22 18:31:39 -05:00
Owner
Copy Link
Copy Source

Originally created by @TheTman7 on GitHub (Oct 11, 2025).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/4528

What would you like to see?

Description:
Currently, when saving and embedding files into a workspace, AnythingLLM automatically chunks all uploaded files according to the configured chunk size and overlap. This behavior applies even when files are already pre-chunked or preprocessed before upload.

For advanced users who preprocess or semantically chunk their data outside of AnythingLLM (for example, using LlamaIndex or LangChain), this automatic chunking can degrade performance and embedding quality. It also makes external preprocessing redundant.

Proposed Solution:
Add a workspace-level or upload-level option, such as:
[ ] Disable automatic chunking (treat each uploaded file as a single chunk)

When enabled:
• AnythingLLM would skip its internal chunking process.
• Each file uploaded would be embedded as-is, assuming it is already within the model’s token limit.
• If the file exceeds the model’s token limit, an informative error message would be displayed.

Use Case:
This feature would help users who:
• Use semantic chunking pipelines (e.g., with LlamaIndex, LangChain, or custom scripts).
• Need fine-grained control over chunk boundaries for retrieval accuracy.
• Want to handle embedding preprocessing externally while leveraging AnythingLLM’s workspace and chat interface.

Example Workflow:
1. Pre-chunk a document using LlamaIndex’s semantic text splitter.
2. Upload those small, preprocessed chunks (e.g., 001.md, 002.md, etc.) to AnythingLLM.
3. Enable “Disable automatic chunking.”
4. AnythingLLM embeds each file directly, preserving intended chunk boundaries.

Benefits:
• Prevents double chunking.
• Improves control over data preprocessing and embedding consistency.
• Enables better integration with advanced semantic chunkers.

Additional Context:
While users can currently simulate this behavior by setting a very large chunk size and overlap 0, it’s not reliable across providers or file types. A dedicated “no chunking” toggle would make this workflow consistent and predictable.

Originally created by @TheTman7 on GitHub (Oct 11, 2025). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/4528 ### What would you like to see? Description: Currently, when saving and embedding files into a workspace, AnythingLLM automatically chunks all uploaded files according to the configured chunk size and overlap. This behavior applies even when files are already pre-chunked or preprocessed before upload. For advanced users who preprocess or semantically chunk their data outside of AnythingLLM (for example, using LlamaIndex or LangChain), this automatic chunking can degrade performance and embedding quality. It also makes external preprocessing redundant. Proposed Solution: Add a workspace-level or upload-level option, such as: [ ] Disable automatic chunking (treat each uploaded file as a single chunk) When enabled: • AnythingLLM would skip its internal chunking process. • Each file uploaded would be embedded as-is, assuming it is already within the model’s token limit. • If the file exceeds the model’s token limit, an informative error message would be displayed. Use Case: This feature would help users who: • Use semantic chunking pipelines (e.g., with LlamaIndex, LangChain, or custom scripts). • Need fine-grained control over chunk boundaries for retrieval accuracy. • Want to handle embedding preprocessing externally while leveraging AnythingLLM’s workspace and chat interface. Example Workflow: 1. Pre-chunk a document using LlamaIndex’s semantic text splitter. 2. Upload those small, preprocessed chunks (e.g., 001.md, 002.md, etc.) to AnythingLLM. 3. Enable “Disable automatic chunking.” 4. AnythingLLM embeds each file directly, preserving intended chunk boundaries. Benefits: • Prevents double chunking. • Improves control over data preprocessing and embedding consistency. • Enables better integration with advanced semantic chunkers. Additional Context: While users can currently simulate this behavior by setting a very large chunk size and overlap 0, it’s not reliable across providers or file types. A dedicated “no chunking” toggle would make this workflow consistent and predictable.
yindo added the enhancementfeature request labels 2026-02-22 18:31:39 -05:00
yindo commented 2026-02-22 18:31:39 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Oct 11, 2025):

Are these files chunked in a way that are also compatible with the limitations of the selected embedding model? Each embedding model has its own limitations on length and metadata too. So in this case, it might make sense to go beyond all of that and directly to the DB as embeddings.

Are you only chunking the data, or also are you embedding it?

@timothycarambat commented on GitHub (Oct 11, 2025): Are these files chunked in a way that are also compatible with the limitations of the selected embedding model? Each embedding model has its own limitations on length and metadata too. So in this case, it might make sense to go beyond all of that and directly to the DB as embeddings. Are you only chunking the data, or also are you embedding it?
yindo commented 2026-02-22 18:31:39 -05:00
Author
Owner
Copy Link
Copy Source

@TheTman7 commented on GitHub (Oct 11, 2025):

Are these files chunked in a way that are also compatible with the limitations of the selected embedding model? Each embedding model has its own limitations on length and metadata too. So in this case, it might make sense to go beyond all of that and directly to the DB as embeddings.

Are you only chunking the data, or also are you embedding it?

Yes, I ended up forking the repo to see if it would work and it seemed to work well with the Azure OpenAi text embedding small. However, this forked version was a quick 'vibe code'.

Right now I am just chunking without the embed files, but I could modify it to create the embed files as well. For the time being I can just insert the embeddings to the db, but thought I'd share the idea anyway!

@TheTman7 commented on GitHub (Oct 11, 2025): > Are these files chunked in a way that are also compatible with the limitations of the selected embedding model? Each embedding model has its own limitations on length and metadata too. So in this case, it might make sense to go beyond all of that and directly to the DB as embeddings. > > Are you only chunking the data, or also are you embedding it? Yes, I ended up forking the repo to see if it would work and it seemed to work well with the Azure OpenAi text embedding small. However, this forked version was a quick 'vibe code'. Right now I am just chunking without the embed files, but I could modify it to create the embed files as well. For the time being I can just insert the embeddings to the db, but thought I'd share the idea anyway!
yindo commented 2026-02-22 18:31:40 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Oct 13, 2025):

I think there is some path or way we could have this be obtainable. It might be directly via the dev API or something. This can be quite hairy to do in the UI and out of scope for the everyday user. This way, you can still have the output, and we can also ensure the vectors are searchable and still have the metadata schema we need to show citations properly.

Likely an extension of the /upload or embed endpoints where we skip parsing and just expect an array of embeddings with associated metadata.

@timothycarambat commented on GitHub (Oct 13, 2025): I think there is some path or way we could have this be obtainable. It might be directly via the dev API or something. This can be quite hairy to do in the UI and out of scope for the everyday user. This way, you can still have the output, and we can also ensure the vectors are searchable and still have the metadata schema we need to show citations properly. Likely an extension of the `/upload` or embed endpoints where we skip parsing and just expect an array of embeddings with associated metadata.
yindo changed title from [FEAT]: Add Option to Disable Chunking for Pre-Chunked or Preprocessed Files to [GH-ISSUE #4528] [FEAT]: Add Option to Disable Chunking for Pre-Chunked or Preprocessed Files 2026-06-05 14:49:03 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label enhancement feature request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#2882
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?