Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-21 01:25:28 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #388] Not being able to set chunk size for local.ai embeddings #220

New Issue
Closed
opened 2026-02-22 18:18:23 -05:00 by yindo · 4 comments
yindo commented 2026-02-22 18:18:23 -05:00
Owner
Copy Link
Copy Source

Originally created by @Amejonah1200 on GitHub (Nov 16, 2023).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/388

https://github.com/Mintplex-Labs/anything-llm/blob/5ad8a5f2d0c545a913319a641877d70aa7b82b09/server/utils/EmbeddingEngines/localAi/index.js#L16

Why is this not configurable using EMBEDDING_MODEL_CHUNK_SIZE or something? Because some embedding models need lower chunk sizes.
For example, sentence-transformers/all-MiniLM-L6-v2 has a default of 256 "word pieces".

Originally created by @Amejonah1200 on GitHub (Nov 16, 2023). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/388 https://github.com/Mintplex-Labs/anything-llm/blob/5ad8a5f2d0c545a913319a641877d70aa7b82b09/server/utils/EmbeddingEngines/localAi/index.js#L16 Why is this not configurable using `EMBEDDING_MODEL_CHUNK_SIZE` or something? Because some embedding models need lower chunk sizes. For example, [sentence-transformers/all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2#intended-uses) has a default of 256 "word pieces".
yindo added the Integration Requestpossible bug labels 2026-02-22 18:18:24 -05:00
yindo closed this issue 2026-02-22 18:18:24 -05:00
yindo commented 2026-02-22 18:18:24 -05:00
Author
Owner
Copy Link
Copy Source

@Amejonah1200 commented on GitHub (Nov 16, 2023):

if(!process.env.EMBEDDING_MODEL_CHUNK_SIZE)
  this.embeddingChunkLimit = 1_000;
else {
  let size = parseInt(process.env.EMBEDDING_MODEL_CHUNK_SIZE)
  if(size === NaN || size <= 0) throw new Error("Invalid int for EMBEDDING_MODEL_CHUNK_SIZE."); 
  this.embeddingChunkLimit = size;
}

Should do the trick.

@Amejonah1200 commented on GitHub (Nov 16, 2023): ```js if(!process.env.EMBEDDING_MODEL_CHUNK_SIZE) this.embeddingChunkLimit = 1_000; else { let size = parseInt(process.env.EMBEDDING_MODEL_CHUNK_SIZE) if(size === NaN || size <= 0) throw new Error("Invalid int for EMBEDDING_MODEL_CHUNK_SIZE."); this.embeddingChunkLimit = size; } ``` Should do the trick.
yindo commented 2026-02-22 18:18:24 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Nov 16, 2023):

This is not the size of a chunk, its how many chunks can be sent at once via the API. Most embedding apis allow you to send many n long chunks at the same time so you dont have to send a request n times.

That is what the 1,000 is actually referring to. I imagine that with LocalAI, it cannot handle more than one request at a time though and that is likely the issue

@timothycarambat commented on GitHub (Nov 16, 2023): This is not the size of a chunk, its how many chunks can be sent at once via the API. Most embedding apis allow you to send many `n` long chunks at the same time so you dont have to send a request `n` times. That is what the 1,000 is actually referring to. I imagine that with LocalAI, it cannot handle more than one request at a time though and that is likely the issue
yindo commented 2026-02-22 18:18:24 -05:00
Author
Owner
Copy Link
Copy Source

@Amejonah1200 commented on GitHub (Nov 17, 2023):

Apart from this, I kinda doubt you can append 2 embeddings, as they would represent 2 different meanings.

@Amejonah1200 commented on GitHub (Nov 17, 2023): Apart from this, I kinda doubt you can append 2 embeddings, as they would represent 2 different meanings.
yindo commented 2026-02-22 18:18:24 -05:00
Author
Owner
Copy Link
Copy Source

@Amejonah1200 commented on GitHub (Nov 17, 2023):

https://github.com/Mintplex-Labs/anything-llm/blob/master/server/utils/vectorDbProviders/qdrant/index.js#L172-L180

I found this code, as said: not every embedder can process such amount of tokens at once.

@Amejonah1200 commented on GitHub (Nov 17, 2023): https://github.com/Mintplex-Labs/anything-llm/blob/master/server/utils/vectorDbProviders/qdrant/index.js#L172-L180 I found this code, as said: not every embedder can process such amount of tokens at once.
yindo changed title from Not being able to set chunk size for local.ai embeddings to [GH-ISSUE #388] Not being able to set chunk size for local.ai embeddings 2026-06-05 14:34:08 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label Integration Request possible bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#220
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?