Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #4374] [FEAT]: vector database and embeder #2789

New Issue
Closed
opened 2026-02-22 18:31:15 -05:00 by yindo · 3 comments
yindo commented 2026-02-22 18:31:15 -05:00
Owner
Copy Link
Copy Source

Originally created by @kalle07 on GitHub (Sep 9, 2025).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/4374

What would you like to see?

if i change the the chunk length
BTW its consiten ?? CHUNK -> token or character ???

here
Image

here
Image

and here
Image

OK if i change ... red warning: vector-database will be changed

Image

now if i go to workspace the doc is still emebdd, why?
now I have only one option, I reset the vector dataspace in the workspace, because if i only remove the document its still chached and i cant emebdd with new saetting, so only way reset ... all is very much clicking arround

Originally created by @kalle07 on GitHub (Sep 9, 2025). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/4374 ### What would you like to see? if i change the the chunk length BTW its consiten ?? CHUNK -> token or character ??? here <img width="711" height="225" alt="Image" src="https://github.com/user-attachments/assets/a85bfe28-dcc1-4902-a9d7-70121b254b5d" /> here <img width="726" height="215" alt="Image" src="https://github.com/user-attachments/assets/dda0bc72-cea8-44af-9533-df6c7aa35143" /> and here <img width="470" height="408" alt="Image" src="https://github.com/user-attachments/assets/98730e5a-975e-4088-9a98-fba94a19a69a" /> OK if i change ... red warning: vector-database will be changed <img width="833" height="473" alt="Image" src="https://github.com/user-attachments/assets/1c9d7222-2012-406e-8768-065b750ed5dc" /> now if i go to workspace the doc is still emebdd, why? now I have only one option, I reset the vector dataspace in the workspace, because if i only remove the document its still chached and i cant emebdd with new saetting, so only way reset ... all is very much clicking arround
yindo added the enhancementfeature request labels 2026-02-22 18:31:16 -05:00
yindo closed this issue 2026-02-22 18:31:16 -05:00
yindo commented 2026-02-22 18:31:16 -05:00
Author
Owner
Copy Link
Copy Source

@shatfield4 commented on GitHub (Sep 9, 2025):

In the first screenshot, that is showing the maximum amount of tokens your LLM can use for both context and response. You set this for an LLM to tell AnythingLLM the maximum amount of tokens we can use.

In the second screenshot, this is for the embedding provider. This would be the equivalent to the max tokens for the LLM but just for the embedding model.

In the third screenshot, this is a setting for the chunk sizes of embedded text. You set this when you want to change how big the text chunk sizes are going to be and the overlap of text between each of the embedded chunks.

When changing the text splitter preferences, the intended behavior is to not unembed the current documents but the cache is cleared so that you can reembed it. We do this to prevent large documents that are already embedded from needing to be reembeded. The new text splitter settings are applied when reembedding the document.

@shatfield4 commented on GitHub (Sep 9, 2025): In the first screenshot, that is showing the maximum amount of tokens your LLM can use for both context and response. You set this for an LLM to tell AnythingLLM the maximum amount of tokens we can use. In the second screenshot, this is for the embedding provider. This would be the equivalent to the max tokens for the LLM but just for the embedding model. In the third screenshot, this is a setting for the chunk sizes of embedded text. You set this when you want to change how big the text chunk sizes are going to be and the overlap of text between each of the embedded chunks. When changing the text splitter preferences, the intended behavior is to not unembed the current documents but the cache is cleared so that you can reembed it. We do this to prevent large documents that are already embedded from needing to be reembeded. The new text splitter settings are applied when reembedding the document.
yindo commented 2026-02-22 18:31:16 -05:00
Author
Owner
Copy Link
Copy Source

@kalle07 commented on GitHub (Sep 10, 2025):

i see ... but its a lot of settings in 4 different menues ;)

time for database splitting, one for each embedder , one for each chunk length

OK ... you say LLM is token and the embder is token...
in the third screen shows up "character" ... what is that exactly ?
because i can set only max number of token i set in embedder -> but token are not character !

@kalle07 commented on GitHub (Sep 10, 2025): i see ... but its a lot of settings in 4 different menues ;) time for database splitting, one for each embedder , one for each chunk length OK ... you say LLM is token and the embder is token... in the third screen shows up "character" ... what is that exactly ? because i can set only max number of token i set in embedder -> but token are not character !
yindo commented 2026-02-22 18:31:16 -05:00
Author
Owner
Copy Link
Copy Source

@kalle07 commented on GitHub (Sep 26, 2025):

what is it in here ???

Image

becasue if you say it characters liek its written why i cant set the max token from my embedder its token and token are always differ from chraracters around 1token aproximate 3 chractrers ?!?

@kalle07 commented on GitHub (Sep 26, 2025): what is it in here ??? <img width="470" height="408" alt="Image" src="https://github.com/user-attachments/assets/b415cd7e-63c4-4041-862a-828e02c7b5c1" /> becasue if you say it characters liek its written why i cant set the max token from my embedder its token and token are always differ from chraracters around 1token aproximate 3 chractrers ?!?
yindo changed title from [FEAT]: vector database and embeder to [GH-ISSUE #4374] [FEAT]: vector database and embeder 2026-06-05 14:48:34 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label enhancement feature request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#2789
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?