Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #4440] [DOCS]: vector database and embeder token vs characters #2832

New Issue
Closed
opened 2026-02-22 18:31:27 -05:00 by yindo · 2 comments
yindo commented 2026-02-22 18:31:27 -05:00
Owner
Copy Link
Copy Source

Originally created by @kalle07 on GitHub (Sep 26, 2025).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/4440

Description

your menue is not consistant!
you show up "token" (more or less) in LLM and in embedder!
but if i choose max text-chunk there is written characters ? what is it?

Image

because the max number i can set here is max token from embedder settings, so it is not consitant!

Originally created by @kalle07 on GitHub (Sep 26, 2025). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/4440 ### Description your menue is not consistant! you show up "token" (more or less) in LLM and in embedder! but if i choose max text-chunk there is written characters ? what is it? <img width="470" height="408" alt="Image" src="https://github.com/user-attachments/assets/0c443480-ee99-44f9-a89c-cc343b44a996" /> because the max number i can set here is max token from embedder settings, so it is not consitant!
yindo added the documentation label 2026-02-22 18:31:27 -05:00
yindo closed this issue 2026-02-22 18:31:27 -05:00
yindo commented 2026-02-22 18:31:27 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Sep 29, 2025):

It is character because, unless we run the models' specific tokenizer, we cannot produce an accurate count - the hint in both does not mention token to avoid that confusion

@timothycarambat commented on GitHub (Sep 29, 2025): It is character because, unless we run the models' specific tokenizer, we cannot produce an accurate count - the hint in both does not mention token to avoid that confusion
yindo commented 2026-02-22 18:31:27 -05:00
Author
Owner
Copy Link
Copy Source

@kalle07 commented on GitHub (Sep 30, 2025):

so my question is ... thats also chararcters because its the max number i can set in text-chunk

Image

here is not mentioned that this are characters.

For the complicated calculation, you could perhaps at least provide an estimated number of token, since characters are independent of language words.

For example, 5000-6000 characters correspond to approximately one page and approximately 1200-1400 token.

@kalle07 commented on GitHub (Sep 30, 2025): so my question is ... thats also chararcters because its the max number i can set in text-chunk <img width="929" height="517" alt="Image" src="https://github.com/user-attachments/assets/4177d7c6-2321-42e8-93e6-24e9eaa6f18e" /> here is not mentioned that this are characters. For the complicated calculation, you could perhaps at least provide an estimated number of token, since characters are independent of language words. For example, 5000-6000 characters correspond to approximately one page and approximately 1200-1400 token.
yindo changed title from [DOCS]: vector database and embeder token vs characters to [GH-ISSUE #4440] [DOCS]: vector database and embeder token vs characters 2026-06-05 14:48:47 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label documentation
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#2832
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?