Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-21 01:25:28 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #1259] [BUG]: Citations not reflecting custom sized embedding #788

New Issue
Closed
opened 2026-02-22 18:21:22 -05:00 by yindo · 7 comments
yindo commented 2026-02-22 18:21:22 -05:00
Owner
Copy Link
Copy Source

Originally created by @RahSwe on GitHub (May 1, 2024).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/1259

How are you running AnythingLLM?

Docker (remote machine)

What happened?

Having started a fresh instance of ALLM and setting the custom chunk size to 800 tokens, the citations shown are much shorter than the actual vector size. As an example:

image

Are there known steps to reproduce?

See above

Originally created by @RahSwe on GitHub (May 1, 2024). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/1259 ### How are you running AnythingLLM? Docker (remote machine) ### What happened? Having started a fresh instance of ALLM and setting the custom chunk size to 800 tokens, the citations shown are much shorter than the actual vector size. As an example: <img width="626" alt="image" src="https://github.com/Mintplex-Labs/anything-llm/assets/163884933/3eec075f-e798-4171-98e7-3579ea166da4"> ### Are there known steps to reproduce? See above
yindo added the enhancement label 2026-02-22 18:21:22 -05:00
yindo closed this issue 2026-02-22 18:21:22 -05:00
yindo commented 2026-02-22 18:21:22 -05:00
Author
Owner
Copy Link
Copy Source

@Propheticus commented on GitHub (May 11, 2024):

It's unexpected, but the chunk length value is in characters not in tokens. So it's smaller by a factor 3 to 4. (So is the sequence length setting for the embedding model btw)
800 tokens would be a setting of ~3000 characters.

@Propheticus commented on GitHub (May 11, 2024): It's unexpected, but the chunk length value is in characters not in tokens. So it's smaller by a factor 3 to 4. (So is the sequence length setting for the embedding model btw) 800 tokens would be a setting of ~3000 characters.
yindo commented 2026-02-22 18:21:22 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (May 11, 2024):

@Propheticus is correct, the spit size is by chars, not tokens. This is actually intentional and known as counting tokens depends on the tokenizer of the model, which we cannot really replicate since we only have access to the tokenizer that comes with tiktoken and the reason we dont rely on that is that it can be expensive to calculate for each embedding.

I do think though we are at a point where we can probably rely on that library more as we for sure "underestimate" the token length since we count by chars, which is indeed off by some factor depending on tokenizer

@timothycarambat commented on GitHub (May 11, 2024): @Propheticus is correct, the spit size is by chars, not tokens. This is actually intentional and known as counting tokens depends on the tokenizer of the model, which we cannot really replicate since we only have access to the tokenizer that comes with `tiktoken` and the reason we dont rely on that is that it can be expensive to calculate for each embedding. I do think though we are at a point where we can probably rely on that library more as we for sure "underestimate" the token length since we count by chars, which is indeed off by some factor depending on tokenizer
yindo commented 2026-02-22 18:21:22 -05:00
Author
Owner
Copy Link
Copy Source

@RahSwe commented on GitHub (May 12, 2024):

@Propheticus @timothycarambat

Mistake on my side, I did not realize the setting was for characters not tokens. Which means that I will now re-vectorize all of my embeddings.

However it looks like the citations are still too small even when counting characters and bot tokens?

@RahSwe commented on GitHub (May 12, 2024): @Propheticus @timothycarambat Mistake on my side, I did not realize the setting was for characters not tokens. Which means that I will now re-vectorize all of my embeddings. However it looks like the citations are still too small even when counting characters and bot tokens?
yindo commented 2026-02-22 18:21:22 -05:00
Author
Owner
Copy Link
Copy Source

@RahSwe commented on GitHub (May 12, 2024):

@Propheticus @timothycarambat

Further testing, it seems that something is off:

  • I increased the chunk size from 800 characters to 4 000 characters (using OpenAI embedding model and LanceDB).
  • I then deleted all vectors using the workspace option
  • I then made new vectors
  • I then made test queries. The citations show numbers of characters in the range 700-900 characters.
@RahSwe commented on GitHub (May 12, 2024): @Propheticus @timothycarambat Further testing, it seems that something is off: * I increased the chunk size from 800 characters to 4 000 characters (using OpenAI embedding model and LanceDB). * I then deleted all vectors using the workspace option * I then made new vectors * I then made test queries. The citations show numbers of characters in the range 700-900 characters.
yindo commented 2026-02-22 18:21:22 -05:00
Author
Owner
Copy Link
Copy Source

@RahSwe commented on GitHub (May 12, 2024):

@Propheticus @timothycarambat

Further testing:

  • I tried not only resetting the workspace vector database but also deleting all the files, re-upload the files and then embed.
  • I now get chunks and citations approximately corresponding to the custom chunk size. However it is not exactly corresponding to the 4 000 character setting, as an example:
image
@RahSwe commented on GitHub (May 12, 2024): @Propheticus @timothycarambat Further testing: * I tried not only resetting the workspace vector database but also **deleting all the files**, **re-upload the files** and then embed. * I now get chunks and citations approximately corresponding to the custom chunk size. However it is not exactly corresponding to the 4 000 character setting, as an example: <img width="605" alt="image" src="https://github.com/Mintplex-Labs/anything-llm/assets/163884933/7a08d579-1e0a-47eb-883d-761515e7bc14">
yindo commented 2026-02-22 18:21:22 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Aug 13, 2024):

This is due to the "Overlap" setting, that is why you are short of the total amount

@timothycarambat commented on GitHub (Aug 13, 2024): This is due to the "Overlap" setting, that is why you are short of the total amount
yindo commented 2026-02-22 18:21:22 -05:00
Author
Owner
Copy Link
Copy Source

@RahSwe commented on GitHub (Aug 13, 2024):

Hi Tim,
I think the reason was that LangChain creates chunks of equal sizes, which means that the chunk size will rarely correspond to the exakt chunk size applied (then the last chunk would differ a lot).
This is probably more noticeable when using larger chunks but nevertheless is not a bug but a mathematical necessity.

13 aug. 2024 kl. 18:05 skrev Timothy Carambat @.***>:

This is due to the "Overlap" setting, that is why you are short of the total amount

—
Reply to this email directly, view it on GitHub https://github.com/Mintplex-Labs/anything-llm/issues/1259#issuecomment-2286611505, or unsubscribe https://github.com/notifications/unsubscribe-auth/BHCK7BNO2JUJTCO6JV4MSETZRIVE3AVCNFSM6AAAAABHCU3OCGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOBWGYYTCNJQGU.
You are receiving this because you authored the thread.

@RahSwe commented on GitHub (Aug 13, 2024): Hi Tim, I think the reason was that LangChain creates chunks of equal sizes, which means that the chunk size will rarely correspond to the exakt chunk size applied (then the last chunk would differ a lot). This is probably more noticeable when using larger chunks but nevertheless is not a bug but a mathematical necessity. > 13 aug. 2024 kl. 18:05 skrev Timothy Carambat ***@***.***>: > > > This is due to the "Overlap" setting, that is why you are short of the total amount > > — > Reply to this email directly, view it on GitHub <https://github.com/Mintplex-Labs/anything-llm/issues/1259#issuecomment-2286611505>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BHCK7BNO2JUJTCO6JV4MSETZRIVE3AVCNFSM6AAAAABHCU3OCGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOBWGYYTCNJQGU>. > You are receiving this because you authored the thread. >
yindo changed title from [BUG]: Citations not reflecting custom sized embedding to [GH-ISSUE #1259] [BUG]: Citations not reflecting custom sized embedding 2026-06-05 14:37:12 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label enhancement
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#788
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?