Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #3069] [BUG]: performance - tokenizeString runs unnecessary when EMBEDDING_ENGINE is not ‘openai’ #1971

New Issue
Closed
opened 2026-02-22 18:27:28 -05:00 by yindo · 8 comments
yindo commented 2026-02-22 18:27:28 -05:00
Owner
Copy Link
Copy Source

Originally created by @louishalbritter on GitHub (Jan 30, 2025).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/3069

How are you running AnythingLLM?

All versions

What happened?

The function tokenizeString is very CPU-intensive. Its only use I found is here to estimate embedding costs for OpenAI:
https://github.com/Mintplex-Labs/anything-llm/blob/e1af72daa73a12ee96035548f81fd364d5da3c4c/frontend/src/components/Modals/ManageWorkspace/Documents/index.jsx#L145-L146
When run against a local LLM provider, this function isn’t necessary, thus saving significant time and energy.

Are there known steps to reproduce?

I'm using this configuration:

      - EMBEDDING_ENGINE=ollama
      - EMBEDDING_BASE_PATH=http://ollama:11434
      - EMBEDDING_MODEL_PREF=nomic-embed-text:latest

When I upload an 80KiB .xlsx file, the process takes too long and results in a timeout.
Without the token estimation, it's embedded within 1.1s.

Originally created by @louishalbritter on GitHub (Jan 30, 2025). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/3069 ### How are you running AnythingLLM? All versions ### What happened? The function `tokenizeString` is very CPU-intensive. Its only use I found is here to estimate embedding costs for OpenAI: https://github.com/Mintplex-Labs/anything-llm/blob/e1af72daa73a12ee96035548f81fd364d5da3c4c/frontend/src/components/Modals/ManageWorkspace/Documents/index.jsx#L145-L146 When run against a local LLM provider, this function isn’t necessary, thus saving significant time and energy. ### Are there known steps to reproduce? I'm using this configuration: ``` - EMBEDDING_ENGINE=ollama - EMBEDDING_BASE_PATH=http://ollama:11434 - EMBEDDING_MODEL_PREF=nomic-embed-text:latest ``` When I upload an 80KiB .xlsx file, the process takes too long and results in a timeout. Without the token estimation, it's embedded within 1.1s.
yindo added the possible bug label 2026-02-22 18:27:28 -05:00
yindo closed this issue 2026-02-22 18:27:28 -05:00
yindo commented 2026-02-22 18:27:29 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Jan 30, 2025):

This is not the case, we also use token count estimate for detecting if the pinned document count exceed the context window of a model on chats.

https://github.com/Mintplex-Labs/anything-llm/blob/e1af72daa73a12ee96035548f81fd364d5da3c4c/server/utils/DocumentManager/index.js#L38-L61

The frontend function you highlight simply is adding costs (which can frankly be removed as we have other paid providers now anyway who arent tracked).

The metadata is also appended to table schema as well so we cannot outright remove it either since it would cause upsert failures. What kind of device are you on? This process is pretty trival to run and I have run this on 500pg pdfs without issue.

@timothycarambat commented on GitHub (Jan 30, 2025): This is not the case, we also use token count estimate for detecting if the pinned document count exceed the context window of a model on chats. https://github.com/Mintplex-Labs/anything-llm/blob/e1af72daa73a12ee96035548f81fd364d5da3c4c/server/utils/DocumentManager/index.js#L38-L61 The frontend function you highlight simply is adding costs (which can frankly be removed as we have other paid providers now anyway who arent tracked). The metadata is also appended to table schema as well so we cannot outright remove it either since it would cause upsert failures. What kind of device are you on? This process is pretty trival to run and I have run this on 500pg pdfs without issue.
yindo commented 2026-02-22 18:27:29 -05:00
Author
Owner
Copy Link
Copy Source

@louishalbritter commented on GitHub (Jan 30, 2025):

Okay, reguarding the device, it shouldn't be the bottleneck. It's a Ryzen 7 7700X with a Radeon RX 6800.
I was running it in docker context though. I'll quickly test the behaviour in the native app since docker might throttle the cpu

@louishalbritter commented on GitHub (Jan 30, 2025): Okay, reguarding the device, it shouldn't be the bottleneck. It's a Ryzen 7 7700X with a Radeon RX 6800. I was running it in docker context though. I'll quickly test the behaviour in the native app since docker might throttle the cpu
yindo commented 2026-02-22 18:27:29 -05:00
Author
Owner
Copy Link
Copy Source

@louishalbritter commented on GitHub (Jan 30, 2025):

The electron App behaves the same. Here's a 10s timeline from the node profiler:
Image

One Core is always at 100% until I stop the process. I waited for 5m now, but it's still not finished to embed those 80kb..

Would you have another code suggestion how to debug or improve that?

@louishalbritter commented on GitHub (Jan 30, 2025): The electron App behaves the same. Here's a 10s timeline from the node profiler: ![Image](https://github.com/user-attachments/assets/14c5d918-151a-4dc9-b0db-a3e3427c206f) One Core is always at 100% until I stop the process. I waited for 5m now, but it's still not finished to embed those 80kb.. Would you have another code suggestion how to debug or improve that?
yindo commented 2026-02-22 18:27:29 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Jan 30, 2025):

Yeah, I just made a change in how that function is called. If you upload a lot of files at once or it is called many times in a row. At the end of the day the encode function runs very fast, but setting up the encoder itself can take a lot of CPU. No reason to not singleton it to get the gains.

Singleton getEncoding vs current impl on a 1kB string for 100 loops

Results:
Optimized version: 42.69ms
Unoptimized version: 22180.34ms
Difference: 22137.65ms
Performance improvement: 519.57x
@timothycarambat commented on GitHub (Jan 30, 2025): Yeah, I just made a change in how that function is called. If you upload a _lot_ of files at once or it is called many times in a row. At the end of the day the `encode` function runs very fast, but setting up the encoder itself can take a lot of CPU. No reason to not singleton it to get the gains. Singleton `getEncoding` vs current impl on a 1kB string for 100 loops ``` Results: Optimized version: 42.69ms Unoptimized version: 22180.34ms Difference: 22137.65ms Performance improvement: 519.57x ```
yindo commented 2026-02-22 18:27:29 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Jan 30, 2025):

Worth mentioning we use this library in the main process as well for the backend, so being able to singleton it as well will lead to more gains as well.

@timothycarambat commented on GitHub (Jan 30, 2025): Worth mentioning we use this library in the main process as well for the backend, so being able to singleton it as well will lead to more gains as well.
yindo commented 2026-02-22 18:27:29 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Jan 30, 2025):

Publishing this fix under the Docker dev tag right now: https://github.com/Mintplex-Labs/anything-llm/actions/runs/13064309729/job/36453865437?pr=3072

The dev build only builds on x86, so you wont be able to run this image without emulation if on arm. Either way, it is worth seeing if this tag resolves your performance bottleneck (even if it is docker imposed)

EDIT: this seems to address throughput, but not a one-off 80kb string, which is what is happening for you

@timothycarambat commented on GitHub (Jan 30, 2025): Publishing this fix under the Docker `dev` tag right now: https://github.com/Mintplex-Labs/anything-llm/actions/runs/13064309729/job/36453865437?pr=3072 The dev build only builds on x86, so you wont be able to run this image without emulation if on `arm`. Either way, it is worth seeing if this tag resolves your performance bottleneck (even if it is docker imposed) EDIT: this seems to address throughput, but not a one-off 80kb string, which is what is happening for you
yindo commented 2026-02-22 18:27:29 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Jan 30, 2025):

Okay, now the PR and newest build will also handle long string encoding and over a certain limit will do estimation.

On a 500pg PDF, this cut time in half with no side effects during upload. So this should make the collector faster since NodeJS is singe threaded and the tokenization is done in the same thread.

@timothycarambat commented on GitHub (Jan 30, 2025): Okay, now the PR and [newest build ](https://github.com/Mintplex-Labs/anything-llm/actions/runs/13064720253/job/36454984359?pr=3072) will _also_ handle long string encoding and over a certain limit will do estimation. On a 500pg PDF, this cut time in half with no side effects during upload. So this should make the collector faster since NodeJS is singe threaded and the tokenization is done in the same thread.
yindo commented 2026-02-22 18:27:29 -05:00
Author
Owner
Copy Link
Copy Source

@louishalbritter commented on GitHub (Jan 31, 2025):

Perfect, solves the issue. Thanks :)

@louishalbritter commented on GitHub (Jan 31, 2025): Perfect, solves the issue. Thanks :)
yindo changed title from [BUG]: performance - `tokenizeString` runs unnecessary when `EMBEDDING_ENGINE` is not ‘openai’ to [GH-ISSUE #3069] [BUG]: performance - `tokenizeString` runs unnecessary when `EMBEDDING_ENGINE` is not ‘openai’ 2026-06-05 14:43:44 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label possible bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#1971
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?