Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #928] [BUG]: White screen crash during indexing/embedding. Too many docs in workspace? Workspace needs a cleaning up function? #559

New Issue
Closed
opened 2026-02-22 18:20:10 -05:00 by yindo · 10 comments
yindo commented 2026-02-22 18:20:10 -05:00
Owner
Copy Link
Copy Source

Originally created by @Tiberius1313 on GitHub (Mar 19, 2024).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/928

How are you running AnythingLLM?

AnythingLLM desktop app

What happened?

Adding a new documents seams to be a bit delayed. Then selecting the file is very laggy. Clicking "move 1 file to workspace" results in a white screen crash.

The issue seams to be caused do to too many files in the Workspace. In my case the error appears once the folder contains more than ~2500 files or a size >100 MB (RAM usage is only at 66%).

If I delete the files in the folder "custum-documents"* or rather push them to an other folder, that removes all files from the workspace and the problem seams to be solved. But I'm not sure if these files are still needed and my workaround is a good solution?

*(C:\Users[name]\AppData\Roaming\anythingllm-desktop\storage\documents\custom-documents).

Are there known steps to reproduce?

System: 14-Core Intel Xeon E5-2690 v4, 3166 MHz on MSI X99S Gaming 7 (MS-7885), 8x 16 GB DDR4-3200 DDR4 SDRAM, NVIDIA GeForce GTX 970, Windows 10 Pro 10.0.19045.4170, Anything 1.3.1

It could be that the error occurs after more than 3000 files in the mentioned folder.

Originally created by @Tiberius1313 on GitHub (Mar 19, 2024). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/928 ### How are you running AnythingLLM? AnythingLLM desktop app ### What happened? Adding a new documents seams to be a bit delayed. Then selecting the file is very laggy. Clicking "move 1 file to workspace" results in a white screen crash. The issue seams to be caused do to too many files in the Workspace. In my case the error appears once the folder contains more than ~2500 files or a size >100 MB (RAM usage is only at 66%). If I delete the files in the folder "custum-documents"* or rather push them to an other folder, that removes all files from the workspace and the problem seams to be solved. But I'm not sure if these files are still needed and my workaround is a good solution? *(C:\Users\[name]\AppData\Roaming\anythingllm-desktop\storage\documents\custom-documents). ### Are there known steps to reproduce? System: 14-Core Intel Xeon E5-2690 v4, 3166 MHz on MSI X99S Gaming 7 (MS-7885), 8x 16 GB DDR4-3200 DDR4 SDRAM, NVIDIA GeForce GTX 970, Windows 10 Pro 10.0.19045.4170, Anything 1.3.1 It could be that the error occurs after more than 3000 files in the mentioned folder.
yindo added the possible bug label 2026-02-22 18:20:10 -05:00
yindo closed this issue 2026-02-22 18:20:10 -05:00
yindo commented 2026-02-22 18:20:10 -05:00
Author
Owner
Copy Link
Copy Source

@shatfield4 commented on GitHub (Mar 19, 2024):

What type of files are you uploading to the document uploader? The crash might be happening due to some hidden files that are incomplete but also what is the reason you are uploading 3000 files at one time? Regarding the lag that happens, this is happening just because you are loading in so many files at a time and your CPU cannot keep up with rendering all 3000 files.

If you want to embed this many files without relying on your CPU to do it, we suggest you use a cloud embedding model like OpenAI since it can handle much more than just your local CPU would.

@shatfield4 commented on GitHub (Mar 19, 2024): What type of files are you uploading to the document uploader? The crash might be happening due to some hidden files that are incomplete but also what is the reason you are uploading 3000 files at one time? Regarding the lag that happens, this is happening just because you are loading in so many files at a time and your CPU cannot keep up with rendering all 3000 files. If you want to embed this many files without relying on your CPU to do it, we suggest you use a cloud embedding model like OpenAI since it can handle much more than just your local CPU would.
yindo commented 2026-02-22 18:20:10 -05:00
Author
Owner
Copy Link
Copy Source

@Tiberius1313 commented on GitHub (Mar 19, 2024):

@shatfield4 thanks you for your response.

I'm uploading PDFs. But I'm not uploading them at the same time. Even if I just add any single file the crash occurs. Once the folder ist emptied the system works fine again.

Is it necessary to keep does files in that folder? As it seams if I delete them I might get a lower quality response, but I'm not sure if that's really the case. Do does files interact with the retrieval process or is for that only the information in the DB been used?

@Tiberius1313 commented on GitHub (Mar 19, 2024): @shatfield4 thanks you for your response. I'm uploading PDFs. But I'm not uploading them at the same time. Even if I just add any single file the crash occurs. Once the folder ist emptied the system works fine again. Is it necessary to keep does files in that folder? As it seams if I delete them I might get a lower quality response, but I'm not sure if that's really the case. Do does files interact with the retrieval process or is for that only the information in the DB been used?
yindo commented 2026-02-22 18:20:11 -05:00
Author
Owner
Copy Link
Copy Source

@shatfield4 commented on GitHub (Mar 20, 2024):

It is necessary to keep the files in that folder because that file is a metadata file and is how the document picker knows which files are in the workspace and available. If you delete those files in the custom-documents folder manually, the documents are still embedded in your vector database so even though your workspace says there are no documents, you will still get context from that document and the RAG will still happen (this does not make the results less accurate by deleting those files).

This makes me think that you have a PDF file that may be corrupt or empty that is causing the crash. Is there a certain file that you can upload to replicate this bug consistently or does this only happen when you just upload lots of documents?

@shatfield4 commented on GitHub (Mar 20, 2024): It is necessary to keep the files in that folder because that file is a metadata file and is how the document picker knows which files are in the workspace and available. If you delete those files in the custom-documents folder manually, the documents are still embedded in your vector database so even though your workspace says there are no documents, you will still get context from that document and the RAG will still happen (this does not make the results less accurate by deleting those files). This makes me think that you have a PDF file that may be corrupt or empty that is causing the crash. Is there a certain file that you can upload to replicate this bug consistently or does this only happen when you just upload lots of documents?
yindo commented 2026-02-22 18:20:11 -05:00
Author
Owner
Copy Link
Copy Source

@Tiberius1313 commented on GitHub (Mar 21, 2024):

Thank you for explaining. I understand now better how the DB an the json Files work together.

Meanwhile I have uploaded and processed new files. First everything worked fine and fluently now that I reached 2600 files the same error started to occurs again. Sometimes after restarting I was able to add new documents but once the folder contained more than 2700 files the error appears every time. That means electing a file in "My Documents" is slow and when I press "Move 1 file to workspace" anythingllm crashes after 20 sec in a white screen.

So I don't think the error is related to a single corrupt file but to the size of the files in the folder "custom-documents".

@Tiberius1313 commented on GitHub (Mar 21, 2024): Thank you for explaining. I understand now better how the DB an the json Files work together. Meanwhile I have uploaded and processed new files. First everything worked fine and fluently now that I reached 2600 files the same error started to occurs again. Sometimes after restarting I was able to add new documents but once the folder contained more than 2700 files the error appears every time. That means electing a file in "My Documents" is slow and when I press "Move 1 file to workspace" anythingllm crashes after 20 sec in a white screen. So I don't think the error is related to a single corrupt file but to the size of the files in the folder "custom-documents".
yindo commented 2026-02-22 18:20:11 -05:00
Author
Owner
Copy Link
Copy Source

@jafrank88 commented on GitHub (Mar 21, 2024):

I am seeing this as well. There is some cliff that is hit around 180k vectors maybe. I can upload far more files, but if I try to embed even one more I get the white screen of death and no Task Manager activity related to background processing. Using Desktop v.1.40

@jafrank88 commented on GitHub (Mar 21, 2024): I am seeing this as well. There is some cliff that is hit around 180k vectors maybe. I can upload far more files, but if I try to embed even one more I get the white screen of death and no Task Manager activity related to background processing. Using Desktop v.1.40
yindo commented 2026-02-22 18:20:11 -05:00
Author
Owner
Copy Link
Copy Source

@Tiberius1313 commented on GitHub (Mar 27, 2024):

Thank you for confirming. But I don't think it's related to the DB and the vectors because when you delete the files from custom-documents the issue is gone and that step did not affect the DB.
I think the bug is simply caused by the Windows File Explores limitation to handle large amount of files. This process seams to be a single core application and when the response takes too long anythingLLM seams to run into a time out error.
Maybe the problem could be solved by storing the files in sub folder structure und the comparison of new and already embedded files could be made by an index?

@Tiberius1313 commented on GitHub (Mar 27, 2024): Thank you for confirming. But I don't think it's related to the DB and the vectors because when you delete the files from custom-documents the issue is gone and that step did not affect the DB. I think the bug is simply caused by the Windows File Explores limitation to handle large amount of files. This process seams to be a single core application and when the response takes too long anythingLLM seams to run into a time out error. Maybe the problem could be solved by storing the files in sub folder structure und the comparison of new and already embedded files could be made by an index?
yindo commented 2026-02-22 18:20:11 -05:00
Author
Owner
Copy Link
Copy Source

@wallartup commented on GitHub (Mar 28, 2024):

You probably hit a vector DB limit somehow.

A fix could be to be able to deploy different instances of lancedb and not have all the documents connected with just one db instance. After a certain amount of vector chunks it "cuts" the db into another vector

@wallartup commented on GitHub (Mar 28, 2024): You probably hit a vector DB limit somehow. A fix could be to be able to deploy different instances of lancedb and not have all the documents connected with just one db instance. After a certain amount of vector chunks it "cuts" the db into another vector
yindo commented 2026-02-22 18:20:11 -05:00
Author
Owner
Copy Link
Copy Source

@wallartup commented on GitHub (Mar 28, 2024):

Look at this:
https://lancedb.github.io/lancedb/ann_indexes/

@wallartup commented on GitHub (Mar 28, 2024): Look at this: https://lancedb.github.io/lancedb/ann_indexes/
yindo commented 2026-02-22 18:20:11 -05:00
Author
Owner
Copy Link
Copy Source

@scimantica commented on GitHub (Aug 27, 2024):

Adding another voice to this - am also seeing the same behaviour above a certain number of documents.

@scimantica commented on GitHub (Aug 27, 2024): Adding another voice to this - am also seeing the same behaviour above a certain number of documents.
yindo commented 2026-02-22 18:20:11 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Sep 19, 2024):

Moved to #2317

@timothycarambat commented on GitHub (Sep 19, 2024): Moved to #2317
yindo changed title from [BUG]: White screen crash during indexing/embedding. Too many docs in workspace? Workspace needs a cleaning up function? to [GH-ISSUE #928] [BUG]: White screen crash during indexing/embedding. Too many docs in workspace? Workspace needs a cleaning up function? 2026-06-05 14:35:59 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label possible bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#559
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?