Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #4776] [FEAT]: UI of RAG Document window on Desktop needs improvements for large corpus of files #3009

New Issue
Closed
opened 2026-02-22 18:32:14 -05:00 by yindo · 2 comments
yindo commented 2026-02-22 18:32:14 -05:00
Owner
Copy Link
Copy Source

Originally created by @uruiamme on GitHub (Dec 12, 2025).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/4776

What would you like to see?

For those with a large corpus, the Upload Document window and its User Interface (UI) become the primary interface with our local LLM besides the query. [This reviewer in on Windows]

On Desktop, this window has a file picker on the left and an upload window on the right or down below. The RAG and collector seem to just use the Upload Document screen, but it currently has some room for UI improvements to give the user insight into the tasks involved.

Main reference: https://docs.anythingllm.com/chatting-with-documents/rag-in-anythingllm

  1. My suggestions concern the File Picker: The current UI's file picker has no way to paste in a URN, Folder, File name, a list of files, or a wildcard. It is all based on selecting the file(s). A drag-and-drop feature is also available. But using this feature immediately begins the process of uploading, which blocks further operations until this process is done. There's no easy way to import or export the list of files selected for uploading, nor to exclude unwanted files based on file size or file type - we are at the mercy of the picker and the OS if our corpus is large and complicated.
  2. My suggestion is to enforce a "GO" button to explicitly tell the Upload process to begin. This will allow the user to select files three different ways: (1) with the picker, (2) dropping them, (3) using a text field that can accept a clipboard paste that intelligently selects files, folders, and knows wildcards. (e.g. C:\DOCS\*.pdf) The user does any or all three of these things, often several times, and then presses "GO" or "UPLOAD" and can watch the progress. Files in this area are in a QUEUE until the GO button is pressed. Advice can be given for a queue larger than a setpoint, like 1 GB, to suggest a smaller size or certain kind of LLM/RAG combinations.
  3. I would like to be able to modify an upload process. There are already checkboxes. Perhaps each folder or file can have a checkbox that toggles red and green. While uploading, the checkboxes change to a 3-color scheme where yellow check and gray text means it has not uploaded and green with black text says that it is done. If a user decides not to upload the file, he selects the box, it turns red, and the filename turns red. The GO button doesn't prevent work being done in other portions of the window.
  4. I would like the file list to show Icon, Filename, Extension or Type, and File size in abbreviated form. Files above a certain setpoint (20 MB default) have a red checkbox and red text and will not be uploaded. Icons for PDF, Text, docs, videos, and spreadsheets should use standard colors and familiar shapes,
  5. Settings on the Upload Document window should include: Maximum size of text/docs, PDFs, videos, and spreadsheets (each defaults to 20 MB or 100 MB but has a field to adjust this). Without this, the user who accidentally selects a few 100 MB .doc or 500 MB .pdf files will be a bit confused why the process is taking so long. And if a video or audio file is inadvertently in the corpus, this could change the upload time a lot.
  6. When a user selects several files and folders, a quick list of what has been selected must appear before he selects "GO." A list of 233 pdfs, 27 docs, 14,888 jpgs, and 47 .mp4 files that total 250 GB might be accidental. Perhaps the user didn't mean to have any images or any videos? He needs to know the breakdown, so a prelimary thread reading and parsing the intended upload needs to be presented next to the "GO" button. This is where advice about avoiding large media files should be mentioned!
  7. No mention is made of document sync on this page. I think it should be mentioned at this point in the UI even if it is experimental.
  8. No mention is made of chunking on this page. I believe it is relevant enough to mention it and explain where to change this setting.
  9. I'm confused by the "Fetch website" button, especially with no help about how this might work on a document upload page. Please move it to its own tab like Data Connectors.
  10. Rename "Documents" tab to "Local Files" please.
  11. Add a progress bar. Now that there is a pre-load and a "GO" button, the UI should now have a quantity of files and a size of files to use as a basis for two versions of percent completion. Please add a percent complete and use one decimal place, e.g. 15.3% of Files Uploaded, 43.2% of Data Uploaded.
  12. I would like a progress field that explicitly states what file is being processed, e.g. "C:\DOC\aligator.docx 23 MB, 10/7/2001" - if the user notices the same doc on his screen for an hour, he can guess that there is an issue with it. Listing its file size would sort of tip him off if it was a mistake.
  13. I would like to know if a document needed a specialized processor, what kind, and what it's doing. So an .mp4 that needs ffmpeg or an audio file that needs a speech recognition app or a pdf needs an OCR - tell the user what's going on.
  14. I would like the end of the progress filed to show a total size of the JSON file "database" of documents.
  15. I would like a timer that indicates the length of time the upload has taken.
  16. I would like all of the same capabilities when the documents are embedded or moved to Workspace and vectors are generated. This is seemingly more important, as the database part of the RAG and all of the parameters, tokens, and parsing are all complicated ... and time-consuming.
  17. I would also suggest a way to see the QUEUE files with their full URN or path. A directory tree of each file's origin. Does AnythingLLM sort of forget where files came from? If it doesn't, then I would like to see this list in the full form, pathnames and all. I could export that to a corpus1.txt file and then make minor changes to it for version 2. Perhaps the queue file could be stored in the documents folder as such a file. You could get all fancy and put it in a .csv or .json with the SHA-256 hash, but the user would have no need for that. Just the file list. And the next time I want to start a new LLM, perhaps I could just import this on the Documents page?
  18. Can you put up memory and CPU usage stats? The OS can do this, too, but it seems like these processes are heavily CPU and memory bound. Some insights could help pinpoint some of these issues. A user might want to put a lengthy upload or embed process into a background process to work or develop something else.
  19. I can't tell if there is a self-healing aspect to the Document window. If a crash or reboot occurs, can there be a way to inform the user the next time he runs the program that a Document Upload or Embed was in progress? And in that window, the GO button now has a "RESET" in red and a "RESUME" button in green? A saved state file could serve as the key to resuming, but only if it has a way to delete it and start over. On this same topic, there should be a way to delete all files embedded or uploaded from this screen in case the user detects an error or corrupt database.
  20. And finally, I would like to know about the most recent error or warning. If a file failed to be opened because it's empty, has a DRM-protected stream, a password, or of the OCR couldn't parse Esperanto, I want to know. Also files that have been locked or deleted, ignored because they are .exe or .dll files, unreadable/corrupted gibberish, or if it's got the same SHA-256 as another file in the database.
Originally created by @uruiamme on GitHub (Dec 12, 2025). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/4776 ### What would you like to see? For those with a large corpus, the Upload Document window and its User Interface (UI) become the primary interface with our local LLM besides the query. [This reviewer in on Windows] On Desktop, this window has a file picker on the left and an upload window on the right or down below. The RAG and collector seem to just use the Upload Document screen, but it currently has some room for UI improvements to give the user insight into the tasks involved. Main reference: [https://docs.anythingllm.com/chatting-with-documents/rag-in-anythingllm](https://docs.anythingllm.com/chatting-with-documents/rag-in-anythingllm) 1. My suggestions concern the File Picker: The current UI's file picker has no way to paste in a URN, Folder, File name, a list of files, or a wildcard. It is all based on selecting the file(s). A drag-and-drop feature is also available. But using this feature immediately begins the process of uploading, which blocks further operations until this process is done. There's no _easy_ way to import or export the list of files selected for uploading, nor to exclude unwanted files based on file size or file type - we are at the mercy of the picker and the OS **if our corpus is large and complicated.** 2. My suggestion is to **enforce a "GO" button** to explicitly tell the Upload process to begin. This will allow the user to select files three different ways: (1) with the picker, (2) dropping them, (3) using a text field that can accept a clipboard paste that intelligently selects files, folders, and knows wildcards. (e.g. `C:\DOCS\*.pdf`) The user does any or all three of these things, often several times, and then presses "GO" or "UPLOAD" and can watch the progress. Files in this area are in a QUEUE until the GO button is pressed. Advice can be given for a queue larger than a setpoint, like 1 GB, to suggest a smaller size or certain kind of LLM/RAG combinations. 3. I would like to be able to **modify an upload process.** There are already checkboxes. Perhaps each folder or file can have a checkbox that toggles red and green. While uploading, the checkboxes change to a 3-color scheme where yellow check and gray text means it has not uploaded and green with black text says that it is done. If a user decides not to upload the file, he selects the box, it turns red, and the filename turns red. The GO button doesn't prevent work being done in other portions of the window. 4. I would like the **file list** to show Icon, Filename, Extension or Type, and File size in abbreviated form. Files above a certain setpoint (20 MB default) have a red checkbox and red text and will not be uploaded. Icons for PDF, Text, docs, videos, and spreadsheets should use standard colors and familiar shapes, 5. Settings on the Upload Document window should include: **Maximum size** of text/docs, PDFs, videos, and spreadsheets (each defaults to 20 MB or 100 MB but has a field to adjust this). Without this, the user who accidentally selects a few 100 MB .doc or 500 MB .pdf files will be a bit confused why the process is taking so long. And if a video or audio file is inadvertently in the corpus, this could change the upload time a lot. 6. **When a user selects several files and folders**, a quick list of what has been selected must appear before he selects "GO." A list of 233 pdfs, 27 docs, 14,888 jpgs, and 47 .mp4 files that total 250 GB might be accidental. Perhaps the user didn't mean to have any images or any videos? He needs to know the breakdown, so a prelimary thread reading and parsing the intended upload needs to be presented next to the "GO" button. This is where advice about avoiding large media files should be mentioned! 7. **No mention is made of [document sync](https://docs.anythingllm.com/beta-preview/active-features/live-document-sync#how-does-document-sync-work-with-local-files) on this page**. I think it should be mentioned at this point in the UI even if it is experimental. 8. **No mention is made of chunking** on this page. I believe it is relevant enough to mention it and explain where to change this setting. 9. I'm confused by the "Fetch website" button, especially with no help about how this might work on a document upload page. Please move it to its own tab like Data Connectors. 10. **Rename "Documents" tab to "Local Files" please.** 11. **Add a progress bar.** Now that there is a pre-load and a "GO" button, the UI should now have a quantity of files and a size of files to use as a basis for two versions of percent completion. Please add a percent complete and use one decimal place, e.g. 15.3% of Files Uploaded, 43.2% of Data Uploaded. 12. I would like a progress field that explicitly states what file is being processed, e.g. "C:\DOC\aligator.docx 23 MB, 10/7/2001" - if the user notices the same doc on his screen for an hour, he can guess that there is an issue with it. Listing its file size would sort of tip him off if it was a mistake. 13. I would like to know if a document needed a specialized processor, what kind, and what it's doing. So an .mp4 that needs ffmpeg or an audio file that needs a speech recognition app or a pdf needs an OCR - tell the user what's going on. 14. I would like the end of the progress filed to show a total size of the JSON file "database" of documents. 15. I would like a timer that indicates the length of time the upload has taken. 16. I would like all of the **same capabilities** when the documents are embedded or moved to Workspace and vectors are generated. This is seemingly more important, as the database part of the RAG and all of the parameters, tokens, and parsing are all complicated ... and time-consuming. 17. I would also suggest a way to see the QUEUE files with their full URN or path. A directory tree of each file's origin. Does AnythingLLM sort of forget where files came from? If it doesn't, then I would like to see this list in the full form, pathnames and all. I could export that to a `corpus1.txt` file and then make minor changes to it for version 2. Perhaps the queue file could be stored in the documents folder as such a file. You could get all fancy and put it in a .csv or .json with the SHA-256 hash, but the user would have no need for that. Just the file list. And the next time I want to start a new LLM, perhaps I could just import this on the Documents page? 18. Can you put up memory and CPU usage stats? The OS can do this, too, but it seems like these processes are heavily CPU and memory bound. Some insights could help pinpoint some of these issues. A user might want to put a lengthy upload or embed process into a background process to work or develop something else. 19. I can't tell if there is a **self-healing aspect** to the Document window. If a crash or reboot occurs, can there be a way to inform the user the next time he runs the program that a Document Upload or Embed was in progress? And in that window, the GO button now has a "RESET" in red and a "RESUME" button in green? A saved state file could serve as the key to resuming, but only if it has a way to delete it and start over. On this same topic, there should be a way to delete all files embedded or uploaded from this screen in case the user detects an error or corrupt database. 20. And finally, I would like to know about the most recent error or warning. If a file failed to be opened because it's empty, has a DRM-protected stream, a password, or of the OCR couldn't parse Esperanto, I want to know. Also files that have been locked or deleted, ignored because they are .exe or .dll files, unreadable/corrupted gibberish, or if it's got the same SHA-256 as another file in the database.
yindo added the enhancementfeature request labels 2026-02-22 18:32:14 -05:00
yindo closed this issue 2026-02-22 18:32:14 -05:00
yindo commented 2026-02-22 18:32:14 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Dec 13, 2025):

I have read all of this, totally understand it, and have taken notes for our internal board. I'm only closing this issue because it is a massive wall of text. It's great feedback, not a great GitHub issue.

@timothycarambat commented on GitHub (Dec 13, 2025): I have read all of this, totally understand it, and have taken notes for our internal board. I'm only closing this issue because it is a massive wall of text. It's great feedback, not a great GitHub issue.
yindo commented 2026-02-22 18:32:15 -05:00
Author
Owner
Copy Link
Copy Source

@uruiamme commented on GitHub (Dec 13, 2025):

Thank you for considering these UI improvements. I love the program so far!

@uruiamme commented on GitHub (Dec 13, 2025): Thank you for considering these UI improvements. I love the program so far!
yindo changed title from [FEAT]: UI of RAG Document window on Desktop needs improvements for large corpus of files to [GH-ISSUE #4776] [FEAT]: UI of RAG Document window on Desktop needs improvements for large corpus of files 2026-06-05 14:49:47 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label enhancement feature request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#3009
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?