Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[PR #2355] [CLOSED] #2317 Reduce workspace popup loading time by 98% when there are many files #3958

New Issue
Closed
opened 2026-02-22 18:34:50 -05:00 by yindo · 0 comments
yindo commented 2026-02-22 18:34:50 -05:00
Owner
Copy Link
Copy Source

📋 Pull Request Information

Original PR: https://github.com/Mintplex-Labs/anything-llm/pull/2355
Author: @blazeyo
Created: 9/23/2024
Status: ❌ Closed

Base: master ← Head: 2317-fix-document-popup-for-large-number-of-files

📝 Commits (3)

  • 4c373f3 #2317 Fetch pinned documents once per folder to reduce the number of queries.
  • 76f7e80 Reorder the lines to keeps const declarations together.
  • 9fd0113 Merge branch 'master' into 2317-fix-document-popup-for-large-number-of-files

📊 Changes

2 files changed (+38 additions, -6 deletions)

View changed files

📝 server/models/documents.js (+3 -1)
📝 server/utils/files/index.js (+35 -5)

📄 Description

Pull Request Type

  • ✨ feat
  • 🐛 fix
  • ♻️ refactor
  • 💄 style
  • 🔨 chore
  • 📝 docs

Relevant Issues

connect #2317

What is in this change?

Reduces the loading time of the documents popup by fetching pinned documents once per folder.

Results on my local instance with ~14k files:

  • loading time
    • before: 105 seconds
    • after: 2 seconds
  • number of queries for pinned documents:
    • before: ~14k
    • after: 8

Before

[0.05ms] Checked/created documentsPath
[1.28ms] Checked DocumentSyncQueue.enabled()
[2.29ms] Read subfolder: custom-documents
[122.83ms] Read subfolder: folder1
[205.92ms] Read subfolder: folder2
[301.90ms] Read subfolder: folder3
[336.27ms] Read subfolder: folder4
[35961.37ms] Read subfolder: folder5
[49814.86ms] Read subfolder: folder6
[51723.69ms] Read subfolder: folder7
[105922.62ms] Finished processing all folders and files

Function call timings summary:
  readFile: 1939.09ms
  parseJSON: 694.86ms
  getPinnedWorkspaces: 100892.30ms
  getWatchedWorkspaces: 6.67ms
  checkCachedVector: 1971.95ms

After

[0.05ms] Checked/created documentsPath
[11.17ms] Checked DocumentSyncQueue.enabled()
[12.08ms] Read subfolder: custom-documents
[54.88ms] Read subfolder: folder1
[65.54ms] Read subfolder: folder2
[74.09ms] Read subfolder: folder3
[90.14ms] Read subfolder: folder4
[648.97ms] Read subfolder: folder5
[781.37ms] Read subfolder: folder6
[817.45ms] Read subfolder: folder7
[1926.20ms] Finished processing all folders and files

Function call timings summary:
  readFile: 734.33ms
  parseJSON: 398.08ms
  getPinnedWorkspaces: 328.55ms
  getWatchedWorkspaces: 1.58ms
  checkCachedVector: 288.89ms

Additional Information

An additional optional parameter select has been added to Document.where.

Developer Validations

  • I ran yarn lint from the root of the repo & committed changes
  • Relevant documentation has been updated
  • I have tested my code functionality
  • Docker build succeeds locally

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/Mintplex-Labs/anything-llm/pull/2355 **Author:** [@blazeyo](https://github.com/blazeyo) **Created:** 9/23/2024 **Status:** ❌ Closed **Base:** `master` ← **Head:** `2317-fix-document-popup-for-large-number-of-files` --- ### 📝 Commits (3) - [`4c373f3`](https://github.com/Mintplex-Labs/anything-llm/commit/4c373f30705deb005d88f4f1decc07c8b95a8278) #2317 Fetch pinned documents once per folder to reduce the number of queries. - [`76f7e80`](https://github.com/Mintplex-Labs/anything-llm/commit/76f7e800793839e04fe9f74701189545d8f37b09) Reorder the lines to keeps const declarations together. - [`9fd0113`](https://github.com/Mintplex-Labs/anything-llm/commit/9fd0113443125373f6c47007fdabb8178e5fc3e8) Merge branch 'master' into 2317-fix-document-popup-for-large-number-of-files ### 📊 Changes **2 files changed** (+38 additions, -6 deletions) <details> <summary>View changed files</summary> 📝 `server/models/documents.js` (+3 -1) 📝 `server/utils/files/index.js` (+35 -5) </details> ### 📄 Description ### Pull Request Type <!-- For change type, change [ ] to [x]. --> - [ ] ✨ feat - [x] 🐛 fix - [ ] ♻️ refactor - [ ] 💄 style - [ ] 🔨 chore - [ ] 📝 docs ### Relevant Issues <!-- Use "resolves #xxx" to auto resolve on merge. Otherwise, please use "connect #xxx" --> connect #2317 ### What is in this change? Reduces the loading time of the documents popup by fetching pinned documents once per folder. Results on my local instance with ~14k files: - loading time - before: 105 seconds - after: 2 seconds - number of queries for pinned documents: - before: ~14k - after: 8 #### Before ``` [0.05ms] Checked/created documentsPath [1.28ms] Checked DocumentSyncQueue.enabled() [2.29ms] Read subfolder: custom-documents [122.83ms] Read subfolder: folder1 [205.92ms] Read subfolder: folder2 [301.90ms] Read subfolder: folder3 [336.27ms] Read subfolder: folder4 [35961.37ms] Read subfolder: folder5 [49814.86ms] Read subfolder: folder6 [51723.69ms] Read subfolder: folder7 [105922.62ms] Finished processing all folders and files Function call timings summary: readFile: 1939.09ms parseJSON: 694.86ms getPinnedWorkspaces: 100892.30ms getWatchedWorkspaces: 6.67ms checkCachedVector: 1971.95ms ``` #### After ``` [0.05ms] Checked/created documentsPath [11.17ms] Checked DocumentSyncQueue.enabled() [12.08ms] Read subfolder: custom-documents [54.88ms] Read subfolder: folder1 [65.54ms] Read subfolder: folder2 [74.09ms] Read subfolder: folder3 [90.14ms] Read subfolder: folder4 [648.97ms] Read subfolder: folder5 [781.37ms] Read subfolder: folder6 [817.45ms] Read subfolder: folder7 [1926.20ms] Finished processing all folders and files Function call timings summary: readFile: 734.33ms parseJSON: 398.08ms getPinnedWorkspaces: 328.55ms getWatchedWorkspaces: 1.58ms checkCachedVector: 288.89ms ``` ### Additional Information An additional optional parameter `select` has been added to `Document.where`. ### Developer Validations <!-- All of the applicable items should be checked. --> - [x] I ran `yarn lint` from the root of the repo & committed changes - [x] Relevant documentation has been updated - [x] I have tested my code functionality - [x] Docker build succeeds locally --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
yindo added the pull-request label 2026-02-22 18:34:51 -05:00
yindo closed this issue 2026-02-22 18:34:51 -05:00
yindo changed title from [PR #2355] #2317 Reduce workspace popup loading time by 98% when there are many files to [PR #2355] [CLOSED] #2317 Reduce workspace popup loading time by 98% when there are many files 2026-06-05 15:16:05 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#3958
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?