Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[PR #2365] [MERGED] PR#2355 Continued + expanded scope #3962

New Issue
Closed
opened 2026-02-22 18:34:51 -05:00 by yindo · 0 comments
yindo commented 2026-02-22 18:34:51 -05:00
Owner
Copy Link
Copy Source

📋 Pull Request Information

Original PR: https://github.com/Mintplex-Labs/anything-llm/pull/2365
Author: @timothycarambat
Created: 9/24/2024
Status: ✅ Merged
Merged: 9/24/2024
Merged by: @timothycarambat

Base: master ← Head: 2317-fix-document-popup-for-large-number-of-files

📝 Commits (4)

  • 4c373f3 #2317 Fetch pinned documents once per folder to reduce the number of queries.
  • 76f7e80 Reorder the lines to keeps const declarations together.
  • 9fd0113 Merge branch 'master' into 2317-fix-document-popup-for-large-number-of-files
  • 1ea9d33 Add some comments to functions

📊 Changes

3 files changed (+89 additions, -32 deletions)

View changed files

📝 server/models/documentSyncQueue.js (+7 -0)
📝 server/models/documents.js (+3 -16)
📝 server/utils/files/index.js (+79 -16)

📄 Description

Original PR by @blazeyo

Pull Request Type

  • ✨ feat
  • 🐛 fix
  • ♻️ refactor
  • 💄 style
  • 🔨 chore
  • 📝 docs

Relevant Issues

resolves #2317
closes #2355
resolves #928

What is in this change?

Reduces the loading time of the documents popup by fetching pinned documents once per folder.

Results on my local instance with ~14k files:

  • loading time
    • before: 105 seconds
    • after: 2 seconds
  • number of queries for pinned documents:
    • before: ~14k
    • after: 8

Before

[0.05ms] Checked/created documentsPath
[1.28ms] Checked DocumentSyncQueue.enabled()
[2.29ms] Read subfolder: custom-documents
[122.83ms] Read subfolder: folder1
[205.92ms] Read subfolder: folder2
[301.90ms] Read subfolder: folder3
[336.27ms] Read subfolder: folder4
[35961.37ms] Read subfolder: folder5
[49814.86ms] Read subfolder: folder6
[51723.69ms] Read subfolder: folder7
[105922.62ms] Finished processing all folders and files

Function call timings summary:
  readFile: 1939.09ms
  parseJSON: 694.86ms
  getPinnedWorkspaces: 100892.30ms
  getWatchedWorkspaces: 6.67ms
  checkCachedVector: 1971.95ms

After

[0.05ms] Checked/created documentsPath
[11.17ms] Checked DocumentSyncQueue.enabled()
[12.08ms] Read subfolder: custom-documents
[54.88ms] Read subfolder: folder1
[65.54ms] Read subfolder: folder2
[74.09ms] Read subfolder: folder3
[90.14ms] Read subfolder: folder4
[648.97ms] Read subfolder: folder5
[781.37ms] Read subfolder: folder6
[817.45ms] Read subfolder: folder7
[1926.20ms] Finished processing all folders and files

Function call timings summary:
  readFile: 734.33ms
  parseJSON: 398.08ms
  getPinnedWorkspaces: 328.55ms
  getWatchedWorkspaces: 1.58ms
  checkCachedVector: 288.89ms

Additional Information

An additional optional parameter select has been added to Document.where.

Developer Validations

  • I ran yarn lint from the root of the repo & committed changes
  • Relevant documentation has been updated
  • I have tested my code functionality
  • Docker build succeeds locally

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/Mintplex-Labs/anything-llm/pull/2365 **Author:** [@timothycarambat](https://github.com/timothycarambat) **Created:** 9/24/2024 **Status:** ✅ Merged **Merged:** 9/24/2024 **Merged by:** [@timothycarambat](https://github.com/timothycarambat) **Base:** `master` ← **Head:** `2317-fix-document-popup-for-large-number-of-files` --- ### 📝 Commits (4) - [`4c373f3`](https://github.com/Mintplex-Labs/anything-llm/commit/4c373f30705deb005d88f4f1decc07c8b95a8278) #2317 Fetch pinned documents once per folder to reduce the number of queries. - [`76f7e80`](https://github.com/Mintplex-Labs/anything-llm/commit/76f7e800793839e04fe9f74701189545d8f37b09) Reorder the lines to keeps const declarations together. - [`9fd0113`](https://github.com/Mintplex-Labs/anything-llm/commit/9fd0113443125373f6c47007fdabb8178e5fc3e8) Merge branch 'master' into 2317-fix-document-popup-for-large-number-of-files - [`1ea9d33`](https://github.com/Mintplex-Labs/anything-llm/commit/1ea9d33d365f4adb447663cf4db91ddaae88ddf0) Add some comments to functions ### 📊 Changes **3 files changed** (+89 additions, -32 deletions) <details> <summary>View changed files</summary> 📝 `server/models/documentSyncQueue.js` (+7 -0) 📝 `server/models/documents.js` (+3 -16) 📝 `server/utils/files/index.js` (+79 -16) </details> ### 📄 Description Original PR by @blazeyo ### Pull Request Type <!-- For change type, change [ ] to [x]. --> - [ ] ✨ feat - [x] 🐛 fix - [ ] ♻️ refactor - [ ] 💄 style - [ ] 🔨 chore - [ ] 📝 docs ### Relevant Issues <!-- Use "resolves #xxx" to auto resolve on merge. Otherwise, please use "connect #xxx" --> resolves #2317 closes #2355 resolves #928 ### What is in this change? Reduces the loading time of the documents popup by fetching pinned documents once per folder. Results on my local instance with ~14k files: - loading time - before: 105 seconds - after: 2 seconds - number of queries for pinned documents: - before: ~14k - after: 8 #### Before ``` [0.05ms] Checked/created documentsPath [1.28ms] Checked DocumentSyncQueue.enabled() [2.29ms] Read subfolder: custom-documents [122.83ms] Read subfolder: folder1 [205.92ms] Read subfolder: folder2 [301.90ms] Read subfolder: folder3 [336.27ms] Read subfolder: folder4 [35961.37ms] Read subfolder: folder5 [49814.86ms] Read subfolder: folder6 [51723.69ms] Read subfolder: folder7 [105922.62ms] Finished processing all folders and files Function call timings summary: readFile: 1939.09ms parseJSON: 694.86ms getPinnedWorkspaces: 100892.30ms getWatchedWorkspaces: 6.67ms checkCachedVector: 1971.95ms ``` #### After ``` [0.05ms] Checked/created documentsPath [11.17ms] Checked DocumentSyncQueue.enabled() [12.08ms] Read subfolder: custom-documents [54.88ms] Read subfolder: folder1 [65.54ms] Read subfolder: folder2 [74.09ms] Read subfolder: folder3 [90.14ms] Read subfolder: folder4 [648.97ms] Read subfolder: folder5 [781.37ms] Read subfolder: folder6 [817.45ms] Read subfolder: folder7 [1926.20ms] Finished processing all folders and files Function call timings summary: readFile: 734.33ms parseJSON: 398.08ms getPinnedWorkspaces: 328.55ms getWatchedWorkspaces: 1.58ms checkCachedVector: 288.89ms ``` ### Additional Information An additional optional parameter `select` has been added to `Document.where`. ### Developer Validations <!-- All of the applicable items should be checked. --> - [x] I ran `yarn lint` from the root of the repo & committed changes - [x] Relevant documentation has been updated - [x] I have tested my code functionality - [x] Docker build succeeds locally --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
yindo added the pull-request label 2026-02-22 18:34:51 -05:00
yindo closed this issue 2026-02-22 18:34:51 -05:00
yindo changed title from [PR #2365] PR#2355 Continued + expanded scope to [PR #2365] [MERGED] PR#2355 Continued + expanded scope 2026-06-05 15:16:07 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#3962
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?