Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[PR #2335] [MERGED] Added an option to fetch issues from gitlab. Made the file fetching a… #3954

New Issue
Closed
opened 2026-02-22 18:34:50 -05:00 by yindo · 0 comments
yindo commented 2026-02-22 18:34:50 -05:00
Owner
Copy Link
Copy Source

📋 Pull Request Information

Original PR: https://github.com/Mintplex-Labs/anything-llm/pull/2335
Author: @blazeyo
Created: 9/20/2024
Status: ✅ Merged
Merged: 9/26/2024
Merged by: @timothycarambat

Base: master ← Head: 2334-fetch-issues-from-gitlab

📝 Commits (8)

  • 4e4fc08 Added an option to fetch issues from gitlab. Made the file fetching asynchornous to improve performance. #2334
  • 0673d41 Fixed a typo in loadGitlabRepo.
  • 3ac0cad Convert issues to markdown.
  • 6d14e64 Fixed an issue with time estimate field names in issueToMarkdown.
  • 2b14628 Merge branch 'master' into 2334-fetch-issues-from-gitlab
  • 5eebbf2 handle rate limits more gracefully + update checkbox to toggle switch
  • e83b989 lint
  • b66e057 Merge branch 'master' into 2334-fetch-issues-from-gitlab

📊 Changes

4 files changed (+328 additions, -110 deletions)

View changed files

📝 collector/utils/extensions/RepoLoader/GitlabRepo/RepoLoader/index.js (+175 -101)
📝 collector/utils/extensions/RepoLoader/GitlabRepo/index.js (+114 -7)
📝 frontend/src/components/Modals/ManageWorkspace/DataConnectors/Connectors/Gitlab/index.jsx (+25 -0)
📝 frontend/src/models/dataConnector.js (+14 -2)

📄 Description

…synchornous to improve performance. #2334

Pull Request Type

  • ✨ feat
  • 🐛 fix
  • ♻️ refactor
  • 💄 style
  • 🔨 chore
  • 📝 docs

Relevant Issues

connect #812
resolves #2334

What is in this change?

  • New "Fetch Issues" Checkbox: Adds an option on the GitLab connector page to fetch all project issues, including associated discussion items (such as comments, assignee changes, etc.).

    Selection_223

  • New fetchNextPage Method: Implements a fetchNextPage method in GitlabRepoLoader to streamline the process of fetching all pages from an API endpoint in a more generic and reusable way.

  • Refactoring: Refactors the getRepoBranches and fetchFilesRecursive methods to utilize the new fetchNextPage logic.

  • Speed Improvements: File fetching is now performed in parallel, resulting in a substantial performance boost—improving speed by an order of magnitude.

Additional Information

There are a few areas that would benefit from further discussion:

  1. EDIT: no longer current. The issues are converted to markdown now.
  2. Page Size Configuration
  • Concurrent fetching significantly boosts performance, but it can strain system resources, particularly if GitLab is hosted on a less powerful server.
  • During testing on a GitLab instance with 8 cores and 16GB of RAM, I fetched a repository containing 6.5k files and 1.7k issues (with up to 150 discussion items each) using 100 items per page. While this worked well, the server's average 5-minute load reached 10.
  • It might be worth considering making the pageSize parameter configurable to allow for smaller page sizes (e.g., 25 items per page) on less capable servers.
  1. Chunk Source & Repository URL
  • Currently, the generateChunkSource function does not include the repository URL in its payload. This might be necessary for the "Automatic Document Content Sync" feature, particularly for self-hosted GitLab instances.
  • If the repo URL is indeed required for this feature to work, I am happy to open an issue and submit a separate PR to address this.

Developer Validations

  • I ran yarn lint from the root of the repo & committed changes
  • Relevant documentation has been updated
  • I have tested my code functionality
  • Docker build succeeds locally

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/Mintplex-Labs/anything-llm/pull/2335 **Author:** [@blazeyo](https://github.com/blazeyo) **Created:** 9/20/2024 **Status:** ✅ Merged **Merged:** 9/26/2024 **Merged by:** [@timothycarambat](https://github.com/timothycarambat) **Base:** `master` ← **Head:** `2334-fetch-issues-from-gitlab` --- ### 📝 Commits (8) - [`4e4fc08`](https://github.com/Mintplex-Labs/anything-llm/commit/4e4fc08f268b89a9f54f3cf1e42c342f854c3bfb) Added an option to fetch issues from gitlab. Made the file fetching asynchornous to improve performance. #2334 - [`0673d41`](https://github.com/Mintplex-Labs/anything-llm/commit/0673d4183ae88e8ff77ca630b2e9e8ad9a21b9f0) Fixed a typo in loadGitlabRepo. - [`3ac0cad`](https://github.com/Mintplex-Labs/anything-llm/commit/3ac0cadc98f0214f4a88e9766fa499ad985bb234) Convert issues to markdown. - [`6d14e64`](https://github.com/Mintplex-Labs/anything-llm/commit/6d14e64a648c42b291ab1134d8fe03b4c16b7802) Fixed an issue with time estimate field names in issueToMarkdown. - [`2b14628`](https://github.com/Mintplex-Labs/anything-llm/commit/2b146285fdddba5ee53eb98a78aee1e1a722bb7b) Merge branch 'master' into 2334-fetch-issues-from-gitlab - [`5eebbf2`](https://github.com/Mintplex-Labs/anything-llm/commit/5eebbf2183323d6590b6f3af8c81867b114001cf) handle rate limits more gracefully + update checkbox to toggle switch - [`e83b989`](https://github.com/Mintplex-Labs/anything-llm/commit/e83b9892632e05cb533ba34473e29caac5dc307c) lint - [`b66e057`](https://github.com/Mintplex-Labs/anything-llm/commit/b66e05754e2c0dd1b06d4c5ab99f6009d62a7b48) Merge branch 'master' into 2334-fetch-issues-from-gitlab ### 📊 Changes **4 files changed** (+328 additions, -110 deletions) <details> <summary>View changed files</summary> 📝 `collector/utils/extensions/RepoLoader/GitlabRepo/RepoLoader/index.js` (+175 -101) 📝 `collector/utils/extensions/RepoLoader/GitlabRepo/index.js` (+114 -7) 📝 `frontend/src/components/Modals/ManageWorkspace/DataConnectors/Connectors/Gitlab/index.jsx` (+25 -0) 📝 `frontend/src/models/dataConnector.js` (+14 -2) </details> ### 📄 Description …synchornous to improve performance. #2334 ### Pull Request Type <!-- For change type, change [ ] to [x]. --> - [x] ✨ feat - [ ] 🐛 fix - [x] ♻️ refactor - [ ] 💄 style - [ ] 🔨 chore - [ ] 📝 docs ### Relevant Issues <!-- Use "resolves #xxx" to auto resolve on merge. Otherwise, please use "connect #xxx" --> connect #812 resolves #2334 ### What is in this change? <!-- Describe the changes in this PR that are impactful to the repo. --> - **New "Fetch Issues" Checkbox**: Adds an option on the GitLab connector page to fetch all project issues, including associated discussion items (such as comments, assignee changes, etc.). ![Selection_223](https://github.com/user-attachments/assets/2cb41033-2a5c-476c-99e6-690fddc0a0a1) - **New `fetchNextPage` Method**: Implements a `fetchNextPage` method in `GitlabRepoLoader` to streamline the process of fetching all pages from an API endpoint in a more generic and reusable way. - **Refactoring**: Refactors the `getRepoBranches` and `fetchFilesRecursive` methods to utilize the new `fetchNextPage` logic. - **Speed Improvements**: File fetching is now performed in parallel, resulting in a substantial performance boost—improving speed by an order of magnitude. ### Additional Information There are a few areas that would benefit from further discussion: 1. EDIT: no longer current. The issues are converted to markdown now. 2. **Page Size Configuration** - Concurrent fetching significantly boosts performance, but it can strain system resources, particularly if GitLab is hosted on a less powerful server. - During testing on a GitLab instance with 8 cores and 16GB of RAM, I fetched a repository containing 6.5k files and 1.7k issues (with up to 150 discussion items each) using 100 items per page. While this worked well, the server's average 5-minute load reached 10. - It might be worth considering making the pageSize parameter configurable to allow for smaller page sizes (e.g., 25 items per page) on less capable servers. 3. **Chunk Source & Repository URL** - Currently, [the generateChunkSource function](https://github.com/Mintplex-Labs/anything-llm/blob/5a3d55db671359613a5d68d98eb7166ca6db4188/collector/utils/extensions/RepoLoader/GitlabRepo/index.js#L133) does not include the repository URL in its payload. This might be necessary for the "Automatic Document Content Sync" feature, particularly for self-hosted GitLab instances. - If the repo URL is indeed required for this feature to work, I am happy to open an issue and submit a separate PR to address this. ### Developer Validations <!-- All of the applicable items should be checked. --> - [x] I ran `yarn lint` from the root of the repo & committed changes - [x] Relevant documentation has been updated - [x] I have tested my code functionality - [x] Docker build succeeds locally --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
yindo added the pull-request label 2026-02-22 18:34:50 -05:00
yindo closed this issue 2026-02-22 18:34:50 -05:00
yindo changed title from [PR #2335] Added an option to fetch issues from gitlab. Made the file fetching a… to [PR #2335] [MERGED] Added an option to fetch issues from gitlab. Made the file fetching a… 2026-06-05 15:16:04 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#3954
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?