Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[PR #5021] [MERGED] fix: GitLab connector infinite loop and rate limit crash for large repos #4865

New Issue
Closed
opened 2026-02-22 18:36:41 -05:00 by yindo · 0 comments
yindo commented 2026-02-22 18:36:41 -05:00
Owner
Copy Link
Copy Source

📋 Pull Request Information

Original PR: https://github.com/Mintplex-Labs/anything-llm/pull/5021
Author: @angelplusultra
Created: 2/18/2026
Status: ✅ Merged
Merged: 2/19/2026
Merged by: @timothycarambat

Base: master ← Head: 5003-bug-gitlab-infinite-loop-and-rate-limit-crash

📝 Commits (4)

  • be18d97 Fix infinite loop and rate limit crashes
  • 7d95950 simplify logic | add max-retries to fetchNextPage and fetchSingleFileContents
  • e704d07 Merge branch 'master' into 5003-bug-gitlab-infinite-loop-and-rate-limit-crash
  • 5a33a10 Merge branch 'master' into 5003-bug-gitlab-infinite-loop-and-rate-limit-crash

📊 Changes

1 file changed (+66 additions, -26 deletions)

View changed files

📝 collector/utils/extensions/RepoLoader/GitlabRepo/RepoLoader/index.js (+66 -26)

📄 Description

Pull Request Type

  • ✨ feat (New feature)
  • 🐛 fix (Bug fix)
  • ♻️ refactor (Code refactoring without changing behavior)
  • 💄 style (UI style changes)
  • 🔨 chore (Build, CI, maintenance)
  • 📝 docs (Documentation updates)

Relevant Issues

resolves #5003

Description

Fixes two bugs in the GitLab connector that occur with large repositories (>10k files):

  1. Infinite loop (pagination): GitLab omits the x-total-pages header for large repos for performance reasons. The old code relied on this header to detect the last page. When missing, pagination reset to page 1 and looped forever. Fixed by using x-next-page as the sole pagination signal — when it's empty, we're on the last page.

  2. Crash on rate limit: The old code checked for status 401 (Unauthorized) instead of 429 (Rate Limit), and called response.json() unconditionally. When GitLab returned a plain-text 429 response like "Rate limit exceeded", JSON.parse threw a SyntaxError, crashing the collector. Fixed by handling 429 in both fetchNextPage and fetchSingleFileContents with retry logic that respects the retry-after header (defaulting to 60s, capped at 3 retries).

Additional Information

  • Also fixed the 401 log message which incorrectly said "Rate limit hit" — it now correctly says "Unauthorized".
  • Added non-OK status guard before response.json() to prevent JSON parse errors on any unexpected error response.

Developer Validations

  • I ran yarn lint from the root of the repo & committed changes
  • Relevant documentation has been updated (if applicable)
  • I have tested my code functionality
  • Docker build succeeds locally

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/Mintplex-Labs/anything-llm/pull/5021 **Author:** [@angelplusultra](https://github.com/angelplusultra) **Created:** 2/18/2026 **Status:** ✅ Merged **Merged:** 2/19/2026 **Merged by:** [@timothycarambat](https://github.com/timothycarambat) **Base:** `master` ← **Head:** `5003-bug-gitlab-infinite-loop-and-rate-limit-crash` --- ### 📝 Commits (4) - [`be18d97`](https://github.com/Mintplex-Labs/anything-llm/commit/be18d9716298beff26db98c6a6394db5cf7d10b0) Fix infinite loop and rate limit crashes - [`7d95950`](https://github.com/Mintplex-Labs/anything-llm/commit/7d959504d9d492ca9b90d1eecf53018392e99990) simplify logic | add max-retries to fetchNextPage and fetchSingleFileContents - [`e704d07`](https://github.com/Mintplex-Labs/anything-llm/commit/e704d0735c9724178f9eda6d17fd1c326eaeb8ff) Merge branch 'master' into 5003-bug-gitlab-infinite-loop-and-rate-limit-crash - [`5a33a10`](https://github.com/Mintplex-Labs/anything-llm/commit/5a33a10d45dcec15d61ba6c0a22f9c1d74cf6d33) Merge branch 'master' into 5003-bug-gitlab-infinite-loop-and-rate-limit-crash ### 📊 Changes **1 file changed** (+66 additions, -26 deletions) <details> <summary>View changed files</summary> 📝 `collector/utils/extensions/RepoLoader/GitlabRepo/RepoLoader/index.js` (+66 -26) </details> ### 📄 Description ### Pull Request Type <!-- For change type, change [ ] to [x]. --> - [ ] ✨ feat (New feature) - [x] 🐛 fix (Bug fix) - [ ] ♻️ refactor (Code refactoring without changing behavior) - [ ] 💄 style (UI style changes) - [ ] 🔨 chore (Build, CI, maintenance) - [ ] 📝 docs (Documentation updates) ### Relevant Issues <!-- Use "resolves #xxx" to auto resolve on merge. Otherwise, please use "connect #xxx" --> resolves #5003 ### Description Fixes two bugs in the GitLab connector that occur with large repositories (>10k files): 1. **Infinite loop (pagination):** GitLab omits the `x-total-pages` header for large repos for performance reasons. The old code relied on this header to detect the last page. When missing, pagination reset to page 1 and looped forever. Fixed by using `x-next-page` as the sole pagination signal — when it's empty, we're on the last page. 2. **Crash on rate limit:** The old code checked for status `401` (Unauthorized) instead of `429` (Rate Limit), and called `response.json()` unconditionally. When GitLab returned a plain-text 429 response like `"Rate limit exceeded"`, `JSON.parse` threw a `SyntaxError`, crashing the collector. Fixed by handling 429 in both `fetchNextPage` and `fetchSingleFileContents` with retry logic that respects the `retry-after` header (defaulting to 60s, capped at 3 retries). ### Additional Information - Also fixed the `401` log message which incorrectly said "Rate limit hit" — it now correctly says "Unauthorized". - Added non-OK status guard before `response.json()` to prevent JSON parse errors on any unexpected error response. ### Developer Validations <!-- All of the applicable items should be checked. --> - [x] I ran `yarn lint` from the root of the repo & committed changes - [ ] Relevant documentation has been updated (if applicable) - [x] I have tested my code functionality - [ ] Docker build succeeds locally --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
yindo added the pull-request label 2026-02-22 18:36:41 -05:00
yindo closed this issue 2026-02-22 18:36:41 -05:00
yindo changed title from [PR #5021] fix: GitLab connector infinite loop and rate limit crash for large repos to [PR #5021] [MERGED] fix: GitLab connector infinite loop and rate limit crash for large repos 2026-06-05 15:20:46 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#4865
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?