Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #5003] [BUG] GitLab Connector: Infinite pagination loop on large repos / JSON crash on Rate Limit #3142

New Issue
Closed
opened 2026-02-22 18:32:47 -05:00 by yindo · 1 comment
yindo commented 2026-02-22 18:32:47 -05:00
Owner
Copy Link
Copy Source

Originally created by @YokoAC on GitHub (Feb 15, 2026).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/5003

Originally assigned to: @angelplusultra on GitHub.

How are you running AnythingLLM?

Local development

What happened?

I have encountered two issues when using the GitLab Connector with large repositories (>10k files):

  1. Infinite Loop (Pagination Bug): GitLab stops sending the x-total-pages header for large repositories for performance reasons (https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/23931, https://gitlab.com/gitlab-org/gitlab/-/issues/361362). The current logic relies on this header to detect the last page. When it is missing, the connector resets to Page 1 and loops forever.
  2. Crash on Rate Limit: The connector attempts to download all files in a page simultaneously using Promise.all. This triggers GitLab's HTTP 429 (Rate Limit) protection. The API returns a plain text error ("Rate limit exceeded"), which causes JSON.parse() to throw a SyntaxError: Unexpected token 'R', crashing the collector.

This is also happening in Docker and using a private Repo.

Are there known steps to reproduce?

I have chosen https://gitlab.com/qemu-project/qemu just as an example, because I was searching for a publicly available repository with more than 10k files. The same would happen for https://gitlab.com/gitlab-org/gitlab-foss.

  1. Start the Collector: Run the collector service (e.g., yarn dev:collector or via Docker).
  2. Configure GitLab Connector:
  3. Open a Workspace in AnythingLLM. Go to the Connectors and select Gitlab. Repo URL: Enter a large public repository URL (e.g., https://gitlab.com/qemu-project/qemu)). Access Token: use a Personal Access Token . (Note: Using no token hits rate limits faster, triggering the crash sooner).
  4. Trigger the Bug by clicking "submit" to start the process
  5. Observe the Logs:
  • Crash: The collector logs SyntaxError: Unexpected token 'R' (due to unhandled 429 Rate Limit response).
  • Infinite Loop: If it survives the crash, it reaches the last page (e.g., Page 118) and then restarts at Page 1 because x-total-pages is missing.

The endless loop. It starts with the following. You can already see, that the x-total-pages is 0 after the first message.
Image

Upon reaching the end, it continues downloading, starting right from the beginning.
Image

I made a quick attempt to solve the issue, which would require some further cleanup: anything-llm\collector\utils\extensions\RepoLoader\GitlabRepo\RepoLoader\index.js:
index.js

Two adjustments for the fetchNextPage function:

  1. Only relay on x-next-page
  2. Check for rate-limit response. If reached, wait a minute, so the limit gets refreshed, then continue downloading (here some smaller changes are still required)

With those changes, I can download all files.

If desired, I could try to clean this up and provide a proper PR.

Originally created by @YokoAC on GitHub (Feb 15, 2026). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/5003 Originally assigned to: @angelplusultra on GitHub. ### How are you running AnythingLLM? Local development ### What happened? I have encountered two issues when using the GitLab Connector with large repositories (>10k files): 1. **Infinite Loop (Pagination Bug)**: GitLab stops sending the x-total-pages header for large repositories for performance reasons (https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/23931, https://gitlab.com/gitlab-org/gitlab/-/issues/361362). The current logic relies on this header to detect the last page. When it is missing, the connector resets to Page 1 and loops forever. 2. **Crash on Rate Limit**: The connector attempts to download all files in a page simultaneously using Promise.all. This triggers GitLab's HTTP 429 (Rate Limit) protection. The API returns a plain text error ("Rate limit exceeded"), which causes JSON.parse() to throw a SyntaxError: Unexpected token 'R', crashing the collector. This is also happening in Docker and using a private Repo. ### Are there known steps to reproduce? I have chosen https://gitlab.com/qemu-project/qemu just as an example, because I was searching for a publicly available repository with more than 10k files. The same would happen for https://gitlab.com/gitlab-org/gitlab-foss. 1. Start the Collector: Run the collector service (e.g., yarn dev:collector or via Docker). 2. Configure GitLab Connector: 3. Open a Workspace in AnythingLLM. Go to the Connectors and select Gitlab. Repo URL: Enter a large public repository URL (e.g., https://gitlab.com/qemu-project/qemu)). Access Token: use a Personal Access Token . (Note: Using no token hits rate limits faster, triggering the crash sooner). 4. Trigger the Bug by clicking "submit" to start the process 5. Observe the Logs: - Crash: The collector logs SyntaxError: Unexpected token 'R' (due to unhandled 429 Rate Limit response). - Infinite Loop: If it survives the crash, it reaches the last page (e.g., Page 118) and then restarts at Page 1 because x-total-pages is missing. The endless loop. It starts with the following. You can already see, that the x-total-pages is 0 after the first message. <img width="981" height="171" alt="Image" src="https://github.com/user-attachments/assets/fce1d057-c175-4762-8f3f-2cd32e7bb2df" /> Upon reaching the end, it continues downloading, starting right from the beginning. <img width="979" height="186" alt="Image" src="https://github.com/user-attachments/assets/22a20845-4bf4-4fe3-a5a7-e0dc1acc25da" /> I made a quick attempt to solve the issue, which would require some further cleanup: anything-llm\collector\utils\extensions\RepoLoader\GitlabRepo\RepoLoader\index.js: [index.js](https://github.com/user-attachments/files/25326714/index.js) Two adjustments for the _fetchNextPage_ function: 1. Only relay on x-next-page 2. Check for rate-limit response. If reached, wait a minute, so the limit gets refreshed, then continue downloading (here some smaller changes are still required) With those changes, I can download all files. If desired, I could try to clean this up and provide a proper PR.
yindo added the possible bugcore-team-only labels 2026-02-22 18:32:47 -05:00
yindo closed this issue 2026-02-22 18:32:47 -05:00
yindo commented 2026-02-22 18:32:48 -05:00
Author
Owner
Copy Link
Copy Source

@jheeanny commented on GitHub (Feb 17, 2026):

Excellent detailed analysis! You’ve correctly identified both root causes:

  1. Pagination — GitLab’s omission of x-total-pages for large repos makes relying on it unsafe. Switching to x-next-page is the right approach.
  2. Rate limit 429 — The plain-text body should be detected and parsed safely, not fed to JSON.parse.

Your proposed changes seem spot‑on. I’d additionally suggest:

  • Add a maximum retry count for rate‑limit handling to avoid potential endless loops
  • Use exponential backoff (e.g., 1s → 2s → 4s…) for retries
  • Add unit tests for pagination termination and 429 handling to prevent regressions

Would you be willing to open a PR with these fixes? The maintainers would likely welcome this. If you need help polishing the PR or writing tests, let me know.

@jheeanny commented on GitHub (Feb 17, 2026): Excellent detailed analysis! You’ve correctly identified both root causes: 1. **Pagination** — GitLab’s omission of `x-total-pages` for large repos makes relying on it unsafe. Switching to `x-next-page` is the right approach. 2. **Rate limit 429** — The plain-text body should be detected and parsed safely, not fed to `JSON.parse`. Your proposed changes seem spot‑on. I’d additionally suggest: - Add a maximum retry count for rate‑limit handling to avoid potential endless loops - Use exponential backoff (e.g., 1s → 2s → 4s…) for retries - Add unit tests for pagination termination and 429 handling to prevent regressions Would you be willing to open a PR with these fixes? The maintainers would likely welcome this. If you need help polishing the PR or writing tests, let me know.
yindo changed title from [BUG] GitLab Connector: Infinite pagination loop on large repos / JSON crash on Rate Limit to [GH-ISSUE #5003] [BUG] GitLab Connector: Infinite pagination loop on large repos / JSON crash on Rate Limit 2026-06-05 14:50:32 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label core-team-only possible bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#3142
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?