Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #2243] [BUG]: GitHub Repo Fetch Failed message appears but then proceeds without notification #1463

New Issue
Closed
opened 2026-02-22 18:24:56 -05:00 by yindo · 4 comments
yindo commented 2026-02-22 18:24:56 -05:00
Owner
Copy Link
Copy Source

Originally created by @Sch0field9 on GitHub (Sep 8, 2024).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/2243

How are you running AnythingLLM?

AnythingLLM desktop app

What happened?

When fetching a specific Github repo it hangs a long time after Access token set! Recursive loading enabled!
after a minute or two the message "Fetch failed" appears. Neither the GUI or Logs indicate any further processes. However after doing nothing for another minute the fetch continues and downloads the files from the repo. This does not happen on all repos

[collector] info: [Github Loader]: Access token set! Recursive loading enabled! [backend] info: [CollectorApi] fetch failed [backend] info: [TELEMETRY SENT] {"event":"extension_invoked","distinctId":"372a5230-9bfd-4c05-a653-ae4d165a0c67","properties":{"type":"github_repo","runtime":"desktop"}} [collector] info: [Github Loader]: Found 3952 source files. Saving...

Are there known steps to reproduce?

No response

Originally created by @Sch0field9 on GitHub (Sep 8, 2024). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/2243 ### How are you running AnythingLLM? AnythingLLM desktop app ### What happened? When fetching a specific Github repo it hangs a long time after Access token set! Recursive loading enabled! after a minute or two the message "Fetch failed" appears. Neither the GUI or Logs indicate any further processes. However after doing nothing for another minute the fetch continues and downloads the files from the repo. This does not happen on all repos `[collector] info: [Github Loader]: Access token set! Recursive loading enabled! [backend] info: [CollectorApi] fetch failed [backend] info: [TELEMETRY SENT] {"event":"extension_invoked","distinctId":"372a5230-9bfd-4c05-a653-ae4d165a0c67","properties":{"type":"github_repo","runtime":"desktop"}} [collector] info: [Github Loader]: Found 3952 source files. Saving...` ### Are there known steps to reproduce? _No response_
yindo added the needs info / can't replicate label 2026-02-22 18:24:56 -05:00
yindo closed this issue 2026-02-22 18:24:56 -05:00
yindo commented 2026-02-22 18:24:56 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Sep 9, 2024):

Is this repo public for repro?

Additionally, is there any more content to the logs or does is simply only output [backend] info: [CollectorApi] fetch failed with nothing else in that line?

Given the source have nearly 4K files, I am willing to bet this is related to downloading 4K files via the PAT and it hitting rate limits.

@timothycarambat commented on GitHub (Sep 9, 2024): Is this repo public for repro? Additionally, is there any more content to the logs or does is simply only output `[backend] info: [CollectorApi] fetch failed ` with nothing else in that line? Given the source have nearly 4K files, I am willing to bet this is related to downloading 4K files via the PAT and it hitting rate limits.
yindo commented 2026-02-22 18:24:56 -05:00
Author
Owner
Copy Link
Copy Source

@Sch0field9 commented on GitHub (Sep 9, 2024):

Those are all the logs I have. I couldn't figure out how to change the log level.
The repo I had the troubles with was Github home-assistant/core:master --
(Not sure if it was core)

However, I had something similar happen when I tried it with the AnythinLLM repo. It startet way faster, but it somewhere in the middle was a fetch failed log entry, only one and it continued downloading. The Frontend had the same behaviour tough. Message came up fetch failed and no other indications that it is still downloading files.

I also suspect it has to do with large repos. My bug reports focus is more about that there is no indication to the user, less that why exactly it failed. And possibly a feature request to be able to change the log level? Since there is also no indication in the info logs, that its still processing.

@Sch0field9 commented on GitHub (Sep 9, 2024): Those are all the logs I have. I couldn't figure out how to change the log level. The repo I had the troubles with was [Github home-assistant/core:master --](https://github.com/home-assistant/core) (Not sure if it was core) However, I had something similar happen when I tried it with the AnythinLLM repo. It startet way faster, but it somewhere in the middle was a fetch failed log entry, only one and it continued downloading. The Frontend had the same behaviour tough. Message came up fetch failed and no other indications that it is still downloading files. I also suspect it has to do with large repos. My bug reports focus is more about that there is no indication to the user, less that why exactly it failed. And possibly a feature request to be able to change the log level? Since there is also no indication in the info logs, that its still processing.
yindo commented 2026-02-22 18:24:56 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Sep 9, 2024):

Adding verbose logging to the tool at least so the process can be seen in logs even if the operator/user cannot see it in UI currently. Did pull in AnythingLLM repo without errors at this time

@timothycarambat commented on GitHub (Sep 9, 2024): Adding verbose logging to the tool at least so the process can be seen in logs even if the operator/user cannot see it in UI currently. Did pull in AnythingLLM repo without errors at this time
yindo changed title from [BUG]: GitHub Repo Fetch Failed message appears but then proceeds without notification to [GH-ISSUE #2243] [BUG]: GitHub Repo Fetch Failed message appears but then proceeds without notification 2026-06-05 14:40:54 -04:00
yindo commented 2026-06-05 14:40:55 -04:00
Author
Owner
Copy Link
Copy Source

@edddeduck commented on GitHub (Feb 26, 2026):

I have reopened the bug as I have found a 100% reproducible test case using a public documentation website that is not too large. I have been able to reproduce this using v1.11.0.

I told it to capture the website for https://docs.mod.io The site has 669 pages.

After about 10 minutes I got a message in the UI saying that the fetch failed. I checked the logs and could see this.

[backend] info: [CollectorApi] fetch failed
[backend] info: [TELEMETRY SENT] {"event":"extension_invoked","distinctId":"ee0a39fb-723b-4ff3-8055-74fa8720132b","properties":{"type":"website_depth","runtime":"docker"}}
prisma:info Starting a sqlite pool with 113 connections.

However I noted the CPU usage was using just over 100% (where 100% indicates one core being fully utilised) so something was obviously happening.

NOTE: I can using another tool check you can load all the pages on the site in under a minute so it feels strange it took close to 30 mins to review the full site, but without extra logging I am not sure what was taking the time here.

After another approximately 20 minutes the logs started exporting the following:

[collector] info: Found 669 links to scrape.
[collector] info: Starting bulk scraping...
[collector] info: Scraping 1/669: https://docs.mod.io
[collector] info: Successfully scraped https://docs.mod.io.
[collector] info: Scraping 2/669: https://docs.mod.io/#__docusaurus_skipToContent_fallback
[collector] info: [TikTokenTokenizer] Input will take too long to encode - estimating
[collector] info: Successfully scraped https://docs.mod.io/#__docusaurus_skipToContent_fallback.
[collector] info: Scraping 3/669: https://docs.mod.io/

It would be nice to get some progress information in the logs (and the UI) as it seems if you have a large capture requested it'll always time out and say the fetch failed even though it's actually just a large site to request.

The tool is great but having to monitor CPU usage in the terminal, the log file and the UI to try and work out what is happening makes it a little hard. The biggest issue is the large delay between the failed message and the found links to scrape message. Based on logs it indicates things have failed but if you check the CPU usage it's clear something was happening so if you're patient it'll work in the end.

Given it would be useful to ingest larger data sets than this one a more robust UX would be a great improvement. Happy to provide more information as needed.

<!-- gh-comment-id:3965744843 --> @edddeduck commented on GitHub (Feb 26, 2026): I have reopened the bug as I have found a 100% reproducible test case using a public documentation website that is not too large. I have been able to reproduce this using v1.11.0. I told it to capture the website for https://docs.mod.io The site has 669 pages. After about 10 minutes I got a message in the UI saying that the fetch failed. I checked the logs and could see this. ``` [backend] info: [CollectorApi] fetch failed [backend] info: [TELEMETRY SENT] {"event":"extension_invoked","distinctId":"ee0a39fb-723b-4ff3-8055-74fa8720132b","properties":{"type":"website_depth","runtime":"docker"}} prisma:info Starting a sqlite pool with 113 connections. ``` However I noted the CPU usage was using just over 100% (where 100% indicates one core being fully utilised) so something was obviously happening. NOTE: I can using another tool check you can load all the pages on the site in under a minute so it feels strange it took close to 30 mins to review the full site, but without extra logging I am not sure what was taking the time here. After another approximately 20 minutes the logs started exporting the following: ``` [collector] info: Found 669 links to scrape. [collector] info: Starting bulk scraping... [collector] info: Scraping 1/669: https://docs.mod.io [collector] info: Successfully scraped https://docs.mod.io. [collector] info: Scraping 2/669: https://docs.mod.io/#__docusaurus_skipToContent_fallback [collector] info: [TikTokenTokenizer] Input will take too long to encode - estimating [collector] info: Successfully scraped https://docs.mod.io/#__docusaurus_skipToContent_fallback. [collector] info: Scraping 3/669: https://docs.mod.io/ ``` It would be nice to get some progress information in the logs (and the UI) as it seems if you have a large capture requested it'll always time out and say the fetch failed even though it's actually just a large site to request. The tool is great but having to monitor CPU usage in the terminal, the log file and the UI to try and work out what is happening makes it a little hard. The biggest issue is the large delay between the failed message and the found links to scrape message. Based on logs it indicates things have failed but if you check the CPU usage it's clear something was happening so if you're patient it'll work in the end. Given it would be useful to ingest larger data sets than this one a more robust UX would be a great improvement. Happy to provide more information as needed.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label needs info / can't replicate
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#1463
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?