Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 22:23:50 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #3055] [BUG]: Bulk Link Scraping #1958

New Issue
Closed
opened 2026-02-22 18:27:25 -05:00 by yindo · 2 comments
yindo commented 2026-02-22 18:27:25 -05:00
Owner
Copy Link
Copy Source

Originally created by @Abhay-2004 on GitHub (Jan 29, 2025).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/3055

How are you running AnythingLLM?

Docker (remote machine)

What happened?

Bulk Link Scrapping doesn't work as expected, data is not scrapped accurately and throws an error. error occurs when the depth is set to 4/5 and links are over 50. Unexpected Unexpected token '<, " <!DOCTYPE".... is not valid JSON. ERROR

Error occurs with a probability of 60%

Are there known steps to reproduce?

Set the depth >=4 and number of links >=50.

Originally created by @Abhay-2004 on GitHub (Jan 29, 2025). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/3055 ### How are you running AnythingLLM? Docker (remote machine) ### What happened? Bulk Link Scrapping doesn't work as expected, data is not scrapped accurately and throws an error. error occurs when the depth is set to 4/5 and links are over 50. Unexpected Unexpected token '<, " <!DOCTYPE".... is not valid JSON. ERROR Error occurs with a probability of 60% ### Are there known steps to reproduce? Set the depth >=4 and number of links >=50.
yindo added the Dockerpossible bugneeds info / can't replicate labels 2026-02-22 18:27:25 -05:00
yindo closed this issue 2026-02-22 18:27:25 -05:00
yindo commented 2026-02-22 18:27:25 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Jan 29, 2025):

Logs from the docker container would be helpful here in determination of the root cause

@timothycarambat commented on GitHub (Jan 29, 2025): Logs from the docker container would be helpful here in determination of the root cause
yindo commented 2026-02-22 18:27:25 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Jan 30, 2025):

TimeoutError: Navigation timeout of 180000 ms exceeded

as well as this being the link: https://indicators.extension.iastate.edu/Indicators/Census/iowa%20%20counties%20%20rural%20urban%20%202020.pdf.

Which is a PDF served from the file, not HTML content, which the link scraper expects. That being said we should handle those situations and that is in this request: #2110

So the issue is not that the link scraper does not work. It lands on pages that are not HTML and therefore cannot parse them. So it hangs. That is the root cause. So we will pull up that old issue that has been sitting unassigned for a while and handle that and it will solve this issue.

@timothycarambat commented on GitHub (Jan 30, 2025): > TimeoutError: Navigation timeout of 180000 ms exceeded as well as this being the link: `https://indicators.extension.iastate.edu/Indicators/Census/iowa%20%20counties%20%20rural%20urban%20%202020.pdf.` Which is a PDF served from the file, not HTML content, which the link scraper expects. _That being said_ we should handle those situations and that is in this request: #2110 So the issue is not that the link scraper does not work. It lands on pages that are not HTML and therefore cannot parse them. So it hangs. That is the root cause. So we will pull up that old issue that has been sitting unassigned for a while and handle that and it will solve this issue.
yindo changed title from [BUG]: Bulk Link Scraping to [GH-ISSUE #3055] [BUG]: Bulk Link Scraping 2026-06-05 14:43:41 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label Docker needs info / can't replicate possible bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#1958
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?