Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #1528] [BUG]: Bulk link scraper tries to fetch child-links from root of website instead of defined origin #983

New Issue
Closed
opened 2026-02-22 18:22:32 -05:00 by yindo · 3 comments
yindo commented 2026-02-22 18:22:32 -05:00
Owner
Copy Link
Copy Source

Originally created by @Propheticus on GitHub (May 24, 2024).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/1528

Originally assigned to: @shatfield4 on GitHub.

How are you running AnythingLLM?

Docker (local)

What happened?

https://discord.com/channels/1114740394715004990/1243578185581334538

The bulk link scraper does not work when the URL you enter is not the root.

If you enter https://www.somesite.com/some/sub with depth 1, it will correctly identify the children of that sub https://www.somesite.com/some/sub/child_#
However, it will then scrape https://www.somesite.com/child_#

E.g.
If you enter https://learn.microsoft.com/en-us/azure/well-architected/reliability
It will try to scrape https://learn.microsoft.com/metrics instead of https://learn.microsoft.com/en-us/azure/well-architected/reliability/metrics

image
image
404 - Page not found\n\nWe couldn't find this page.

Are there known steps to reproduce?

Enter a URL which is not the root website / homepage.

Originally created by @Propheticus on GitHub (May 24, 2024). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/1528 Originally assigned to: @shatfield4 on GitHub. ### How are you running AnythingLLM? Docker (local) ### What happened? https://discord.com/channels/1114740394715004990/1243578185581334538 The bulk link scraper does not work when the URL you enter is not the root. If you enter `https://www.somesite.com/some/sub` with depth 1, it will correctly identify the children of that sub `https://www.somesite.com/some/sub/child_#` However, it will then scrape `https://www.somesite.com/child_#` E.g. If you enter `https://learn.microsoft.com/en-us/azure/well-architected/reliability` It will try to scrape `https://learn.microsoft.com/metrics` instead of `https://learn.microsoft.com/en-us/azure/well-architected/reliability/metrics` ![image](https://github.com/Mintplex-Labs/anything-llm/assets/6628064/295070d8-cf6f-416b-ae21-d840a88cf671) ![image](https://github.com/Mintplex-Labs/anything-llm/assets/6628064/fd025886-d6e0-4cc1-a945-ff10dbd4c0ae) `404 - Page not found\n\nWe couldn't find this page.` ### Are there known steps to reproduce? Enter a URL which is not the root website / homepage.
yindo added the bug label 2026-02-22 18:22:32 -05:00
yindo closed this issue 2026-02-22 18:22:32 -05:00
yindo commented 2026-02-22 18:22:32 -05:00
Author
Owner
Copy Link
Copy Source

@shatfield4 commented on GitHub (May 24, 2024):

This is the intended behavior, the bulk scraper is meant to "spider web" out and crawl all links on the page. It first scrapes the first link provided, then looks for all links on that page and will scrape them all.

@shatfield4 commented on GitHub (May 24, 2024): This is the intended behavior, the bulk scraper is meant to "spider web" out and crawl all links on the page. It first scrapes the first link provided, then looks for all links on that page and will scrape them all.
yindo commented 2026-02-22 18:22:32 -05:00
Author
Owner
Copy Link
Copy Source

@Propheticus commented on GitHub (May 24, 2024):

What if I want to start the spidering to start from a subpage? I don't want to spiderweb all of Microsoft learn, just the subpages of the well architected framework.

Also it's trying to scrape pages that don't exist. It's taking a child of the subpage but then appends that child name to the root. That can't be as intended @shatfield4, is it?

@Propheticus commented on GitHub (May 24, 2024): What if I want to start the spidering to start from a subpage? I don't want to spiderweb all of Microsoft learn, just the subpages of the well architected framework. Also it's trying to scrape pages that don't exist. It's taking a child of the subpage but then appends that child name to the root. That can't be as intended @shatfield4, is it?
yindo commented 2026-02-22 18:22:32 -05:00
Author
Owner
Copy Link
Copy Source

@Propheticus commented on GitHub (May 25, 2024):

Let me put it more simply:

The page to be scraped (with 1 level deep spiderwebbing) is https://learn.microsoft.com/en-us/azure/well-architected/reliability
This page contains an HREF to a child called metrics
image
That means we want to scrape https://learn.microsoft.com/en-us/azure/well-architected/reliability/metrics

The bulk scraper however tries to scrape https://learn.microsoft.com/metrics which does not exist.

Something is going wrong in determining the full URL to be scraped. Perhaps https://github.com/Mintplex-Labs/anything-llm/blob/c24b79c9d1c349f9977d065c0f54bfeaa3dfe22a/collector/utils/extensions/WebsiteDepth/index.js#L13 ?

@Propheticus commented on GitHub (May 25, 2024): Let me put it more simply: The page to be scraped (with 1 level deep spiderwebbing) is `https://learn.microsoft.com/en-us/azure/well-architected/reliability` This page contains an HREF to a child called `metrics` ![image](https://github.com/Mintplex-Labs/anything-llm/assets/6628064/92ccb52f-66ef-4f3d-beef-f9c3c46236ec) That means we want to scrape `https://learn.microsoft.com/en-us/azure/well-architected/reliability/metrics` The bulk scraper however tries to scrape `https://learn.microsoft.com/metrics` which does not exist. Something is going wrong in determining the full URL to be scraped. Perhaps https://github.com/Mintplex-Labs/anything-llm/blob/c24b79c9d1c349f9977d065c0f54bfeaa3dfe22a/collector/utils/extensions/WebsiteDepth/index.js#L13 ?
yindo changed title from [BUG]: Bulk link scraper tries to fetch child-links from root of website instead of defined origin to [GH-ISSUE #1528] [BUG]: Bulk link scraper tries to fetch child-links from root of website instead of defined origin 2026-06-05 14:38:18 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#983
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?