Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #1997] [BUG]: Depth setting for Bulk Link Scraper has no effect #1297

New Issue
Closed
opened 2026-02-22 18:24:09 -05:00 by yindo · 6 comments
yindo commented 2026-02-22 18:24:09 -05:00
Owner
Copy Link
Copy Source

Originally created by @mPyKen on GitHub (Jul 30, 2024).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/1997

How are you running AnythingLLM?

Kubernetes

What happened?

Bulk Link Scraper discovers more links than there are on the page even though depth is set to 1.

Are there known steps to reproduce?

Open https://www.google.com on your browser and check that the html has 32 occurances of href.
Start Bulk Link Scraper on google with depth = 1 and Max Links = #occurances of href + 1 = 33 Max Links = 100.
Then check logs:

[collector] info: Discovering links...
[collector] info: Found 100 links to scrape.
[collector] info: Starting bulk scraping...

I can try to fix this if contributions are welcome

Originally created by @mPyKen on GitHub (Jul 30, 2024). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/1997 ### How are you running AnythingLLM? Kubernetes ### What happened? Bulk Link Scraper discovers more links than there are on the page even though depth is set to 1. ### Are there known steps to reproduce? Open `https://www.google.com` on your browser and check that the html has 32 occurances of `href`. Start Bulk Link Scraper on google with `depth = 1` and ~`Max Links = #occurances of href + 1 = 33`~ `Max Links = 100`. Then check logs: ``` [collector] info: Discovering links... [collector] info: Found 100 links to scrape. [collector] info: Starting bulk scraping... ``` I can try to fix this if contributions are welcome
yindo added the possible bug label 2026-02-22 18:24:09 -05:00
yindo closed this issue 2026-02-22 18:24:09 -05:00
yindo commented 2026-02-22 18:24:10 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Jul 30, 2024):

its +1 because the root site counts as a link so that we add the origin site as well.

@timothycarambat commented on GitHub (Jul 30, 2024): its +1 because the root site counts as a link so that we add the origin site as well.
yindo commented 2026-02-22 18:24:10 -05:00
Author
Owner
Copy Link
Copy Source

@mPyKen commented on GitHub (Jul 30, 2024):

Fair enough, +1 makes sense. Let me update my reproduction steps: Is there any reason it detects more than 33 links when I set Max Links = 100 and depth = 1?

[collector] info: Discovering links...
[collector] info: Found 100 links to scrape.
[collector] info: Starting bulk scraping...
[collector] info: Scraping 1/100: https://www.google.com/imghp
...
@mPyKen commented on GitHub (Jul 30, 2024): Fair enough, +1 makes sense. Let me update my reproduction steps: Is there any reason it detects more than 33 links when I set `Max Links = 100` and `depth = 1`? ``` [collector] info: Discovering links... [collector] info: Found 100 links to scrape. [collector] info: Starting bulk scraping... [collector] info: Scraping 1/100: https://www.google.com/imghp ... ```
yindo commented 2026-02-22 18:24:10 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Jul 30, 2024):

@shatfield4 any ideas?

@timothycarambat commented on GitHub (Jul 30, 2024): @shatfield4 any ideas?
yindo commented 2026-02-22 18:24:10 -05:00
Author
Owner
Copy Link
Copy Source

@mPyKen commented on GitHub (Jul 30, 2024):

Hmm to me, it looks like depth is indeed being ignored in discoverLinks (condition currentLevel < depth will always be met). If it's ok I can try to fix this and create a PR

@mPyKen commented on GitHub (Jul 30, 2024): Hmm to me, it looks like depth is indeed being ignored in [discoverLinks](https://github.com/Mintplex-Labs/anything-llm/blob/3856a4d19cf925ff8177e3fcc122eeedb0198a5f/collector/utils/extensions/WebsiteDepth/index.js#L21-L40) (condition `currentLevel < depth` will always be met). If it's ok I can try to fix this and create a PR
yindo commented 2026-02-22 18:24:10 -05:00
Author
Owner
Copy Link
Copy Source

@mPyKen commented on GitHub (Aug 11, 2024):

Please let me know if this is a won't fix, help wanted or confirmation needed

@mPyKen commented on GitHub (Aug 11, 2024): Please let me know if this is a won't fix, help wanted or confirmation needed
yindo commented 2026-02-22 18:24:10 -05:00
Author
Owner
Copy Link
Copy Source

@shatfield4 commented on GitHub (Aug 12, 2024):

@mPyKen I just created a fix for this in #2096, thank you for pointing this out!

@shatfield4 commented on GitHub (Aug 12, 2024): @mPyKen I just created a fix for this in #2096, thank you for pointing this out!
yindo changed title from [BUG]: Depth setting for Bulk Link Scraper has no effect to [GH-ISSUE #1997] [BUG]: Depth setting for Bulk Link Scraper has no effect 2026-06-05 14:40:01 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label possible bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#1297
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?