Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-21 01:25:28 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #1570] [FEAT]: Bulk Link scraper improvement suggestions #1016

New Issue
Open
opened 2026-02-22 18:22:41 -05:00 by yindo · 2 comments
yindo commented 2026-02-22 18:22:41 -05:00
Owner
Copy Link
Copy Source

Originally created by @wallartup on GitHub (May 30, 2024).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/1570

How are you running AnythingLLM?

Docker (local)

What happened?

First of all, I love the idea of recurssively scraping a lot of content via a bulk link scraper.

I think it needs to be rethinked a little, I would recommend not giving people the ability to choose the amount of links to scrape but rather have a full recursive scrape of the whole website. The reason is that you already have the ability to choose what links you want to scrape IF you choose to go one on one.

I would argue that no one knows the amount of links they want to scrape (because you don't control what the scraper finds) within a full website so it becomes negligent because when you are scraping you want everything on that page or subdomain or folder.

Explanation:

  1. Many times you have a usecase where the whole website needs to be scraped, you would usually not scrape 10 or 50 pages (and not knowing which pages have been scraped) but rather everything that the "scraper" finds on that url. Make a recurssive scraper for everything! Remove the option to add amount of links
  2. When we are scraping we only get the popup saying "scraping, this might take some time". I would rather make this a brand new menu alternative in the backend called "Scraped websites" where you can see the amount of pages that are scraped, stop the scraping, pause the scraping or "play" the scraping again.
  3. These files does not need to land in the document folder, they should be in a separate folder and EMBEDDED automatically.
  4. Make scraped websites available via the settings menu scraped websites. Put a similar three dots as adding users to a workspace, this way you can now add different scraped websites to different workspaces outside of the document management solution.
  5. In the data connectors bulk links "menu" -> create a link to the admin pages to see the scapings that are happening, that are done and/or paused.

I tried adding 50000 links to scrape and I have no idea now how to shut it down :) haha. It even said scrape failed and then it continues scraping in the background. This is why the above is key.

Are there known steps to reproduce?

No response

Originally created by @wallartup on GitHub (May 30, 2024). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/1570 ### How are you running AnythingLLM? Docker (local) ### What happened? First of all, I love the idea of recurssively scraping a lot of content via a bulk link scraper. I think it needs to be rethinked a little, I would recommend not giving people the ability to choose the amount of links to scrape but rather have a full recursive scrape of the whole website. The reason is that you already have the ability to choose what links you want to scrape IF you choose to go one on one. I would argue that no one knows the amount of links they want to scrape (because you don't control what the scraper finds) within a full website so it becomes negligent because when you are scraping you want everything on that page or subdomain or folder. Explanation: 1. Many times you have a usecase where the whole website needs to be scraped, you would usually not scrape 10 or 50 pages (and not knowing which pages have been scraped) but rather everything that the "scraper" finds on that url. Make a recurssive scraper for everything! Remove the option to add amount of links 2. When we are scraping we only get the popup saying "scraping, this might take some time". I would rather make this a brand new menu alternative in the backend called "Scraped websites" where you can see the amount of pages that are scraped, stop the scraping, pause the scraping or "play" the scraping again. 3. These files does not need to land in the document folder, they should be in a separate folder and EMBEDDED automatically. 4. Make scraped websites available via the settings menu scraped websites. Put a similar three dots as adding users to a workspace, this way you can now add different scraped websites to different workspaces outside of the document management solution. 5. In the data connectors bulk links "menu" -> create a link to the admin pages to see the scapings that are happening, that are done and/or paused. I tried adding 50000 links to scrape and I have no idea now how to shut it down :) haha. It even said scrape failed and then it continues scraping in the background. This is why the above is key. ### Are there known steps to reproduce? _No response_
yindo added the enhancement label 2026-02-22 18:22:41 -05:00
yindo commented 2026-02-22 18:22:41 -05:00
Author
Owner
Copy Link
Copy Source

@SupercaliG commented on GitHub (Jun 1, 2024):

I second this!

@SupercaliG commented on GitHub (Jun 1, 2024): I second this!
yindo commented 2026-02-22 18:22:41 -05:00
Author
Owner
Copy Link
Copy Source

@Alminc91 commented on GitHub (Oct 4, 2024):

Just one tip meanwhile how I control or monitor the scraping:
in your docker container you can run docker-compose logs --tail=200 --follow

Then you will see how many links are scraped from total, left and which link is currently scraped. At least a nice temporary solution :).

@Alminc91 commented on GitHub (Oct 4, 2024): Just one tip meanwhile how I control or monitor the scraping: in your docker container you can run docker-compose logs --tail=200 --follow Then you will see how many links are scraped from total, left and which link is currently scraped. At least a nice temporary solution :).
yindo changed title from [FEAT]: Bulk Link scraper improvement suggestions to [GH-ISSUE #1570] [FEAT]: Bulk Link scraper improvement suggestions 2026-06-05 14:38:30 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label enhancement
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#1016
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?