Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #1515] [FEAT]: Build Link Scraper - Be able to specify a list of Whilelisted domains. #974

New Issue
Closed
opened 2026-02-22 18:22:29 -05:00 by yindo · 5 comments
yindo commented 2026-02-22 18:22:29 -05:00
Owner
Copy Link
Copy Source

Originally created by @atljoseph on GitHub (May 23, 2024).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/1515

What would you like to see?

Need to have this feature focus on a singe domain, or a limited number of domains.

image

I don't want this to link down to 4chan for some reason LOL.
Is facing customers.

Originally created by @atljoseph on GitHub (May 23, 2024). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/1515 ### What would you like to see? Need to have this feature focus on a singe domain, or a limited number of domains. ![image](https://github.com/Mintplex-Labs/anything-llm/assets/25017621/5739d036-bab5-4156-b958-36ce39765241) I don't want this to link down to 4chan for some reason LOL. Is facing customers.
yindo added the enhancementfeature request labels 2026-02-22 18:22:29 -05:00
yindo closed this issue 2026-02-22 18:22:29 -05:00
yindo commented 2026-02-22 18:22:29 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (May 23, 2024):

That is what this feature does. It does not leave the parent origin that you assign. For example linking reddit will only scrape child links that also resolve on reddit.com hostnames. It will not child-follow every post on the main page

@timothycarambat commented on GitHub (May 23, 2024): That is what this feature does. It does not leave the parent origin that you assign. For example linking reddit will only scrape child links that also resolve on `reddit.com` hostnames. It will not child-follow every post on the main page
yindo commented 2026-02-22 18:22:30 -05:00
Author
Owner
Copy Link
Copy Source

@atljoseph commented on GitHub (May 23, 2024):

Oh ok. I thought it might be the case, but couldn’t find anything in
the app stating that fact. Cool!!

On Thu, May 23, 2024 at 3:53 PM Timothy Carambat @.***>
wrote:

That is what this feature does. It does not leave the parent origin that
you assign. For example linking reddit will only scrape child links that
also resolve on reddit.com hostnames. It will not child-follow every post
on the main page

—
Reply to this email directly, view it on GitHub
https://github.com/Mintplex-Labs/anything-llm/issues/1515#issuecomment-2127914156,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AF632FN6AQDDECO3TZVAYYDZDZCL5AVCNFSM6AAAAABIGG3ETOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMRXHEYTIMJVGY
.
You are receiving this because you authored the thread.Message ID:
@.***>

@atljoseph commented on GitHub (May 23, 2024): Oh ok. I thought it might be the case, but couldn’t find anything in the app stating that fact. Cool!! On Thu, May 23, 2024 at 3:53 PM Timothy Carambat ***@***.***> wrote: > That is what this feature does. It does not leave the parent origin that > you assign. For example linking reddit will only scrape child links that > also resolve on reddit.com hostnames. It will not child-follow every post > on the main page > > — > Reply to this email directly, view it on GitHub > <https://github.com/Mintplex-Labs/anything-llm/issues/1515#issuecomment-2127914156>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AF632FN6AQDDECO3TZVAYYDZDZCL5AVCNFSM6AAAAABIGG3ETOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMRXHEYTIMJVGY> > . > You are receiving this because you authored the thread.Message ID: > ***@***.***> >
yindo commented 2026-02-22 18:22:30 -05:00
Author
Owner
Copy Link
Copy Source

@atljoseph commented on GitHub (May 23, 2024):

Curious. What happens to pdf files or other documents encountered while scraping? I presume video files are skipped (maybe a way to send that to gpt4o in the future to embed).

@atljoseph commented on GitHub (May 23, 2024): Curious. What happens to pdf files or other documents encountered while scraping? I presume video files are skipped (maybe a way to send that to gpt4o in the future to embed).
yindo commented 2026-02-22 18:22:30 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (May 24, 2024):

If you enter in a direct PDF link, it wont download it and parse it - it would just ignore it. I would like to add the ability to get the mime and parse automatically, would probably be a nice to have

@timothycarambat commented on GitHub (May 24, 2024): If you enter in a direct PDF link, it wont download it and parse it - it would just ignore it. I would like to add the ability to get the mime and parse automatically, would probably be a nice to have
yindo commented 2026-02-22 18:22:30 -05:00
Author
Owner
Copy Link
Copy Source

@atljoseph commented on GitHub (May 24, 2024):

Can’t have everything… yet. Appreciate it :)

On Fri, May 24, 2024 at 12:26 AM Timothy Carambat @.***>
wrote:

If you enter in a direct PDF link, it wont download it and parse it - it
would just ignore it. I would like to add the ability to get the mime and
parse automatically, would probably be a nice to have

—
Reply to this email directly, view it on GitHub
https://github.com/Mintplex-Labs/anything-llm/issues/1515#issuecomment-2128497282,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AF632FJDFYBJ7ZWO23LD23TZD26O3AVCNFSM6AAAAABIGG3ETOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMRYGQ4TOMRYGI
.
You are receiving this because you authored the thread.Message ID:
@.***>

@atljoseph commented on GitHub (May 24, 2024): Can’t have everything… yet. Appreciate it :) On Fri, May 24, 2024 at 12:26 AM Timothy Carambat ***@***.***> wrote: > If you enter in a direct PDF link, it wont download it and parse it - it > would just ignore it. I would like to add the ability to get the mime and > parse automatically, would probably be a nice to have > > — > Reply to this email directly, view it on GitHub > <https://github.com/Mintplex-Labs/anything-llm/issues/1515#issuecomment-2128497282>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AF632FJDFYBJ7ZWO23LD23TZD26O3AVCNFSM6AAAAABIGG3ETOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMRYGQ4TOMRYGI> > . > You are receiving this because you authored the thread.Message ID: > ***@***.***> >
yindo changed title from [FEAT]: Build Link Scraper - Be able to specify a list of Whilelisted domains. to [GH-ISSUE #1515] [FEAT]: Build Link Scraper - Be able to specify a list of Whilelisted domains. 2026-06-05 14:38:14 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label enhancement feature request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#974
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?