Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-18 19:34:33 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #5078] [BUG]: Bulk Link Scraper doesn't accept self signed cert #4902

New Issue
Closed
opened 2026-06-05 14:50:46 -04:00 by yindo · 0 comments
yindo commented 2026-06-05 14:50:46 -04:00
Owner
Copy Link
Copy Source

Originally created by @kevinend51 on GitHub (Feb 26, 2026).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/5078

How are you running AnythingLLM?

Docker (remote machine)

What happened?

Failed to use Data Connectors -> Bulk Link Scraper to scrape self signed website, the error log is listed below:

[collector] info: Found 1 links to scrape.
[collector] info: Starting bulk scraping...
[collector] info: Scraping 1/1: https://*****.inc
[collector] error: Failed to scrape https://*****.inc. Error: net::ERR_CERT_AUTHORITY_INVALID at https://*****.inc
    at navigate (file:///app/collector/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/Frame.js:173:27)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Deferred.race (file:///app/collector/node_modules/puppeteer-core/lib/esm/puppeteer/util/Deferred.js:80:20)
    at async CdpFrame.goto (file:///app/collector/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/Frame.js:139:25)
    at async CdpPage.goto (file:///app/collector/node_modules/puppeteer-core/lib/esm/puppeteer/api/Page.js:568:20)
    at async PuppeteerWebBaseLoader._scrape (/app/collector/node_modules/langchain/dist/document_loaders/web/puppeteer.cjs:49:9)
    at async PuppeteerWebBaseLoader.load (/app/collector/node_modules/langchain/dist/document_loaders/web/puppeteer.cjs:74:22)
    at async bulkScrapePages (/app/collector/utils/extensions/WebsiteDepth/index.js:102:20)
    at async websiteScraper (/app/collector/utils/extensions/WebsiteDepth/index.js:164:23)
    at async /app/collector/extensions/index.js:124:29
[collector] info: Scraped 0 pages.

However, scraping the same website using Documents -> Fetch website works. I found the Documents -> Fetch website is configured to ignore https error in this commit. While the Bulk Link Scraper is lack of this option.

Are there known steps to reproduce?

Use Data Connectors -> Bulk Link Scraper to scrape self signed website. "Successfully scraped 0 pages!" is prompted, scrape failed.

Originally created by @kevinend51 on GitHub (Feb 26, 2026). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/5078 ### How are you running AnythingLLM? Docker (remote machine) ### What happened? Failed to use Data Connectors -> Bulk Link Scraper to scrape self signed website, the error log is listed below: ``` [collector] info: Found 1 links to scrape. [collector] info: Starting bulk scraping... [collector] info: Scraping 1/1: https://*****.inc [collector] error: Failed to scrape https://*****.inc. Error: net::ERR_CERT_AUTHORITY_INVALID at https://*****.inc at navigate (file:///app/collector/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/Frame.js:173:27) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async Deferred.race (file:///app/collector/node_modules/puppeteer-core/lib/esm/puppeteer/util/Deferred.js:80:20) at async CdpFrame.goto (file:///app/collector/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/Frame.js:139:25) at async CdpPage.goto (file:///app/collector/node_modules/puppeteer-core/lib/esm/puppeteer/api/Page.js:568:20) at async PuppeteerWebBaseLoader._scrape (/app/collector/node_modules/langchain/dist/document_loaders/web/puppeteer.cjs:49:9) at async PuppeteerWebBaseLoader.load (/app/collector/node_modules/langchain/dist/document_loaders/web/puppeteer.cjs:74:22) at async bulkScrapePages (/app/collector/utils/extensions/WebsiteDepth/index.js:102:20) at async websiteScraper (/app/collector/utils/extensions/WebsiteDepth/index.js:164:23) at async /app/collector/extensions/index.js:124:29 [collector] info: Scraped 0 pages. ``` However, scraping the same website using Documents -> Fetch website works. I found the Documents -> Fetch website is configured to ignore https error in this [commit](https://github.com/Mintplex-Labs/anything-llm/commit/619f6b3884e22283c1de571da4dddb7d92553d8b). While the Bulk Link Scraper is lack of this option. ### Are there known steps to reproduce? Use Data Connectors -> Bulk Link Scraper to scrape self signed website. "Successfully scraped 0 pages!" is prompted, scrape failed.
yindo added the possible bug label 2026-06-05 14:50:46 -04:00
yindo closed this issue 2026-06-05 14:50:46 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label possible bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#4902
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?