Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #2758] [BUG]: Scraping failed #1771

New Issue
Closed
opened 2026-02-22 18:26:27 -05:00 by yindo · 0 comments
yindo commented 2026-02-22 18:26:27 -05:00
Owner
Copy Link
Copy Source

Originally created by @mrsimpson on GitHub (Dec 3, 2024).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/2758

Originally assigned to: @shatfield4 on GitHub.

How are you running AnythingLLM?

All versions

What happened?

What I did

Just tried out AnythingLLM for the first time. I wanted to test the website scraping with the naive-ui documentation (https://www.naiveui.com/en-US/os-theme/docs/introduction) to apply RAG on it.

Added the URL to the Bulk-link-scraper.
After a short delay, there is a success-message which says "successfully scraped 0 pages".

Retrieving the page in the browser works fine.

What I expected

I expected it to scrape documentation from 20 components which a human finds as links on the above page. If something goes wrong, I expected a meaningful error message,

What I observed

  • This seems to be quite a special type of webpage which is impossible for puppeteer to retrieve. I can at least scrape other webpages.
  • Container log
2024-12-03 21:56:24 [collector] info: Discovering links...
2024-12-03 21:56:25 [collector] info: Found 1 links to scrape.
2024-12-03 21:56:25 [collector] info: Starting bulk scraping...
2024-12-03 21:56:25 [collector] info: Scraping 1/1: https://www.naiveui.com/en-US/os-theme/docs/introduction
2024-12-03 21:56:25 [collector] error: Failed to scrape https://www.naiveui.com/en-US/os-theme/docs/introduction. TypeError: Cannot read properties of undefined (reading 'length')
2024-12-03 21:56:25     at bulkScrapePages (/app/collector/utils/extensions/WebsiteDepth/index.js:105:20)
2024-12-03 21:56:25     at async websiteScraper (/app/collector/utils/extensions/WebsiteDepth/index.js:160:23)
2024-12-03 21:56:25     at async /app/collector/extensions/index.js:122:29
2024-12-03 21:56:25 [collector] info: Scraped 0 pages.
  • I started frontend, server and collector as separate processes from my IDE and set a breakpoint at the scraping. I noticed that puppeteer fails with a "websocket hung up".
    This seems not to be handled properly

Are there known steps to reproduce?

Open the bulk scraping tool
enter the URL https://www.naiveui.com/en-US/os-theme/docs/introduction with default parameters
scrape

Originally created by @mrsimpson on GitHub (Dec 3, 2024). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/2758 Originally assigned to: @shatfield4 on GitHub. ### How are you running AnythingLLM? All versions ### What happened? #### What I did Just tried out AnythingLLM for the first time. I wanted to test the website scraping with the naive-ui documentation (https://www.naiveui.com/en-US/os-theme/docs/introduction) to apply RAG on it. Added the URL to the Bulk-link-scraper. After a short delay, there is a success-message which says "successfully scraped 0 pages". Retrieving the page in the browser works fine. #### What I expected I expected it to scrape documentation from 20 components which a human finds as links on the above page. If something goes wrong, I expected a meaningful error message, #### What I observed - This seems to be quite a special type of webpage which is impossible for puppeteer to retrieve. I can at least scrape other webpages. - Container log ``` 2024-12-03 21:56:24 [collector] info: Discovering links... 2024-12-03 21:56:25 [collector] info: Found 1 links to scrape. 2024-12-03 21:56:25 [collector] info: Starting bulk scraping... 2024-12-03 21:56:25 [collector] info: Scraping 1/1: https://www.naiveui.com/en-US/os-theme/docs/introduction 2024-12-03 21:56:25 [collector] error: Failed to scrape https://www.naiveui.com/en-US/os-theme/docs/introduction. TypeError: Cannot read properties of undefined (reading 'length') 2024-12-03 21:56:25 at bulkScrapePages (/app/collector/utils/extensions/WebsiteDepth/index.js:105:20) 2024-12-03 21:56:25 at async websiteScraper (/app/collector/utils/extensions/WebsiteDepth/index.js:160:23) 2024-12-03 21:56:25 at async /app/collector/extensions/index.js:122:29 2024-12-03 21:56:25 [collector] info: Scraped 0 pages. ``` - I started frontend, server and collector as separate processes from my IDE and set a breakpoint at the scraping. I noticed that puppeteer fails with a "websocket hung up". This seems not to be handled properly ### Are there known steps to reproduce? Open the bulk scraping tool enter the URL https://www.naiveui.com/en-US/os-theme/docs/introduction with default parameters scrape
yindo added the possible buginvestigating labels 2026-02-22 18:26:27 -05:00
yindo closed this issue 2026-02-22 18:26:27 -05:00
yindo changed title from [BUG]: Scraping failed to [GH-ISSUE #2758] [BUG]: Scraping failed 2026-06-05 14:42:36 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label investigating possible bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#1771
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?