Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #3848] [BUG]: Bulk Link Scraper #2458

New Issue
Closed
opened 2026-02-22 18:29:45 -05:00 by yindo · 2 comments
yindo commented 2026-02-22 18:29:45 -05:00
Owner
Copy Link
Copy Source

Originally created by @ghost on GitHub (May 19, 2025).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/3848

How are you running AnythingLLM?

AnythingLLM desktop app

What happened?

[backend] info: [EncryptionManager] Loaded existing key & salt for encrypting arbitrary data.

[collector] info: Discovering links...

[collector] info: Found 1 links to scrape.

[collector] info: Starting bulk scraping...

[collector] info: Scraping 1/1: https://tr.wikipedia.org/wiki/Türkçe

[collector] error: getPageContent failed to be fetched by electron - falling back to fetch! Error: An object could not be cloned.

    at ParentPort.postMessage (node:electron/js2c/utility_init:2:4761)

    at ao (C:\AnythingLLM\resources\backend\collector.js:20:418)

    at ai (C:\AnythingLLM\resources\backend\collector.js:47:5845)

    at ii (C:\AnythingLLM\resources\backend\collector.js:47:6920)

    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

    at async C:\AnythingLLM\resources\backend\collector.js:47:8582

[collector] error: getPageContent failed to be fetched by any method. TypeError: Failed to parse URL from function link() { [native code] }

    at Object.fetch (node:internal/deps/undici/undici:11457:11)

    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

    at async ao (C:\AnythingLLM\resources\backend\collector.js:20:969)

    at async ai (C:\AnythingLLM\resources\backend\collector.js:47:5839)

    at async ii (C:\AnythingLLM\resources\backend\collector.js:47:6914)

    at async C:\AnythingLLM\resources\backend\collector.js:47:8582

[collector] error: Failed to scrape https://tr.wikipedia.org/wiki/Türkçe. TypeError: Cannot read properties of null (reading 'length')

    at ai (C:\AnythingLLM\resources\backend\collector.js:47:5857)

    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

    at async ii (C:\AnythingLLM\resources\backend\collector.js:47:6914)

    at async C:\AnythingLLM\resources\backend\collector.js:47:8582

[collector] info: Scraped 0 pages.

[backend] info: [TELEMETRY SENT] {"event":"extension_invoked","distinctId":"a8223daf-c968-4a40-b1ae-ca92d4e2ca54","properties":{"type":"website_depth"}}

Are there known steps to reproduce?

No response

Originally created by @ghost on GitHub (May 19, 2025). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/3848 ### How are you running AnythingLLM? AnythingLLM desktop app ### What happened? [backend] info: [EncryptionManager] Loaded existing key & salt for encrypting arbitrary data. [collector] info: Discovering links... [collector] info: Found 1 links to scrape. [collector] info: Starting bulk scraping... [collector] info: Scraping 1/1: https://tr.wikipedia.org/wiki/Türkçe [collector] error: getPageContent failed to be fetched by electron - falling back to fetch! Error: An object could not be cloned.     at ParentPort.postMessage (node:electron/js2c/utility_init:2:4761)     at ao (C:\AnythingLLM\resources\backend\collector.js:20:418)     at ai (C:\AnythingLLM\resources\backend\collector.js:47:5845)     at ii (C:\AnythingLLM\resources\backend\collector.js:47:6920)     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)     at async C:\AnythingLLM\resources\backend\collector.js:47:8582 [collector] error: getPageContent failed to be fetched by any method. TypeError: Failed to parse URL from function link() { [native code] }     at Object.fetch (node:internal/deps/undici/undici:11457:11)     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)     at async ao (C:\AnythingLLM\resources\backend\collector.js:20:969)     at async ai (C:\AnythingLLM\resources\backend\collector.js:47:5839)     at async ii (C:\AnythingLLM\resources\backend\collector.js:47:6914)     at async C:\AnythingLLM\resources\backend\collector.js:47:8582 [collector] error: Failed to scrape https://tr.wikipedia.org/wiki/Türkçe. TypeError: Cannot read properties of null (reading 'length')     at ai (C:\AnythingLLM\resources\backend\collector.js:47:5857)     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)     at async ii (C:\AnythingLLM\resources\backend\collector.js:47:6914)     at async C:\AnythingLLM\resources\backend\collector.js:47:8582 [collector] info: Scraped 0 pages. [backend] info: [TELEMETRY SENT] {"event":"extension_invoked","distinctId":"a8223daf-c968-4a40-b1ae-ca92d4e2ca54","properties":{"type":"website_depth"}} ### Are there known steps to reproduce? _No response_
yindo added the possible bug label 2026-02-22 18:29:45 -05:00
yindo closed this issue 2026-02-22 18:29:45 -05:00
yindo commented 2026-02-22 18:29:46 -05:00
Author
Owner
Copy Link
Copy Source

@dbhorst commented on GitHub (May 19, 2025):

Same here on macOS 15, anythingLLM Desktop app. Happens when using the bulk link scraper.

{
    "level": "info",
    "message": "Scraping 1/1: https://www.keycloak.org/documentation",
    "service": "collector"
}
{
    "level": "error",
    "message": "getPageContent failed to be fetched by electron - falling back to fetch! Error: An object could not be cloned.\n    at ParentPort.postMessage (node:electron/js2c/utility_init:2:4761)\n    at ao (/Applications/AnythingLLM.app/Contents/Resources/backend/collector.js:20:418)\n    at ai (/Applications/AnythingLLM.app/Contents/Resources/backend/collector.js:47:5845)\n    at ii (/Applications/AnythingLLM.app/Contents/Resources/backend/collector.js:47:6920)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async /Applications/AnythingLLM.app/Contents/Resources/backend/collector.js:47:8582",
    "service": "collector"
}
{
    "level": "error",
    "message": "getPageContent failed to be fetched by any method. TypeError: Failed to parse URL from function link() { [native code] }\n    at Object.fetch (node:internal/deps/undici/undici:11457:11)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async ao (/Applications/AnythingLLM.app/Contents/Resources/backend/collector.js:20:969)\n    at async ai (/Applications/AnythingLLM.app/Contents/Resources/backend/collector.js:47:5839)\n    at async ii (/Applications/AnythingLLM.app/Contents/Resources/backend/collector.js:47:6914)\n    at async /Applications/AnythingLLM.app/Contents/Resources/backend/collector.js:47:8582",
    "service": "collector"
}
{
    "level": "error",
    "message": "Failed to scrape https://www.keycloak.org/documentation. TypeError: Cannot read properties of null (reading 'length')\n    at ai (/Applications/AnythingLLM.app/Contents/Resources/backend/collector.js:47:5857)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async ii (/Applications/AnythingLLM.app/Contents/Resources/backend/collector.js:47:6914)\n    at async /Applications/AnythingLLM.app/Contents/Resources/backend/collector.js:47:8582",
    "service": "collector"
}
{
    "level": "info",
    "message": "Scraped 0 pages.",
    "service": "collector"
}

Are there known steps to reproduce?

  1. New Workspace
  2. Add documents button
  3. Data connectors tab
  4. Select Build Link Scraper
  5. Enter URL, e.g. https://www.keycloak.org/documentation
  6. Click submit
@dbhorst commented on GitHub (May 19, 2025): Same here on macOS 15, anythingLLM Desktop app. Happens when using the bulk link scraper. ``` { "level": "info", "message": "Scraping 1/1: https://www.keycloak.org/documentation", "service": "collector" } { "level": "error", "message": "getPageContent failed to be fetched by electron - falling back to fetch! Error: An object could not be cloned.\n at ParentPort.postMessage (node:electron/js2c/utility_init:2:4761)\n at ao (/Applications/AnythingLLM.app/Contents/Resources/backend/collector.js:20:418)\n at ai (/Applications/AnythingLLM.app/Contents/Resources/backend/collector.js:47:5845)\n at ii (/Applications/AnythingLLM.app/Contents/Resources/backend/collector.js:47:6920)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async /Applications/AnythingLLM.app/Contents/Resources/backend/collector.js:47:8582", "service": "collector" } { "level": "error", "message": "getPageContent failed to be fetched by any method. TypeError: Failed to parse URL from function link() { [native code] }\n at Object.fetch (node:internal/deps/undici/undici:11457:11)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async ao (/Applications/AnythingLLM.app/Contents/Resources/backend/collector.js:20:969)\n at async ai (/Applications/AnythingLLM.app/Contents/Resources/backend/collector.js:47:5839)\n at async ii (/Applications/AnythingLLM.app/Contents/Resources/backend/collector.js:47:6914)\n at async /Applications/AnythingLLM.app/Contents/Resources/backend/collector.js:47:8582", "service": "collector" } { "level": "error", "message": "Failed to scrape https://www.keycloak.org/documentation. TypeError: Cannot read properties of null (reading 'length')\n at ai (/Applications/AnythingLLM.app/Contents/Resources/backend/collector.js:47:5857)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async ii (/Applications/AnythingLLM.app/Contents/Resources/backend/collector.js:47:6914)\n at async /Applications/AnythingLLM.app/Contents/Resources/backend/collector.js:47:8582", "service": "collector" } { "level": "info", "message": "Scraped 0 pages.", "service": "collector" } ``` **Are there known steps to reproduce?** 1. New Workspace 2. Add documents button 3. Data connectors tab 4. Select Build Link Scraper 5. Enter URL, e.g. https://www.keycloak.org/documentation 6. Click submit
yindo commented 2026-02-22 18:29:46 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (May 19, 2025):

Thank you for the repro - was a side effect of how we use the background browser process in electron as to why this only impacted desktop. Will be patched in next release 1.8.2 👍

@timothycarambat commented on GitHub (May 19, 2025): Thank you for the repro - was a side effect of how we use the background browser process in electron as to why this only impacted desktop. Will be patched in next release 1.8.2 👍
yindo changed title from [BUG]: Bulk Link Scraper to [GH-ISSUE #3848] [BUG]: Bulk Link Scraper 2026-06-05 14:46:41 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label possible bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#2458
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?