Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #2475] [BUG]: Failed Scrapper #1607

New Issue
Closed
opened 2026-02-22 18:25:38 -05:00 by yindo · 3 comments
yindo commented 2026-02-22 18:25:38 -05:00
Owner
Copy Link
Copy Source

Originally created by @carlscutt on GitHub (Oct 15, 2024).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/2475

How are you running AnythingLLM?

Docker (local)

What happened?

The set up is Docker + anythingll installed on the local machine with Ollama running locally. All functions work bar one. Entering a URL and selecting "Fetch website" fails. https://www.sharks.org or http://www.sharks.org. I see "Scraping Link" Then: Error uploading link. A processing error occurred.

Are there known steps to reproduce?

No response

Originally created by @carlscutt on GitHub (Oct 15, 2024). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/2475 ### How are you running AnythingLLM? Docker (local) ### What happened? The set up is Docker + anythingll installed on the local machine with Ollama running locally. All functions work bar one. Entering a URL and selecting "Fetch website" fails. https://www.sharks.org or http://www.sharks.org. I see "Scraping Link" Then: Error uploading link. A processing error occurred. ### Are there known steps to reproduce? _No response_
yindo added the possible bug label 2026-02-22 18:25:38 -05:00
yindo closed this issue 2026-02-22 18:25:38 -05:00
yindo commented 2026-02-22 18:25:38 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Oct 15, 2024):

Please pull the logs from the container so we can discover more definitive root cause as this is a generic error at this time.

Willing to bet you will get something about the chromium process not running due to root or something along those lines. Be sure you are using the --cap-add SYS_ADMIN

example:

export STORAGE_LOCATION=$HOME/anythingllm && \
mkdir -p $STORAGE_LOCATION && \
touch "$STORAGE_LOCATION/.env" && \
docker run -d -p 3001:3001 \
--cap-add SYS_ADMIN \
-v ${STORAGE_LOCATION}:/app/server/storage \
-v ${STORAGE_LOCATION}/.env:/app/server/.env \
-e STORAGE_DIR="/app/server/storage" \
mintplexlabs/anythingllm

Otherwise, closing until we find something more "bug" related or actionable 👍

@timothycarambat commented on GitHub (Oct 15, 2024): Please pull the logs from the container so we can discover more definitive root cause as this is a generic error at this time. Willing to bet you will get something about the chromium process not running due to `root` or something along those lines. Be sure you are using the `--cap-add SYS_ADMIN` example: ``` export STORAGE_LOCATION=$HOME/anythingllm && \ mkdir -p $STORAGE_LOCATION && \ touch "$STORAGE_LOCATION/.env" && \ docker run -d -p 3001:3001 \ --cap-add SYS_ADMIN \ -v ${STORAGE_LOCATION}:/app/server/storage \ -v ${STORAGE_LOCATION}/.env:/app/server/.env \ -e STORAGE_DIR="/app/server/storage" \ mintplexlabs/anythingllm ``` Otherwise, closing until we find something more "bug" related or actionable 👍
yindo commented 2026-02-22 18:25:39 -05:00
Author
Owner
Copy Link
Copy Source

@carlscutt commented on GitHub (Oct 17, 2024):

Very sorry to take so long.

2024-10-17T12:47:21.107238864Z [collector] info: -- Working URL
https://www.sharks.org --
2024-10-17T12:47:49.126363969Z [collector] error: getPageContent failed to
be fetched by puppeteer - falling back to fetch! Error:
net::ERR_NAME_NOT_RESOLVED at https://www.sharks.org
2024-10-17T12:47:49.126436244Z at navigate
(file:///app/collector/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/Frame.js:173:27)
2024-10-17T12:47:49.126446058Z at process.processTicksAndRejections
(node:internal/process/task_queues:95:5)
2024-10-17T12:47:49.126452678Z at async Deferred.race
(file:///app/collector/node_modules/puppeteer-core/lib/esm/puppeteer/util/Deferred.js:80:20)
2024-10-17T12:47:49.126459563Z at async CdpFrame.goto
(file:///app/collector/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/Frame.js:139:25)
2024-10-17T12:47:49.126465124Z at async CdpPage.goto
(file:///app/collector/node_modules/puppeteer-core/lib/esm/puppeteer/api/Page.js:568:20)
2024-10-17T12:47:49.126470703Z at async PuppeteerWebBaseLoader._scrape
(/app/collector/node_modules/langchain/dist/document_loaders/web/puppeteer.cjs:49:9)
2024-10-17T12:47:49.126476659Z at async PuppeteerWebBaseLoader.load
(/app/collector/node_modules/langchain/dist/document_loaders/web/puppeteer.cjs:74:22)
2024-10-17T12:47:49.126483227Z at async getPageContent
(/app/collector/processLink/convert/generic.js:72:18)
2024-10-17T12:47:49.126488959Z at async scrapeGenericUrl
(/app/collector/processLink/convert/generic.js:11:19)
2024-10-17T12:47:49.126494302Z at async processLink
(/app/collector/processLink/index.js:6:10)
2024-10-17T12:47:59.226376570Z [collector] error: getPageContent failed to
be fetched by any method. TypeError: fetch failed
2024-10-17T12:47:59.226661395Z at
node:internal/deps/undici/undici:12618:11
2024-10-17T12:47:59.226677329Z at process.processTicksAndRejections
(node:internal/process/task_queues:95:5)
2024-10-17T12:47:59.226688471Z at async getPageContent
(/app/collector/processLink/convert/generic.js:87:22)
2024-10-17T12:47:59.226698467Z at async scrapeGenericUrl
(/app/collector/processLink/convert/generic.js:11:19)
2024-10-17T12:47:59.226707723Z at async processLink
(/app/collector/processLink/index.js:6:10)
2024-10-17T12:47:59.226717713Z at async /app/collector/index.js:67:51
2024-10-17T12:47:59.230978562Z [collector] error: TypeError: Cannot read
properties of null (reading 'length')
2024-10-17T12:47:59.231064942Z at scrapeGenericUrl
(/app/collector/processLink/convert/generic.js:13:16)
2024-10-17T12:47:59.231079268Z at process.processTicksAndRejections
(node:internal/process/task_queues:95:5)
2024-10-17T12:47:59.231090119Z at async processLink
(/app/collector/processLink/index.js:6:10)
2024-10-17T12:47:59.231100203Z at async /app/collector/index.js:67:51

On Tue, 15 Oct 2024 at 21:24, Timothy Carambat @.***>
wrote:

Please pull the logs from the container so we can discover more definitive
root cause as this is a generic error at this time.

Willing to bet you will get something about the chromium process not
running due to root or something along those lines. Be sure you are using
the --cap-add SYS_ADMIN

example:

export STORAGE_LOCATION=$HOME/anythingllm &&
mkdir -p $STORAGE_LOCATION &&
touch "$STORAGE_LOCATION/.env" &&
docker run -d -p 3001:3001
--cap-add SYS_ADMIN
-v ${STORAGE_LOCATION}:/app/server/storage
-v ${STORAGE_LOCATION}/.env:/app/server/.env
-e STORAGE_DIR="/app/server/storage"
mintplexlabs/anythingllm

Otherwise, closing until we find something more "bug" related or
actionable 👍

—
Reply to this email directly, view it on GitHub
https://github.com/Mintplex-Labs/anything-llm/issues/2475#issuecomment-2414830793,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/BKCM7AI4NGV67XT6SBMW5ALZ3VTVDAVCNFSM6AAAAABP62XXFSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMJUHAZTANZZGM
.
You are receiving this because you authored the thread.Message ID:
@.***>

--
Carl Scutt
+34 687 697 628
@.***

@carlscutt commented on GitHub (Oct 17, 2024): Very sorry to take so long. 2024-10-17T12:47:21.107238864Z [collector] info: -- Working URL https://www.sharks.org -- 2024-10-17T12:47:49.126363969Z [collector] error: getPageContent failed to be fetched by puppeteer - falling back to fetch! Error: net::ERR_NAME_NOT_RESOLVED at https://www.sharks.org 2024-10-17T12:47:49.126436244Z at navigate (file:///app/collector/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/Frame.js:173:27) 2024-10-17T12:47:49.126446058Z at process.processTicksAndRejections (node:internal/process/task_queues:95:5) 2024-10-17T12:47:49.126452678Z at async Deferred.race (file:///app/collector/node_modules/puppeteer-core/lib/esm/puppeteer/util/Deferred.js:80:20) 2024-10-17T12:47:49.126459563Z at async CdpFrame.goto (file:///app/collector/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/Frame.js:139:25) 2024-10-17T12:47:49.126465124Z at async CdpPage.goto (file:///app/collector/node_modules/puppeteer-core/lib/esm/puppeteer/api/Page.js:568:20) 2024-10-17T12:47:49.126470703Z at async PuppeteerWebBaseLoader._scrape (/app/collector/node_modules/langchain/dist/document_loaders/web/puppeteer.cjs:49:9) 2024-10-17T12:47:49.126476659Z at async PuppeteerWebBaseLoader.load (/app/collector/node_modules/langchain/dist/document_loaders/web/puppeteer.cjs:74:22) 2024-10-17T12:47:49.126483227Z at async getPageContent (/app/collector/processLink/convert/generic.js:72:18) 2024-10-17T12:47:49.126488959Z at async scrapeGenericUrl (/app/collector/processLink/convert/generic.js:11:19) 2024-10-17T12:47:49.126494302Z at async processLink (/app/collector/processLink/index.js:6:10) 2024-10-17T12:47:59.226376570Z [collector] error: getPageContent failed to be fetched by any method. TypeError: fetch failed 2024-10-17T12:47:59.226661395Z at node:internal/deps/undici/undici:12618:11 2024-10-17T12:47:59.226677329Z at process.processTicksAndRejections (node:internal/process/task_queues:95:5) 2024-10-17T12:47:59.226688471Z at async getPageContent (/app/collector/processLink/convert/generic.js:87:22) 2024-10-17T12:47:59.226698467Z at async scrapeGenericUrl (/app/collector/processLink/convert/generic.js:11:19) 2024-10-17T12:47:59.226707723Z at async processLink (/app/collector/processLink/index.js:6:10) 2024-10-17T12:47:59.226717713Z at async /app/collector/index.js:67:51 2024-10-17T12:47:59.230978562Z [collector] error: TypeError: Cannot read properties of null (reading 'length') 2024-10-17T12:47:59.231064942Z at scrapeGenericUrl (/app/collector/processLink/convert/generic.js:13:16) 2024-10-17T12:47:59.231079268Z at process.processTicksAndRejections (node:internal/process/task_queues:95:5) 2024-10-17T12:47:59.231090119Z at async processLink (/app/collector/processLink/index.js:6:10) 2024-10-17T12:47:59.231100203Z at async /app/collector/index.js:67:51 On Tue, 15 Oct 2024 at 21:24, Timothy Carambat ***@***.***> wrote: > Please pull the logs from the container so we can discover more definitive > root cause as this is a generic error at this time. > > Willing to bet you will get something about the chromium process not > running due to root or something along those lines. Be sure you are using > the --cap-add SYS_ADMIN > > example: > > export STORAGE_LOCATION=$HOME/anythingllm && \ > mkdir -p $STORAGE_LOCATION && \ > touch "$STORAGE_LOCATION/.env" && \ > docker run -d -p 3001:3001 \ > --cap-add SYS_ADMIN \ > -v ${STORAGE_LOCATION}:/app/server/storage \ > -v ${STORAGE_LOCATION}/.env:/app/server/.env \ > -e STORAGE_DIR="/app/server/storage" \ > mintplexlabs/anythingllm > > Otherwise, closing until we find something more "bug" related or > actionable 👍 > > — > Reply to this email directly, view it on GitHub > <https://github.com/Mintplex-Labs/anything-llm/issues/2475#issuecomment-2414830793>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/BKCM7AI4NGV67XT6SBMW5ALZ3VTVDAVCNFSM6AAAAABP62XXFSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMJUHAZTANZZGM> > . > You are receiving this because you authored the thread.Message ID: > ***@***.***> > -- Carl Scutt +34 687 697 628 ***@***.***
yindo commented 2026-02-22 18:25:39 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Oct 17, 2024):

net::ERR_NAME_NOT_RESOLVED at https://www.sharks.org/

This happens for both puppeteer and regular fetch, which normal indicates the HTTP/HTTPS connection itself is not resolving. Are you using a VPN or anything like that? Looks like the site does not resolve and that is the entire reason

@timothycarambat commented on GitHub (Oct 17, 2024): > net::ERR_NAME_NOT_RESOLVED at https://www.sharks.org/ This happens for both puppeteer and regular fetch, which normal indicates the HTTP/HTTPS connection itself is not resolving. Are you using a VPN or anything like that? Looks like the site does not resolve and that is the entire reason
yindo changed title from [BUG]: Failed Scrapper to [GH-ISSUE #2475] [BUG]: Failed Scrapper 2026-06-05 14:41:42 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label possible bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#1607
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?