Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 22:23:50 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #2400] [BUG]: Bulk Link Scraper not wokring. #1558

New Issue
Closed
opened 2026-02-22 18:25:24 -05:00 by yindo · 4 comments
yindo commented 2026-02-22 18:25:24 -05:00
Owner
Copy Link
Copy Source

Originally created by @amitshuklabag on GitHub (Oct 1, 2024).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/2400

How are you running AnythingLLM?

Docker (remote machine)

What happened?

While using the Bulk Link Scraper it shows the below error:

app-anything-llm-1 | [collector] error: Failed to get page links from https://melbadigital.fi/. Error: Could not find Chrome (ver. 119.0.6045.105). This can occur if either app-anything-llm-1 | 1. you did not perform an installation before running the script (e.g. npm install) or app-anything-llm-1 | 2. your cache path is incorrectly configured (which is: /root/.cache/puppeteer). app-anything-llm-1 | For (2), check out our guide on configuring puppeteer at https://pptr.dev/guides/configuration. app-anything-llm-1 | at ChromeLauncher.resolveExecutablePath (file:///app/collector/node_modules/puppeteer-core/lib/esm/puppeteer/node/ProductLauncher.js:260:27) app-anything-llm-1 | at ChromeLauncher.executablePath (file:///app/collector/node_modules/puppeteer-core/lib/esm/puppeteer/node/ChromeLauncher.js:197:25) app-anything-llm-1 | at ChromeLauncher.computeLaunchArguments (file:///app/collector/node_modules/puppeteer-core/lib/esm/puppeteer/node/ChromeLauncher.js:91:37) app-anything-llm-1 | at async ChromeLauncher.launch (file:///app/collector/node_modules/puppeteer-core/lib/esm/puppeteer/node/ProductLauncher.js:53:28) app-anything-llm-1 | at async PuppeteerWebBaseLoader._scrape (/app/collector/node_modules/langchain/dist/document_loaders/web/puppeteer.cjs:42:25) app-anything-llm-1 | at async PuppeteerWebBaseLoader.load (/app/collector/node_modules/langchain/dist/document_loaders/web/puppeteer.cjs:74:22) app-anything-llm-1 | at async getPageLinks (/app/collector/utils/extensions/WebsiteDepth/index.js:51:18) app-anything-llm-1 | at async discoverLinks (/app/collector/utils/extensions/WebsiteDepth/index.js:22:22) app-anything-llm-1 | at async websiteScraper (/app/collector/utils/extensions/WebsiteDepth/index.js:151:25) app-anything-llm-1 | at async /app/collector/extensions/index.js:124:29

Are there known steps to reproduce?

Deploy the AnythingLLM using your docker image mintplexlabs/anythingllm with the latest or 1.1.1 tag.

Go to Data connection -> Bulk Link Scraper -> Website URL -> pass it https://melbadigital.fi/ -> click the button "Submit"

You will get the same error.

Here is my docker-compose:
`version: "3.3"

services:
anything-llm:
user: 0:0
image: mintplexlabs/anythingllm:latest
restart: always
cap_add:
- SYS_ADMIN
volumes:
- "./docker.env:/app/server/.env"
- "./storage:/app/server/storage"
- "./collector/hotdir/:/app/collector/hotdir"
- "./collector/outputs/:/app/collector/outputs"
ports:
- "172.17.0.1:52840:3001"
env_file:
- ./docker.env
extra_hosts:
- "xxxxxx:xxxxxxxxxxxx"

volumes:
storage:
driver: local
driver_opts:
type: none
device: ${PWD}/storage
o: bind`

Originally created by @amitshuklabag on GitHub (Oct 1, 2024). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/2400 ### How are you running AnythingLLM? Docker (remote machine) ### What happened? While using the Bulk Link Scraper it shows the below error: `app-anything-llm-1 | [collector] error: Failed to get page links from https://melbadigital.fi/. Error: Could not find Chrome (ver. 119.0.6045.105). This can occur if either app-anything-llm-1 | 1. you did not perform an installation before running the script (e.g. `npm install`) or app-anything-llm-1 | 2. your cache path is incorrectly configured (which is: /root/.cache/puppeteer). app-anything-llm-1 | For (2), check out our guide on configuring puppeteer at https://pptr.dev/guides/configuration. app-anything-llm-1 | at ChromeLauncher.resolveExecutablePath (file:///app/collector/node_modules/puppeteer-core/lib/esm/puppeteer/node/ProductLauncher.js:260:27) app-anything-llm-1 | at ChromeLauncher.executablePath (file:///app/collector/node_modules/puppeteer-core/lib/esm/puppeteer/node/ChromeLauncher.js:197:25) app-anything-llm-1 | at ChromeLauncher.computeLaunchArguments (file:///app/collector/node_modules/puppeteer-core/lib/esm/puppeteer/node/ChromeLauncher.js:91:37) app-anything-llm-1 | at async ChromeLauncher.launch (file:///app/collector/node_modules/puppeteer-core/lib/esm/puppeteer/node/ProductLauncher.js:53:28) app-anything-llm-1 | at async PuppeteerWebBaseLoader._scrape (/app/collector/node_modules/langchain/dist/document_loaders/web/puppeteer.cjs:42:25) app-anything-llm-1 | at async PuppeteerWebBaseLoader.load (/app/collector/node_modules/langchain/dist/document_loaders/web/puppeteer.cjs:74:22) app-anything-llm-1 | at async getPageLinks (/app/collector/utils/extensions/WebsiteDepth/index.js:51:18) app-anything-llm-1 | at async discoverLinks (/app/collector/utils/extensions/WebsiteDepth/index.js:22:22) app-anything-llm-1 | at async websiteScraper (/app/collector/utils/extensions/WebsiteDepth/index.js:151:25) app-anything-llm-1 | at async /app/collector/extensions/index.js:124:29` ### Are there known steps to reproduce? Deploy the AnythingLLM using your docker image mintplexlabs/anythingllm with the latest or 1.1.1 tag. Go to Data connection -> Bulk Link Scraper -> Website URL -> pass it https://melbadigital.fi/ -> click the button "Submit" You will get the same error. Here is my docker-compose: `version: "3.3" services: anything-llm: user: 0:0 image: mintplexlabs/anythingllm:latest restart: always cap_add: - SYS_ADMIN volumes: - "./docker.env:/app/server/.env" - "./storage:/app/server/storage" - "./collector/hotdir/:/app/collector/hotdir" - "./collector/outputs/:/app/collector/outputs" ports: - "172.17.0.1:52840:3001" env_file: - ./docker.env extra_hosts: - "xxxxxx:xxxxxxxxxxxx" volumes: storage: driver: local driver_opts: type: none device: ${PWD}/storage o: bind`
yindo added the needs info / can't replicate label 2026-02-22 18:25:24 -05:00
yindo closed this issue 2026-02-22 18:25:24 -05:00
yindo commented 2026-02-22 18:25:24 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Oct 1, 2024):

  • "./collector/hotdir/:/app/collector/hotdir"
  • "./collector/outputs/:/app/collector/outputs"

Those mounts are not needed and could be messing with how the Puppeteer cache location is found.

Attempting to replicate:

  • docker pull mintplexlabs/anythingllm
export STORAGE_LOCATION=$HOME/anythingllm && \
mkdir -p $STORAGE_LOCATION && \
touch "$STORAGE_LOCATION/.env" && \
docker run -d -p 3001:3001 \
--cap-add SYS_ADMIN \
-v ${STORAGE_LOCATION}:/app/server/storage \
-v ${STORAGE_LOCATION}/.env:/app/server/.env \
-e STORAGE_DIR="/app/server/storage" \
mintplexlabs/anythingllm

Open workspace, go to data connector, enter bulk link scrape with website - get the following:

[collector] info: Discovering links...
[collector] info: Found 20 links to scrape.
[collector] info: Starting bulk scraping...
[collector] info: Scraping 1/20: https://melbadigital.fi
[collector] info: Successfully scraped https://melbadigital.fi.
[collector] info: Scraping 2/20: https://melbadigital.fi/
.....
@timothycarambat commented on GitHub (Oct 1, 2024): > - "./collector/hotdir/:/app/collector/hotdir" > - "./collector/outputs/:/app/collector/outputs" Those mounts are not needed and could be messing with how the Puppeteer cache location is found. Attempting to replicate: - `docker pull mintplexlabs/anythingllm` ```bash export STORAGE_LOCATION=$HOME/anythingllm && \ mkdir -p $STORAGE_LOCATION && \ touch "$STORAGE_LOCATION/.env" && \ docker run -d -p 3001:3001 \ --cap-add SYS_ADMIN \ -v ${STORAGE_LOCATION}:/app/server/storage \ -v ${STORAGE_LOCATION}/.env:/app/server/.env \ -e STORAGE_DIR="/app/server/storage" \ mintplexlabs/anythingllm ``` Open workspace, go to data connector, enter bulk link scrape with website - get the following: ``` [collector] info: Discovering links... [collector] info: Found 20 links to scrape. [collector] info: Starting bulk scraping... [collector] info: Scraping 1/20: https://melbadigital.fi [collector] info: Successfully scraped https://melbadigital.fi. [collector] info: Scraping 2/20: https://melbadigital.fi/ ..... ```
yindo commented 2026-02-22 18:25:24 -05:00
Author
Owner
Copy Link
Copy Source

@amitshuklabag commented on GitHub (Oct 3, 2024):

Based on your suggestion I have tried it with docker-compose, but the issues are now different.

version: "3.8"
services:
  anything-llm:
    image: mintplexlabs/anythingllm
    restart: always
    cap_add:
      - SYS_ADMIN
    user: "0:0"   # Run as root to avoid permission issues
    volumes:
      - "./storage:/app/server/storage"     
      - "./docker.env:/app/server/.env"     
    environment:
      - STORAGE_DIR="/app/server/storage"   
    ports:
      - "172.17.0.1:3001:3001"

It returns the error: "Error: Response could not be completed"

But when I try it with

docker run -d -p 172.17.0.1:3001:3001 \
--cap-add SYS_ADMIN \
-v ./storage:/app/server/storage \
-v ./docker.env:/app/server/.env \
-e STORAGE_DIR="/app/server/storage" \
--restart always \
mintplexlabs/anythingllm

it works fine.

So, could you please check and let me know what was wrong in my docker-compose, or maybe you can try it with the docker-compose file and send me which one is working.

@amitshuklabag commented on GitHub (Oct 3, 2024): Based on your suggestion I have tried it with docker-compose, but the issues are now different. ``` version: "3.8" services: anything-llm: image: mintplexlabs/anythingllm restart: always cap_add: - SYS_ADMIN user: "0:0" # Run as root to avoid permission issues volumes: - "./storage:/app/server/storage" - "./docker.env:/app/server/.env" environment: - STORAGE_DIR="/app/server/storage" ports: - "172.17.0.1:3001:3001" ``` It returns the error: "Error: Response could not be completed" But when I try it with ``` docker run -d -p 172.17.0.1:3001:3001 \ --cap-add SYS_ADMIN \ -v ./storage:/app/server/storage \ -v ./docker.env:/app/server/.env \ -e STORAGE_DIR="/app/server/storage" \ --restart always \ mintplexlabs/anythingllm ``` it works fine. So, could you please check and let me know what was wrong in my docker-compose, or maybe you can try it with the docker-compose file and send me which one is working.
yindo commented 2026-02-22 18:25:25 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Oct 3, 2024):

It is likely just this line in your compose:

user: "0:0"

You should not be running the whole image as root, since this is insecure for chromium, thus fails to scrape any websites via puppeteer

@timothycarambat commented on GitHub (Oct 3, 2024): It is likely just this line in your compose: > user: "0:0" You should not be running the whole image as root, since this is insecure for chromium, thus fails to scrape any websites via puppeteer
yindo commented 2026-02-22 18:25:25 -05:00
Author
Owner
Copy Link
Copy Source

@amitshuklabag commented on GitHub (Oct 4, 2024):

Thanks for all of your help, it's working now.

@amitshuklabag commented on GitHub (Oct 4, 2024): Thanks for all of your help, it's working now.
yindo changed title from [BUG]: Bulk Link Scraper not wokring. to [GH-ISSUE #2400] [BUG]: Bulk Link Scraper not wokring. 2026-06-05 14:41:25 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label needs info / can't replicate
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#1558
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?