langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 12:05:30 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Watercrawl is not configured despite is configured #20278

New Issue
Closed
opened 2026-02-21 20:06:39 -05:00 by yindo · 6 comments
yindo commented 2026-02-21 20:06:39 -05:00
Owner
Copy Link
Copy Source

Originally created by @chenyuAC on GitHub (Nov 12, 2025).

Self Checks

  • I have read the Contributing Guide and Language Policy.
  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report, otherwise it will be closed.
  • 【中文用户 & Non English User】请使用英语提交，否则会被关闭 ：）
  • Please do not modify this template :) and fill in all the required fields.

Dify version

1.9.2

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

  1. Create knowledge
  2. Sync from Website
  3. Select a provider - Watercrawl
  4. Configure Watercrawl

✔️ Expected Behavior

Able to use Watercrawl

❌ Actual Behavior

Still shows Watercrawl is not configured

Originally created by @chenyuAC on GitHub (Nov 12, 2025). ### Self Checks - [x] I have read the [Contributing Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) and [Language Policy](https://github.com/langgenius/dify/issues/1542). - [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report, otherwise it will be closed. - [x] 【中文用户 & Non English User】请使用英语提交，否则会被关闭 ：） - [x] Please do not modify this template :) and fill in all the required fields. ### Dify version 1.9.2 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce 1. Create knowledge 2. Sync from Website 3. Select a provider - Watercrawl 4. Configure Watercrawl ### ✔️ Expected Behavior Able to use Watercrawl ### ❌ Actual Behavior Still shows `Watercrawl is not configured`
yindo added the 🐞 bug label 2026-02-21 20:06:39 -05:00
yindo closed this issue 2026-02-21 20:06:39 -05:00
yindo commented 2026-02-21 20:06:39 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Nov 12, 2025):

Hi @chenyuAC! I'm Dosu and I’m helping the dify team.

The "Watercrawl is not configured" message is controlled by a frontend environment variable, not by checking backend credentials or actual provider setup. For Watercrawl to show as configured, you need to set ENABLE_WEBSITE_WATERCRAWL=true (or NEXT_PUBLIC_ENABLE_WEBSITE_WATERCRAWL=true) in your .env file or Docker environment, and make sure this variable is passed to both the frontend and backend containers. Recent updates to the Docker templates ensure this variable is included by default, but if your setup is missing it or it isn't applied, Watercrawl will appear as "not configured" even if you've set up credentials or API keys correctly. Restart your containers after updating the environment variables to apply changes.

If you already set the variable and still see the issue, double-check that your frontend build picked up the change and that your backend credentials (API key for Watercrawl) are correctly configured in the provider settings. The UI only checks the env variable, so mismatches between UI and backend config can happen.
Reference: docker-compose env config
Reference: frontend config logic

To reply, just mention @dosu.

How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other  Chat with Dosu Join Discord Share on X

@dosubot[bot] commented on GitHub (Nov 12, 2025): <!-- Greeting --> Hi @chenyuAC! I'm [Dosu](https://go.dosu.dev/dosubot) and I’m helping the dify team. <!-- Answer --> The "Watercrawl is not configured" message is controlled by a frontend environment variable, not by checking backend credentials or actual provider setup. For Watercrawl to show as configured, you need to set `ENABLE_WEBSITE_WATERCRAWL=true` (or `NEXT_PUBLIC_ENABLE_WEBSITE_WATERCRAWL=true`) in your `.env` file or Docker environment, and make sure this variable is passed to both the frontend and backend containers. Recent updates to the Docker templates ensure this variable is included by default, but if your setup is missing it or it isn't applied, Watercrawl will appear as "not configured" even if you've set up credentials or API keys correctly. Restart your containers after updating the environment variables to apply changes. If you already set the variable and still see the issue, double-check that your frontend build picked up the change and that your backend credentials (API key for Watercrawl) are correctly configured in the provider settings. The UI only checks the env variable, so mismatches between UI and backend config can happen. [Reference: docker-compose env config](https://github.com/langgenius/dify/pull/27704) [Reference: frontend config logic](https://github.com/langgenius/dify/blob/b76e17b25d51fa98cf9108a3d0a65a2d8b44ae68/web/config/index.ts#L375-L379) <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- <sup>How did I do? [Good](https://app.dosu.dev/response-feedback/3d1e0365-a0fa-4771-80ab-d4a0e71c338e?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/3d1e0365-a0fa-4771-80ab-d4a0e71c338e?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/3d1e0365-a0fa-4771-80ab-d4a0e71c338e?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/3d1e0365-a0fa-4771-80ab-d4a0e71c338e?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/3d1e0365-a0fa-4771-80ab-d4a0e71c338e?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/3d1e0365-a0fa-4771-80ab-d4a0e71c338e?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/3d1e0365-a0fa-4771-80ab-d4a0e71c338e?feedback_type=other)</sup>&nbsp;&nbsp;[![Chat with Dosu](https://dosu.dev/dosu-chat-badge.svg)](https://app.dosu.dev/6088a4f4-073a-43f3-b3d4-4f06fb0c05a0/ask?utm_source=github)&nbsp;[![Join Discord](https://img.shields.io/badge/join-5865F2?logo=discord&logoColor=white&label=)](https://go.dosu.dev/discord-bot)&nbsp;[![Share on X](https://img.shields.io/badge/X-share-black)](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/langgenius/dify/issues/28129)
yindo commented 2026-02-21 20:06:39 -05:00
Author
Owner
Copy Link
Copy Source

@chenyuAC commented on GitHub (Nov 12, 2025):

@dosu the env variables you mentioned are set to true.

A quick analysis of the reveals:

Root Cause

In api/services/datasource_provider_service.py:817-821, the get_hard_code_datasource_credentials() method only includes these plugin IDs in its hardcoded list:

if datasource.plugin_id in [
    "langgenius/firecrawl_datasource",
    "langgenius/notion_datasource",
    "langgenius/jina_datasource",
]:

The "langgenius/watercrawl_datasource" plugin is missing from this list!

How the Bug Manifests

  1. The frontend calls /auth/plugin/datasource/default-list to get available datasources
  2. This endpoint calls get_hard_code_datasource_credentials() which only returns datasources in the hardcoded list
  3. Watercrawl is excluded, so it returns an empty credentials_list for Watercrawl
  4. In web/app/components/datasets/create/website/index.tsx:42-47, the frontend filters for providers where item.credentials_list.length > 0
  5. Since Watercrawl has no credentials in the list (due to the backend bug), it's excluded from availableProviders
  6. At line 140, when !source, the NoData component is shown with the "not configured" message
@chenyuAC commented on GitHub (Nov 12, 2025): @dosu the env variables you mentioned are set to true. A quick analysis of the reveals: ## Root Cause In [`api/services/datasource_provider_service.py:817-821`](https://github.com/langgenius/dify/blob/ec871819e46de82f2f9ee33a71a61d96a58bdb37/api/services/datasource_provider_service.py#L817-L821), the `get_hard_code_datasource_credentials()` method only includes these plugin IDs in its hardcoded list: ```python if datasource.plugin_id in [ "langgenius/firecrawl_datasource", "langgenius/notion_datasource", "langgenius/jina_datasource", ]: ``` **The `"langgenius/watercrawl_datasource"` plugin is missing from this list!** ## How the Bug Manifests 1. The frontend calls `/auth/plugin/datasource/default-list` to get available datasources 2. This endpoint calls `get_hard_code_datasource_credentials()` which only returns datasources in the hardcoded list 3. Watercrawl is excluded, so it returns an empty `credentials_list` for Watercrawl 4. In [`web/app/components/datasets/create/website/index.tsx:42-47`](https://github.com/langgenius/dify/blob/ec871819e46de82f2f9ee33a71a61d96a58bdb37/web/app/components/datasets/create/website/index.tsx#L42-L47), the frontend filters for providers where `item.credentials_list.length > 0` 5. Since Watercrawl has no credentials in the list (due to the backend bug), it's excluded from `availableProviders` 6. At [line 140](https://github.com/langgenius/dify/blob/ec871819e46de82f2f9ee33a71a61d96a58bdb37/web/app/components/datasets/create/website/index.tsx#L140), when `!source`, the NoData component is shown with the "not configured" message
yindo commented 2026-02-21 20:06:39 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Nov 12, 2025):

Your root cause analysis is spot on: Watercrawl is missing from the hardcoded plugin ID list in get_hard_code_datasource_credentials(), which leads to the frontend filtering it out and showing "not configured" even when credentials are set.

You can safely fix this by adding "langgenius/watercrawl_datasource" to the list in that function. This change is isolated to the datasource credentials listing logic and will make Watercrawl available in the frontend as intended—no side effects or special handling required, as long as Watercrawl's credential schema matches the other providers' conventions. After updating, restart your backend service to apply the change. This will resolve the UI issue immediately and let you use Watercrawl as expected.
Reference: code logic and impact

To reply, just mention @dosu.

How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other  Chat with Dosu Join Discord Share on X

@dosubot[bot] commented on GitHub (Nov 12, 2025): <!-- Answer --> Your root cause analysis is spot on: Watercrawl is missing from the hardcoded plugin ID list in get_hard_code_datasource_credentials(), which leads to the frontend filtering it out and showing "not configured" even when credentials are set. You can safely fix this by adding "langgenius/watercrawl_datasource" to the list in that function. This change is isolated to the datasource credentials listing logic and will make Watercrawl available in the frontend as intended—no side effects or special handling required, as long as Watercrawl's credential schema matches the other providers' conventions. After updating, restart your backend service to apply the change. This will resolve the UI issue immediately and let you use Watercrawl as expected. [Reference: code logic and impact](https://github.com/langgenius/dify/blob/b76e17b25d51fa98cf9108a3d0a65a2d8b44ae68/api/services/datasource_provider_service.py#L37-L991) <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- <sup>How did I do? [Good](https://app.dosu.dev/response-feedback/4a1d66a9-4f28-4e2b-900a-ac87f2b99e51?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/4a1d66a9-4f28-4e2b-900a-ac87f2b99e51?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/4a1d66a9-4f28-4e2b-900a-ac87f2b99e51?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/4a1d66a9-4f28-4e2b-900a-ac87f2b99e51?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/4a1d66a9-4f28-4e2b-900a-ac87f2b99e51?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/4a1d66a9-4f28-4e2b-900a-ac87f2b99e51?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/4a1d66a9-4f28-4e2b-900a-ac87f2b99e51?feedback_type=other)</sup>&nbsp;&nbsp;[![Chat with Dosu](https://dosu.dev/dosu-chat-badge.svg)](https://app.dosu.dev/6088a4f4-073a-43f3-b3d4-4f06fb0c05a0/ask?utm_source=github)&nbsp;[![Join Discord](https://img.shields.io/badge/join-5865F2?logo=discord&logoColor=white&label=)](https://go.dosu.dev/discord-bot)&nbsp;[![Share on X](https://img.shields.io/badge/X-share-black)](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/langgenius/dify/issues/28129)
yindo commented 2026-02-21 20:06:39 -05:00
Author
Owner
Copy Link
Copy Source

@80tech commented on GitHub (Dec 11, 2025):

hi @chenyuAC @dosu,

I encountered the same issue on version 1.11.0. Following your instructions, I initially tried adding langgenius/watercrawl_datasource to /app/api/services/datasource_provider_service.py, but this did not work. However, after changing the entry to watercrawl/watercrawl_datasource, I was successfully able to configure Watercrawl on the "Sync from Website" page.

When I started the crawling process, it showed an unknown error: 'NoneType' object has no attribute 'encode'. Do you have any suggestions for resolving this?

Thanks

@80tech commented on GitHub (Dec 11, 2025): hi @chenyuAC @dosu, I encountered the same issue on version 1.11.0. Following your instructions, I initially tried adding langgenius/watercrawl_datasource to /app/api/services/datasource_provider_service.py, but this did not work. However, after changing the entry to watercrawl/watercrawl_datasource, I was successfully able to configure Watercrawl on the "Sync from Website" page. When I started the crawling process, it showed an unknown error: 'NoneType' object has no attribute 'encode'. Do you have any suggestions for resolving this? Thanks
yindo commented 2026-02-21 20:06:40 -05:00
Author
Owner
Copy Link
Copy Source

@E218PQ commented on GitHub (Dec 12, 2025):

@dosu the env variables you mentioned are set to true.

A quick analysis of the reveals:

Root Cause

In api/services/datasource_provider_service.py:817-821, the method only includes these plugin IDs in its hardcoded list:get_hard_code_datasource_credentials()

if datasource.plugin_id in [
"langgenius/firecrawl_datasource",
"langgenius/notion_datasource",
"langgenius/jina_datasource",
]:
The "langgenius/watercrawl_datasource" plugin is missing from this list!

How the Bug Manifests

  1. The frontend calls to get available datasources/auth/plugin/datasource/default-list
  2. This endpoint calls which only returns datasources in the hardcoded listget_hard_code_datasource_credentials()
  3. Watercrawl is excluded, so it returns an empty for Watercrawlcredentials_list
  4. In web/app/components/datasets/create/website/index.tsx:42-47, the frontend filters for providers where item.credentials_list.length > 0
  5. Since Watercrawl has no credentials in the list (due to the backend bug), it's excluded from availableProviders
  6. At line 140, when , the NoData component is shown with the "not configured" message!source

@chenyuAC
Excuse me, I encountered the same problem, but I followed the instructions and added "langgenius/waterscrawl_datasource" to api/services/datasourcesprovider_dervice. py. After restarting, it doesn't seem to have changed. Is there any other operation I missed?

@E218PQ commented on GitHub (Dec 12, 2025): > [@dosu](https://github.com/dosu) the env variables you mentioned are set to true. > > A quick analysis of the reveals: > > ## Root Cause > In [`api/services/datasource_provider_service.py:817-821`](https://github.com/langgenius/dify/blob/ec871819e46de82f2f9ee33a71a61d96a58bdb37/api/services/datasource_provider_service.py#L817-L821), the method only includes these plugin IDs in its hardcoded list:`get_hard_code_datasource_credentials()` > > if datasource.plugin_id in [ > "langgenius/firecrawl_datasource", > "langgenius/notion_datasource", > "langgenius/jina_datasource", > ]: > **The `"langgenius/watercrawl_datasource"` plugin is missing from this list!** > > ## How the Bug Manifests > 1. The frontend calls to get available datasources`/auth/plugin/datasource/default-list` > 2. This endpoint calls which only returns datasources in the hardcoded list`get_hard_code_datasource_credentials()` > 3. Watercrawl is excluded, so it returns an empty for Watercrawl`credentials_list` > 4. In [`web/app/components/datasets/create/website/index.tsx:42-47`](https://github.com/langgenius/dify/blob/ec871819e46de82f2f9ee33a71a61d96a58bdb37/web/app/components/datasets/create/website/index.tsx#L42-L47), the frontend filters for providers where `item.credentials_list.length > 0` > 5. Since Watercrawl has no credentials in the list (due to the backend bug), it's excluded from `availableProviders` > 6. At [line 140](https://github.com/langgenius/dify/blob/ec871819e46de82f2f9ee33a71a61d96a58bdb37/web/app/components/datasets/create/website/index.tsx#L140), when , the NoData component is shown with the "not configured" message`!source` @chenyuAC Excuse me, I encountered the same problem, but I followed the instructions and added "langgenius/waterscrawl_datasource" to api/services/datasourcesprovider_dervice. py. After restarting, it doesn't seem to have changed. Is there any other operation I missed?
yindo commented 2026-02-21 20:06:40 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Jan 18, 2026):

Hi, @chenyuAC. I'm Dosu, and I'm helping the Dify team manage their backlog and am marking this issue as stale.

Issue Summary:

  • You reported that Watercrawl shows as "not configured" in Dify v1.9.2 self-hosted via Docker despite proper setup.
  • The root cause was identified as the backend function get_hard_code_datasource_credentials() omitting the Watercrawl plugin ID from its list.
  • Adding the correct plugin ID and restarting the backend resolved the issue for v1.9.2.
  • Another user noted that in v1.11.0 the plugin ID changed and reported a new crawling error, seeking further assistance.
  • The issue remains unresolved regarding the new error in later versions.

Next Steps:

  • Please confirm if this issue is still relevant with the latest version of Dify, and if so, feel free to keep the discussion open by commenting here.
  • Otherwise, I will automatically close this issue in 15 days.

Thank you for your understanding and contribution!

@dosubot[bot] commented on GitHub (Jan 18, 2026): Hi, @chenyuAC. I'm [Dosu](https://dosu.dev), and I'm helping the Dify team manage their backlog and am marking this issue as stale. **Issue Summary:** - You reported that Watercrawl shows as "not configured" in Dify v1.9.2 self-hosted via Docker despite proper setup. - The root cause was identified as the backend function `get_hard_code_datasource_credentials()` omitting the Watercrawl plugin ID from its list. - Adding the correct plugin ID and restarting the backend resolved the issue for v1.9.2. - Another user noted that in v1.11.0 the plugin ID changed and reported a new crawling error, seeking further assistance. - The issue remains unresolved regarding the new error in later versions. **Next Steps:** - Please confirm if this issue is still relevant with the latest version of Dify, and if so, feel free to keep the discussion open by commenting here. - Otherwise, I will automatically close this issue in 15 days. Thank you for your understanding and contribution!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#20278
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?