mirror of
https://github.com/langgenius/dify-official-plugins.git
synced 2026-07-21 17:45:23 -04:00
The WaterCrawl is not available in the knowledge base in versions 1.9.0 and 2.0.0. #668
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @itq5 on GitHub (Sep 23, 2025).
Self Checks
Dify version
1.9.0
Cloud or Self Hosted
Self Hosted (Docker)
Steps to reproduce
The WaterCrawl is not available in the knowledge base in versions 1.9.0 and 2.0.0.
✔️ Expected Behavior
WaterCrawl should normally be classified into the data source category
❌ Actual Behavior
No response
@dosubot[bot] commented on GitHub (Sep 23, 2025):
WaterCrawl is not available as a selectable data source in the knowledge base for Dify versions 1.9.0 and 2.0.0 because it is not registered or implemented as a data source in the codebase for these releases. While WaterCrawl is referenced in localization files and has provider and authentication logic present, it lacks the dedicated modules and registration steps that other data sources (like FireCrawl) have, which is why it does not appear in the UI or integration list. There is no evidence it was intentionally removed or reclassified, and no workaround or manual integration method is documented for these versions. To enable WaterCrawl as a data source, code changes would be required to add the necessary modules and registration logic, similar to other supported sources.
For reference, WaterCrawl was previously integrated as a knowledge base source in earlier versions (e.g., v1.2.0) alongside other website crawling providers, but this status does not extend to 1.9.0 or 2.0.0 [source].
To reply, just mention @dosu.
How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other
@dosubot[bot] commented on GitHub (Sep 23, 2025):
WaterCrawl is supported in the backend and API for Dify 1.9.0 and 2.0.0, but its visibility in the knowledge base UI is controlled by a frontend feature flag called
ENABLE_WEBSITE_WATERCRAWL, which defaults to false in self-hosted deployments. This is why you don't see it as a data source unless you explicitly enable it. To make WaterCrawl available, set the environment variableNEXT_PUBLIC_ENABLE_WEBSITE_WATERCRAWL=truein your deployment and restart the frontend service. No backend or plugin changes are needed—this is just a configuration issue. Other website crawling tools like FireCrawl and JinaReader are enabled by default because their flags are set to true see source.If WaterCrawl still doesn't appear after enabling the flag, try clearing your browser cache and restarting your containers to ensure the new config is picked up.
To reply, just mention @dosu.
How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other
@itq5 commented on GitHub (Sep 24, 2025):
Adding NEXT_PUBLIC_ENABLE_WEBSITE_WATERCRAWL=true in .env also did not restore
@dosubot[bot] commented on GitHub (Sep 24, 2025):
The WaterCrawl feature flag (NEXT_PUBLIC_ENABLE_WEBSITE_WATERCRAWL) is only read by the frontend at build time, not at runtime. In Docker/self-hosted setups, after adding or changing this variable in your .env file, you must rebuild the frontend image and redeploy it for the change to take effect—just restarting the container is not enough. Make sure the variable is present in the environment at build time (e.g., in .env.local or the correct .env file used during the Docker build) and then rebuild and restart your frontend container. This is a common pattern for Next.js apps and is confirmed by the Dify frontend code and Dockerfile design see details.
If you already did this and it still doesn't appear, double-check that the .env file is in the correct location and is being picked up during the build process.
To reply, just mention @dosu.
How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other
@itq5 commented on GitHub (Sep 25, 2025):
I followed the steps as follows, and there is an issue with the official image
The images versions are all 1.9.0
Actually, checking in web/.env, NEXT_PUBLIC_ENABLE_WEBSITE_WATERCRAWL=true
@dosubot[bot] commented on GitHub (Oct 12, 2025):
Hi, @itq5. I'm Dosu, and I'm helping the dify-official-plugins team manage their backlog and am marking this issue as stale.
Issue Summary:
Next Steps:
Thank you for your understanding and contribution!