langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-08-27 11:11:21 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Improve error handling for WaterCrawl API limitations (403 errors) #13027

New Issue
Closed
opened 2026-02-21 19:10:13 -05:00 by yindo · 2 comments
yindo commented 2026-02-21 19:10:13 -05:00
Owner
Copy Link
Copy Source

Originally created by @amirasaran on GitHub (Apr 16, 2025).

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:)
  • Please do not modify this template :) and fill in all the required fields.

Dify version

Version 1.2.0

Cloud or Self Hosted

Cloud

Steps to reproduce

Use WaterCrawl on a free plan within Dify
Exceed one of the following:

  • Daily or monthly page limits
  • Crawl depth over 2
  • Run more than 1 crawl concurrently

Observe the error message in Dify

✔️ Expected Behavior

The application should catch these specific 403 responses and provide a clear, user-friendly explanation of what went wrong, along with a suggestion to upgrade the plan or reduce crawl parameters.

Suggested Fix: Implement error-specific messaging for WaterCrawl integration when a 403 status is returned.

❌ Actual Behavior

When using the WaterCrawl.dev integration as a knowledge base source, users on the free plan may encounter 403 errors due to plan restrictions (e.g., exceeded crawl limits or depth restrictions). Currently, these errors are not clearly communicated to the user.


Traceback (most recent call last):
  File "/Users/crazywoola/Program/dify/api/.venv/lib/python3.12/site-packages/flask/app.py", line 917, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/crazywoola/Program/dify/api/.venv/lib/python3.12/site-packages/flask/app.py", line 902, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/crazywoola/Program/dify/api/.venv/lib/python3.12/site-packages/flask_restful/__init__.py", line 489, in wrapper
    resp = resource(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/crazywoola/Program/dify/api/.venv/lib/python3.12/site-packages/flask/views.py", line 110, in view
    return current_app.ensure_sync(self.dispatch_request)(**kwargs)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/crazywoola/Program/dify/api/.venv/lib/python3.12/site-packages/flask_restful/__init__.py", line 604, in dispatch_request
    resp = meth(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/crazywoola/Program/dify/api/controllers/console/wraps.py", line 198, in decorated
    return view(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/crazywoola/Program/dify/api/libs/login.py", line 94, in decorated_view
    return current_app.ensure_sync(func)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/crazywoola/Program/dify/api/controllers/console/wraps.py", line 30, in decorated
    return view(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/crazywoola/Program/dify/api/controllers/console/datasets/website.py", line 32, in post
    raise WebsiteCrawlError(str(e))
controllers.console.datasets.error.WebsiteCrawlError: 500 Internal Server Error: 403 Client Error: Forbidden for url: https://app.watercrawl.dev/api/v1/core/crawl-requests/
Originally created by @amirasaran on GitHub (Apr 16, 2025). ### Self Checks - [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [x] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:) - [x] Please do not modify this template :) and fill in all the required fields. ### Dify version Version 1.2.0 ### Cloud or Self Hosted Cloud ### Steps to reproduce Use WaterCrawl on a free plan within Dify Exceed one of the following: - Daily or monthly page limits - Crawl depth over 2 - Run more than 1 crawl concurrently Observe the error message in Dify ### ✔️ Expected Behavior The application should catch these specific 403 responses and provide a clear, user-friendly explanation of what went wrong, along with a suggestion to upgrade the plan or reduce crawl parameters. Suggested Fix: Implement error-specific messaging for WaterCrawl integration when a 403 status is returned. ### ❌ Actual Behavior When using the WaterCrawl.dev integration as a knowledge base source, users on the free plan may encounter 403 errors due to plan restrictions (e.g., exceeded crawl limits or depth restrictions). Currently, these errors are not clearly communicated to the user. ``` Traceback (most recent call last): File "/Users/crazywoola/Program/dify/api/.venv/lib/python3.12/site-packages/flask/app.py", line 917, in full_dispatch_request rv = self.dispatch_request() ^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/crazywoola/Program/dify/api/.venv/lib/python3.12/site-packages/flask/app.py", line 902, in dispatch_request return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) # type: ignore[no-any-return] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/crazywoola/Program/dify/api/.venv/lib/python3.12/site-packages/flask_restful/__init__.py", line 489, in wrapper resp = resource(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/crazywoola/Program/dify/api/.venv/lib/python3.12/site-packages/flask/views.py", line 110, in view return current_app.ensure_sync(self.dispatch_request)(**kwargs) # type: ignore[no-any-return] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/crazywoola/Program/dify/api/.venv/lib/python3.12/site-packages/flask_restful/__init__.py", line 604, in dispatch_request resp = meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/Users/crazywoola/Program/dify/api/controllers/console/wraps.py", line 198, in decorated return view(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/Users/crazywoola/Program/dify/api/libs/login.py", line 94, in decorated_view return current_app.ensure_sync(func)(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/crazywoola/Program/dify/api/controllers/console/wraps.py", line 30, in decorated return view(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/Users/crazywoola/Program/dify/api/controllers/console/datasets/website.py", line 32, in post raise WebsiteCrawlError(str(e)) controllers.console.datasets.error.WebsiteCrawlError: 500 Internal Server Error: 403 Client Error: Forbidden for url: https://app.watercrawl.dev/api/v1/core/crawl-requests/ ```
yindo added the 💪 enhancementstale labels 2026-02-21 19:10:13 -05:00
yindo closed this issue 2026-02-21 19:10:13 -05:00
yindo commented 2026-02-21 19:10:13 -05:00
Author
Owner
Copy Link
Copy Source

@amirasaran commented on GitHub (Apr 16, 2025):

This issue originally reported here: https://github.com/langgenius/dify/pull/16396#issuecomment-2808153727

@amirasaran commented on GitHub (Apr 16, 2025): This issue originally reported here: https://github.com/langgenius/dify/pull/16396#issuecomment-2808153727
yindo commented 2026-02-21 19:10:14 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (May 17, 2025):

Hi, @amirasaran. I'm Dosu, and I'm helping the Dify team manage their backlog. I'm marking this issue as stale.

Issue Summary:

  • The issue concerns unclear error handling in the WaterCrawl API integration.
  • Users on the free plan encounter 403 errors due to crawl limits or depth restrictions.
  • You suggested improving error messages to inform users about limitations and provide guidance.
  • A related discussion was referenced in a previous pull request comment.

Next Steps:

  • Please let me know if this issue is still relevant to the latest version of the Dify repository by commenting here.
  • If there is no further activity, this issue will be automatically closed in 15 days.

Thank you for your understanding and contribution!

@dosubot[bot] commented on GitHub (May 17, 2025): Hi, @amirasaran. I'm [Dosu](https://dosu.dev), and I'm helping the Dify team manage their backlog. I'm marking this issue as stale. **Issue Summary:** - The issue concerns unclear error handling in the WaterCrawl API integration. - Users on the free plan encounter 403 errors due to crawl limits or depth restrictions. - You suggested improving error messages to inform users about limitations and provide guidance. - A related discussion was referenced in a previous pull request comment. **Next Steps:** - Please let me know if this issue is still relevant to the latest version of the Dify repository by commenting here. - If there is no further activity, this issue will be automatically closed in 15 days. Thank you for your understanding and contribution!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label stale 💪 enhancement
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#13027
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?