langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 20:15:31 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

WebscraperTool can not bypass Cloudflare #4579

New Issue
Closed
opened 2026-02-21 18:07:06 -05:00 by yindo · 2 comments
yindo commented 2026-02-21 18:07:06 -05:00
Owner
Copy Link
Copy Source

Originally created by @Weaxs on GitHub (Jul 12, 2024).

Self Checks

  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
  • Please do not modify this template :) and fill in all the required fields.

1. Is this request related to a challenge you're experiencing? Tell me about your story.

WebscraperTool can not bypass the Cloudflare basic WAF protection.

Whether use webscraper tool in workflow or in agent, this problem can be reproduced.

Cloudflare JavaScript detections

image

image

image

2. Additional context or comments

I found some module for fix this, It may be useful for reference:

3. Can you help us with this feature?

  • I am interested in contributing to this feature.
Originally created by @Weaxs on GitHub (Jul 12, 2024). ### Self Checks - [X] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [X] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [X] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:） - [X] Please do not modify this template :) and fill in all the required fields. ### 1. Is this request related to a challenge you're experiencing? Tell me about your story. WebscraperTool can not bypass the Cloudflare basic WAF protection. Whether use webscraper tool in workflow or in agent, this problem can be reproduced. [Cloudflare JavaScript detections](https://developers.cloudflare.com/bots/reference/javascript-detections/) ![image](https://github.com/user-attachments/assets/22566cf2-ca5c-44bc-99a7-8c26a6b31e1c) ![image](https://github.com/user-attachments/assets/e9206129-59eb-4657-892d-1bf9baf67c71) ![image](https://github.com/user-attachments/assets/cb4f3c00-d488-4415-9fb2-69963db9e3f0) ### 2. Additional context or comments I found some module for fix this, It may be useful for reference: - https://github.com/Anorov/cloudflare-scrape - https://github.com/VeNoMouS/cloudscraper ### 3. Can you help us with this feature? - [X] I am interested in contributing to this feature.
yindo added the 💪 enhancement label 2026-02-21 18:07:06 -05:00
yindo closed this issue 2026-02-21 18:07:06 -05:00
yindo commented 2026-02-21 18:07:06 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Jul 12, 2024):

To modify the WebscraperTool to bypass Cloudflare's basic WAF protection, you can integrate the cloudscraper module into the _invoke method. Here’s how you can do it:

  1. Install the cloudscraper module:

    pip install cloudscraper

  2. Modify the WebscraperTool class to use cloudscraper for fetching the webpage:

from typing import Any, Union
import cloudscraper

from core.tools.entities.tool_entities import ToolInvokeMessage
from core.tools.errors import ToolInvokeError
from core.tools.tool.builtin_tool import BuiltinTool

class WebscraperTool(BuiltinTool):
    def _invoke(self,
                user_id: str,
                tool_parameters: dict[str, Any],
                ) -> Union[ToolInvokeMessage, list[ToolInvokeMessage]]:
        """
            invoke tools
        """
        try:
            url = tool_parameters.get('url', '')
            user_agent = tool_parameters.get('user_agent', '')
            if not url:
                return self.create_text_message('Please input url')

            # Use cloudscraper to get webpage
            scraper = cloudscraper.create_scraper()
            headers = {'User-Agent': user_agent} if user_agent else {}
            result = scraper.get(url, headers=headers).text

            if tool_parameters.get('generate_summary'):
                # summarize and return
                return self.create_text_message(self.summary(user_id=user_id, content=result))
            else:
                # return full webpage
                return self.create_text_message(result)
        except Exception as e:
            raise ToolInvokeError(str(e))

This modification uses cloudscraper to handle the HTTP request, which can bypass Cloudflare's basic WAF protection. The cloudscraper.create_scraper() function creates a session that can handle Cloudflare's anti-bot page. The headers dictionary is used to set the User-Agent if provided [1].

To continue talking to Dosu, mention @dosu.

@dosubot[bot] commented on GitHub (Jul 12, 2024): <!-- Answer --> To modify the `WebscraperTool` to bypass Cloudflare's basic WAF protection, you can integrate the `cloudscraper` module into the `_invoke` method. Here’s how you can do it: 1. **Install the `cloudscraper` module**: ```sh pip install cloudscraper ``` 2. **Modify the `WebscraperTool` class** to use `cloudscraper` for fetching the webpage: ```python from typing import Any, Union import cloudscraper from core.tools.entities.tool_entities import ToolInvokeMessage from core.tools.errors import ToolInvokeError from core.tools.tool.builtin_tool import BuiltinTool class WebscraperTool(BuiltinTool): def _invoke(self, user_id: str, tool_parameters: dict[str, Any], ) -> Union[ToolInvokeMessage, list[ToolInvokeMessage]]: """ invoke tools """ try: url = tool_parameters.get('url', '') user_agent = tool_parameters.get('user_agent', '') if not url: return self.create_text_message('Please input url') # Use cloudscraper to get webpage scraper = cloudscraper.create_scraper() headers = {'User-Agent': user_agent} if user_agent else {} result = scraper.get(url, headers=headers).text if tool_parameters.get('generate_summary'): # summarize and return return self.create_text_message(self.summary(user_id=user_id, content=result)) else: # return full webpage return self.create_text_message(result) except Exception as e: raise ToolInvokeError(str(e)) ``` This modification uses `cloudscraper` to handle the HTTP request, which can bypass Cloudflare's basic WAF protection. The `cloudscraper.create_scraper()` function creates a session that can handle Cloudflare's anti-bot page. The `headers` dictionary is used to set the `User-Agent` if provided <sup>[[1]](https://github.com/langgenius/dify/blob/main/api/core/tools/provider/builtin/webscraper/tools/webscraper.py)</sup>. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.*
yindo commented 2026-02-21 18:07:06 -05:00
Author
Owner
Copy Link
Copy Source

@Weaxs commented on GitHub (Jul 16, 2024):

I config the Cloudflare WAF rules for skipping webscraper requests, but it's not work.

image

image

@Weaxs commented on GitHub (Jul 16, 2024): I config the Cloudflare WAF rules for skipping webscraper requests, but it's not work. ![image](https://github.com/user-attachments/assets/3ef25f42-bd72-40c9-81cf-b9f1d5ca7b58) ![image](https://github.com/user-attachments/assets/7db31d15-03d6-4bfd-a6bf-2849c5f545df)
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 💪 enhancement
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#4579
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?