langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-25 21:47:07 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Why not support by default for scrapingbee,serper.dev,serpstack api's for google searching instead of just giving one option as serp only #902

New Issue
Closed
opened 2026-02-21 17:28:57 -05:00 by yindo · 3 comments
yindo commented 2026-02-21 17:28:57 -05:00
Owner
Copy Link
Copy Source

Originally created by @Greatz08 on GitHub (Jan 13, 2024).

Originally assigned to: @takatost on GitHub.

Self Checks

Description of the new feature / enhancement

Atleast devs please add more api support by default for google search, flowise has also given atleast serper.dev option with which we can easily cost effectively search with google as by default serp api is costly as compared to other mentioned options which have some decent free and paid options for google search so please add support atleast for serper.dev api if not the other ones so that we can easily put api key and use google search with that instead of only depending on serp api. I know there are api extension option but tbh i havent understood completely how to use it. i did try my best to get api from mentioned websites and i tried to create end point but was getting some issuesso couldnt understand how to solve it because putting api key is easy but connection with endpoint i couldnt understand and tried to set it up but got some different error codes :-)) so please for now atleast add support for serper.dev and if possible for others too and then make any small video or EASY GUIDE WITH PROPER WORKING COMPLETE EXAMPLE to make other understand how to use it in future to connect with other api services.
Again Thanks for the hardwork and hope to see this request get fulfilled as soon as possible :-)) btw you can take reference from how flowise implemented serper.dev api if you get any issues i have tested there and it worked like a charm but this foss solution truly is awesome and i want it to grow more so will try to contribute for now with ideas.

Scenario when this would be used?

:-)

Supporting information

No response

Originally created by @Greatz08 on GitHub (Jan 13, 2024). Originally assigned to: @takatost on GitHub. ### Self Checks - [X] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [X] I confirm that I am using English to file this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). ### Description of the new feature / enhancement Atleast devs please add more api support by default for google search, flowise has also given atleast serper.dev option with which we can easily cost effectively search with google as by default serp api is costly as compared to other mentioned options which have some decent free and paid options for google search so please add support atleast for serper.dev api if not the other ones so that we can easily put api key and use google search with that instead of only depending on serp api. I know there are api extension option but tbh i havent understood completely how to use it. i did try my best to get api from mentioned websites and i tried to create end point but was getting some issuesso couldnt understand how to solve it because putting api key is easy but connection with endpoint i couldnt understand and tried to set it up but got some different error codes :-)) so please for now atleast add support for serper.dev and if possible for others too and then make any small video or EASY GUIDE WITH PROPER WORKING COMPLETE EXAMPLE to make other understand how to use it in future to connect with other api services. Again Thanks for the hardwork and hope to see this request get fulfilled as soon as possible :-)) btw you can take reference from how flowise implemented serper.dev api if you get any issues i have tested there and it worked like a charm but this foss solution truly is awesome and i want it to grow more so will try to contribute for now with ideas. ### Scenario when this would be used? :-) ### Supporting information _No response_
yindo added the 💪 enhancement label 2026-02-21 17:28:57 -05:00
yindo closed this issue 2026-02-21 17:28:57 -05:00
yindo commented 2026-02-21 17:28:57 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Jan 13, 2024):

🤖

Hello @HakaishinShwet!

I'm Dosu, your friendly neighborhood bot helper. My purpose is to assist you with any concerns or questions you might have about dify, and to help you become a contributor, if you'd like. While you're waiting for a human maintainer to check into this, I'm here to help you as best I can. Let's make the open-source community a better place, one issue at a time!

Let's dive into your issue soon!

Thank you for your suggestion and detailed explanation. Adding support for additional APIs like scrapingbee, serper.dev, and serpstack could indeed provide more cost-effective options for Google searching.

To implement this, new provider classes similar to the SerpAPIToolProvider class would need to be created for each of the new APIs. Here's a basic example of what a new provider for scrapingbee might look like:

from core.tool.provider.base import BaseToolProvider
from models.tool import ToolProviderName

class ScrapingBeeToolProvider(BaseToolProvider):
    def get_provider_name(self) -> ToolProviderName:
        return ToolProviderName.SCRAPINGBEE

    def get_credentials(self, obfuscated: bool = False) -> Optional[dict]:
        tool_provider = self.get_provider(must_enabled=True)
        if not tool_provider:
            return None

        credentials = tool_provider.credentials
        if not credentials:
            return None

        if credentials.get('api_key'):
            credentials['api_key'] = self.decrypt_token(credentials.get('api_key'), obfuscated)

        return credentials

    def credentials_to_func_kwargs(self) -> Optional[dict]:
        credentials = self.get_credentials()
        if not credentials:
            return None

        return {
            'scrapingbee_api_key': credentials.get('api_key')
        }

    def credentials_validate(self, credentials: dict):
        if 'api_key' not in credentials or not credentials.get('api_key'):
            raise ToolValidateFailedError("ScrapingBee api_key is required.")

        api_key = credentials.get('api_key')

        try:
            # Replace with actual ScrapingBee API wrapper
            ScrapingBeeAPIWrapper(scrapingbee_api_key=api_key).run(query='test')
        except Exception as e:
            raise ToolValidateFailedError("ScrapingBee api_key is invalid. {}".format(e))

    def encrypt_credentials(self, credentials: dict) -> Optional[dict]:
        credentials['api_key'] = self.encrypt_token(credentials.get('api_key'))
        return credentials

This is a basic example and the actual implementation would depend on the specifics of the new APIs. You would also need to create the corresponding ToolProviderName enum values and implement the actual API wrappers (like ScrapingBeeAPIWrapper in the example).

As for the current structure of the dify codebase, it is structured around a class named ExternalApi which inherits from the Api class of the flask_restful library. This class is used to handle API integrations and manage errors that may occur during the process. You can find more details about this in the api/libs/external_api.py file.

Unfortunately, I couldn't find specific details about how the current serp API is integrated into the dify codebase. However, the general approach would be similar to the example provided above for the ScrapingBee API.

I hope this helps! If you have any further questions or need more clarification, feel free to ask.

Sources

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

@dosubot[bot] commented on GitHub (Jan 13, 2024): _🤖_ <!-- Greeting --> Hello @HakaishinShwet! I'm [Dosu](https://dosu.dev), your friendly neighborhood bot helper. My purpose is to assist you with any concerns or questions you might have about dify, and to help you become a contributor, if you'd like. While you're waiting for a human maintainer to check into this, I'm here to help you as best I can. Let's make the open-source community a better place, one issue at a time! Let's dive into your issue soon! <!-- Answer --> Thank you for your suggestion and detailed explanation. Adding support for additional APIs like scrapingbee, serper.dev, and serpstack could indeed provide more cost-effective options for Google searching. To implement this, new provider classes similar to the `SerpAPIToolProvider` class would need to be created for each of the new APIs. Here's a basic example of what a new provider for scrapingbee might look like: ```py from core.tool.provider.base import BaseToolProvider from models.tool import ToolProviderName class ScrapingBeeToolProvider(BaseToolProvider): def get_provider_name(self) -> ToolProviderName: return ToolProviderName.SCRAPINGBEE def get_credentials(self, obfuscated: bool = False) -> Optional[dict]: tool_provider = self.get_provider(must_enabled=True) if not tool_provider: return None credentials = tool_provider.credentials if not credentials: return None if credentials.get('api_key'): credentials['api_key'] = self.decrypt_token(credentials.get('api_key'), obfuscated) return credentials def credentials_to_func_kwargs(self) -> Optional[dict]: credentials = self.get_credentials() if not credentials: return None return { 'scrapingbee_api_key': credentials.get('api_key') } def credentials_validate(self, credentials: dict): if 'api_key' not in credentials or not credentials.get('api_key'): raise ToolValidateFailedError("ScrapingBee api_key is required.") api_key = credentials.get('api_key') try: # Replace with actual ScrapingBee API wrapper ScrapingBeeAPIWrapper(scrapingbee_api_key=api_key).run(query='test') except Exception as e: raise ToolValidateFailedError("ScrapingBee api_key is invalid. {}".format(e)) def encrypt_credentials(self, credentials: dict) -> Optional[dict]: credentials['api_key'] = self.encrypt_token(credentials.get('api_key')) return credentials ``` This is a basic example and the actual implementation would depend on the specifics of the new APIs. You would also need to create the corresponding `ToolProviderName` enum values and implement the actual API wrappers (like `ScrapingBeeAPIWrapper` in the example). As for the current structure of the dify codebase, it is structured around a class named `ExternalApi` which inherits from the `Api` class of the `flask_restful` library. This class is used to handle API integrations and manage errors that may occur during the process. You can find more details about this in the [api/libs/external_api.py](https://github.com/langgenius/dify/blob/main/api/libs/external_api.py) file. Unfortunately, I couldn't find specific details about how the current serp API is integrated into the dify codebase. However, the general approach would be similar to the example provided above for the ScrapingBee API. I hope this helps! If you have any further questions or need more clarification, feel free to ask. ## Sources - [api/core/tool/provider/serpapi_provider.py](https://github.com/langgenius/dify/blob/main/api/core/tool/provider/serpapi_provider.py) - [api/libs/external_api.py](https://github.com/langgenius/dify/blob/main/api/libs/external_api.py) --- This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant. If you want to continue the conversation, start your reply with @dosu-bot.
yindo commented 2026-02-21 17:28:57 -05:00
Author
Owner
Copy Link
Copy Source

@crazywoola commented on GitHub (Jan 14, 2024):

We will have the functions in next few releases. Not only the serp api, but also some other cool stuffs.

@crazywoola commented on GitHub (Jan 14, 2024): We will have the functions in next few releases. Not only the serp api, but also some other cool stuffs.
yindo commented 2026-02-21 17:28:57 -05:00
Author
Owner
Copy Link
Copy Source

@hamxa0077 commented on GitHub (Feb 25, 2025):

I’m looking for a way to scrape data using the Serper.dev API,s there an existing integration for this, or would anyone have suggestions on how to achieve this using the Serperdev API?

@hamxa0077 commented on GitHub (Feb 25, 2025): I’m looking for a way to scrape data using the Serper.dev API,s there an existing integration for this, or would anyone have suggestions on how to achieve this using the Serperdev API?
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 💪 enhancement
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#902
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?