langgenius/dify-plugins
1
0
Fork 0
mirror of https://github.com/langgenius/dify-plugins.git synced 2026-07-22 01:55:36 -04:00
Code Issues 119 Packages Projects Releases Wiki Activity

[PR #646] [CLOSED] Add Bright Data plugin v1.0.0 #1085

New Issue
Closed
opened 2026-02-22 17:47:34 -05:00 by yindo · 0 comments
yindo commented 2026-02-22 17:47:34 -05:00
Owner
Copy Link
Copy Source

📋 Pull Request Information

Original PR: https://github.com/langgenius/dify-plugins/pull/646
Author: @Idanvilenski
Created: 6/8/2025
Status: ❌ Closed

Base: main ← Head: main

📝 Commits (10+)

📊 Changes

17 files changed (+1254 additions, -0 deletions)

View changed files

➕ idanvilenski/brightdata-dify-plugin/LICENSE (+44 -0)
➕ idanvilenski/brightdata-dify-plugin/PRIVACY.md (+42 -0)
➕ idanvilenski/brightdata-dify-plugin/README.md (+134 -0)
➕ idanvilenski/brightdata-dify-plugin/assets/demo1.PNG (+1 -0)
➕ idanvilenski/brightdata-dify-plugin/brightdata_plugin.difypkg (+0 -0)
➕ idanvilenski/brightdata-dify-plugin/brightdata_plugin/_assets/icon.svg (+1 -0)
➕ idanvilenski/brightdata-dify-plugin/brightdata_plugin/main.py (+6 -0)
➕ idanvilenski/brightdata-dify-plugin/brightdata_plugin/manifest.yaml (+41 -0)
➕ idanvilenski/brightdata-dify-plugin/brightdata_plugin/provider/brightdata.py (+20 -0)
➕ idanvilenski/brightdata-dify-plugin/brightdata_plugin/provider/brightdata.yaml (+36 -0)
➕ idanvilenski/brightdata-dify-plugin/brightdata_plugin/requirements.txt (+2 -0)
➕ idanvilenski/brightdata-dify-plugin/brightdata_plugin/tools/scrape_as_markdown.py (+57 -0)
➕ idanvilenski/brightdata-dify-plugin/brightdata_plugin/tools/scrape_as_markdown.yaml (+38 -0)
➕ idanvilenski/brightdata-dify-plugin/brightdata_plugin/tools/search_engine.py (+75 -0)
➕ idanvilenski/brightdata-dify-plugin/brightdata_plugin/tools/search_engine.yaml (+69 -0)
➕ idanvilenski/brightdata-dify-plugin/brightdata_plugin/tools/structured_data_feeds.py (+638 -0)
➕ idanvilenski/brightdata-dify-plugin/brightdata_plugin/tools/structured_data_feeds.yaml (+50 -0)

📄 Description

Plugin Submission Form

1. Metadata

2. Submission Type

  • New plugin submission
  • Version update for existing plugin

3. Description

This plugin provides comprehensive web scraping and data extraction capabilities powered by Bright Data's enterprise-grade infrastructure with intelligent auto-detection. It supports 50+ platforms including major e-commerce sites (Amazon, eBay, Walmart), social media platforms (LinkedIn, Instagram, YouTube, TikTok), business intelligence sources (Crunchbase, ZoomInfo), and search engines.

Key Features:

  • Universal Web Scraping: Convert any webpage to clean markdown format perfect for content analysis and documentation extraction
  • Structured Data Extraction: Pre-built extractors for 50+ popular platforms with intelligent auto-detection
  • Search Engine Integration: Get search results from Google, Bing, and Yandex with clean formatting
  • Enterprise-Grade Infrastructure: Powered by Bright Data's proxy network with advanced bot detection bypass
  • Cache-Optimized: Utilizes cached data when available for faster and more reliable results

Supported Platforms:

  • E-commerce: Amazon Products/Reviews/Search, eBay, Walmart, Best Buy, Etsy, Zara, Home Depot
  • Social Media: Instagram Profiles/Posts/Reels/Comments, Facebook Posts/Marketplace/Events, TikTok Profiles/Posts/Shop/Comments, YouTube Profiles/Videos/Comments, X (Twitter) Posts
  • Professional Networks: LinkedIn People/Company Profiles, Job Listings, Posts, People Search
  • Business Intelligence: Crunchbase Company Data, ZoomInfo Company Profiles
  • Maps & Reviews: Google Maps Reviews, Booking Hotel Listings, Zillow Properties
  • News & Content: Reuters News, Reddit Posts
  • App Stores: Google Play Store, Apple App Store
  • Financial: Yahoo Finance Business Data

Technical Capabilities:

  • Smart Content Processing: Automatically detects content type and applies appropriate extraction methods
  • Multi-format Output: JSON, Markdown, and Raw data formats
  • Error Handling: Robust error handling with retry mechanisms
  • Rate Limiting: Built-in rate limiting to respect target websites
  • Privacy Compliant: No personal data collection, transparent data handling

Use Cases:

  • E-commerce Monitoring: Track product prices, availability, and competitor analysis
  • Lead Generation: Extract business information and contact details from professional platforms
  • Content Research: Gather articles, news, and social media content for analysis
  • Market Research: Monitor competitor websites, social media presence, and market trends
  • SEO Analysis: Track search engine results and keyword rankings
  • Academic Research: Extract structured data from various online sources
  • Business Intelligence: Gather company information and industry insights

This plugin brings significant value to the Dify ecosystem by providing enterprise-grade web scraping capabilities that were previously unavailable, enabling users to build sophisticated data collection and analysis workflows without complex infrastructure setup.

4. Checklist

  • I have read and followed the Publish to Dify Marketplace guidelines
  • I have read and comply with the Plugin Developer Agreement
  • I confirm my plugin works properly on both Dify Community Edition and Cloud Version
  • I confirm my plugin has been thoroughly tested for completeness and functionality
  • My plugin brings new value to Dify

5. Documentation Checklist

Please confirm that your plugin README includes all necessary information:

  • Step-by-step setup instructions
  • Detailed usage instructions
  • All required APIs and credentials are clearly listed
  • Connection requirements and configuration details
  • Link to the repository for the plugin source code

6. Privacy Protection Information

Privacy Policy

  • I confirm that I have prepared and included a privacy policy in my plugin package based on the Plugin Privacy Protection Guidelines.

Based on Dify Plugin Privacy Protection Guidelines:

Data Collection

This plugin does not collect any user personal data. The plugin operates as a proxy service that:

  • URLs and Search Queries: Processes user-provided URLs and search queries solely for the purpose of web scraping and data extraction
  • Scraped Content: Temporarily processes publicly available web content to extract and format data as requested
  • API Credentials: Securely handles Bright Data API credentials provided by users for authentication purposes only

Data Handling Practices

  • No Personal Data Storage: The plugin does not store, log, or retain any user personal information
  • Temporary Processing: All data processing occurs in real-time during the scraping operation and is not persisted
  • Third-Party Service: Data is processed through Bright Data's infrastructure according to their privacy policy
  • Public Data Only: The plugin only accesses publicly available web content as specified by user input

Third-Party Data Processing

This plugin utilizes Bright Data's Web Scraping Services:

  • Service Provider: Bright Data Ltd.
  • Data Processed: URLs, search queries, and publicly available web content
  • Purpose: Web scraping and data extraction services
  • Privacy Policy: https://brightdata.com/privacy
  • Data Retention: Bright Data may temporarily cache scraped content according to their policies

User Control

  • Users maintain full control over what data is scraped by specifying URLs and search parameters
  • All scraped data is returned directly to the user through the Dify workflow
  • Users can review Bright Data's privacy policy for detailed information about their data processing practices

7. Technical Specifications

System Requirements

  • Runtime: Python 3.11+
  • Memory: 256MB
  • Storage: 10MB
  • Architecture: Supports amd64 and arm64

Dependencies

  • Bright Data API: Requires valid API token for web scraping services
  • External Services: Bright Data's proxy network and data extraction infrastructure
  • Network: HTTPS outbound connections required

Security Features

  • Secure API Handling: Encrypted transmission of API credentials
  • Input Validation: Comprehensive validation of URLs and parameters
  • Error Isolation: Errors are contained and do not affect other workflow components
  • Rate Limiting: Built-in protection against excessive API usage

8. Plugin Value Proposition

Unique Value

This plugin provides enterprise-grade web scraping capabilities that are currently unavailable in the Dify ecosystem:

  1. Comprehensive Platform Support: Pre-built extractors for 50+ major platforms vs. generic scraping tools
  2. Advanced Bot Detection Bypass: Utilizes Bright Data's proxy network to access protected content
  3. Structured Data Output: Returns clean, structured data instead of raw HTML
  4. Cache Optimization: Leverages cached data for faster, more reliable results
  5. Professional Infrastructure: Enterprise-grade reliability and performance

Differentiation from Existing Tools

  • vs. Basic Web Scrapers: Provides structured extraction for specific platforms with intelligent content detection
  • vs. Search Tools: Offers comprehensive data extraction beyond simple search results
  • vs. Generic APIs: Includes advanced proxy rotation and bot detection bypass capabilities
  • vs. Manual Data Collection: Automates complex data extraction workflows with enterprise reliability

9. Maintenance Commitment

Ongoing Support

  • Issue Resolution: Committed to addressing user-reported issues within 48 hours
  • Feature Requests: Regular evaluation and implementation of community feature requests
  • API Updates: Monitoring and updating for changes in Bright Data API and supported platforms
  • Documentation: Maintaining comprehensive documentation and usage examples

Update Schedule

  • Minor Updates: Bug fixes and small improvements as needed
  • Major Updates: New platform support and feature additions quarterly
  • Security Updates: Immediate response to security-related issues
  • API Migration: Proactive updates for API changes with advance notice

10. Contact Information

Additional Notes: This plugin has been thoroughly tested across multiple platforms and use cases. The integration with Bright Data provides enterprise-grade reliability and performance that significantly enhances Dify's data collection capabilities. The plugin follows all Dify development guidelines and security best practices.

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/langgenius/dify-plugins/pull/646 **Author:** [@Idanvilenski](https://github.com/Idanvilenski) **Created:** 6/8/2025 **Status:** ❌ Closed **Base:** `main` ← **Head:** `main` --- ### 📝 Commits (10+) - [`9bd3116`](https://github.com/langgenius/dify-plugins/commit/9bd311683d9fa032d9d014ff801043c8bbf34eae) Create LICENSE - [`5e7c987`](https://github.com/langgenius/dify-plugins/commit/5e7c9873d6b2ab2688ea80aa1b434b9d09c774d2) Add files via upload - [`3fe755b`](https://github.com/langgenius/dify-plugins/commit/3fe755be9256050f0c965a4bcd780b9eadd7e27b) Rename brightdata_plugin (1).difypkg to brightdata_plugin.difypkg - [`51fab1f`](https://github.com/langgenius/dify-plugins/commit/51fab1fdc73b8ce9f430095cb0ef444b17db406d) Create PRIVACY.md - [`d68ce9f`](https://github.com/langgenius/dify-plugins/commit/d68ce9f0101a501093f35fb3720a58bd96030e6e) Create GUIDE.md - [`57931ea`](https://github.com/langgenius/dify-plugins/commit/57931ea1e684f2d3e6bdb076b3cae5e40c4cff88) Create README.md - [`ddc0a29`](https://github.com/langgenius/dify-plugins/commit/ddc0a29a138b1db88453b885c5a2347c6e0304de) Create demo1.PNG - [`f0fce4e`](https://github.com/langgenius/dify-plugins/commit/f0fce4e4b603f97a7a533ad53d31e0564aaa941c) Create search_engine.yaml - [`c935b50`](https://github.com/langgenius/dify-plugins/commit/c935b50c770ecaf54d45e928146af8f33f06b1ce) Add files via upload - [`43363ca`](https://github.com/langgenius/dify-plugins/commit/43363ca0aaf806da10d2cc1a7e7a4548e8d97c9a) Create bright-data.yaml ### 📊 Changes **17 files changed** (+1254 additions, -0 deletions) <details> <summary>View changed files</summary> ➕ `idanvilenski/brightdata-dify-plugin/LICENSE` (+44 -0) ➕ `idanvilenski/brightdata-dify-plugin/PRIVACY.md` (+42 -0) ➕ `idanvilenski/brightdata-dify-plugin/README.md` (+134 -0) ➕ `idanvilenski/brightdata-dify-plugin/assets/demo1.PNG` (+1 -0) ➕ `idanvilenski/brightdata-dify-plugin/brightdata_plugin.difypkg` (+0 -0) ➕ `idanvilenski/brightdata-dify-plugin/brightdata_plugin/_assets/icon.svg` (+1 -0) ➕ `idanvilenski/brightdata-dify-plugin/brightdata_plugin/main.py` (+6 -0) ➕ `idanvilenski/brightdata-dify-plugin/brightdata_plugin/manifest.yaml` (+41 -0) ➕ `idanvilenski/brightdata-dify-plugin/brightdata_plugin/provider/brightdata.py` (+20 -0) ➕ `idanvilenski/brightdata-dify-plugin/brightdata_plugin/provider/brightdata.yaml` (+36 -0) ➕ `idanvilenski/brightdata-dify-plugin/brightdata_plugin/requirements.txt` (+2 -0) ➕ `idanvilenski/brightdata-dify-plugin/brightdata_plugin/tools/scrape_as_markdown.py` (+57 -0) ➕ `idanvilenski/brightdata-dify-plugin/brightdata_plugin/tools/scrape_as_markdown.yaml` (+38 -0) ➕ `idanvilenski/brightdata-dify-plugin/brightdata_plugin/tools/search_engine.py` (+75 -0) ➕ `idanvilenski/brightdata-dify-plugin/brightdata_plugin/tools/search_engine.yaml` (+69 -0) ➕ `idanvilenski/brightdata-dify-plugin/brightdata_plugin/tools/structured_data_feeds.py` (+638 -0) ➕ `idanvilenski/brightdata-dify-plugin/brightdata_plugin/tools/structured_data_feeds.yaml` (+50 -0) </details> ### 📄 Description # Plugin Submission Form ## 1. Metadata * **Plugin Author**: idanvilenski * **Plugin Name**: Bright Data Web Unlocking Plugin * **Repository URL**: https://github.com/idanvilenski/BrightData_Dify_Plugin ## 2. Submission Type - [x] New plugin submission - [ ] Version update for existing plugin ## 3. Description This plugin provides comprehensive web scraping and data extraction capabilities powered by Bright Data's enterprise-grade infrastructure with intelligent auto-detection. It supports 50+ platforms including major e-commerce sites (Amazon, eBay, Walmart), social media platforms (LinkedIn, Instagram, YouTube, TikTok), business intelligence sources (Crunchbase, ZoomInfo), and search engines. ### Key Features: * **Universal Web Scraping**: Convert any webpage to clean markdown format perfect for content analysis and documentation extraction * **Structured Data Extraction**: Pre-built extractors for 50+ popular platforms with intelligent auto-detection * **Search Engine Integration**: Get search results from Google, Bing, and Yandex with clean formatting * **Enterprise-Grade Infrastructure**: Powered by Bright Data's proxy network with advanced bot detection bypass * **Cache-Optimized**: Utilizes cached data when available for faster and more reliable results ### Supported Platforms: * **E-commerce**: Amazon Products/Reviews/Search, eBay, Walmart, Best Buy, Etsy, Zara, Home Depot * **Social Media**: Instagram Profiles/Posts/Reels/Comments, Facebook Posts/Marketplace/Events, TikTok Profiles/Posts/Shop/Comments, YouTube Profiles/Videos/Comments, X (Twitter) Posts * **Professional Networks**: LinkedIn People/Company Profiles, Job Listings, Posts, People Search * **Business Intelligence**: Crunchbase Company Data, ZoomInfo Company Profiles * **Maps & Reviews**: Google Maps Reviews, Booking Hotel Listings, Zillow Properties * **News & Content**: Reuters News, Reddit Posts * **App Stores**: Google Play Store, Apple App Store * **Financial**: Yahoo Finance Business Data ### Technical Capabilities: * **Smart Content Processing**: Automatically detects content type and applies appropriate extraction methods * **Multi-format Output**: JSON, Markdown, and Raw data formats * **Error Handling**: Robust error handling with retry mechanisms * **Rate Limiting**: Built-in rate limiting to respect target websites * **Privacy Compliant**: No personal data collection, transparent data handling ### Use Cases: * **E-commerce Monitoring**: Track product prices, availability, and competitor analysis * **Lead Generation**: Extract business information and contact details from professional platforms * **Content Research**: Gather articles, news, and social media content for analysis * **Market Research**: Monitor competitor websites, social media presence, and market trends * **SEO Analysis**: Track search engine results and keyword rankings * **Academic Research**: Extract structured data from various online sources * **Business Intelligence**: Gather company information and industry insights This plugin brings significant value to the Dify ecosystem by providing enterprise-grade web scraping capabilities that were previously unavailable, enabling users to build sophisticated data collection and analysis workflows without complex infrastructure setup. ## 4. Checklist - [x] I have read and followed the Publish to Dify Marketplace guidelines - [x] I have read and comply with the Plugin Developer Agreement - [x] I confirm my plugin works properly on both Dify Community Edition and Cloud Version - [x] I confirm my plugin has been thoroughly tested for completeness and functionality - [x] My plugin brings new value to Dify ## 5. Documentation Checklist Please confirm that your plugin README includes all necessary information: - [x] Step-by-step setup instructions - [x] Detailed usage instructions - [x] All required APIs and credentials are clearly listed - [x] Connection requirements and configuration details - [x] Link to the repository for the plugin source code ## 6. Privacy Protection Information ### Privacy Policy - [x] I confirm that I have prepared and included a privacy policy in my plugin package based on the Plugin Privacy Protection Guidelines. Based on Dify Plugin Privacy Protection Guidelines: ### Data Collection This plugin **does not collect any user personal data**. The plugin operates as a proxy service that: * **URLs and Search Queries**: Processes user-provided URLs and search queries solely for the purpose of web scraping and data extraction * **Scraped Content**: Temporarily processes publicly available web content to extract and format data as requested * **API Credentials**: Securely handles Bright Data API credentials provided by users for authentication purposes only ### Data Handling Practices * **No Personal Data Storage**: The plugin does not store, log, or retain any user personal information * **Temporary Processing**: All data processing occurs in real-time during the scraping operation and is not persisted * **Third-Party Service**: Data is processed through Bright Data's infrastructure according to their privacy policy * **Public Data Only**: The plugin only accesses publicly available web content as specified by user input ### Third-Party Data Processing This plugin utilizes **Bright Data's Web Scraping Services**: * **Service Provider**: Bright Data Ltd. * **Data Processed**: URLs, search queries, and publicly available web content * **Purpose**: Web scraping and data extraction services * **Privacy Policy**: https://brightdata.com/privacy * **Data Retention**: Bright Data may temporarily cache scraped content according to their policies ### User Control * Users maintain full control over what data is scraped by specifying URLs and search parameters * All scraped data is returned directly to the user through the Dify workflow * Users can review Bright Data's privacy policy for detailed information about their data processing practices ## 7. Technical Specifications ### System Requirements * **Runtime**: Python 3.11+ * **Memory**: 256MB * **Storage**: 10MB * **Architecture**: Supports amd64 and arm64 ### Dependencies * **Bright Data API**: Requires valid API token for web scraping services * **External Services**: Bright Data's proxy network and data extraction infrastructure * **Network**: HTTPS outbound connections required ### Security Features * **Secure API Handling**: Encrypted transmission of API credentials * **Input Validation**: Comprehensive validation of URLs and parameters * **Error Isolation**: Errors are contained and do not affect other workflow components * **Rate Limiting**: Built-in protection against excessive API usage ## 8. Plugin Value Proposition ### Unique Value This plugin provides **enterprise-grade web scraping capabilities** that are currently unavailable in the Dify ecosystem: 1. **Comprehensive Platform Support**: Pre-built extractors for 50+ major platforms vs. generic scraping tools 2. **Advanced Bot Detection Bypass**: Utilizes Bright Data's proxy network to access protected content 3. **Structured Data Output**: Returns clean, structured data instead of raw HTML 4. **Cache Optimization**: Leverages cached data for faster, more reliable results 5. **Professional Infrastructure**: Enterprise-grade reliability and performance ### Differentiation from Existing Tools * **vs. Basic Web Scrapers**: Provides structured extraction for specific platforms with intelligent content detection * **vs. Search Tools**: Offers comprehensive data extraction beyond simple search results * **vs. Generic APIs**: Includes advanced proxy rotation and bot detection bypass capabilities * **vs. Manual Data Collection**: Automates complex data extraction workflows with enterprise reliability ## 9. Maintenance Commitment ### Ongoing Support * **Issue Resolution**: Committed to addressing user-reported issues within 48 hours * **Feature Requests**: Regular evaluation and implementation of community feature requests * **API Updates**: Monitoring and updating for changes in Bright Data API and supported platforms * **Documentation**: Maintaining comprehensive documentation and usage examples ### Update Schedule * **Minor Updates**: Bug fixes and small improvements as needed * **Major Updates**: New platform support and feature additions quarterly * **Security Updates**: Immediate response to security-related issues * **API Migration**: Proactive updates for API changes with advance notice ## 10. Contact Information * **GitHub**: @idanvilenski * **Repository**: https://github.com/idanvilenski/BrightData_Dify_Plugin * **Issues**: Please report issues through GitHub Issues * **Support**: Available for community support and plugin assistance --- **Additional Notes**: This plugin has been thoroughly tested across multiple platforms and use cases. The integration with Bright Data provides enterprise-grade reliability and performance that significantly enhances Dify's data collection capabilities. The plugin follows all Dify development guidelines and security best practices. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
yindo added the pull-request label 2026-02-22 17:47:34 -05:00
yindo closed this issue 2026-02-22 17:47:35 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Plugin Request
Plugin Request
Plugin Request
bug
duplicate
enhancement
pull-request
Mirrored from GitHub Pull Request
 question
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify-plugins#1085
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?