langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 20:15:31 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

[PR #5725] feat: Spider web scraper & crawler tool #25032

New Issue
Closed
opened 2026-02-21 20:24:09 -05:00 by yindo · 0 comments
yindo commented 2026-02-21 20:24:09 -05:00
Owner
Copy Link
Copy Source

Original Pull Request: https://github.com/langgenius/dify/pull/5725

State: closed
Merged: Yes

Description

Add Spider, the fastest open source scraper & crawler that returns LLM-ready data.

Twitter: @WilliamEspegren

Type of Change

Please delete options that are not relevant.

  • New feature (non-breaking change which adds functionality)

How Has This Been Tested?

I have created a new workflow in the UI then tested it manually with all the different parameters.

image

Suggested Checklist:

  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • My changes generate no new warnings
  • I ran dev/reformat(backend) and cd web && npx lint-staged(frontend) to appease the lint gods

This PR was inspired by #3819

**Original Pull Request:** https://github.com/langgenius/dify/pull/5725 **State:** closed **Merged:** Yes --- # Description Add [Spider](https://spider.cloud/), the [fastest](https://github.com/spider-rs/spider/blob/main/benches/BENCHMARKS.md#benchmark-results) open source scraper & crawler that returns LLM-ready data. Twitter: [@WilliamEspegren](https://x.com/WilliamEspegren) ## Type of Change Please delete options that are not relevant. - [x] New feature (non-breaking change which adds functionality) # How Has This Been Tested? I have created a new workflow in the UI then tested it manually with all the different parameters. ![image](https://github.com/langgenius/dify/assets/131612909/6dc9c012-59bd-4dcb-a21b-f94f143ab915) # Suggested Checklist: - [x] I have performed a self-review of my own code - [ ] I have commented my code, particularly in hard-to-understand areas - [x] My changes generate no new warnings - [x] I ran `dev/reformat`(backend) and `cd web && npx lint-staged`(frontend) to appease the lint gods This PR was inspired by #3819
yindo added the pull-request label 2026-02-21 20:24:09 -05:00
yindo closed this issue 2026-02-21 20:24:09 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#25032
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?