langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-20 09:10:05 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

[PR #3988] feat: Firecrawl Web Extractor #24355

New Issue
Closed
opened 2026-02-21 20:22:51 -05:00 by yindo · 0 comments
yindo commented 2026-02-21 20:22:51 -05:00
Owner
Copy Link
Copy Source

Original Pull Request: https://github.com/langgenius/dify/pull/3988

State: closed
Merged: No

Description

Dify x Firecrawl collab for creating knowledge base from web pages. Firecrawl team went ahead and created the Firecrawl web extractor which takes care of scraping and crawling website - turning them into llm-ready markdown.

There are 3 modes to help yall implement the front-end: crawl, scrape and crawl_return_urls.

  • Crawl: crawls the entire website and outputs a list of documents
  • Scrape: scrapes a single page only and outputs a list of document with 1 document
  • Crawl Return Urls: Optional: This is a helpful util for your team to be able to display which urls the person wants to scrape when they insert a website. That way you can show them a list and have them pick which ones they want ingested as a data source. You can then just feed those urls to the /scrape endpoint.

Questions / Doubts:

  • We were a bit unsure about the auth side of the web extractor. We were going to follow the Notion example but because Notion authenticates via OAuth and we only do a Bearer api key, didn't make too much sense to copy that. So we ended up not touching that side of things for now. It would be good to get some clarification on what the best approach is there but yall can probably implement that way faster than we can so feel free to tackle it.

I ended up leaving some [REVIEW] tags in some places as we weren't too sure there.

Type of Change

  • New feature (non-breaking change which adds functionality)
  • This change requires a documentation update, included: Dify Document

How Has This Been Tested?

  • Unit tests for Firecrawl web extractor

TODO:

  • Firecrawl Web extractor (Firecrawl team)
  • Firecrawl Authentication / Bearer Auth data source (?) (Prop Dify team, but Firecrawl team can help if needed (just need further info))
  • Front-end for web loader (Dify team)

ccing: @guchenhe @rafaelsideguide

**Original Pull Request:** https://github.com/langgenius/dify/pull/3988 **State:** closed **Merged:** No --- # Description Dify x Firecrawl collab for creating knowledge base from web pages. Firecrawl team went ahead and created the Firecrawl web extractor which takes care of scraping and crawling website - turning them into llm-ready markdown. There are 3 modes to help yall implement the front-end: `crawl`, `scrape` and `crawl_return_urls`. - Crawl: crawls the entire website and outputs a list of documents - Scrape: scrapes a single page only and outputs a list of document with 1 document - Crawl Return Urls: Optional: This is a helpful util for your team to be able to display which urls the person wants to scrape when they insert a website. That way you can show them a list and have them pick which ones they want ingested as a data source. You can then just feed those urls to the /scrape endpoint. Questions / Doubts: - We were a bit unsure about the auth side of the web extractor. We were going to follow the Notion example but because Notion authenticates via OAuth and we only do a Bearer api key, didn't make too much sense to copy that. So we ended up not touching that side of things for now. It would be good to get some clarification on what the best approach is there but yall can probably implement that way faster than we can so feel free to tackle it. I ended up leaving some [REVIEW] tags in some places as we weren't too sure there. ## Type of Change - [X] New feature (non-breaking change which adds functionality) - [X] This change requires a documentation update, included: [Dify Document](https://github.com/langgenius/dify-docs) ## # How Has This Been Tested? - [X] Unit tests for Firecrawl web extractor TODO: - [X] Firecrawl Web extractor (Firecrawl team) - [ ] Firecrawl Authentication / Bearer Auth data source (?) (Prop Dify team, but Firecrawl team can help if needed (just need further info)) - [ ] Front-end for web loader (Dify team) ccing: @guchenhe @rafaelsideguide
yindo added the pull-request label 2026-02-21 20:22:51 -05:00
yindo closed this issue 2026-02-21 20:22:51 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#24355
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?