langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 12:05:30 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Firecrawl plugin fails credential validation if GET / or GET /v1 does not return JSON (self-hosted Firecrawl) #16062

New Issue
Closed
opened 2026-02-21 19:24:30 -05:00 by yindo · 4 comments
yindo commented 2026-02-21 19:24:30 -05:00
Owner
Copy Link
Copy Source

Originally created by @takpanda on GitHub (Aug 11, 2025).

Self Checks

  • I have read the Contributing Guide and Language Policy.
  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report, otherwise it will be closed.
  • 【中文用户 & Non English User】请使用英语提交，否则会被关闭 ：）
  • Please do not modify this template :) and fill in all the required fields.

Dify version

v1.7.2

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

  1. Self-host Firecrawl (http://xxxxx:3002)
  2. In Dify (v1.7.2), set:
  3. Save configuration.

✔️ Expected Behavior

  • Credential validation should not assume GET / or GET /v1 returns JSON.
  • Or, the plugin should validate credentials by calling a JSON-returning endpoint (e.g. /v1/crawl with a HEAD or minimal payload).

❌ Actual Behavior

  • On save, Dify tries to validate credentials.
  • Firecrawl responds to GET / with 200 HTML, or GET /v1 with 404 HTML.
  • Dify attempts to parse it as JSON → fails with:
ERROR [Dummy-26] [base.py:211] - Error in stream reponse for plugin {'code': -500, 'message': '{"message":"{\\"args\\":{},\\"error_type\\":\\"ToolProviderCredentialValidationError\\",\\"message\\":\\"Expecting value: line 1 column 1 (char 0)\\"}","error_type":"PluginInvokeError","args":null}', 'data': None}
  • Knowledge Base addition should work if crawling works via API.
ERROR [Dummy-27] [app.py:875] - Exception on /console/api/website/crawl [POST]
Traceback (most recent call last):
  File "/app/api/controllers/console/datasets/website.py", line 36, in post
    result = WebsiteService.crawl_url(api_request)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/services/website_service.py", line 151, in crawl_url
    return cls._crawl_with_firecrawl(request=request, api_key=api_key, config=config)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/services/website_service.py", line 180, in _crawl_with_firecrawl
    job_id = firecrawl_app.crawl_url(request.url, params)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/rag/extractor/firecrawl/firecrawl_app.py", line 52, in crawl_url
    self._handle_error(response, "start crawl job")
  File "/app/api/core/rag/extractor/firecrawl/firecrawl_app.py", line 128, in _handle_error
    raise Exception(f"Failed to {action}. Status code: {response.status_code}. Error: {error_message}")  # type: ignore[return]
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Exception: Failed to start crawl job. Status code: 400. Error: Bad Request

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/api/.venv/lib/python3.12/site-packages/flask/app.py", line 917, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/.venv/lib/python3.12/site-packages/flask/app.py", line 902, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/.venv/lib/python3.12/site-packages/flask_restful/__init__.py", line 489, in wrapper
    resp = resource(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/.venv/lib/python3.12/site-packages/flask/views.py", line 110, in view
    return current_app.ensure_sync(self.dispatch_request)(**kwargs)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/.venv/lib/python3.12/site-packages/flask_restful/__init__.py", line 604, in dispatch_request
    resp = meth(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/controllers/console/wraps.py", line 210, in decorated
    return view(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/libs/login.py", line 61, in decorated_view
    return current_app.ensure_sync(func)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/controllers/console/wraps.py", line 31, in decorated
    return view(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/controllers/console/datasets/website.py", line 38, in post
    raise WebsiteCrawlError(str(e))
controllers.console.datasets.error.WebsiteCrawlError: 500 Internal Server Error: Failed to start crawl job. Status code: 400. Error: Bad Request

Environment:

  • Dify version: 1.7.2 (self-hosted)
  • Firecrawl version: self-hosted, API v1
  • Network: no proxy/firewall issues, POST /v1/crawl works via curl
Originally created by @takpanda on GitHub (Aug 11, 2025). ### Self Checks - [x] I have read the [Contributing Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) and [Language Policy](https://github.com/langgenius/dify/issues/1542). - [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report, otherwise it will be closed. - [x] 【中文用户 & Non English User】请使用英语提交，否则会被关闭 ：） - [x] Please do not modify this template :) and fill in all the required fields. ### Dify version v1.7.2 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce 1. Self-host Firecrawl (http://xxxxx:3002) 2. In Dify (v1.7.2), set: * Base URL: http://xxxxx:3002 * API Key: fc-... 3. Save configuration. ### ✔️ Expected Behavior * Credential validation should not assume GET / or GET /v1 returns JSON. * Or, the plugin should validate credentials by calling a JSON-returning endpoint (e.g. /v1/crawl with a HEAD or minimal payload). ### ❌ Actual Behavior * On save, Dify tries to validate credentials. * Firecrawl responds to GET / with 200 HTML, or GET /v1 with 404 HTML. * Dify attempts to parse it as JSON → fails with: ``` ERROR [Dummy-26] [base.py:211] - Error in stream reponse for plugin {'code': -500, 'message': '{"message":"{\\"args\\":{},\\"error_type\\":\\"ToolProviderCredentialValidationError\\",\\"message\\":\\"Expecting value: line 1 column 1 (char 0)\\"}","error_type":"PluginInvokeError","args":null}', 'data': None} ``` * Knowledge Base addition should work if crawling works via API. ``` ERROR [Dummy-27] [app.py:875] - Exception on /console/api/website/crawl [POST] Traceback (most recent call last): File "/app/api/controllers/console/datasets/website.py", line 36, in post result = WebsiteService.crawl_url(api_request) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/api/services/website_service.py", line 151, in crawl_url return cls._crawl_with_firecrawl(request=request, api_key=api_key, config=config) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/api/services/website_service.py", line 180, in _crawl_with_firecrawl job_id = firecrawl_app.crawl_url(request.url, params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/api/core/rag/extractor/firecrawl/firecrawl_app.py", line 52, in crawl_url self._handle_error(response, "start crawl job") File "/app/api/core/rag/extractor/firecrawl/firecrawl_app.py", line 128, in _handle_error raise Exception(f"Failed to {action}. Status code: {response.status_code}. Error: {error_message}") # type: ignore[return] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Exception: Failed to start crawl job. Status code: 400. Error: Bad Request During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/app/api/.venv/lib/python3.12/site-packages/flask/app.py", line 917, in full_dispatch_request rv = self.dispatch_request() ^^^^^^^^^^^^^^^^^^^^^^^ File "/app/api/.venv/lib/python3.12/site-packages/flask/app.py", line 902, in dispatch_request return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) # type: ignore[no-any-return] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/api/.venv/lib/python3.12/site-packages/flask_restful/__init__.py", line 489, in wrapper resp = resource(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/api/.venv/lib/python3.12/site-packages/flask/views.py", line 110, in view return current_app.ensure_sync(self.dispatch_request)(**kwargs) # type: ignore[no-any-return] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/api/.venv/lib/python3.12/site-packages/flask_restful/__init__.py", line 604, in dispatch_request resp = meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/app/api/controllers/console/wraps.py", line 210, in decorated return view(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/app/api/libs/login.py", line 61, in decorated_view return current_app.ensure_sync(func)(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/api/controllers/console/wraps.py", line 31, in decorated return view(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/app/api/controllers/console/datasets/website.py", line 38, in post raise WebsiteCrawlError(str(e)) controllers.console.datasets.error.WebsiteCrawlError: 500 Internal Server Error: Failed to start crawl job. Status code: 400. Error: Bad Request ``` Environment: * Dify version: 1.7.2 (self-hosted) * Firecrawl version: self-hosted, API v1 * Network: no proxy/firewall issues, POST /v1/crawl works via curl
yindo added the good first issue label 2026-02-21 19:24:30 -05:00
yindo closed this issue 2026-02-21 19:24:30 -05:00
yindo commented 2026-02-21 19:24:30 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Aug 11, 2025):

Hi @takpanda! I'm Dosu and I’m helping the dify team.

The credential validation issue happens because Dify expects a JSON response from the Firecrawl API, but if the base URL is set with a trailing slash (like http://host:port/), Dify may call GET / and receive HTML, causing a JSON parsing error. The recommended workaround is to remove the trailing slash from your Firecrawl base URL in Dify (use http://host:port instead of http://host:port/). This change has resolved the issue for other users and allows credential validation to succeed [source].

Dify’s actual credential validation logic uses a POST request to /v1/crawl with a test payload and expects a JSON response with a 200 status code. If you still get a 400 Bad Request, check that your Firecrawl instance accepts the payload and that your API key and permissions are correct [source].

If you’re running into persistent issues, make sure you’re on the latest versions of both Dify and Firecrawl, as recent updates have improved compatibility and error handling. If POST /v1/crawl works via curl but fails in Dify, double-check your configuration and Firecrawl logs for clues.

To reply, just mention @dosu.

How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other  Chat with Dosu Join Discord Share on X

@dosubot[bot] commented on GitHub (Aug 11, 2025): <!-- Greeting --> Hi @takpanda! I'm [Dosu](https://go.dosu.dev/dosubot) and I’m helping the dify team. <!-- Answer --> The credential validation issue happens because Dify expects a JSON response from the Firecrawl API, but if the base URL is set with a trailing slash (like `http://host:port/`), Dify may call GET / and receive HTML, causing a JSON parsing error. The recommended workaround is to remove the trailing slash from your Firecrawl base URL in Dify (use `http://host:port` instead of `http://host:port/`). This change has resolved the issue for other users and allows credential validation to succeed [[source](https://github.com/langgenius/dify/issues/12961)]. Dify’s actual credential validation logic uses a POST request to `/v1/crawl` with a test payload and expects a JSON response with a 200 status code. If you still get a 400 Bad Request, check that your Firecrawl instance accepts the payload and that your API key and permissions are correct [[source](https://github.com/langgenius/dify/pull/22705)]. If you’re running into persistent issues, make sure you’re on the latest versions of both Dify and Firecrawl, as recent updates have improved compatibility and error handling. If POST `/v1/crawl` works via curl but fails in Dify, double-check your configuration and Firecrawl logs for clues. <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- <sup>How did I do? [Good](https://app.dosu.dev/response-feedback/d3643bf0-fafb-45fb-ab9d-3016ae4e6280?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/d3643bf0-fafb-45fb-ab9d-3016ae4e6280?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/d3643bf0-fafb-45fb-ab9d-3016ae4e6280?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/d3643bf0-fafb-45fb-ab9d-3016ae4e6280?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/d3643bf0-fafb-45fb-ab9d-3016ae4e6280?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/d3643bf0-fafb-45fb-ab9d-3016ae4e6280?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/d3643bf0-fafb-45fb-ab9d-3016ae4e6280?feedback_type=other)</sup>&nbsp;&nbsp;[![Chat with Dosu](https://dosu.dev/dosu-chat-badge.svg)](https://app.dosu.dev/6088a4f4-073a-43f3-b3d4-4f06fb0c05a0/ask?utm_source=github)&nbsp;[![Join Discord](https://img.shields.io/badge/join-5865F2?logo=discord&logoColor=white&label=)](https://go.dosu.dev/discord-bot)&nbsp;[![Share on X](https://img.shields.io/badge/X-share-black)](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/langgenius/dify/issues/23762)
yindo commented 2026-02-21 19:24:31 -05:00
Author
Owner
Copy Link
Copy Source

@takpanda commented on GitHub (Sep 3, 2025):

The authentication issue has been resolved.
However, the crawling problem still persists.
For your reference, crawling works correctly in my environment with v0.14.2.

Thank you for your continued support.

@takpanda commented on GitHub (Sep 3, 2025): The authentication issue has been resolved. However, the crawling problem still persists. For your reference, crawling works correctly in my environment with v0.14.2. Thank you for your continued support.
yindo commented 2026-02-21 19:24:31 -05:00
Author
Owner
Copy Link
Copy Source

@yoshio-sugiyama-rb commented on GitHub (Sep 4, 2025):

It seems integration key is invalid.

API Reference

@yoshio-sugiyama-rb commented on GitHub (Sep 4, 2025): It seems [`integration`](https://github.com/langgenius/dify/blob/8d5f788f2b4a8b92ee108b10974eb1833973c72a/api/core/rag/extractor/firecrawl/firecrawl_app.py#L43) key is invalid. [API Reference](https://docs.firecrawl.dev/api-reference/v1-endpoint/crawl-post)
yindo commented 2026-02-21 19:24:31 -05:00
Author
Owner
Copy Link
Copy Source

@kingemperorcz commented on GitHub (Nov 4, 2025):

The issue seems still on. How to resolve it.

@kingemperorcz commented on GitHub (Nov 4, 2025): The issue seems still on. How to resolve it.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label good first issue
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#16062
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?