langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-08-27 21:10:04 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Firecrawl API Key not working, getting 500 error in Dify #4343

New Issue
Closed
opened 2026-02-21 18:05:56 -05:00 by yindo · 6 comments
yindo commented 2026-02-21 18:05:56 -05:00
Owner
Copy Link
Copy Source

Originally created by @Edison-A-N on GitHub (Jun 30, 2024).

Originally assigned to: @JohnJyong on GitHub.

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
  • Please do not modify this template :) and fill in all the required fields.

Dify version

eab0ac3a1

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

Enter the correct API Key in Firecrawl
Add the Firecrawl API Key in Difi Agent Knowledge
Observe if Difi throws a 500 error

✔️ Expected Behavior

No response

❌ Actual Behavior

worker logs:

api_1                    | 2024-06-30 09:07:14,849.849 ERROR [Dummy-156] [app.py:838] - Exception on /console/api/api-key-auth/data-source/binding [POST]
api_1                    | Traceback (most recent call last):
api_1                    |   File "/app/api/controllers/console/auth/data_source_bearer_auth.py", line 51, in post
api_1                    |     ApiKeyAuthService.create_provider_auth(current_user.current_tenant_id, args)
api_1                    |   File "/app/api/services/auth/api_key_auth_service.py", line 21, in create_provider_auth
api_1                    |     auth_result = ApiKeyAuthFactory(args['provider'], args['credentials']).validate_credentials()
api_1                    |   File "/app/api/services/auth/api_key_auth_factory.py", line 14, in validate_credentials
api_1                    |     return self.auth.validate_credentials()
api_1                    |   File "/app/api/services/auth/firecrawl.py", line 37, in validate_credentials
api_1                    |     self._handle_error(response)
api_1                    |   File "/app/api/services/auth/firecrawl.py", line 54, in _handle_error
api_1                    |     error_message = json.loads(response.text).get('error', 'Unknown error occurred')
api_1                    |   File "/usr/local/lib/python3.10/json/__init__.py", line 346, in loads
api_1                    |     return _default_decoder.decode(s)
api_1                    |   File "/usr/local/lib/python3.10/json/decoder.py", line 337, in decode
api_1                    |     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
api_1                    |   File "/usr/local/lib/python3.10/json/decoder.py", line 355, in raw_decode
api_1                    |     raise JSONDecodeError("Expecting value", s, err.value) from None
api_1                    | json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
api_1                    | 
api_1                    | During handling of the above exception, another exception occurred:
api_1                    | 
api_1                    | Traceback (most recent call last):
api_1                    |   File "/app/api/.venv/lib/python3.10/site-packages/flask/app.py", line 880, in full_dispatch_request
api_1                    |     rv = self.dispatch_request()
api_1                    |   File "/app/api/.venv/lib/python3.10/site-packages/flask/app.py", line 865, in dispatch_request
api_1                    |     return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
api_1                    |   File "/app/api/.venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 489, in wrapper
api_1                    |     resp = resource(*args, **kwargs)
api_1                    |   File "/app/api/.venv/lib/python3.10/site-packages/flask/views.py", line 110, in view
api_1                    |     return current_app.ensure_sync(self.dispatch_request)(**kwargs)  # type: ignore[no-any-return]
api_1                    |   File "/app/api/.venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 604, in dispatch_request
api_1                    |     resp = meth(*args, **kwargs)
api_1                    |   File "/app/api/controllers/console/setup.py", line 74, in decorated
api_1                    |     return view(*args, **kwargs)
api_1                    |   File "/app/api/libs/login.py", line 91, in decorated_view
api_1                    |     return current_app.ensure_sync(func)(*args, **kwargs)
api_1                    |   File "/app/api/controllers/console/wraps.py", line 21, in decorated
api_1                    |     return view(*args, **kwargs)
api_1                    |   File "/app/api/controllers/console/auth/data_source_bearer_auth.py", line 53, in post
api_1                    |     raise ApiKeyAuthFailedError(str(e))
api_1                    | controllers.console.auth.error.ApiKeyAuthFailedError: 500 Internal Server Error: Expecting value: line 1 column 1 (char 0)
Originally created by @Edison-A-N on GitHub (Jun 30, 2024). Originally assigned to: @JohnJyong on GitHub. ### Self Checks - [X] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [X] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [X] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [X] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:） - [X] Please do not modify this template :) and fill in all the required fields. ### Dify version eab0ac3a1 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce Enter the correct API Key in Firecrawl Add the Firecrawl API Key in Difi Agent Knowledge Observe if Difi throws a 500 error ### ✔️ Expected Behavior _No response_ ### ❌ Actual Behavior worker logs: ``` api_1 | 2024-06-30 09:07:14,849.849 ERROR [Dummy-156] [app.py:838] - Exception on /console/api/api-key-auth/data-source/binding [POST] api_1 | Traceback (most recent call last): api_1 | File "/app/api/controllers/console/auth/data_source_bearer_auth.py", line 51, in post api_1 | ApiKeyAuthService.create_provider_auth(current_user.current_tenant_id, args) api_1 | File "/app/api/services/auth/api_key_auth_service.py", line 21, in create_provider_auth api_1 | auth_result = ApiKeyAuthFactory(args['provider'], args['credentials']).validate_credentials() api_1 | File "/app/api/services/auth/api_key_auth_factory.py", line 14, in validate_credentials api_1 | return self.auth.validate_credentials() api_1 | File "/app/api/services/auth/firecrawl.py", line 37, in validate_credentials api_1 | self._handle_error(response) api_1 | File "/app/api/services/auth/firecrawl.py", line 54, in _handle_error api_1 | error_message = json.loads(response.text).get('error', 'Unknown error occurred') api_1 | File "/usr/local/lib/python3.10/json/__init__.py", line 346, in loads api_1 | return _default_decoder.decode(s) api_1 | File "/usr/local/lib/python3.10/json/decoder.py", line 337, in decode api_1 | obj, end = self.raw_decode(s, idx=_w(s, 0).end()) api_1 | File "/usr/local/lib/python3.10/json/decoder.py", line 355, in raw_decode api_1 | raise JSONDecodeError("Expecting value", s, err.value) from None api_1 | json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) api_1 | api_1 | During handling of the above exception, another exception occurred: api_1 | api_1 | Traceback (most recent call last): api_1 | File "/app/api/.venv/lib/python3.10/site-packages/flask/app.py", line 880, in full_dispatch_request api_1 | rv = self.dispatch_request() api_1 | File "/app/api/.venv/lib/python3.10/site-packages/flask/app.py", line 865, in dispatch_request api_1 | return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) # type: ignore[no-any-return] api_1 | File "/app/api/.venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 489, in wrapper api_1 | resp = resource(*args, **kwargs) api_1 | File "/app/api/.venv/lib/python3.10/site-packages/flask/views.py", line 110, in view api_1 | return current_app.ensure_sync(self.dispatch_request)(**kwargs) # type: ignore[no-any-return] api_1 | File "/app/api/.venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 604, in dispatch_request api_1 | resp = meth(*args, **kwargs) api_1 | File "/app/api/controllers/console/setup.py", line 74, in decorated api_1 | return view(*args, **kwargs) api_1 | File "/app/api/libs/login.py", line 91, in decorated_view api_1 | return current_app.ensure_sync(func)(*args, **kwargs) api_1 | File "/app/api/controllers/console/wraps.py", line 21, in decorated api_1 | return view(*args, **kwargs) api_1 | File "/app/api/controllers/console/auth/data_source_bearer_auth.py", line 53, in post api_1 | raise ApiKeyAuthFailedError(str(e)) api_1 | controllers.console.auth.error.ApiKeyAuthFailedError: 500 Internal Server Error: Expecting value: line 1 column 1 (char 0) ```
yindo added the 🐞 bug label 2026-02-21 18:05:56 -05:00
yindo closed this issue 2026-02-21 18:05:56 -05:00
yindo commented 2026-02-21 18:05:56 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Jun 30, 2024):

To resolve the 500 Internal Server Error when using the Firecrawl API Key in Dify, follow these steps:

  1. Check the API Key: Ensure that the API key is correctly provided and valid.
  2. Verify the Base URL: Make sure the base_url is correct and the Firecrawl service is up and running.
  3. Inspect the Error Message: The error message indicates a JSON decoding issue, which suggests that the response from the Firecrawl API might not be in JSON format. Check the response from the Firecrawl API to understand the root cause.

The _handle_error method in the FirecrawlApp class captures the error message from the response and raises an exception with it. Here is the relevant part of the code:

def _handle_error(self, response, action):
    error_message = response.json().get('error', 'Unknown error occurred')
    raise Exception(f'Failed to {action}. Status code: {response.status_code}. Error: {error_message}')

Additionally, ensure that Firecrawl is properly configured in Dify:

  1. Navigate to the Firecrawl Configuration Section:

    • Go to the section where you can configure Firecrawl in Dify.

  2. Enter the API Key:

    • Enter your API key from firecrawl.dev.

  3. Configure Firecrawl Options:

    • Set various options such as crawl sub-pages, limit, max depth, exclude paths, include only paths, and extract only main content.

  4. Run Firecrawl:

    • Once configured, run Firecrawl by clicking the "运行" button.

If the error persists, you may need to contact Firecrawl support for further assistance.

For more details, you can refer to the Dify documentation: Firecrawl docs [1][2][3].

To continue talking to Dosu, mention @dosu.

@dosubot[bot] commented on GitHub (Jun 30, 2024): <!-- Answer --> To resolve the 500 Internal Server Error when using the Firecrawl API Key in Dify, follow these steps: 1. **Check the API Key**: Ensure that the API key is correctly provided and valid. 2. **Verify the Base URL**: Make sure the `base_url` is correct and the Firecrawl service is up and running. 3. **Inspect the Error Message**: The error message indicates a JSON decoding issue, which suggests that the response from the Firecrawl API might not be in JSON format. Check the response from the Firecrawl API to understand the root cause. The `_handle_error` method in the `FirecrawlApp` class captures the error message from the response and raises an exception with it. Here is the relevant part of the code: ```python def _handle_error(self, response, action): error_message = response.json().get('error', 'Unknown error occurred') raise Exception(f'Failed to {action}. Status code: {response.status_code}. Error: {error_message}') ``` Additionally, ensure that Firecrawl is properly configured in Dify: 1. **Navigate to the Firecrawl Configuration Section**: - Go to the section where you can configure Firecrawl in Dify. 2. **Enter the API Key**: - Enter your API key from firecrawl.dev. 3. **Configure Firecrawl Options**: - Set various options such as crawl sub-pages, limit, max depth, exclude paths, include only paths, and extract only main content. 4. **Run Firecrawl**: - Once configured, run Firecrawl by clicking the "运行" button. If the error persists, you may need to contact Firecrawl support for further assistance. For more details, you can refer to the Dify documentation: [Firecrawl docs](https://docs.dify.ai/v/zh-hans/guides/knowledge-base/sync_from_website) <sup>[[1]](https://github.com/langgenius/dify/blob/main/api/services/auth/firecrawl.py)</sup><sup>[[2]](https://github.com/langgenius/dify/blob/main/api/core/tools/provider/builtin/firecrawl/firecrawl_appx.py)</sup><sup>[[3]](https://github.com/langgenius/dify/blob/main/api/core/rag/extractor/firecrawl/firecrawl_app.py)</sup>. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.*
yindo commented 2026-02-21 18:05:56 -05:00
Author
Owner
Copy Link
Copy Source

@JohnJyong commented on GitHub (Jun 30, 2024):

This problem cannot be reproduced. Please confirm whether your local connection to Firecrawl can be successful， thanks @Edison-A-N

@JohnJyong commented on GitHub (Jun 30, 2024): This problem cannot be reproduced. Please confirm whether your local connection to Firecrawl can be successful， thanks @Edison-A-N
yindo commented 2026-02-21 18:05:56 -05:00
Author
Owner
Copy Link
Copy Source

@Edison-A-N commented on GitHub (Jul 1, 2024):

Thank you. It seems I only need to fill in the Key, and don't need to fill in the base_url. This way it should work successfully. Thank you very much.

@Edison-A-N commented on GitHub (Jul 1, 2024): Thank you. It seems I only need to fill in the Key, and don't need to fill in the base_url. This way it should work successfully. Thank you very much.
yindo commented 2026-02-21 18:05:56 -05:00
Author
Owner
Copy Link
Copy Source

@kawtarazel commented on GitHub (Nov 21, 2024):

i got the firecrawl API key but it tells me (API KEY not provided ), even it's correct
it works with antrhopic.
my project aims to scrap an arabic website.
https://colab.research.google.com/github/MohammedNasserAhmed/AINARABIC/blob/main/Arabic_Web_Scraping_with_Anthropic_and_Firecrawl_APIs.ipynb#scrollTo=PSMgwoRWnofY.

@kawtarazel commented on GitHub (Nov 21, 2024): i got the firecrawl API key but it tells me (API KEY not provided ), even it's correct it works with antrhopic. my project aims to scrap an arabic website. https://colab.research.google.com/github/MohammedNasserAhmed/AINARABIC/blob/main/Arabic_Web_Scraping_with_Anthropic_and_Firecrawl_APIs.ipynb#scrollTo=PSMgwoRWnofY.
yindo commented 2026-02-21 18:05:57 -05:00
Author
Owner
Copy Link
Copy Source

@wycoal commented on GitHub (May 11, 2025):

谢谢。看来我只需要填写 Key，不需要填写 base_url。这样应该就能成功了。非常感谢。

delete base url, but display :Failed to authorize. Status code: 401. Error: Unauthorized: Invalid token

@wycoal commented on GitHub (May 11, 2025): > 谢谢。看来我只需要填写 Key，不需要填写 base_url。这样应该就能成功了。非常感谢。 delete base url, but display :Failed to authorize. Status code: 401. Error: Unauthorized: Invalid token
yindo commented 2026-02-21 18:05:57 -05:00
Author
Owner
Copy Link
Copy Source

@wycoal commented on GitHub (May 11, 2025):

I have the similar problem, local firecrawl, input apiKey and baseUrl with http://host.docker.internal:3002, it display Failed to authorize. Status code: 500. Error: An unexpected error occurred. Please contact help@firecrawl.com for help. Your exception ID is 26feb1a7153e47b58324d914778dc80.
the firecrawl works with curl -X GET http://localhost:3002/test

@wycoal commented on GitHub (May 11, 2025): I have the similar problem, local firecrawl, input apiKey and baseUrl with http://host.docker.internal:3002, it display Failed to authorize. Status code: 500. Error: An unexpected error occurred. Please contact help@firecrawl.com for help. Your exception ID is 26feb1a7153e47b58324d914778dc80. the firecrawl works with curl -X GET http://localhost:3002/test
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#4343
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?