mirror of
https://github.com/langgenius/dify-official-plugins.git
synced 2026-07-21 17:45:23 -04:00
Error reported when connecting to the vllm-launched service using the OpenAI-API-compatible plugin. #297
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @suntao2015005848 on GitHub (May 19, 2025).
Self Checks
Dify version
1.4.0
Cloud or Self Hosted
Self Hosted (Docker)
Steps to reproduce
The vLLM model is deployed on 10.2.201.8. I tested the API locally using Postman, and it worked fine. Additionally, the Dify instance set up on 10.2.201.8 with OpenAI-API-compatible also functioned normally. However, when I tried to register via OpenAI-API-compatible on my other server at 192.168.42.68, it reported an error. I’m certain there are no network issues between 192.168.42.68 and 10.2.201.8, as the Xinference service on 10.2.201.8 is also accessible to 192.168.42.68.
vllm :
python3 -m vllm.entrypoints.openai.api_server
--model /xinference/my_model/QwQ-32B
--served-model-name QwQ-32B
--tensor-parallel-size 4
--trust-remote-code
--max_model_len 40960
--host 0.0.0.0
--port 8501
--dtype bfloat16
--enable-auto-tool-choice
--tool-call-parser hermes
--gpu-memory-utilization 0.9
erroe report :
docker-api-1 | 2025-05-19 08:19:40.177 ERROR [Dummy-2] [models.py:162] - Failed to save model credentials, tenant_id: 9d679366-403d-4498-9939-2f37e1d25f32, model: QwQ-32B, model_type: llm
docker-api-1 | Traceback (most recent call last):
docker-api-1 | File "/app/api/controllers/console/workspace/models.py", line 154, in post
docker-api-1 | model_provider_service.save_model_credentials(
docker-api-1 | File "/app/api/services/model_provider_service.py", line 237, in save_model_credentials
docker-api-1 | provider_configuration.add_or_update_custom_model_credentials(
docker-api-1 | File "/app/api/core/entities/provider_configuration.py", line 422, in add_or_update_custom_model_credentials
docker-api-1 | provider_model_record, credentials = self.custom_model_credentials_validate(model_type, model, credentials)
docker-api-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
docker-api-1 | File "/app/api/core/entities/provider_configuration.py", line 402, in custom_model_credentials_validate
docker-api-1 | credentials = model_provider_factory.model_credentials_validate(
docker-api-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
docker-api-1 | File "/app/api/core/model_runtime/model_providers/model_provider_factory.py", line 191, in model_credentials_validate
docker-api-1 | self.plugin_model_manager.validate_model_credentials(
docker-api-1 | File "/app/api/core/plugin/impl/model.py", line 137, in validate_model_credentials
docker-api-1 | for resp in response:
docker-api-1 | ^^^^^^^^
docker-api-1 | File "/app/api/core/plugin/impl/base.py", line 192, in _request_with_plugin_daemon_response_stream
docker-api-1 | self._handle_plugin_daemon_error(error.error_type, error.message)
docker-api-1 | File "/app/api/core/plugin/impl/base.py", line 222, in _handle_plugin_daemon_error
docker-api-1 | raise CredentialsValidateFailedError(error_object.get("message"))
docker-api-1 | core.model_runtime.errors.validate.CredentialsValidateFailedError: An error occurred during credentials validation: HTTPConnectionPool(host='10.2.201.8', port=8501): Max retries exceeded with url: /v1/chat/completions (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7ff1d51198b0>, 'Connection to 10.2.201.8 timed out. (connect timeout=10)'))
✔️ Expected Behavior
No response
❌ Actual Behavior
error
@dosubot[bot] commented on GitHub (May 19, 2025):
I found a similar resolved issue that might help you. The suggested solution for a similar problem was to ensure that containers spawned by Docker Compose do not inherit HTTP proxy settings from the Docker host, as this can cause connectivity issues. Additionally, before using the OpenAI-API-compatible model, you should turn on LAN services in LM Studio's settings and verify that the API endpoint URL is correctly set [1].
In your case, the error message indicates a
ConnectTimeoutError, which suggests that the service athttp://10.2.201.8:8501/v1is not reachable. Here are a few steps you can take to troubleshoot this issue:Check Network Connectivity: Ensure that the machine running the Docker container can reach the IP address
10.2.201.8on port8501. You can use tools likepingorcurlto test connectivity.Verify Service Availability: Make sure that the service you are trying to connect to is running and listening on the specified IP and port. You can check this by accessing the endpoint directly from a browser or using a tool like
curl.Review Docker Network Settings: If you are using Docker, ensure that the network settings allow communication between the containers and the host network.
Check Proxy Settings: If your environment uses a proxy, ensure that the proxy settings are correctly configured and not interfering with the connection.
Increase Timeout: If the service is slow to respond, consider increasing the connection timeout settings in your application to allow more time for the connection to be established.
To continue talking to Dosu, mention @dosu.
Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other
@suntao2015005848 commented on GitHub (May 19, 2025):
@dosu The vLLM model is deployed on 10.2.201.8. I tested the API locally using Postman, and it worked fine. Additionally, the Dify instance set up on 10.2.201.8 with OpenAI-API-compatible also functioned normally. However, when I tried to register via OpenAI-API-compatible on my other server at 192.168.42.68, it reported an error. I’m certain there are no network issues between 192.168.42.68 and 10.2.201.8, as the Xinference service on 10.2.201.8 is also accessible to 192.168.42.68.
@dosubot[bot] commented on GitHub (May 19, 2025):
The error you're encountering might be related to the
ConnectTimeoutError, which can occur due to several reasons. Here are some steps to help troubleshoot and resolve the issue:Verify API Endpoint Configuration: Ensure that the API endpoint URL is correctly configured in Dify for the OpenAI-API-compatible service. Double-check the endpoint URL and make sure it matches the one you tested successfully with Postman.
Network Configuration: Although you mentioned there are no network issues, it's worth verifying that the Docker container on server
192.168.42.68can indeed reach10.2.201.8on port8501. You can usecurlortelnetfrom within the Docker container to test connectivity.Docker Network Settings: Ensure that the Docker network settings allow communication between the containers and the host network. You might need to adjust the Docker Compose network configuration to ensure proper routing.
Proxy Settings: Check if there are any proxy settings configured in your environment that might be affecting the connection. Proxy settings can sometimes lead to connection issues.
Timeout Settings: Consider increasing the connection timeout settings in Dify. The default connection timeout is 10 seconds, which might be insufficient if the service is slow to respond. You can adjust these settings in the
CodeExecutionSandboxConfigclass within theapi/configs/feature/__init__.pyfile [1].By following these steps, you should be able to identify and resolve the connectivity issue between your servers.
To continue talking to Dosu, mention @dosu.
Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other