langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 12:05:30 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Vertex AI API failed with squid proxy #5643

New Issue
Closed
opened 2026-02-21 18:12:00 -05:00 by yindo · 3 comments
yindo commented 2026-02-21 18:12:00 -05:00
Owner
Copy Link
Copy Source

Originally created by @tealeeseng on GitHub (Sep 16, 2024).

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
  • Please do not modify this template :) and fill in all the required fields.

Dify version

0.8.0

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

DIFY dockers are running with external squid proxy configured on API docker via HTTPS_PROXY environmental parameter. The setup works with Azure GPT models and self hosted llama3.1-8b model under vllm (openAI api compliant).

For gemini-1.5 flash model hosted in google Cloud's vertex AI, the DIFY API docker failed with error message, "[vertex_ai] Server Unavailable Error, 503 failed to connect to all addresses; last error: UNKNOWN: ipv4:142.251.10.95:443: Failed to connect to remote host: Timeout occurred: FD Shutdown" in chatbot App UI.

Screenshot from 2024-09-16 10-58-31

/

✔️ Expected Behavior

chatbot App UI behave normally

❌ Actual Behavior

the DIFY API docker failed with error message, "[vertex_ai] Server Unavailable Error, 503 failed to connect to all addresses; last error: UNKNOWN: ipv4:142.251.10.95:443: Failed to connect to remote host: Timeout occurred: FD Shutdown" in chatbot App UI.

Originally created by @tealeeseng on GitHub (Sep 16, 2024). ### Self Checks - [X] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [X] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [X] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [X] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:） - [X] Please do not modify this template :) and fill in all the required fields. ### Dify version 0.8.0 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce DIFY dockers are running with external squid proxy configured on API docker via HTTPS_PROXY environmental parameter. The setup works with Azure GPT models and self hosted llama3.1-8b model under vllm (openAI api compliant). For gemini-1.5 flash model hosted in google Cloud's vertex AI, the DIFY API docker failed with error message, "[vertex_ai] Server Unavailable Error, 503 failed to connect to all addresses; last error: UNKNOWN: ipv4:142.251.10.95:443: Failed to connect to remote host: Timeout occurred: FD Shutdown" in chatbot App UI. ![Screenshot from 2024-09-16 10-58-31](https://github.com/user-attachments/assets/d4396d46-e11d-4cbd-bd13-38f517df63c4) / ### ✔️ Expected Behavior chatbot App UI behave normally ### ❌ Actual Behavior the DIFY API docker failed with error message, "[vertex_ai] Server Unavailable Error, 503 failed to connect to all addresses; last error: UNKNOWN: ipv4:142.251.10.95:443: Failed to connect to remote host: Timeout occurred: FD Shutdown" in chatbot App UI.
yindo added the 🐞 bug label 2026-02-21 18:12:00 -05:00
yindo closed this issue 2026-02-21 18:12:00 -05:00
yindo commented 2026-02-21 18:12:00 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Sep 16, 2024):

Hello @tealeeseng! I'm here to help you with any bugs, questions, or contributions you have for the project. Let's tackle this issue together!

I found some similar issues and discussions that might be relevant to your problem:

  1. connection error: request error, status_code: 503 [1]
  2. Can not set Vertex AI API [2]
  3. connection error when set up model provider [3]

Additionally, there are some closed issues that might provide insights:

  1. [google] Server Unavailable Error, 504 Deadline Exceeded and Error: Minified React error #185 [4]
  2. Model Provider add Google Gemini failed! [5]

For your specific issue with the Squid proxy and Vertex AI, here are some steps you can take:

  1. Verify Configuration:
    Ensure that the vertex_project_id, vertex_location, and vertex_service_account_key (if not using Application Default Credentials) are correctly set in the configuration. Here is the relevant configuration schema from the vertex_ai.yaml:

    provider: vertex_ai
label:
  en_US: Vertex AI | Google Cloud Platform
description:
  en_US: Vertex AI in Google Cloud Platform.
icon_small:
  en_US: icon_s_en.svg
icon_large:
  en_US: icon_l_en.png
background: "#FCFDFF"
help:
  title:
    en_US: Get your Access Details from Google
  url:
    en_US: https://cloud.google.com/vertex-ai/
supported_model_types:
  - llm
  - text-embedding
configurate_methods:
  - predefined-model
provider_credential_schema:
  credential_form_schemas:
    - variable: vertex_project_id
      label:
        en_US: Project ID
      type: text-input
      required: true
      placeholder:
        en_US: Enter your Google Cloud Project ID
    - variable: vertex_location
      label:
        en_US: Location
      type: text-input
      required: true
      placeholder:
        en_US: Enter your Google Cloud Location
    - variable: vertex_service_account_key
      label:
        en_US: Service Account Key (Leave blank if you use Application Default Credentials)
      type: secret-input
      required: false
      placeholder:
        en_US: Enter your Google Cloud Service Account Key in base64 format

  2. Check Proxy Configuration:
    Ensure that the HTTPS_PROXY environmental parameter is correctly set in your Docker setup. Here are the steps to configure it:

    • Add the HTTPS_PROXY variable to your environment configuration. This variable should be set to the address of your HTTPS proxy.

      Example for .env file:

      HTTPS_PROXY=http://your-proxy-address:port

      Example for docker-compose.yml:

      version: '3.8'
services:
  api:
    image: your-api-image
    environment:
      HTTPS_PROXY: http://your-proxy-address:port

    • Ensure that the proxy is accessible from within the Docker network.

  3. Network and Firewall Rules:
    Ensure that your network and firewall rules allow connections to the Vertex AI endpoints. This might involve allowing outbound traffic on port 443 to the IP addresses used by Vertex AI.

  4. Retry Mechanisms:
    Dify implements specific error handling and retry mechanisms for handling 503 errors. Ensure that these mechanisms are in place and properly configured. The _invoke_error_mapping property in the VertexAiLargeLanguageModel class maps various exceptions to unified error types, including handling 503 errors [6].

By following these steps, you should be able to resolve the connectivity issue with the Vertex AI API when using a Squid proxy. If the issue persists, please provide more details from the debug logs for further assistance.

To continue talking to Dosu, mention @dosu.

@dosubot[bot] commented on GitHub (Sep 16, 2024): <!-- Greeting --> Hello @tealeeseng! I'm here to help you with any bugs, questions, or contributions you have for the project. Let's tackle this issue together! <!-- Answer --> I found some similar issues and discussions that might be relevant to your problem: 1. [connection error: request error, status_code: 503](https://github.com/langgenius/dify/discussions/6181) <sup>[[1]](https://github.com/langgenius/dify/discussions/6181)</sup> 2. [Can not set Vertex AI API](https://github.com/langgenius/dify/discussions/7975) <sup>[[2]](https://github.com/langgenius/dify/discussions/7975)</sup> 3. [connection error when set up model provider](https://github.com/langgenius/dify/discussions/7839) <sup>[[3]](https://github.com/langgenius/dify/discussions/7839)</sup> Additionally, there are some closed issues that might provide insights: 1. [[google] Server Unavailable Error, 504 Deadline Exceeded and Error: Minified React error #185](https://github.com/langgenius/dify/issues/4232) <sup>[[4]](https://github.com/langgenius/dify/issues/4232)</sup> 2. [Model Provider add Google Gemini failed!](https://github.com/langgenius/dify/issues/4571) <sup>[[5]](https://github.com/langgenius/dify/issues/4571)</sup> For your specific issue with the Squid proxy and Vertex AI, here are some steps you can take: 1. **Verify Configuration**: Ensure that the `vertex_project_id`, `vertex_location`, and `vertex_service_account_key` (if not using Application Default Credentials) are correctly set in the configuration. Here is the relevant configuration schema from the `vertex_ai.yaml`: ```yaml provider: vertex_ai label: en_US: Vertex AI | Google Cloud Platform description: en_US: Vertex AI in Google Cloud Platform. icon_small: en_US: icon_s_en.svg icon_large: en_US: icon_l_en.png background: "#FCFDFF" help: title: en_US: Get your Access Details from Google url: en_US: https://cloud.google.com/vertex-ai/ supported_model_types: - llm - text-embedding configurate_methods: - predefined-model provider_credential_schema: credential_form_schemas: - variable: vertex_project_id label: en_US: Project ID type: text-input required: true placeholder: en_US: Enter your Google Cloud Project ID - variable: vertex_location label: en_US: Location type: text-input required: true placeholder: en_US: Enter your Google Cloud Location - variable: vertex_service_account_key label: en_US: Service Account Key (Leave blank if you use Application Default Credentials) type: secret-input required: false placeholder: en_US: Enter your Google Cloud Service Account Key in base64 format ``` 2. **Check Proxy Configuration**: Ensure that the `HTTPS_PROXY` environmental parameter is correctly set in your Docker setup. Here are the steps to configure it: - Add the `HTTPS_PROXY` variable to your environment configuration. This variable should be set to the address of your HTTPS proxy. Example for `.env` file: ```env HTTPS_PROXY=http://your-proxy-address:port ``` Example for `docker-compose.yml`: ```yaml version: '3.8' services: api: image: your-api-image environment: HTTPS_PROXY: http://your-proxy-address:port ``` - Ensure that the proxy is accessible from within the Docker network. 3. **Network and Firewall Rules**: Ensure that your network and firewall rules allow connections to the Vertex AI endpoints. This might involve allowing outbound traffic on port 443 to the IP addresses used by Vertex AI. 4. **Retry Mechanisms**: Dify implements specific error handling and retry mechanisms for handling 503 errors. Ensure that these mechanisms are in place and properly configured. The `_invoke_error_mapping` property in the `VertexAiLargeLanguageModel` class maps various exceptions to unified error types, including handling 503 errors <sup>[[6]](https://github.com/langgenius/dify/blob/main/api/core/model_runtime/model_providers/vertex_ai/llm/llm.py)</sup>. By following these steps, you should be able to resolve the connectivity issue with the Vertex AI API when using a Squid proxy. If the issue persists, please provide more details from the debug logs for further assistance. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.*
yindo commented 2026-02-21 18:12:00 -05:00
Author
Owner
Copy Link
Copy Source

@tealeeseng commented on GitHub (Dec 16, 2024):

Tried step 2 with squid proxy yet the same error message, "[vertex_ai] Server Unavailable Error, 503 failed to connect to all addresses; last error: UNKNOWN: ipv4:142.251.10.95:443: Failed to connect to remote host: Timeout occurred: FD Shutdown" shown in chatbot App UI.

@tealeeseng commented on GitHub (Dec 16, 2024): Tried step 2 with squid proxy yet the same error message, "[vertex_ai] Server Unavailable Error, 503 failed to connect to all addresses; last error: UNKNOWN: ipv4:142.251.10.95:443: Failed to connect to remote host: Timeout occurred: FD Shutdown" shown in chatbot App UI.
yindo commented 2026-02-21 18:12:00 -05:00
Author
Owner
Copy Link
Copy Source

@tomy-kyu commented on GitHub (Feb 16, 2025):

In order to access the Gemini API via Squid Proxy, it seems that it is important that Squid Proxy is able to interpret gRPC and is compatible with HTTP/2. If Squid Proxy is not compatible with gRPC, it will not be possible to communicate with the Gemini API/Vertex AI as it is.
However, it seems that REST is used for image uploads in Vertex AI.

In order to communicate with the Gemini API/Vertex AI API using Squid, which does not support HTTP/2, it is necessary to manually change to REST mode. However, it seems that this was not enough for the request processing of Vertex AI.

The following measures have made it possible to use the Gemini API to a certain extent. I will describe them for reference.
The target is the _generate function in api/core/model_runtime/model_providers/google/llm/llm.py.

I hope that these comments will lead to better handling of Dify in the future.

@tomy-kyu commented on GitHub (Feb 16, 2025): In order to access the Gemini API via Squid Proxy, it seems that it is important that Squid Proxy is able to interpret gRPC and is compatible with HTTP/2. If Squid Proxy is not compatible with gRPC, it will not be possible to communicate with the Gemini API/Vertex AI as it is. However, it seems that REST is used for image uploads in Vertex AI. In order to communicate with the Gemini API/Vertex AI API using Squid, which does not support HTTP/2, it is necessary to manually change to REST mode. However, it seems that this was not enough for the request processing of Vertex AI. The following measures have made it possible to use the Gemini API to a certain extent. I will describe them for reference. The target is the _generate function in api/core/model_runtime/model_providers/google/llm/llm.py. - Add transport=“rest” to configure (https://github.com/google-gemini/generative-ai-python/blob/8849d4f46010ce4ae68243c4f8a44a138b56598f/docs/api/google/generativeai/configure.md) - Explicitly set the processing argument stream to be passed to the result variable to False - Remove the conditional branch when the stream variable is set to True I hope that these comments will lead to better handling of Dify in the future.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#5643
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?