langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-22 04:25:29 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

NVIDIA Triton Inference Server HTTP API Error #7966

New Issue
Closed
opened 2026-02-21 18:23:17 -05:00 by yindo · 8 comments
yindo commented 2026-02-21 18:23:17 -05:00
Owner
Copy Link
Copy Source

Originally created by @gonzal40 on GitHub (Feb 4, 2025).

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
  • Please do not modify this template :) and fill in all the required fields.

Dify version

v0.15.2 and v1.0.0 Beta

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

  1. Deploy clean installation of Triton Inference Server in Docker from Nvidia NGC container (https://catalog.ngc.nvidia.com/orgs/nvidia/containers/tritonserver)
  2. Deploy clean installation with docker compose (https://docs.dify.ai/getting-started/install-self-hosted/docker-compose)
  3. Configure Dify model provider using NVIDIA Triton Inference Server model provider plugin
  4. Receive the following error when saving configuration: An error occurred during connection: An error occurred during connection: Client error '400 Bad Request' for url 'http://192.168.20.145:8000/v2/models/Meta-Llama-3.1-8B-Instruct-ONNX-INT4/generate' For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/400
  5. Nvidia's documentation for Triton Inference Server defines error 400 as:TRITONSERVER_ERROR_UNKNOWN,
    TRITONSERVER_ERROR_INVALID_ARG,
    TRITONSERVER_ERROR_ALREADY_EXISTS,
    TRITONSERVER_ERROR_CANCELLED
    400
    Bad Request (default for other errors)

✔️ Expected Behavior

Expected behavior is for the model provider configuration to save after confirming connectivity to the Triton Inference Sever. According to the Triton Inference Server documentation (https://github.com/triton-inference-server/server/blob/main/docs/protocol/extension_parameters.md), it appears the HTTP API should specify /v2/models/{model name}/infer when making a call for inference.

❌ Actual Behavior

Instead of calling the HTTP API in the way Triton Inference Server expects (/v2/models/{model name}/infer), the Dify plugin calls /v2/models/{model name}/generate which appears to be causing the 400 error.

Originally created by @gonzal40 on GitHub (Feb 4, 2025). ### Self Checks - [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [x] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:） - [x] Please do not modify this template :) and fill in all the required fields. ### Dify version v0.15.2 and v1.0.0 Beta ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce 1. Deploy clean installation of Triton Inference Server in Docker from Nvidia NGC container (https://catalog.ngc.nvidia.com/orgs/nvidia/containers/tritonserver) 2. Deploy clean installation with docker compose (https://docs.dify.ai/getting-started/install-self-hosted/docker-compose) 3. Configure Dify model provider using NVIDIA Triton Inference Server model provider plugin 4. Receive the following error when saving configuration: An error occurred during connection: An error occurred during connection: Client error '400 Bad Request' for url 'http://192.168.20.145:8000/v2/models/Meta-Llama-3.1-8B-Instruct-ONNX-INT4/generate' For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/400 5. Nvidia's documentation for Triton Inference Server defines error 400 as:TRITONSERVER_ERROR_UNKNOWN, TRITONSERVER_ERROR_INVALID_ARG, TRITONSERVER_ERROR_ALREADY_EXISTS, TRITONSERVER_ERROR_CANCELLED 400 Bad Request (default for other errors) ### ✔️ Expected Behavior Expected behavior is for the model provider configuration to save after confirming connectivity to the Triton Inference Sever. According to the Triton Inference Server documentation (https://github.com/triton-inference-server/server/blob/main/docs/protocol/extension_parameters.md), it appears the HTTP API should specify /v2/models/{model name}/infer when making a call for inference. ### ❌ Actual Behavior Instead of calling the HTTP API in the way Triton Inference Server expects (/v2/models/{model name}/infer), the Dify plugin calls /v2/models/{model name}/generate which appears to be causing the 400 error.
yindo added the 🐞 bug label 2026-02-21 18:23:17 -05:00
yindo closed this issue 2026-02-21 18:23:17 -05:00
yindo commented 2026-02-21 18:23:17 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Feb 4, 2025):

Hey there, @gonzal40! I'm here to help you with any bugs, questions, or contributions. Let's tackle this issue together!

To resolve the 400 Bad Request error you're encountering, you need to modify the endpoint being called in the _generate method of the TritonInferenceAILargeLanguageModel class. The Dify plugin is incorrectly calling the endpoint /v2/models/<model name>/generate instead of the correct /v2/models/<model name>/infer.

Here's how you can fix it:

  1. Locate the _generate method in the TritonInferenceAILargeLanguageModel class, which is implemented in the file api/core/model_runtime/model_providers/triton_inference_server/llm/llm.py.

  2. Change the endpoint from "generate" to "infer" in the POST request:

response = post(
    str(URL(credentials["server_url"]) / "v2" / "models" / model / "infer"),
    json={
        "text_input": self._convert_prompt_message_to_text(prompt_messages),
        "max_tokens": model_parameters.get("max_tokens", 512),
        "parameters": {"stream": False, **parameters},
    },
    timeout=(10, 120),
)

This change should correct the endpoint being called and resolve the 400 Bad Request error [1].

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other

@dosubot[bot] commented on GitHub (Feb 4, 2025): <!-- Greeting --> Hey there, @gonzal40! I'm here to help you with any bugs, questions, or contributions. Let's tackle this issue together! <!-- Answer --> To resolve the 400 Bad Request error you're encountering, you need to modify the endpoint being called in the `_generate` method of the `TritonInferenceAILargeLanguageModel` class. The Dify plugin is incorrectly calling the endpoint `/v2/models/<model name>/generate` instead of the correct `/v2/models/<model name>/infer`. Here's how you can fix it: 1. Locate the `_generate` method in the `TritonInferenceAILargeLanguageModel` class, which is implemented in the file `api/core/model_runtime/model_providers/triton_inference_server/llm/llm.py`. 2. Change the endpoint from `"generate"` to `"infer"` in the POST request: ```python response = post( str(URL(credentials["server_url"]) / "v2" / "models" / model / "infer"), json={ "text_input": self._convert_prompt_message_to_text(prompt_messages), "max_tokens": model_parameters.get("max_tokens", 512), "parameters": {"stream": False, **parameters}, }, timeout=(10, 120), ) ``` This change should correct the endpoint being called and resolve the 400 Bad Request error <sup>[[1]](https://github.com/langgenius/dify/blob/main/api/core/model_runtime/model_providers/triton_inference_server/llm/llm.py)</sup>. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.* --- <sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/3624001f-4a1e-4ce6-8243-4efc5264178b?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/3624001f-4a1e-4ce6-8243-4efc5264178b?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/3624001f-4a1e-4ce6-8243-4efc5264178b?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/3624001f-4a1e-4ce6-8243-4efc5264178b?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/3624001f-4a1e-4ce6-8243-4efc5264178b?feedback_type=hallucination) | [Other](https://app.dosu.dev/response-feedback/3624001f-4a1e-4ce6-8243-4efc5264178b?feedback_type=other)</sup>
yindo commented 2026-02-21 18:23:17 -05:00
Author
Owner
Copy Link
Copy Source

@gonzal40 commented on GitHub (Feb 4, 2025):

@dosu - I edited the llm.py file as suggested and the exact same error message appears. Perhaps this suggests that "generate" is defined elsewhere?

@gonzal40 commented on GitHub (Feb 4, 2025): @dosu - I edited the llm.py file as suggested and the exact same error message appears. Perhaps this suggests that "generate" is defined elsewhere?
yindo commented 2026-02-21 18:23:17 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Feb 4, 2025):

It's possible that the "generate" endpoint is referenced elsewhere in the codebase. To investigate further, you can search the entire codebase for occurrences of the term "generate" to identify any other references or definitions. This can be done using a command like grep in the terminal or using the search functionality in your code editor.

Additionally, ensure that the changes you made to the llm.py file were saved and that the application was restarted to apply those changes. If the issue persists, there might be other parts of the code or configuration files that need to be updated.

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other

@dosubot[bot] commented on GitHub (Feb 4, 2025): <!-- Answer --> It's possible that the "generate" endpoint is referenced elsewhere in the codebase. To investigate further, you can search the entire codebase for occurrences of the term "generate" to identify any other references or definitions. This can be done using a command like `grep` in the terminal or using the search functionality in your code editor. Additionally, ensure that the changes you made to the `llm.py` file were saved and that the application was restarted to apply those changes. If the issue persists, there might be other parts of the code or configuration files that need to be updated. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.* --- <sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/5125ab0e-7085-428f-b15c-5f50bc2d4849?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/5125ab0e-7085-428f-b15c-5f50bc2d4849?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/5125ab0e-7085-428f-b15c-5f50bc2d4849?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/5125ab0e-7085-428f-b15c-5f50bc2d4849?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/5125ab0e-7085-428f-b15c-5f50bc2d4849?feedback_type=hallucination) | [Other](https://app.dosu.dev/response-feedback/5125ab0e-7085-428f-b15c-5f50bc2d4849?feedback_type=other)</sup>
yindo commented 2026-02-21 18:23:17 -05:00
Author
Owner
Copy Link
Copy Source

@crazywoola commented on GitHub (Mar 7, 2025):

@crazywoola commented on GitHub (Mar 7, 2025): - Please open an issue in this repo instead, because we need more information regarding the tool's version. - https://github.com/langgenius/dify-official-plugins/
yindo commented 2026-02-21 18:23:17 -05:00
Author
Owner
Copy Link
Copy Source

@AnyangAngus commented on GitHub (Mar 26, 2025):

@dosu - I edited the llm.py file as suggested and the exact same error message appears. Perhaps this suggests that "generate" is defined elsewhere?

@gonzal40 meet the same problem , Did you solve this request url issue?

@AnyangAngus commented on GitHub (Mar 26, 2025): > [@dosu](https://github.com/dosu) - I edited the llm.py file as suggested and the exact same error message appears. Perhaps this suggests that "generate" is defined elsewhere? @gonzal40 meet the same problem , Did you solve this request url issue?
yindo commented 2026-02-21 18:23:17 -05:00
Author
Owner
Copy Link
Copy Source

@gonzal40 commented on GitHub (Mar 27, 2025):

No, I haven’t resolved the issue. I received the following comment a few days ago, but haven’t had time to get back into this.crazywoola left a comment (langgenius/dify#13192)Please open an issue in this repo instead, because we need more information regarding the tool's version.https://github.com/langgenius/dify-official-plugins/On Mar 26, 2025, at 10:10 PM, AnyangAngus @.***> wrote:

@dosu - I edited the llm.py file as suggested and the exact same error message appears. Perhaps this suggests that "generate" is defined elsewhere?

@gonzal40 meet the same problem , Did you solve this request url issue?—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>

AnyangAngus left a comment (langgenius/dify#13192)

@dosu - I edited the llm.py file as suggested and the exact same error message appears. Perhaps this suggests that "generate" is defined elsewhere?

@gonzal40 meet the same problem , Did you solve this request url issue?

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>

@gonzal40 commented on GitHub (Mar 27, 2025): No, I haven’t resolved the issue. I received the following comment a few days ago, but haven’t had time to get back into this.crazywoola left a comment (langgenius/dify#13192)Please open an issue in this repo instead, because we need more information regarding the tool's version.https://github.com/langgenius/dify-official-plugins/On Mar 26, 2025, at 10:10 PM, AnyangAngus ***@***.***> wrote: @dosu - I edited the llm.py file as suggested and the exact same error message appears. Perhaps this suggests that "generate" is defined elsewhere? @gonzal40 meet the same problem , Did you solve this request url issue?—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***> AnyangAngus left a comment (langgenius/dify#13192) @dosu - I edited the llm.py file as suggested and the exact same error message appears. Perhaps this suggests that "generate" is defined elsewhere? @gonzal40 meet the same problem , Did you solve this request url issue? —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>
yindo commented 2026-02-21 18:23:17 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Mar 27, 2025):

To address the issue with the "generate" endpoint, you should search the entire Dify codebase for any other references to "generate" that might be causing the problem. This can be done using a command like grep in the terminal or the search functionality in your code editor. It's possible that there are other files or modules referencing this endpoint, which need to be updated to "infer" as well.

Additionally, ensure that any changes made to the code are saved and that the application is restarted to apply those changes. If the problem persists, consider checking configuration files or environment variables that might influence the endpoint being used for model inference [1].

A maintainer has already been looped in, so you might also want to follow up on their suggestion to open an issue in the Dify official plugins repository for further assistance. This will help gather more information regarding the tool's version and any other specifics that might be relevant to resolving the issue.

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other

@dosubot[bot] commented on GitHub (Mar 27, 2025): <!-- Answer --> To address the issue with the "generate" endpoint, you should search the entire Dify codebase for any other references to "generate" that might be causing the problem. This can be done using a command like `grep` in the terminal or the search functionality in your code editor. It's possible that there are other files or modules referencing this endpoint, which need to be updated to "infer" as well. Additionally, ensure that any changes made to the code are saved and that the application is restarted to apply those changes. If the problem persists, consider checking configuration files or environment variables that might influence the endpoint being used for model inference <sup>[[1]](https://github.com/langgenius/dify/issues/11816)</sup>. A maintainer has already been looped in, so you might also want to follow up on their suggestion to open an issue in the Dify official plugins repository for further assistance. This will help gather more information regarding the tool's version and any other specifics that might be relevant to resolving the issue. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.* --- <sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/70ae0770-02cd-42bb-a3fd-6091d45f9270?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/70ae0770-02cd-42bb-a3fd-6091d45f9270?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/70ae0770-02cd-42bb-a3fd-6091d45f9270?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/70ae0770-02cd-42bb-a3fd-6091d45f9270?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/70ae0770-02cd-42bb-a3fd-6091d45f9270?feedback_type=hallucination) | [Bug Report](https://app.dosu.dev/response-feedback/70ae0770-02cd-42bb-a3fd-6091d45f9270?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/70ae0770-02cd-42bb-a3fd-6091d45f9270?feedback_type=other)</sup>
yindo commented 2026-02-21 18:23:17 -05:00
Author
Owner
Copy Link
Copy Source

@gonzal40 commented on GitHub (Mar 29, 2025):

@crazywoola @AnyangAngus I opened a new issue as requested after updating after returning to this after a few weeks. I updated all software to current versions and the issue persists. The new issue can be accessed at the following link: https://github.com/langgenius/dify-official-plugins/issues/605.

As an additional point of interest, it looks like Nvidia recently released a successor to the Triton Inference Server called Dynamo. Details can be found at the following link: https://www.nvidia.com/en-us/ai/dynamo/

@gonzal40 commented on GitHub (Mar 29, 2025): @crazywoola @AnyangAngus I opened a new issue as requested after updating after returning to this after a few weeks. I updated all software to current versions and the issue persists. The new issue can be accessed at the following link: https://github.com/langgenius/dify-official-plugins/issues/605. As an additional point of interest, it looks like Nvidia recently released a successor to the Triton Inference Server called Dynamo. Details can be found at the following link: https://www.nvidia.com/en-us/ai/dynamo/
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#7966
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?