langgenius/dify-official-plugins
1
0
Fork 0
mirror of https://github.com/langgenius/dify-official-plugins.git synced 2026-07-21 17:45:23 -04:00
Code Issues 54 Packages Projects Releases Wiki Activity

NVIDIA Triton Inference Server HTTP API Error #148

New Issue
Closed
opened 2026-02-16 10:18:14 -05:00 by yindo · 2 comments
yindo commented 2026-02-16 10:18:14 -05:00
Owner
Copy Link
Copy Source

Originally created by @gonzal40 on GitHub (Mar 29, 2025).

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues Dify issues & Dify Official Plugins, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
  • Please do not modify this template :) and fill in all the required fields.

Dify version

v0.15.2. v1.0.0 Beta, v1.1.3

Plugin version

0.02

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

  1. Deploy clean installation of Triton Inference Server in Docker (nvcr.io/nvidia/tritonserver:25.02-py3-igpu) from Nvidia NGC container (https://catalog.ngc.nvidia.com/orgs/nvidia/containers/tritonserver)
  2. Deploy clean installation with docker compose (https://docs.dify.ai/getting-started/install-self-hosted/docker-compose)
  3. Configure Dify model provider using NVIDIA Triton Inference Server model provider plugin
  4. Receive the following error when saving configuration: An error occurred during connection: An error occurred during connection: Client error '400 Bad Request' for url 'http://192.168.20.145:8000/v2/models/Meta-Llama-3.1-8B-Instruct-ONNX-INT4/generate' For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/400
  5. Same error is presented when attempting to connect to Triton Inference Server with any other LLM model.

Note: Nvidia's documentation for Triton Inference Server defines error 400 as:TRITONSERVER_ERROR_UNKNOWN,
TRITONSERVER_ERROR_INVALID_ARG,
TRITONSERVER_ERROR_ALREADY_EXISTS,
TRITONSERVER_ERROR_CANCELLED
400
Bad Request (default for other errors)

✔️ Error log

This is the only error presented in the web interface when attempting to save my configuration details in the Triton Server plugin:

An error occurred during connection: An error occurred during connection: Client error '400 Bad Request' for url 'http://192.168.20.145:8000/v2/models/Meta-Llama-3.1-8B-Instruct-ONNX-INT4/generate' For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/400

✔️ Expected Behavior

Expected behavior is for the model provider configuration to save after confirming connectivity to the Triton Inference Sever. According to the Triton Inference Server documentation (https://github.com/triton-inference-server/server/blob/main/docs/protocol/extension_parameters.md), it appears the HTTP API should specify /v2/models/{model name}/infer when making a call for inference.

❌ Actual Behavior

Instead of calling the HTTP API in the way Triton Inference Server expects (/v2/models/{model name}/infer), the Dify plugin calls /v2/models/{model name}/generate which appears to be causing the 400 error.

Originally created by @gonzal40 on GitHub (Mar 29, 2025). ### Self Checks - [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [x] I have searched for existing issues [Dify issues](https://github.com/langgenius/dify/issues) & [Dify Official Plugins](https://github.com/langgenius/dify-official-plugins/issues), including closed ones. - [x] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [x] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:） - [x] Please do not modify this template :) and fill in all the required fields. ### Dify version v0.15.2. v1.0.0 Beta, v1.1.3 ### Plugin version 0.02 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce 1. Deploy clean installation of Triton Inference Server in Docker (nvcr.io/nvidia/tritonserver:25.02-py3-igpu) from Nvidia NGC container (https://catalog.ngc.nvidia.com/orgs/nvidia/containers/tritonserver) 2. Deploy clean installation with docker compose (https://docs.dify.ai/getting-started/install-self-hosted/docker-compose) 3. Configure Dify model provider using NVIDIA Triton Inference Server model provider plugin 4. Receive the following error when saving configuration: An error occurred during connection: An error occurred during connection: Client error '400 Bad Request' for url 'http://192.168.20.145:8000/v2/models/Meta-Llama-3.1-8B-Instruct-ONNX-INT4/generate' For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/400 5. Same error is presented when attempting to connect to Triton Inference Server with any other LLM model. Note: Nvidia's documentation for Triton Inference Server defines error 400 as:TRITONSERVER_ERROR_UNKNOWN, TRITONSERVER_ERROR_INVALID_ARG, TRITONSERVER_ERROR_ALREADY_EXISTS, TRITONSERVER_ERROR_CANCELLED 400 Bad Request (default for other errors) ### ✔️ Error log This is the only error presented in the web interface when attempting to save my configuration details in the Triton Server plugin: An error occurred during connection: An error occurred during connection: Client error '400 Bad Request' for url 'http://192.168.20.145:8000/v2/models/Meta-Llama-3.1-8B-Instruct-ONNX-INT4/generate' For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/400 ✔️ Expected Behavior Expected behavior is for the model provider configuration to save after confirming connectivity to the Triton Inference Sever. According to the Triton Inference Server documentation (https://github.com/triton-inference-server/server/blob/main/docs/protocol/extension_parameters.md), it appears the HTTP API should specify /v2/models/{model name}/infer when making a call for inference. ❌ Actual Behavior Instead of calling the HTTP API in the way Triton Inference Server expects (/v2/models/{model name}/infer), the Dify plugin calls /v2/models/{model name}/generate which appears to be causing the 400 error.
yindo added the bug label 2026-02-16 10:18:14 -05:00
yindo closed this issue 2026-02-16 10:18:14 -05:00
yindo commented 2026-02-16 10:18:15 -05:00
Author
Owner
Copy Link
Copy Source

@AnyangAngus commented on GitHub (Mar 30, 2025):

meet same problem.
/infer and /generate both can send request to tritonserver but the json formats are different,
I change the code in plugin container manually but the url did not change any more. https://github.com/langgenius/dify-official-plugins/blob/main/models/triton_inference_server/models/llm/llm.py#L189

so how can I change code in plugin container?

@AnyangAngus commented on GitHub (Mar 30, 2025): meet same problem. /infer and /generate both can send request to tritonserver but the json formats are different, I change the code in plugin container manually but the url did not change any more. https://github.com/langgenius/dify-official-plugins/blob/main/models/triton_inference_server/models/llm/llm.py#L189 so how can I change code in plugin container?
yindo commented 2026-02-16 10:18:15 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Aug 15, 2025):

Hi, @gonzal40. I'm Dosu, and I'm helping the dify-official-plugins team manage their backlog and am marking this issue as stale.

Issue Summary:

  • You reported a 400 Bad Request error when using the NVIDIA Triton Inference Server model provider plugin in a self-hosted Docker setup.
  • Another user confirmed the same issue and highlighted differences in JSON formats between /infer and /generate endpoints.
  • Attempts to manually update the plugin container code did not take effect.
  • No responses or solutions have been provided by maintainers so far.

Next Steps:

  • Please let me know if this issue is still relevant with the latest version of the dify-official-plugins repository by commenting here.
  • If I do not hear back within 5 days, I will automatically close this issue.

Thank you for your understanding and contribution!

@dosubot[bot] commented on GitHub (Aug 15, 2025): Hi, @gonzal40. I'm [Dosu](https://dosu.dev), and I'm helping the dify-official-plugins team manage their backlog and am marking this issue as stale. **Issue Summary:** - You reported a 400 Bad Request error when using the NVIDIA Triton Inference Server model provider plugin in a self-hosted Docker setup. - Another user confirmed the same issue and highlighted differences in JSON formats between `/infer` and `/generate` endpoints. - Attempts to manually update the plugin container code did not take effect. - No responses or solutions have been provided by maintainers so far. **Next Steps:** - Please let me know if this issue is still relevant with the latest version of the dify-official-plugins repository by commenting here. - If I do not hear back within 5 days, I will automatically close this issue. Thank you for your understanding and contribution!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
bug
bug
dependencies
documentation
enhancement
good first issue
invalid
pull-request
Mirrored from GitHub Pull Request
 question
stale
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify-official-plugins#148
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?