langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-22 12:36:14 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Can not load models from OpenLLM #706

New Issue
Closed
opened 2026-02-21 17:28:06 -05:00 by yindo · 5 comments
yindo commented 2026-02-21 17:28:06 -05:00
Owner
Copy Link
Copy Source

Originally created by @hahchenchen on GitHub (Nov 16, 2023).

Originally assigned to: @takatost on GitHub.

Dify version

0.3.30

Cloud or Self Hosted

Self Hosted

Steps to reproduce

按照教程上的，使用openllm启动模型：
openllm start opt --model_id facebook/opt-125m -p 3333

然后在dify上添加。报错
image

✔️ Expected Behavior

成功添加

❌ Actual Behavior

{"code": "invalid_param", "message": "'responses'", "status": 400}

Originally created by @hahchenchen on GitHub (Nov 16, 2023). Originally assigned to: @takatost on GitHub. ### Dify version 0.3.30 ### Cloud or Self Hosted Self Hosted ### Steps to reproduce 按照教程上的，使用openllm启动模型： openllm start opt --model_id facebook/opt-125m -p 3333 然后在dify上添加。报错 ![image](https://github.com/langgenius/dify/assets/22194004/f04e72b0-a76a-46d7-8c2b-6e177f988892) ### ✔️ Expected Behavior 成功添加 ### ❌ Actual Behavior {"code": "invalid_param", "message": "'responses'", "status": 400}
yindo added the 🐞 bug label 2026-02-21 17:28:06 -05:00
yindo closed this issue 2026-02-21 17:28:06 -05:00
yindo commented 2026-02-21 17:28:06 -05:00
Author
Owner
Copy Link
Copy Source

@hahchenchen commented on GitHub (Nov 16, 2023):

@crazywoola hi，can you help that？

@hahchenchen commented on GitHub (Nov 16, 2023): @crazywoola hi，can you help that？
yindo commented 2026-02-21 17:28:06 -05:00
Author
Owner
Copy Link
Copy Source

@crazywoola commented on GitHub (Nov 16, 2023):

Can you post the full parameters you filled in the form?

@crazywoola commented on GitHub (Nov 16, 2023): Can you post the full parameters you filled in the form?
yindo commented 2026-02-21 17:28:07 -05:00
Author
Owner
Copy Link
Copy Source

@hahchenchen commented on GitHub (Nov 16, 2023):

image

@hahchenchen commented on GitHub (Nov 16, 2023): ![image](https://github.com/langgenius/dify/assets/22194004/e2c2aad8-34cf-40f6-9a76-8ca2961aa0d7)
yindo commented 2026-02-21 17:28:07 -05:00
Author
Owner
Copy Link
Copy Source

@crazywoola commented on GitHub (Nov 16, 2023):

image.
I see, we will fix this later.

vLLM is not available. Note that PyTorch backend is not as performant as vLLM and you should always consider using vLLM for production.
To disable these warnings, set 'OPENLLM_DISABLE_WARNING=True'
"auto" and "half" are not supported on CPU. OpenLLM will default fallback to "float32".
🚀Tip: run 'openllm build facebook/opt-125m --serialization legacy' to create a BentoLLM for 'facebook/opt-125m'
2023-11-16T21:49:39+0800 [INFO] [cli] Prometheus metrics for HTTP BentoServer from "_service:svc" can be accessed at http://localhost:3333/metrics.
2023-11-16T21:49:39+0800 [INFO] [cli] Starting production HTTP BentoServer from "_service:svc" listening on http://0.0.0.0:3333 (Press CTRL+C to quit)
2023-11-16T21:50:00+0800 [INFO] [runner:llm-opt-runner:1] _ (scheme=http,method=POST,path=http://127.0.0.1:8000/generate_iterator,type=application/octet-stream,length=1695) (status=404,type=text/plain; charset=utf-8,length=9) 0.573ms (trace=4e046c3035158d9b5b0de5ea5e78494b,span=562a85432a93206e,sampled=1,service.name=llm-opt-runner)
2023-11-16T21:50:00+0800 [ERROR] [api_server:llm-opt-service:3] Exception on /v1/generate [POST] (trace=4e046c3035158d9b5b0de5ea5e78494b,span=fe1c6657c44d3831,sampled=1,service.name=llm-opt-service)
Traceback (most recent call last):
File "/Users/minibanana/anaconda3/lib/python3.10/site-packages/bentoml/_internal/server/http_app.py", line 341, in api_func
output = await api.func(*args)
File "/Users/minibanana/anaconda3/lib/python3.10/site-packages/openllm/_service.py", line 37, in generate_v1
return (await llm.generate(**llm_model_class(**input_dict).model_dump())).model_dump()
File "/Users/minibanana/anaconda3/lib/python3.10/site-packages/openllm/_llm.py", line 508, in generate
raise RuntimeError('No result is returned.')
RuntimeError: No result is returned.
2023-11-16T21:50:00+0800 [INFO] [api_server:llm-opt-service:3] 192.168.31.241:56201 (scheme=http,method=POST,path=/v1/generate,type=application/json,length=56) (status=500,type=application/json,length=110) 124.968ms (trace=4e046c3035158d9b5b0de5ea5e78494b,span=fe1c6657c44d3831,sampled=1,service.name=llm-opt-service)
2023-11-16T21:50:01+0800 [INFO] [runner:llm-opt-runner:1] _ (scheme=http,method=POST,path=http://127.0.0.1:8000/generate_iterator,type=application/octet-stream,length=1695) (status=404,type=text/plain; charset=utf-8,length=9) 0.343ms (trace=b62f935d79de28397801e6d5fc0c8f1c,span=5a67082cc9f2cc4a,sampled=1,service.name=llm-opt-runner)
2023-11-16T21:50:01+0800 [ERROR] [api_server:llm-opt-service:6] Exception on /v1/generate [POST] (trace=b62f935d79de28397801e6d5fc0c8f1c,span=332012a07c9ccde7,sampled=1,service.name=llm-opt-service)
Traceback (most recent call last):
File "/Users/minibanana/anaconda3/lib/python3.10/site-packages/bentoml/_internal/server/http_app.py", line 341, in api_func
output = await api.func(*args)
File "/Users/minibanana/anaconda3/lib/python3.10/site-packages/openllm/_service.py", line 37, in generate_v1
return (await llm.generate(**llm_model_class(**input_dict).model_dump())).model_dump()
File "/Users/minibanana/anaconda3/lib/python3.10/site-packages/openllm/_llm.py", line 508, in generate
raise RuntimeError('No result is returned.')
RuntimeError: No result is returned.
2023-11-16T21:50:01+0800 [INFO] [api_server:llm-opt-service:6] 192.168.31.241:56203 (scheme=http,method=POST,path=/v1/generate,type=application/json,length=56) (status=500,type=application/json,length=110) 87.994ms (trace=b62f935d79de28397801e6d5fc0c8f1c,span=332012a07c9ccde7,sampled=1,service.name=llm-opt-service)

@takatost Please take a look at this.

@crazywoola commented on GitHub (Nov 16, 2023): <img width="676" alt="image" src="https://github.com/langgenius/dify/assets/100913391/fad33329-4017-466d-a444-8ef3475e5afd">. I see, we will fix this later. ``` vLLM is not available. Note that PyTorch backend is not as performant as vLLM and you should always consider using vLLM for production. To disable these warnings, set 'OPENLLM_DISABLE_WARNING=True' "auto" and "half" are not supported on CPU. OpenLLM will default fallback to "float32". 🚀Tip: run 'openllm build facebook/opt-125m --serialization legacy' to create a BentoLLM for 'facebook/opt-125m' 2023-11-16T21:49:39+0800 [INFO] [cli] Prometheus metrics for HTTP BentoServer from "_service:svc" can be accessed at http://localhost:3333/metrics. 2023-11-16T21:49:39+0800 [INFO] [cli] Starting production HTTP BentoServer from "_service:svc" listening on http://0.0.0.0:3333 (Press CTRL+C to quit) 2023-11-16T21:50:00+0800 [INFO] [runner:llm-opt-runner:1] _ (scheme=http,method=POST,path=http://127.0.0.1:8000/generate_iterator,type=application/octet-stream,length=1695) (status=404,type=text/plain; charset=utf-8,length=9) 0.573ms (trace=4e046c3035158d9b5b0de5ea5e78494b,span=562a85432a93206e,sampled=1,service.name=llm-opt-runner) 2023-11-16T21:50:00+0800 [ERROR] [api_server:llm-opt-service:3] Exception on /v1/generate [POST] (trace=4e046c3035158d9b5b0de5ea5e78494b,span=fe1c6657c44d3831,sampled=1,service.name=llm-opt-service) Traceback (most recent call last): File "/Users/minibanana/anaconda3/lib/python3.10/site-packages/bentoml/_internal/server/http_app.py", line 341, in api_func output = await api.func(*args) File "/Users/minibanana/anaconda3/lib/python3.10/site-packages/openllm/_service.py", line 37, in generate_v1 return (await llm.generate(**llm_model_class(**input_dict).model_dump())).model_dump() File "/Users/minibanana/anaconda3/lib/python3.10/site-packages/openllm/_llm.py", line 508, in generate raise RuntimeError('No result is returned.') RuntimeError: No result is returned. 2023-11-16T21:50:00+0800 [INFO] [api_server:llm-opt-service:3] 192.168.31.241:56201 (scheme=http,method=POST,path=/v1/generate,type=application/json,length=56) (status=500,type=application/json,length=110) 124.968ms (trace=4e046c3035158d9b5b0de5ea5e78494b,span=fe1c6657c44d3831,sampled=1,service.name=llm-opt-service) 2023-11-16T21:50:01+0800 [INFO] [runner:llm-opt-runner:1] _ (scheme=http,method=POST,path=http://127.0.0.1:8000/generate_iterator,type=application/octet-stream,length=1695) (status=404,type=text/plain; charset=utf-8,length=9) 0.343ms (trace=b62f935d79de28397801e6d5fc0c8f1c,span=5a67082cc9f2cc4a,sampled=1,service.name=llm-opt-runner) 2023-11-16T21:50:01+0800 [ERROR] [api_server:llm-opt-service:6] Exception on /v1/generate [POST] (trace=b62f935d79de28397801e6d5fc0c8f1c,span=332012a07c9ccde7,sampled=1,service.name=llm-opt-service) Traceback (most recent call last): File "/Users/minibanana/anaconda3/lib/python3.10/site-packages/bentoml/_internal/server/http_app.py", line 341, in api_func output = await api.func(*args) File "/Users/minibanana/anaconda3/lib/python3.10/site-packages/openllm/_service.py", line 37, in generate_v1 return (await llm.generate(**llm_model_class(**input_dict).model_dump())).model_dump() File "/Users/minibanana/anaconda3/lib/python3.10/site-packages/openllm/_llm.py", line 508, in generate raise RuntimeError('No result is returned.') RuntimeError: No result is returned. 2023-11-16T21:50:01+0800 [INFO] [api_server:llm-opt-service:6] 192.168.31.241:56203 (scheme=http,method=POST,path=/v1/generate,type=application/json,length=56) (status=500,type=application/json,length=110) 87.994ms (trace=b62f935d79de28397801e6d5fc0c8f1c,span=332012a07c9ccde7,sampled=1,service.name=llm-opt-service) ``` @takatost Please take a look at this.
yindo commented 2026-02-21 17:28:07 -05:00
Author
Owner
Copy Link
Copy Source

@takatost commented on GitHub (Nov 17, 2023):

It could be due to the incompatibility between the latest version of OpenLLM and the previous API. I tested the version we integrated at that time, 0.3.6, and it was working fine. However, the new version is throwing an error. Let's see how we can fix it.

@takatost commented on GitHub (Nov 17, 2023): It could be due to the incompatibility between the latest version of OpenLLM and the previous API. I tested the version we integrated at that time, 0.3.6, and it was working fine. However, the new version is throwing an error. Let's see how we can fix it.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#706
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?