langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 12:05:30 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Ollama Num GPU option not consistent with allowed values #3819

New Issue
Closed
opened 2026-02-21 18:02:24 -05:00 by yindo · 0 comments
yindo commented 2026-02-21 18:02:24 -05:00
Owner
Copy Link
Copy Source

Originally created by @rothnic on GitHub (May 28, 2024).

Originally assigned to: @crazywoola on GitHub.

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • Please do not modify this template :) and fill in all the required fields.

Dify version

0.6.8

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

Setup:

  • Setup ollama (did this on a windows machine) and pulled llama3 and phi3 and tested models locally that running prompts with ollama run or through the api without any additional options passed that the model was fully loaded on the gpu
    • You can check if it was loaded by inspecting the ollama logs and seeing how many layers were loaded, which was 33/33 layers in my case
  • Add ollama model to dify (llama3 and phi3 used in my case)
  • Create a simple chatbot using the ollama model

Issue:

  • At this point, I'm not sure why it felt like i was having trouble, but I though that I left the options unset and it wasn't being fully loaded on the gpu and was running slowly (i might have jumped straight to setting Num GPU to 1)
  • So, i looked at the options and saw num_gpu, which seemed to suggest that setting to 1 would mean use the gpu or to control how many gpus are used
  • However, after setting to 1, this is when i looked at the logs and found that only 1/33 layers were being loaded on the gpu
  • Next, i tried setting that value to 33, but the user interface doesn't support that
  • Next, i disabled that option in the dify chatbot for the llama3 model, clicked publish, clicked restart, then things started working as expected

✔️ Expected Behavior

  1. I'd expect for the option to support the range of values that ollama allows. It currently is allowing values between 0 and 1. Ollama's code suggests it should only be allowed as integer values and should be the number of layers, which suggests any integer value up to the number of layers in the model (which might not be known, so should not be limited to any max integer value)
  2. The user interface should only support integers. The user interface in Dify seems to be configured to support that option as an integer, but the input component seems to not limit to only integer values.
image

❌ Actual Behavior

  • Num GPU doesn't provide as much explanation as it could, which is understandable given that ollama doesn't have an api docs page that describes this completely. The note about use on macs kind of makes things confusing given that the tooltip and the name of the option sounds like setting to 1 enables using 1 gpu.
  • The input allows setting non integer values
  • The input doesn't allow setting the number of layers to send to the gpu, since it is limited to a max of 1
Originally created by @rothnic on GitHub (May 28, 2024). Originally assigned to: @crazywoola on GitHub. ### Self Checks - [X] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [X] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [X] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [X] Please do not modify this template :) and fill in all the required fields. ### Dify version 0.6.8 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce Setup: * Setup ollama (did this on a windows machine) and pulled llama3 and phi3 and tested models locally that running prompts with ollama run <modelname> or through the api without any additional options passed that the model was fully loaded on the gpu * You can check if it was loaded by inspecting the ollama logs and seeing how many layers were loaded, which was 33/33 layers in my case * Add ollama model to dify (llama3 and phi3 used in my case) * Create a simple chatbot using the ollama model Issue: * At this point, I'm not sure why it felt like i was having trouble, but I though that I left the options unset and it wasn't being fully loaded on the gpu and was running slowly (i might have jumped straight to setting Num GPU to 1) * So, i looked at the options and saw `num_gpu`, which seemed to suggest that setting to 1 would mean use the gpu or to control how many gpus are used * However, after setting to 1, this is when i looked at the logs and found that only 1/33 layers were being loaded on the gpu * Next, i tried setting that value to 33, but the user interface doesn't support that * Next, i disabled that option in the dify chatbot for the llama3 model, clicked publish, clicked restart, then things started working as expected ### ✔️ Expected Behavior 1. I'd expect for the option to support the range of values that ollama allows. It currently is allowing values between 0 and 1. [Ollama's code ](https://github.com/ollama/ollama/blob/9db0996ed458e64ee6814d3c276bd5cb57c208ca/cmd/interactive.go#L172) suggests it should only be allowed as integer values and should be the number of layers, which suggests any integer value up to the number of layers in the model (which might not be known, so should not be limited to any max integer value) 2. The user interface should only support integers. The user interface in Dify [seems to be configured to support that option as an integer](https://github.com/langgenius/dify/blob/332baca538feb9e50b99e0bca7bcb2b816bdb66c/api/core/model_runtime/model_providers/ollama/llm/llm.py#L535), but the input component seems to not limit to only integer values. <img width="512" alt="image" src="https://github.com/langgenius/dify/assets/452052/501a66a2-672f-4ae5-b80e-45b554e95777"> ### ❌ Actual Behavior * Num GPU doesn't provide as much explanation as it could, which is understandable given that ollama doesn't have an api docs page that describes this completely. The note about use on macs kind of makes things confusing given that the tooltip and the name of the option sounds like setting to 1 enables using 1 gpu. * The input allows setting non integer values * The input doesn't allow setting the number of layers to send to the gpu, since it is [limited to a max of 1](https://github.com/langgenius/dify/blob/332baca538feb9e50b99e0bca7bcb2b816bdb66c/api/core/model_runtime/model_providers/ollama/llm/llm.py#L535)
yindo added the 🐞 bug label 2026-02-21 18:02:24 -05:00
yindo closed this issue 2026-02-21 18:02:24 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#3819
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?