langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-24 13:06:09 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Ollama based models are "not" providing full tokens (stops around 150 or so) on self-hosted docker(dify) based environment #1058

New Issue
Closed
opened 2026-02-21 17:29:45 -05:00 by yindo · 12 comments
yindo commented 2026-02-21 17:29:45 -05:00
Owner
Copy Link
Copy Source

Originally created by @ashmanpan on GitHub (Feb 17, 2024).

Originally assigned to: @takatost, @crazywoola on GitHub.

Self Checks

Dify version

0.5.4

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

  1. run models on Ollama , locally on Mac M3
  2. Create app use Ollama based model, I used llama2
  3. use 2 model debugging view, use chat-gpt-4 as 2nd model to compare
  4. set max tokens on both models as 4096
  5. ask questions which will relatively long answers, you will see ChatGPT giving full details of the question in the answer but Ollama based locale model stooped after 150 or so tokens

✔️ Expected Behavior

Ollama based model also should give full long response like ChatGPT , till it reaches to max token 4096

❌ Actual Behavior

Ollama based model using dify is stopping in between the response after 150 or so tokens .

I tried Ollama same model on the terminal with the same questions and i can see full response from the model. it's not Ollama issues.

Originally created by @ashmanpan on GitHub (Feb 17, 2024). Originally assigned to: @takatost, @crazywoola on GitHub. ### Self Checks - [X] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [X] I confirm that I am using English to file this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). ### Dify version 0.5.4 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce 1. run models on Ollama , locally on Mac M3 2. Create app use Ollama based model, I used llama2 3. use 2 model debugging view, use chat-gpt-4 as 2nd model to compare 4. set max tokens on both models as 4096 5. ask questions which will relatively long answers, you will see ChatGPT giving full details of the question in the answer but Ollama based locale model stooped after 150 or so tokens ### ✔️ Expected Behavior Ollama based model also should give full long response like ChatGPT , till it reaches to max token 4096 ### ❌ Actual Behavior Ollama based model using dify is stopping in between the response after 150 or so tokens . I tried Ollama same model on the terminal with the same questions and i can see full response from the model. it's not Ollama issues.
yindo added the 🐞 bug label 2026-02-21 17:29:45 -05:00
yindo closed this issue 2026-02-21 17:29:45 -05:00
yindo commented 2026-02-21 17:29:45 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Feb 17, 2024):

🤖

Hey there, @ashmanpan! 👋 I'm Dosu, your friendly neighborhood bot here to help you tackle bugs, answer your questions, and guide you on contributing while we wait for a human maintainer to swing by. I'm diving into your issue with the Ollama models and will get back to you with a full answer shortly!

@dosubot[bot] commented on GitHub (Feb 17, 2024): _🤖_ <!-- Greeting --> Hey there, @ashmanpan! 👋 I'm [Dosu](https://dosu.dev), your friendly neighborhood bot here to help you tackle bugs, answer your questions, and guide you on contributing while we wait for a human maintainer to swing by. I'm diving into your issue with the Ollama models and will get back to you with a full answer shortly!
yindo commented 2026-02-21 17:29:45 -05:00
Author
Owner
Copy Link
Copy Source

@ashmanpan commented on GitHub (Feb 17, 2024):

detaiils and snpshot is avilable here - https://discord.com/channels/1082486657678311454/1206116751801327616

@ashmanpan commented on GitHub (Feb 17, 2024): detaiils and snpshot is avilable here - https://discord.com/channels/1082486657678311454/1206116751801327616
yindo commented 2026-02-21 17:29:46 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Feb 17, 2024):

🤖

Based on the information you've provided, it seems like the issue you're experiencing might be related to the 'num_predict' parameter in the Ollama model configuration. This parameter controls the maximum number of tokens to predict when generating text. Even though you've set the maximum token limit to 4096, the 'num_predict' parameter might be limiting the number of tokens generated by the model.

In the Dify version 0.5.4 codebase, the default 'num_predict' parameter value for Ollama based models is set to 128. Here's the relevant code snippet:

ParameterRule(
    name='num_predict',
    use_template='max_tokens',
    label=I18nObject(en_US="Num Predict"),
    type=ParameterType.INT,
    help=I18nObject(en_US="Maximum number of tokens to predict when generating text. "
                          "(Default: 128, -1 = infinite generation, -2 = fill context)"),
    default=128,
    min=-2,
    max=int(credentials.get('max_tokens', 4096)),
),

You can find this in the api/core/model_runtime/model_providers/ollama/llm/llm.py file.

If you want the model to generate more tokens, you might need to adjust the 'num_predict' parameter to a higher value or set it to -1 for infinite generation.

Unfortunately, I can't view the additional details and snapshots you've provided at the link as I'm unable to access URLs. If you could provide more information or any error messages you're encountering directly in this thread, it would be helpful in further diagnosing the issue.

Please note that you're using Dify version 0.5.4, and the latest version is 0.5.5. It might be worth updating to the latest version to see if the issue persists, as it could have been addressed in the latest release.

I hope this helps! If you have any more questions or need further clarification, feel free to ask.

Sources

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

@dosubot[bot] commented on GitHub (Feb 17, 2024): _🤖_ <!-- Answer --> Based on the information you've provided, it seems like the issue you're experiencing might be related to the 'num_predict' parameter in the Ollama model configuration. This parameter controls the maximum number of tokens to predict when generating text. Even though you've set the maximum token limit to 4096, the 'num_predict' parameter might be limiting the number of tokens generated by the model. In the Dify version 0.5.4 codebase, the default 'num_predict' parameter value for Ollama based models is set to 128. Here's the relevant code snippet: ```py ParameterRule( name='num_predict', use_template='max_tokens', label=I18nObject(en_US="Num Predict"), type=ParameterType.INT, help=I18nObject(en_US="Maximum number of tokens to predict when generating text. " "(Default: 128, -1 = infinite generation, -2 = fill context)"), default=128, min=-2, max=int(credentials.get('max_tokens', 4096)), ), ``` You can find this in the [api/core/model_runtime/model_providers/ollama/llm/llm.py](https://github.com/langgenius/dify/blob/main/api/core/model_runtime/model_providers/ollama/llm/llm.py) file. If you want the model to generate more tokens, you might need to adjust the 'num_predict' parameter to a higher value or set it to -1 for infinite generation. Unfortunately, I can't view the additional details and snapshots you've provided at the link as I'm unable to access URLs. If you could provide more information or any error messages you're encountering directly in this thread, it would be helpful in further diagnosing the issue. Please note that you're using Dify version 0.5.4, and the latest version is 0.5.5. It might be worth updating to the latest version to see if the issue persists, as it could have been addressed in the latest release. I hope this helps! If you have any more questions or need further clarification, feel free to ask. ## Sources - [api/core/model_runtime/model_providers/ollama/llm/llm.py](https://github.com/langgenius/dify/blob/main/api/core/model_runtime/model_providers/ollama/llm/llm.py) <!-- Dosu Comment Footer --> <details open> <summary> #### About Dosu </summary> This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant. If you want to continue the conversation, start your reply with @dosu-bot. </details>
yindo commented 2026-02-21 17:29:46 -05:00
Author
Owner
Copy Link
Copy Source

@takatost commented on GitHub (Feb 17, 2024):

You can try adjusting the Num Predict model parameter to the appropriate tokens size for generating texts.

@takatost commented on GitHub (Feb 17, 2024): You can try adjusting the `Num Predict` model parameter to the appropriate tokens size for generating texts.
yindo commented 2026-02-21 17:29:46 -05:00
Author
Owner
Copy Link
Copy Source

@ashmanpan commented on GitHub (Feb 18, 2024):

I have already deployed dify on docker containers, which specific container this file has, how can I edit, or what's the way to change this?

Other question, why this kind low number as default , this is local llm no one is going to charge because of the large no of tokens so why ?

@ashmanpan commented on GitHub (Feb 18, 2024): I have already deployed dify on docker containers, which specific container this file has, how can I edit, or what's the way to change this? Other question, why this kind low number as default , this is local llm no one is going to charge because of the large no of tokens so why ?
yindo commented 2026-02-21 17:29:46 -05:00
Author
Owner
Copy Link
Copy Source

@ashmanpan commented on GitHub (Feb 18, 2024):

image image image image
@ashmanpan commented on GitHub (Feb 18, 2024): <img width="1267" alt="image" src="https://github.com/langgenius/dify/assets/7510337/6cbeb377-3d2b-4e68-b931-e67d297881f0"> <img width="637" alt="image" src="https://github.com/langgenius/dify/assets/7510337/90c6de73-8b3e-44cb-b97c-aee8c41be95b"> <img width="1222" alt="image" src="https://github.com/langgenius/dify/assets/7510337/5b245e39-9f9f-4a67-8e21-f25b694d6338"> <img width="1230" alt="image" src="https://github.com/langgenius/dify/assets/7510337/dc21d222-be36-4720-876f-60c0da96bb04">
yindo commented 2026-02-21 17:29:46 -05:00
Author
Owner
Copy Link
Copy Source

@ashmanpan commented on GitHub (Feb 18, 2024):

below snapshot are after upgarde to 0.5.5

image
@ashmanpan commented on GitHub (Feb 18, 2024): below snapshot are after upgarde to 0.5.5 <img width="1927" alt="image" src="https://github.com/langgenius/dify/assets/7510337/67d9ffa1-6131-4816-bb9f-6985d4480265">
yindo commented 2026-02-21 17:29:46 -05:00
Author
Owner
Copy Link
Copy Source

@ashmanpan commented on GitHub (Feb 18, 2024):

image

fist 2 models are running locally on ollama - you can clearly see the diffrence.

@ashmanpan commented on GitHub (Feb 18, 2024): <img width="1859" alt="image" src="https://github.com/langgenius/dify/assets/7510337/c750ce7d-93dd-47d5-b987-a862e2b65795"> fist 2 models are running locally on ollama - you can clearly see the diffrence.
yindo commented 2026-02-21 17:29:46 -05:00
Author
Owner
Copy Link
Copy Source

@takatost commented on GitHub (Feb 18, 2024):

image This number is set according to the default values provided in the Ollama API document, you can change this value here.
@takatost commented on GitHub (Feb 18, 2024): <img width="597" alt="image" src="https://github.com/langgenius/dify/assets/5485478/95059438-9120-4a49-adac-472f83fe672f"> This number is set according to the default values provided in the Ollama API document, you can change this value here.
yindo commented 2026-02-21 17:29:46 -05:00
Author
Owner
Copy Link
Copy Source

@ashmanpan commented on GitHub (Feb 18, 2024):

Issue resolved . i will close soon after some testing

On Mon, Feb 19, 2024 at 10:47 AM takatost @.***> wrote:

image.png (view on web)
https://github.com/langgenius/dify/assets/5485478/95059438-9120-4a49-adac-472f83fe672f
This number is set according to the default values provided in the Ollama
API document, you can change this value here.

—
Reply to this email directly, view it on GitHub
https://github.com/langgenius/dify/issues/2461#issuecomment-1951601441,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABZJSQM4X2L4FDPHUSAT4Q3YUK4KTAVCNFSM6AAAAABDNZQ7HGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNJRGYYDCNBUGE
.
You are receiving this because you were mentioned.Message ID:
@.***>

@ashmanpan commented on GitHub (Feb 18, 2024): Issue resolved . i will close soon after some testing On Mon, Feb 19, 2024 at 10:47 AM takatost ***@***.***> wrote: > image.png (view on web) > <https://github.com/langgenius/dify/assets/5485478/95059438-9120-4a49-adac-472f83fe672f> > This number is set according to the default values provided in the Ollama > API document, you can change this value here. > > — > Reply to this email directly, view it on GitHub > <https://github.com/langgenius/dify/issues/2461#issuecomment-1951601441>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/ABZJSQM4X2L4FDPHUSAT4Q3YUK4KTAVCNFSM6AAAAABDNZQ7HGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNJRGYYDCNBUGE> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> >
yindo commented 2026-02-21 17:29:46 -05:00
Author
Owner
Copy Link
Copy Source

@akhilmadhumenon commented on GitHub (Apr 10, 2024):

When we tend to set a limit on num_predict, the sentences get cut off mid way, and the model does not complete the sentence. How can we ensure that it will complete the sentence whilst maintaining the maximum token limit?

@akhilmadhumenon commented on GitHub (Apr 10, 2024): When we tend to set a limit on num_predict, the sentences get cut off mid way, and the model does not complete the sentence. How can we ensure that it will complete the sentence whilst maintaining the maximum token limit?
yindo commented 2026-02-21 17:29:46 -05:00
Author
Owner
Copy Link
Copy Source

@sivertheisholt commented on GitHub (May 4, 2024):

When we tend to set a limit on num_predict, the sentences get cut off mid way, and the model does not complete the sentence. How can we ensure that it will complete the sentence whilst maintaining the maximum token limit?

Did you figure out this? I have the same problem, cutting off mid sentence when setting the num_predict to 64.

@sivertheisholt commented on GitHub (May 4, 2024): > When we tend to set a limit on num_predict, the sentences get cut off mid way, and the model does not complete the sentence. How can we ensure that it will complete the sentence whilst maintaining the maximum token limit? Did you figure out this? I have the same problem, cutting off mid sentence when setting the num_predict to 64.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#1058
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?