langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-22 04:25:29 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

The max tokens of a model deployed using Xinference can only be set to 2048 #1026

New Issue
Closed
opened 2026-02-21 17:29:33 -05:00 by yindo · 8 comments
yindo commented 2026-02-21 17:29:33 -05:00
Owner
Copy Link
Copy Source

Originally created by @lileiseven on GitHub (Feb 3, 2024).

Self Checks

Dify version

0.5.3

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

  1. Deploy model chatglm-6b-32k using xinference
  2. In the Dify application, I use the model chatglm-6b-32k deployed by xinference

✔️ Expected Behavior

In orchestrate, the parameter max tokens can be set to 32k

❌ Actual Behavior

In orchestrate, the parameter max tokens can only be set to 2048

Originally created by @lileiseven on GitHub (Feb 3, 2024). ### Self Checks - [X] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [X] I confirm that I am using English to file this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). ### Dify version 0.5.3 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce 1. Deploy model chatglm-6b-32k using xinference 2. In the Dify application, I use the model chatglm-6b-32k deployed by xinference ### ✔️ Expected Behavior In orchestrate, the parameter max tokens can be set to 32k ### ❌ Actual Behavior In orchestrate, the parameter max tokens can only be set to 2048
yindo added the 🐞 bug label 2026-02-21 17:29:33 -05:00
yindo closed this issue 2026-02-21 17:29:33 -05:00
yindo commented 2026-02-21 17:29:33 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Feb 3, 2024):

🤖

Hey @lileiseven, great to see you back! Hope you're doing awesome today 😊🚀

Thank you for bringing this to our attention. The max_tokens parameter in the chatglm-6b-32k model configuration specifies the maximum number of tokens that the model can generate for a single output. According to the YAML configuration file for the chatglm-6b-32k model, the max_tokens parameter can be set up to 32,000. Here is the relevant part of the YAML configuration:

parameter_rules:
  - name: max_tokens
    use_template: max_tokens
    required: true
    default: 2000
    min: 1
    max: 32000

If you're experiencing a limitation of 2048 tokens in Orchestrate when using the chatglm-6b-32k model deployed by Xinference, it might be due to a restriction in the Orchestrate or Xinference system, not in the model itself.

I recommend checking the configuration of your Orchestrate and Xinference systems to ensure they are not imposing this limit. If the issue persists, it might be beneficial to check the logs for any error messages or warnings that could provide more insight into the issue.

Please let me know if you need further assistance.

Sources

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

@dosubot[bot] commented on GitHub (Feb 3, 2024): _🤖_ <!-- Greeting --> Hey @lileiseven, great to see you back! Hope you're doing awesome today 😊🚀 <!-- Answer --> Thank you for bringing this to our attention. The `max_tokens` parameter in the chatglm-6b-32k model configuration specifies the maximum number of tokens that the model can generate for a single output. According to the YAML configuration file for the chatglm-6b-32k model, the `max_tokens` parameter can be set up to 32,000. Here is the relevant part of the YAML configuration: ```yaml parameter_rules: - name: max_tokens use_template: max_tokens required: true default: 2000 min: 1 max: 32000 ``` If you're experiencing a limitation of 2048 tokens in Orchestrate when using the chatglm-6b-32k model deployed by Xinference, it might be due to a restriction in the Orchestrate or Xinference system, not in the model itself. I recommend checking the configuration of your Orchestrate and Xinference systems to ensure they are not imposing this limit. If the issue persists, it might be beneficial to check the logs for any error messages or warnings that could provide more insight into the issue. Please let me know if you need further assistance. ## Sources - [api/core/model_runtime/model_providers/chatglm/llm/chatglm2-6b-32k.yaml](https://github.com/langgenius/dify/blob/main/api/core/model_runtime/model_providers/chatglm/llm/chatglm2-6b-32k.yaml) - [api/core/model_runtime/model_providers/chatglm/llm/chatglm3-6b-32k.yaml](https://github.com/langgenius/dify/blob/main/api/core/model_runtime/model_providers/chatglm/llm/chatglm3-6b-32k.yaml) --- This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant. If you want to continue the conversation, start your reply with @dosu-bot.
yindo commented 2026-02-21 17:29:33 -05:00
Author
Owner
Copy Link
Copy Source

@Yeuoly commented on GitHub (Feb 3, 2024):

This issue has already been fixed in #2336. BTW, max_tokens is different from context_size, max_tokens specifies a maximum limit for the maximum current output length, not a context limit.

@Yeuoly commented on GitHub (Feb 3, 2024): This issue has already been fixed in #2336. BTW, max_tokens is different from context_size, max_tokens specifies a maximum limit for the maximum current output length, not a context limit.
yindo commented 2026-02-21 17:29:33 -05:00
Author
Owner
Copy Link
Copy Source

@lileiseven commented on GitHub (Feb 3, 2024):

I have upgraded to dify 0.5.3, but the max token can still only be set to 2048.

@lileiseven commented on GitHub (Feb 3, 2024): I have upgraded to dify 0.5.3, but the max token can still only be set to 2048.
yindo commented 2026-02-21 17:29:33 -05:00
Author
Owner
Copy Link
Copy Source

@crazywoola commented on GitHub (Feb 3, 2024):

I have upgraded to dify 0.5.3, but the max token can still only be set to 2048.

BTW, max_tokens is different from context_size, max_tokens specifies a maximum limit for the maximum current output length, not a context limit.

It means output tokens is 2k, the whole context window is 32k

@crazywoola commented on GitHub (Feb 3, 2024): > I have upgraded to dify 0.5.3, but the max token can still only be set to 2048. > BTW, max_tokens is different from context_size, max_tokens specifies a maximum limit for the maximum current output length, not a context limit. It means output tokens is 2k, the whole context window is 32k
yindo commented 2026-02-21 17:29:34 -05:00
Author
Owner
Copy Link
Copy Source

@littlewwwhite commented on GitHub (Feb 23, 2024):

I have upgraded to dify 0.5.3, but the max token can still only be set to 2048.

BTW, max_tokens is different from context_size, max_tokens specifies a maximum limit for the maximum current output length, not a context limit.

It means output tokens is 2k, the whole context window is 32k

How could I promote the output token 2k to longer? 4k 8k?

@littlewwwhite commented on GitHub (Feb 23, 2024): > > I have upgraded to dify 0.5.3, but the max token can still only be set to 2048. > > > BTW, max_tokens is different from context_size, max_tokens specifies a maximum limit for the maximum current output length, not a context limit. > > It means output tokens is 2k, the whole context window is 32k How could I promote the output token 2k to longer? 4k 8k?
yindo commented 2026-02-21 17:29:34 -05:00
Author
Owner
Copy Link
Copy Source

@crazywoola commented on GitHub (Feb 23, 2024):

@lileiseven Something like this.
image

@crazywoola commented on GitHub (Feb 23, 2024): @lileiseven Something like this. <img width="694" alt="image" src="https://github.com/langgenius/dify/assets/100913391/35e85ce5-9815-4734-ba17-b19221590559">
yindo commented 2026-02-21 17:29:34 -05:00
Author
Owner
Copy Link
Copy Source

@littlewwwhite commented on GitHub (Feb 25, 2024):

@lileiseven Something like this. image

Thanks. Maybe that works on ollama, but the xinference don't have such options, I can only set less than 2048 in dify's interface, is that a problem about xinference?

@littlewwwhite commented on GitHub (Feb 25, 2024): > @lileiseven Something like this. <img alt="image" width="694" src="https://private-user-images.githubusercontent.com/100913391/307484410-35e85ce5-9815-4734-ba17-b19221590559.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MDg4NDM2MjIsIm5iZiI6MTcwODg0MzMyMiwicGF0aCI6Ii8xMDA5MTMzOTEvMzA3NDg0NDEwLTM1ZTg1Y2U1LTk4MTUtNDczNC1iYTE3LWIxOTIyMTU5MDU1OS5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwMjI1JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDIyNVQwNjQyMDJaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT0xYjg5OWQxZDMzMGU3NGExZDk4ZDg0ZjczODdkZjdkOTc2MGIwOTVlZWYzODY3N2FlZTg2M2VkM2NlOTVhODNmJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.PCoDRKvMaTbFnf8mpUJxoDWUub6f9zfF7OuEZOQcs3I"> Thanks. Maybe that works on ollama, but the xinference don't have such options, I can only set less than 2048 in dify's interface, is that a problem about xinference?
yindo commented 2026-02-21 17:29:34 -05:00
Author
Owner
Copy Link
Copy Source

@Yeuoly commented on GitHub (Feb 25, 2024):

@lileiseven Something like this. image

Thanks. Maybe that works on ollama, but the xinference don't have such options, I can only set less than 2048 in dify's interface, is that a problem about xinference?

Yes, there is not a context size options in xinference, however, dify will try to fetch it from xinference directly, you can visit http://your_xinference_endpoint/v1/models/model_uid to see the context size of your model.

@Yeuoly commented on GitHub (Feb 25, 2024): > > @lileiseven Something like this. <img alt="image" width="694" src="https://private-user-images.githubusercontent.com/100913391/307484410-35e85ce5-9815-4734-ba17-b19221590559.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MDg4NDM2MjIsIm5iZiI6MTcwODg0MzMyMiwicGF0aCI6Ii8xMDA5MTMzOTEvMzA3NDg0NDEwLTM1ZTg1Y2U1LTk4MTUtNDczNC1iYTE3LWIxOTIyMTU5MDU1OS5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwMjI1JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDIyNVQwNjQyMDJaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT0xYjg5OWQxZDMzMGU3NGExZDk4ZDg0ZjczODdkZjdkOTc2MGIwOTVlZWYzODY3N2FlZTg2M2VkM2NlOTVhODNmJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.PCoDRKvMaTbFnf8mpUJxoDWUub6f9zfF7OuEZOQcs3I"> > > Thanks. Maybe that works on ollama, but the xinference don't have such options, I can only set less than 2048 in dify's interface, is that a problem about xinference? Yes, there is not a context size options in xinference, however, dify will try to fetch it from xinference directly, you can visit http://your_xinference_endpoint/v1/models/model_uid to see the context size of your model.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#1026
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?