langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 20:15:31 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Remove the 2048 token limit on API calls for longer generation capability #2784

New Issue
Closed
opened 2026-02-21 17:49:57 -05:00 by yindo · 0 comments
yindo commented 2026-02-21 17:49:57 -05:00
Owner
Copy Link
Copy Source

Originally created by @LateralusAtCrezam on GitHub (Apr 29, 2024).

Self Checks

  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • Pleas do not modify this template :) and fill in all the required fields.

1. Is this request related to a challenge you're experiencing?

This is a followup issue from Issue #3672 but applicable globally, agnostic of model provider. With the fixes suggested, I was able to extract up to 9-10,000 tokens per response on xinference. The issue still exists on AWS Bedrock, Azure OpenAI and Google Gemini, however.

2. Describe the feature you'd like to see

For research purposes, I would like to see this limitation removed globally or at least on my particular install, where the max tokens for ANY response are not capped to 2,048 tokens by default, and the limit is instead configurable depending upon which model I'm using. For now, xinference with OpenHermes 2.5 is my control, which was able to achieve 7,000+ tokens with stability. My first test case is that I'm using AWS Bedrock with Llama 2 70B Chat, which I know is capable of more than 2,048 tokens per response as the base model. For my second point of comparison, I'm using Google Gemini through the Model Providers. I will be incorporating more and more test subjects as research progresses.

3. How will this feature improve your workflow or experience?

This allows me to research entropy and generation instability of LLMs with generation length as the north-star and develop strategies to counteract, or balance out instability in responses approaching 7,000+ tokens.

4. Additional context or comments

No response

5. Can you help us with this feature?

  • I am interested in contributing to this feature.
Originally created by @LateralusAtCrezam on GitHub (Apr 29, 2024). ### Self Checks - [X] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [X] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [X] Pleas do not modify this template :) and fill in all the required fields. ### 1. Is this request related to a challenge you're experiencing? This is a followup issue from Issue #3672 but applicable globally, agnostic of model provider. With the fixes suggested, I was able to extract up to 9-10,000 tokens per response on xinference. The issue still exists on AWS Bedrock, Azure OpenAI and Google Gemini, however. ### 2. Describe the feature you'd like to see For research purposes, I would like to see this limitation removed globally or at least on my particular install, where the max tokens for ANY response are not capped to 2,048 tokens by default, and the limit is instead configurable depending upon which model I'm using. For now, xinference with OpenHermes 2.5 is my control, which was able to achieve 7,000+ tokens with stability. My first test case is that I'm using AWS Bedrock with Llama 2 70B Chat, which I know is capable of more than 2,048 tokens per response as the base model. For my second point of comparison, I'm using Google Gemini through the Model Providers. I will be incorporating more and more test subjects as research progresses. ### 3. How will this feature improve your workflow or experience? This allows me to research entropy and generation instability of LLMs with generation length as the north-star and develop strategies to counteract, or balance out instability in responses approaching 7,000+ tokens. ### 4. Additional context or comments _No response_ ### 5. Can you help us with this feature? - [ ] I am interested in contributing to this feature.
yindo added the 💪 enhancement label 2026-02-21 17:49:57 -05:00
yindo closed this issue 2026-02-21 17:49:57 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 💪 enhancement
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#2784
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?