langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-22 20:45:26 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Model load balancing usage issues #5745

New Issue
Closed
opened 2026-02-21 18:12:24 -05:00 by yindo · 11 comments
yindo commented 2026-02-21 18:12:24 -05:00
Owner
Copy Link
Copy Source

Originally created by @yunfeng13 on GitHub (Sep 23, 2024).

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
  • Please do not modify this template :) and fill in all the required fields.

Dify version

0.8.3

Cloud or Self Hosted

Cloud

Steps to reproduce

After load balancing is enabled, the key used is always the latest one added. Why is the default key not used? Shouldn't it be used one by one? I want to know when the default key will be used, when load balancing will take effect, and when the current is limited.

✔️ Expected Behavior

After load balancing is enabled, the key used is always the latest one added. Why is the default key not used? Shouldn't it be used one by one? I want to know when the default key will be used, when load balancing will take effect, and when the current is limited.

❌ Actual Behavior

After load balancing is enabled, the key used is always the latest one added. Why is the default key not used? Shouldn't it be used one by one? I want to know when the default key will be used, when load balancing will take effect, and when the current is limited.

Originally created by @yunfeng13 on GitHub (Sep 23, 2024). ### Self Checks - [X] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [X] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [X] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [X] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:） - [X] Please do not modify this template :) and fill in all the required fields. ### Dify version 0.8.3 ### Cloud or Self Hosted Cloud ### Steps to reproduce After load balancing is enabled, the key used is always the latest one added. Why is the default key not used? Shouldn't it be used one by one? I want to know when the default key will be used, when load balancing will take effect, and when the current is limited. ### ✔️ Expected Behavior After load balancing is enabled, the key used is always the latest one added. Why is the default key not used? Shouldn't it be used one by one? I want to know when the default key will be used, when load balancing will take effect, and when the current is limited. ### ❌ Actual Behavior After load balancing is enabled, the key used is always the latest one added. Why is the default key not used? Shouldn't it be used one by one? I want to know when the default key will be used, when load balancing will take effect, and when the current is limited.
yindo added the 🐞 bug label 2026-02-21 18:12:24 -05:00
yindo closed this issue 2026-02-21 18:12:24 -05:00
yindo commented 2026-02-21 18:12:24 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Sep 23, 2024):

The default key is not used in load balancing because load balancing requires multiple API keys to distribute the requests. The default key is the credential added when initially configuring the model provider. Load balancing will take effect and the current will be limited when at least one additional model credential is added to the configuration [1].

For more details, you might find this similar discussion helpful: What will happen to the API when load balancing is enabled and rate limiting is enabled? [2].

To continue talking to Dosu, mention @dosu.

@dosubot[bot] commented on GitHub (Sep 23, 2024): <!-- Answer --> The default key is not used in load balancing because load balancing requires multiple API keys to distribute the requests. The default key is the credential added when initially configuring the model provider. Load balancing will take effect and the current will be limited when at least one additional model credential is added to the configuration <sup>[[1]](https://github.com/langgenius/dify-docs/blob/main/en/guides/model-configuration/load-balancing.md)</sup>. For more details, you might find this similar discussion helpful: [What will happen to the API when load balancing is enabled and rate limiting is enabled?](https://github.com/langgenius/dify/discussions/8314) <sup>[[2]](https://github.com/langgenius/dify/discussions/8314)</sup>. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.*
yindo commented 2026-02-21 18:12:24 -05:00
Author
Owner
Copy Link
Copy Source

@yunfeng13 commented on GitHub (Sep 23, 2024):

@StudyingLover @dosu
image
I set up two additional keys, but I still use one key and don’t use both together.

@yunfeng13 commented on GitHub (Sep 23, 2024): @StudyingLover @dosu ![image](https://github.com/user-attachments/assets/47c9572d-3213-4f78-9c75-24cfb37b3d57) I set up two additional keys, but I still use one key and don’t use both together.
yindo commented 2026-02-21 18:12:25 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Sep 23, 2024):

To ensure that load balancing works correctly with multiple API keys, follow these steps:

  1. Add Configuration: Make sure you have added the additional API keys by clicking the Add Configuration button in the load balancing settings.
  2. Save Changes: Ensure that you save the changes after adding the new keys.
  3. Enable Load Balancing: Verify that the load balancing feature is enabled.

Dify uses the Round-robin strategy by default for load balancing, which means it should distribute requests evenly across all configured keys [1]. If you still see only one key being used, double-check that all keys are correctly added and saved in the configuration.

If the issue persists, there might be a problem with how the keys are being recognized or saved. In that case, reviewing the configuration steps or checking for any known issues in the documentation might help.

To continue talking to Dosu, mention @dosu.

@dosubot[bot] commented on GitHub (Sep 23, 2024): <!-- Answer --> To ensure that load balancing works correctly with multiple API keys, follow these steps: 1. **Add Configuration**: Make sure you have added the additional API keys by clicking the **Add Configuration** button in the load balancing settings. 2. **Save Changes**: Ensure that you save the changes after adding the new keys. 3. **Enable Load Balancing**: Verify that the load balancing feature is enabled. Dify uses the Round-robin strategy by default for load balancing, which means it should distribute requests evenly across all configured keys <sup>[[1]](https://github.com/langgenius/dify-docs/blob/main/en/guides/model-configuration/load-balancing.md)</sup>. If you still see only one key being used, double-check that all keys are correctly added and saved in the configuration. If the issue persists, there might be a problem with how the keys are being recognized or saved. In that case, reviewing the configuration steps or checking for any known issues in the documentation might help. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.*
yindo commented 2026-02-21 18:12:25 -05:00
Author
Owner
Copy Link
Copy Source

@yunfeng13 commented on GitHub (Sep 23, 2024):

@StudyingLover @dosu
But it was indeed saved successfully, but there was no load balancing, but the first key was always used.

@yunfeng13 commented on GitHub (Sep 23, 2024): @StudyingLover @dosu But it was indeed saved successfully, but there was no load balancing, but the first key was always used.
yindo commented 2026-02-21 18:12:25 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Sep 23, 2024):

The first key is always used instead of load balancing across multiple keys because at least one additional model credential must be added to save and enable load balancing. If only the default API key is configured without adding additional keys, load balancing will not be activated, and the system will continue to use the first key. Additionally, it is important to note that model load balancing is a paid feature and requires a subscription to a SaaS paid service or the purchase of an enterprise version to enable this functionality [1].

If you have already added additional keys and saved the configuration successfully, but still see only the first key being used, it might be worth checking the following:

  1. Verify Configuration: Double-check that all additional keys are correctly added and saved in the configuration.
  2. Subscription: Ensure that your subscription includes the load balancing feature.
  3. Logs and Debugging: Use the logging and debugging features in Dify to diagnose the issue. The Logs feature records all interaction processes, and integrating with Langfuse can provide detailed trace information [2][3].

If everything seems correct and the issue persists, there might be a deeper problem with how the keys are being recognized or saved.

To continue talking to Dosu, mention @dosu.

@dosubot[bot] commented on GitHub (Sep 23, 2024): <!-- Answer --> The first key is always used instead of load balancing across multiple keys because **at least one additional model credential** must be added to save and enable load balancing. If only the default API key is configured without adding additional keys, load balancing will not be activated, and the system will continue to use the first key. Additionally, it is important to note that model load balancing is a paid feature and requires a subscription to a SaaS paid service or the purchase of an enterprise version to enable this functionality <sup>[[1]](https://github.com/langgenius/dify-docs/blob/main/en/guides/model-configuration/load-balancing.md)</sup>. If you have already added additional keys and saved the configuration successfully, but still see only the first key being used, it might be worth checking the following: 1. **Verify Configuration**: Double-check that all additional keys are correctly added and saved in the configuration. 2. **Subscription**: Ensure that your subscription includes the load balancing feature. 3. **Logs and Debugging**: Use the logging and debugging features in Dify to diagnose the issue. The Logs feature records all interaction processes, and integrating with Langfuse can provide detailed trace information <sup>[[2]](https://github.com/langgenius/dify-docs/blob/main/jp/guides/biao-zhu/logs.md)</sup><sup>[[3]](https://github.com/langgenius/dify-docs/blob/main/en/guides/monitoring/integrate-external-ops-tools/integrate-langfuse.md)</sup>. If everything seems correct and the issue persists, there might be a deeper problem with how the keys are being recognized or saved. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.*
yindo commented 2026-02-21 18:12:25 -05:00
Author
Owner
Copy Link
Copy Source

@yunfeng13 commented on GitHub (Sep 23, 2024):

@StudyingLover @dosu
Can you tell me which part of the load balancing code is in, that is, use the key saved by the load balancing to request

@yunfeng13 commented on GitHub (Sep 23, 2024): @StudyingLover @dosu Can you tell me which part of the load balancing code is in, that is, use the key saved by the load balancing to request
yindo commented 2026-02-21 18:12:25 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Sep 23, 2024):

The part of the load balancing code responsible for using the key saved by the load balancing to make requests is:

provider_model_credentials_cache = ProviderCredentialsCache(
    tenant_id=load_balancing_model_config.tenant_id,
    identity_id=load_balancing_model_config.id,
    cache_type=ProviderCredentialsCacheType.LOAD_BALANCING_MODEL,
)

# Get cached provider model credentials
cached_provider_model_credentials = provider_model_credentials_cache.get()

This code snippet retrieves the cached provider model credentials using the key saved by the load balancing configuration. The ProviderCredentialsCache class is used to manage the cache, and the get method fetches the credentials based on the tenant_id and identity_id from the load_balancing_model_config [1][2].

To continue talking to Dosu, mention @dosu.

@dosubot[bot] commented on GitHub (Sep 23, 2024): <!-- Answer --> The part of the load balancing code responsible for using the key saved by the load balancing to make requests is: ```python provider_model_credentials_cache = ProviderCredentialsCache( tenant_id=load_balancing_model_config.tenant_id, identity_id=load_balancing_model_config.id, cache_type=ProviderCredentialsCacheType.LOAD_BALANCING_MODEL, ) # Get cached provider model credentials cached_provider_model_credentials = provider_model_credentials_cache.get() ``` This code snippet retrieves the cached provider model credentials using the key saved by the load balancing configuration. The `ProviderCredentialsCache` class is used to manage the cache, and the `get` method fetches the credentials based on the `tenant_id` and `identity_id` from the `load_balancing_model_config` <sup>[[1]](https://github.com/langgenius/dify/blob/main/api/core/provider_manager.py)</sup><sup>[[2]](https://github.com/langgenius/dify/blob/main/api/services/model_load_balancing_service.py)</sup>. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.*
yindo commented 2026-02-21 18:12:25 -05:00
Author
Owner
Copy Link
Copy Source

@yunfeng13 commented on GitHub (Sep 23, 2024):

@StudyingLover @dosu
So it means that if I don’t have it in redis, will there be a situation where only one key is used?

@yunfeng13 commented on GitHub (Sep 23, 2024): @StudyingLover @dosu So it means that if I don’t have it in redis, will there be a situation where only one key is used?
yindo commented 2026-02-21 18:12:25 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Sep 23, 2024):

Yes, if you don’t have the load balancing configuration in Redis, there will be a situation where only one key is used. This is because the ModelInstance class relies on the LBModelManager for load balancing. If the load_balancing_manager is not set (i.e., None), the _round_robin_invoke method will directly call the function with the provided credentials without any load balancing.

Here is the relevant code snippet:

def _round_robin_invoke(self, function: Callable, *args, **kwargs):
    """
    Round-robin invoke
    :param function: function to invoke
    :param args: function args
    :param kwargs: function kwargs
    :return:
    """
    if not self.load_balancing_manager:
        return function(*args, **kwargs)

If self.load_balancing_manager is None, the function is invoked directly, meaning only one set of credentials (one key) will be used [1].

To continue talking to Dosu, mention @dosu.

@dosubot[bot] commented on GitHub (Sep 23, 2024): <!-- Answer --> Yes, if you don’t have the load balancing configuration in Redis, there will be a situation where only one key is used. This is because the `ModelInstance` class relies on the `LBModelManager` for load balancing. If the `load_balancing_manager` is not set (i.e., `None`), the `_round_robin_invoke` method will directly call the function with the provided credentials without any load balancing. Here is the relevant code snippet: ```python def _round_robin_invoke(self, function: Callable, *args, **kwargs): """ Round-robin invoke :param function: function to invoke :param args: function args :param kwargs: function kwargs :return: """ if not self.load_balancing_manager: return function(*args, **kwargs) ``` If `self.load_balancing_manager` is `None`, the function is invoked directly, meaning only one set of credentials (one key) will be used <sup>[[1]](https://github.com/langgenius/dify/blob/main/api/core/model_manager.py)</sup>. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.*
yindo commented 2026-02-21 18:12:25 -05:00
Author
Owner
Copy Link
Copy Source

@yunfeng13 commented on GitHub (Sep 23, 2024):

@StudyingLover @dosu The saved load balancing configuration is now in the table load_balancing_model_configs, but the load balancing fails and the default key is always used. What may be the reason?

@yunfeng13 commented on GitHub (Sep 23, 2024): @StudyingLover @dosu The saved load balancing configuration is now in the table load_balancing_model_configs, but the load balancing fails and the default key is always used. What may be the reason?
yindo commented 2026-02-21 18:12:25 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Sep 23, 2024):

The load balancing fails and the default key is always used because at least one additional model credential must be added to save and enable load balancing. If no additional credentials are added, the system will default to using the initial API key.

Additionally, the system checks for the existence of an "inherit" configuration, which represents the provider or model custom credentials. If this "inherit" configuration does not exist, it initializes one. This ensures that the load balancing configuration is properly set up.

Moreover, if the encrypted_config field in the load_balancing_model_config is not populated, the system will skip that configuration unless it is the "inherit" configuration, which is added with empty credentials [1][2][3].

To continue talking to Dosu, mention @dosu.

@dosubot[bot] commented on GitHub (Sep 23, 2024): <!-- Answer --> The load balancing fails and the default key is always used because **at least one additional model credential** must be added to save and enable load balancing. If no additional credentials are added, the system will default to using the initial API key. Additionally, the system checks for the existence of an "inherit" configuration, which represents the provider or model custom credentials. If this "inherit" configuration does not exist, it initializes one. This ensures that the load balancing configuration is properly set up. Moreover, if the `encrypted_config` field in the `load_balancing_model_config` is not populated, the system will skip that configuration unless it is the "__inherit__" configuration, which is added with empty credentials <sup>[[1]](https://github.com/langgenius/dify-docs/blob/main/en/guides/model-configuration/load-balancing.md)</sup><sup>[[2]](https://github.com/langgenius/dify/blob/main/api/services/model_load_balancing_service.py)</sup><sup>[[3]](https://github.com/langgenius/dify/blob/main/api/core/provider_manager.py)</sup>. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.*
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#5745
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?