langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 12:05:30 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

A significant bug when processing in batch mode that includes multiple LLM results. #5202

New Issue
Closed
opened 2026-02-21 18:09:50 -05:00 by yindo · 8 comments
yindo commented 2026-02-21 18:09:50 -05:00
Owner
Copy Link
Copy Source

Originally created by @NirvanaProduct on GitHub (Aug 25, 2024).

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
  • Please do not modify this template :) and fill in all the required fields.

Dify version

0.7.1

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

Design a workflow where the output variable at the end contains the results of multiple LLM runs, and then run batches.
screenshot-20240825-200151
screenshot-20240825-161327

✔️ Expected Behavior

Get the sum of multiple LLM output results.

❌ Actual Behavior

Only received the content from one of the LLM outputs.

Originally created by @NirvanaProduct on GitHub (Aug 25, 2024). ### Self Checks - [X] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [X] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [X] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [X] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:） - [X] Please do not modify this template :) and fill in all the required fields. ### Dify version 0.7.1 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce Design a workflow where the output variable at the end contains the results of multiple LLM runs, and then run batches. ![screenshot-20240825-200151](https://github.com/user-attachments/assets/65595f49-a5f5-4784-94b7-715a66d8ac7c) ![screenshot-20240825-161327](https://github.com/user-attachments/assets/f7a24d05-6040-44bb-baba-113b63beaff5) ### ✔️ Expected Behavior Get the sum of multiple LLM output results. ### ❌ Actual Behavior Only received the content from one of the LLM outputs.
yindo closed this issue 2026-02-21 18:09:50 -05:00
yindo commented 2026-02-21 18:09:51 -05:00
Author
Owner
Copy Link
Copy Source

@NirvanaProduct commented on GitHub (Aug 25, 2024):

screenshot-20240825-161942

@NirvanaProduct commented on GitHub (Aug 25, 2024): ![screenshot-20240825-161942](https://github.com/user-attachments/assets/9ba7f656-6271-46b7-8926-3bacd0859a1b)
yindo commented 2026-02-21 18:09:51 -05:00
Author
Owner
Copy Link
Copy Source

@NirvanaProduct commented on GitHub (Aug 25, 2024):

Using workflow mode because only workflow mode can use batch processing functionality. In workflow mode, content can only be output through "end".

@NirvanaProduct commented on GitHub (Aug 25, 2024): Using workflow mode because only workflow mode can use batch processing functionality. In workflow mode, content can only be output through "end".
yindo commented 2026-02-21 18:09:51 -05:00
Author
Owner
Copy Link
Copy Source

@NirvanaProduct commented on GitHub (Aug 25, 2024):

screenshot-20240825-161327

@NirvanaProduct commented on GitHub (Aug 25, 2024): ![screenshot-20240825-161327](https://github.com/user-attachments/assets/985ea231-9b37-44b4-b37b-909e3301a1dd)
yindo commented 2026-02-21 18:09:51 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Aug 25, 2024):

@NirvanaProduct 你好! 我是Dosu，一个可以帮助你解决bug、回答问题并成为贡献者的机器人。在等待人类维护者时，我可以帮助你。

To resolve the issue where only one LLM output is received instead of the combined results from multiple LLMs when processing in batch mode using workflow mode in Dify version 0.7.1, you need to ensure that the _invoke method correctly handles the invocation of multiple models and combines their results.

Here is a possible approach to modify the _invoke method to handle multiple LLMs and combine their results:

class VertexAiLargeLanguageModel(LargeLanguageModel):

    def _invoke(self, models: list[str], credentials: dict,
                prompt_messages: list[PromptMessage], model_parameters: dict,
                tools: Optional[list[PromptMessageTool]] = None, stop: Optional[list[str]] = None,
                stream: bool = True, user: Optional[str] = None) \
            -> Union[LLMResult, Generator]:
        """
        Invoke multiple large language models and combine their results

        :param models: list of model names
        :param credentials: model credentials
        :param prompt_messages: prompt messages
        :param model_parameters: model parameters
        :param tools: tools for tool calling
        :param stop: stop words
        :param stream: is stream response
        :param user: unique user id
        :return: combined response or stream response chunk generator result
        """
        results = []
        for model in models:
            if "claude" in model:
                result = self._generate_anthropic(model, credentials, prompt_messages, model_parameters, stop, stream, user)
            else:
                result = self._generate(model, credentials, prompt_messages, model_parameters, tools, stop, stream, user)
            
            if isinstance(result, Generator):
                results.extend(list(result))
            else:
                results.append(result)
        
        # Combine results from multiple models
        combined_result = self._combine_results(results)
        return combined_result

    def _combine_results(self, results: list[Union[LLMResult, LLMResultChunk]]) -> Union[LLMResult, Generator]:
        """
        Combine results from multiple LLMs

        :param results: list of results from multiple LLMs
        :return: combined result
        """
        combined_content = ""
        combined_usage = LLMUsage()
        
        for result in results:
            if isinstance(result, LLMResult):
                combined_content += result.message.content
                combined_usage.prompt_tokens += result.usage.prompt_tokens
                combined_usage.completion_tokens += result.usage.completion_tokens
                combined_usage.total_tokens += result.usage.total_tokens
                combined_usage.total_price += result.usage.total_price
            elif isinstance(result, LLMResultChunk):
                combined_content += result.delta.message.content
                combined_usage.prompt_tokens += result.delta.usage.prompt_tokens
                combined_usage.completion_tokens += result.delta.usage.completion_tokens
                combined_usage.total_tokens += result.delta.usage.total_tokens
                combined_usage.total_price += result.delta.usage.total_price
        
        combined_message = AssistantPromptMessage(content=combined_content)
        combined_result = LLMResult(
            model="combined",
            prompt_messages=[],
            message=combined_message,
            usage=combined_usage
        )
        
        return combined_result

This code modifies the _invoke method to accept a list of models and then invokes each model individually. The results from each model are collected and combined using the _combine_results method. The _combine_results method aggregates the content and usage statistics from each result to form a single combined result [1][2][3].

To continue talking to Dosu, mention @dosu.

@dosubot[bot] commented on GitHub (Aug 25, 2024): <!-- Greeting --> @NirvanaProduct 你好! 我是[Dosu](https://dosu.dev)，一个可以帮助你解决bug、回答问题并成为贡献者的机器人。在等待人类维护者时，我可以帮助你。 <!-- Answer --> To resolve the issue where only one LLM output is received instead of the combined results from multiple LLMs when processing in batch mode using workflow mode in Dify version 0.7.1, you need to ensure that the `_invoke` method correctly handles the invocation of multiple models and combines their results. Here is a possible approach to modify the `_invoke` method to handle multiple LLMs and combine their results: ```python class VertexAiLargeLanguageModel(LargeLanguageModel): def _invoke(self, models: list[str], credentials: dict, prompt_messages: list[PromptMessage], model_parameters: dict, tools: Optional[list[PromptMessageTool]] = None, stop: Optional[list[str]] = None, stream: bool = True, user: Optional[str] = None) \ -> Union[LLMResult, Generator]: """ Invoke multiple large language models and combine their results :param models: list of model names :param credentials: model credentials :param prompt_messages: prompt messages :param model_parameters: model parameters :param tools: tools for tool calling :param stop: stop words :param stream: is stream response :param user: unique user id :return: combined response or stream response chunk generator result """ results = [] for model in models: if "claude" in model: result = self._generate_anthropic(model, credentials, prompt_messages, model_parameters, stop, stream, user) else: result = self._generate(model, credentials, prompt_messages, model_parameters, tools, stop, stream, user) if isinstance(result, Generator): results.extend(list(result)) else: results.append(result) # Combine results from multiple models combined_result = self._combine_results(results) return combined_result def _combine_results(self, results: list[Union[LLMResult, LLMResultChunk]]) -> Union[LLMResult, Generator]: """ Combine results from multiple LLMs :param results: list of results from multiple LLMs :return: combined result """ combined_content = "" combined_usage = LLMUsage() for result in results: if isinstance(result, LLMResult): combined_content += result.message.content combined_usage.prompt_tokens += result.usage.prompt_tokens combined_usage.completion_tokens += result.usage.completion_tokens combined_usage.total_tokens += result.usage.total_tokens combined_usage.total_price += result.usage.total_price elif isinstance(result, LLMResultChunk): combined_content += result.delta.message.content combined_usage.prompt_tokens += result.delta.usage.prompt_tokens combined_usage.completion_tokens += result.delta.usage.completion_tokens combined_usage.total_tokens += result.delta.usage.total_tokens combined_usage.total_price += result.delta.usage.total_price combined_message = AssistantPromptMessage(content=combined_content) combined_result = LLMResult( model="combined", prompt_messages=[], message=combined_message, usage=combined_usage ) return combined_result ``` This code modifies the `_invoke` method to accept a list of models and then invokes each model individually. The results from each model are collected and combined using the `_combine_results` method. The `_combine_results` method aggregates the content and usage statistics from each result to form a single combined result <sup>[[1]](https://github.com/langgenius/dify/blob/main/api/core/model_runtime/model_providers/vertex_ai/llm/llm.py)</sup><sup>[[2]](https://github.com/langgenius/dify/blob/main/api/core/model_runtime/model_providers/bedrock/llm/llm.py)</sup><sup>[[3]](https://github.com/langgenius/dify/blob/main/api/core/workflow/nodes/parameter_extractor/parameter_extractor_node.py)</sup>. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.*
yindo commented 2026-02-21 18:09:51 -05:00
Author
Owner
Copy Link
Copy Source

@crazywoola commented on GitHub (Aug 25, 2024):

I guess this works for you.

image image image
@crazywoola commented on GitHub (Aug 25, 2024): I guess this works for you. <img width="1712" alt="image" src="https://github.com/user-attachments/assets/972b2ef1-d1f9-44d7-a433-2bb26250c3cc"> <img width="1548" alt="image" src="https://github.com/user-attachments/assets/282474b8-8d26-4d22-a627-57b5c4552fe0"> <img width="573" alt="image" src="https://github.com/user-attachments/assets/efa76b57-8678-4903-9703-489a6dd210d9">
yindo commented 2026-02-21 18:09:51 -05:00
Author
Owner
Copy Link
Copy Source

@NirvanaProduct commented on GitHub (Aug 26, 2024):

@crazywoola Thank you very much for your answer, but there are still unresolved errors.
When my output is short enough, the method you mentioned works. (As shown in Figure 1, run batch mode)
However, the content I need to output is very long and requires more than 300 seconds to complete. During the waiting process, it will be judged as output timeout by batch processing, and I will still be unable to complete my task. (As shown in Figure 2, run batch mode)
screenshot-20240826-183117

screenshot-20240826-181832

@NirvanaProduct commented on GitHub (Aug 26, 2024): @crazywoola Thank you very much for your answer, but there are still unresolved errors. When my output is short enough, the method you mentioned works. (As shown in Figure 1, run batch mode) However, the content I need to output is very long and requires more than 300 seconds to complete. During the waiting process, it will be judged as output timeout by batch processing, and I will still be unable to complete my task. (As shown in Figure 2, run batch mode) ![screenshot-20240826-183117](https://github.com/user-attachments/assets/0f1dcf2a-70c7-4f78-baea-6a5b3b1f1d77) ![screenshot-20240826-181832](https://github.com/user-attachments/assets/4903c4d7-791f-43e3-ac37-8932509288eb)
yindo commented 2026-02-21 18:09:51 -05:00
Author
Owner
Copy Link
Copy Source

@NirvanaProduct commented on GitHub (Aug 26, 2024):

screenshot-20240826-182528

@NirvanaProduct commented on GitHub (Aug 26, 2024): ![screenshot-20240826-182528](https://github.com/user-attachments/assets/0f26a550-4c2d-407a-830a-7f582d831bfe)
yindo commented 2026-02-21 18:09:51 -05:00
Author
Owner
Copy Link
Copy Source

@NirvanaProduct commented on GitHub (Aug 26, 2024):

run batch mode

@NirvanaProduct commented on GitHub (Aug 26, 2024): run batch mode
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#5202
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?