langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 12:05:30 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

When creating a workflow, if multiple documents are passed through a document extractor and sent to the LLM, the LLM may only summarize one document and ignore the others. #8556

New Issue
Closed
opened 2026-02-21 18:26:18 -05:00 by yindo · 8 comments
yindo commented 2026-02-21 18:26:18 -05:00
Owner
Copy Link
Copy Source

Originally created by @zpf2000 on GitHub (Feb 24, 2025).

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
  • Please do not modify this template :) and fill in all the required fields.

Dify version

latest

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

When creating a workflow, if multiple documents are passed through a document extractor and sent to the LLM, the LLM may only summarize one document and ignore the others.

✔️ Expected Behavior

No response

❌ Actual Behavior

No response

Originally created by @zpf2000 on GitHub (Feb 24, 2025). ### Self Checks - [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [x] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:） - [x] Please do not modify this template :) and fill in all the required fields. ### Dify version latest ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce When creating a workflow, if multiple documents are passed through a document extractor and sent to the LLM, the LLM may only summarize one document and ignore the others. ### ✔️ Expected Behavior _No response_ ### ❌ Actual Behavior _No response_
yindo added the 🤔 cant-reproduce label 2026-02-21 18:26:18 -05:00
yindo closed this issue 2026-02-21 18:26:18 -05:00
yindo commented 2026-02-21 18:26:18 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Feb 24, 2025):

Hello, @zpf2000! 👋 I'm here to help you with bug reports, answer questions, and assist you in becoming a contributor. If you need any assistance, feel free to reach out!

To ensure that the LLM summarizes all documents when multiple documents are passed through a document extractor in a workflow, make sure the input variable is of type Array[File]. This allows the document extractor node to handle multiple files by extracting their contents and converting them into an array of strings, which can then be passed to the LLM for summarization. The _run method in the DocumentExtractorNode class processes each document in the list, ensuring that all documents are summarized [1][2].

Additionally, check the configurations of the LLM node, such as prompt templates and model parameters, to ensure they are set up to handle multiple document summaries effectively [3]. If the issue persists, review the error handling and logging in the document extractor and LLM nodes for any insights into why some documents might be ignored [1].

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other

@dosubot[bot] commented on GitHub (Feb 24, 2025): <!-- Greeting --> Hello, @zpf2000! 👋 I'm here to help you with bug reports, answer questions, and assist you in becoming a contributor. If you need any assistance, feel free to reach out! <!-- Answer --> To ensure that the LLM summarizes all documents when multiple documents are passed through a document extractor in a workflow, make sure the input variable is of type `Array[File]`. This allows the document extractor node to handle multiple files by extracting their contents and converting them into an array of strings, which can then be passed to the LLM for summarization. The `_run` method in the `DocumentExtractorNode` class processes each document in the list, ensuring that all documents are summarized <sup>[[1]](https://github.com/langgenius/dify/blob/main/api/core/workflow/nodes/document_extractor/node.py)</sup><sup>[[2]](https://github.com/langgenius/dify-docs/blob/main/en/guides/workflow/node/doc-extractor.md)</sup>. Additionally, check the configurations of the LLM node, such as prompt templates and model parameters, to ensure they are set up to handle multiple document summaries effectively <sup>[[3]](https://github.com/langgenius/dify/blob/main/api/core/workflow/nodes/llm/node.py)</sup>. If the issue persists, review the error handling and logging in the document extractor and LLM nodes for any insights into why some documents might be ignored <sup>[[1]](https://github.com/langgenius/dify/blob/main/api/core/workflow/nodes/document_extractor/node.py)</sup>. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.* --- <sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/4340191e-a7f8-43bc-a1a4-0037392e3cf4?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/4340191e-a7f8-43bc-a1a4-0037392e3cf4?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/4340191e-a7f8-43bc-a1a4-0037392e3cf4?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/4340191e-a7f8-43bc-a1a4-0037392e3cf4?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/4340191e-a7f8-43bc-a1a4-0037392e3cf4?feedback_type=hallucination) | [Bug Report](https://app.dosu.dev/response-feedback/4340191e-a7f8-43bc-a1a4-0037392e3cf4?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/4340191e-a7f8-43bc-a1a4-0037392e3cf4?feedback_type=other)</sup>
yindo commented 2026-02-21 18:26:18 -05:00
Author
Owner
Copy Link
Copy Source

@zpf2000 commented on GitHub (Feb 24, 2025):

Thanks for reply. I confirm that each document is being extracted because the output of the extractor and the input to the LLM are consistent, both in the Array[File] format, and I have also ensured that their content is correct. Additionally, when I want the LLM to summarize the documents into a table, it sometimes ignores my system prompt and summarizes the content into a list or other formats instead. @dosu

@zpf2000 commented on GitHub (Feb 24, 2025): Thanks for reply. I confirm that each document is being extracted because the output of the extractor and the input to the LLM are consistent, both in the `Array[File]` format, and I have also ensured that their content is correct. Additionally, when I want the LLM to summarize the documents into a table, it sometimes ignores my system prompt and summarizes the content into a list or other formats instead. @dosu
yindo commented 2026-02-21 18:26:18 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Feb 24, 2025):

If the LLM is ignoring your system prompt and summarizing the content into an unintended format, there are a few potential causes and solutions to consider:

  1. Prompt Configuration: Ensure that your prompt templates are correctly configured. The LLM node allows for advanced configuration options, including using Jinja-2 templates for complex prompts. Make sure the system prompt is clearly defined and correctly formatted [1].

  2. Model Parameters: Check the model parameters such as temperature, TopP, and response format. These parameters can influence the output style. For instance, a higher temperature might lead to more creative outputs, which could deviate from the expected format [2].

  3. Error Handling and Logging: Review the error handling and logging mechanisms in the LLM node. Specific errors like InvalidVariableTypeError or NoPromptFoundError could indicate issues with how prompts are being processed. These logs can provide insights into why the system prompt might be ignored [3].

  4. Advanced Settings: If you are using memory or context features, ensure they are configured correctly. Misconfigured memory settings might affect how the LLM interprets the prompt [1].

By reviewing these areas, you can better understand why the LLM might be ignoring the system prompt and adjust the configuration to achieve the desired output format.

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other

@dosubot[bot] commented on GitHub (Feb 24, 2025): <!-- Answer --> If the LLM is ignoring your system prompt and summarizing the content into an unintended format, there are a few potential causes and solutions to consider: 1. **Prompt Configuration**: Ensure that your prompt templates are correctly configured. The LLM node allows for advanced configuration options, including using Jinja-2 templates for complex prompts. Make sure the system prompt is clearly defined and correctly formatted <sup>[[1]](https://github.com/langgenius/dify-docs/blob/main/en/guides/workflow/node/llm.md)</sup>. 2. **Model Parameters**: Check the model parameters such as temperature, TopP, and response format. These parameters can influence the output style. For instance, a higher temperature might lead to more creative outputs, which could deviate from the expected format <sup>[[2]](https://github.com/langgenius/dify/blob/main/web/app/components/workflow/nodes/llm/use-config.ts)</sup>. 3. **Error Handling and Logging**: Review the error handling and logging mechanisms in the LLM node. Specific errors like `InvalidVariableTypeError` or `NoPromptFoundError` could indicate issues with how prompts are being processed. These logs can provide insights into why the system prompt might be ignored <sup>[[3]](https://github.com/langgenius/dify/blob/main/api/core/workflow/nodes/llm/node.py)</sup>. 4. **Advanced Settings**: If you are using memory or context features, ensure they are configured correctly. Misconfigured memory settings might affect how the LLM interprets the prompt <sup>[[1]](https://github.com/langgenius/dify-docs/blob/main/en/guides/workflow/node/llm.md)</sup>. By reviewing these areas, you can better understand why the LLM might be ignoring the system prompt and adjust the configuration to achieve the desired output format. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.* --- <sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/f15b3c91-fd4f-4837-98f2-62f92aa1fd11?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/f15b3c91-fd4f-4837-98f2-62f92aa1fd11?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/f15b3c91-fd4f-4837-98f2-62f92aa1fd11?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/f15b3c91-fd4f-4837-98f2-62f92aa1fd11?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/f15b3c91-fd4f-4837-98f2-62f92aa1fd11?feedback_type=hallucination) | [Bug Report](https://app.dosu.dev/response-feedback/f15b3c91-fd4f-4837-98f2-62f92aa1fd11?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/f15b3c91-fd4f-4837-98f2-62f92aa1fd11?feedback_type=other)</sup>
yindo commented 2026-02-21 18:26:18 -05:00
Author
Owner
Copy Link
Copy Source

@crazywoola commented on GitHub (Feb 24, 2025):

Please provide the DSL to help us investigate. And there is no Expected Behavior and Actual Behavior, which is quite hard to undersrtand what is going on here. Please also consider the LLM might not smart enough to guess what you are trying to do, please provide the prompt as well.

@crazywoola commented on GitHub (Feb 24, 2025): Please provide the DSL to help us investigate. And there is no Expected Behavior and Actual Behavior, which is quite hard to undersrtand what is going on here. Please also consider the LLM might not smart enough to guess what you are trying to do, please provide the prompt as well.
yindo commented 2026-02-21 18:26:18 -05:00
Author
Owner
Copy Link
Copy Source

@zpf2000 commented on GitHub (Feb 24, 2025):

demo.md
Here is the corresponding DSL. The system prompt is: "You are a professional document analysis assistant responsible for extracting key information from multiple PDF documents and summarizing it into a structured Markdown table. Your output should only include the Markdown table, with no additional information. You need to extract key information from the provided text content and present it in a clear table format. Please note that these document contents are independent; do not mix their information."

The user prompt is: "Below are the contents of {{#1740362373703.length#}} independent documents. Each document's content has been extracted as a string and stored in an array. The length of the array is {{#1740362373703.length#}}, indicating there are {{#1740362373703.length#}} documents. Based on this content, please extract key information from each document and generate a summary Markdown table. Only output this table, with no redundant messages. The input data format and content are as follows: {{#context#}}. Please ensure the table content is concise, accurate, and that the information for each document is clearly distinguishable."

I placed the document content in the context of the LLM because if I directly used the Exactor.text output from the extractor, the text would become Unicode. Using the context ensures that the content received by the LLM is correct.

@zpf2000 commented on GitHub (Feb 24, 2025): [demo.md](https://github.com/user-attachments/files/18937083/demo.md) Here is the corresponding DSL. The system prompt is: "You are a professional document analysis assistant responsible for extracting key information from multiple PDF documents and summarizing it into a structured Markdown table. Your output should only include the Markdown table, with no additional information. You need to extract key information from the provided text content and present it in a clear table format. Please note that these document contents are independent; do not mix their information." The user prompt is: "Below are the contents of {{#1740362373703.length#}} independent documents. Each document's content has been extracted as a string and stored in an array. The length of the array is {{#1740362373703.length#}}, indicating there are {{#1740362373703.length#}} documents. Based on this content, please extract key information from each document and generate a summary Markdown table. Only output this table, with no redundant messages. The input data format and content are as follows: {{#context#}}. Please ensure the table content is concise, accurate, and that the information for each document is clearly distinguishable." I placed the document content in the context of the LLM because if I directly used the `Exactor.text` output from the extractor, the text would become **Unicode**. Using the context ensures that the content received by the LLM is correct.
yindo commented 2026-02-21 18:26:18 -05:00
Author
Owner
Copy Link
Copy Source

@crazywoola commented on GitHub (Feb 24, 2025):

Here are something you should know:

  • The provided workflow looks good to me.
  • DeepSeek R1 is good doesn't mean 1.5b is good enough. It's just a toy version of deepseek-r1.
  • DeepSeek R1 doesn't have a system role, you should conbine them together in the user instead.
@crazywoola commented on GitHub (Feb 24, 2025): Here are something you should know: - The provided workflow looks good to me. - DeepSeek R1 is good doesn't mean 1.5b is good enough. It's just a toy version of deepseek-r1. - DeepSeek R1 doesn't have a `system` role, you should conbine them together in the `user` instead.
yindo commented 2026-02-21 18:26:19 -05:00
Author
Owner
Copy Link
Copy Source

@zpf2000 commented on GitHub (Feb 24, 2025):

Thank you for your response. I wanted to test whether the functionality could be implemented using a lightweight version first, before considering accuracy. However, it seems that its inability to fully follow the prompt and generate a Markdown table is due to the version issue, is that correct? Additionally, placing variables in the context—is this the correct approach? Why does directly calling the variable cause the text to turn into Unicode?

@zpf2000 commented on GitHub (Feb 24, 2025): Thank you for your response. I wanted to test whether the functionality could be implemented using a lightweight version first, before considering accuracy. However, it seems that its inability to fully follow the prompt and generate a Markdown table is due to the version issue, is that correct? Additionally, placing variables in the context—is this the correct approach? Why does directly calling the variable cause the text to turn into Unicode?
yindo commented 2026-02-21 18:26:19 -05:00
Author
Owner
Copy Link
Copy Source

@crazywoola commented on GitHub (Feb 24, 2025):

Q1: Yes
Q2: Yes
Q3: because the data pass to the LLM is encoded, this is an expected behavior, it doesn't matter the charset is utf or unicode.

@crazywoola commented on GitHub (Feb 24, 2025): Q1: Yes Q2: Yes Q3: because the data pass to the LLM is encoded, this is an expected behavior, it doesn't matter the charset is utf or unicode.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🤔 cant-reproduce
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#8556
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?