langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 20:15:31 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

The result of PDF text segmentation is incorrect, and the cross-page content will be forcibly segmented. #8911

New Issue
Closed
opened 2026-02-21 18:28:33 -05:00 by yindo · 16 comments
yindo commented 2026-02-21 18:28:33 -05:00
Owner
Copy Link
Copy Source

Originally created by @ssim2023 on GitHub (Mar 3, 2025).

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
  • Please do not modify this template :) and fill in all the required fields.

Dify version

0.15.3

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

Image

Image

✔️ Expected Behavior

Segmented content is not affected by spreads

❌ Actual Behavior

A complete piece of content will be forcibly segmented when it encounters spreads

Originally created by @ssim2023 on GitHub (Mar 3, 2025). ### Self Checks - [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [x] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:） - [x] Please do not modify this template :) and fill in all the required fields. ### Dify version 0.15.3 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce ![Image](https://github.com/user-attachments/assets/07efdb3b-9e0f-4294-b612-ed0d9ab60da4) ![Image](https://github.com/user-attachments/assets/5af90a7c-e8ff-49ab-92df-4bd07cbc0852) ### ✔️ Expected Behavior Segmented content is not affected by spreads ### ❌ Actual Behavior A complete piece of content will be forcibly segmented when it encounters spreads
yindo added the 🐞 bug label 2026-02-21 18:28:33 -05:00
yindo closed this issue 2026-02-21 18:28:33 -05:00
yindo commented 2026-02-21 18:28:34 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Mar 3, 2025):

Hello @ssim2023! I'm here to help you with any issues or questions you have about Dify. I can assist you with bug resolution, answering queries, and even guide you on how to become a contributor. Let's tackle this challenge together!

The issue you're experiencing with PDF text segmentation in Dify, where cross-page content is forcibly segmented, might be related to the default settings for text segmentation. Dify uses a chunk delimiter (default is \n) and a maximum chunk length (default is 500 tokens) to segment text. If these limits are exceeded, the system enforces chunking, which could lead to the segmentation of cross-page content [1].

To address this, you can adjust the configuration options to improve segmentation accuracy. Consider increasing the MAXIMUM_CHUNK_TOKEN_LENGTH to allow larger chunks, which might help retain more context across pages. Additionally, setting an Overlapping Chunk Length can enhance information retention and retrieval accuracy, which is crucial for cross-page content [2].

If you need more control over the segmentation process, you can define custom chunking rules using regex or switch to a custom mode to manually adjust chunking rules and delimiters [2].

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other

@dosubot[bot] commented on GitHub (Mar 3, 2025): <!-- Greeting --> Hello @ssim2023! I'm here to help you with any issues or questions you have about Dify. I can assist you with bug resolution, answering queries, and even guide you on how to become a contributor. Let's tackle this challenge together! <!-- Answer --> The issue you're experiencing with PDF text segmentation in Dify, where cross-page content is forcibly segmented, might be related to the default settings for text segmentation. Dify uses a chunk delimiter (default is `\n`) and a maximum chunk length (default is 500 tokens) to segment text. If these limits are exceeded, the system enforces chunking, which could lead to the segmentation of cross-page content <sup>[[1]](https://github.com/langgenius/dify-docs/blob/main/jp/guides/knowledge-base/create-knowledge-and-upload-documents/chunking-and-cleaning-text.md)</sup>. To address this, you can adjust the configuration options to improve segmentation accuracy. Consider increasing the **MAXIMUM_CHUNK_TOKEN_LENGTH** to allow larger chunks, which might help retain more context across pages. Additionally, setting an **Overlapping Chunk Length** can enhance information retention and retrieval accuracy, which is crucial for cross-page content <sup>[[2]](https://github.com/langgenius/dify-docs/blob/main/en/guides/knowledge-base/knowledge-and-documents-maintenance/maintain-knowledge-documents.md)</sup>. If you need more control over the segmentation process, you can define custom chunking rules using regex or switch to a custom mode to manually adjust chunking rules and delimiters <sup>[[2]](https://github.com/langgenius/dify-docs/blob/main/en/guides/knowledge-base/knowledge-and-documents-maintenance/maintain-knowledge-documents.md)</sup>. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.* --- <sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/2a904593-61cc-4ea7-97f8-0cea0575b330?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/2a904593-61cc-4ea7-97f8-0cea0575b330?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/2a904593-61cc-4ea7-97f8-0cea0575b330?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/2a904593-61cc-4ea7-97f8-0cea0575b330?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/2a904593-61cc-4ea7-97f8-0cea0575b330?feedback_type=hallucination) | [Bug Report](https://app.dosu.dev/response-feedback/2a904593-61cc-4ea7-97f8-0cea0575b330?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/2a904593-61cc-4ea7-97f8-0cea0575b330?feedback_type=other)</sup>
yindo commented 2026-02-21 18:28:34 -05:00
Author
Owner
Copy Link
Copy Source

@ssim2023 commented on GitHub (Mar 3, 2025):

As shown in the picture, I used the parent-child segmentation mode. The maximum token I set for the parent segment is 4000. After segmentation, it is obvious that the length of chunk-2 is far from being segmented across pages before reaching this limit. Therefore, this is not a problem of setting the maximum token impact. It is a forced segmentation of PDF across pages

@ssim2023 commented on GitHub (Mar 3, 2025): As shown in the picture, I used the parent-child segmentation mode. The maximum token I set for the parent segment is 4000. After segmentation, it is obvious that the length of chunk-2 is far from being segmented across pages before reaching this limit. Therefore, this is not a problem of setting the maximum token impact. It is a forced segmentation of PDF across pages
yindo commented 2026-02-21 18:28:34 -05:00
Author
Owner
Copy Link
Copy Source

@ssim2023 commented on GitHub (Mar 3, 2025):

@dosubot

@ssim2023 commented on GitHub (Mar 3, 2025): @dosubot
yindo commented 2026-02-21 18:28:34 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Mar 3, 2025):

The issue you're encountering with forced segmentation across pages in Dify, despite setting the maximum token length for the parent segment to 4000, is likely due to the system's enforced maximum segment length of 1000 tokens. This limit is set by the INDEXING_MAX_SEGMENTATION_TOKENS_LENGTH configuration, which defaults to 1000 tokens, regardless of any higher settings you might apply in the UI [1][2][3].

To address this, you can override the default setting by modifying the .env file in your Dify setup. Change the INDEXING_MAX_SEGMENTATION_TOKENS_LENGTH to 4000 and restart Docker Compose to apply the changes [4]. This should allow for longer segments and potentially resolve the forced segmentation issue you're experiencing.

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other

@dosubot[bot] commented on GitHub (Mar 3, 2025): <!-- Answer --> The issue you're encountering with forced segmentation across pages in Dify, despite setting the maximum token length for the parent segment to 4000, is likely due to the system's enforced maximum segment length of 1000 tokens. This limit is set by the `INDEXING_MAX_SEGMENTATION_TOKENS_LENGTH` configuration, which defaults to 1000 tokens, regardless of any higher settings you might apply in the UI <sup>[[1]](https://github.com/langgenius/dify-docs/blob/main/jp/guides/knowledge-base/create-knowledge-and-upload-documents/chunking-and-cleaning-text.md)</sup><sup>[[2]](https://github.com/langgenius/dify-docs/blob/main/en/guides/knowledge-base/create-knowledge-and-upload-documents/2.-choose-a-chunk-mode.md)</sup><sup>[[3]](https://github.com/langgenius/dify/issues/12916)</sup>. To address this, you can override the default setting by modifying the `.env` file in your Dify setup. Change the `INDEXING_MAX_SEGMENTATION_TOKENS_LENGTH` to 4000 and restart Docker Compose to apply the changes <sup>[[4]](https://github.com/langgenius/dify/issues/12995)</sup>. This should allow for longer segments and potentially resolve the forced segmentation issue you're experiencing. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.* --- <sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/dfc83249-84c7-4f20-a1c2-f02df3083fb2?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/dfc83249-84c7-4f20-a1c2-f02df3083fb2?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/dfc83249-84c7-4f20-a1c2-f02df3083fb2?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/dfc83249-84c7-4f20-a1c2-f02df3083fb2?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/dfc83249-84c7-4f20-a1c2-f02df3083fb2?feedback_type=hallucination) | [Bug Report](https://app.dosu.dev/response-feedback/dfc83249-84c7-4f20-a1c2-f02df3083fb2?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/dfc83249-84c7-4f20-a1c2-f02df3083fb2?feedback_type=other)</sup>
yindo commented 2026-02-21 18:28:34 -05:00
Author
Owner
Copy Link
Copy Source

@ssim2023 commented on GitHub (Mar 3, 2025):

@dosubot I have set the INDEXING_MAX_SEGMENTATION_TOKENS_LENGTH to 8000 in the .env file and restarted the service, but I still can't avoid this problem

Image

@ssim2023 commented on GitHub (Mar 3, 2025): @dosubot I have set the INDEXING_MAX_SEGMENTATION_TOKENS_LENGTH to 8000 in the .env file and restarted the service, but I still can't avoid this problem ![Image](https://github.com/user-attachments/assets/434b1b00-4186-4709-83ab-67bcad674b97)
yindo commented 2026-02-21 18:28:34 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Mar 3, 2025):

The forced segmentation across pages in Dify persists because the maximum allowed segment length is enforced at 1000 tokens, regardless of setting INDEXING_MAX_SEGMENTATION_TOKENS_LENGTH to 8000 in the .env file. The valid range for segment length is between 50 and 1000 tokens, as enforced both in the UI and at the system level [1].

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other

@dosubot[bot] commented on GitHub (Mar 3, 2025): <!-- Answer --> The forced segmentation across pages in Dify persists because the maximum allowed segment length is enforced at 1000 tokens, regardless of setting `INDEXING_MAX_SEGMENTATION_TOKENS_LENGTH` to 8000 in the `.env` file. The valid range for segment length is between 50 and 1000 tokens, as enforced both in the UI and at the system level <sup>[[1]](https://github.com/langgenius/dify/issues/12916)</sup>. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.* --- <sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/45a13027-1586-43e0-84d8-137a78d75348?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/45a13027-1586-43e0-84d8-137a78d75348?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/45a13027-1586-43e0-84d8-137a78d75348?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/45a13027-1586-43e0-84d8-137a78d75348?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/45a13027-1586-43e0-84d8-137a78d75348?feedback_type=hallucination) | [Bug Report](https://app.dosu.dev/response-feedback/45a13027-1586-43e0-84d8-137a78d75348?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/45a13027-1586-43e0-84d8-137a78d75348?feedback_type=other)</sup>
yindo commented 2026-02-21 18:28:34 -05:00
Author
Owner
Copy Link
Copy Source

@ssim2023 commented on GitHub (Mar 3, 2025):

@dosubot I think there are still problems with code processing for two reasons: 1. Even according to the default maximum number of tokens, Chunk-2 in my picture is only 148 characters, which does not exceed the set value, but it is segmented in advance. 2. When I use long paragraphs or txt in other pages, a parent segment can be divided very large, which shows that the current 148 characters are segmented unexpectedly

@ssim2023 commented on GitHub (Mar 3, 2025): @dosubot I think there are still problems with code processing for two reasons: 1. Even according to the default maximum number of tokens, Chunk-2 in my picture is only 148 characters, which does not exceed the set value, but it is segmented in advance. 2. When I use long paragraphs or txt in other pages, a parent segment can be divided very large, which shows that the current 148 characters are segmented unexpectedly
yindo commented 2026-02-21 18:28:34 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Mar 3, 2025):

The unexpected segmentation you're experiencing, where a chunk is only 148 characters, might be due to specific conditions or rules in the Dify codebase that trigger segmentation beyond just token limits. Here are a few possibilities to consider:

  1. Segmentation Rules: There might be specific rules or conditions in the code that enforce segmentation based on factors other than token count, such as page boundaries or specific delimiters in the PDF content.

  2. PDF Handling: PDFs can have unique formatting and metadata that might affect how text is extracted and segmented. This could lead to unexpected behavior compared to plain text or other document formats.

  3. Known Issues: There is a known issue related to segmentation recognition when using the create_by_text endpoint with parent-child mode. This might be affecting how segments are processed, especially if the JSON payload structure is incorrect [1].

  4. Configuration Overrides: Ensure that there are no other configuration settings or overrides that might be affecting segmentation behavior, such as default delimiters or other processing rules.

If these possibilities don't address the issue, it might be beneficial to review the specific code handling PDF segmentation or check for any updates or patches that might address this behavior.

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other

@dosubot[bot] commented on GitHub (Mar 3, 2025): <!-- Answer --> The unexpected segmentation you're experiencing, where a chunk is only 148 characters, might be due to specific conditions or rules in the Dify codebase that trigger segmentation beyond just token limits. Here are a few possibilities to consider: 1. **Segmentation Rules**: There might be specific rules or conditions in the code that enforce segmentation based on factors other than token count, such as page boundaries or specific delimiters in the PDF content. 2. **PDF Handling**: PDFs can have unique formatting and metadata that might affect how text is extracted and segmented. This could lead to unexpected behavior compared to plain text or other document formats. 3. **Known Issues**: There is a known issue related to segmentation recognition when using the `create_by_text` endpoint with parent-child mode. This might be affecting how segments are processed, especially if the JSON payload structure is incorrect <sup>[[1]](https://github.com/langgenius/dify/issues/13013)</sup>. 4. **Configuration Overrides**: Ensure that there are no other configuration settings or overrides that might be affecting segmentation behavior, such as default delimiters or other processing rules. If these possibilities don't address the issue, it might be beneficial to review the specific code handling PDF segmentation or check for any updates or patches that might address this behavior. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.* --- <sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/25e88d82-967a-4928-93f5-b7ad4be2605a?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/25e88d82-967a-4928-93f5-b7ad4be2605a?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/25e88d82-967a-4928-93f5-b7ad4be2605a?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/25e88d82-967a-4928-93f5-b7ad4be2605a?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/25e88d82-967a-4928-93f5-b7ad4be2605a?feedback_type=hallucination) | [Bug Report](https://app.dosu.dev/response-feedback/25e88d82-967a-4928-93f5-b7ad4be2605a?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/25e88d82-967a-4928-93f5-b7ad4be2605a?feedback_type=other)</sup>
yindo commented 2026-02-21 18:28:35 -05:00
Author
Owner
Copy Link
Copy Source

@ssim2023 commented on GitHub (Mar 3, 2025):

@dosubot This kind of accidental forced segmentation may affect the vectorization and then affect the accuracy and completeness of the retrieved data, so it is also a problem I want to solve. Would you consider fixing this problem? (In addition, when I choose the father-child segmentation, and the parent paragraph chooses the full text, there will be no forced segmentation across pages, which also shows that the current problem can be specially corrected)

@ssim2023 commented on GitHub (Mar 3, 2025): @dosubot This kind of accidental forced segmentation may affect the vectorization and then affect the accuracy and completeness of the retrieved data, so it is also a problem I want to solve. Would you consider fixing this problem? (In addition, when I choose the father-child segmentation, and the parent paragraph chooses the full text, there will be no forced segmentation across pages, which also shows that the current problem can be specially corrected)
yindo commented 2026-02-21 18:28:35 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Mar 3, 2025):

To resolve the issue of accidental forced segmentation in Dify, which affects vectorization and data retrieval accuracy, you should upgrade Dify to version v0.15.2. This version addresses the problem where custom segmentation settings revert to default values. Additionally, ensure that the JSON payload structure is correct when using the Knowledge Base API's create_by_text endpoint with parent-child mode. The parent_mode and subchunk_segmentation parameters should be placed under the rules object, not directly under process_rule. Here's the correct structure:

{
    "process_rule": {
        "mode": "hierarchical",
        "rules": {
            "segmentation": {
                "separator": "whatever",
                "max_tokens": 2000
            },
            "parent_mode": "paragraph",
            "subchunk_segmentation": {
                "separator": "***",
                "max_tokens": 500
            }
        }
    }
}

By using the correct JSON structure and the father-child segmentation mode, which employs a dual-layer segmentation structure, you can maintain larger text units as parent chunks for context while using smaller child chunks for precise retrieval. This approach helps avoid the issue and ensures accurate data retrieval [1][2][3].

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other

@dosubot[bot] commented on GitHub (Mar 3, 2025): <!-- Answer --> To resolve the issue of accidental forced segmentation in Dify, which affects vectorization and data retrieval accuracy, you should upgrade Dify to version v0.15.2. This version addresses the problem where custom segmentation settings revert to default values. Additionally, ensure that the JSON payload structure is correct when using the Knowledge Base API's `create_by_text` endpoint with parent-child mode. The `parent_mode` and `subchunk_segmentation` parameters should be placed under the `rules` object, not directly under `process_rule`. Here's the correct structure: ```json { "process_rule": { "mode": "hierarchical", "rules": { "segmentation": { "separator": "whatever", "max_tokens": 2000 }, "parent_mode": "paragraph", "subchunk_segmentation": { "separator": "***", "max_tokens": 500 } } } } ``` By using the correct JSON structure and the father-child segmentation mode, which employs a dual-layer segmentation structure, you can maintain larger text units as parent chunks for context while using smaller child chunks for precise retrieval. This approach helps avoid the issue and ensures accurate data retrieval <sup>[[1]](https://github.com/langgenius/dify/issues/13007)</sup><sup>[[2]](https://github.com/langgenius/dify-docs/blob/main/zh_CN/guides/knowledge-base/create-knowledge-and-upload-documents/chunking-and-cleaning-text.md)</sup><sup>[[3]](https://github.com/langgenius/dify/issues/13013)</sup>. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.* --- <sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/c9d49515-9730-44d9-8f6e-53b339f0eb2f?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/c9d49515-9730-44d9-8f6e-53b339f0eb2f?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/c9d49515-9730-44d9-8f6e-53b339f0eb2f?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/c9d49515-9730-44d9-8f6e-53b339f0eb2f?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/c9d49515-9730-44d9-8f6e-53b339f0eb2f?feedback_type=hallucination) | [Bug Report](https://app.dosu.dev/response-feedback/c9d49515-9730-44d9-8f6e-53b339f0eb2f?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/c9d49515-9730-44d9-8f6e-53b339f0eb2f?feedback_type=other)</sup>
yindo commented 2026-02-21 18:28:35 -05:00
Author
Owner
Copy Link
Copy Source

@ssim2023 commented on GitHub (Mar 3, 2025):

@dosubot The version I'm currently using is 0.15.3 😀, after 0.15.2

@ssim2023 commented on GitHub (Mar 3, 2025): @dosubot The version I'm currently using is 0.15.3 😀, after 0.15.2
yindo commented 2026-02-21 18:28:35 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Mar 3, 2025):

Since you're already using version 0.15.3, which should include the fixes from 0.15.2, the issue might be more complex. Here are a few additional steps you can take:

  1. Review Code for Segmentation Logic: Check the specific code handling PDF segmentation for any conditions that might trigger unexpected segmentation. Look for logic related to page boundaries or specific delimiters.

  2. Debugging: Enable detailed logging to trace how the segmentation is being applied. This might help identify any unexpected behavior or conditions leading to forced segmentation.

  3. Community and Support: Consider reaching out to the community or support channels for insights or similar experiences. There might be undocumented nuances or recent patches that could help.

  4. Custom Handling: If feasible, implement a custom segmentation handler that better suits your specific needs, especially if the default behavior doesn't align with your requirements.

If these steps don't resolve the issue, it might be necessary to report this as a potential bug for further investigation by the development team.

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other

@dosubot[bot] commented on GitHub (Mar 3, 2025): <!-- Answer --> Since you're already using version 0.15.3, which should include the fixes from 0.15.2, the issue might be more complex. Here are a few additional steps you can take: 1. **Review Code for Segmentation Logic**: Check the specific code handling PDF segmentation for any conditions that might trigger unexpected segmentation. Look for logic related to page boundaries or specific delimiters. 2. **Debugging**: Enable detailed logging to trace how the segmentation is being applied. This might help identify any unexpected behavior or conditions leading to forced segmentation. 3. **Community and Support**: Consider reaching out to the community or support channels for insights or similar experiences. There might be undocumented nuances or recent patches that could help. 4. **Custom Handling**: If feasible, implement a custom segmentation handler that better suits your specific needs, especially if the default behavior doesn't align with your requirements. If these steps don't resolve the issue, it might be necessary to report this as a potential bug for further investigation by the development team. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.* --- <sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/d52e99de-7993-4856-a170-964f6587f2c3?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/d52e99de-7993-4856-a170-964f6587f2c3?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/d52e99de-7993-4856-a170-964f6587f2c3?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/d52e99de-7993-4856-a170-964f6587f2c3?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/d52e99de-7993-4856-a170-964f6587f2c3?feedback_type=hallucination) | [Bug Report](https://app.dosu.dev/response-feedback/d52e99de-7993-4856-a170-964f6587f2c3?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/d52e99de-7993-4856-a170-964f6587f2c3?feedback_type=other)</sup>
yindo commented 2026-02-21 18:28:35 -05:00
Author
Owner
Copy Link
Copy Source

@ssim2023 commented on GitHub (Mar 3, 2025):

@dosu Help me see this question

@ssim2023 commented on GitHub (Mar 3, 2025): @dosu Help me see this question
yindo commented 2026-02-21 18:28:35 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Mar 3, 2025):

The issue you're experiencing with forced segmentation across pages in Dify, despite setting the maximum token length, might be due to specific conditions in the codebase that enforce segmentation beyond token limits. Here are some points to consider:

  1. Segmentation Rules: Dify may have rules that trigger segmentation based on page boundaries or specific delimiters, not just token count [1].

  2. PDF Handling: The way Dify handles PDF metadata and formatting during text extraction could affect segmentation behavior, potentially leading to unexpected results [2].

  3. Configuration Limits: The maximum segment length is enforced at 1000 tokens, regardless of higher settings in the .env file. This limit is set by the INDEXING_MAX_SEGMENTATION_TOKENS_LENGTH configuration [3].

  4. Parent-Child Mode: Using the father-child segmentation mode with the full document as the parent chunk can help avoid forced segmentation across pages, as it maintains larger text units for context [2].

If these considerations don't resolve the issue, reviewing the specific code handling PDF segmentation or checking for any recent patches might be necessary.

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other

@dosubot[bot] commented on GitHub (Mar 3, 2025): <!-- Answer --> The issue you're experiencing with forced segmentation across pages in Dify, despite setting the maximum token length, might be due to specific conditions in the codebase that enforce segmentation beyond token limits. Here are some points to consider: 1. **Segmentation Rules**: Dify may have rules that trigger segmentation based on page boundaries or specific delimiters, not just token count <sup>[[1]](https://github.com/langgenius/dify-docs/blob/main/jp/guides/knowledge-base/create-knowledge-and-upload-documents/chunking-and-cleaning-text.md)</sup>. 2. **PDF Handling**: The way Dify handles PDF metadata and formatting during text extraction could affect segmentation behavior, potentially leading to unexpected results <sup>[[2]](https://github.com/langgenius/dify-docs/blob/main/en/guides/knowledge-base/create-knowledge-and-upload-documents/2.-choose-a-chunk-mode.md)</sup>. 3. **Configuration Limits**: The maximum segment length is enforced at 1000 tokens, regardless of higher settings in the `.env` file. This limit is set by the `INDEXING_MAX_SEGMENTATION_TOKENS_LENGTH` configuration <sup>[[3]](https://github.com/langgenius/dify/issues/12916)</sup>. 4. **Parent-Child Mode**: Using the father-child segmentation mode with the full document as the parent chunk can help avoid forced segmentation across pages, as it maintains larger text units for context <sup>[[2]](https://github.com/langgenius/dify-docs/blob/main/en/guides/knowledge-base/create-knowledge-and-upload-documents/2.-choose-a-chunk-mode.md)</sup>. If these considerations don't resolve the issue, reviewing the specific code handling PDF segmentation or checking for any recent patches might be necessary. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.* --- <sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/2a4a2727-1f9d-49a7-aa41-c0ae7382abcf?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/2a4a2727-1f9d-49a7-aa41-c0ae7382abcf?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/2a4a2727-1f9d-49a7-aa41-c0ae7382abcf?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/2a4a2727-1f9d-49a7-aa41-c0ae7382abcf?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/2a4a2727-1f9d-49a7-aa41-c0ae7382abcf?feedback_type=hallucination) | [Bug Report](https://app.dosu.dev/response-feedback/2a4a2727-1f9d-49a7-aa41-c0ae7382abcf?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/2a4a2727-1f9d-49a7-aa41-c0ae7382abcf?feedback_type=other)</sup>
yindo commented 2026-02-21 18:28:35 -05:00
Author
Owner
Copy Link
Copy Source

@ahahaboom233 commented on GitHub (Mar 12, 2025):

I have the same problem, my knowledge didn't reach the limit of the number of tokens, but it was cut in advance, does this have anything to do with the splitter I set, I set it to ====.
@dosu

@ahahaboom233 commented on GitHub (Mar 12, 2025): I have the same problem, my knowledge didn't reach the limit of the number of tokens, but it was cut in advance, does this have anything to do with the splitter I set, I set it to ====. @dosu
yindo commented 2026-02-21 18:28:35 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Apr 13, 2025):

Hi, @ssim2023. I'm Dosu, and I'm helping the Dify team manage their backlog. I'm marking this issue as stale.

Issue Summary

  • Issue with PDF text segmentation across pages in Dify version 0.15.3.
  • Occurs in a self-hosted Docker environment despite high maximum token length.
  • Configuration adjustments in .env file did not resolve the issue.
  • Another user, "ahahaboom233," reported a similar problem, possibly linked to custom splitter settings.

Next Steps

  • Please confirm if this issue is still relevant with the latest version of Dify. If so, you can keep the discussion open by commenting here.
  • If there is no further activity, this issue will be automatically closed in 15 days.

Thank you for your understanding and contribution!

@dosubot[bot] commented on GitHub (Apr 13, 2025): Hi, @ssim2023. I'm [Dosu](https://dosu.dev), and I'm helping the Dify team manage their backlog. I'm marking this issue as stale. **Issue Summary** - Issue with PDF text segmentation across pages in Dify version 0.15.3. - Occurs in a self-hosted Docker environment despite high maximum token length. - Configuration adjustments in `.env` file did not resolve the issue. - Another user, "ahahaboom233," reported a similar problem, possibly linked to custom splitter settings. **Next Steps** - Please confirm if this issue is still relevant with the latest version of Dify. If so, you can keep the discussion open by commenting here. - If there is no further activity, this issue will be automatically closed in 15 days. Thank you for your understanding and contribution!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#8911
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?