langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 12:05:30 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Markdown file parsing error, chunk is not good. #4249

New Issue
Closed
opened 2026-02-21 18:05:23 -05:00 by yindo · 15 comments
yindo commented 2026-02-21 18:05:23 -05:00
Owner
Copy Link
Copy Source

Originally created by @zesen-lee on GitHub (Jun 25, 2024).

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • Please do not modify this template :) and fill in all the required fields.

Dify version

0.6.11

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

import https://github.com/blankbro/mqtt-storm/blob/master/README.md

image

✔️ Expected Behavior

md截图

Keep the picture link in markdown.
Keep markdown format.

❌ Actual Behavior

image
image

Originally created by @zesen-lee on GitHub (Jun 25, 2024). ### Self Checks - [X] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [X] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [X] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [X] Please do not modify this template :) and fill in all the required fields. ### Dify version 0.6.11 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce import https://github.com/blankbro/mqtt-storm/blob/master/README.md ![image](https://github.com/langgenius/dify/assets/31715758/cb32383d-45ea-4401-b602-501a013578c4) ### ✔️ Expected Behavior <img width="877" alt="md截图" src="https://github.com/langgenius/dify/assets/31715758/d1c3258a-8feb-4f9a-90ee-4f475710e12c"> Keep the picture link in markdown. Keep markdown format. ### ❌ Actual Behavior ![image](https://github.com/langgenius/dify/assets/31715758/3a659f45-2706-4eae-bbdb-c5f7960defaf) ![image](https://github.com/langgenius/dify/assets/31715758/8455023f-d0d4-45c5-88df-32768f18fd6f)
yindo added the 🐞 bug label 2026-02-21 18:05:23 -05:00
yindo closed this issue 2026-02-21 18:05:23 -05:00
yindo commented 2026-02-21 18:05:23 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Jun 25, 2024):

Hello, @blankbro! I'm here to help you with any bugs, questions, or contributions!

To resolve the "chunk is not good" Markdown file parsing error in Dify version 0.6.11 when importing the README.md file from the mqtt-storm GitHub repository, follow these steps:

  1. Check for Unclosed Tags: Ensure that all HTML tags within the Markdown file are properly closed.
  2. Validate Markdown Syntax: Use a Markdown validator to check for any syntax errors.
  3. Simplify Content: Temporarily remove complex elements like tables, images, and HTML to identify if any specific part is causing the issue.

Here is an example of a simplified and properly structured Markdown file:

# Project Title

![cover-image](https://example.com/image.png)

<p align="center">
  <a href="https://example.com">Link 1</a> ·
  <a href="https://example.com">Link 2</a> ·
  <a href="https://example.com">Link 3</a>
</p>

## Features

- Feature 1
- Feature 2
- Feature 3

## Usage

Instructions on how to use the project.

## Contributing

Guidelines for contributing to the project.

## License

Information about the project's license.

If the error persists, it may be necessary to debug further by isolating specific sections of the Markdown file to identify the problematic chunk [1].

Additionally, Dify uses the react-markdown library for Markdown parsing. Known issues with react-markdown version 8.0.6 include:

  1. HTML Parsing: By default, react-markdown does not support parsing HTML within Markdown.
  2. Plugins Compatibility: Some plugins or extensions might not be fully compatible.
  3. Syntax Highlighting: Additional configuration or plugins like react-syntax-highlighter might be needed [2].

Dify also provides built-in tools for validating or sanitizing Markdown files before parsing them, such as replacing continuous spaces, newlines, and tabs, and deleting all URLs and email addresses [3].

For more detailed error messages, check the Logs section in the Dify application, accessible via the left-side navigation [4].

To continue talking to Dosu, mention @dosu.

@dosubot[bot] commented on GitHub (Jun 25, 2024): <!-- Greeting --> Hello, @blankbro! I'm here to help you with any bugs, questions, or contributions! <!-- Answer --> To resolve the "chunk is not good" Markdown file parsing error in Dify version 0.6.11 when importing the README.md file from the mqtt-storm GitHub repository, follow these steps: 1. **Check for Unclosed Tags**: Ensure that all HTML tags within the Markdown file are properly closed. 2. **Validate Markdown Syntax**: Use a Markdown validator to check for any syntax errors. 3. **Simplify Content**: Temporarily remove complex elements like tables, images, and HTML to identify if any specific part is causing the issue. Here is an example of a simplified and properly structured Markdown file: ```md # Project Title ![cover-image](https://example.com/image.png) <p align="center"> <a href="https://example.com">Link 1</a> · <a href="https://example.com">Link 2</a> · <a href="https://example.com">Link 3</a> </p> ## Features - Feature 1 - Feature 2 - Feature 3 ## Usage Instructions on how to use the project. ## Contributing Guidelines for contributing to the project. ## License Information about the project's license. ``` If the error persists, it may be necessary to debug further by isolating specific sections of the Markdown file to identify the problematic chunk <sup>[[1]](https://github.com/langgenius/dify/blob/main/README_ES.md)</sup>. Additionally, Dify uses the `react-markdown` library for Markdown parsing. Known issues with `react-markdown` version 8.0.6 include: 1. **HTML Parsing**: By default, `react-markdown` does not support parsing HTML within Markdown. 2. **Plugins Compatibility**: Some plugins or extensions might not be fully compatible. 3. **Syntax Highlighting**: Additional configuration or plugins like `react-syntax-highlighter` might be needed <sup>[[2]](https://github.com/langgenius/dify/blob/main/web/package.json)</sup>. Dify also provides built-in tools for validating or sanitizing Markdown files before parsing them, such as replacing continuous spaces, newlines, and tabs, and deleting all URLs and email addresses <sup>[[3]](https://github.com/langgenius/dify-docs/blob/main/en/guides/knowledge-base/create-knowledge-and-upload-documents.md)</sup>. For more detailed error messages, check the **Logs** section in the Dify application, accessible via the left-side navigation <sup>[[4]](https://github.com/langgenius/dify-docs/blob/main/en/guides/knowledge-base/maintain-dataset-via-api.md)</sup>. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.*
yindo commented 2026-02-21 18:05:23 -05:00
Author
Owner
Copy Link
Copy Source

@crazywoola commented on GitHub (Jun 25, 2024):

Please use the custom separate identifier to do it. :)

@crazywoola commented on GitHub (Jun 25, 2024): Please use the custom separate identifier to do it. :)
yindo commented 2026-02-21 18:05:23 -05:00
Author
Owner
Copy Link
Copy Source

@zesen-lee commented on GitHub (Jun 25, 2024):

Please use the custom separate identifier to do it. :)

@crazywoola

image
image

Is that it? But my problem still exists.

@zesen-lee commented on GitHub (Jun 25, 2024): > Please use the custom separate identifier to do it. :) @crazywoola ![image](https://github.com/langgenius/dify/assets/31715758/45efa4b4-cb58-40e8-8bf4-1236a82135c1) ![image](https://github.com/langgenius/dify/assets/31715758/df182f11-f34d-481c-af55-4bd63b9f64d9) Is that it? But my problem still exists.
yindo commented 2026-02-21 18:05:23 -05:00
Author
Owner
Copy Link
Copy Source

@crazywoola commented on GitHub (Jun 29, 2024):

Yes, you need to choose which identifier you would like to use. For example, you could fill in ###

@crazywoola commented on GitHub (Jun 29, 2024): Yes, you need to choose which identifier you would like to use. For example, you could fill in ` ###`
yindo commented 2026-02-21 18:05:23 -05:00
Author
Owner
Copy Link
Copy Source

@zesen-lee commented on GitHub (Jun 29, 2024):

Yes, you need to choose which identifier you would like to use. For example, you could fill in ###

image

problem still exists.

@zesen-lee commented on GitHub (Jun 29, 2024): > Yes, you need to choose which identifier you would like to use. For example, you could fill in ` ###` <img width="941" alt="image" src="https://github.com/langgenius/dify/assets/31715758/1b13948a-8767-46e1-bd4a-d4d2298bfad9"> problem still exists.
yindo commented 2026-02-21 18:05:24 -05:00
Author
Owner
Copy Link
Copy Source

@crazywoola commented on GitHub (Jun 29, 2024):

image Your doc is segmenting by `##`, I mean for example `###`, you should change it according to your needs.
@crazywoola commented on GitHub (Jun 29, 2024): <img width="844" alt="image" src="https://github.com/langgenius/dify/assets/100913391/ff0c282d-5a55-4e23-8fba-9a83c2d7e329"> Your doc is segmenting by `##`, I mean for example `###`, you should change it according to your needs.
yindo commented 2026-02-21 18:05:24 -05:00
Author
Owner
Copy Link
Copy Source

@zesen-lee commented on GitHub (Jun 30, 2024):

image Your doc is segmenting by ##, I mean for example ###, you should change it according to your needs.

image problem still exists.
@zesen-lee commented on GitHub (Jun 30, 2024): > <img alt="image" width="844" src="https://private-user-images.githubusercontent.com/100913391/344442393-ff0c282d-5a55-4e23-8fba-9a83c2d7e329.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTk3MjA1NTgsIm5iZiI6MTcxOTcyMDI1OCwicGF0aCI6Ii8xMDA5MTMzOTEvMzQ0NDQyMzkzLWZmMGMyODJkLTVhNTUtNGUyMy04ZmJhLTlhODNjMmQ3ZTMyOS5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNjMwJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDYzMFQwNDA0MThaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT01N2UxYThmMDcyYjFhNmE1OGUzODE4YmNkNjdkYzNlYWVlYWZiYWRkMGE4ZjY2NGE4NWZjNzI0MmY4YmIzMjM1JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.IroAPHJPnsgBD5ZcK5oP2FquTiNxljQbmNNymvWfy38"> Your doc is segmenting by `##`, I mean for example `###`, you should change it according to your needs. <img width="1417" alt="image" src="https://github.com/langgenius/dify/assets/31715758/ad2f5b84-e6c0-4d3f-be70-485cd076a2c2"> problem still exists.
yindo commented 2026-02-21 18:05:24 -05:00
Author
Owner
Copy Link
Copy Source

@tangyoha commented on GitHub (Jul 2, 2024):

@crazywoola hi, After my investigation, I found that it was caused by the commit https://github.com/langgenius/dify/pull/2528. The md file was processed and the pictures and links were automatically removed.
image

@tangyoha commented on GitHub (Jul 2, 2024): @crazywoola hi, After my investigation, I found that it was caused by the commit https://github.com/langgenius/dify/pull/2528. The md file was processed and the pictures and links were automatically removed. ![image](https://github.com/langgenius/dify/assets/39958403/219bdc20-a3b3-4609-9954-e50ac61a978d)
yindo commented 2026-02-21 18:05:24 -05:00
Author
Owner
Copy Link
Copy Source

@zesen-lee commented on GitHub (Jul 16, 2024):

I have upgraded to version 0.6.14, and the issue with image links being removed has been resolved.
However, at the same time, a new problem has been discovered: the code blocks have been cut off.

image

@zesen-lee commented on GitHub (Jul 16, 2024): I have upgraded to version 0.6.14, and the issue with image links being removed has been resolved. However, at the same time, a new problem has been discovered: the code blocks have been cut off. ![image](https://github.com/user-attachments/assets/9e399788-9df4-4027-9462-4f527e6545b6)
yindo commented 2026-02-21 18:05:24 -05:00
Author
Owner
Copy Link
Copy Source

@tangyoha commented on GitHub (Jul 16, 2024):

@blankbro hi， can you give me a example？

@tangyoha commented on GitHub (Jul 16, 2024): @blankbro hi， can you give me a example？
yindo commented 2026-02-21 18:05:24 -05:00
Author
Owner
Copy Link
Copy Source

@zesen-lee commented on GitHub (Jul 16, 2024):

@blankbro hi， can you give me a example？

this md file https://github.com/blankbro/mqtt-storm/blob/master/README.md

@zesen-lee commented on GitHub (Jul 16, 2024): > @blankbro hi， can you give me a example？ this md file https://github.com/blankbro/mqtt-storm/blob/master/README.md
yindo commented 2026-02-21 18:05:24 -05:00
Author
Owner
Copy Link
Copy Source

@tangyoha commented on GitHub (Jul 17, 2024):

@blankbro hi, The code is split. Are you using this split document?

image Your doc is segmenting by ##, I mean for example ###, you should change it according to your needs.

image problem still exists.

@tangyoha commented on GitHub (Jul 17, 2024): @blankbro hi, The code is split. Are you using this split document? > > <img alt="image" width="54.145835876464844" height="21" src="https://private-user-images.githubusercontent.com/100913391/344442393-ff0c282d-5a55-4e23-8fba-9a83c2d7e329.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTk3MjA1NTgsIm5iZiI6MTcxOTcyMDI1OCwicGF0aCI6Ii8xMDA5MTMzOTEvMzQ0NDQyMzkzLWZmMGMyODJkLTVhNTUtNGUyMy04ZmJhLTlhODNjMmQ3ZTMyOS5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNjMwJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDYzMFQwNDA0MThaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT01N2UxYThmMDcyYjFhNmE1OGUzODE4YmNkNjdkYzNlYWVlYWZiYWRkMGE4ZjY2NGE4NWZjNzI0MmY4YmIzMjM1JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.IroAPHJPnsgBD5ZcK5oP2FquTiNxljQbmNNymvWfy38"> Your doc is segmenting by `##`, I mean for example `###`, you should change it according to your needs. > > <img alt="image" width="1417" src="https://private-user-images.githubusercontent.com/31715758/344442654-ad2f5b84-e6c0-4d3f-be70-485cd076a2c2.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjEyMDY1MzgsIm5iZiI6MTcyMTIwNjIzOCwicGF0aCI6Ii8zMTcxNTc1OC8zNDQ0NDI2NTQtYWQyZjViODQtZTZjMC00ZDNmLWJlNzAtNDg1Y2QwNzZhMmMyLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA3MTclMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNzE3VDA4NTAzOFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWEyNmRiNGE0N2Y3ZTA3Yjg1ZWY5NGNhOTc3YjYxM2Y0Mzk4YTRlYTk4Yjg3MjFmMWRhOTA2MWUzMTBmYzgxNzgmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.JizZAREkkkLeLkvlZ6qYlDV4Ax-YXP7FMLR7ayJ2RBQ"> problem still exists.
yindo commented 2026-02-21 18:05:24 -05:00
Author
Owner
Copy Link
Copy Source

@zesen-lee commented on GitHub (Jul 17, 2024):

@blankbro hi, The code is split. Are you using this split document?

@tangyoha Yes, you can to try.

@zesen-lee commented on GitHub (Jul 17, 2024): > @blankbro hi, The code is split. Are you using this split document? @tangyoha Yes, you can to try.
yindo commented 2026-02-21 18:05:24 -05:00
Author
Owner
Copy Link
Copy Source

@tangyoha commented on GitHub (Jul 18, 2024):

@blankbro hi, The problem does exist, please raise another issue.

@tangyoha commented on GitHub (Jul 18, 2024): @blankbro hi, The problem does exist, please raise another issue.
yindo commented 2026-02-21 18:05:24 -05:00
Author
Owner
Copy Link
Copy Source

@zesen-lee commented on GitHub (Jul 18, 2024):

@blankbro hi, The problem does exist, please raise another issue.

new issue is https://github.com/langgenius/dify/issues/6421

@zesen-lee commented on GitHub (Jul 18, 2024): > @blankbro hi, The problem does exist, please raise another issue. new issue is https://github.com/langgenius/dify/issues/6421
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#4249
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?