langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-08-24 22:11:36 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Markdown ingest is weird: text before first header is missing #7283

New Issue
Closed
opened 2026-02-21 18:19:48 -05:00 by yindo · 1 comment
yindo commented 2026-02-21 18:19:48 -05:00
Owner
Copy Link
Copy Source

Originally created by @cpwan on GitHub (Dec 18, 2024).

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
  • Please do not modify this template :) and fill in all the required fields.

Dify version

0.14.1

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

Ingest the following markdown into "Knowledge"

Any text above the first # is gone!

# apple 

## banana

## Cherry

### dify

i am a bad boy

===

lol???

The rendered example looks like this
image

✔️ Expected Behavior

The Any text above the first # is gone! text should be ingested.

❌ Actual Behavior

It is gone. As a consequence, there is no chunk for any text ahead of the first #.

Originally created by @cpwan on GitHub (Dec 18, 2024). ### Self Checks - [X] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [X] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [X] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [X] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:） - [X] Please do not modify this template :) and fill in all the required fields. ### Dify version 0.14.1 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce Ingest the following markdown into "Knowledge" ```md Any text above the first # is gone! # apple ## banana ## Cherry ### dify i am a bad boy === lol??? ``` The rendered example looks like this ![image](https://github.com/user-attachments/assets/0d38b94a-8594-492d-863f-c2300dafa5d0) ### ✔️ Expected Behavior The `Any text above the first # is gone!` text should be ingested. ### ❌ Actual Behavior It is gone. As a consequence, there is no chunk for any text ahead of the first #.
yindo added the 🐞 bug label 2026-02-21 18:19:48 -05:00
yindo closed this issue 2026-02-21 18:19:48 -05:00
yindo commented 2026-02-21 18:19:48 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Dec 18, 2024):

The issue you're experiencing with markdown ingestion in Dify, where text before the first markdown header (#) is not being ingested, might be related to how the markdown processing is implemented. Unfortunately, I couldn't find specific details in the codebase about how text before the first header is handled during ingestion.

However, the MarkdownExtractor class is responsible for loading markdown files and converting them into Document objects, which involves parsing the markdown into tuples of headers and their corresponding text content [1]. This might be where the issue arises, as the text before the first header might not be included in these tuples.

To address this, you might need to look into the MarkdownExtractor class and see if there's a way to modify it to include text before the first header. Alternatively, you could consider adding a header at the very beginning of your markdown content as a temporary workaround.

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other

@dosubot[bot] commented on GitHub (Dec 18, 2024): <!-- Answer --> The issue you're experiencing with markdown ingestion in Dify, where text before the first markdown header (`#`) is not being ingested, might be related to how the markdown processing is implemented. Unfortunately, I couldn't find specific details in the codebase about how text before the first header is handled during ingestion. However, the `MarkdownExtractor` class is responsible for loading markdown files and converting them into `Document` objects, which involves parsing the markdown into tuples of headers and their corresponding text content <sup>[[1]](https://github.com/langgenius/dify/blob/main/api/core/rag/extractor/markdown_extractor.py)</sup>. This might be where the issue arises, as the text before the first header might not be included in these tuples. To address this, you might need to look into the `MarkdownExtractor` class and see if there's a way to modify it to include text before the first header. Alternatively, you could consider adding a header at the very beginning of your markdown content as a temporary workaround. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.* --- <sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/99210dca-406a-4bb2-983a-70c007ef03b7?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/99210dca-406a-4bb2-983a-70c007ef03b7?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/99210dca-406a-4bb2-983a-70c007ef03b7?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/99210dca-406a-4bb2-983a-70c007ef03b7?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/99210dca-406a-4bb2-983a-70c007ef03b7?feedback_type=hallucination) | [Other](https://app.dosu.dev/response-feedback/99210dca-406a-4bb2-983a-70c007ef03b7?feedback_type=other)</sup>
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#7283
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?