[PR #1685] [MERGED] feat(tools): add markdown_chunker - Advanced Markdown Chunker #1779

New Issue

2026-02-22T18:07:36-05:00

yindo commented

2026-02-22 18:07:36 -05:00

📋 Pull Request Information

Original PR: https://github.com/langgenius/dify-plugins/pull/1685
Author: @asukhodko
Created: 12/7/2025
Status: ✅ Merged
Merged: 12/24/2025
Merged by: @crazywoola

Base: main ← Head: main

📝 Commits (5)

a9217cd feat(tools): add markdown_chunker - Advanced Markdown Chunker
ef48635 Merge branch 'langgenius:main' into main
6b6e52f feat(tools): add markdown_chunker - Advanced Markdown Chunker
9c3e968 Merge branch 'langgenius:main' into main
b5a40ca feat(tools): add markdown_chunker - Advanced Markdown Chunker

📊 Changes

1 file changed (+0 additions, -0 deletions)

View changed files

➕ asukhodko/markdown-chunker-2.1.4.difypkg (+0 -0)

📄 Description

Structure-aware Markdown chunking for RAG with auto-strategy selection, adaptive sizing, hierarchical chunks, and rich metadata support.

Plugin Submission Form

1. Metadata

Plugin Author: asukhodko
Plugin Name: markdown_chunker
Repository URL: https://github.com/asukhodko/dify-markdown-chunker

2. Submission Type

New plugin submission
Version update for existing plugin

3. Description

Advanced Markdown Chunker is a structure-aware, multi-strategy Markdown chunking tool for Dify RAG workflows. It analyzes each document, automatically selects the most suitable chunking strategy from four specialized modes, and can adapt chunk size based on content complexity to keep chunks both structurally coherent and retrieval-friendly. :contentReference[oaicite:1]{index=1}

By default, each chunk can embed metadata (header paths, content type, line numbers, overlap context) directly into the chunk text, improving semantic retrieval quality in vector databases. All processing happens locally inside Dify; no external APIs or LLM calls are used for chunking. :contentReference[oaicite:2]{index=2}

Core capability: content-aware, multi-strategy chunking

The plugin inspects the document structure and content mix (code, lists, headings, plain text) and chooses between four internal strategies:

Strategy	Auto-activates when	Optimized for
Code-Aware	`code_ratio ≥ 30%`	API docs, technical references, tutorials
List-Aware	`list_ratio > 40%`	Changelogs, feature lists, task lists, outlines
Structural	`≥ 3` hierarchical header levels detected	Documentation, guides, specifications
Fallback	default for everything else	Mixed content and simple text documents

Each strategy preserves Markdown semantics: headers, lists, tables, and code blocks are kept intact, and the chunker avoids splitting in unnatural places (e.g., mid-function or in the middle of a table row). :contentReference[oaicite:3]{index=3}

Key features

List-aware chunking – preserves nested list hierarchies and binds context (intro paragraphs + their lists), which is crucial for changelogs, release notes, and feature documentation where list structure carries meaning. :contentReference[oaicite:4]{index=4}
Code-aware chunking – treats fenced code blocks as atomic units, preserves language metadata, and intelligently binds code to its explanations (Before/After patterns, Code+Output pairs, sequential examples), avoiding mid-function splits that hurt comprehension. :contentReference[oaicite:5]{index=5}
Structure-preserving segmentation – respects heading hierarchy, keeps tables and lists intact, supports nested fencing (quadruple/quintuple backticks and tildes), and avoids breaking Markdown blocks in unnatural positions. :contentReference[oaicite:6]{index=6}
Adaptive chunk sizing (optional) – can automatically adjust chunk size based on content complexity:
- code-heavy content → larger chunks (up to ~1.5× base size),
- simple text → smaller chunks (down to ~0.5× base size),
  with configurable weights and bounds; disabled by default for backward compatibility. :contentReference[oaicite:7]{index=7}
Hierarchical chunks (parent–child) – supports parent–child relationships between chunks for multi-level retrieval (overview vs detail), with O(1) lookup and programmatic navigation (ancestors, children, siblings), while remaining backward compatible with flat chunking. :contentReference[oaicite:8]{index=8}
Streaming processing for large files – can process large Markdown files in a streaming fashion (e.g. >10MB) with controlled memory usage and smart window boundary detection, keeping chunk quality stable even on long documents. :contentReference[oaicite:9]{index=9}
LaTeX & formulas as atomic blocks – preserves LaTeX display math ($$...$$) and environments (\begin{equation}, \begin{align}, etc.) as whole units across all strategies, which is critical for scientific and technical documents. :contentReference[oaicite:10]{index=10}
Table grouping – optionally groups related tables in the same chunk with configurable distance and section-awareness, which is useful for API docs (parameters/response/error tables). :contentReference[oaicite:11]{index=11}
Adaptive overlap – dynamically scales contextual overlap with chunk size (up to ~35%), maintaining consistent retrieval quality across small and large chunks. :contentReference[oaicite:12]{index=12}
Rich embedded metadata – each chunk can include content type, header path, source line numbers, and overlap context, improving filtering, debugging and retrieval behavior in vector stores. :contentReference[oaicite:13]{index=13}
High reliability & test coverage – over 800 tests (including property-based tests with Hypothesis) validate correctness, edge cases, and performance regressions; automatic fallback mode ensures graceful degradation on errors. :contentReference[oaicite:14]{index=14}
Fully local processing – all parsing and chunking are performed locally within the user’s Dify instance. No external API calls, no LLM dependency for chunking, and no data leaves the configured infrastructure. :contentReference[oaicite:15]{index=15}

Typical use cases

RAG knowledge bases built on mixed Markdown content
API and SDK documentation with heavy code examples
Changelogs and release notes with nested feature lists
Technical specifications, scientific papers, and architecture docs (including LaTeX formulas and structured tables)
Large Markdown manuals and multi-section guides that benefit from hierarchical chunks and streaming processing

4. Checklist

I have read and followed the Publish to Dify Marketplace guidelines
I have read and comply with the Plugin Developer Agreement
I confirm my plugin works properly on both Dify Community Edition and Cloud Version
I confirm my plugin has been thoroughly tested for completeness and functionality
My plugin brings new value to Dify

5. Documentation Checklist

Please confirm that your plugin README includes all necessary information:

Step-by-step setup instructions
Detailed usage instructions
All required APIs and credentials are clearly listed
Connection requirements and configuration details
Link to the repository for the plugin source code

6. Privacy Protection Information

Based on Dify Plugin Privacy Protection Guidelines:

Data Collection

This plugin does NOT collect or store any personal data by default. All Markdown parsing and chunking are performed locally within the user's Dify instance:

No external APIs are called during chunking.
No request payloads or chunks are sent to third-party services.
All intermediate and final data stay within the configured Dify infrastructure.

Privacy Policy

I confirm that I have prepared and included a privacy policy in my plugin package based on the Plugin Privacy Protection Guidelines

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/langgenius/dify-plugins/pull/1685 **Author:** [@asukhodko](https://github.com/asukhodko) **Created:** 12/7/2025 **Status:** ✅ Merged **Merged:** 12/24/2025 **Merged by:** [@crazywoola](https://github.com/crazywoola) **Base:** `main` ← **Head:** `main` --- ### 📝 Commits (5) - [`a9217cd`](https://github.com/langgenius/dify-plugins/commit/a9217cd55415dbabc06ee6c5194935fd265737f5) feat(tools): add markdown_chunker - Advanced Markdown Chunker - [`ef48635`](https://github.com/langgenius/dify-plugins/commit/ef48635d9646b71f70c8102deab5eeb2568a1264) Merge branch 'langgenius:main' into main - [`6b6e52f`](https://github.com/langgenius/dify-plugins/commit/6b6e52f2ccabb7b65f2af917ab616212754440fd) feat(tools): add markdown_chunker - Advanced Markdown Chunker - [`9c3e968`](https://github.com/langgenius/dify-plugins/commit/9c3e968eb2c7adf7e6163177bf80614854260f64) Merge branch 'langgenius:main' into main - [`b5a40ca`](https://github.com/langgenius/dify-plugins/commit/b5a40ca74c4c82c72b31663c720dc8b5e5dacafb) feat(tools): add markdown_chunker - Advanced Markdown Chunker ### 📊 Changes **1 file changed** (+0 additions, -0 deletions) <details> <summary>View changed files</summary> ➕ `asukhodko/markdown-chunker-2.1.4.difypkg` (+0 -0) </details> ### 📄 Description Structure-aware Markdown chunking for RAG with auto-strategy selection, adaptive sizing, hierarchical chunks, and rich metadata support. # Plugin Submission Form ## 1. Metadata  - **Plugin Author**: asukhodko - **Plugin Name**: markdown_chunker - **Repository URL**: https://github.com/asukhodko/dify-markdown-chunker ## 2. Submission Type - [x] New plugin submission - [ ] Version update for existing plugin ## 3. Description Advanced Markdown Chunker is a structure-aware, multi-strategy Markdown chunking tool for Dify RAG workflows. It analyzes each document, automatically selects the most suitable chunking strategy from four specialized modes, and can adapt chunk size based on content complexity to keep chunks both structurally coherent and retrieval-friendly. :contentReference[oaicite:1]{index=1} By default, each chunk can embed metadata (header paths, content type, line numbers, overlap context) directly into the chunk text, improving semantic retrieval quality in vector databases. All processing happens locally inside Dify; no external APIs or LLM calls are used for chunking. :contentReference[oaicite:2]{index=2} ### Core capability: content-aware, multi-strategy chunking The plugin inspects the document structure and content mix (code, lists, headings, plain text) and chooses between four internal strategies: | Strategy | Auto-activates when | Optimized for | |----------------|-----------------------------------------------|---------------------------------------------------| | **Code-Aware** | `code_ratio ≥ 30%` | API docs, technical references, tutorials | | **List-Aware** | `list_ratio > 40%` | Changelogs, feature lists, task lists, outlines | | **Structural** | `≥ 3` hierarchical header levels detected | Documentation, guides, specifications | | **Fallback** | default for everything else | Mixed content and simple text documents | Each strategy preserves Markdown semantics: headers, lists, tables, and code blocks are kept intact, and the chunker avoids splitting in unnatural places (e.g., mid-function or in the middle of a table row). :contentReference[oaicite:3]{index=3} ### Key features - **List-aware chunking** – preserves nested list hierarchies and binds context (intro paragraphs + their lists), which is crucial for changelogs, release notes, and feature documentation where list structure carries meaning. :contentReference[oaicite:4]{index=4} - **Code-aware chunking** – treats fenced code blocks as atomic units, preserves language metadata, and intelligently binds code to its explanations (Before/After patterns, Code+Output pairs, sequential examples), avoiding mid-function splits that hurt comprehension. :contentReference[oaicite:5]{index=5} - **Structure-preserving segmentation** – respects heading hierarchy, keeps tables and lists intact, supports nested fencing (quadruple/quintuple backticks and tildes), and avoids breaking Markdown blocks in unnatural positions. :contentReference[oaicite:6]{index=6} - **Adaptive chunk sizing (optional)** – can automatically adjust chunk size based on content complexity: - code-heavy content → larger chunks (up to ~1.5× base size), - simple text → smaller chunks (down to ~0.5× base size), with configurable weights and bounds; disabled by default for backward compatibility. :contentReference[oaicite:7]{index=7} - **Hierarchical chunks (parent–child)** – supports parent–child relationships between chunks for multi-level retrieval (overview vs detail), with O(1) lookup and programmatic navigation (ancestors, children, siblings), while remaining backward compatible with flat chunking. :contentReference[oaicite:8]{index=8} - **Streaming processing for large files** – can process large Markdown files in a streaming fashion (e.g. >10MB) with controlled memory usage and smart window boundary detection, keeping chunk quality stable even on long documents. :contentReference[oaicite:9]{index=9} - **LaTeX & formulas as atomic blocks** – preserves LaTeX display math (`$$...$$`) and environments (`\begin{equation}`, `\begin{align}`, etc.) as whole units across all strategies, which is critical for scientific and technical documents. :contentReference[oaicite:10]{index=10} - **Table grouping** – optionally groups related tables in the same chunk with configurable distance and section-awareness, which is useful for API docs (parameters/response/error tables). :contentReference[oaicite:11]{index=11} - **Adaptive overlap** – dynamically scales contextual overlap with chunk size (up to ~35%), maintaining consistent retrieval quality across small and large chunks. :contentReference[oaicite:12]{index=12} - **Rich embedded metadata** – each chunk can include content type, header path, source line numbers, and overlap context, improving filtering, debugging and retrieval behavior in vector stores. :contentReference[oaicite:13]{index=13} - **High reliability & test coverage** – over 800 tests (including property-based tests with Hypothesis) validate correctness, edge cases, and performance regressions; automatic fallback mode ensures graceful degradation on errors. :contentReference[oaicite:14]{index=14} - **Fully local processing** – all parsing and chunking are performed locally within the user’s Dify instance. No external API calls, no LLM dependency for chunking, and no data leaves the configured infrastructure. :contentReference[oaicite:15]{index=15} ### Typical use cases - RAG knowledge bases built on mixed Markdown content - API and SDK documentation with heavy code examples - Changelogs and release notes with nested feature lists - Technical specifications, scientific papers, and architecture docs (including LaTeX formulas and structured tables) - Large Markdown manuals and multi-section guides that benefit from hierarchical chunks and streaming processing ## 4. Checklist - [x] I have read and followed the Publish to Dify Marketplace guidelines - [x] I have read and comply with the Plugin Developer Agreement - [x] I confirm my plugin works properly on both Dify Community Edition and Cloud Version - [x] I confirm my plugin has been thoroughly tested for completeness and functionality - [x] My plugin brings new value to Dify ## 5. Documentation Checklist Please confirm that your plugin README includes all necessary information: - [x] Step-by-step setup instructions - [x] Detailed usage instructions - [x] All required APIs and credentials are clearly listed - [x] Connection requirements and configuration details - [x] Link to the repository for the plugin source code ## 6. Privacy Protection Information Based on Dify Plugin Privacy Protection [Guidelines](https://docs.dify.ai/plugins/publish-plugins/publish-to-dify-marketplace/plugin-privacy-protection-guidelines): ### Data Collection This plugin does NOT collect or store any personal data by default. All Markdown parsing and chunking are performed locally within the user's Dify instance: - No external APIs are called during chunking. - No request payloads or chunks are sent to third-party services. - All intermediate and final data stay within the configured Dify infrastructure. ### Privacy Policy - [x] I confirm that I have prepared and included a privacy policy in my plugin package based on the Plugin Privacy Protection Guidelines --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

yindo added the pull-request label 2026-02-22 18:07:36 -05:00

yindo closed this issue

2026-02-22 18:07:36 -05:00

yindo referenced this issue

2026-02-22 18:09:50 -05:00

[PR #1779] [MERGED] Feat/base #1850

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: langgenius/dify-plugins#1779