langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-22 04:25:29 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Discrepancy in Parent-Child Segmentation Between API and Frontend Document Upload #21247

New Issue
Open
opened 2026-02-21 20:11:36 -05:00 by yindo · 6 comments
yindo commented 2026-02-21 20:11:36 -05:00
Owner
Copy Link
Copy Source

Originally created by @suntao2015005848 on GitHub (Dec 24, 2025).

Originally assigned to: @JohnJyong on GitHub.

Self Checks

  • I have read the Contributing Guide and Language Policy.
  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report, otherwise it will be closed.
  • 【中文用户 & Non English User】请使用英语提交，否则会被关闭 ：）
  • Please do not modify this template :) and fill in all the required fields.

Dify version

1.11.1

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

Why do documents uploaded to the knowledge base via the API, with parent-child segmentation parameters provided, result in different segmentation outcomes compared to directly uploading documents through the frontend for indexing?
pipeline：

Image

Segmentation parameters used via the Dify frontend operation:

Image Image

Via API (endpoint: https://dify.xxx.com.cn/v1/datasets/{dataset_id}/document/create-by-file):

{
"indexing_technique": "high_quality",
"doc_form": "hierarchical_model",
"process_rule": {
"mode": "hierarchical",
"rules": {
"pre_processing_rules": [
{
"id": "remove_extra_spaces",
"enabled": true
},
{
"id": "remove_urls_emails",
"enabled": true
}
],
"parent_mode": "paragraph",
"segmentation": {
"separator": "\n\n",
"max_tokens": 3000,
"chunk_overlap": 0
},
"subchunk_segmentation": {
"separator": "\n",
"max_tokens": 500,
"chunk_overlap": 0
}
},
"limits": {
"indexing_max_segmentation_tokens_length": 8000
}
}
}
api_res:

Image

✔️ Expected Behavior

API and frontend segmenting should be the same.

❌ Actual Behavior

A 12MB document was split into 790 segments via the API, while using the client-side parent-child document segmentation resulted in only 88 segments.

Originally created by @suntao2015005848 on GitHub (Dec 24, 2025). Originally assigned to: @JohnJyong on GitHub. ### Self Checks - [x] I have read the [Contributing Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) and [Language Policy](https://github.com/langgenius/dify/issues/1542). - [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report, otherwise it will be closed. - [x] 【中文用户 & Non English User】请使用英语提交，否则会被关闭 ：） - [x] Please do not modify this template :) and fill in all the required fields. ### Dify version 1.11.1 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce Why do documents uploaded to the knowledge base via the API, with parent-child segmentation parameters provided, result in different segmentation outcomes compared to directly uploading documents through the frontend for indexing? pipeline： <img width="1168" height="458" alt="Image" src="https://github.com/user-attachments/assets/b90fdf51-ca1b-4431-a2d8-1ca1e3b3ea96" /> Segmentation parameters used via the Dify frontend operation: <img width="777" height="635" alt="Image" src="https://github.com/user-attachments/assets/77efee8a-c362-440a-ba5b-1a7772b35532" /> <img width="357" height="738" alt="Image" src="https://github.com/user-attachments/assets/d6b87be0-e234-4957-9094-c06c0aed4eda" /> Via API (endpoint: https://dify.xxx.com.cn/v1/datasets/{dataset_id}/document/create-by-file): { "indexing_technique": "high_quality", "doc_form": "hierarchical_model", "process_rule": { "mode": "hierarchical", "rules": { "pre_processing_rules": [ { "id": "remove_extra_spaces", "enabled": true }, { "id": "remove_urls_emails", "enabled": true } ], "parent_mode": "paragraph", "segmentation": { "separator": "\n\n", "max_tokens": 3000, "chunk_overlap": 0 }, "subchunk_segmentation": { "separator": "\n", "max_tokens": 500, "chunk_overlap": 0 } }, "limits": { "indexing_max_segmentation_tokens_length": 8000 } } } api_res: <img width="372" height="548" alt="Image" src="https://github.com/user-attachments/assets/54de59d2-fe27-418b-b59f-e11de0eb2635" /> ### ✔️ Expected Behavior API and frontend segmenting should be the same. ### ❌ Actual Behavior A 12MB document was split into **790 segments via the API**, while using the client-side parent-child document segmentation resulted in **only 88 segments**.
yindo added the 🐞 bug label 2026-02-21 20:11:36 -05:00
yindo commented 2026-02-21 20:11:36 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Dec 24, 2025):

The segmentation discrepancy between API and frontend uploads in Dify almost always comes down to subtle differences in the process_rule payload structure or parameter values sent to the backend. Both API and frontend use the same backend logic for hierarchical (parent-child) chunking, but the API is strict about payload structure and will silently fall back to defaults or flat segmentation if required fields are missing or mis-nested—even if the API returns 200 OK. This can result in a much higher segment count or unexpected chunking outcomes, as you observed.

To ensure consistent segmentation, your API payload must exactly match what the frontend sends. The most common pitfalls are:

  • Not setting "mode": "hierarchical" in process_rule.
  • Missing or misplacing parent_mode, segmentation, or subchunk_segmentation—all must be direct children of rules.
  • Using different separators, max_tokens, or chunk_overlap values than the frontend.
  • Omitting preprocessing rules, which can change the text before chunking.

Here’s a minimal working example for hierarchical segmentation that matches the frontend structure:

{
  "indexing_technique": "high_quality",
  "doc_form": "hierarchical_model",
  "process_rule": {
    "mode": "hierarchical",
    "rules": {
      "pre_processing_rules": [
        { "id": "remove_extra_spaces", "enabled": true },
        { "id": "remove_urls_emails", "enabled": true }
      ],
      "segmentation": {
        "separator": "\n\n",
        "max_tokens": 3000,
        "chunk_overlap": 0
      },
      "parent_mode": "paragraph",
      "subchunk_segmentation": {
        "separator": "\n",
        "max_tokens": 500,
        "chunk_overlap": 0
      }
    }
  }
}

Double-check that your API payload matches this structure exactly—especially the nesting of fields. Even small differences (like putting parent_mode inside segmentation, or omitting subchunk_segmentation) will cause the backend to ignore your intended settings and use defaults, leading to segmentation mismatches [source] [source].

If you still see discrepancies after confirming the payload, compare the effective process_rule and segmentation parameters in the document details UI for both API and frontend uploads. If they differ, the backend likely defaulted some values due to a payload issue. For persistent issues, check backend logs for validation errors or silent fallbacks—recent Dify versions have improved error reporting for these cases [source].

Matching the API payload to the frontend structure should resolve the segmentation count mismatch.

To reply, just mention @dosu.

How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other

Chat with Dosu Open in Cursor Join Discord Share on X

@dosubot[bot] commented on GitHub (Dec 24, 2025): <!-- Answer --> The segmentation discrepancy between API and frontend uploads in Dify almost always comes down to subtle differences in the process_rule payload structure or parameter values sent to the backend. Both API and frontend use the same backend logic for hierarchical (parent-child) chunking, but the API is strict about payload structure and will silently fall back to defaults or flat segmentation if required fields are missing or mis-nested—even if the API returns 200 OK. This can result in a much higher segment count or unexpected chunking outcomes, as you observed. To ensure consistent segmentation, your API payload must exactly match what the frontend sends. The most common pitfalls are: - Not setting `"mode": "hierarchical"` in `process_rule`. - Missing or misplacing `parent_mode`, `segmentation`, or `subchunk_segmentation`—all must be direct children of `rules`. - Using different separators, max_tokens, or chunk_overlap values than the frontend. - Omitting preprocessing rules, which can change the text before chunking. Here’s a minimal working example for hierarchical segmentation that matches the frontend structure: ```json { "indexing_technique": "high_quality", "doc_form": "hierarchical_model", "process_rule": { "mode": "hierarchical", "rules": { "pre_processing_rules": [ { "id": "remove_extra_spaces", "enabled": true }, { "id": "remove_urls_emails", "enabled": true } ], "segmentation": { "separator": "\n\n", "max_tokens": 3000, "chunk_overlap": 0 }, "parent_mode": "paragraph", "subchunk_segmentation": { "separator": "\n", "max_tokens": 500, "chunk_overlap": 0 } } } } ``` Double-check that your API payload matches this structure exactly—especially the nesting of fields. Even small differences (like putting `parent_mode` inside `segmentation`, or omitting `subchunk_segmentation`) will cause the backend to ignore your intended settings and use defaults, leading to segmentation mismatches [[source]](https://github.com/langgenius/dify/issues/23933) [[source]](https://github.com/langgenius/dify/issues/22270). If you still see discrepancies after confirming the payload, compare the effective process_rule and segmentation parameters in the document details UI for both API and frontend uploads. If they differ, the backend likely defaulted some values due to a payload issue. For persistent issues, check backend logs for validation errors or silent fallbacks—recent Dify versions have improved error reporting for these cases [[source]](https://github.com/langgenius/dify/issues/28763). Matching the API payload to the frontend structure should resolve the segmentation count mismatch. <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- <sup>How did I do? [Good](https://app.dosu.dev/response-feedback/f93664dc-cf0e-4a3a-8de5-a1c063c4782b?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/f93664dc-cf0e-4a3a-8de5-a1c063c4782b?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/f93664dc-cf0e-4a3a-8de5-a1c063c4782b?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/f93664dc-cf0e-4a3a-8de5-a1c063c4782b?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/f93664dc-cf0e-4a3a-8de5-a1c063c4782b?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/f93664dc-cf0e-4a3a-8de5-a1c063c4782b?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/f93664dc-cf0e-4a3a-8de5-a1c063c4782b?feedback_type=other)</sup> [![Chat with Dosu](https://dosu.dev/dosu-chat-badge.svg)](https://app.dosu.dev/6088a4f4-073a-43f3-b3d4-4f06fb0c05a0/ask?utm_source=github)&nbsp;[![Open in Cursor](https://dosu.dev/dosu-in-cursor.svg)](https://cursor.com/link/prompt?text=The%20segmentation%20discrepancy%20between%20API%20and%20frontend%20uploads%20in%20Dify%20almost%20always%20comes%20down%20to%20subtle%20differences%20in%20the%20process_rule%20payload%20structure%20or%20parameter%20values%20sent%20to%20the%20backend.%20Both%20API%20and%20frontend%20use%20the%20same%20backend%20logic%20for%20hierarchical%20%28parent-child%29%20chunking%2C%20but%20the%20API%20is%20strict%20about%20payload%20structure%20and%20will%20silently%20fall%20back%20to%20defaults%20or%20flat%20segmentation%20if%20required%20fields%20are%20missing%20or%20mis-nested%E2%80%94even%20if%20the%20API%20returns%20200%20OK.%20This%20can%20result%20in%20a%20much%20higher%20segment%20count%20or%20unexpected%20chunking%20outcomes%2C%20as%20you%20observed.%0A%0ATo%20ensure%20consistent%20segmentation%2C%20your%20API%20payload%20must%20exactly%20match%20what%20the%20frontend%20sends.%20The%20most%20common%20pitfalls%20are%3A%0A%0A-%20Not%20setting%20%60%22mode%22%3A%20%22hierarchical%22%60%20in%20%60process_rule%60.%0A-%20Missing%20or%20misplacing%20%60parent_mode%60%2C%20%60segmentation%60%2C%20or%20%60subchunk_segmentation%60%E2%80%94all%20must%20be%20direct%20children%20of%20%60rules%60.%0A-%20Using%20different%20separators%2C%20max_tokens%2C%20or%20chunk_overlap%20values%20than%20the%20frontend.%0A-%20Omitting%20preprocessing%20rules%2C%20which%20can%20change%20the%20text%20before%20chunking.%0A%0AHere%E2%80%99s%20a%20minimal%20working%20example%20for%20hierarchical%20segmentation%20that%20matches%20the%20frontend%20structure%3A%0A%0A%60%60%60json%0A%7B%0A%20%20%22indexing_technique%22%3A%20%22high_quality%22%2C%0A%20%20%22doc_form%22%3A%20%22hierarchical_model%22%2C%0A%20%20%22process_rule%22%3A%20%7B%0A%20%20%20%20%22mode%22%3A%20%22hierarchical%22%2C%0A%20%20%20%20%22rules%22%3A%20%7B%0A%20%20%20%20%20%20%22pre_processing_rules%22%3A%20%5B%0A%20%20%20%20%20%20%20%20%7B%20%22id%22%3A%20%22remove_extra_spaces%22%2C%20%22enabled%22%3A%20true%20%7D%2C%0A%20%20%20%20%20%20%20%20%7B%20%22id%22%3A%20%22remove_urls_emails%22%2C%20%22enabled%22%3A%20true%20%7D%0A%20%20%20%20%20%20%5D%2C%0A%20%20%20%20%20%20%22segmentation%22%3A%20%7B%0A%20%20%20%20%20%20%20%20%22separator%22%3A%20%22%5Cn%5Cn%22%2C%0A%20%20%20%20%20%20%20%20%22max_tokens%22%3A%203000%2C%0A%20%20%20%20%20%20%20%20%22chunk_overlap%22%3A%200%0A%20%20%20%20%20%20%7D%2C%0A%20%20%20%20%20%20%22parent_mode%22%3A%20%22paragraph%22%2C%0A%20%20%20%20%20%20%22subchunk_segmentation%22%3A%20%7B%0A%20%20%20%20%20%20%20%20%22separator%22%3A%20%22%5Cn%22%2C%0A%20%20%20%20%20%20%20%20%22max_tokens%22%3A%20500%2C%0A%20%20%20%20%20%20%20%20%22chunk_overlap%22%3A%200%0A%20%20%20%20%20%20%7D%0A%20%20%20%20%7D%0A%20%20%7D%0A%7D%0A%60%60%60%0A%0ADouble-check%20that%20your%20API%20payload%20matches%20this%20structure%20exactly%E2%80%94especially%20the%20nesting%20of%20fields.%20Even%20small%20differences%20%28like%20putting%20%60parent_mode%60%20inside%20%60segmentation%60%2C%20or%20omitting%20%60subchunk_segmentation%60%29%20will%20cause%20the%20backend%20to%20ignore%20your%20intended%20settings%20and%20use%20defaults%2C%20leading%20to%20segmentation%20mismatches%20%5B%5Bsource%5D%5D%28https%3A//github.com/langgenius/dify/issues/23933%29%20%5B%5Bsource%5D%5D%28https%3A//github.com/langgenius/dify/issues/22270%29.%0A%0AIf%20you%20still%20see%20discrepancies%20after%20confirming%20the%20payload%2C%20compare%20the%20effective%20process_rule%20and%20segmentation%20parameters%20in%20the%20document%20details%20UI%20for%20both%20API%20and%20frontend%20uploads.%20If%20they%20differ%2C%20the%20backend%20likely%20defaulted%20some%20values%20due%20to%20a%20payload%20issue.%20For%20persistent%20issues%2C%20check%20backend%20logs%20for%20validation%20errors%20or%20silent%20fallbacks%E2%80%94recent%20Dify%20versions%20have%20improved%20error%20reporting%20for%20these%20cases%20%5B%5Bsource%5D%5D%28https%3A//github.com/langgenius/dify/issues/28763%29.%0A%0AMatching%20the%20API%20payload%20to%20the%20frontend%20structure%20should%20resolve%20the%20segmentation%20count%20mismatch.)&nbsp;[![Join Discord](https://img.shields.io/badge/join-5865F2?logo=discord&logoColor=white&label=)](https://go.dosu.dev/discord-bot)&nbsp;[![Share on X](https://img.shields.io/badge/X-share-black)](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/langgenius/dify/issues/30071)
yindo commented 2026-02-21 20:11:36 -05:00
Author
Owner
Copy Link
Copy Source

@suntao2015005848 commented on GitHub (Dec 24, 2025):

Dify API 和前端上传之间的分段差异几乎总是源于发送到后端的 process_rule 有效负载结构或参数值的细微差别。API 和前端都使用相同的后端逻辑进行层级（父子）分块，但 API 对有效负载结构要求严格，如果缺少必填字段或嵌套错误，即使 API 返回 200 OK，也会静默回退到默认值或扁平分段。正如您所观察到的，这会导致分段数量远高于预期或出现意料之外的分块结果。

为确保数据分段的一致性，您的 API 请求负载必须与前端发送的内容完全一致。最常见的陷阱包括：

  • 未"mode": "hierarchical"设置process_rule。
  • 缺失或错放的parent_mode，，segmentation或subchunk_segmentation——所有这些都必须是的直接子项rules。
  • 使用与前端不同的分隔符、最大标记数或块重叠值。
  • 省略预处理规则，这些规则可能会在分块之前改变文本。

以下是一个与前端结构相匹配的最小层级分割工作示例：

{
"indexing_technique": "high_quality",
"doc_form": "hierarchical_model",
"process_rule": {
"mode": "hierarchical",
"rules": {
"pre_processing_rules": [
{ "id": "remove_extra_spaces", "enabled": true },
{ "id": "remove_urls_emails", "enabled": true }
],
"segmentation": {
"separator": "\n\n",
"max_tokens": 3000,
"chunk_overlap": 0
},
"parent_mode": "paragraph",
"subchunk_segmentation": {
"separator": "\n",
"max_tokens": 500,
"chunk_overlap": 0
}
}
}
}
请仔细检查您的 API 有效负载是否与此结构完全匹配——尤其是字段的嵌套。即使是微小的差异（例如将字段放在 <head>``parent_mode内segmentation，或省略字段subchunk_segmentation），也会导致后端忽略您的预期设置并使用默认值，从而导致分段不匹配[来源] [来源]。

如果在确认有效负载后仍然发现差异，请比较 API 和前端上传的文档详情 UI 中的有效 process_rule 和 segmentation 参数。如果它们不同，则后端可能由于有效负载问题而使用了默认值。对于持续存在的问题，请检查后端日志中是否存在验证错误或静默回退——最新版本的 Dify 已改进了此类情况的错误报告[来源]。

将 API 有效负载与前端结构匹配应该可以解决分段计数不匹配的问题。

回复时只需提及@dosu即可。

我做得怎么样？好|无关|错误|冗长|幻觉|举报 🐛 |其他

与 Dosu 聊天 在光标中打开 加入 Discord 分享至 X

I checked the API used for indexing after uploading a document to the knowledge base through frontend operations via the browser console: http://192.168.41.68/console/api/datasets/process-rule?document_id=b0826553-00ee-4c71-846a-3cdfc8034378.
The output is as follows:

json
{
"mode": "hierarchical",
"rules": {
"pre_processing_rules": [
{
"id": "remove_extra_spaces",
"enabled": true
},
{
"id": "remove_urls_emails",
"enabled": true
}
],
"segmentation": {
"separator": "\n\n",
"max_tokens": 3000,
"chunk_overlap": 0
},
"parent_mode": "paragraph",
"subchunk_segmentation": {
"separator": "\n",
"max_tokens": 500,
"chunk_overlap": 0
}
},
"limits": {
"indexing_max_segmentation_tokens_length": 8000
}
}
This is exactly the same as the parameters I passed in via the API. Why is there such a significant difference?How should I go about resolving this issue?

@suntao2015005848 commented on GitHub (Dec 24, 2025): > Dify API 和前端上传之间的分段差异几乎总是源于发送到后端的 process_rule 有效负载结构或参数值的细微差别。API 和前端都使用相同的后端逻辑进行层级（父子）分块，但 API 对有效负载结构要求严格，如果缺少必填字段或嵌套错误，即使 API 返回 200 OK，也会静默回退到默认值或扁平分段。正如您所观察到的，这会导致分段数量远高于预期或出现意料之外的分块结果。 > > 为确保数据分段的一致性，您的 API 请求负载必须与前端发送的内容完全一致。最常见的陷阱包括： > > * 未`"mode": "hierarchical"`设置`process_rule`。 > * 缺失或错放的`parent_mode`，，`segmentation`或`subchunk_segmentation`——所有这些都必须是的直接子项`rules`。 > * 使用与前端不同的分隔符、最大标记数或块重叠值。 > * 省略预处理规则，这些规则可能会在分块之前改变文本。 > > 以下是一个与前端结构相匹配的最小层级分割工作示例： > > { > "indexing_technique": "high_quality", > "doc_form": "hierarchical_model", > "process_rule": { > "mode": "hierarchical", > "rules": { > "pre_processing_rules": [ > { "id": "remove_extra_spaces", "enabled": true }, > { "id": "remove_urls_emails", "enabled": true } > ], > "segmentation": { > "separator": "\n\n", > "max_tokens": 3000, > "chunk_overlap": 0 > }, > "parent_mode": "paragraph", > "subchunk_segmentation": { > "separator": "\n", > "max_tokens": 500, > "chunk_overlap": 0 > } > } > } > } > 请仔细检查您的 API 有效负载是否与此结构完全匹配——尤其是字段的嵌套。即使是微小的差异（例如将字段放在 `<head>``parent_mode`内`segmentation`，或省略字段`subchunk_segmentation`），也会导致后端忽略您的预期设置并使用默认值，从而导致分段不匹配[[来源] ](https://github.com/langgenius/dify/issues/23933) [[来源]](https://github.com/langgenius/dify/issues/22270)。 > > 如果在确认有效负载后仍然发现差异，请比较 API 和前端上传的文档详情 UI 中的有效 process_rule 和 segmentation 参数。如果它们不同，则后端可能由于有效负载问题而使用了默认值。对于持续存在的问题，请检查后端日志中是否存在验证错误或静默回退——最新版本的 Dify 已改进了此类情况的错误报告[[来源]](https://github.com/langgenius/dify/issues/28763)。 > > 将 API 有效负载与前端结构匹配应该可以解决分段计数不匹配的问题。 > > _回复时只需提及[@dosu](https://go.dosu.dev/dosubot)即可。_ > > 我做得怎么样？[好](https://app.dosu.dev/response-feedback/f93664dc-cf0e-4a3a-8de5-a1c063c4782b?feedback_type=great_response)|[无关](https://app.dosu.dev/response-feedback/f93664dc-cf0e-4a3a-8de5-a1c063c4782b?feedback_type=irrelevant_answer)|[错误](https://app.dosu.dev/response-feedback/f93664dc-cf0e-4a3a-8de5-a1c063c4782b?feedback_type=incorrect_sources)|[冗长](https://app.dosu.dev/response-feedback/f93664dc-cf0e-4a3a-8de5-a1c063c4782b?feedback_type=too_verbose)|[幻觉](https://app.dosu.dev/response-feedback/f93664dc-cf0e-4a3a-8de5-a1c063c4782b?feedback_type=hallucination)|[举报 🐛](https://app.dosu.dev/response-feedback/f93664dc-cf0e-4a3a-8de5-a1c063c4782b?feedback_type=bug_report) |[其他](https://app.dosu.dev/response-feedback/f93664dc-cf0e-4a3a-8de5-a1c063c4782b?feedback_type=other) > > [![与 Dosu 聊天](https://camo.githubusercontent.com/7c571478962a6b8a5d47cac74b6824c75e4f6731bba52377e6007f9150462923/68747470733a2f2f646f73752e6465762f646f73752d636861742d62616467652e737667)](https://app.dosu.dev/6088a4f4-073a-43f3-b3d4-4f06fb0c05a0/ask?utm_source=github) [![在光标中打开](https://camo.githubusercontent.com/bdf7dbded65ee585887f47e5fd6fa099ea0933088b4b03d02e83b4def50154d4/68747470733a2f2f646f73752e6465762f646f73752d696e2d637572736f722e737667)](https://cursor.com/link/prompt?text=The%20segmentation%20discrepancy%20between%20API%20and%20frontend%20uploads%20in%20Dify%20almost%20always%20comes%20down%20to%20subtle%20differences%20in%20the%20process_rule%20payload%20structure%20or%20parameter%20values%20sent%20to%20the%20backend.%20Both%20API%20and%20frontend%20use%20the%20same%20backend%20logic%20for%20hierarchical%20%28parent-child%29%20chunking%2C%20but%20the%20API%20is%20strict%20about%20payload%20structure%20and%20will%20silently%20fall%20back%20to%20defaults%20or%20flat%20segmentation%20if%20required%20fields%20are%20missing%20or%20mis-nested%E2%80%94even%20if%20the%20API%20returns%20200%20OK.%20This%20can%20result%20in%20a%20much%20higher%20segment%20count%20or%20unexpected%20chunking%20outcomes%2C%20as%20you%20observed.%0A%0ATo%20ensure%20consistent%20segmentation%2C%20your%20API%20payload%20must%20exactly%20match%20what%20the%20frontend%20sends.%20The%20most%20common%20pitfalls%20are%3A%0A%0A-%20Not%20setting%20%60%22mode%22%3A%20%22hierarchical%22%60%20in%20%60process_rule%60.%0A-%20Missing%20or%20misplacing%20%60parent_mode%60%2C%20%60segmentation%60%2C%20or%20%60subchunk_segmentation%60%E2%80%94all%20must%20be%20direct%20children%20of%20%60rules%60.%0A-%20Using%20different%20separators%2C%20max_tokens%2C%20or%20chunk_overlap%20values%20than%20the%20frontend.%0A-%20Omitting%20preprocessing%20rules%2C%20which%20can%20change%20the%20text%20before%20chunking.%0A%0AHere%E2%80%99s%20a%20minimal%20working%20example%20for%20hierarchical%20segmentation%20that%20matches%20the%20frontend%20structure%3A%0A%0A%60%60%60json%0A%7B%0A%20%20%22indexing_technique%22%3A%20%22high_quality%22%2C%0A%20%20%22doc_form%22%3A%20%22hierarchical_model%22%2C%0A%20%20%22process_rule%22%3A%20%7B%0A%20%20%20%20%22mode%22%3A%20%22hierarchical%22%2C%0A%20%20%20%20%22rules%22%3A%20%7B%0A%20%20%20%20%20%20%22pre_processing_rules%22%3A%20%5B%0A%20%20%20%20%20%20%20%20%7B%20%22id%22%3A%20%22remove_extra_spaces%22%2C%20%22enabled%22%3A%20true%20%7D%2C%0A%20%20%20%20%20%20%20%20%7B%20%22id%22%3A%20%22remove_urls_emails%22%2C%20%22enabled%22%3A%20true%20%7D%0A%20%20%20%20%20%20%5D%2C%0A%20%20%20%20%20%20%22segmentation%22%3A%20%7B%0A%20%20%20%20%20%20%20%20%22separator%22%3A%20%22%5Cn%5Cn%22%2C%0A%20%20%20%20%20%20%20%20%22max_tokens%22%3A%203000%2C%0A%20%20%20%20%20%20%20%20%22chunk_overlap%22%3A%200%0A%20%20%20%20%20%20%7D%2C%0A%20%20%20%20%20%20%22parent_mode%22%3A%20%22paragraph%22%2C%0A%20%20%20%20%20%20%22subchunk_segmentation%22%3A%20%7B%0A%20%20%20%20%20%20%20%20%22separator%22%3A%20%22%5Cn%22%2C%0A%20%20%20%20%20%20%20%20%22max_tokens%22%3A%20500%2C%0A%20%20%20%20%20%20%20%20%22chunk_overlap%22%3A%200%0A%20%20%20%20%20%20%7D%0A%20%20%20%20%7D%0A%20%20%7D%0A%7D%0A%60%60%60%0A%0ADouble-check%20that%20your%20API%20payload%20matches%20this%20structure%20exactly%E2%80%94especially%20the%20nesting%20of%20fields.%20Even%20small%20differences%20%28like%20putting%20%60parent_mode%60%20inside%20%60segmentation%60%2C%20or%20omitting%20%60subchunk_segmentation%60%29%20will%20cause%20the%20backend%20to%20ignore%20your%20intended%20settings%20and%20use%20defaults%2C%20leading%20to%20segmentation%20mismatches%20%5B%5Bsource%5D%5D%28https%3A//github.com/langgenius/dify/issues/23933%29%20%5B%5Bsource%5D%5D%28https%3A//github.com/langgenius/dify/issues/22270%29.%0A%0AIf%20you%20still%20see%20discrepancies%20after%20confirming%20the%20payload%2C%20compare%20the%20effective%20process_rule%20and%20segmentation%20parameters%20in%20the%20document%20details%20UI%20for%20both%20API%20and%20frontend%20uploads.%20If%20they%20differ%2C%20the%20backend%20likely%20defaulted%20some%20values%20due%20to%20a%20payload%20issue.%20For%20persistent%20issues%2C%20check%20backend%20logs%20for%20validation%20errors%20or%20silent%20fallbacks%E2%80%94recent%20Dify%20versions%20have%20improved%20error%20reporting%20for%20these%20cases%20%5B%5Bsource%5D%5D%28https%3A//github.com/langgenius/dify/issues/28763%29.%0A%0AMatching%20the%20API%20payload%20to%20the%20frontend%20structure%20should%20resolve%20the%20segmentation%20count%20mismatch.) [![加入 Discord](https://camo.githubusercontent.com/7d2066700925db1e370d956834f9ea0e5bc92cecbe84963fbd2a363e45091b4e/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6a6f696e2d3538363546323f6c6f676f3d646973636f7264266c6f676f436f6c6f723d7768697465266c6162656c3d)](https://go.dosu.dev/discord-bot) [![分享至 X](https://camo.githubusercontent.com/a8bd7f3fcf3f5e3bd124eee9ae50ae2ac17b40971726665ac2121d48f9aee155/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f582d73686172652d626c61636b)](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/langgenius/dify/issues/30071) I checked the API used for indexing after uploading a document to the knowledge base through frontend operations via the browser console: http://192.168.41.68/console/api/datasets/process-rule?document_id=b0826553-00ee-4c71-846a-3cdfc8034378. The output is as follows: json { "mode": "hierarchical", "rules": { "pre_processing_rules": [ { "id": "remove_extra_spaces", "enabled": true }, { "id": "remove_urls_emails", "enabled": true } ], "segmentation": { "separator": "\n\n", "max_tokens": 3000, "chunk_overlap": 0 }, "parent_mode": "paragraph", "subchunk_segmentation": { "separator": "\n", "max_tokens": 500, "chunk_overlap": 0 } }, "limits": { "indexing_max_segmentation_tokens_length": 8000 } } This is exactly the same as the parameters I passed in via the API. Why is there such a significant difference?How should I go about resolving this issue?
yindo commented 2026-02-21 20:11:36 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Dec 24, 2025):

Based on the backend code and multiple issue reports, both API and frontend uploads in Dify use the exact same segmentation and preprocessing pipeline. If your process_rule structure and values are truly identical, segmentation results should also be identical—unless there is a subtle difference in the file content as received by the backend (for example, encoding, line endings, or invisible characters) or a silent fallback to defaults due to a hidden validation issue [source] [source] [source].

Here’s what you can do next:

  1. Ensure you are uploading the exact same file (byte-for-byte) via both API and frontend. Even a small difference in encoding or line endings can affect how the backend splits the document.
  2. Compare the raw extracted text (after preprocessing, before segmentation) for both uploads. If possible, use the Dify UI’s segment preview or export feature to inspect the actual chunk boundaries and content.
  3. Check backend logs for warnings or silent fallbacks during the API upload. Even if the API returns 200 OK, the backend may have defaulted to flat segmentation or ignored some settings if it detected a subtle issue.
  4. If you are on Dify v1.11.1, consider updating—recent versions have improved validation and error reporting for process_rule payloads, which helps catch silent fallbacks [source].

If after all these checks the process_rule and file content are truly identical, but the segment counts still differ, it’s likely due to a subtle difference in the file as received by the backend or a rare bug in the segmentation logic. Comparing the actual segment content and checking logs should help pinpoint the root cause.

To reply, just mention @dosu.

How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other

Chat with Dosu Open in Cursor Join Discord Share on X

@dosubot[bot] commented on GitHub (Dec 24, 2025): <!-- Answer --> Based on the backend code and multiple issue reports, both API and frontend uploads in Dify use the exact same segmentation and preprocessing pipeline. If your process_rule structure and values are truly identical, segmentation results should also be identical—unless there is a subtle difference in the file content as received by the backend (for example, encoding, line endings, or invisible characters) or a silent fallback to defaults due to a hidden validation issue [[source]](https://github.com/langgenius/dify/issues/23933) [[source]](https://github.com/langgenius/dify/issues/22270) [[source]](https://github.com/langgenius/dify/issues/28763). Here’s what you can do next: 1. **Ensure you are uploading the exact same file (byte-for-byte) via both API and frontend**. Even a small difference in encoding or line endings can affect how the backend splits the document. 2. **Compare the raw extracted text (after preprocessing, before segmentation) for both uploads**. If possible, use the Dify UI’s segment preview or export feature to inspect the actual chunk boundaries and content. 3. **Check backend logs for warnings or silent fallbacks** during the API upload. Even if the API returns 200 OK, the backend may have defaulted to flat segmentation or ignored some settings if it detected a subtle issue. 4. **If you are on Dify v1.11.1, consider updating**—recent versions have improved validation and error reporting for process_rule payloads, which helps catch silent fallbacks [[source]](https://github.com/langgenius/dify/pull/28810). If after all these checks the process_rule and file content are truly identical, but the segment counts still differ, it’s likely due to a subtle difference in the file as received by the backend or a rare bug in the segmentation logic. Comparing the actual segment content and checking logs should help pinpoint the root cause. <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- <sup>How did I do? [Good](https://app.dosu.dev/response-feedback/ea542fc9-0d26-485c-adb1-6b22d4f6d180?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/ea542fc9-0d26-485c-adb1-6b22d4f6d180?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/ea542fc9-0d26-485c-adb1-6b22d4f6d180?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/ea542fc9-0d26-485c-adb1-6b22d4f6d180?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/ea542fc9-0d26-485c-adb1-6b22d4f6d180?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/ea542fc9-0d26-485c-adb1-6b22d4f6d180?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/ea542fc9-0d26-485c-adb1-6b22d4f6d180?feedback_type=other)</sup> [![Chat with Dosu](https://dosu.dev/dosu-chat-badge.svg)](https://app.dosu.dev/6088a4f4-073a-43f3-b3d4-4f06fb0c05a0/ask?utm_source=github)&nbsp;[![Open in Cursor](https://dosu.dev/dosu-in-cursor.svg)](https://cursor.com/link/prompt?text=Based%20on%20the%20backend%20code%20and%20multiple%20issue%20reports%2C%20both%20API%20and%20frontend%20uploads%20in%20Dify%20use%20the%20exact%20same%20segmentation%20and%20preprocessing%20pipeline.%20If%20your%20process_rule%20structure%20and%20values%20are%20truly%20identical%2C%20segmentation%20results%20should%20also%20be%20identical%E2%80%94unless%20there%20is%20a%20subtle%20difference%20in%20the%20file%20content%20as%20received%20by%20the%20backend%20%28for%20example%2C%20encoding%2C%20line%20endings%2C%20or%20invisible%20characters%29%20or%20a%20silent%20fallback%20to%20defaults%20due%20to%20a%20hidden%20validation%20issue%20%5B%5Bsource%5D%5D%28https%3A//github.com/langgenius/dify/issues/23933%29%20%5B%5Bsource%5D%5D%28https%3A//github.com/langgenius/dify/issues/22270%29%20%5B%5Bsource%5D%5D%28https%3A//github.com/langgenius/dify/issues/28763%29.%0A%0AHere%E2%80%99s%20what%20you%20can%20do%20next%3A%0A%0A1.%20%2A%2AEnsure%20you%20are%20uploading%20the%20exact%20same%20file%20%28byte-for-byte%29%20via%20both%20API%20and%20frontend%2A%2A.%20Even%20a%20small%20difference%20in%20encoding%20or%20line%20endings%20can%20affect%20how%20the%20backend%20splits%20the%20document.%0A2.%20%2A%2ACompare%20the%20raw%20extracted%20text%20%28after%20preprocessing%2C%20before%20segmentation%29%20for%20both%20uploads%2A%2A.%20If%20possible%2C%20use%20the%20Dify%20UI%E2%80%99s%20segment%20preview%20or%20export%20feature%20to%20inspect%20the%20actual%20chunk%20boundaries%20and%20content.%0A3.%20%2A%2ACheck%20backend%20logs%20for%20warnings%20or%20silent%20fallbacks%2A%2A%20during%20the%20API%20upload.%20Even%20if%20the%20API%20returns%20200%20OK%2C%20the%20backend%20may%20have%20defaulted%20to%20flat%20segmentation%20or%20ignored%20some%20settings%20if%20it%20detected%20a%20subtle%20issue.%0A4.%20%2A%2AIf%20you%20are%20on%20Dify%20v1.11.1%2C%20consider%20updating%2A%2A%E2%80%94recent%20versions%20have%20improved%20validation%20and%20error%20reporting%20for%20process_rule%20payloads%2C%20which%20helps%20catch%20silent%20fallbacks%20%5B%5Bsource%5D%5D%28https%3A//github.com/langgenius/dify/pull/28810%29.%0A%0AIf%20after%20all%20these%20checks%20the%20process_rule%20and%20file%20content%20are%20truly%20identical%2C%20but%20the%20segment%20counts%20still%20differ%2C%20it%E2%80%99s%20likely%20due%20to%20a%20subtle%20difference%20in%20the%20file%20as%20received%20by%20the%20backend%20or%20a%20rare%20bug%20in%20the%20segmentation%20logic.%20Comparing%20the%20actual%20segment%20content%20and%20checking%20logs%20should%20help%20pinpoint%20the%20root%20cause.)&nbsp;[![Join Discord](https://img.shields.io/badge/join-5865F2?logo=discord&logoColor=white&label=)](https://go.dosu.dev/discord-bot)&nbsp;[![Share on X](https://img.shields.io/badge/X-share-black)](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/langgenius/dify/issues/30071)
yindo commented 2026-02-21 20:11:36 -05:00
Author
Owner
Copy Link
Copy Source

@suntao2015005848 commented on GitHub (Dec 24, 2025):

为何通过前端页面，

根据遥控器代码和多个问题报告，Dify 中的 API 和前置上传都使用了数十个的分割和分割流程。如果您的 process_rule 结构和值确实相同，则分割结果也应该相同——除非接收到的文件内容存在梯度（例如，编码、换行符或不可见字符），或者隐藏由于的验证问题而静默回退到默认值[来源] [来源] [来源]。
接下来你可以这样做：

  1. 请确保通过API和前置上传的文件数十（字节）。即使是编码或换行符的计算梯度，也影响了表格对文档的分割方式。
  2. 比较两次上传的原始提取文本（重建后、分割前）。如果可以，请使用Dify用户界面的分割预览或导出功能来检查实际的文本块边界和内容。
  3. 检查日志日志，查看 API 上传过程中是否存在警告或静默回退。即使 API 返回 200 OK，报表也可能检测到一些问题而默认采用纸张分区或忽略某些设置。
  4. 如果您使用的是 Dify v1.11.1，请考虑更新——最近的版本改进了 process_rule 有效负载的验证和错误报告，这有助于捕获静默回[来源]。

如果经过所有这些检查后，处理规则和文件内容确实完全相同，但段计数不同，则很可能仍然是由于接收到的文件存在严重差异，或者分段逻辑中存在罕见的错误。比较实际的段内容并检查日志有助于查找根本原因。
回复时只需引用@dosu即可。
我在做什么？好|关联|错误|怎么样冗长|幻觉|报告🐛 |其他
与 Dosu 聊天 在窗口中打开 加入不和谐 分享至X

Different Processing Mechanisms:
Frontend Upload: Goes through the RAG Pipeline workflow, which uses an optimized processing chain.
API Upload: Goes through basic document indexing, using the standard processing flow.So this leads to different document processing results. This is a knowledge base processing bug.

@suntao2015005848 commented on GitHub (Dec 24, 2025): > 为何通过前端页面， > > > 根据遥控器代码和多个问题报告，Dify 中的 API 和前置上传都使用了数十个的分割和分割流程。如果您的 process_rule 结构和值确实相同，则分割结果也应该相同——除非接收到的文件内容存在梯度（例如，编码、换行符或不可见字符），或者隐藏由于的验证问题而静默回退到默认值[[来源] ](https://github.com/langgenius/dify/issues/23933) [[来源] ](https://github.com/langgenius/dify/issues/22270) [[来源]](https://github.com/langgenius/dify/issues/28763)。 > > 接下来你可以这样做： > > > > 1. **请确保通过API和前置上传的文件数十（字节）**。即使是编码或换行符的计算梯度，也影响了表格对文档的分割方式。 > > 2. **比较两次上传的原始提取文本（重建后、分割前）**。如果可以，请使用Dify用户界面的分割预览或导出功能来检查实际的文本块边界和内容。 > > 3. **检查日志日志，查看 API 上传过程中是否存在警告或静默回退**。即使 API 返回 200 OK，报表也可能检测到一些问题而默认采用纸张分区或忽略某些设置。 > > 4. **如果您使用的是 Dify v1.11.1，请考虑更新**——最近的版本改进了 process_rule 有效负载的验证和错误报告，这有助于捕获静默回[[来源]](https://github.com/langgenius/dify/pull/28810)。 > > > > 如果经过所有这些检查后，处理规则和文件内容确实完全相同，但段计数不同，则很可能仍然是由于接收到的文件存在严重差异，或者分段逻辑中存在罕见的错误。比较实际的段内容并检查日志有助于查找根本原因。 > > _回复时只需引用[@dosu](https://go.dosu.dev/dosubot)即可。_ > > 我在做什么？[好](https://app.dosu.dev/response-feedback/ea542fc9-0d26-485c-adb1-6b22d4f6d180?feedback_type=great_response)|[关联](https://app.dosu.dev/response-feedback/ea542fc9-0d26-485c-adb1-6b22d4f6d180?feedback_type=irrelevant_answer)|[错误](https://app.dosu.dev/response-feedback/ea542fc9-0d26-485c-adb1-6b22d4f6d180?feedback_type=incorrect_sources)|[怎么样冗长](https://app.dosu.dev/response-feedback/ea542fc9-0d26-485c-adb1-6b22d4f6d180?feedback_type=too_verbose)|[幻觉](https://app.dosu.dev/response-feedback/ea542fc9-0d26-485c-adb1-6b22d4f6d180?feedback_type=hallucination)|[报告🐛](https://app.dosu.dev/response-feedback/ea542fc9-0d26-485c-adb1-6b22d4f6d180?feedback_type=bug_report) |[其他](https://app.dosu.dev/response-feedback/ea542fc9-0d26-485c-adb1-6b22d4f6d180?feedback_type=other) > > [![与 Dosu 聊天](https://camo.githubusercontent.com/7c571478962a6b8a5d47cac74b6824c75e4f6731bba52377e6007f9150462923/68747470733a2f2f646f73752e6465762f646f73752d636861742d62616467652e737667)](https://app.dosu.dev/6088a4f4-073a-43f3-b3d4-4f06fb0c05a0/ask?utm_source=github) [![在窗口中打开](https://camo.githubusercontent.com/bdf7dbded65ee585887f47e5fd6fa099ea0933088b4b03d02e83b4def50154d4/68747470733a2f2f646f73752e6465762f646f73752d696e2d637572736f722e737667)](https://cursor.com/link/prompt?text=Based%20on%20the%20backend%20code%20and%20multiple%20issue%20reports%2C%20both%20API%20and%20frontend%20uploads%20in%20Dify%20use%20the%20exact%20same%20segmentation%20and%20preprocessing%20pipeline.%20If%20your%20process_rule%20structure%20and%20values%20are%20truly%20identical%2C%20segmentation%20results%20should%20also%20be%20identical%E2%80%94unless%20there%20is%20a%20subtle%20difference%20in%20the%20file%20content%20as%20received%20by%20the%20backend%20%28for%20example%2C%20encoding%2C%20line%20endings%2C%20or%20invisible%20characters%29%20or%20a%20silent%20fallback%20to%20defaults%20due%20to%20a%20hidden%20validation%20issue%20%5B%5Bsource%5D%5D%28https%3A//github.com/langgenius/dify/issues/23933%29%20%5B%5Bsource%5D%5D%28https%3A//github.com/langgenius/dify/issues/22270%29%20%5B%5Bsource%5D%5D%28https%3A//github.com/langgenius/dify/issues/28763%29.%0A%0AHere%E2%80%99s%20what%20you%20can%20do%20next%3A%0A%0A1.%20%2A%2AEnsure%20you%20are%20uploading%20the%20exact%20same%20file%20%28byte-for-byte%29%20via%20both%20API%20and%20frontend%2A%2A.%20Even%20a%20small%20difference%20in%20encoding%20or%20line%20endings%20can%20affect%20how%20the%20backend%20splits%20the%20document.%0A2.%20%2A%2ACompare%20the%20raw%20extracted%20text%20%28after%20preprocessing%2C%20before%20segmentation%29%20for%20both%20uploads%2A%2A.%20If%20possible%2C%20use%20the%20Dify%20UI%E2%80%99s%20segment%20preview%20or%20export%20feature%20to%20inspect%20the%20actual%20chunk%20boundaries%20and%20content.%0A3.%20%2A%2ACheck%20backend%20logs%20for%20warnings%20or%20silent%20fallbacks%2A%2A%20during%20the%20API%20upload.%20Even%20if%20the%20API%20returns%20200%20OK%2C%20the%20backend%20may%20have%20defaulted%20to%20flat%20segmentation%20or%20ignored%20some%20settings%20if%20it%20detected%20a%20subtle%20issue.%0A4.%20%2A%2AIf%20you%20are%20on%20Dify%20v1.11.1%2C%20consider%20updating%2A%2A%E2%80%94recent%20versions%20have%20improved%20validation%20and%20error%20reporting%20for%20process_rule%20payloads%2C%20which%20helps%20catch%20silent%20fallbacks%20%5B%5Bsource%5D%5D%28https%3A//github.com/langgenius/dify/pull/28810%29.%0A%0AIf%20after%20all%20these%20checks%20the%20process_rule%20and%20file%20content%20are%20truly%20identical%2C%20but%20the%20segment%20counts%20still%20differ%2C%20it%E2%80%99s%20likely%20due%20to%20a%20subtle%20difference%20in%20the%20file%20as%20received%20by%20the%20backend%20or%20a%20rare%20bug%20in%20the%20segmentation%20logic.%20Comparing%20the%20actual%20segment%20content%20and%20checking%20logs%20should%20help%20pinpoint%20the%20root%20cause.) [![加入不和谐](https://camo.githubusercontent.com/7d2066700925db1e370d956834f9ea0e5bc92cecbe84963fbd2a363e45091b4e/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6a6f696e2d3538363546323f6c6f676f3d646973636f7264266c6f676f436f6c6f723d7768697465266c6162656c3d)](https://go.dosu.dev/discord-bot) [![分享至X](https://camo.githubusercontent.com/a8bd7f3fcf3f5e3bd124eee9ae50ae2ac17b40971726665ac2121d48f9aee155/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f582d73686172652d626c61636b)](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/langgenius/dify/issues/30071) > > Different Processing Mechanisms: Frontend Upload: Goes through the RAG Pipeline workflow, which uses an optimized processing chain. API Upload: Goes through basic document indexing, using the standard processing flow.So this leads to different document processing results. This is a knowledge base processing bug.
yindo commented 2026-02-21 20:11:36 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Dec 24, 2025):

根据 Dify 后端代码和最新的 PR 验证，API 上传和前端上传（无论是否走 RAG pipeline）最终都会走同一套后端文档处理和分段逻辑：两者都会把 process_rule 存到数据库，然后由统一的 IndexingRunner 进行提取、清洗、分段和入库。不存在“前端优化链路”或“API 基础链路”导致分段逻辑不同的情况，所有分段和预处理都严格依赖于数据库中的 process_rule 配置，并且两种上传方式都会触发同样的异步分段任务【详见源码分析】。

如果你确认 process_rule 完全一致，理论上分段结果也应一致。出现分段数差异，常见原因有：

  • 上传的文件内容实际有细微差异（如编码、换行符、隐藏字符等），导致分段点不同。
  • process_rule 字段结构或嵌套有微小差异，API 端容易因字段缺失或嵌套错误被后端静默回退为默认分段（如扁平分段），即使返回 200 OK【相关 PR #28810 已增强校验】。
  • 某些 aggressive 的预处理规则（如 remove_urls_emails）会清理掉大量内容，影响分段数。

建议排查步骤：

  1. 确认两次上传的文件完全一致（可用 md5sum 校验）。
  2. 对比 process_rule 的结构和内容，确保所有字段和嵌套都与前端一致。
  3. 用 Dify UI 的分段预览或导出功能，分别查看两次上传后预处理和分段前的原始文本内容。
  4. 检查后端日志，确认 API 上传时没有校验/回退警告。
  5. 升级到包含 PR #28810 的新版本，避免因 process_rule 校验不严导致的静默回退。

如果经过上述排查，process_rule 和文件内容都完全一致但分段数仍不同，极有可能是文件内容在传输或解析过程中有细微差异，或极少数情况下后端分段逻辑有 bug。此时建议导出实际分段内容进行比对，定位根因。

参考：源码分析与 PR 说明

To reply, just mention @dosu.

How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other

Chat with Dosu Open in Cursor Join Discord Share on X

@dosubot[bot] commented on GitHub (Dec 24, 2025): <!-- Answer --> 根据 Dify 后端代码和最新的 PR 验证，API 上传和前端上传（无论是否走 RAG pipeline）最终都会走同一套后端文档处理和分段逻辑：两者都会把 process_rule 存到数据库，然后由统一的 IndexingRunner 进行提取、清洗、分段和入库。不存在“前端优化链路”或“API 基础链路”导致分段逻辑不同的情况，所有分段和预处理都严格依赖于数据库中的 process_rule 配置，并且两种上传方式都会触发同样的异步分段任务【详见源码分析】。 如果你确认 process_rule 完全一致，理论上分段结果也应一致。出现分段数差异，常见原因有： - 上传的文件内容实际有细微差异（如编码、换行符、隐藏字符等），导致分段点不同。 - process_rule 字段结构或嵌套有微小差异，API 端容易因字段缺失或嵌套错误被后端静默回退为默认分段（如扁平分段），即使返回 200 OK【相关 PR #28810 已增强校验】。 - 某些 aggressive 的预处理规则（如 remove_urls_emails）会清理掉大量内容，影响分段数。 建议排查步骤： 1. 确认两次上传的文件完全一致（可用 md5sum 校验）。 2. 对比 process_rule 的结构和内容，确保所有字段和嵌套都与前端一致。 3. 用 Dify UI 的分段预览或导出功能，分别查看两次上传后预处理和分段前的原始文本内容。 4. 检查后端日志，确认 API 上传时没有校验/回退警告。 5. 升级到包含 PR #28810 的新版本，避免因 process_rule 校验不严导致的静默回退。 如果经过上述排查，process_rule 和文件内容都完全一致但分段数仍不同，极有可能是文件内容在传输或解析过程中有细微差异，或极少数情况下后端分段逻辑有 bug。此时建议导出实际分段内容进行比对，定位根因。 参考：[源码分析与 PR 说明](https://github.com/langgenius/dify/pull/28810) <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- <sup>How did I do? [Good](https://app.dosu.dev/response-feedback/9c912fe8-adab-4cf1-ba0b-6754aafda399?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/9c912fe8-adab-4cf1-ba0b-6754aafda399?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/9c912fe8-adab-4cf1-ba0b-6754aafda399?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/9c912fe8-adab-4cf1-ba0b-6754aafda399?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/9c912fe8-adab-4cf1-ba0b-6754aafda399?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/9c912fe8-adab-4cf1-ba0b-6754aafda399?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/9c912fe8-adab-4cf1-ba0b-6754aafda399?feedback_type=other)</sup> [![Chat with Dosu](https://dosu.dev/dosu-chat-badge.svg)](https://app.dosu.dev/6088a4f4-073a-43f3-b3d4-4f06fb0c05a0/ask?utm_source=github)&nbsp;[![Open in Cursor](https://dosu.dev/dosu-in-cursor.svg)](https://cursor.com/link/prompt?text=%E6%A0%B9%E6%8D%AE%20Dify%20%E5%90%8E%E7%AB%AF%E4%BB%A3%E7%A0%81%E5%92%8C%E6%9C%80%E6%96%B0%E7%9A%84%20PR%20%E9%AA%8C%E8%AF%81%EF%BC%8CAPI%20%E4%B8%8A%E4%BC%A0%E5%92%8C%E5%89%8D%E7%AB%AF%E4%B8%8A%E4%BC%A0%EF%BC%88%E6%97%A0%E8%AE%BA%E6%98%AF%E5%90%A6%E8%B5%B0%20RAG%20pipeline%EF%BC%89%E6%9C%80%E7%BB%88%E9%83%BD%E4%BC%9A%E8%B5%B0%E5%90%8C%E4%B8%80%E5%A5%97%E5%90%8E%E7%AB%AF%E6%96%87%E6%A1%A3%E5%A4%84%E7%90%86%E5%92%8C%E5%88%86%E6%AE%B5%E9%80%BB%E8%BE%91%EF%BC%9A%E4%B8%A4%E8%80%85%E9%83%BD%E4%BC%9A%E6%8A%8A%20process_rule%20%E5%AD%98%E5%88%B0%E6%95%B0%E6%8D%AE%E5%BA%93%EF%BC%8C%E7%84%B6%E5%90%8E%E7%94%B1%E7%BB%9F%E4%B8%80%E7%9A%84%20IndexingRunner%20%E8%BF%9B%E8%A1%8C%E6%8F%90%E5%8F%96%E3%80%81%E6%B8%85%E6%B4%97%E3%80%81%E5%88%86%E6%AE%B5%E5%92%8C%E5%85%A5%E5%BA%93%E3%80%82%E4%B8%8D%E5%AD%98%E5%9C%A8%E2%80%9C%E5%89%8D%E7%AB%AF%E4%BC%98%E5%8C%96%E9%93%BE%E8%B7%AF%E2%80%9D%E6%88%96%E2%80%9CAPI%20%E5%9F%BA%E7%A1%80%E9%93%BE%E8%B7%AF%E2%80%9D%E5%AF%BC%E8%87%B4%E5%88%86%E6%AE%B5%E9%80%BB%E8%BE%91%E4%B8%8D%E5%90%8C%E7%9A%84%E6%83%85%E5%86%B5%EF%BC%8C%E6%89%80%E6%9C%89%E5%88%86%E6%AE%B5%E5%92%8C%E9%A2%84%E5%A4%84%E7%90%86%E9%83%BD%E4%B8%A5%E6%A0%BC%E4%BE%9D%E8%B5%96%E4%BA%8E%E6%95%B0%E6%8D%AE%E5%BA%93%E4%B8%AD%E7%9A%84%20process_rule%20%E9%85%8D%E7%BD%AE%EF%BC%8C%E5%B9%B6%E4%B8%94%E4%B8%A4%E7%A7%8D%E4%B8%8A%E4%BC%A0%E6%96%B9%E5%BC%8F%E9%83%BD%E4%BC%9A%E8%A7%A6%E5%8F%91%E5%90%8C%E6%A0%B7%E7%9A%84%E5%BC%82%E6%AD%A5%E5%88%86%E6%AE%B5%E4%BB%BB%E5%8A%A1%E3%80%90%E8%AF%A6%E8%A7%81%E6%BA%90%E7%A0%81%E5%88%86%E6%9E%90%E3%80%91%E3%80%82%0A%0A%E5%A6%82%E6%9E%9C%E4%BD%A0%E7%A1%AE%E8%AE%A4%20process_rule%20%E5%AE%8C%E5%85%A8%E4%B8%80%E8%87%B4%EF%BC%8C%E7%90%86%E8%AE%BA%E4%B8%8A%E5%88%86%E6%AE%B5%E7%BB%93%E6%9E%9C%E4%B9%9F%E5%BA%94%E4%B8%80%E8%87%B4%E3%80%82%E5%87%BA%E7%8E%B0%E5%88%86%E6%AE%B5%E6%95%B0%E5%B7%AE%E5%BC%82%EF%BC%8C%E5%B8%B8%E8%A7%81%E5%8E%9F%E5%9B%A0%E6%9C%89%EF%BC%9A%0A%0A-%20%E4%B8%8A%E4%BC%A0%E7%9A%84%E6%96%87%E4%BB%B6%E5%86%85%E5%AE%B9%E5%AE%9E%E9%99%85%E6%9C%89%E7%BB%86%E5%BE%AE%E5%B7%AE%E5%BC%82%EF%BC%88%E5%A6%82%E7%BC%96%E7%A0%81%E3%80%81%E6%8D%A2%E8%A1%8C%E7%AC%A6%E3%80%81%E9%9A%90%E8%97%8F%E5%AD%97%E7%AC%A6%E7%AD%89%EF%BC%89%EF%BC%8C%E5%AF%BC%E8%87%B4%E5%88%86%E6%AE%B5%E7%82%B9%E4%B8%8D%E5%90%8C%E3%80%82%0A-%20process_rule%20%E5%AD%97%E6%AE%B5%E7%BB%93%E6%9E%84%E6%88%96%E5%B5%8C%E5%A5%97%E6%9C%89%E5%BE%AE%E5%B0%8F%E5%B7%AE%E5%BC%82%EF%BC%8CAPI%20%E7%AB%AF%E5%AE%B9%E6%98%93%E5%9B%A0%E5%AD%97%E6%AE%B5%E7%BC%BA%E5%A4%B1%E6%88%96%E5%B5%8C%E5%A5%97%E9%94%99%E8%AF%AF%E8%A2%AB%E5%90%8E%E7%AB%AF%E9%9D%99%E9%BB%98%E5%9B%9E%E9%80%80%E4%B8%BA%E9%BB%98%E8%AE%A4%E5%88%86%E6%AE%B5%EF%BC%88%E5%A6%82%E6%89%81%E5%B9%B3%E5%88%86%E6%AE%B5%EF%BC%89%EF%BC%8C%E5%8D%B3%E4%BD%BF%E8%BF%94%E5%9B%9E%20200%20OK%E3%80%90%E7%9B%B8%E5%85%B3%20PR%20%2328810%20%E5%B7%B2%E5%A2%9E%E5%BC%BA%E6%A0%A1%E9%AA%8C%E3%80%91%E3%80%82%0A-%20%E6%9F%90%E4%BA%9B%20aggressive%20%E7%9A%84%E9%A2%84%E5%A4%84%E7%90%86%E8%A7%84%E5%88%99%EF%BC%88%E5%A6%82%20remove_urls_emails%EF%BC%89%E4%BC%9A%E6%B8%85%E7%90%86%E6%8E%89%E5%A4%A7%E9%87%8F%E5%86%85%E5%AE%B9%EF%BC%8C%E5%BD%B1%E5%93%8D%E5%88%86%E6%AE%B5%E6%95%B0%E3%80%82%0A%0A%E5%BB%BA%E8%AE%AE%E6%8E%92%E6%9F%A5%E6%AD%A5%E9%AA%A4%EF%BC%9A%0A%0A1.%20%E7%A1%AE%E8%AE%A4%E4%B8%A4%E6%AC%A1%E4%B8%8A%E4%BC%A0%E7%9A%84%E6%96%87%E4%BB%B6%E5%AE%8C%E5%85%A8%E4%B8%80%E8%87%B4%EF%BC%88%E5%8F%AF%E7%94%A8%20md5sum%20%E6%A0%A1%E9%AA%8C%EF%BC%89%E3%80%82%0A2.%20%E5%AF%B9%E6%AF%94%20process_rule%20%E7%9A%84%E7%BB%93%E6%9E%84%E5%92%8C%E5%86%85%E5%AE%B9%EF%BC%8C%E7%A1%AE%E4%BF%9D%E6%89%80%E6%9C%89%E5%AD%97%E6%AE%B5%E5%92%8C%E5%B5%8C%E5%A5%97%E9%83%BD%E4%B8%8E%E5%89%8D%E7%AB%AF%E4%B8%80%E8%87%B4%E3%80%82%0A3.%20%E7%94%A8%20Dify%20UI%20%E7%9A%84%E5%88%86%E6%AE%B5%E9%A2%84%E8%A7%88%E6%88%96%E5%AF%BC%E5%87%BA%E5%8A%9F%E8%83%BD%EF%BC%8C%E5%88%86%E5%88%AB%E6%9F%A5%E7%9C%8B%E4%B8%A4%E6%AC%A1%E4%B8%8A%E4%BC%A0%E5%90%8E%E9%A2%84%E5%A4%84%E7%90%86%E5%92%8C%E5%88%86%E6%AE%B5%E5%89%8D%E7%9A%84%E5%8E%9F%E5%A7%8B%E6%96%87%E6%9C%AC%E5%86%85%E5%AE%B9%E3%80%82%0A4.%20%E6%A3%80%E6%9F%A5%E5%90%8E%E7%AB%AF%E6%97%A5%E5%BF%97%EF%BC%8C%E7%A1%AE%E8%AE%A4%20API%20%E4%B8%8A%E4%BC%A0%E6%97%B6%E6%B2%A1%E6%9C%89%E6%A0%A1%E9%AA%8C/%E5%9B%9E%E9%80%80%E8%AD%A6%E5%91%8A%E3%80%82%0A5.%20%E5%8D%87%E7%BA%A7%E5%88%B0%E5%8C%85%E5%90%AB%20PR%20%2328810%20%E7%9A%84%E6%96%B0%E7%89%88%E6%9C%AC%EF%BC%8C%E9%81%BF%E5%85%8D%E5%9B%A0%20process_rule%20%E6%A0%A1%E9%AA%8C%E4%B8%8D%E4%B8%A5%E5%AF%BC%E8%87%B4%E7%9A%84%E9%9D%99%E9%BB%98%E5%9B%9E%E9%80%80%E3%80%82%0A%0A%E5%A6%82%E6%9E%9C%E7%BB%8F%E8%BF%87%E4%B8%8A%E8%BF%B0%E6%8E%92%E6%9F%A5%EF%BC%8Cprocess_rule%20%E5%92%8C%E6%96%87%E4%BB%B6%E5%86%85%E5%AE%B9%E9%83%BD%E5%AE%8C%E5%85%A8%E4%B8%80%E8%87%B4%E4%BD%86%E5%88%86%E6%AE%B5%E6%95%B0%E4%BB%8D%E4%B8%8D%E5%90%8C%EF%BC%8C%E6%9E%81%E6%9C%89%E5%8F%AF%E8%83%BD%E6%98%AF%E6%96%87%E4%BB%B6%E5%86%85%E5%AE%B9%E5%9C%A8%E4%BC%A0%E8%BE%93%E6%88%96%E8%A7%A3%E6%9E%90%E8%BF%87%E7%A8%8B%E4%B8%AD%E6%9C%89%E7%BB%86%E5%BE%AE%E5%B7%AE%E5%BC%82%EF%BC%8C%E6%88%96%E6%9E%81%E5%B0%91%E6%95%B0%E6%83%85%E5%86%B5%E4%B8%8B%E5%90%8E%E7%AB%AF%E5%88%86%E6%AE%B5%E9%80%BB%E8%BE%91%E6%9C%89%20bug%E3%80%82%E6%AD%A4%E6%97%B6%E5%BB%BA%E8%AE%AE%E5%AF%BC%E5%87%BA%E5%AE%9E%E9%99%85%E5%88%86%E6%AE%B5%E5%86%85%E5%AE%B9%E8%BF%9B%E8%A1%8C%E6%AF%94%E5%AF%B9%EF%BC%8C%E5%AE%9A%E4%BD%8D%E6%A0%B9%E5%9B%A0%E3%80%82%0A%0A%E5%8F%82%E8%80%83%EF%BC%9A%5B%E6%BA%90%E7%A0%81%E5%88%86%E6%9E%90%E4%B8%8E%20PR%20%E8%AF%B4%E6%98%8E%5D%28https%3A//github.com/langgenius/dify/pull/28810%29)&nbsp;[![Join Discord](https://img.shields.io/badge/join-5865F2?logo=discord&logoColor=white&label=)](https://go.dosu.dev/discord-bot)&nbsp;[![Share on X](https://img.shields.io/badge/X-share-black)](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/langgenius/dify/issues/30071)
yindo commented 2026-02-21 20:11:36 -05:00
Author
Owner
Copy Link
Copy Source

@suntao2015005848 commented on GitHub (Dec 24, 2025):

Based on the backend code and multiple issue reports, both API and frontend uploads in Dify use the exact same segmentation and preprocessing pipeline. If your process_rule structure and values are truly identical, segmentation results should also be identical—unless there is a subtle difference in the file content as received by the backend (for example, encoding, line endings, or invisible characters) or a silent fallback to defaults due to a hidden validation issue [source] [source] [source].

Here’s what you can do next:

  1. Ensure you are uploading the exact same file (byte-for-byte) via both API and frontend. Even a small difference in encoding or line endings can affect how the backend splits the document.
  2. Compare the raw extracted text (after preprocessing, before segmentation) for both uploads. If possible, use the Dify UI’s segment preview or export feature to inspect the actual chunk boundaries and content.
  3. Check backend logs for warnings or silent fallbacks during the API upload. Even if the API returns 200 OK, the backend may have defaulted to flat segmentation or ignored some settings if it detected a subtle issue.
  4. If you are on Dify v1.11.1, consider updating—recent versions have improved validation and error reporting for process_rule payloads, which helps catch silent fallbacks [source].

If after all these checks the process_rule and file content are truly identical, but the segment counts still differ, it’s likely due to a subtle difference in the file as received by the backend or a rare bug in the segmentation logic. Comparing the actual segment content and checking logs should help pinpoint the root cause.

To reply, just mention @dosu.

How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other

Chat with Dosu Open in Cursor Join Discord Share on X

My RAG Pipeline workflow skips the Dify Extractor and directly uses the Document Extractor node.

技术文档知识库_父子分段.txt

@suntao2015005848 commented on GitHub (Dec 24, 2025): > Based on the backend code and multiple issue reports, both API and frontend uploads in Dify use the exact same segmentation and preprocessing pipeline. If your process_rule structure and values are truly identical, segmentation results should also be identical—unless there is a subtle difference in the file content as received by the backend (for example, encoding, line endings, or invisible characters) or a silent fallback to defaults due to a hidden validation issue [[source]](https://github.com/langgenius/dify/issues/23933) [[source]](https://github.com/langgenius/dify/issues/22270) [[source]](https://github.com/langgenius/dify/issues/28763). > > Here’s what you can do next: > > 1. **Ensure you are uploading the exact same file (byte-for-byte) via both API and frontend**. Even a small difference in encoding or line endings can affect how the backend splits the document. > 2. **Compare the raw extracted text (after preprocessing, before segmentation) for both uploads**. If possible, use the Dify UI’s segment preview or export feature to inspect the actual chunk boundaries and content. > 3. **Check backend logs for warnings or silent fallbacks** during the API upload. Even if the API returns 200 OK, the backend may have defaulted to flat segmentation or ignored some settings if it detected a subtle issue. > 4. **If you are on Dify v1.11.1, consider updating**—recent versions have improved validation and error reporting for process_rule payloads, which helps catch silent fallbacks [[source]](https://github.com/langgenius/dify/pull/28810). > > If after all these checks the process_rule and file content are truly identical, but the segment counts still differ, it’s likely due to a subtle difference in the file as received by the backend or a rare bug in the segmentation logic. Comparing the actual segment content and checking logs should help pinpoint the root cause. > > _To reply, just mention [@dosu](https://go.dosu.dev/dosubot)._ > > How did I do? [Good](https://app.dosu.dev/response-feedback/ea542fc9-0d26-485c-adb1-6b22d4f6d180?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/ea542fc9-0d26-485c-adb1-6b22d4f6d180?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/ea542fc9-0d26-485c-adb1-6b22d4f6d180?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/ea542fc9-0d26-485c-adb1-6b22d4f6d180?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/ea542fc9-0d26-485c-adb1-6b22d4f6d180?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/ea542fc9-0d26-485c-adb1-6b22d4f6d180?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/ea542fc9-0d26-485c-adb1-6b22d4f6d180?feedback_type=other) > > [![Chat with Dosu](https://camo.githubusercontent.com/7c571478962a6b8a5d47cac74b6824c75e4f6731bba52377e6007f9150462923/68747470733a2f2f646f73752e6465762f646f73752d636861742d62616467652e737667)](https://app.dosu.dev/6088a4f4-073a-43f3-b3d4-4f06fb0c05a0/ask?utm_source=github) [![Open in Cursor](https://camo.githubusercontent.com/bdf7dbded65ee585887f47e5fd6fa099ea0933088b4b03d02e83b4def50154d4/68747470733a2f2f646f73752e6465762f646f73752d696e2d637572736f722e737667)](https://cursor.com/link/prompt?text=Based%20on%20the%20backend%20code%20and%20multiple%20issue%20reports%2C%20both%20API%20and%20frontend%20uploads%20in%20Dify%20use%20the%20exact%20same%20segmentation%20and%20preprocessing%20pipeline.%20If%20your%20process_rule%20structure%20and%20values%20are%20truly%20identical%2C%20segmentation%20results%20should%20also%20be%20identical%E2%80%94unless%20there%20is%20a%20subtle%20difference%20in%20the%20file%20content%20as%20received%20by%20the%20backend%20%28for%20example%2C%20encoding%2C%20line%20endings%2C%20or%20invisible%20characters%29%20or%20a%20silent%20fallback%20to%20defaults%20due%20to%20a%20hidden%20validation%20issue%20%5B%5Bsource%5D%5D%28https%3A//github.com/langgenius/dify/issues/23933%29%20%5B%5Bsource%5D%5D%28https%3A//github.com/langgenius/dify/issues/22270%29%20%5B%5Bsource%5D%5D%28https%3A//github.com/langgenius/dify/issues/28763%29.%0A%0AHere%E2%80%99s%20what%20you%20can%20do%20next%3A%0A%0A1.%20%2A%2AEnsure%20you%20are%20uploading%20the%20exact%20same%20file%20%28byte-for-byte%29%20via%20both%20API%20and%20frontend%2A%2A.%20Even%20a%20small%20difference%20in%20encoding%20or%20line%20endings%20can%20affect%20how%20the%20backend%20splits%20the%20document.%0A2.%20%2A%2ACompare%20the%20raw%20extracted%20text%20%28after%20preprocessing%2C%20before%20segmentation%29%20for%20both%20uploads%2A%2A.%20If%20possible%2C%20use%20the%20Dify%20UI%E2%80%99s%20segment%20preview%20or%20export%20feature%20to%20inspect%20the%20actual%20chunk%20boundaries%20and%20content.%0A3.%20%2A%2ACheck%20backend%20logs%20for%20warnings%20or%20silent%20fallbacks%2A%2A%20during%20the%20API%20upload.%20Even%20if%20the%20API%20returns%20200%20OK%2C%20the%20backend%20may%20have%20defaulted%20to%20flat%20segmentation%20or%20ignored%20some%20settings%20if%20it%20detected%20a%20subtle%20issue.%0A4.%20%2A%2AIf%20you%20are%20on%20Dify%20v1.11.1%2C%20consider%20updating%2A%2A%E2%80%94recent%20versions%20have%20improved%20validation%20and%20error%20reporting%20for%20process_rule%20payloads%2C%20which%20helps%20catch%20silent%20fallbacks%20%5B%5Bsource%5D%5D%28https%3A//github.com/langgenius/dify/pull/28810%29.%0A%0AIf%20after%20all%20these%20checks%20the%20process_rule%20and%20file%20content%20are%20truly%20identical%2C%20but%20the%20segment%20counts%20still%20differ%2C%20it%E2%80%99s%20likely%20due%20to%20a%20subtle%20difference%20in%20the%20file%20as%20received%20by%20the%20backend%20or%20a%20rare%20bug%20in%20the%20segmentation%20logic.%20Comparing%20the%20actual%20segment%20content%20and%20checking%20logs%20should%20help%20pinpoint%20the%20root%20cause.) [![Join Discord](https://camo.githubusercontent.com/7d2066700925db1e370d956834f9ea0e5bc92cecbe84963fbd2a363e45091b4e/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6a6f696e2d3538363546323f6c6f676f3d646973636f7264266c6f676f436f6c6f723d7768697465266c6162656c3d)](https://go.dosu.dev/discord-bot) [![Share on X](https://camo.githubusercontent.com/a8bd7f3fcf3f5e3bd124eee9ae50ae2ac17b40971726665ac2121d48f9aee155/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f582d73686172652d626c61636b)](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/langgenius/dify/issues/30071) My RAG Pipeline workflow skips the Dify Extractor and directly uses the Document Extractor node. [技术文档知识库_父子分段.txt](https://github.com/user-attachments/files/24323370/_.txt)
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#21247
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?