langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 20:15:31 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

[Bug]Chunking Setting in file web, The previously added chunk data will be lost. #15829

New Issue
Closed
opened 2026-02-21 19:23:32 -05:00 by yindo · 8 comments
yindo commented 2026-02-21 19:23:32 -05:00
Owner
Copy Link
Copy Source

Originally created by @Mzs-code on GitHub (Aug 1, 2025).

Originally assigned to: @JohnJyong on GitHub.

Self Checks

  • I have read the Contributing Guide and Language Policy.
  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report, otherwise it will be closed.
  • 【中文用户 & Non English User】请使用英语提交，否则会被关闭 ：）
  • Please do not modify this template :) and fill in all the required fields.

Dify version

1.7.1

Cloud or Self Hosted

Cloud, Self Hosted (Docker)

Steps to reproduce

This is a very dangerous bug.😭
We lost all the test data :(

  1. upload file in knowledge
    Image

  2. add chunk use Web or API
    Image

  3. Save Chunking Setting(Whether modified or not)
    Image
    Image

  4. Previously added chunk will lost
    Image

https://github.com/user-attachments/assets/b223f2ec-25e9-43c8-a4c5-094ab03e529a

✔️ Expected Behavior

Previously added chunk no lost.

❌ Actual Behavior

No response

Originally created by @Mzs-code on GitHub (Aug 1, 2025). Originally assigned to: @JohnJyong on GitHub. ### Self Checks - [x] I have read the [Contributing Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) and [Language Policy](https://github.com/langgenius/dify/issues/1542). - [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report, otherwise it will be closed. - [x] 【中文用户 & Non English User】请使用英语提交，否则会被关闭 ：） - [x] Please do not modify this template :) and fill in all the required fields. ### Dify version 1.7.1 ### Cloud or Self Hosted Cloud, Self Hosted (Docker) ### Steps to reproduce This is a very dangerous bug.😭 We lost all the test data :( 1. upload file in knowledge <img width="2908" height="912" alt="Image" src="https://github.com/user-attachments/assets/f52bc39e-eb63-454c-bf3e-275f6e3d50e2" /> 2. add chunk use Web or API <img width="918" height="782" alt="Image" src="https://github.com/user-attachments/assets/84f5c0ab-4ae8-41e2-b214-c8ef5c985837" /> 3. Save Chunking Setting(Whether modified or not) <img width="1374" height="1582" alt="Image" src="https://github.com/user-attachments/assets/e559b85b-1138-46e3-b943-d82588057999" /> <img width="2060" height="1792" alt="Image" src="https://github.com/user-attachments/assets/64af5b63-c80a-4067-b3f5-3773bf5b041c" /> 4. Previously added chunk will lost <img width="956" height="700" alt="Image" src="https://github.com/user-attachments/assets/4603c11d-7521-4080-bbbc-5aacecd10270" /> https://github.com/user-attachments/assets/b223f2ec-25e9-43c8-a4c5-094ab03e529a ### ✔️ Expected Behavior Previously added chunk no lost. ### ❌ Actual Behavior _No response_
yindo added the 🐞 bug label 2026-02-21 19:23:32 -05:00
yindo closed this issue 2026-02-21 19:23:32 -05:00
yindo commented 2026-02-21 19:23:32 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Aug 1, 2025):

This data loss happens because saving the chunking settings in the web UI triggers backend logic that can wipe and rebuild all chunk data—even if you didn't actually change any settings. The backend treats every save as a potential change and may launch a reindexing task that deletes and recreates chunks and their vector indexes, which leads to the loss of previously added chunk data if safeguards aren't in place to detect no-op saves or preserve custom chunks. There's currently no explicit protection against this in the workflow, which is why you're seeing this dangerous behavior in 1.7.1 [code reference].

Until this is fixed, avoid saving chunking settings unless absolutely necessary, and always back up your data before making changes. There are no documented workarounds or fixes for this specific issue yet. If you need to recover lost data, you would have to restore from a backup.

To reply, just mention @dosu.

How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other  Chat with Dosu Join Discord Share on X

@dosubot[bot] commented on GitHub (Aug 1, 2025): <!-- Answer --> This data loss happens because saving the chunking settings in the web UI triggers backend logic that can wipe and rebuild all chunk data—even if you didn't actually change any settings. The backend treats every save as a potential change and may launch a reindexing task that deletes and recreates chunks and their vector indexes, which leads to the loss of previously added chunk data if safeguards aren't in place to detect no-op saves or preserve custom chunks. There's currently no explicit protection against this in the workflow, which is why you're seeing this dangerous behavior in 1.7.1 [[code reference](https://github.com/langgenius/dify/blob/f78b903a4946e0a142d0e8ffe78d78d590b4d9f7/api/services/dataset_service.py)]. Until this is fixed, avoid saving chunking settings unless absolutely necessary, and always back up your data before making changes. There are no documented workarounds or fixes for this specific issue yet. If you need to recover lost data, you would have to restore from a backup. <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- <sup>How did I do? [Good](https://app.dosu.dev/response-feedback/c922e83a-b836-41e4-9b4e-f9c6760ed616?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/c922e83a-b836-41e4-9b4e-f9c6760ed616?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/c922e83a-b836-41e4-9b4e-f9c6760ed616?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/c922e83a-b836-41e4-9b4e-f9c6760ed616?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/c922e83a-b836-41e4-9b4e-f9c6760ed616?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/c922e83a-b836-41e4-9b4e-f9c6760ed616?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/c922e83a-b836-41e4-9b4e-f9c6760ed616?feedback_type=other)</sup>&nbsp;&nbsp;[![Chat with Dosu](https://dosu.dev/dosu-chat-badge.svg)](https://app.dosu.dev/6088a4f4-073a-43f3-b3d4-4f06fb0c05a0/ask?utm_source=github)&nbsp;[![Join Discord](https://img.shields.io/badge/join-5865F2?logo=discord&logoColor=white&label=)](https://go.dosu.dev/discord-bot)&nbsp;[![Share on X](https://img.shields.io/badge/X-share-black)](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/langgenius/dify/issues/23279)
yindo commented 2026-02-21 19:23:32 -05:00
Author
Owner
Copy Link
Copy Source

@JohnJyong commented on GitHub (Aug 1, 2025):

the action re-chunk will splite your original file and cover your current documen .

@JohnJyong commented on GitHub (Aug 1, 2025): the action `re-chunk` will splite your original file and cover your current documen .
yindo commented 2026-02-21 19:23:32 -05:00
Author
Owner
Copy Link
Copy Source

@Mzs-code commented on GitHub (Aug 1, 2025):

@JohnJyong I understand your explanation.
But, It is no described anywhere, 'change Chunking Setting will base original file '

Intuitively from the use of products, When the function of add chunk(web or api) is provided,
the action re-chunk base latest data will be better.

On the other hand, updating the knowledge base step by step is more realistic.

@Mzs-code commented on GitHub (Aug 1, 2025): @JohnJyong I understand your explanation. But, It is no described anywhere, 'change Chunking Setting will base original file ' Intuitively from the use of products, When the function of add chunk(web or api) is provided, the action re-chunk base latest data will be better. On the other hand, updating the knowledge base step by step is more realistic.
yindo commented 2026-02-21 19:23:32 -05:00
Author
Owner
Copy Link
Copy Source

@Mzs-code commented on GitHub (Aug 1, 2025):

Because the built-in Delimiter function is not satisfied, we uploaded 1000 chunks using api.

@Mzs-code commented on GitHub (Aug 1, 2025): Because the built-in Delimiter function is not satisfied, we uploaded 1000 chunks using api.
yindo commented 2026-02-21 19:23:32 -05:00
Author
Owner
Copy Link
Copy Source

@Mzs-code commented on GitHub (Aug 1, 2025):

I suggest that a strong prompt should be added to the save&process button.
This is the simplest way at present.

@Mzs-code commented on GitHub (Aug 1, 2025): I suggest that a strong prompt should be added to the save&process button. This is the simplest way at present.
yindo commented 2026-02-21 19:23:32 -05:00
Author
Owner
Copy Link
Copy Source

@quicksandznzn commented on GitHub (Aug 3, 2025):

the action re-chunk will splite your original file and cover your current documen .

Would it be possible to add chunk identifiers to document_segments to distinguish between manually added chunks and automatically segmented ones, so that during re-chunking we only reprocess the automatically segmented chunks?

@quicksandznzn commented on GitHub (Aug 3, 2025): > the action `re-chunk` will splite your original file and cover your current documen . Would it be possible to add chunk identifiers to document_segments to distinguish between manually added chunks and automatically segmented ones, so that during re-chunking we only reprocess the automatically segmented chunks?
yindo commented 2026-02-21 19:23:33 -05:00
Author
Owner
Copy Link
Copy Source

@Mzs-code commented on GitHub (Aug 4, 2025):

the action re-chunk will splite your original file and cover your current documen .

Would it be possible to add chunk identifiers to document_segments to distinguish between manually added chunks and automatically segmented ones, so that during re-chunking we only reprocess the automatically segmented chunks?

Good Idea!

@Mzs-code commented on GitHub (Aug 4, 2025): > > the action `re-chunk` will splite your original file and cover your current documen . > > Would it be possible to add chunk identifiers to document_segments to distinguish between manually added chunks and automatically segmented ones, so that during re-chunking we only reprocess the automatically segmented chunks? Good Idea!
yindo commented 2026-02-21 19:23:33 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Sep 4, 2025):

Hi, @Mzs-code. I'm Dosu, and I'm helping the Dify team manage their backlog and am marking this issue as stale.

Issue Summary:

  • You reported a critical bug in version 1.7.1 where saving chunking settings in the web UI causes loss of previously added chunk data.
  • The backend rebuilds all chunks on every save, even if unchanged, leading to data loss.
  • A contributor explained this behavior and advised avoiding saving chunking settings unless necessary and backing up data.
  • You and others discussed the need for clearer warnings and suggested improvements like distinguishing manually added chunks from auto-segmented ones.
  • The issue remains open with no current fix or workaround.

Next Steps:

  • Please let me know if this issue is still relevant with the latest version of Dify by commenting here to keep the discussion open.
  • Otherwise, this issue will be automatically closed in 15 days.

Thank you for your understanding and contribution!

@dosubot[bot] commented on GitHub (Sep 4, 2025): Hi, @Mzs-code. I'm [Dosu](https://dosu.dev), and I'm helping the Dify team manage their backlog and am marking this issue as stale. **Issue Summary:** - You reported a critical bug in version 1.7.1 where saving chunking settings in the web UI causes loss of previously added chunk data. - The backend rebuilds all chunks on every save, even if unchanged, leading to data loss. - A contributor explained this behavior and advised avoiding saving chunking settings unless necessary and backing up data. - You and others discussed the need for clearer warnings and suggested improvements like distinguishing manually added chunks from auto-segmented ones. - The issue remains open with no current fix or workaround. **Next Steps:** - Please let me know if this issue is still relevant with the latest version of Dify by commenting here to keep the discussion open. - Otherwise, this issue will be automatically closed in 15 days. Thank you for your understanding and contribution!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#15829
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?