langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 03:55:25 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

PR change might not work as expected #11883

New Issue
Closed
opened 2026-02-21 19:04:33 -05:00 by yindo · 3 comments
yindo commented 2026-02-21 19:04:33 -05:00
Owner
Copy Link
Copy Source

Originally created by @DavideDelbianco on GitHub (Mar 20, 2025).

Originally assigned to: @JohnJyong on GitHub.

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
  • Please do not modify this template :) and fill in all the required fields.

Dify version

1.1.0

Cloud or Self Hosted

Cloud

Steps to reproduce

Issue #16256 was closed with PR #16307
I think the command that was added will not work as expected, please double check.

From the source code of this PR it seems to me that there's no filtering logic for the built-in metadata keys
Image

Those keys were/are present in the doc_metadata field
Image

And the UI seems to throw an error when you attempt to add a custom metadata key that's already part of the built-in ones

Image

✔️ Expected Behavior

Orphaned metadata should be migrated to new system

❌ Actual Behavior

The migration process could generate invalid metadata, redundant with build-in keys or not work at all because of error caused by duplicated keys

Originally created by @DavideDelbianco on GitHub (Mar 20, 2025). Originally assigned to: @JohnJyong on GitHub. ### Self Checks - [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [x] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:） - [x] Please do not modify this template :) and fill in all the required fields. ### Dify version 1.1.0 ### Cloud or Self Hosted Cloud ### Steps to reproduce Issue #16256 was closed with PR #16307 I think the command that was added will not work as expected, please double check. From the source code of this PR it seems to me that there's no filtering logic for the built-in metadata keys ![Image](https://github.com/user-attachments/assets/9202695c-639c-4489-adbc-d06dc94176e1) Those keys were/are present in the doc_metadata field ![Image](https://github.com/user-attachments/assets/ce4967bd-269a-409a-b012-c16bfe26c023) And the UI seems to throw an error when you attempt to add a custom metadata key that's already part of the built-in ones ![Image](https://github.com/user-attachments/assets/ee03cb6f-cff3-4a2c-9e3f-a1b32d26aee2) ### ✔️ Expected Behavior Orphaned metadata should be migrated to new system ### ❌ Actual Behavior The migration process could generate invalid metadata, redundant with build-in keys or not work at all because of error caused by duplicated keys
yindo added the 🐞 bugcloud labels 2026-02-21 19:04:33 -05:00
yindo closed this issue 2026-02-21 19:04:34 -05:00
yindo commented 2026-02-21 19:04:34 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Mar 20, 2025):

The issue you're encountering seems to be related to the handling of metadata keys during the migration process in PR #16307. Specifically, it appears that there might be a lack of filtering logic for built-in metadata keys, which could lead to conflicts when custom metadata keys overlap with these built-in ones.

In the current implementation, the system differentiates between built-in and custom metadata keys based on their location in the Metadata panel, activation method, and other characteristics [1]. However, if the migration process does not properly account for these differences, it could result in errors or redundant metadata.

To address this, you should ensure that the migration script includes logic to filter out or handle built-in metadata keys separately from custom ones. This could involve:

  1. Validation Checks: Implementing checks to prevent the addition of custom metadata keys that conflict with built-in keys. The code already includes some validation to prevent duplicate metadata names by checking against existing metadata and built-in fields [2].

  2. Error Handling: Ensuring that the application gracefully handles errors when a user attempts to add a custom metadata key that conflicts with a built-in key. This might involve providing clear error messages or preventing the action altogether.

  3. Migration Logic: Reviewing the migration logic to ensure it correctly identifies and processes built-in metadata keys, possibly by adding specific filtering or handling mechanisms for these keys.

By addressing these areas, you can help ensure that the migration process works as expected and avoids generating invalid or redundant metadata.

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other

@dosubot[bot] commented on GitHub (Mar 20, 2025): <!-- Answer --> The issue you're encountering seems to be related to the handling of metadata keys during the migration process in PR #16307. Specifically, it appears that there might be a lack of filtering logic for built-in metadata keys, which could lead to conflicts when custom metadata keys overlap with these built-in ones. In the current implementation, the system differentiates between built-in and custom metadata keys based on their location in the Metadata panel, activation method, and other characteristics <sup>[[1]](https://github.com/langgenius/dify-docs/blob/517bc5b3e25a7ea8476f612a8df0adf47049d489/en/guides/knowledge-base/metadata.md#L57-L107)</sup>. However, if the migration process does not properly account for these differences, it could result in errors or redundant metadata. To address this, you should ensure that the migration script includes logic to filter out or handle built-in metadata keys separately from custom ones. This could involve: 1. **Validation Checks**: Implementing checks to prevent the addition of custom metadata keys that conflict with built-in keys. The code already includes some validation to prevent duplicate metadata names by checking against existing metadata and built-in fields <sup>[[2]](https://github.com/langgenius/dify/blob/abeaea4f79a6dbabb46a9d79224189b8c7fac828/api/services/metadata_service.py)</sup>. 2. **Error Handling**: Ensuring that the application gracefully handles errors when a user attempts to add a custom metadata key that conflicts with a built-in key. This might involve providing clear error messages or preventing the action altogether. 3. **Migration Logic**: Reviewing the migration logic to ensure it correctly identifies and processes built-in metadata keys, possibly by adding specific filtering or handling mechanisms for these keys. By addressing these areas, you can help ensure that the migration process works as expected and avoids generating invalid or redundant metadata. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.* --- <sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/01f230c4-4831-49fd-b2ca-b941a8997024?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/01f230c4-4831-49fd-b2ca-b941a8997024?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/01f230c4-4831-49fd-b2ca-b941a8997024?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/01f230c4-4831-49fd-b2ca-b941a8997024?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/01f230c4-4831-49fd-b2ca-b941a8997024?feedback_type=hallucination) | [Bug Report](https://app.dosu.dev/response-feedback/01f230c4-4831-49fd-b2ca-b941a8997024?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/01f230c4-4831-49fd-b2ca-b941a8997024?feedback_type=other)</sup>
yindo commented 2026-02-21 19:04:34 -05:00
Author
Owner
Copy Link
Copy Source

@DavideDelbianco commented on GitHub (Mar 20, 2025):

There's also some more leftovers from the old metadata implementation in
api/controllers/console/datasets/dataset_document.py
api/services/dataset_service.py

Image

@DavideDelbianco commented on GitHub (Mar 20, 2025): There's also some more leftovers from the old metadata implementation in `api/controllers/console/datasets/dataset_document.py` `api/services/dataset_service.py` ![Image](https://github.com/user-attachments/assets/7ae5a8fd-8913-406b-b9f9-9e14031edfe0)
yindo commented 2026-02-21 19:04:34 -05:00
Author
Owner
Copy Link
Copy Source

@JohnJyong commented on GitHub (Mar 20, 2025):

@DavideDelbianco thanks for the reminder， i have add the built-in field check in below pr #16371

@JohnJyong commented on GitHub (Mar 20, 2025): @DavideDelbianco thanks for the reminder， i have add the built-in field check in below pr #16371
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label cloud 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#11883
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?