langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-08-27 11:11:21 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

[PR #27754] fix(rag): include built-in metadata fields in automatic filtering #31856

New Issue
Open
opened 2026-02-21 20:50:16 -05:00 by yindo · 0 comments
yindo commented 2026-02-21 20:50:16 -05:00
Owner
Copy Link
Copy Source

Original Pull Request: https://github.com/langgenius/dify/pull/27754

State: open
Merged: No

Fixes #27753

Summary

Fixed automatic metadata filtering to properly support built-in metadata fields when built_in_field_enabled is enabled for a dataset.

Problem: The _automatic_metadata_filter_func method only queried custom metadata fields from the dataset_metadatas table, ignoring built-in fields (document_name, uploader, upload_date, last_update_date, source) even when they were enabled.

Solution:

  • Added _get_all_metadata_fields() private method to both DatasetRetrieval and KnowledgeRetrievalNode classes
  • This method queries both custom metadata fields and built-in fields (when enabled)
  • Refactored _automatic_metadata_filter_func to use the new helper method
  • Eliminated code duplication between the two modules

Impact: Automatic metadata filtering now works correctly with both custom and built-in metadata fields, enabling queries like "find documents uploaded by user X" or "show files uploaded in 2024".

Changes

  • Modified api/core/rag/retrieval/dataset_retrieval.py

    • Added BuiltInField import
    • Added _get_all_metadata_fields() method (lines 966-987)
    • Updated _automatic_metadata_filter_func() to use the new method

  • Modified api/core/workflow/nodes/knowledge_retrieval/knowledge_retrieval_node.py

    • Added BuiltInField import
    • Added _get_all_metadata_fields() method (lines 524-547)
    • Updated _automatic_metadata_filter_func() to use the new method

Checklist

  • This change requires a documentation update, included: Dify Document
  • I understand that this PR may be closed in case there was no previous discussion or issues. (This doesn't apply to typos!)
  • I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
  • I've updated the documentation accordingly.
  • I ran `dev/reformat`(backend) and `cd web && npx lint-staged`(frontend) to appease the lint gods
**Original Pull Request:** https://github.com/langgenius/dify/pull/27754 **State:** open **Merged:** No --- Fixes #27753 ## Summary Fixed automatic metadata filtering to properly support built-in metadata fields when `built_in_field_enabled` is enabled for a dataset. **Problem**: The `_automatic_metadata_filter_func` method only queried custom metadata fields from the `dataset_metadatas` table, ignoring built-in fields (document_name, uploader, upload_date, last_update_date, source) even when they were enabled. **Solution**: - Added `_get_all_metadata_fields()` private method to both `DatasetRetrieval` and `KnowledgeRetrievalNode` classes - This method queries both custom metadata fields and built-in fields (when enabled) - Refactored `_automatic_metadata_filter_func` to use the new helper method - Eliminated code duplication between the two modules **Impact**: Automatic metadata filtering now works correctly with both custom and built-in metadata fields, enabling queries like "find documents uploaded by user X" or "show files uploaded in 2024". ## Changes - Modified `api/core/rag/retrieval/dataset_retrieval.py` - Added `BuiltInField` import - Added `_get_all_metadata_fields()` method (lines 966-987) - Updated `_automatic_metadata_filter_func()` to use the new method - Modified `api/core/workflow/nodes/knowledge_retrieval/knowledge_retrieval_node.py` - Added `BuiltInField` import - Added `_get_all_metadata_fields()` method (lines 524-547) - Updated `_automatic_metadata_filter_func()` to use the new method ## Checklist - [ ] This change requires a documentation update, included: [Dify Document](https://github.com/langgenius/dify-docs) - [x] I understand that this PR may be closed in case there was no previous discussion or issues. (This doesn't apply to typos!) - [x] I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change. - [x] I've updated the documentation accordingly. - [x] I ran \`dev/reformat\`(backend) and \`cd web && npx lint-staged\`(frontend) to appease the lint gods
yindo added the pull-request label 2026-02-21 20:50:16 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#31856
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?