langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-08-24 22:11:36 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

metadata filtering is not valid in workflow knowledge retrive node #14404

New Issue
Closed
opened 2026-02-21 19:17:05 -05:00 by yindo · 6 comments
yindo commented 2026-02-21 19:17:05 -05:00
Owner
Copy Link
Copy Source

Originally created by @winqihe on GitHub (Jun 3, 2025).

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:)
  • Please do not modify this template :) and fill in all the required fields.

Dify version

1.3.1

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

I use knowledge retrive node in workflow , its work, when I use metadata filtering , its not work, in this node ,return result: [] , its not correct, you know , I disabled metadata filtering , retrive is correct , I checked my filtering condition in twice , this metadata filtering feature cant work corrent

✔️ Expected Behavior

knowledge retrive node use metadata filtering return result should return retrive data

❌ Actual Behavior

return result : []

Originally created by @winqihe on GitHub (Jun 3, 2025). ### Self Checks - [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [x] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:) - [x] Please do not modify this template :) and fill in all the required fields. ### Dify version 1.3.1 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce I use knowledge retrive node in workflow , it`s work, when I use metadata filtering , it`s not work, in this node ,return result: [] , it`s not correct, you know , I disabled metadata filtering , retrive is correct , I checked my filtering condition in twice , this metadata filtering feature can`t work corrent ### ✔️ Expected Behavior knowledge retrive node use metadata filtering return result should return retrive data ### ❌ Actual Behavior return result : []
yindo added the 🐞 bug label 2026-02-21 19:17:05 -05:00
yindo closed this issue 2026-02-21 19:17:05 -05:00
yindo commented 2026-02-21 19:17:05 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Jun 3, 2025):

There are no existing issues or documentation specifically addressing metadata filtering problems in the knowledge retrieve node, but I reviewed the code and logic for this feature in Dify 1.3.1.

Metadata filtering in the knowledge retrieve node supports three modes: disabled, automatic, and manual. When enabled, the node builds filter conditions based on your configuration and applies them to restrict which documents are retrieved. If the filter conditions do not match any documents, the result will be an empty list, as you observed. This can happen if the metadata field names, values, or operators do not exactly match what is stored in your document metadata, or if the filter logic is too restrictive or misconfigured. Supported operators include string, number, and time comparisons such as "contains", "is", "=", "before", etc. [source]

Here are some troubleshooting steps:

  • Double-check that the metadata field names and values in your filtering conditions exactly match those in your documents' metadata (case, type, and formatting matter).
  • Ensure you are using supported comparison operators for the data type of each metadata field.
  • If possible, inspect the actual metadata stored in your database for the relevant documents to confirm the fields and values.
  • Try a very broad filter (for example, a single condition with "not empty") to see if any documents are returned, then narrow down to your intended filter.

The code does not provide explicit error messages if a filter is invalid or does not match any documents—it simply returns an empty result. If you continue to get empty results even with a broad filter, there may be a mismatch between your filter configuration and the actual document metadata, or an issue with how metadata is being indexed or stored [source].

If you can share an example of your metadata filtering condition and a sample of your document metadata, I can help you further pinpoint the issue.

To reply, just mention my name @dosu.

How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other  Join Discord Share on X

@dosubot[bot] commented on GitHub (Jun 3, 2025): <!-- Answer --> There are no existing issues or documentation specifically addressing metadata filtering problems in the knowledge retrieve node, but I reviewed the code and logic for this feature in Dify 1.3.1. Metadata filtering in the knowledge retrieve node supports three modes: disabled, automatic, and manual. When enabled, the node builds filter conditions based on your configuration and applies them to restrict which documents are retrieved. If the filter conditions do not match any documents, the result will be an empty list, as you observed. This can happen if the metadata field names, values, or operators do not exactly match what is stored in your document metadata, or if the filter logic is too restrictive or misconfigured. Supported operators include string, number, and time comparisons such as "contains", "is", "=", "before", etc. [[source]](https://github.com/langgenius/dify/blob/275e86a26c6b86409c391432a010fe0b1096b88e/api/core/rag/entities/metadata_entities.py) Here are some troubleshooting steps: - Double-check that the metadata field names and values in your filtering conditions exactly match those in your documents' metadata (case, type, and formatting matter). - Ensure you are using supported comparison operators for the data type of each metadata field. - If possible, inspect the actual metadata stored in your database for the relevant documents to confirm the fields and values. - Try a very broad filter (for example, a single condition with "not empty") to see if any documents are returned, then narrow down to your intended filter. The code does not provide explicit error messages if a filter is invalid or does not match any documents—it simply returns an empty result. If you continue to get empty results even with a broad filter, there may be a mismatch between your filter configuration and the actual document metadata, or an issue with how metadata is being indexed or stored [[source]](https://github.com/langgenius/dify/blob/275e86a26c6b86409c391432a010fe0b1096b88e/api/core/workflow/nodes/knowledge_retrieval/knowledge_retrieval_node.py). If you can share an example of your metadata filtering condition and a sample of your document metadata, I can help you further pinpoint the issue. <!-- Dosu Comment Footer --> *To reply, just mention my name [@dosu](https://go.dosu.dev/dosubot).* --- <sup>How did I do? [Good](https://app.dosu.dev/response-feedback/3802e51e-630d-43ed-81b7-d20d5136c2b3?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/3802e51e-630d-43ed-81b7-d20d5136c2b3?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/3802e51e-630d-43ed-81b7-d20d5136c2b3?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/3802e51e-630d-43ed-81b7-d20d5136c2b3?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/3802e51e-630d-43ed-81b7-d20d5136c2b3?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/3802e51e-630d-43ed-81b7-d20d5136c2b3?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/3802e51e-630d-43ed-81b7-d20d5136c2b3?feedback_type=other)</sup>&nbsp;&nbsp;[![Join Discord](https://img.shields.io/badge/join-5865F2?logo=discord&logoColor=white&label=)](https://go.dosu.dev/discord-bot)&nbsp;[![Share on X](https://img.shields.io/badge/X-share-black)](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/langgenius/dify/issues/20621)
yindo commented 2026-02-21 19:17:05 -05:00
Author
Owner
Copy Link
Copy Source

@winqihe commented on GitHub (Jun 3, 2025):

sure, I used broad filter , condition "not empty" ,but not match any documents ,when I disabled this feature (metadata filtering), I can match some document ,so how to solve this question?

@winqihe commented on GitHub (Jun 3, 2025): sure, I used broad filter , condition "not empty" ,but not match any documents ,when I disabled this feature (metadata filtering), I can match some document ,so how to solve this question?
yindo commented 2026-02-21 19:17:05 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Jun 3, 2025):

The "not empty" metadata filter in Dify only matches documents where the specified metadata key both exists and is not null in the database. It does not consider empty strings, empty lists, or empty objects as "empty"—only missing keys or explicit null values count as empty. If your documents do not have the relevant metadata key set at all during ingestion, or if the key is present but set to an empty string or similar, the "not empty" filter will not match them, resulting in an empty result set even with a broad filter. This is a common source of confusion and can make the filter appear broken when it is actually working as designed [source].

To resolve this, check your document ingestion process to ensure that the metadata key you want to filter on is always set and not left unset or null. If you have existing datasets where this might not be the case, recreating the datasets has been shown to resolve similar filtering problems [source].

If you need the filter to treat empty strings, empty lists, or empty objects as "empty", you would need to adjust the ingestion logic to store those as null, or modify the filtering logic to explicitly check for those cases.

In summary: the "not empty" filter only matches documents where the metadata key exists and is not null. Make sure your documents meet this requirement, or recreate your datasets to ensure metadata consistency.

To reply, just mention my name @dosu.

How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other  Join Discord Share on X

@dosubot[bot] commented on GitHub (Jun 3, 2025): <!-- Answer --> The "not empty" metadata filter in Dify only matches documents where the specified metadata key both exists and is not null in the database. It does not consider empty strings, empty lists, or empty objects as "empty"—only missing keys or explicit null values count as empty. If your documents do not have the relevant metadata key set at all during ingestion, or if the key is present but set to an empty string or similar, the "not empty" filter will not match them, resulting in an empty result set even with a broad filter. This is a common source of confusion and can make the filter appear broken when it is actually working as designed [[source]](https://github.com/langgenius/dify/blob/275e86a26c6b86409c391432a010fe0b1096b88e/api/core/rag/retrieval/dataset_retrieval.py). To resolve this, check your document ingestion process to ensure that the metadata key you want to filter on is always set and not left unset or null. If you have existing datasets where this might not be the case, recreating the datasets has been shown to resolve similar filtering problems [[source]](https://github.com/langgenius/dify/issues/16177). If you need the filter to treat empty strings, empty lists, or empty objects as "empty", you would need to adjust the ingestion logic to store those as null, or modify the filtering logic to explicitly check for those cases. In summary: the "not empty" filter only matches documents where the metadata key exists and is not null. Make sure your documents meet this requirement, or recreate your datasets to ensure metadata consistency. <!-- Dosu Comment Footer --> *To reply, just mention my name [@dosu](https://go.dosu.dev/dosubot).* --- <sup>How did I do? [Good](https://app.dosu.dev/response-feedback/7e5cf16a-0a3a-4198-81ce-3bfc02ecc12d?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/7e5cf16a-0a3a-4198-81ce-3bfc02ecc12d?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/7e5cf16a-0a3a-4198-81ce-3bfc02ecc12d?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/7e5cf16a-0a3a-4198-81ce-3bfc02ecc12d?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/7e5cf16a-0a3a-4198-81ce-3bfc02ecc12d?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/7e5cf16a-0a3a-4198-81ce-3bfc02ecc12d?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/7e5cf16a-0a3a-4198-81ce-3bfc02ecc12d?feedback_type=other)</sup>&nbsp;&nbsp;[![Join Discord](https://img.shields.io/badge/join-5865F2?logo=discord&logoColor=white&label=)](https://go.dosu.dev/discord-bot)&nbsp;[![Share on X](https://img.shields.io/badge/X-share-black)](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/langgenius/dify/issues/20621)
yindo commented 2026-02-21 19:17:05 -05:00
Author
Owner
Copy Link
Copy Source

@winqihe commented on GitHub (Jun 3, 2025):

I double checked my filter condition , but this filter still not work, it`s also not match any document

@winqihe commented on GitHub (Jun 3, 2025): I double checked my filter condition , but this filter still not work, it`s also not match any document
yindo commented 2026-02-21 19:17:05 -05:00
Author
Owner
Copy Link
Copy Source

@winqihe commented on GitHub (Jun 4, 2025):

I do three unit test ，but all it`s not match any document

@winqihe commented on GitHub (Jun 4, 2025): I do three unit test ，but all it`s not match any document
yindo commented 2026-02-21 19:17:05 -05:00
Author
Owner
Copy Link
Copy Source

@berg-wang commented on GitHub (Jun 19, 2025):

I met this problem too! My dify version is 1.1.3

@berg-wang commented on GitHub (Jun 19, 2025): I met this problem too! My dify version is 1.1.3
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#14404
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?