langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-23 21:15:32 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Chatbot works fine with one dataset, but result in nothing with multiple datasets, and no datasets retrieved #15540

New Issue
Closed
opened 2026-02-21 19:22:12 -05:00 by yindo · 7 comments
yindo commented 2026-02-21 19:22:12 -05:00
Owner
Copy Link
Copy Source

Originally created by @R1310328554 on GitHub (Jul 21, 2025).

Self Checks

  • I have read the Contributing Guide and Language Policy.
  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report, otherwise it will be closed.
  • 【中文用户 & Non English User】请使用英语提交，否则会被关闭 ：）
  • Please do not modify this template :) and fill in all the required fields.

Dify version

1.5

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

#  Create a dataset with some documents,  do the 'Retrieval Test',  the result is good . The dataset works fine.
#  Create a Chatbot with the dataset,  test it using the 'Talk to Bot' inputbox. the result is nice. The Chatbot works fine.

My Chatbot's prompt is as following:

`You are a seasoned technical expert and business specialist with extensive experience. You possess comprehensive knowledge of all aspects of the company, including products, customers, management, technology, and sales.

Based on user inquiries, you will sequentially search all knowledge bases. If matching content segments are found, you will:

Sort them by relevance

Quote the most appropriate segment(s)

Formulate a precise, using strictly the found documentation

Critical rules:

NEVER fabricate information - if no matching content exists, respond: "I couldn't find matching documentation and am unable to answer this question."

Follow this workflow:
a. Thoroughly analyze the question's core requirements
b. Search all knowledge bases
c. Sort found segments by relevance
d. Compose a structured response using ONLY verified content
e. Annotate each knowledge point with its source document ID

Maintain responses that are:

Concise and accurate

Strictly fact-based

Free from XML tags

Devoid of speculative content

Output format requirements:
precedes each verified information unit. When no references exist, deliver only the standardized "unable to answer" response.`

#  Create another dataset with some other documents,  do the 'Retrieval Test'.

# Add the second dataset  to the Chatbot, test it using the same questions as in the previous test steps, find that no datasets were retrieved, and the result is very bad.

✔️ Expected Behavior

When I Add a new dataset to the Chatbot , asking the same questions as in the previous test steps should return the same or better result.

❌ Actual Behavior

When I Add a new dataset to the Chatbot , asking the same questions as in the previous test steps , it returned very poor answers or fixed response as specified the the prompt:

"I couldn't find matching documentation and am unable to answer this question."

So, it seems that the Chatbot can't work with multiple datasets , or it will retrieve no datasets ， result in nothing.

Originally created by @R1310328554 on GitHub (Jul 21, 2025). ### Self Checks - [x] I have read the [Contributing Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) and [Language Policy](https://github.com/langgenius/dify/issues/1542). - [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report, otherwise it will be closed. - [x] 【中文用户 & Non English User】请使用英语提交，否则会被关闭 ：） - [x] Please do not modify this template :) and fill in all the required fields. ### Dify version 1.5 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce ``` # Create a dataset with some documents, do the 'Retrieval Test', the result is good . The dataset works fine. # Create a Chatbot with the dataset, test it using the 'Talk to Bot' inputbox. the result is nice. The Chatbot works fine. My Chatbot's prompt is as following: `You are a seasoned technical expert and business specialist with extensive experience. You possess comprehensive knowledge of all aspects of the company, including products, customers, management, technology, and sales. Based on user inquiries, you will sequentially search all knowledge bases. If matching content segments are found, you will: Sort them by relevance Quote the most appropriate segment(s) Formulate a precise, using strictly the found documentation Critical rules: NEVER fabricate information - if no matching content exists, respond: "I couldn't find matching documentation and am unable to answer this question." Follow this workflow: a. Thoroughly analyze the question's core requirements b. Search all knowledge bases c. Sort found segments by relevance d. Compose a structured response using ONLY verified content e. Annotate each knowledge point with its source document ID Maintain responses that are: Concise and accurate Strictly fact-based Free from XML tags Devoid of speculative content Output format requirements: precedes each verified information unit. When no references exist, deliver only the standardized "unable to answer" response.` # Create another dataset with some other documents, do the 'Retrieval Test'. # Add the second dataset to the Chatbot, test it using the same questions as in the previous test steps, find that no datasets were retrieved, and the result is very bad. ``` ### ✔️ Expected Behavior When I Add a new dataset to the Chatbot , asking the same questions as in the previous test steps should return the same or better result. ### ❌ Actual Behavior When I Add a new dataset to the Chatbot , asking the same questions as in the previous test steps , it returned very poor answers or fixed response as specified the the prompt: "I couldn't find matching documentation and am unable to answer this question." So, it seems that the Chatbot can't work with multiple datasets , or it will retrieve no datasets ， result in nothing.
yindo added the 👻 feat:rag label 2026-02-21 19:22:12 -05:00
yindo closed this issue 2026-02-21 19:22:12 -05:00
yindo commented 2026-02-21 19:22:12 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Jul 21, 2025):

This issue is likely caused by strict filtering and compatibility checks in Dify v1.5's multi-dataset retrieval logic. When multiple datasets are attached, retrieval can fail if the datasets have different indexing techniques or embedding models and a reranking model is not set, or if metadata filters exclude all documents. Each dataset may work alone, but together, these checks can result in no results being returned.

To maximize the chance of successful multi-dataset retrieval in v1.5, ensure all datasets use the same indexing technique and embedding model, set a reranking model if required, and review metadata filters to make sure they aren't excluding all documents. Also, verify that each dataset has available, enabled, and non-archived documents.

Recent versions have introduced features like an "Aggregate All" switch and validation fixes that improve multi-dataset support, so if configuration alone doesn't resolve the issue, consider upgrading to a newer version where these improvements are available (source).

To reply, just mention @dosu.

How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other  Join Discord Share on X

@dosubot[bot] commented on GitHub (Jul 21, 2025): <!-- Answer --> This issue is likely caused by strict filtering and compatibility checks in Dify v1.5's multi-dataset retrieval logic. When multiple datasets are attached, retrieval can fail if the datasets have different indexing techniques or embedding models and a reranking model is not set, or if metadata filters exclude all documents. Each dataset may work alone, but together, these checks can result in no results being returned. To maximize the chance of successful multi-dataset retrieval in v1.5, ensure all datasets use the same indexing technique and embedding model, set a reranking model if required, and review metadata filters to make sure they aren't excluding all documents. Also, verify that each dataset has available, enabled, and non-archived documents. Recent versions have introduced features like an "Aggregate All" switch and validation fixes that improve multi-dataset support, so if configuration alone doesn't resolve the issue, consider upgrading to a newer version where these improvements are available ([source](https://github.com/langgenius/dify/pull/22200)). <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- <sup>How did I do? [Good](https://app.dosu.dev/response-feedback/6dc7eef1-851c-4e45-b563-04f371187389?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/6dc7eef1-851c-4e45-b563-04f371187389?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/6dc7eef1-851c-4e45-b563-04f371187389?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/6dc7eef1-851c-4e45-b563-04f371187389?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/6dc7eef1-851c-4e45-b563-04f371187389?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/6dc7eef1-851c-4e45-b563-04f371187389?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/6dc7eef1-851c-4e45-b563-04f371187389?feedback_type=other)</sup>&nbsp;&nbsp;[![Join Discord](https://img.shields.io/badge/join-5865F2?logo=discord&logoColor=white&label=)](https://go.dosu.dev/discord-bot)&nbsp;[![Share on X](https://img.shields.io/badge/X-share-black)](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/langgenius/dify/issues/22733)
yindo commented 2026-02-21 19:22:12 -05:00
Author
Owner
Copy Link
Copy Source

@R1310328554 commented on GitHub (Jul 21, 2025):

How did I know that when Chatbot with multiple datasets then no datasets was retrieved? cuz the dataset's 'Retrieval Test' page have 'Records', when added multiple datasets in a Chatbot , I found that neither datasets will record the questions， So I conclude that no datasets was retrieved when test Chatbot with multiple datasets.

@R1310328554 commented on GitHub (Jul 21, 2025): How did I know that when Chatbot with multiple datasets then no datasets was retrieved? cuz the dataset's 'Retrieval Test' page have 'Records', when added multiple datasets in a Chatbot , I found that neither datasets will record the questions， So I conclude that no datasets was retrieved when test Chatbot with multiple datasets.
yindo commented 2026-02-21 19:22:12 -05:00
Author
Owner
Copy Link
Copy Source

@crazywoola commented on GitHub (Jul 21, 2025):

When I Add a new dataset to the Chatbot , asking the same questions in the previous test steps should return the same or better result.

This is not predictable by default, it depends on the LLM itself.
Prompt can not guarantee 100% accuracy, if you want to achieve better results with multiple datasets.

Here are few suggestions you might need to know:

  • Refine the description of the knowledge
  • Use a workflow or chatflow with if/else node to deal with the no answer situation.
@crazywoola commented on GitHub (Jul 21, 2025): > When I Add a new dataset to the Chatbot , asking the same questions in the previous test steps should return the same or better result. This is not predictable by default, it depends on the LLM itself. Prompt can not guarantee 100% accuracy, if you want to achieve better results with multiple datasets. Here are few suggestions you might need to know: - Refine the description of the knowledge - Use a workflow or chatflow with if/else node to deal with the no answer situation.
yindo commented 2026-02-21 19:22:12 -05:00
Author
Owner
Copy Link
Copy Source

@R1310328554 commented on GitHub (Jul 21, 2025):

I just cannt understand that Chatbot with one dataset works fine, but Chatbot with multiple datasets then asking the same questions as I did in the previous test step resulting very poor answers @crazywoola

@R1310328554 commented on GitHub (Jul 21, 2025): I just cannt understand that Chatbot with one dataset works fine, but Chatbot with multiple datasets then asking the same questions as I did in the previous test step resulting very poor answers @crazywoola
yindo commented 2026-02-21 19:22:12 -05:00
Author
Owner
Copy Link
Copy Source

@R1310328554 commented on GitHub (Jul 21, 2025):

Could it be a bug or not? I remember that the v1.3 seems have no such problems.

@R1310328554 commented on GitHub (Jul 21, 2025): Could it be a bug or not? I remember that the v1.3 seems have no such problems.
yindo commented 2026-02-21 19:22:12 -05:00
Author
Owner
Copy Link
Copy Source

@R1310328554 commented on GitHub (Jul 21, 2025):

@dosubot

@R1310328554 commented on GitHub (Jul 21, 2025): @dosubot
yindo commented 2026-02-21 19:22:12 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Jul 21, 2025):

This is expected in Dify v1.5 due to stricter multi-dataset retrieval logic. When you attach multiple datasets, Dify requires all datasets to use the same indexing technique and embedding model—unless you configure a reranking model. If these requirements aren't met, retrieval silently fails and you get no results, even if each dataset works fine alone. This is a regression from v1.3, which was more permissive.

To fix this, make sure all datasets use the same indexing and embedding settings, or enable a reranking model. Also, check that metadata filters aren't excluding all documents. Recent versions have improved multi-dataset support and added features like "Aggregate All" to help with this scenario, so upgrading may help if configuration alone doesn't resolve it. For best results, write clear, distinct knowledge base descriptions—this helps the AI select the right dataset when multiple are present. More details are in the retrieval logic code and docs.

To reply, just mention @dosu.

How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other  Join Discord Share on X

@dosubot[bot] commented on GitHub (Jul 21, 2025): <!-- Answer --> This is expected in Dify v1.5 due to stricter multi-dataset retrieval logic. When you attach multiple datasets, Dify requires all datasets to use the same indexing technique and embedding model—unless you configure a reranking model. If these requirements aren't met, retrieval silently fails and you get no results, even if each dataset works fine alone. This is a regression from v1.3, which was more permissive. To fix this, make sure all datasets use the same indexing and embedding settings, or enable a reranking model. Also, check that metadata filters aren't excluding all documents. Recent versions have improved multi-dataset support and added features like "Aggregate All" to help with this scenario, so upgrading may help if configuration alone doesn't resolve it. For best results, write clear, distinct knowledge base descriptions—this helps the AI select the right dataset when multiple are present. More details are in the [retrieval logic code](https://github.com/langgenius/dify/blob/eb06de0921938dfa7743f12b443246f8c657b6d5/api/core/rag/retrieval/dataset_retrieval.py) and [docs](https://github.com/crazywoola/dify-docs-mintlify/blob/a6647b33df40ace51422b11f355ec6e2618bc05e/en/features/datasets/README.mdx). <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- <sup>How did I do? [Good](https://app.dosu.dev/response-feedback/f13dfb71-5d65-4575-b7d4-034f0d1da637?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/f13dfb71-5d65-4575-b7d4-034f0d1da637?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/f13dfb71-5d65-4575-b7d4-034f0d1da637?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/f13dfb71-5d65-4575-b7d4-034f0d1da637?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/f13dfb71-5d65-4575-b7d4-034f0d1da637?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/f13dfb71-5d65-4575-b7d4-034f0d1da637?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/f13dfb71-5d65-4575-b7d4-034f0d1da637?feedback_type=other)</sup>&nbsp;&nbsp;[![Join Discord](https://img.shields.io/badge/join-5865F2?logo=discord&logoColor=white&label=)](https://go.dosu.dev/discord-bot)&nbsp;[![Share on X](https://img.shields.io/badge/X-share-black)](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/langgenius/dify/issues/22733)
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 👻 feat:rag
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#15540
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?