langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 03:55:25 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

🐛 Bug Report: Retriever returns fewer chunks than configured Top K value #22253

New Issue
Open
opened 2026-02-21 20:16:19 -05:00 by yindo · 1 comment
yindo commented 2026-02-21 20:16:19 -05:00
Owner
Copy Link
Copy Source

Originally created by @pablomart on GitHub (Feb 19, 2026).

Self Checks

  • I have read the Contributing Guide and Language Policy.
  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report, otherwise it will be closed.
  • 【中文用户 & Non English User】请使用英语提交，否则会被关闭 ：）
  • Please do not modify this template :) and fill in all the required fields.

Dify version

1.13.0

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

Description

When the Top K parameter is set to 10 in the retrieval configuration, the knowledge base retriever only returns 5 chunks instead of the expected 10. This happens consistently regardless of the retrieval method used.

Steps to Reproduce

  1. Open a Chatbot or Workflow application that uses a Knowledge Base.
  2. Go to the Retrieval Configuration settings.
  3. Select Hybrid Search (or any retrieval method).
  4. Set Top K to 10.
  5. Send a query that should match documents in the knowledge base.
  6. Inspect the retrieved context — only 5 paragraphs/chunks are returned.

✔️ Expected Behavior

The retriever should return up to 10 chunks as configured by the Top K parameter.

❌ Actual Behavior

Only 5 chunks are returned despite Top K being set to 10. The UI even shows the label "5 Párrafos recuperados" (5 paragraphs retrieved).

Image

Screenshots

Screenshot shows Top K = 10 configured with Hybrid Search (Weighted Score, Semantic 0.7 / Keyword 0.3), but only 5 chunks are returned in the results panel.

Configuration Details

  • Retrieval method: Hybrid Search (Búsqueda Híbrida) — Recommended
  • Reranking strategy: Weighted Score (Puntuación ponderada)
  • Semantic weight: 0.7
  • Keyword weight: 0.3
  • Top K: 10
  • Score Threshold: 0 (disabled)
  • Knowledge base documents: Multiple PDFs

Additional Context

It is unclear whether the limitation is:

  • A hard cap somewhere in the retrieval pipeline overriding the Top K value.
  • A deduplication or post-processing step that silently drops chunks.
  • A UI-only display issue where more chunks are retrieved but not shown.
  • A bug specific to Hybrid Search with Weighted Score reranking.

This directly impacts RAG quality, as users expecting broader context coverage are silently receiving fewer results than configured.

Originally created by @pablomart on GitHub (Feb 19, 2026). ### Self Checks - [x] I have read the [Contributing Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) and [Language Policy](https://github.com/langgenius/dify/issues/1542). - [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report, otherwise it will be closed. - [x] 【中文用户 & Non English User】请使用英语提交，否则会被关闭 ：） - [x] Please do not modify this template :) and fill in all the required fields. ### Dify version 1.13.0 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce ### Description When the **Top K** parameter is set to `10` in the retrieval configuration, the knowledge base retriever only returns **5 chunks** instead of the expected 10. This happens consistently regardless of the retrieval method used. ### Steps to Reproduce 1. Open a Chatbot or Workflow application that uses a Knowledge Base. 2. Go to the **Retrieval Configuration** settings. 3. Select **Hybrid Search** (or any retrieval method). 4. Set **Top K** to `10`. 5. Send a query that should match documents in the knowledge base. 6. Inspect the retrieved context — only **5 paragraphs/chunks** are returned. ### ✔️ Expected Behavior The retriever should return up to **10 chunks** as configured by the Top K parameter. ### ❌ Actual Behavior Only **5 chunks** are returned despite Top K being set to `10`. The UI even shows the label **"5 Párrafos recuperados" (5 paragraphs retrieved)**. <img width="1206" height="642" alt="Image" src="https://github.com/user-attachments/assets/6dad0e15-d762-4909-9299-3d23c8038e77" /> ### Screenshots > *Screenshot shows Top K = 10 configured with Hybrid Search (Weighted Score, Semantic 0.7 / Keyword 0.3), but only 5 chunks are returned in the results panel.* ### Configuration Details - **Retrieval method:** Hybrid Search (Búsqueda Híbrida) — Recommended - **Reranking strategy:** Weighted Score (Puntuación ponderada) - **Semantic weight:** 0.7 - **Keyword weight:** 0.3 - **Top K:** 10 - **Score Threshold:** 0 (disabled) - **Knowledge base documents:** Multiple PDFs ### Additional Context It is unclear whether the limitation is: - A **hard cap** somewhere in the retrieval pipeline overriding the Top K value. - A **deduplication or post-processing step** that silently drops chunks. - A **UI-only display issue** where more chunks are retrieved but not shown. - A bug specific to **Hybrid Search** with Weighted Score reranking. This directly impacts RAG quality, as users expecting broader context coverage are silently receiving fewer results than configured.
yindo added the 🐞 bug label 2026-02-21 20:16:19 -05:00
yindo commented 2026-02-21 20:16:19 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Feb 19, 2026):

This is a known issue that's been reported before. There are several factors causing fewer chunks to be returned than your configured Top K:

Most likely causes for your setup:

  1. Hybrid Search Deduplication - When both vector and full-text search return the same documents, deduplication removes duplicates, keeping only the highest-scoring version. This can reduce results by up to 50%.

  2. Weighted Score Reranking Bug - In v1.13.0, there was a bug where text search scores were incorrectly treated as vector search scores during weighted reranking, affecting your 0.7/0.3 weighted scoring.

  3. Hardcoded Top K Fallback - A bug caused Top K to fall back to 2 in certain code paths, overriding user settings.

Recommended solutions:

  1. Upgrade from v1.13.0 - Several critical fixes were merged after this version: PR #24879 (hardcoded Top K), PR #26360 (deduplication), and PR #28126 (weighted scoring).

  2. Quick test - Temporarily switch to Vector Search only (disable hybrid) to see if you get all 10 results. If so, the issue is with hybrid search deduplication/reranking.

  3. Full reindex workaround - Export and re-import your knowledge base to trigger a complete reindex, which can resolve inconsistent retrieval counts.

This isn't a UI display issue - the retrieval pipeline is genuinely returning fewer chunks due to deduplication and filtering stages applied before the final Top K limit.

To reply, just mention @dosu.

How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other

Chat with Dosu Open in Cursor Join Discord Share on X

@dosubot[bot] commented on GitHub (Feb 19, 2026): <!-- Answer --> This is a **known issue** that's been reported before. There are several factors causing fewer chunks to be returned than your configured Top K: **Most likely causes for your setup:** 1. **Hybrid Search Deduplication** - When both vector and full-text search return the same documents, [deduplication removes duplicates](https://github.com/langgenius/dify/blob/fbacb9f7a279611190f0476256d94dff9a38d991/api/core/rag/datasource/retrieval_service.py#L155-L196), keeping only the highest-scoring version. This can reduce results by up to 50%. 2. **Weighted Score Reranking Bug** - In v1.13.0, there was a bug where [text search scores were incorrectly treated as vector search scores](https://github.com/langgenius/dify/pull/28126) during weighted reranking, affecting your 0.7/0.3 weighted scoring. 3. **Hardcoded Top K Fallback** - A bug caused [Top K to fall back to 2](https://github.com/langgenius/dify/pull/24879) in certain code paths, overriding user settings. **Recommended solutions:** 1. **Upgrade from v1.13.0** - Several critical fixes were merged after this version: [PR #24879](https://github.com/langgenius/dify/pull/24879) (hardcoded Top K), [PR #26360](https://github.com/langgenius/dify/pull/26360) (deduplication), and [PR #28126](https://github.com/langgenius/dify/pull/28126) (weighted scoring). 2. **Quick test** - Temporarily switch to **Vector Search only** (disable hybrid) to see if you get all 10 results. If so, the issue is with hybrid search deduplication/reranking. 3. **Full reindex workaround** - [Export and re-import your knowledge base](https://github.com/langgenius/dify/issues/31505) to trigger a complete reindex, which can resolve inconsistent retrieval counts. This isn't a UI display issue - the retrieval pipeline is genuinely returning fewer chunks due to deduplication and filtering stages applied before the final Top K limit. <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- <sup>How did I do? [Good](https://app.dosu.dev/response-feedback/c4cc5ad1-a137-4691-ba6c-ef8a8e01bb34?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/c4cc5ad1-a137-4691-ba6c-ef8a8e01bb34?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/c4cc5ad1-a137-4691-ba6c-ef8a8e01bb34?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/c4cc5ad1-a137-4691-ba6c-ef8a8e01bb34?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/c4cc5ad1-a137-4691-ba6c-ef8a8e01bb34?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/c4cc5ad1-a137-4691-ba6c-ef8a8e01bb34?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/c4cc5ad1-a137-4691-ba6c-ef8a8e01bb34?feedback_type=other)</sup> [![Chat with Dosu](https://dosu.dev/dosu-chat-badge.svg)](https://app.dosu.dev/6088a4f4-073a-43f3-b3d4-4f06fb0c05a0/ask?utm_source=github)&nbsp;[![Open in Cursor](https://dosu.dev/dosu-in-cursor.svg)](https://cursor.com/link/prompt?text=This%20is%20a%20%2A%2Aknown%20issue%2A%2A%20that%27s%20been%20reported%20before.%20There%20are%20several%20factors%20causing%20fewer%20chunks%20to%20be%20returned%20than%20your%20configured%20Top%20K%3A%0A%0A%2A%2AMost%20likely%20causes%20for%20your%20setup%3A%2A%2A%0A%0A1.%20%2A%2AHybrid%20Search%20Deduplication%2A%2A%20-%20When%20both%20vector%20and%20full-text%20search%20return%20the%20same%20documents%2C%20%5Bdeduplication%20removes%20duplicates%5D%28https%3A//github.com/langgenius/dify/blob/fbacb9f7a279611190f0476256d94dff9a38d991/api/core/rag/datasource/retrieval_service.py%23L155-L196%29%2C%20keeping%20only%20the%20highest-scoring%20version.%20This%20can%20reduce%20results%20by%20up%20to%2050%25.%0A%0A2.%20%2A%2AWeighted%20Score%20Reranking%20Bug%2A%2A%20-%20In%20v1.13.0%2C%20there%20was%20a%20bug%20where%20%5Btext%20search%20scores%20were%20incorrectly%20treated%20as%20vector%20search%20scores%5D%28https%3A//github.com/langgenius/dify/pull/28126%29%20during%20weighted%20reranking%2C%20affecting%20your%200.7/0.3%20weighted%20scoring.%0A%0A3.%20%2A%2AHardcoded%20Top%20K%20Fallback%2A%2A%20-%20A%20bug%20caused%20%5BTop%20K%20to%20fall%20back%20to%202%5D%28https%3A//github.com/langgenius/dify/pull/24879%29%20in%20certain%20code%20paths%2C%20overriding%20user%20settings.%0A%0A%2A%2ARecommended%20solutions%3A%2A%2A%0A%0A1.%20%2A%2AUpgrade%20from%20v1.13.0%2A%2A%20-%20Several%20critical%20fixes%20were%20merged%20after%20this%20version%3A%20%5BPR%20%2324879%5D%28https%3A//github.com/langgenius/dify/pull/24879%29%20%28hardcoded%20Top%20K%29%2C%20%5BPR%20%2326360%5D%28https%3A//github.com/langgenius/dify/pull/26360%29%20%28deduplication%29%2C%20and%20%5BPR%20%2328126%5D%28https%3A//github.com/langgenius/dify/pull/28126%29%20%28weighted%20scoring%29.%0A%0A2.%20%2A%2AQuick%20test%2A%2A%20-%20Temporarily%20switch%20to%20%2A%2AVector%20Search%20only%2A%2A%20%28disable%20hybrid%29%20to%20see%20if%20you%20get%20all%2010%20results.%20If%20so%2C%20the%20issue%20is%20with%20hybrid%20search%20deduplication/reranking.%0A%0A3.%20%2A%2AFull%20reindex%20workaround%2A%2A%20-%20%5BExport%20and%20re-import%20your%20knowledge%20base%5D%28https%3A//github.com/langgenius/dify/issues/31505%29%20to%20trigger%20a%20complete%20reindex%2C%20which%20can%20resolve%20inconsistent%20retrieval%20counts.%0A%0AThis%20isn%27t%20a%20UI%20display%20issue%20-%20the%20retrieval%20pipeline%20is%20genuinely%20returning%20fewer%20chunks%20due%20to%20deduplication%20and%20filtering%20stages%20applied%20before%20the%20final%20Top%20K%20limit.)&nbsp;[![Join Discord](https://img.shields.io/badge/join-5865F2?logo=discord&logoColor=white&label=)](https://go.dosu.dev/discord-bot)&nbsp;[![Share on X](https://img.shields.io/badge/X-share-black)](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/langgenius/dify/issues/32421)
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#22253
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?