langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 20:15:31 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

The Rerank model in RAG needs to support independent score_threshold and top_k #6927

New Issue
Closed
opened 2026-02-21 18:17:59 -05:00 by yindo · 9 comments
yindo commented 2026-02-21 18:17:59 -05:00
Owner
Copy Link
Copy Source

Originally created by @hustyichi on GitHub (Nov 25, 2024).

Originally assigned to: @laipz8200 on GitHub.

Self Checks

  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
  • Please do not modify this template :) and fill in all the required fields.

1. Is this request related to a challenge you're experiencing? Tell me about your story.

Currently, the retrieval model and the re-ranking stage share the score_threshold and top_k parameters, which currently have the following problems:

  1. The retrieval model and the re-ranking model have different score distributions and need to use different thresholds, which cannot be supported under the current solution;
  2. In order to ensure the integrity of the information, as much text as possible is generally recalled during the retrieval stage, and compressed to a relatively small number after the re-ranking is completed to match the input context of the large model. Currently, when top_k is shared, if top_k is set too small, the retrieval stage cannot recall enough content, affecting the recall rate, or if top_k is set too large, it exceeds the context of the large model;

2. Additional context or comments

No response

3. Can you help us with this feature?

  • I am interested in contributing to this feature.
Originally created by @hustyichi on GitHub (Nov 25, 2024). Originally assigned to: @laipz8200 on GitHub. ### Self Checks - [X] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [X] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [X] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:） - [X] Please do not modify this template :) and fill in all the required fields. ### 1. Is this request related to a challenge you're experiencing? Tell me about your story. Currently, the retrieval model and the re-ranking stage share the score_threshold and top_k parameters, which currently have the following problems: 1. The retrieval model and the re-ranking model have different score distributions and need to use different thresholds, which cannot be supported under the current solution; 2. In order to ensure the integrity of the information, as much text as possible is generally recalled during the retrieval stage, and compressed to a relatively small number after the re-ranking is completed to match the input context of the large model. Currently, when top_k is shared, if top_k is set too small, the retrieval stage cannot recall enough content, affecting the recall rate, or if top_k is set too large, it exceeds the context of the large model; ### 2. Additional context or comments _No response_ ### 3. Can you help us with this feature? - [ ] I am interested in contributing to this feature.
yindo added the 💪 enhancement👻 feat:rag labels 2026-02-21 18:17:59 -05:00
yindo closed this issue 2026-02-21 18:17:59 -05:00
yindo commented 2026-02-21 18:17:59 -05:00
Author
Owner
Copy Link
Copy Source

@nadirvishun commented on GitHub (Nov 25, 2024):

A lot of people have been giving feedback for a long time, like: #7187, and the officials are aware of it, but are slow to change it.

@nadirvishun commented on GitHub (Nov 25, 2024): A lot of people have been giving feedback for a long time, like: [#7187](https://github.com/langgenius/dify/issues/7187), and the officials are aware of it, but are slow to change it.
yindo commented 2026-02-21 18:17:59 -05:00
Author
Owner
Copy Link
Copy Source

@laipz8200 commented on GitHub (Dec 3, 2024):

I’ll ping our team about this feature request.

@laipz8200 commented on GitHub (Dec 3, 2024): I’ll ping our team about this feature request.
yindo commented 2026-02-21 18:17:59 -05:00
Author
Owner
Copy Link
Copy Source

@laipz8200 commented on GitHub (Dec 6, 2024):

Hi! The top_k and score_threshold settings in Workflow, Chatbot, or Agent do not override the configurations in the knowledge base. For your mentioned scenario, you can configure a higher value in the knowledge base, allowing Dify to retrieve more data during the search phase. Then, set lower values in Workflow, Chatbot, or Agent. This way, Dify will rerank the results and discard unnecessary ones after retrieval.

@laipz8200 commented on GitHub (Dec 6, 2024): Hi! The `top_k` and `score_threshold` settings in Workflow, Chatbot, or Agent do not override the configurations in the knowledge base. For your mentioned scenario, you can configure a higher value in the knowledge base, allowing Dify to retrieve more data during the search phase. Then, set lower values in Workflow, Chatbot, or Agent. This way, Dify will rerank the results and discard unnecessary ones after retrieval.
yindo commented 2026-02-21 18:17:59 -05:00
Author
Owner
Copy Link
Copy Source

@nadirvishun commented on GitHub (Dec 7, 2024):

I reviewed the process again, and there are two settings here:

  • Settings in the knowledge base, where its parameter TopK participates in both the first-stage vector retrieval and the rerank retrieval of the knowledge base itself.
  • Retrieval settings in chatFlow's knowledge base, where its parameter TopK is only used for reranking the documents aggregated from the multiple knowledge base retrievals again. Its value determines the maximum number of fragments passed to the large model.

image

Suppose I have 3 knowledge bases:

  • According to the current process, the knowledge base TopK is 50, rerank also return 50 records(useless effort, because everything will be reranked again afterwards), while the retrieval setting in chatFlow is 8. If aggregated, it would be 50+50+50=150, and then rerank out 8 records from 150. The data involved in the final rerank would be a lot, which should affect the speed.
  • If we distinguish search TopK as 50 and rerank TopN as 8 in the knowledge base, and the retrieval setting in chatFlow is 8, if aggregated, it would be 8+8+8=24, and then rerank out 8 records from 24. This would obviously be better.

I don't know if what I wrote above is accurate.

@nadirvishun commented on GitHub (Dec 7, 2024): I reviewed the process again, and there are two settings here: - Settings in the knowledge base, where its parameter `TopK` participates in both the first-stage vector retrieval and the rerank retrieval of the knowledge base itself. - Retrieval settings in chatFlow's knowledge base, where its parameter `TopK` is only used for reranking the documents aggregated from the multiple knowledge base retrievals again. Its value determines the maximum number of fragments passed to the large model. ![image](https://github.com/user-attachments/assets/e6ce16e6-c152-4c2b-b6cd-2506cdb09e19) Suppose I have `3` knowledge bases: - According to the current process, the knowledge base `TopK` is `50`, rerank also return `50` records(useless effort, because everything will be reranked again afterwards), while the retrieval setting in chatFlow is `8`. If aggregated, it would be `50+50+50=150`, and then rerank out `8` records from `150`. The data involved in the final rerank would be a lot, which should affect the speed. - If we distinguish search `TopK` as `50` and rerank `TopN` as `8` in the knowledge base, and the retrieval setting in chatFlow is `8`, if aggregated, it would be `8+8+8=24`, and then rerank out `8` records from `24`. This would obviously be better. I don't know if what I wrote above is accurate.
yindo commented 2026-02-21 18:17:59 -05:00
Author
Owner
Copy Link
Copy Source

@laipz8200 commented on GitHub (Dec 8, 2024):

You are correct, but this would result in a knowledge base filled with too many configurations. So, we don't have any plans to change it right now.

@laipz8200 commented on GitHub (Dec 8, 2024): You are correct, but this would result in a knowledge base filled with too many configurations. So, we don't have any plans to change it right now.
yindo commented 2026-02-21 18:18:00 -05:00
Author
Owner
Copy Link
Copy Source

@nadirvishun commented on GitHub (Dec 8, 2024):

@laipz8200
At least you can remerge the pull request: #11305

@nadirvishun commented on GitHub (Dec 8, 2024): @laipz8200 At least you can remerge the pull request: [#11305](https://github.com/langgenius/dify/pull/11305)
yindo commented 2026-02-21 18:18:00 -05:00
Author
Owner
Copy Link
Copy Source

@laipz8200 commented on GitHub (Dec 8, 2024):

@laipz8200 At least you can remerge the pull request: #11305

We won't merge it because its design is not as we expect, but you can still keep it in your version if you want.

@laipz8200 commented on GitHub (Dec 8, 2024): > @laipz8200 At least you can remerge the pull request: [#11305](https://github.com/langgenius/dify/pull/11305) We won't merge it because its design is not as we expect, but you can still keep it in your version if you want.
yindo commented 2026-02-21 18:18:00 -05:00
Author
Owner
Copy Link
Copy Source

@k1eNdn commented on GitHub (Dec 19, 2024):

I reviewed the process again, and there are two settings here:

  • Settings in the knowledge base, where its parameter TopK participates in both the first-stage vector retrieval and the rerank retrieval of the knowledge base itself.
  • Retrieval settings in chatFlow's knowledge base, where its parameter TopK is only used for reranking the documents aggregated from the multiple knowledge base retrievals again. Its value determines the maximum number of fragments passed to the large model.

image

Suppose I have 3 knowledge bases:

  • According to the current process, the knowledge base TopK is 50, rerank also return 50 records(useless effort, because everything will be reranked again afterwards), while the retrieval setting in chatFlow is 8. If aggregated, it would be 50+50+50=150, and then rerank out 8 records from 150. The data involved in the final rerank would be a lot, which should affect the speed.
  • If we distinguish search TopK as 50 and rerank TopN as 8 in the knowledge base, and the retrieval setting in chatFlow is 8, if aggregated, it would be 8+8+8=24, and then rerank out 8 records from 24. This would obviously be better.

I don't know if what I wrote above is accurate.

I also consider what you mentioned. In the current ChatFlow process, we need to rerank twice: first in the knowledge base and then in the aggregated chunks from multiple datasets. The first rerank is not necessary.

@k1eNdn commented on GitHub (Dec 19, 2024): > I reviewed the process again, and there are two settings here: > > * Settings in the knowledge base, where its parameter `TopK` participates in both the first-stage vector retrieval and the rerank retrieval of the knowledge base itself. > * Retrieval settings in chatFlow's knowledge base, where its parameter `TopK` is only used for reranking the documents aggregated from the multiple knowledge base retrievals again. Its value determines the maximum number of fragments passed to the large model. > > ![image](https://private-user-images.githubusercontent.com/5761981/393496870-e6ce16e6-c152-4c2b-b6cd-2506cdb09e19.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MzQ2NjQ4ODgsIm5iZiI6MTczNDY2NDU4OCwicGF0aCI6Ii81NzYxOTgxLzM5MzQ5Njg3MC1lNmNlMTZlNi1jMTUyLTRjMmItYjZjZC0yNTA2Y2RiMDllMTkucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MTIyMCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDEyMjBUMDMxNjI4WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9NWNiN2JkZmEyZWQxNWVmOTQ4YWYxNTMwMjk3NmIzM2UxYzM3M2YyZTUyOGQ4NDE1YjAzZDYyMTIwYzUxYmZkMiZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QifQ.NecLc7lhPjx7EQeq0kz5KdhS6eCPH9D9Ef8k7-KyY2Q) > > Suppose I have `3` knowledge bases: > > * According to the current process, the knowledge base `TopK` is `50`, rerank also return `50` records(useless effort, because everything will be reranked again afterwards), while the retrieval setting in chatFlow is `8`. If aggregated, it would be `50+50+50=150`, and then rerank out `8` records from `150`. The data involved in the final rerank would be a lot, which should affect the speed. > * If we distinguish search `TopK` as `50` and rerank `TopN` as `8` in the knowledge base, and the retrieval setting in chatFlow is `8`, if aggregated, it would be `8+8+8=24`, and then rerank out `8` records from `24`. This would obviously be better. > > I don't know if what I wrote above is accurate. I also consider what you mentioned. In the current ChatFlow process, we need to rerank twice: first in the knowledge base and then in the aggregated chunks from multiple datasets. The first rerank is not necessary.
yindo commented 2026-02-21 18:18:00 -05:00
Author
Owner
Copy Link
Copy Source

@laipz8200 commented on GitHub (Dec 19, 2024):

@hustyichi Thank you for your response. We understand the issue but believe this approach helps prevent adding too many settings to the knowledge base feature, making it easier for users to use. I have forwarded this issue to our product team, and it may change in the future.

@laipz8200 commented on GitHub (Dec 19, 2024): @hustyichi Thank you for your response. We understand the issue but believe this approach helps prevent adding too many settings to the knowledge base feature, making it easier for users to use. I have forwarded this issue to our product team, and it may change in the future.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 👻 feat:rag 💪 enhancement
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#6927
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?