langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-20 09:10:05 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Does Dify have plans to support rerank and embedding models launched by vLLM? #7301

New Issue
Closed
opened 2026-02-21 18:19:54 -05:00 by yindo · 5 comments
yindo commented 2026-02-21 18:19:54 -05:00
Owner
Copy Link
Copy Source

Originally created by @massif-01 on GitHub (Dec 19, 2024).

Self Checks

  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
  • Please do not modify this template :) and fill in all the required fields.

1. Is this request related to a challenge you're experiencing? Tell me about your story.

Does Dify have plans to support rerank and embedding models launched by vLLM?
When deploying large models and Dify locally on performance-limited devices (such as Orin X or a Mac mini with 64GB of RAM), using Xinference to launch the models can consume precious memory. Using VLLM to launch various models together is a better choice.

2. Additional context or comments

No response

3. Can you help us with this feature?

  • I am interested in contributing to this feature.
Originally created by @massif-01 on GitHub (Dec 19, 2024). ### Self Checks - [X] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [X] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [X] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:） - [X] Please do not modify this template :) and fill in all the required fields. ### 1. Is this request related to a challenge you're experiencing? Tell me about your story. Does Dify have plans to support rerank and embedding models launched by vLLM? When deploying large models and Dify locally on performance-limited devices (such as Orin X or a Mac mini with 64GB of RAM), using Xinference to launch the models can consume precious memory. Using VLLM to launch various models together is a better choice. ### 2. Additional context or comments _No response_ ### 3. Can you help us with this feature? - [ ] I am interested in contributing to this feature.
yindo added the 🙋‍♂️ question label 2026-02-21 18:19:54 -05:00
yindo closed this issue 2026-02-21 18:19:54 -05:00
yindo commented 2026-02-21 18:19:54 -05:00
Author
Owner
Copy Link
Copy Source

@thomas-hiddenpeak commented on GitHub (Dec 19, 2024):

+1

@thomas-hiddenpeak commented on GitHub (Dec 19, 2024): +1
yindo commented 2026-02-21 18:19:54 -05:00
Author
Owner
Copy Link
Copy Source

@crazywoola commented on GitHub (Dec 19, 2024):

Because of this https://github.com/langgenius/dify/issues/11588, we no longer accept the prs related to model runtimes.
We are going to launch v1.0 in the near future. After that, I think the community could help with that.

@crazywoola commented on GitHub (Dec 19, 2024): Because of this https://github.com/langgenius/dify/issues/11588, we no longer accept the prs related to model runtimes. We are going to launch v1.0 in the near future. After that, I think the community could help with that.
yindo commented 2026-02-21 18:19:54 -05:00
Author
Owner
Copy Link
Copy Source

@thomas-hiddenpeak commented on GitHub (Dec 20, 2024):

vLLM rerank task api :

  • Route: /score, Methods: POST
  • Route: /v1/score, Methods: POST
    I try to modify the route /v1/score to /v1/rerank
    but dify show me the notice paramate error

waiting for your v1.0 😭

@thomas-hiddenpeak commented on GitHub (Dec 20, 2024): vLLM rerank task api : - Route: /score, Methods: POST - Route: /v1/score, Methods: POST I try to modify the route /v1/score to /v1/rerank but dify show me the notice paramate error waiting for your v1.0 😭
yindo commented 2026-02-21 18:19:54 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Jan 22, 2025):

Hi, @massif-01. I'm Dosu, and I'm helping the Dify team manage their backlog. I'm marking this issue as stale.

Issue Summary

  • You inquired about Dify's plans to support rerank and embedding models from vLLM for better memory efficiency.
  • HiddenPeak supported your inquiry and provided details about vLLM's rerank task API, noting a parameter error when modifying the route.
  • Crazywoola mentioned that due to a related issue, pull requests for model runtimes are not being accepted, but community help might be possible post Dify v1.0 launch.
  • HiddenPeak expressed anticipation for the Dify v1.0 release.

Next Steps

  • Please let us know if this issue is still relevant to the latest version of the Dify repository by commenting here.
  • If there is no further activity, this issue will be automatically closed in 15 days.

Thank you for your understanding and contribution!

@dosubot[bot] commented on GitHub (Jan 22, 2025): Hi, @massif-01. I'm [Dosu](https://dosu.dev), and I'm helping the Dify team manage their backlog. I'm marking this issue as stale. **Issue Summary** - You inquired about Dify's plans to support rerank and embedding models from vLLM for better memory efficiency. - HiddenPeak supported your inquiry and provided details about vLLM's rerank task API, noting a parameter error when modifying the route. - Crazywoola mentioned that due to a related issue, pull requests for model runtimes are not being accepted, but community help might be possible post Dify v1.0 launch. - HiddenPeak expressed anticipation for the Dify v1.0 release. **Next Steps** - Please let us know if this issue is still relevant to the latest version of the Dify repository by commenting here. - If there is no further activity, this issue will be automatically closed in 15 days. Thank you for your understanding and contribution!
yindo commented 2026-02-21 18:19:55 -05:00
Author
Owner
Copy Link
Copy Source

@hundredwz commented on GitHub (Feb 16, 2025):

vLLM rerank task api :

  • Route: /score, Methods: POST
  • Route: /v1/score, Methods: POST
    I try to modify the route /v1/score to /v1/rerank
    but dify show me the notice paramate error

waiting for your v1.0 😭

vLLM 0.7.2 now support /v1/rerank.

However, we will face 404 error when trying to add rerank model to dify, since there is a bug in rerank.py

we could modify the code in L67 to following

 data = {"model": model_name, "query": query, "documents": docs, "return_documents": True}
  if top_n is not None:
      data["top_n"] = top_n

If premitted, I could submit pr to fix it.

@hundredwz commented on GitHub (Feb 16, 2025): > vLLM rerank task api : > > * Route: /score, Methods: POST > * Route: /v1/score, Methods: POST > I try to modify the route /v1/score to /v1/rerank > but dify show me the notice paramate error > > waiting for your v1.0 😭 vLLM 0.7.2 now support [/v1/rerank](https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html#rerank-api). However, we will face 404 error when trying to add rerank model to dify, since there is a bug in [rerank.py](https://github.com/langgenius/dify/blob/main/api/core/model_runtime/model_providers/openai_api_compatible/rerank/rerank.py#L67) we could modify the code in L67 to following ``` data = {"model": model_name, "query": query, "documents": docs, "return_documents": True} if top_n is not None: data["top_n"] = top_n ``` If premitted, I could submit pr to fix it.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🙋‍♂️ question
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#7301
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?