langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 12:05:30 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

How to implement custom LLM model and use it in Dify? #1078

New Issue
Closed
opened 2026-02-21 17:29:53 -05:00 by yindo · 2 comments
yindo commented 2026-02-21 17:29:53 -05:00
Owner
Copy Link
Copy Source

Originally created by @ntvuongg on GitHub (Feb 21, 2024).

Self Checks

  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • Pleas do not modify this template :) and fill in all the required fields.

Provide a description of the help you need

Hi everyone, I have successfully self-hosted Dify. I want to use my custom llm model instead of using existing model which access through API. Any help?

Originally created by @ntvuongg on GitHub (Feb 21, 2024). ### Self Checks - [X] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [X] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [X] Pleas do not modify this template :) and fill in all the required fields. ### Provide a description of the help you need Hi everyone, I have successfully self-hosted Dify. I want to use my custom llm model instead of using existing model which access through API. Any help?
yindo closed this issue 2026-02-21 17:29:53 -05:00
yindo commented 2026-02-21 17:29:53 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Feb 21, 2024):

🤖

Hey @ntvuongg, nice to see you diving into new adventures with Dify! How's everything going?

Based on the information you've provided and the context from the Dify repository, it seems like you want to use your custom LLM model in a self-hosted Dify instead of the existing model accessed through API.

To achieve this, you would need to modify the _invoke method in the HuggingfaceHubLargeLanguageModel class. This method currently uses the InferenceClient from the huggingface_hub package to make requests to the Hugging Face API. If you want to use your own model, you need to replace this client with your own implementation that loads and uses your model.

Here's a simplified example of how you might do this:

def _invoke(self, model: str, credentials: dict, prompt_messages: list[PromptMessage], model_parameters: dict,
            tools: Optional[list[PromptMessageTool]] = None, stop: Optional[list[str]] = None, stream: bool = True,
            user: Optional[str] = None) -> Union[LLMResult, Generator]:

    # Load your custom model here
    my_model = load_my_model(model)

    # Convert the prompt messages to a single string
    prompt = self._convert_messages_to_prompt(prompt_messages)

    # Use your model to generate a response
    response = my_model.generate(prompt, **model_parameters)

    # Handle the response
    if stream:
        return self._handle_generate_stream_response(model, credentials, prompt_messages, response)

    return self._handle_generate_response(model, credentials, prompt_messages, response)

In this example, load_my_model is a function that loads your custom model, and my_model.generate is a method that generates a response from your model. You'll need to replace these with your actual model loading and generation code.

Please note that this is a simplified example and you might need to adjust it to fit your specific needs, especially if your model's generation method requires different parameters or returns a different type of response.

You can find more details about the HuggingfaceHubLargeLanguageModel class and its methods in the api/core/model_runtime/model_providers/huggingface_hub/llm/llm.py file.

I hope this helps! If you have any other questions or need further clarification, feel free to ask.

Sources

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

@dosubot[bot] commented on GitHub (Feb 21, 2024): _🤖_ <!-- Greeting --> Hey @ntvuongg, nice to see you diving into new adventures with Dify! How's everything going? <!-- Answer --> Based on the information you've provided and the context from the Dify repository, it seems like you want to use your custom LLM model in a self-hosted Dify instead of the existing model accessed through API. To achieve this, you would need to modify the `_invoke` method in the `HuggingfaceHubLargeLanguageModel` class. This method currently uses the `InferenceClient` from the `huggingface_hub` package to make requests to the Hugging Face API. If you want to use your own model, you need to replace this client with your own implementation that loads and uses your model. Here's a simplified example of how you might do this: ```py def _invoke(self, model: str, credentials: dict, prompt_messages: list[PromptMessage], model_parameters: dict, tools: Optional[list[PromptMessageTool]] = None, stop: Optional[list[str]] = None, stream: bool = True, user: Optional[str] = None) -> Union[LLMResult, Generator]: # Load your custom model here my_model = load_my_model(model) # Convert the prompt messages to a single string prompt = self._convert_messages_to_prompt(prompt_messages) # Use your model to generate a response response = my_model.generate(prompt, **model_parameters) # Handle the response if stream: return self._handle_generate_stream_response(model, credentials, prompt_messages, response) return self._handle_generate_response(model, credentials, prompt_messages, response) ``` In this example, `load_my_model` is a function that loads your custom model, and `my_model.generate` is a method that generates a response from your model. You'll need to replace these with your actual model loading and generation code. Please note that this is a simplified example and you might need to adjust it to fit your specific needs, especially if your model's generation method requires different parameters or returns a different type of response. You can find more details about the `HuggingfaceHubLargeLanguageModel` class and its methods in the [api/core/model_runtime/model_providers/huggingface_hub/llm/llm.py](https://github.com/langgenius/dify/blob/main/api/core/model_runtime/model_providers/huggingface_hub/llm/llm.py) file. I hope this helps! If you have any other questions or need further clarification, feel free to ask. ## Sources - [Can we support the locally deployed private models directly through HTTP, instead of models deployed through frameworks(LocalAI，Ollama etc.)??](https://github.com/langgenius/dify/issues/2198) - [能否支持本地的LLM？](https://github.com/langgenius/dify/issues/852) - [Documentation/Roadmap for self-hosted Model providers](https://github.com/langgenius/dify/issues/97) - [api/core/model_runtime/model_providers/huggingface_hub/llm/llm.py](https://github.com/langgenius/dify/blob/main/api/core/model_runtime/model_providers/huggingface_hub/llm/llm.py) <!-- Dosu Comment Footer --> <details open> <summary> #### About Dosu </summary> This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant. If you want to continue the conversation, start your reply with @dosu-bot. </details>
yindo commented 2026-02-21 17:29:53 -05:00
Author
Owner
Copy Link
Copy Source

@crazywoola commented on GitHub (Feb 21, 2024):

Please refer the page https://github.com/langgenius/dify/tree/main/api/core/model_runtime to see how to implement your own provider.
Or you can use xinference and localai to deploy yours. See this https://docs.dify.ai/tutorials/model-configuration/xinference for details.

@crazywoola commented on GitHub (Feb 21, 2024): Please refer the page https://github.com/langgenius/dify/tree/main/api/core/model_runtime to see how to implement your own provider. Or you can use xinference and localai to deploy yours. See this https://docs.dify.ai/tutorials/model-configuration/xinference for details.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#1078
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?