langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 20:15:31 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

How do I add speech-to-text and text-to-speech models when using xinference? #1022

New Issue
Closed
opened 2026-02-21 17:29:32 -05:00 by yindo · 5 comments
yindo commented 2026-02-21 17:29:32 -05:00
Owner
Copy Link
Copy Source

Originally created by @andylzming on GitHub (Feb 2, 2024).

Self Checks

Dify version

0.5.2

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

How do I add speech-to-text and text-to-speech models when using xinference? Xinference already supports Image model and Audio model.

20

21

22

✔️ Expected Behavior

No response

❌ Actual Behavior

No response

Originally created by @andylzming on GitHub (Feb 2, 2024). ### Self Checks - [X] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [X] I confirm that I am using English to file this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). ### Dify version 0.5.2 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce How do I add speech-to-text and text-to-speech models when using xinference? Xinference already supports Image model and Audio model. ![20](https://github.com/langgenius/dify/assets/21255047/80a3714c-0f1f-4a4b-9936-60237ebc7517) ![21](https://github.com/langgenius/dify/assets/21255047/cdcffa9c-3fad-44f9-9db9-ef8a156e312d) ![22](https://github.com/langgenius/dify/assets/21255047/e55f9228-7a1f-4a8b-82f7-e0f351e0c684) ### ✔️ Expected Behavior _No response_ ### ❌ Actual Behavior _No response_
yindo added the 📚 documentation label 2026-02-21 17:29:32 -05:00
yindo closed this issue 2026-02-21 17:29:32 -05:00
yindo commented 2026-02-21 17:29:32 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Feb 2, 2024):

🤖

Hey @andylzming! Great to see you back here, diving into more adventures with dify. 🚀

To add speech-to-text and text-to-speech models when using xinference in a self-hosted Docker environment with Dify version 0.5.2, you need to extend the provided TongyiText2SpeechModel class and implement the required methods.

Here is a simplified example of how you can do this:

class MyText2SpeechModel(TongyiText2SpeechModel):
    def _invoke(self, model: str, credentials: dict, content_text: str, streaming: bool, user: Optional[str] = None) -> any:
        # Your implementation here
        pass

    def validate_credentials(self, model: str, credentials: dict, user: Optional[str] = None) -> None:
        # Your implementation here
        pass

    def _tts_invoke(self, model: str, credentials: dict, content_text: str, user: Optional[str] = None) -> Response:
        # Your implementation here
        pass

    def _tts_invoke_streaming(self, model: str, credentials: dict, content_text: str, user: Optional[str] = None) -> any:
        # Your implementation here
        pass

    def _process_sentence(self, sentence: str, model: str, credentials: dict, audio_type: str):
        # Your implementation here
        pass

You need to replace the # Your implementation here comments with your own code. The _invoke method is used to invoke the text-to-speech model, _tts_invoke and _tts_invoke_streaming are used to process the text and convert it to speech, and _process_sentence is used to process each sentence individually. The validate_credentials method is used to validate the credentials for the text-to-speech model.

Once you have implemented these methods, you can use your new class in your Docker environment. Make sure to provide the necessary credentials and other parameters when invoking the model.

You can find more details about the TongyiText2SpeechModel class in the dify codebase.

Please note that I'm unable to view the images you've attached. If you need further assistance, could you please provide more details or context in text form?

Sources

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

@dosubot[bot] commented on GitHub (Feb 2, 2024): _🤖_ <!-- Greeting --> Hey @andylzming! Great to see you back here, diving into more adventures with dify. 🚀 <!-- Answer --> To add speech-to-text and text-to-speech models when using xinference in a self-hosted Docker environment with Dify version 0.5.2, you need to extend the provided `TongyiText2SpeechModel` class and implement the required methods. Here is a simplified example of how you can do this: ```python class MyText2SpeechModel(TongyiText2SpeechModel): def _invoke(self, model: str, credentials: dict, content_text: str, streaming: bool, user: Optional[str] = None) -> any: # Your implementation here pass def validate_credentials(self, model: str, credentials: dict, user: Optional[str] = None) -> None: # Your implementation here pass def _tts_invoke(self, model: str, credentials: dict, content_text: str, user: Optional[str] = None) -> Response: # Your implementation here pass def _tts_invoke_streaming(self, model: str, credentials: dict, content_text: str, user: Optional[str] = None) -> any: # Your implementation here pass def _process_sentence(self, sentence: str, model: str, credentials: dict, audio_type: str): # Your implementation here pass ``` You need to replace the `# Your implementation here` comments with your own code. The `_invoke` method is used to invoke the text-to-speech model, `_tts_invoke` and `_tts_invoke_streaming` are used to process the text and convert it to speech, and `_process_sentence` is used to process each sentence individually. The `validate_credentials` method is used to validate the credentials for the text-to-speech model. Once you have implemented these methods, you can use your new class in your Docker environment. Make sure to provide the necessary credentials and other parameters when invoking the model. You can find more details about the `TongyiText2SpeechModel` class in the [dify codebase](https://github.com/langgenius/dify/blob/main/api/core/model_runtime/model_providers/tongyi/tts/tts.py). Please note that I'm unable to view the images you've attached. If you need further assistance, could you please provide more details or context in text form? ## Sources - [api/core/model_runtime/model_providers/tongyi/tts/tts.py](https://github.com/langgenius/dify/blob/main/api/core/model_runtime/model_providers/tongyi/tts/tts.py) - [api/core/model_runtime/model_providers/xinference/rerank/rerank.py](https://github.com/langgenius/dify/blob/main/api/core/model_runtime/model_providers/xinference/rerank/rerank.py) --- This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant. If you want to continue the conversation, start your reply with @dosu-bot.
yindo commented 2026-02-21 17:29:32 -05:00
Author
Owner
Copy Link
Copy Source

@crazywoola commented on GitHub (Feb 2, 2024):

See https://docs.dify.ai/tutorials/model-configuration/xinference#starting-xinference for more details. Please use the right template to fire a bug.

@crazywoola commented on GitHub (Feb 2, 2024): See https://docs.dify.ai/tutorials/model-configuration/xinference#starting-xinference for more details. Please use the right template to fire a bug.
yindo commented 2026-02-21 17:29:32 -05:00
Author
Owner
Copy Link
Copy Source

@andylzming commented on GitHub (Feb 5, 2024):

@crazywoola Is the functionality of adding Image models and Audit models (including speech-to-text, text-to-speech) through xinference set to be implemented in the next version?

@andylzming commented on GitHub (Feb 5, 2024): @crazywoola Is the functionality of adding Image models and Audit models (including speech-to-text, text-to-speech) through xinference set to be implemented in the next version?
yindo commented 2026-02-21 17:29:32 -05:00
Author
Owner
Copy Link
Copy Source

@takatost commented on GitHub (Feb 7, 2024):

@crazywoola Is the functionality of adding Image models and Audit models (including speech-to-text, text-to-speech) through xinference set to be implemented in the next version?

We might integrate these models in our spare time, not sure though. If you're interested, you can contribute to us and we would really appreciate it!

@takatost commented on GitHub (Feb 7, 2024): > @crazywoola Is the functionality of adding Image models and Audit models (including speech-to-text, text-to-speech) through xinference set to be implemented in the next version? We might integrate these models in our spare time, not sure though. If you're interested, you can contribute to us and we would really appreciate it!
yindo commented 2026-02-21 17:29:32 -05:00
Author
Owner
Copy Link
Copy Source

@leslie2046 commented on GitHub (Mar 30, 2024):

#3045

@leslie2046 commented on GitHub (Mar 30, 2024): #3045
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 📚 documentation
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#1022
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?