langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-08-27 21:10:04 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Cannot add openai compatible stt model. #5820

New Issue
Closed
opened 2026-02-21 18:12:44 -05:00 by yindo · 11 comments
yindo commented 2026-02-21 18:12:44 -05:00
Owner
Copy Link
Copy Source

Originally created by @tetyoka on GitHub (Sep 26, 2024).

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
  • Please do not modify this template :) and fill in all the required fields.

Dify version

0.8.3

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

Just add OpenAI-API-compatible Speech2text model.

I'm using faster-whisper-server for local OA-compatible API server.
This server is running on another docker container in same computer.

It looks like every API request was succeeded. However, Speech2text model does not appear in model list.
image

It may be caused by my local whisper server. But since Dify app tells me "Operation successful", it should be Dify's bug.
image

✔️ Expected Behavior

STT model appear in model list. If operation failed, I get notification.

❌ Actual Behavior

All process looks like succeeded. STT model does not appear.

Originally created by @tetyoka on GitHub (Sep 26, 2024). ### Self Checks - [X] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [X] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [X] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [X] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:） - [X] Please do not modify this template :) and fill in all the required fields. ### Dify version 0.8.3 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce Just add OpenAI-API-compatible Speech2text model. I'm using [faster-whisper-server](https://github.com/fedirz/faster-whisper-server) for local OA-compatible API server. This server is running on another docker container in same computer. It looks like every API request was succeeded. However, Speech2text model does not appear in model list. ![image](https://github.com/user-attachments/assets/65be23aa-3537-4ce1-9297-fead04328849) It may be caused by my local whisper server. But since Dify app tells me "Operation successful", it should be Dify's bug. ![image](https://github.com/user-attachments/assets/e64d5ce3-35dc-45a6-839d-bb361dcb931b) ### ✔️ Expected Behavior STT model appear in model list. If operation failed, I get notification. ### ❌ Actual Behavior All process looks like succeeded. STT model does not appear.
yindo added the 🐞 bug🤔 cant-reproduce labels 2026-02-21 18:12:44 -05:00
yindo closed this issue 2026-02-21 18:12:44 -05:00
yindo commented 2026-02-21 18:12:44 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Sep 26, 2024):

Hey @tetyoka! I'm here to help you with any bugs, questions, or contributions you have for Dify. Let's get this issue sorted out together!

There are a few similar issues and discussions related to adding OpenAI-compatible speech-to-text (STT) models in Dify:

  1. No voice to text feature in my 0.8.3 version after self-host deployment [1].
  2. Help: How can the locally deployed Qwen 1.5-72b model be added to Dify [2].

A common cause for the issue where the OpenAI-compatible Speech2text model does not appear in the model list, despite successful API requests and notifications, is the failure of credentials validation. The credentials validation might be failing, which prevents the model from being listed.

Ensure that the openai_api_key provided in the credentials is correct and valid. If the credentials are invalid, the model will not be listed. Additionally, make sure that the local faster-whisper-server is properly configured and accessible.

Here is a relevant snippet from the test script that includes a validation step:

def test_validate_credentials():
    model = OpenAISpeech2TextModel()

    with pytest.raises(CredentialsValidateFailedError):
        model.validate_credentials(model="whisper-1", credentials={"openai_api_key": "invalid_key"})

    model.validate_credentials(model="whisper-1", credentials={"openai_api_key": os.environ.get("OPENAI_API_KEY")})

The OAICompatSpeech2TextModel class in the Dify codebase shows that the _invoke method is responsible for making the API call to the speech-to-text service. This method constructs the request using the provided api_key and endpoint_url from the credentials. If the endpoint_url is not correctly configured or if the API key is invalid, the request will fail, and the model will not be listed.

def _invoke(self, model: str, credentials: dict, file: IO[bytes], user: Optional[str] = None) -> str:
    headers = {}

    api_key = credentials.get("api_key")
    if api_key:
        headers["Authorization"] = f"Bearer {api_key}"

    endpoint_url = credentials.get("endpoint_url")
    if not endpoint_url.endswith("/"):
        endpoint_url += "/"
    endpoint_url = urljoin(endpoint_url, "audio/transcriptions")

    payload = {"model": model}
    files = [("file", file)]
    response = requests.post(endpoint_url, headers=headers, data=payload, files=files)

    if response.status_code != 200:
        raise InvokeBadRequestError(response.text)
    response_data = response.json()
    return response_data["text"]

Make sure that the endpoint_url and api_key in the credentials are correctly set and that the local faster-whisper-server is running and accessible at the specified endpoint_url [3][4][5].

To continue talking to Dosu, mention @dosu.

@dosubot[bot] commented on GitHub (Sep 26, 2024): <!-- Greeting --> Hey @tetyoka! I'm here to help you with any bugs, questions, or contributions you have for Dify. Let's get this issue sorted out together! <!-- Answer --> There are a few similar issues and discussions related to adding OpenAI-compatible speech-to-text (STT) models in Dify: 1. [No voice to text feature in my 0.8.3 version after self-host deployment](https://github.com/langgenius/dify/discussions/8642) <sup>[[1]](https://github.com/langgenius/dify/discussions/8642)</sup>. 2. [Help: How can the locally deployed Qwen 1.5-72b model be added to Dify](https://github.com/langgenius/dify/discussions/7524) <sup>[[2]](https://github.com/langgenius/dify/discussions/7524)</sup>. A common cause for the issue where the OpenAI-compatible Speech2text model does not appear in the model list, despite successful API requests and notifications, is the failure of credentials validation. The credentials validation might be failing, which prevents the model from being listed. Ensure that the `openai_api_key` provided in the credentials is correct and valid. If the credentials are invalid, the model will not be listed. Additionally, make sure that the local faster-whisper-server is properly configured and accessible. Here is a relevant snippet from the test script that includes a validation step: ```python def test_validate_credentials(): model = OpenAISpeech2TextModel() with pytest.raises(CredentialsValidateFailedError): model.validate_credentials(model="whisper-1", credentials={"openai_api_key": "invalid_key"}) model.validate_credentials(model="whisper-1", credentials={"openai_api_key": os.environ.get("OPENAI_API_KEY")}) ``` The `OAICompatSpeech2TextModel` class in the Dify codebase shows that the `_invoke` method is responsible for making the API call to the speech-to-text service. This method constructs the request using the provided `api_key` and `endpoint_url` from the credentials. If the `endpoint_url` is not correctly configured or if the API key is invalid, the request will fail, and the model will not be listed. ```python def _invoke(self, model: str, credentials: dict, file: IO[bytes], user: Optional[str] = None) -> str: headers = {} api_key = credentials.get("api_key") if api_key: headers["Authorization"] = f"Bearer {api_key}" endpoint_url = credentials.get("endpoint_url") if not endpoint_url.endswith("/"): endpoint_url += "/" endpoint_url = urljoin(endpoint_url, "audio/transcriptions") payload = {"model": model} files = [("file", file)] response = requests.post(endpoint_url, headers=headers, data=payload, files=files) if response.status_code != 200: raise InvokeBadRequestError(response.text) response_data = response.json() return response_data["text"] ``` Make sure that the `endpoint_url` and `api_key` in the credentials are correctly set and that the local faster-whisper-server is running and accessible at the specified `endpoint_url` <sup>[[3]](https://github.com/langgenius/dify/blob/main/api/tests/integration_tests/model_runtime/openai_api_compatible/test_speech2text.py)</sup><sup>[[4]](https://github.com/langgenius/dify/blob/main/api/tests/integration_tests/model_runtime/openai/test_speech2text.py)</sup><sup>[[5]](https://github.com/langgenius/dify/blob/main/api/core/model_runtime/model_providers/openai_api_compatible/speech2text/speech2text.py)</sup>. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.*
yindo commented 2026-02-21 18:12:44 -05:00
Author
Owner
Copy Link
Copy Source

@crazywoola commented on GitHub (Sep 26, 2024):

I think you need to refresh the page to see the newly added model.

@crazywoola commented on GitHub (Sep 26, 2024): I think you need to refresh the page to see the newly added model.
yindo commented 2026-02-21 18:12:44 -05:00
Author
Owner
Copy Link
Copy Source

@tetyoka commented on GitHub (Sep 26, 2024):

I did. The model does not appear.

@tetyoka commented on GitHub (Sep 26, 2024): I did. The model does not appear.
yindo commented 2026-02-21 18:12:44 -05:00
Author
Owner
Copy Link
Copy Source

@crazywoola commented on GitHub (Sep 26, 2024):

QQ_1727361096875

Here？

@crazywoola commented on GitHub (Sep 26, 2024): ![QQ_1727361096875](https://github.com/user-attachments/assets/c63a9423-d4ef-4dbe-9bd1-c56a84b88c9f) Here？
yindo commented 2026-02-21 18:12:45 -05:00
Author
Owner
Copy Link
Copy Source

@tetyoka commented on GitHub (Sep 26, 2024):

image
It says 0 model.
And also.
image

@tetyoka commented on GitHub (Sep 26, 2024): ![image](https://github.com/user-attachments/assets/aca7c2c7-6bfb-4b70-aaf4-6f7d8d1a2097) It says 0 model. And also. ![image](https://github.com/user-attachments/assets/81e6f766-5a87-4607-b4ac-70414e35610c)
yindo commented 2026-02-21 18:12:45 -05:00
Author
Owner
Copy Link
Copy Source

@tetyoka commented on GitHub (Sep 26, 2024):

Despite my STT situation, I can add OA-compatible LLM API(oobagooba).
image
LLM API is running on another computer.

@tetyoka commented on GitHub (Sep 26, 2024): Despite my STT situation, I can add OA-compatible LLM API(oobagooba). ![image](https://github.com/user-attachments/assets/b2182537-f57c-4fd2-a745-413aa7abd185) LLM API is running on another computer.
yindo commented 2026-02-21 18:12:45 -05:00
Author
Owner
Copy Link
Copy Source

@tetyoka commented on GitHub (Oct 2, 2024):

I updated Dify to 0.9.1. It still happen.

@tetyoka commented on GitHub (Oct 2, 2024): I updated Dify to 0.9.1. It still happen.
yindo commented 2026-02-21 18:12:45 -05:00
Author
Owner
Copy Link
Copy Source

@zhao85 commented on GitHub (Oct 3, 2024):

that's because current openai-api-compatible/openai code of speech2text has not implement get_customizable_model_schema method. I have copied the code from localai to fix it.

@zhao85 commented on GitHub (Oct 3, 2024): that's because current openai-api-compatible/openai code of speech2text has not implement get_customizable_model_schema method. I have copied the code from localai to fix it.
yindo commented 2026-02-21 18:12:45 -05:00
Author
Owner
Copy Link
Copy Source

@tetyoka commented on GitHub (Oct 4, 2024):

I will try your patch!

@tetyoka commented on GitHub (Oct 4, 2024): I will try your patch!
yindo commented 2026-02-21 18:12:45 -05:00
Author
Owner
Copy Link
Copy Source

@andresvidal commented on GitHub (Jan 27, 2025):

Thanks for the awesome project! I'm not sure if this is related, but I keep getting an error from OpenAI models hosted in a different base_url. When I run the same "workflow" with Ollama Llama3.2 it works but when I swap it out for an OpenAi version I get this:

using OpenAi with a different host api
[openai] Error: 1 validation error for LLMResultChunk model Input should be a valid string [type=string_type, input_value=None, input_type=NoneType] For further information visit https://errors.pydantic.dev/2.9/v/string_type

Using Ollama with a different host api
{ "text": "", ..., "finish_reason": "Non-JSON encountered.", ...}

@andresvidal commented on GitHub (Jan 27, 2025): Thanks for the awesome project! I'm not sure if this is related, but I keep getting an error from OpenAI models hosted in a different base_url. When I run the same "workflow" with Ollama Llama3.2 it works but when I swap it out for an OpenAi version I get this: **using OpenAi with a different host api** [openai] Error: 1 validation error for LLMResultChunk model Input should be a valid string [type=string_type, input_value=None, input_type=NoneType] For further information visit https://errors.pydantic.dev/2.9/v/string_type **Using Ollama with a different host api** { "text": "", ..., "finish_reason": "Non-JSON encountered.", ...}
yindo commented 2026-02-21 18:12:45 -05:00
Author
Owner
Copy Link
Copy Source

@sean-escaped commented on GitHub (Mar 3, 2025):

Thanks for the awesome project! I'm not sure if this is related, but I keep getting an error from OpenAI models hosted in a different base_url. When I run the same "workflow" with Ollama Llama3.2 it works but when I swap it out for an OpenAi version I get this:

using OpenAi with a different host api [openai] Error: 1 validation error for LLMResultChunk model Input should be a valid string [type=string_type, input_value=None, input_type=NoneType] For further information visit https://errors.pydantic.dev/2.9/v/string_type

Using Ollama with a different host api { "text": "", ..., "finish_reason": "Non-JSON encountered.", ...}

I have the same issue

@sean-escaped commented on GitHub (Mar 3, 2025): > Thanks for the awesome project! I'm not sure if this is related, but I keep getting an error from OpenAI models hosted in a different base_url. When I run the same "workflow" with Ollama Llama3.2 it works but when I swap it out for an OpenAi version I get this: > > **using OpenAi with a different host api** [openai] Error: 1 validation error for LLMResultChunk model Input should be a valid string [type=string_type, input_value=None, input_type=NoneType] For further information visit https://errors.pydantic.dev/2.9/v/string_type > > **Using Ollama with a different host api** { "text": "", ..., "finish_reason": "Non-JSON encountered.", ...} I have the same issue
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🐞 bug 🤔 cant-reproduce
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#5820
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?