langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-23 13:05:55 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Unable to Upload and Process Audio Using Multimodal Models (GPT Audio Preview, Gemini 1.5 Pro) #7152

New Issue
Closed
opened 2026-02-21 18:19:09 -05:00 by yindo · 3 comments
yindo commented 2026-02-21 18:19:09 -05:00
Owner
Copy Link
Copy Source

Originally created by @ottoradiologia on GitHub (Dec 11, 2024).

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
  • Please do not modify this template :) and fill in all the required fields.

Dify version

13.2

Cloud or Self Hosted

Cloud

Steps to reproduce

I’m experiencing an issue where audio files cannot be properly uploaded and processed by the multimodal models (GPT audio preview and Gemini 1.5 Pro) through the Dify platform. When I attempt to upload an audio file, it successfully generates a URL, but in subsequent follow-up messages, no output or processed content is returned. The input field appears empty, and the audio is never processed.

✔️ Expected Behavior

After uploading the audio file and providing the generated URL in a follow-up request, the multimodal models should transcribe or otherwise process the audio, returning the corresponding output (e.g., transcription text).

❌ Actual Behavior

The response is empty, and no audio processing occurs, despite the upload being successful.

Additional Information:

The issue persists across different audio file formats.
Re-attempting with different files does not resolve the problem.
Other non-audio inputs work as expected in the follow-up messages.
Request for Assistance:
Could you please investigate this issue and advise on whether it is a known bug, a configuration error, or if additional permissions are required to enable audio processing on the Dify platform?

Originally created by @ottoradiologia on GitHub (Dec 11, 2024). ### Self Checks - [X] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [X] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [X] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [X] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:） - [X] Please do not modify this template :) and fill in all the required fields. ### Dify version 13.2 ### Cloud or Self Hosted Cloud ### Steps to reproduce I’m experiencing an issue where audio files cannot be properly uploaded and processed by the multimodal models (GPT audio preview and Gemini 1.5 Pro) through the Dify platform. When I attempt to upload an audio file, it successfully generates a URL, but in subsequent follow-up messages, no output or processed content is returned. The input field appears empty, and the audio is never processed. ### ✔️ Expected Behavior After uploading the audio file and providing the generated URL in a follow-up request, the multimodal models should transcribe or otherwise process the audio, returning the corresponding output (e.g., transcription text). ### ❌ Actual Behavior The response is empty, and no audio processing occurs, despite the upload being successful. Additional Information: The issue persists across different audio file formats. Re-attempting with different files does not resolve the problem. Other non-audio inputs work as expected in the follow-up messages. Request for Assistance: Could you please investigate this issue and advise on whether it is a known bug, a configuration error, or if additional permissions are required to enable audio processing on the Dify platform?
yindo added the 💪 enhancementstt & tts labels 2026-02-21 18:19:09 -05:00
yindo closed this issue 2026-02-21 18:19:09 -05:00
yindo commented 2026-02-21 18:19:10 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Dec 11, 2024):

The issue you're experiencing, where audio files are uploaded successfully but not processed, could be due to several factors. Here are some potential causes and solutions:

  1. File Format and Size: Ensure that the audio files are in a supported format. Dify supports formats like flac, mp3, mp4, mpeg, mpga, m4a, ogg, wav, and webm. Also, check that the file size does not exceed the upload limit [1][2][3].

  2. S3 Upload and URL Generation: The process involves generating a URL and uploading the file to an S3 bucket. If the URL is generated but the file is not processed, there might be an issue with the S3 upload or the subsequent steps. Ensure that the S3 bucket permissions and configurations are correct [4].

  3. Transcription Job: After uploading, a transcription job is initiated. If this job fails, it could be due to incorrect parameters or configuration issues. Check the logs for any specific error messages related to the transcription job [4].

  4. Error Messages: Review any error messages or logs generated during the process. Common errors include NoAudioUploadedError, AudioTooLargeError, and UnsupportedAudioTypeError, among others [5][6].

  5. AWS SageMaker Endpoint: If you're using AWS for processing, ensure that the SageMaker endpoint is correctly set up and accessible. The endpoint processes the audio file and returns the transcribed text [4][7].

  6. Configuration and Permissions: Verify that all necessary configurations and permissions are in place for audio processing. This includes ensuring that any required AWS credentials and permissions are correctly configured.

If these steps do not resolve the issue, it might be helpful to check for any updates or patches for Dify version 13.2 that address known bugs related to audio processing.

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other

@dosubot[bot] commented on GitHub (Dec 11, 2024): <!-- Answer --> The issue you're experiencing, where audio files are uploaded successfully but not processed, could be due to several factors. Here are some potential causes and solutions: 1. **File Format and Size**: Ensure that the audio files are in a supported format. Dify supports formats like flac, mp3, mp4, mpeg, mpga, m4a, ogg, wav, and webm. Also, check that the file size does not exceed the upload limit <sup>[[1]](https://github.com/langgenius/dify/blob/main/api/core/model_runtime/model_providers/gitee_ai/speech2text/whisper-large.yaml)</sup><sup>[[2]](https://github.com/langgenius/dify/blob/main/api/core/model_runtime/model_providers/tencent/speech2text/tencent.yaml)</sup><sup>[[3]](https://github.com/langgenius/dify/blob/main/api/core/model_runtime/model_providers/groq/speech2text/distil-whisper-large-v3-en.yaml)</sup>. 2. **S3 Upload and URL Generation**: The process involves generating a URL and uploading the file to an S3 bucket. If the URL is generated but the file is not processed, there might be an issue with the S3 upload or the subsequent steps. Ensure that the S3 bucket permissions and configurations are correct <sup>[[4]](https://github.com/langgenius/dify/blob/main/api/core/tools/provider/builtin/aws/tools/transcribe_asr.py)</sup>. 3. **Transcription Job**: After uploading, a transcription job is initiated. If this job fails, it could be due to incorrect parameters or configuration issues. Check the logs for any specific error messages related to the transcription job <sup>[[4]](https://github.com/langgenius/dify/blob/main/api/core/tools/provider/builtin/aws/tools/transcribe_asr.py)</sup>. 4. **Error Messages**: Review any error messages or logs generated during the process. Common errors include `NoAudioUploadedError`, `AudioTooLargeError`, and `UnsupportedAudioTypeError`, among others <sup>[[5]](https://github.com/langgenius/dify/blob/main/api/services/errors/audio.py)</sup><sup>[[6]](https://github.com/langgenius/dify/blob/main/api/controllers/service_api/app/audio.py)</sup>. 5. **AWS SageMaker Endpoint**: If you're using AWS for processing, ensure that the SageMaker endpoint is correctly set up and accessible. The endpoint processes the audio file and returns the transcribed text <sup>[[4]](https://github.com/langgenius/dify/blob/main/api/core/tools/provider/builtin/aws/tools/transcribe_asr.py)</sup><sup>[[7]](https://github.com/langgenius/dify/blob/main/api/core/model_runtime/model_providers/sagemaker/speech2text/speech2text.py)</sup>. 6. **Configuration and Permissions**: Verify that all necessary configurations and permissions are in place for audio processing. This includes ensuring that any required AWS credentials and permissions are correctly configured. If these steps do not resolve the issue, it might be helpful to check for any updates or patches for Dify version 13.2 that address known bugs related to audio processing. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.* --- <sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/abfce17d-b79f-430b-adfa-f30265b28af6?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/abfce17d-b79f-430b-adfa-f30265b28af6?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/abfce17d-b79f-430b-adfa-f30265b28af6?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/abfce17d-b79f-430b-adfa-f30265b28af6?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/abfce17d-b79f-430b-adfa-f30265b28af6?feedback_type=hallucination) | [Other](https://app.dosu.dev/response-feedback/abfce17d-b79f-430b-adfa-f30265b28af6?feedback_type=other)</sup>
yindo commented 2026-02-21 18:19:10 -05:00
Author
Owner
Copy Link
Copy Source

@hjlarry commented on GitHub (Dec 11, 2024):

Hi, I don't know why gpt-audio-preview not work, but this https://github.com/langgenius/dify/pull/11497 will support gemini models understand audio file.
84b04409bc6ca7cfaae090c79a7f33f

@hjlarry commented on GitHub (Dec 11, 2024): Hi, I don't know why gpt-audio-preview not work, but this https://github.com/langgenius/dify/pull/11497 will support gemini models understand audio file. <img width="422" alt="84b04409bc6ca7cfaae090c79a7f33f" src="https://github.com/user-attachments/assets/1d243496-8a95-48c4-ab29-0b7ad5c3414c" />
yindo commented 2026-02-21 18:19:10 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Jan 12, 2025):

Hi, @ottoradiologia. I'm Dosu, and I'm helping the Dify team manage their backlog. I'm marking this issue as stale.

Issue Summary

  • Audio files upload successfully but are not processed by multimodal models.
  • Potential causes and solutions suggested include checking file format, size, S3 permissions, transcription parameters, and SageMaker configuration.
  • A pull request mentioned by "hjlarry" may enable Gemini models to understand audio files, potentially addressing part of the issue.

Next Steps

  • Please confirm if this issue is still relevant to the latest version of the Dify repository by commenting here.
  • If there is no further activity, this issue will be automatically closed in 15 days.

Thank you for your understanding and contribution!

@dosubot[bot] commented on GitHub (Jan 12, 2025): Hi, @ottoradiologia. I'm [Dosu](https://dosu.dev), and I'm helping the Dify team manage their backlog. I'm marking this issue as stale. **Issue Summary** - Audio files upload successfully but are not processed by multimodal models. - Potential causes and solutions suggested include checking file format, size, S3 permissions, transcription parameters, and SageMaker configuration. - A pull request mentioned by "hjlarry" may enable Gemini models to understand audio files, potentially addressing part of the issue. **Next Steps** - Please confirm if this issue is still relevant to the latest version of the Dify repository by commenting here. - If there is no further activity, this issue will be automatically closed in 15 days. Thank you for your understanding and contribution!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label stt & tts 💪 enhancement
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#7152
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?