mirror of
https://github.com/langgenius/dify-official-plugins.git
synced 2026-07-21 17:45:23 -04:00
How to add a multimodal LLM like qwen3-omni-flash that supports audio input? #846
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @305758409 on GitHub (Dec 1, 2025).
Self Checks
1. Is this request related to a challenge you're experiencing? Tell me about your story.
I am trying to integrate a multimodal LLM (specifically qwen3-omni-flash) that supports audio input into Dify for a project requiring audio processing capabilities. However, I’ve encountered several obstacles. First, the Tongyi Qianwen plugin does not include a model similar to qwen3-omni-flash, and I was unable to add it manually through the plugin. Next, I attempted to use the OpenAI-API-compatible provider, but the LLM type dropdown only includes a "Vision" support option—no audio-specific option is available. I tried selecting "Vision" support and providing audio input, but the model did not receive or process the audio data.
I noticed that the Gemini plugin in Dify does support audio input, confirming that Dify itself is capable of handling full multimodal models. The issue seems to be that existing plugins do not support adding a model like qwen3-omni-flash.
I would like to know what options are available to work around this. For example:
Can this be achieved through an HTTP node?
Is there another existing plugin that can be used?
Would I need to develop a custom plugin, modify an existing one, or make changes to the Dify source code directly?
Any guidance on the recommended approach would be greatly appreciated.
2. Additional context or comments
Dify version: [1.10.1]
Relevant plugins tested: Tongyi Qianwen, OpenAI-API-compatible, Gemini
Audio input format attempted: [WAV .etc]
Reference: Gemini plugin demonstrates audio input support in Dify.
If custom development is required, are there any documentation or examples for adding new multimodal model providers?
3. Can you help us with this feature?
@dosubot[bot] commented on GitHub (Dec 1, 2025):
Hi @305758409! I'm Dosu and I’m helping the dify-official-plugins team.
To add a multimodal LLM like qwen3-omni-flash with audio input support to Dify, you’ll need to develop a custom plugin or extend an existing one—there’s no out-of-the-box solution or official example for this yet. The Gemini plugin is a good reference: it handles audio by converting prompt messages into SDK content parts, uploading files (audio, image, etc.), and passing their URIs and MIME types to the model API. Audio input is represented as a specific content type and flows through the same upload pipeline as other files, with MIME types preserved so the model can consume the stream (Gemini plugin code, Gemini plugin tests).
Dify’s plugin architecture supports multimodal features, including audio, at both the data model and API levels. To integrate qwen3-omni-flash, you would: define the provider and credential schema, add a YAML entry describing the model and its modalities, implement the model logic to handle audio input (upload, URI, MIME type), and register the plugin so Dify recognizes it as a provider. You may need to tune modality-specific helpers and ensure the UI supports audio input (plugin architecture, model entities, plugin client).
HTTP node or direct API calls could be a workaround, but there’s no documentation confirming this approach for audio input models. The most robust path is to follow the Gemini plugin’s structure and adapt it for qwen3-omni-flash, ensuring you handle audio uploads and prompt construction in a similar way.
If you need a template, the Gemini plugin’s code and tests provide a practical example of how to handle audio and other modalities in a Dify plugin.
To reply, just mention @dosu.
How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other