mirror of
https://github.com/langgenius/dify-official-plugins.git
synced 2026-07-25 05:25:37 -04:00
Test the self-deployed Gemini API #639
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @ronge1012 on GitHub (Sep 11, 2025).
Self Checks
Dify version
version:1.7.1
Cloud or Self Hosted
Self Hosted (Docker)
Steps to reproduce
Testing Process
test version:1.7.1
Test Workflow:
Test results: It works.
Using gpt4o-audio, which has audio capabilities (small icon), enables new fields related to audio.
Using the official Gemini API with audio capabilities (small icon) allows for new audio fields.
Using the self-hosted version of Gemini, it lacks audio capabilities (small icon) and has no fields related to audio.
When using the self-hosted version of Gemini to recognize images, the model possesses visual capabilities and is capable of identifying images.
This is the model we provide.
We examined the source code and found that the section calling the OpenAI-compatible interface is encrypted, making it impossible for us to modify it ourselves. So, is it possible to add audio capabilities to the OpenAI-compatible functionality?
✔️ Expected Behavior
Give me a solution
❌ Actual Behavior
No response
@dosubot[bot] commented on GitHub (Sep 11, 2025):
Hi @ronge1012! I'm Dosu and I’m helping the dify team.
Currently, Dify exposes model capabilities like audio through the
featuresfield in the model schema, which is set during provider/model registration. For your self-hosted Gemini model, audio features are missing because the schema does not includeModelFeature.AUDIO, and there is no configuration or plugin mechanism to enable this without code changes. Even if you could patch the schema, the backend must support audio endpoints for it to work—otherwise, the UI fields will not function as expected.There are ongoing efforts to standardize media content types (including audio) for Gemini and OpenAI-compatible models, but these changes are not yet merged and are unavailable in Dify 1.7.1 [source]. The OpenAI-compatible plugin also lacks audio processing logic, so adding audio capability would require updating the code to handle audio prompt messages [source].
If you need speech-to-text or TTS features, you can enable them in the system provider's settings and the app's Function settings, then publish the app for changes to take effect [source]. However, this does not enable audio input/output for the Gemini model itself.
In summary: enabling audio for self-hosted Gemini via the OpenAI-compatible interface is not possible without code changes or waiting for upstream updates that add this support. If you need audio now, use a model and provider that already advertises audio capability in Dify.
To reply, just mention @dosu.
How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other

@dosubot[bot] commented on GitHub (Sep 11, 2025):
The self-hosted Gemini API in Dify currently lacks audio capabilities because the underlying Gemini model and its API do not expose audio endpoints or features. Dify's OpenAI-compatible interface can only surface audio support if both the model and its provider configuration explicitly support it. While Dify's infrastructure does recognize audio and vision features at the metadata/configuration level, the self-hosted Gemini implementation does not include the necessary audio processing routines or annotate audio support, so the UI and API do not show audio fields or icons. There is no hardcoded technical barrier in Dify's code; the limitation is with the self-hosted Gemini model/API itself, not the Dify integration. To enable audio, the Gemini model/API you deploy must support audio features, and the provider configuration must expose them—currently, this is not the case for self-hosted Gemini setups1 2 3 .
To reply, just mention @dosu.
How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other

OpenAI plugin missing audio support ↩︎
Model capability system in Dify ↩︎
Gemini plugin config and limitations ↩︎
@SenWeiV commented on GitHub (Sep 16, 2025):
hi! so you mean that this issue is not the platform issue? they need to make the self-hosted Gemini api to support the audio features?
@SenWeiV commented on GitHub (Sep 16, 2025):
cool! I think this might be a good solution. with the update of the dify and "open ai compatible plugin", this problem should be solved. Hope this can be done soon!
@newnewliu commented on GitHub (Sep 16, 2025):
DEAL WITH IT! IT BOTHER ME A LOT!
@Alwaysoffline commented on GitHub (Sep 16, 2025):
I also encountered the same problem, hope you can pay attention to it.
@dosubot[bot] commented on GitHub (Oct 2, 2025):
Hi, @ronge1012. I'm Dosu, and I'm helping the dify-official-plugins team manage their backlog and am marking this issue as stale.
Issue Summary:
Next Steps:
Thank you for your understanding and contribution!