langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 20:15:31 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

OpenAI Plugin Fails to Send Multimodal Messages with Images #12095

New Issue
Closed
opened 2026-02-21 19:05:44 -05:00 by yindo · 20 comments
yindo commented 2026-02-21 19:05:44 -05:00
Owner
Copy Link
Copy Source

Originally created by @leozanee on GitHub (Mar 25, 2025).

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:)
  • Please do not modify this template :) and fill in all the required fields.

Dify version

1.1.3

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

Title

OpenAI Plugin Fails to Send Multimodal Messages with Images

Description

The plugin exhibits inconsistent behavior in an unknown scenario, causing the OpenAI plugin to malfunction and fail to send multimodal messages containing images.

Steps to Reproduce

  1. Use the OpenAI plugin (versions 0.0.8 - 0.0.13).
  2. Attempt to send a multimodal message with an image (e.g., content including both text and an image).
  3. Check the logs and observe the error.

Error Logs

When sending a multimodal message with an image, the following error appears in the logs:

api-1            | /app/api/.venv/lib/python3.12/site-packages/pydantic/main.py:390: UserWarning: Pydantic serializer warnings:
api-1            |   PydanticSerializationUnexpectedValue: Expected `str` but got `list` with value `[TextPromptMessageContent...il=<DETAIL.LOW: 'low'>)]` - serialized value may not be as expected
api-1            |   PydanticSerializationUnexpectedValue: Expected `PromptMessageContent` but got `TextPromptMessageContent` with value `TextPromptMessageContent(...ata='can you see this?')` - serialized value may not be as expected
api-1            |   return self.__pydantic_serializer__.to_python(
plugin_daemon-1  | [GIN] 2025/03/25 - 07:07:21 | 200 |    9.872525ms |      172.19.0.9 | POST     "/plugin/3559813e-c1d2-4fdc-ada2-e05ac28e4f45/dispatch/llm/num_tokens"
api-1            | 2025-03-25 07:07:21.159 ERROR [Thread-3 (_generate_worker)] [app_generator.py:246] - Unknown Error when generating
api-1            | Traceback (most recent call last):
api-1            |   File "/app/api/core/app/apps/chat/app_generator.py", line 226, in _generate_worker
api-1            |     runner.run(
api-1            |   File "/app/api/core/app/apps/chat/app_runner.py", line 69, in run
api-1            |     self.get_pre_calculate_rest_tokens(
api-1            |   File "/app/api/core/app/apps/base_app_runner.py", line 90, in get_pre_calculate_rest_tokens
api-1            |     prompt_tokens = model_instance.get_llm_num_tokens(prompt_messages)
api-1            |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
api-1            |   File "/app/api/core/model_manager.py", line 195, in get_llm_num_tokens
api-1            |     self._round_robin_invoke(
api-1            |   File "/app/api/core/model_manager.py", line 370, in _round_robin_invoke
api-1            |     return function(*args, **kwargs)
api-1            |            ^^^^^^^^^^^^^^^^^^^^^^^^^
api-1            |   File "/app/api/core/model_runtime/model_providers/__base/large_language_model.py", line 299, in get_num_tokens
api-1            |     return plugin_model_manager.get_llm_num_tokens(
api-1            |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
api-1            |   File "/app/api/core/plugin/manager/model.py", line 231, in get_llm_num_tokens
api-1            |     for resp in response:
api-1            |                 ^^^^^^^^
api-1            |   File "/app/api/core/plugin/manager/base.py", line 189, in _request_with_plugin_daemon_response_stream
api-1            |     self._handle_plugin_daemon_error(error.error_type, error.message)
api-1            |   File "/app/api/core/plugin/manager/base.py", line 221, in _handle_plugin_daemon_error
api-1            |     raise PluginInvokeError(description=message)
api-1            | core.plugin.manager.exc.PluginInvokeError: PluginInvokeError: {"args":{},"error_type":"ValidationError","message":"1 validation error for ModelGetLLMNumTokens\nprompt_messages.content.type\n  Input should be <PromptMessageContentType.TEXT: 'text'> [type=literal_error, input_value='text', input_type=str]\n    For further information visit https://errors.pydantic.dev/2.8/v/literal_error"}
api-1            | 2025-03-25 07:07:21.169 ERROR [Dummy-2] [base_app_generate_response_converter.py:123] - {"args":{},"error_type":"ValidationError","message":"1 validation error for ModelGetLLMNumTokens\nprompt_messages.content.type\n  Input should be <PromptMessageContentType.TEXT: 'text'> [type=literal_error, input_value='text', input_type=str]\n    For further information visit https://errors.pydantic.dev/2.8/v/literal_error"}

Test Results

  • Tested the following third-party provider plugins:
    • Gemini-0.0.8: Works normally, no errors.
    • Anthropic-0.0.12: Works normally, no errors.
    • OpenAI (0.0.8 - 0.0.13): Errors occur, only with the OpenAI plugin.
  • In Dify Cloud-Hosted Environment:
    • Upgrading the OpenAI plugin to 0.0.13 reproduces the issue.
    • Rolling back to 0.0.8 resolves the issue, and it works normally.
  • In Local Docker-Hosted Environment:
    • Even rolling back to 0.0.8 does not resolve the issue; the error persists.

Analysis

  • Suspect that changes in the plugin runtime environment are causing the issue.
  • The problem was initially encountered while developing a new third-party provider plugin:
    • Works fine in debug mode.
    • After packaging, the same error occurs.
  • Reviewed the source code of the OpenAI and Anthropic plugins but found no significant logical differences.
  • The root cause of the error is unclear and may be related to Pydantic’s serialization validation logic or compatibility issues with the plugin environment.

Request for Assistance

Since the OpenAI plugin is an official implementation and more representative, but the cause of the issue remains elusive, I’m seeking help to identify and resolve the problem. Thank you!

✔️ Expected Behavior

No response

❌ Actual Behavior

No response

Originally created by @leozanee on GitHub (Mar 25, 2025). ### Self Checks - [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [x] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:) - [x] Please do not modify this template :) and fill in all the required fields. ### Dify version 1.1.3 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce ### Title OpenAI Plugin Fails to Send Multimodal Messages with Images ### Description The plugin exhibits inconsistent behavior in an unknown scenario, causing the OpenAI plugin to malfunction and fail to send multimodal messages containing images. ### Steps to Reproduce 1. Use the OpenAI plugin (versions 0.0.8 - 0.0.13). 2. Attempt to send a multimodal message with an image (e.g., content including both text and an image). 3. Check the logs and observe the error. ### Error Logs When sending a multimodal message with an image, the following error appears in the logs: ``` api-1 | /app/api/.venv/lib/python3.12/site-packages/pydantic/main.py:390: UserWarning: Pydantic serializer warnings: api-1 | PydanticSerializationUnexpectedValue: Expected `str` but got `list` with value `[TextPromptMessageContent...il=<DETAIL.LOW: 'low'>)]` - serialized value may not be as expected api-1 | PydanticSerializationUnexpectedValue: Expected `PromptMessageContent` but got `TextPromptMessageContent` with value `TextPromptMessageContent(...ata='can you see this?')` - serialized value may not be as expected api-1 | return self.__pydantic_serializer__.to_python( plugin_daemon-1 | [GIN] 2025/03/25 - 07:07:21 | 200 | 9.872525ms | 172.19.0.9 | POST "/plugin/3559813e-c1d2-4fdc-ada2-e05ac28e4f45/dispatch/llm/num_tokens" api-1 | 2025-03-25 07:07:21.159 ERROR [Thread-3 (_generate_worker)] [app_generator.py:246] - Unknown Error when generating api-1 | Traceback (most recent call last): api-1 | File "/app/api/core/app/apps/chat/app_generator.py", line 226, in _generate_worker api-1 | runner.run( api-1 | File "/app/api/core/app/apps/chat/app_runner.py", line 69, in run api-1 | self.get_pre_calculate_rest_tokens( api-1 | File "/app/api/core/app/apps/base_app_runner.py", line 90, in get_pre_calculate_rest_tokens api-1 | prompt_tokens = model_instance.get_llm_num_tokens(prompt_messages) api-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ api-1 | File "/app/api/core/model_manager.py", line 195, in get_llm_num_tokens api-1 | self._round_robin_invoke( api-1 | File "/app/api/core/model_manager.py", line 370, in _round_robin_invoke api-1 | return function(*args, **kwargs) api-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^ api-1 | File "/app/api/core/model_runtime/model_providers/__base/large_language_model.py", line 299, in get_num_tokens api-1 | return plugin_model_manager.get_llm_num_tokens( api-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ api-1 | File "/app/api/core/plugin/manager/model.py", line 231, in get_llm_num_tokens api-1 | for resp in response: api-1 | ^^^^^^^^ api-1 | File "/app/api/core/plugin/manager/base.py", line 189, in _request_with_plugin_daemon_response_stream api-1 | self._handle_plugin_daemon_error(error.error_type, error.message) api-1 | File "/app/api/core/plugin/manager/base.py", line 221, in _handle_plugin_daemon_error api-1 | raise PluginInvokeError(description=message) api-1 | core.plugin.manager.exc.PluginInvokeError: PluginInvokeError: {"args":{},"error_type":"ValidationError","message":"1 validation error for ModelGetLLMNumTokens\nprompt_messages.content.type\n Input should be <PromptMessageContentType.TEXT: 'text'> [type=literal_error, input_value='text', input_type=str]\n For further information visit https://errors.pydantic.dev/2.8/v/literal_error"} api-1 | 2025-03-25 07:07:21.169 ERROR [Dummy-2] [base_app_generate_response_converter.py:123] - {"args":{},"error_type":"ValidationError","message":"1 validation error for ModelGetLLMNumTokens\nprompt_messages.content.type\n Input should be <PromptMessageContentType.TEXT: 'text'> [type=literal_error, input_value='text', input_type=str]\n For further information visit https://errors.pydantic.dev/2.8/v/literal_error"} ``` ### Test Results - Tested the following third-party provider plugins: - **Gemini-0.0.8**: Works normally, no errors. - **Anthropic-0.0.12**: Works normally, no errors. - **OpenAI (0.0.8 - 0.0.13)**: Errors occur, only with the OpenAI plugin. - In **Dify Cloud-Hosted Environment**: - Upgrading the OpenAI plugin to 0.0.13 reproduces the issue. - Rolling back to 0.0.8 resolves the issue, and it works normally. - In **Local Docker-Hosted Environment**: - Even rolling back to 0.0.8 does not resolve the issue; the error persists. ### Analysis - Suspect that changes in the plugin runtime environment are causing the issue. - The problem was initially encountered while developing a new third-party provider plugin: - Works fine in debug mode. - After packaging, the same error occurs. - Reviewed the source code of the OpenAI and Anthropic plugins but found no significant logical differences. - The root cause of the error is unclear and may be related to Pydantic’s serialization validation logic or compatibility issues with the plugin environment. ### Request for Assistance Since the OpenAI plugin is an official implementation and more representative, but the cause of the issue remains elusive, I’m seeking help to identify and resolve the problem. Thank you! ### ✔️ Expected Behavior _No response_ ### ❌ Actual Behavior _No response_
yindo added the 🐞 buggood first issue labels 2026-02-21 19:05:44 -05:00
yindo closed this issue 2026-02-21 19:05:44 -05:00
yindo commented 2026-02-21 19:05:44 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Mar 25, 2025):

The issue you're encountering with the OpenAI plugin failing to send multimodal messages with images seems to be related to Pydantic's serialization and validation logic. The error logs indicate that there are unexpected values during serialization, specifically with the PromptMessageContent type, which is expected to be TEXT but is receiving other types like TextPromptMessageContent.

Here are a few steps you can take to troubleshoot and potentially resolve the issue:

  1. Check Pydantic Model Definitions: Ensure that the Pydantic models used for handling multimodal messages are correctly defined and that the expected types are properly set. The error suggests a mismatch in expected types, so verifying the model definitions might help.

  2. Review Serialization Logic: Look into the serialization logic for multimodal messages in the OpenAI plugin. There might be differences in how the OpenAI plugin handles serialization compared to other plugins like Gemini and Anthropic.

  3. Plugin Version Compatibility: Since the issue persists across different versions of the OpenAI plugin in a local Docker-hosted environment, there might be compatibility issues with the specific environment setup. Consider testing in a different environment or with different Docker configurations.

  4. Pydantic Version: Ensure that the version of Pydantic being used is compatible with the OpenAI plugin. There might be changes in Pydantic's behavior between versions that affect serialization and validation.

  5. Error Handling in Plugin Manager: The Dify platform's plugin manager handles errors by raising specific exceptions based on the error type. Review the _handle_plugin_daemon_error method to understand how errors are being processed and if there are any discrepancies in handling multimodal messages [1].

By focusing on these areas, you might be able to identify the root cause of the issue and implement a solution.

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other

@dosubot[bot] commented on GitHub (Mar 25, 2025): <!-- Answer --> The issue you're encountering with the OpenAI plugin failing to send multimodal messages with images seems to be related to Pydantic's serialization and validation logic. The error logs indicate that there are unexpected values during serialization, specifically with the `PromptMessageContent` type, which is expected to be `TEXT` but is receiving other types like `TextPromptMessageContent`. Here are a few steps you can take to troubleshoot and potentially resolve the issue: 1. **Check Pydantic Model Definitions**: Ensure that the Pydantic models used for handling multimodal messages are correctly defined and that the expected types are properly set. The error suggests a mismatch in expected types, so verifying the model definitions might help. 2. **Review Serialization Logic**: Look into the serialization logic for multimodal messages in the OpenAI plugin. There might be differences in how the OpenAI plugin handles serialization compared to other plugins like Gemini and Anthropic. 3. **Plugin Version Compatibility**: Since the issue persists across different versions of the OpenAI plugin in a local Docker-hosted environment, there might be compatibility issues with the specific environment setup. Consider testing in a different environment or with different Docker configurations. 4. **Pydantic Version**: Ensure that the version of Pydantic being used is compatible with the OpenAI plugin. There might be changes in Pydantic's behavior between versions that affect serialization and validation. 5. **Error Handling in Plugin Manager**: The Dify platform's plugin manager handles errors by raising specific exceptions based on the error type. Review the `_handle_plugin_daemon_error` method to understand how errors are being processed and if there are any discrepancies in handling multimodal messages <sup>[[1]](https://github.com/langgenius/dify/blob/main/api/core/plugin/manager/base.py)</sup>. By focusing on these areas, you might be able to identify the root cause of the issue and implement a solution. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.* --- <sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/475c575d-8292-46f1-8d70-17e503a110ef?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/475c575d-8292-46f1-8d70-17e503a110ef?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/475c575d-8292-46f1-8d70-17e503a110ef?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/475c575d-8292-46f1-8d70-17e503a110ef?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/475c575d-8292-46f1-8d70-17e503a110ef?feedback_type=hallucination) | [Bug Report](https://app.dosu.dev/response-feedback/475c575d-8292-46f1-8d70-17e503a110ef?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/475c575d-8292-46f1-8d70-17e503a110ef?feedback_type=other)</sup>
yindo commented 2026-02-21 19:05:44 -05:00
Author
Owner
Copy Link
Copy Source

@leozanee commented on GitHub (Mar 25, 2025):

This issue seems related to an existing one. #16353

@leozanee commented on GitHub (Mar 25, 2025): This issue seems related to an existing one. #16353
yindo commented 2026-02-21 19:05:44 -05:00
Author
Owner
Copy Link
Copy Source

@leozanee commented on GitHub (Mar 25, 2025):

very hard to fix, because error just shown when plugin packaged

@leozanee commented on GitHub (Mar 25, 2025): very hard to fix, because error just shown when plugin packaged
yindo commented 2026-02-21 19:05:45 -05:00
Author
Owner
Copy Link
Copy Source

@Ken-Mok commented on GitHub (Mar 25, 2025):

have the same issue when using Azure OpenAI 0.0.10

@Ken-Mok commented on GitHub (Mar 25, 2025): have the same issue when using Azure OpenAI 0.0.10
yindo commented 2026-02-21 19:05:45 -05:00
Author
Owner
Copy Link
Copy Source

@xingxi521 commented on GitHub (Mar 26, 2025):

I'm waiting for the repair to be completed.

@xingxi521 commented on GitHub (Mar 26, 2025): I'm waiting for the repair to be completed.
yindo commented 2026-02-21 19:05:45 -05:00
Author
Owner
Copy Link
Copy Source

@lichao4Java commented on GitHub (Mar 26, 2025):

https://github.com/langgenius/dify/issues/16816

same issues with me .

I'm waiting for the repair to be completed.

@lichao4Java commented on GitHub (Mar 26, 2025): https://github.com/langgenius/dify/issues/16816 same issues with me . I'm waiting for the repair to be completed.
yindo commented 2026-02-21 19:05:45 -05:00
Author
Owner
Copy Link
Copy Source

@lichao4Java commented on GitHub (Mar 26, 2025):

@leozanee Is there any progress？

@lichao4Java commented on GitHub (Mar 26, 2025): @leozanee Is there any progress？
yindo commented 2026-02-21 19:05:45 -05:00
Author
Owner
Copy Link
Copy Source

@kurokobo commented on GitHub (Mar 26, 2025):

https://github.com/langgenius/dify/issues/16353#issuecomment-2754785232

@kurokobo commented on GitHub (Mar 26, 2025): https://github.com/langgenius/dify/issues/16353#issuecomment-2754785232
yindo commented 2026-02-21 19:05:45 -05:00
Author
Owner
Copy Link
Copy Source

@leozanee commented on GitHub (Mar 26, 2025):

Issue Report

Title: plugin_daemon: ValidationError (Enum vs String) on /llm/invoke with Multimodal prompt_messages

Labels: bug, plugin, multimodal, validation

Environment:

  • Dify Version: 1.1.3
  • Deployment: Self Hosted (Docker)
  • Plugin: Official OpenAI Plugin (0.0.13)
  • Issue Context: Error occurs when sending multimodal messages (text + image) to a chat application using a plugin-based LLM.

Description:

Hi! First off, a huge thank you to the Dify team and community maintainers for all your amazing work—I know you’re super busy keeping everything running smoothly.

I’ve been wrestling with this issue for a few days and could really use some guidance, even just a small nudge, so I can contribute back to fixing it.

Here’s what I’ve found so far after some trial and error:

when using a chat application with a plugin model—tested with OpenAI’s vision-capable models—sending a message with both text and an image triggers a ValidationError in the plugin_daemon service during the /dispatch/llm/invoke call. It seems tied to a type mismatch in Pydantic validation, specifically around the type field in the prompt_messages.content list. I’d love to hear your thoughts!

Steps to Reproduce:

  1. Deploy Dify in Docker host mode.
  2. Set up the official OpenAI plugin with valid credentials.
  3. Create a Dify Chat Application.
  4. Configure it to use an OpenAI vision model (e.g., gpt-4o-mini) via the plugin openai.
  5. Send a message with text (e.g., "describe this image") and an uploaded image.
  6. Check the api-1 container logs.

Expected Behavior:

The multimodal message processes smoothly, the /llm/invoke call to plugin_daemon works, and the plugin model returns a response.

Actual Behavior:

The chat fails to generate a response. In the api-1 logs, I see a core.plugin.manager.exc.PluginInvokeError wrapping a ValidationError from plugin_daemon:

core.plugin.manager.exc.PluginInvokeError: PluginInvokeError: {"args":{},"error_type":"ValidationError","message":"1 validation error for ModelInvokeLLMRequest\nprompt_messages.content.type\n  Input should be \u003cPromptMessageContentType.TEXT: 'text'\u003e [type=literal_error, input_value='text', input_type=str]\n    For further information visit https://errors.pydantic.dev/2.8/v/literal_error"}

Interestingly, this happens after the /dispatch/llm/invoke endpoint returns an HTTP 200 OK, so it feels like the issue kicks in during the daemon’s internal parsing or validation.

What I’ve Observed:

From my debugging, it looks like the problem ties back to how plugin_daemon handles the JSON payload for /llm/invoke:

  1. API Side Looks Good: The Dify API (api/core/plugin/manager/model.py) seems to serialize prompt_messages correctly into JSON, with enums like PromptMessageContentType.TEXT sent as strings (e.g., "type": "text"). I logged the payload before it’s sent, and it checks out.
  2. Daemon Trips Up: Even with a valid JSON payload, plugin_daemon throws the error, expecting an Enum object (<PromptMessageContentType.TEXT: 'text'>) instead of the string "text" it receives.
  3. Possible Culprit: I suspect this is tied to Pydantic model definitions in the dify_plugin package—maybe something like dify_plugin.entities.model.message.PromptMessageContent or dify_plugin.core.entities.plugin.request.ModelInvokeLLMRequest—where the type field isn’t set up to handle string inputs from JSON properly.

Troubleshooting Journey:

I’ve tried a few angles to narrow this down, and here’s where I’ve landed:

  1. First Clue (/llm/num_tokens): I initially hit this ValidationError on /llm/num_tokens. Tweaking the API (api/core/app/apps/base_app_runner.py) to send simplified, text-only prompt_messages for those calls fixed it there—proof the daemon can handle requests if I sidestep the tricky structure.
  2. Shift to /llm/invoke: After that fix, the error popped up in /llm/invoke, which needs to send the full multimodal prompt_messages.
  3. Payload Tweaks: I adjusted the API (api/core/plugin/manager/model.py) to ensure /invoke sends proper string enums (e.g., "type": "text", "type": "image"), but the error stuck around.
  4. Plugin-Side Efforts: I played with a custom plugin’s data handling (llm.py), but no dice—the error hits before the plugin code even runs.
  5. Poking dify_plugin: I even hacked around in the plugin_daemon container (via Docker exec/cp), tweaking Pydantic validators in dify_plugin files like message.py and request.py. I tried forcing string acceptance or converting strings to enums, but I couldn’t pin down the root cause and kept hitting the same wall.

Where I’m At:

After multiple rounds of digging, I’m stumped on pinpointing the exact spot in dify_plugin causing this validation hiccup. It feels like a mismatch between how the JSON strings are sent and what the daemon’s Pydantic models expect, but my attempts to fix it locally haven’t cracked it. One extra observation: I’ve noticed that even plugins from vendors that don’t seem to have this issue can still trigger warnings in the logs (another puzzling thing—why do some plugin implementations only log warnings instead of errors?). This makes me wonder if the problem might have a broader impact beyond just my setup.

I’d really appreciate any fresh eyes on this—maybe someone familiar with the plugin_daemon or dify_plugin internals could point me in the right direction? I’m happy to provide more logs or details if that helps! Looking forward to sorting this out together—thanks again for taking a look!

@leozanee commented on GitHub (Mar 26, 2025): ## Issue Report **Title:** `plugin_daemon`: `ValidationError` (Enum vs String) on `/llm/invoke` with Multimodal `prompt_messages` **Labels:** `bug`, `plugin`, `multimodal`, `validation` **Environment:** * **Dify Version:** 1.1.3 * **Deployment:** Self Hosted (Docker) * **Plugin:** Official OpenAI Plugin (0.0.13) * **Issue Context:** Error occurs when sending multimodal messages (text + image) to a chat application using a plugin-based LLM. **Description:** Hi! First off, a huge thank you to the Dify team and community maintainers for all your amazing work—I know you’re super busy keeping everything running smoothly. I’ve been wrestling with this issue for a few days and could really use some guidance, even just a small nudge, so I can contribute back to fixing it. **Here’s what I’ve found so far after some trial and error:** when using a chat application with a plugin model—tested with OpenAI’s vision-capable models—sending a message with both text and an image triggers a `ValidationError` in the `plugin_daemon` service during the `/dispatch/llm/invoke` call. It seems tied to a type mismatch in Pydantic validation, specifically around the `type` field in the `prompt_messages.content` list. I’d love to hear your thoughts! **Steps to Reproduce:** 1. Deploy Dify in Docker host mode. 2. Set up the official OpenAI plugin with valid credentials. 3. Create a Dify Chat Application. 4. Configure it to use an OpenAI vision model (e.g., `gpt-4o-mini`) via the plugin `openai`. 5. Send a message with text (e.g., "describe this image") and an uploaded image. 6. Check the `api-1` container logs. **Expected Behavior:** The multimodal message processes smoothly, the `/llm/invoke` call to `plugin_daemon` works, and the plugin model returns a response. **Actual Behavior:** The chat fails to generate a response. In the `api-1` logs, I see a `core.plugin.manager.exc.PluginInvokeError` wrapping a `ValidationError` from `plugin_daemon`: ``` core.plugin.manager.exc.PluginInvokeError: PluginInvokeError: {"args":{},"error_type":"ValidationError","message":"1 validation error for ModelInvokeLLMRequest\nprompt_messages.content.type\n Input should be \u003cPromptMessageContentType.TEXT: 'text'\u003e [type=literal_error, input_value='text', input_type=str]\n For further information visit https://errors.pydantic.dev/2.8/v/literal_error"} ``` Interestingly, this happens *after* the `/dispatch/llm/invoke` endpoint returns an HTTP `200 OK`, so it feels like the issue kicks in during the daemon’s internal parsing or validation. **What I’ve Observed:** From my debugging, it looks like the problem ties back to how `plugin_daemon` handles the JSON payload for `/llm/invoke`: 1. **API Side Looks Good:** The Dify API (`api/core/plugin/manager/model.py`) seems to serialize `prompt_messages` correctly into JSON, with enums like `PromptMessageContentType.TEXT` sent as strings (e.g., `"type": "text"`). I logged the payload before it’s sent, and it checks out. 2. **Daemon Trips Up:** Even with a valid JSON payload, `plugin_daemon` throws the error, expecting an Enum object (`<PromptMessageContentType.TEXT: 'text'>`) instead of the string `"text"` it receives. 3. **Possible Culprit:** I suspect this is tied to Pydantic model definitions in the `dify_plugin` package—maybe something like `dify_plugin.entities.model.message.PromptMessageContent` or `dify_plugin.core.entities.plugin.request.ModelInvokeLLMRequest`—where the `type` field isn’t set up to handle string inputs from JSON properly. **Troubleshooting Journey:** I’ve tried a few angles to narrow this down, and here’s where I’ve landed: 1. **First Clue (`/llm/num_tokens`):** I initially hit this `ValidationError` on `/llm/num_tokens`. Tweaking the API (`api/core/app/apps/base_app_runner.py`) to send simplified, text-only `prompt_messages` for those calls fixed it there—proof the daemon can handle requests if I sidestep the tricky structure. 2. **Shift to `/llm/invoke`:** After that fix, the error popped up in `/llm/invoke`, which needs to send the full multimodal `prompt_messages`. 3. **Payload Tweaks:** I adjusted the API (`api/core/plugin/manager/model.py`) to ensure `/invoke` sends proper string enums (e.g., `"type": "text"`, `"type": "image"`), but the error stuck around. 4. **Plugin-Side Efforts:** I played with a custom plugin’s data handling (`llm.py`), but no dice—the error hits before the plugin code even runs. 5. **Poking `dify_plugin`:** I even hacked around in the `plugin_daemon` container (via Docker exec/cp), tweaking Pydantic validators in `dify_plugin` files like `message.py` and `request.py`. I tried forcing string acceptance or converting strings to enums, but I couldn’t pin down the root cause and kept hitting the same wall. **Where I’m At:** After multiple rounds of digging, I’m stumped on pinpointing the exact spot in `dify_plugin` causing this validation hiccup. It feels like a mismatch between how the JSON strings are sent and what the daemon’s Pydantic models expect, but my attempts to fix it locally haven’t cracked it. One extra observation: I’ve noticed that even plugins from vendors that don’t seem to have this issue can still trigger warnings in the logs (another puzzling thing—why do some plugin implementations only log warnings instead of errors?). This makes me wonder if the problem might have a broader impact beyond just my setup. I’d really appreciate any fresh eyes on this—maybe someone familiar with the `plugin_daemon` or `dify_plugin` internals could point me in the right direction? I’m happy to provide more logs or details if that helps! Looking forward to sorting this out together—thanks again for taking a look!
yindo commented 2026-02-21 19:05:45 -05:00
Author
Owner
Copy Link
Copy Source

@kurokobo commented on GitHub (Mar 26, 2025):

Temporary workarounds have been released. For those facing this issue, upgrade your plugin and give it another try :)

  • Azure OpenAI: >= 0.0.11
  • OpenAI: >= 0.0.14
  • OpenRouter: >=0.0.6
@kurokobo commented on GitHub (Mar 26, 2025): Temporary workarounds have been released. For those facing this issue, upgrade your plugin and give it another try :) - Azure OpenAI: **>= 0.0.11** - OpenAI: **>= 0.0.14** - OpenRouter: **>=0.0.6**
yindo commented 2026-02-21 19:05:45 -05:00
Author
Owner
Copy Link
Copy Source

@xingxi521 commented on GitHub (Mar 26, 2025):

What do I need to do to upgrade the plugin when I am using docker locally deployed @kurokobo

@xingxi521 commented on GitHub (Mar 26, 2025): What do I need to do to upgrade the plugin when I am using docker locally deployed @kurokobo
yindo commented 2026-02-21 19:05:45 -05:00
Author
Owner
Copy Link
Copy Source

@kurokobo commented on GitHub (Mar 26, 2025):

@xingxi521
https://docs.dify.ai/plugins/manage-plugins#upgrading-plugins

@kurokobo commented on GitHub (Mar 26, 2025): @xingxi521 https://docs.dify.ai/plugins/manage-plugins#upgrading-plugins
yindo commented 2026-02-21 19:05:46 -05:00
Author
Owner
Copy Link
Copy Source

@xingxi521 commented on GitHub (Mar 26, 2025):

Thank you very much

@xingxi521 commented on GitHub (Mar 26, 2025): Thank you very much
yindo commented 2026-02-21 19:05:46 -05:00
Author
Owner
Copy Link
Copy Source

@natsu12 commented on GitHub (Mar 26, 2025):

@kurokobo The latest version of OpenAI plugin is still 0.0.13

Image

@natsu12 commented on GitHub (Mar 26, 2025): @kurokobo The latest version of OpenAI plugin is still 0.0.13 ![Image](https://github.com/user-attachments/assets/39b9280b-97aa-459c-9cf0-681acda0beb1)
yindo commented 2026-02-21 19:05:46 -05:00
Author
Owner
Copy Link
Copy Source

@leozanee commented on GitHub (Mar 27, 2025):

thanks to @laipz8200 fixed it in sdk https://github.com/langgenius/dify-plugin-sdks/pull/73#issue-2951260715

seems the problem fixed , so this issue closed

@leozanee commented on GitHub (Mar 27, 2025): thanks to @laipz8200 fixed it in sdk https://github.com/langgenius/dify-plugin-sdks/pull/73#issue-2951260715 seems the problem fixed , so this issue closed
yindo commented 2026-02-21 19:05:46 -05:00
Author
Owner
Copy Link
Copy Source

@lichao4Java commented on GitHub (Mar 27, 2025):

Temporary workarounds have been released. For those facing this issue, upgrade your plugin and give it another try :)

  • Azure OpenAI: >= 0.0.11
  • OpenAI: >= 0.0.14
  • OpenRouter: >=0.0.6

The latest version of OpenAI plugin is still 0.0.13

@lichao4Java commented on GitHub (Mar 27, 2025): > Temporary workarounds have been released. For those facing this issue, upgrade your plugin and give it another try :) > > * Azure OpenAI: **>= 0.0.11** > * OpenAI: **>= 0.0.14** > * OpenRouter: **>=0.0.6** The latest version of OpenAI plugin is still 0.0.13
yindo commented 2026-02-21 19:05:46 -05:00
Author
Owner
Copy Link
Copy Source

@kurokobo commented on GitHub (Mar 27, 2025):

I am asking the Dfy team here https://github.com/langgenius/dify/issues/16816#issuecomment-2758219720

@kurokobo commented on GitHub (Mar 27, 2025): I am asking the Dfy team here https://github.com/langgenius/dify/issues/16816#issuecomment-2758219720
yindo commented 2026-02-21 19:05:46 -05:00
Author
Owner
Copy Link
Copy Source

@kurokobo commented on GitHub (Mar 27, 2025):

@lichao4Java
Now it's available on the marketplace 😃

@kurokobo commented on GitHub (Mar 27, 2025): @lichao4Java Now it's available on the marketplace 😃
yindo commented 2026-02-21 19:05:46 -05:00
Author
Owner
Copy Link
Copy Source

@lichao4Java commented on GitHub (Mar 28, 2025):

thanks..

发件人: kurokobo @.>
发送时间: 2025年3月28日 3:09
收件人: langgenius/dify @.>
抄送: chris_lee @.>; Mention @.>
主题: Re: [langgenius/dify] OpenAI Plugin Fails to Send Multimodal Messages with Images (Issue #16726)

@lichao4Javahttps://github.com/lichao4Java
Now it's available on the marketplace 😃

—
Reply to this email directly, view it on GitHubhttps://github.com/langgenius/dify/issues/16726#issuecomment-2760063264, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ADE6BEWDIVEXUVJ6QCESBGD2WS4QPAVCNFSM6AAAAABZWZ5TASVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDONRQGA3DGMRWGQ.
You are receiving this because you were mentioned.Message ID: @.***>

[kurokobo]kurokobo left a comment (langgenius/dify#16726)https://github.com/langgenius/dify/issues/16726#issuecomment-2760063264

@lichao4Javahttps://github.com/lichao4Java
Now it's available on the marketplace 😃

—
Reply to this email directly, view it on GitHubhttps://github.com/langgenius/dify/issues/16726#issuecomment-2760063264, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ADE6BEWDIVEXUVJ6QCESBGD2WS4QPAVCNFSM6AAAAABZWZ5TASVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDONRQGA3DGMRWGQ.
You are receiving this because you were mentioned.Message ID: @.***>

@lichao4Java commented on GitHub (Mar 28, 2025): thanks.. ________________________________ 发件人: kurokobo ***@***.***> 发送时间: 2025年3月28日 3:09 收件人: langgenius/dify ***@***.***> 抄送: chris_lee ***@***.***>; Mention ***@***.***> 主题: Re: [langgenius/dify] OpenAI Plugin Fails to Send Multimodal Messages with Images (Issue #16726) @lichao4Java<https://github.com/lichao4Java> Now it's available on the marketplace 😃 — Reply to this email directly, view it on GitHub<https://github.com/langgenius/dify/issues/16726#issuecomment-2760063264>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ADE6BEWDIVEXUVJ6QCESBGD2WS4QPAVCNFSM6AAAAABZWZ5TASVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDONRQGA3DGMRWGQ>. You are receiving this because you were mentioned.Message ID: ***@***.***> [kurokobo]kurokobo left a comment (langgenius/dify#16726)<https://github.com/langgenius/dify/issues/16726#issuecomment-2760063264> @lichao4Java<https://github.com/lichao4Java> Now it's available on the marketplace 😃 — Reply to this email directly, view it on GitHub<https://github.com/langgenius/dify/issues/16726#issuecomment-2760063264>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ADE6BEWDIVEXUVJ6QCESBGD2WS4QPAVCNFSM6AAAAABZWZ5TASVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDONRQGA3DGMRWGQ>. You are receiving this because you were mentioned.Message ID: ***@***.***>
yindo commented 2026-02-21 19:05:46 -05:00
Author
Owner
Copy Link
Copy Source

@lichao4Java commented on GitHub (Apr 3, 2025):

when i upgrade openai openrouter plugin. and dify was crashed ....
look here
https://github.com/langgenius/dify/issues/17384#issuecomment-2775165109

@kurokobo

@lichao4Java commented on GitHub (Apr 3, 2025): when i upgrade openai openrouter plugin. and dify was crashed .... look here https://github.com/langgenius/dify/issues/17384#issuecomment-2775165109 @kurokobo
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label good first issue 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#12095
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?