langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-22 12:36:14 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Ollama vision is not working #5491

New Issue
Closed
opened 2026-02-21 18:11:13 -05:00 by yindo · 8 comments
yindo commented 2026-02-21 18:11:13 -05:00
Owner
Copy Link
Copy Source

Originally created by @ruzzzz6312 on GitHub (Sep 9, 2024).

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
  • Please do not modify this template :) and fill in all the required fields.

Dify version

0.7.0

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

Add Ollama "Llava" model (visual)

✔️ Expected Behavior

To be able to chat about the image

❌ Actual Behavior

2 options, either:
Screenshot 2024-09-09 at 3 22 45 PM
[ollama] Error: API request failed with status code 400: {"error":"illegal base64 data at input byte 5"}

Or:
Screenshot_2024-09-06_at_6 09 13_PM

Query or prefix prompt is too long, you can reduce the prefix prompt, or shrink the max token, or switch to a lim with a larger token limit size.

Originally created by @ruzzzz6312 on GitHub (Sep 9, 2024). ### Self Checks - [X] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [X] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [X] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [X] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:） - [X] Please do not modify this template :) and fill in all the required fields. ### Dify version 0.7.0 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce Add Ollama "Llava" model (visual) ### ✔️ Expected Behavior To be able to chat about the image ### ❌ Actual Behavior 2 options, either: <img width="567" alt="Screenshot 2024-09-09 at 3 22 45 PM" src="https://github.com/user-attachments/assets/0fdba97c-68b4-4bdf-b588-a9ecea4dbb27"> [ollama] Error: API request failed with status code 400: {"error":"illegal base64 data at input byte 5"} Or: <img width="1884" alt="Screenshot_2024-09-06_at_6 09 13_PM" src="https://github.com/user-attachments/assets/55dc5830-6a71-4cc8-bf12-537979630dc4"> Query or prefix prompt is too long, you can reduce the prefix prompt, or shrink the max token, or switch to a lim with a larger token limit size.
yindo added the 🐞 bug label 2026-02-21 18:11:13 -05:00
yindo closed this issue 2026-02-21 18:11:13 -05:00
yindo commented 2026-02-21 18:11:14 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Sep 9, 2024):

The errors you're encountering with the Ollama "Llava" model in Dify version 0.7.0 when self-hosted on Docker are likely due to the Ollama service not being accessible from the Docker container. This is because localhost usually refers to the container itself, not the host machine or other containers.

To resolve this issue, you need to expose the Ollama service to the network. You can do this by replacing localhost with host.docker.internal in the service URL:

http://host.docker.internal:11434

Alternatively, you can use the local network IP address of the host machine. For example:

http://192.168.1.100:11434

or the Docker host IP address:

http://172.17.0.1:11434

If you are running Ollama on macOS, you may need to set the environment variable using launchctl:

launchctl setenv OLLAMA_HOST "0.0.0.0"

For Linux, if Ollama is running as a systemd service, you should edit the service configuration:

[Service]
Environment="OLLAMA_HOST=0.0.0.0"

Then reload systemd and restart Ollama:

systemctl daemon-reload
systemctl restart ollama

For Windows, set the environment variable through the system environment variables settings and restart Ollama.

Additionally, ensure that the model configuration in Dify is set correctly. For example, when configuring the model in Dify, you should input the accessible Ollama service address under 設定 > モデル供給元 > Ollama:

  • モデル名称：llava
  • 基本 URL：http://<your-ollama-endpoint-domain>:11434
  • モデルタイプ：対話
  • モデルのコンテキスト長：4096
  • 最大トークン上限：4096
  • ビジョン対応：はい

Make sure to replace <your-ollama-endpoint-domain> with the appropriate network address. This setup should help in resolving the errors and warnings you are encountering [1][2][3].

To continue talking to Dosu, mention @dosu.

@dosubot[bot] commented on GitHub (Sep 9, 2024): <!-- Answer --> The errors you're encountering with the Ollama "Llava" model in Dify version 0.7.0 when self-hosted on Docker are likely due to the Ollama service not being accessible from the Docker container. This is because `localhost` usually refers to the container itself, not the host machine or other containers. To resolve this issue, you need to expose the Ollama service to the network. You can do this by replacing `localhost` with `host.docker.internal` in the service URL: ```bash http://host.docker.internal:11434 ``` Alternatively, you can use the local network IP address of the host machine. For example: ```bash http://192.168.1.100:11434 ``` or the Docker host IP address: ```bash http://172.17.0.1:11434 ``` If you are running Ollama on macOS, you may need to set the environment variable using `launchctl`: ```bash launchctl setenv OLLAMA_HOST "0.0.0.0" ``` For Linux, if Ollama is running as a systemd service, you should edit the service configuration: ```ini [Service] Environment="OLLAMA_HOST=0.0.0.0" ``` Then reload systemd and restart Ollama: ```bash systemctl daemon-reload systemctl restart ollama ``` For Windows, set the environment variable through the system environment variables settings and restart Ollama. Additionally, ensure that the model configuration in Dify is set correctly. For example, when configuring the model in Dify, you should input the accessible Ollama service address under `設定 > モデル供給元 > Ollama`: - モデル名称：`llava` - 基本 URL：`http://<your-ollama-endpoint-domain>:11434` - モデルタイプ：`対話` - モデルのコンテキスト長：`4096` - 最大トークン上限：`4096` - ビジョン対応：`はい` Make sure to replace `<your-ollama-endpoint-domain>` with the appropriate network address. This setup should help in resolving the errors and warnings you are encountering <sup>[[1]](https://github.com/langgenius/dify-docs/blob/main/en/guides/model-configuration/ollama.md)</sup><sup>[[2]](https://github.com/langgenius/dify-docs/blob/main/zh_CN/guides/model-configuration/ollama.md)</sup><sup>[[3]](https://github.com/langgenius/dify-docs/blob/main/jp/guides/model-configuration/ollama.md)</sup>. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.*
yindo commented 2026-02-21 18:11:14 -05:00
Author
Owner
Copy Link
Copy Source

@ruzzzz6312 commented on GitHub (Sep 9, 2024):

Ollama IS ACCESSIBLE, text / chat works just fine

@ruzzzz6312 commented on GitHub (Sep 9, 2024): Ollama IS ACCESSIBLE, text / chat works just fine
yindo commented 2026-02-21 18:11:14 -05:00
Author
Owner
Copy Link
Copy Source

@ruzzzz6312 commented on GitHub (Sep 9, 2024):

model is 'LLAVA:13B' and 'LLAVA:34B'

@ruzzzz6312 commented on GitHub (Sep 9, 2024): model is 'LLAVA:13B' and 'LLAVA:34B'
yindo commented 2026-02-21 18:11:14 -05:00
Author
Owner
Copy Link
Copy Source

@crazywoola commented on GitHub (Sep 9, 2024):

This is because tokens will be calculated as tokens as well, so a image should be base64 strings and filled into the prompt which might lead to this error query or prefix is too long. You can reduce the image's size to make it pass.

@crazywoola commented on GitHub (Sep 9, 2024): This is because tokens will be calculated as tokens as well, so a image should be base64 strings and filled into the prompt which might lead to this error `query or prefix is too long`. You can reduce the image's size to make it pass.
yindo commented 2026-02-21 18:11:14 -05:00
Author
Owner
Copy Link
Copy Source

@ruzzzz6312 commented on GitHub (Sep 9, 2024):

so I have 2 issues, both never appear when working on the same workflow with GPT-4o, so the problem is related to ollama's module (I guess)

This is because tokens will be calculated as tokens as well, so a image should be base64 strings and filled into the prompt which might lead to this error query or prefix is too long. You can reduce the image's size to make it pass.

@ruzzzz6312 commented on GitHub (Sep 9, 2024): so I have 2 issues, both never appear when working on the same workflow with GPT-4o, so the problem is related to ollama's module (I guess) > This is because tokens will be calculated as tokens as well, so a image should be base64 strings and filled into the prompt which might lead to this error `query or prefix is too long`. You can reduce the image's size to make it pass.
yindo commented 2026-02-21 18:11:14 -05:00
Author
Owner
Copy Link
Copy Source

@crazywoola commented on GitHub (Sep 9, 2024):

The vision function in ollama is not quite same as the gpt-4o, gpt-4o supports adding files by a parameter while the ollama does not, so it will be translated as base64 string.

@crazywoola commented on GitHub (Sep 9, 2024): The vision function in ollama is not quite same as the gpt-4o, gpt-4o supports adding files by a parameter while the ollama does not, so it will be translated as base64 string.
yindo commented 2026-02-21 18:11:14 -05:00
Author
Owner
Copy Link
Copy Source

@ruzzzz6312 commented on GitHub (Sep 9, 2024):

sorry guys, why completed? Error is still there, any workaround to use vision with ollama?

@ruzzzz6312 commented on GitHub (Sep 9, 2024): sorry guys, why completed? Error is still there, any workaround to use vision with ollama?
yindo commented 2026-02-21 18:11:14 -05:00
Author
Owner
Copy Link
Copy Source

@kwiksher commented on GitHub (Sep 11, 2024):

ollama with image_url fails. Paste image link fails "https://cloud.dify.ai/logo/logo-site.png"

image

but Upload from computer works.

@kwiksher commented on GitHub (Sep 11, 2024): ollama with image_url fails. Paste image link fails "https://cloud.dify.ai/logo/logo-site.png" ![image](https://github.com/user-attachments/assets/f20c87bb-a666-40e4-8411-d95fab6333a1) but Upload from computer works.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#5491
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?