Possible workaround for the issues with multimodal... #38

New Issue

2026-02-15T16:27:38-05:00

yindo commented

2026-02-15 16:27:38 -05:00

Originally created by @joakimeriksson on GitHub (Feb 26, 2024).

Multimodal request seems to currently ignore the initial prompt - the multimodal example in this repo is not really working if you change the prompt. This is a changed version that seems to work - but includes a workaround:

import sys
import random
import httpx
from ollama import chat


latest = httpx.get('https://xkcd.com/info.0.json')
latest.raise_for_status()
num = random.randint(1, latest.json().get('num'))
comic = httpx.get(f'https://xkcd.com/{num}/info.0.json')
comic.raise_for_status()

print(f'xkcd #{comic.json().get("num")}: {comic.json().get("alt")}')
print(f'link: https://xkcd.com/{num}')
print('---')

raw = httpx.get(comic.json().get('img'))
raw.raise_for_status()

res = chat('llava', messages = [
  {			'role': 'user',
        	        'images': [raw.content],
			'content': ' '
   },
  {			'role': 'user',
        	        'images': [],
			'content': ' what is to the left in the image?'
  }
], stream=False)
print(res['message']['content'])

Originally created by @joakimeriksson on GitHub (Feb 26, 2024). Multimodal request seems to currently ignore the initial prompt - the multimodal example in this repo is not really working if you change the prompt. This is a changed version that seems to work - but includes a workaround: ```python import sys import random import httpx from ollama import chat latest = httpx.get('https://xkcd.com/info.0.json') latest.raise_for_status() num = random.randint(1, latest.json().get('num')) comic = httpx.get(f'https://xkcd.com/{num}/info.0.json') comic.raise_for_status() print(f'xkcd #{comic.json().get("num")}: {comic.json().get("alt")}') print(f'link: https://xkcd.com/{num}') print('---') raw = httpx.get(comic.json().get('img')) raw.raise_for_status() res = chat('llava', messages = [ { 'role': 'user', 'images': [raw.content], 'content': ' ' }, { 'role': 'user', 'images': [], 'content': ' what is to the left in the image?' } ], stream=False) print(res['message']['content']) ```

yindo closed this issue

2026-02-15 16:27:38 -05:00

yindo commented

2026-02-15 16:27:38 -05:00

@eliranwong commented on GitHub (Feb 28, 2024):

Thanks, I also noted aout the ignorance, but haven't thought about this workaround

@eliranwong commented on GitHub (Feb 28, 2024): Thanks, I also noted aout the ignorance, but haven't thought about this workaround

yindo commented

2026-02-15 16:27:38 -05:00

@mxyng commented on GitHub (Mar 6, 2024):

This should be fixed upstream in ollama by prepending the image tag rather than appending. This should be fixed by https://github.com/ollama/ollama/pull/2789 which is released in 0.1.28

@mxyng commented on GitHub (Mar 6, 2024): This should be fixed upstream in ollama by prepending the image tag rather than appending. This should be fixed by https://github.com/ollama/ollama/pull/2789 which is released in 0.1.28

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: ollama/ollama-python#38