How to send image to vision model #144

New Issue

2026-02-15T16:28:16-05:00

yindo commented

2026-02-15 16:28:16 -05:00

Originally created by @engdante on GitHub (Sep 17, 2024).

Please tell me the correct way to send an image to the vision model.

this is my function:

def generate_image_description(image_path):
prompt = f"Describe the content of this image: {image_path}."
response = client.chat(model='llava-phi3:3.8b', messages=[
{
'role': 'user',
'content': prompt,
},
])
return response['message']['content']

Originally created by @engdante on GitHub (Sep 17, 2024). Please tell me the correct way to send an image to the vision model. this is my function: def generate_image_description(image_path): prompt = f"Describe the content of this image: **{image_path}**." response = client.chat(model=**'llava-phi3:3.8b**', messages=[ { 'role': 'user', 'content': prompt, }, ]) return response['message']['content']

yindo closed this issue

2026-02-15 16:28:16 -05:00

yindo commented

2026-02-15 16:28:17 -05:00

@karanravindra commented on GitHub (Nov 5, 2024):

Please refer to the definition of a "chat message" in the python code Message Type Dict.

The image can be passed in using the "images" key in your message dictionary. The "images" key is a sequence of "bytes" or "path-like str".

Here is an example:

import ollama

response = ollama.chat(
    model="moondream",
    messages=[
        {"role": "user", "content": "Describe the image", "images": ["./cat.jpeg"]}
    ],
)

print(response["message"]['role'])
print(response["message"]['content'])

@karanravindra commented on GitHub (Nov 5, 2024): Please refer to the definition of a "chat message" in the python code [Message Type Dict](https://github.com/ollama/ollama-python/blob/ebe332b29d5c65aeccfadd4151bf6059ded7049b/ollama/_types.py#L77C7-L77C14). The image can be passed in using the "images" key in your message dictionary. The "images" key is a sequence of "bytes" or "path-like str". Here is an example: ```python import ollama response = ollama.chat( model="moondream", messages=[ {"role": "user", "content": "Describe the image", "images": ["./cat.jpeg"]} ], ) print(response["message"]['role']) print(response["message"]['content']) ```

yindo commented

2026-02-15 16:28:18 -05:00

@pnmartinez commented on GitHub (Mar 29, 2025):

Just my grain of salt for future searches: for gemma3, this is what worked for me using the Python ollama lib:

for img_path in paths:
  try:
    # Read the image file as binary data
    with open(img_path, 'rb') as img_file:
        img_data = img_file.read()
    
    # Convert image to base64 for Ollama
    img_base64 = base64.b64encode(img_data).decode('utf-8')
    
    # Use the correct approach as per Gemma 3 documentation - images at top level
    response = ollama.generate(
        model='gemma3:12b',
        prompt="What's this? Provide a description without leading or trailing text.",
        images=[img_base64],  # Pass base64 encoded image data at top level
        options={"temperature": 0.1}  # Lower temperature for more consistent output
    )
    
    # Extract the caption from the response
    caption = response['response'].strip()

@pnmartinez commented on GitHub (Mar 29, 2025): Just my grain of salt for future searches: for gemma3, this is what worked for me using the Python `ollama` lib: ```py for img_path in paths: try: # Read the image file as binary data with open(img_path, 'rb') as img_file: img_data = img_file.read() # Convert image to base64 for Ollama img_base64 = base64.b64encode(img_data).decode('utf-8') # Use the correct approach as per Gemma 3 documentation - images at top level response = ollama.generate( model='gemma3:12b', prompt="What's this? Provide a description without leading or trailing text.", images=[img_base64], # Pass base64 encoded image data at top level options={"temperature": 0.1} # Lower temperature for more consistent output ) # Extract the caption from the response caption = response['response'].strip()

yindo commented

2026-02-15 16:28:18 -05:00

@myyim commented on GitHub (Apr 1, 2025):

This works for me: https://medium.com/p/bb4696663701

@myyim commented on GitHub (Apr 1, 2025): This works for me: https://medium.com/p/bb4696663701

yindo commented

2026-02-15 16:28:18 -05:00

@ParthSareen commented on GitHub (Apr 1, 2025):

@pnmartinez @myyim does the example https://github.com/ollama/ollama-python/blob/main/examples/multimodal-chat.py not work for you guys?

@ParthSareen commented on GitHub (Apr 1, 2025): @pnmartinez @myyim does the example https://github.com/ollama/ollama-python/blob/main/examples/multimodal-chat.py not work for you guys?

yindo commented

2026-02-15 16:28:18 -05:00

@garricklw commented on GitHub (Apr 7, 2025):

Just found out in my own testing, if you want to call it against the Ollama object's chat function, you need to pass it as an "ImageDocument"

llm = Ollama(model="gemma3", request_timeout=360.0)
response = llm.chat([ChatMessage("What's this? Provide a description without leading or trailing text.", additional_kwargs={"images": [ImageDocument(image_path=image_path)]})])

@garricklw commented on GitHub (Apr 7, 2025): Just found out in my own testing, if you want to call it against the Ollama object's chat function, you need to pass it as an "ImageDocument" `llm = Ollama(model="gemma3", request_timeout=360.0)` `response = llm.chat([ChatMessage("What's this? Provide a description without leading or trailing text.", additional_kwargs={"images": [ImageDocument(image_path=image_path)]})])`

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: ollama/ollama-python#144