A few questions about vision models (multimodal) #192

New Issue

2026-02-15T16:28:35-05:00

yindo commented

2026-02-15 16:28:35 -05:00

Originally created by @DanilZittser on GitHub (Dec 25, 2024).

Hello, developers of ollama-python!

Thank you for your hard work and dedication to building such an awesome library. Keep it up! 🚀

I have a few questions about using multimodal models.

1️⃣ Where can I find up-to-date information about what prompt techniques the model supports?

For example, the llama3.2-vision description page doesn't say anything about the model supporting prompt techniques like system prompt, few-shot examples or structured output. Am I right in thinking that for each model you need to go to the official GitHub page?

For example, on this issue I learned that the model does not support few-shot prompting.

===

2️⃣ Should the same way of representing an image as bytes be used for few-shot examples and the actual request?

messages = [
    # <system_prompt>
    {
        'role': 'system',
        'content': system_prompt,
    },
    # </system_prompt>
    # <few_shot>
    {
        'role': 'user',
        'images': ['dummy.jpg']  # <-- 1 path-like str
    },
    {
        'role': 'assistant',
        'content': '{"dummy_output": [1, 2]}',
    },
    # </few_shot>
    # <actual_request>
    {
        'role': 'user',
        'images': [numpy_image.tobytes()]  # <-- 2 numpy built-in method
    }
    # </actual_request>
]

The problem is this:

def main() -> None:
    img = np.random.randint(low=0, high=256, size=(240, 320, 3), dtype=np.uint8)
    cv2.imwrite('dummy.jpg', img)

    assert Path('dummy.jpg').read_bytes() == img.tobytes()  # AssertionError
    assert Path('dummy.jpg').read_bytes() == cv2.imread('dummy.jpg').tobytes()  # AssertionError

===

ollama server version

ollama version is 0.5.2-rc3-0-g581a4a5-dirty

ollama client version

Name: ollama
Version: 0.4.4
Summary: The official Python client for Ollama.
Home-page: https://ollama.com
Author: Ollama
Author-email: hello@ollama.com
License: MIT
Requires: httpx, pydantic
Required-by:

Originally created by @DanilZittser on GitHub (Dec 25, 2024). Hello, developers of `ollama-python`! Thank you for your hard work and dedication to building such an awesome library. Keep it up! 🚀 I have a few questions about using multimodal models. :one: Where can I find up-to-date information about what prompt techniques the model supports? For example, the [llama3.2-vision](https://ollama.com/library/llama3.2-vision) description page doesn't say anything about the model supporting prompt techniques like system prompt, few-shot examples or structured output. Am I right in thinking that for each model you need to go to the official GitHub page? For example, on this [issue](https://github.com/meta-llama/llama-models/issues/163) I learned that the model does not support few-shot prompting. === :two: Should the same way of representing an image as bytes be used for few-shot examples and the actual request? ```python messages = [ # <system_prompt> { 'role': 'system', 'content': system_prompt, }, # </system_prompt> # <few_shot> { 'role': 'user', 'images': ['dummy.jpg'] # <-- 1 path-like str }, { 'role': 'assistant', 'content': '{"dummy_output": [1, 2]}', }, # </few_shot> # <actual_request> { 'role': 'user', 'images': [numpy_image.tobytes()] # <-- 2 numpy built-in method } # </actual_request> ] ``` The problem is this: ```python def main() -> None: img = np.random.randint(low=0, high=256, size=(240, 320, 3), dtype=np.uint8) cv2.imwrite('dummy.jpg', img) assert Path('dummy.jpg').read_bytes() == img.tobytes() # AssertionError assert Path('dummy.jpg').read_bytes() == cv2.imread('dummy.jpg').tobytes() # AssertionError ``` === <details> <summary>ollama server version</summary> ``` ollama version is 0.5.2-rc3-0-g581a4a5-dirty ``` </details> <details> <summary>ollama client version</summary> ``` Name: ollama Version: 0.4.4 Summary: The official Python client for Ollama. Home-page: https://ollama.com Author: Ollama Author-email: hello@ollama.com License: MIT Requires: httpx, pydantic Required-by: ``` </details>

yindo commented

2026-02-15 16:28:35 -05:00

@TemugeB commented on GitHub (Dec 27, 2024):

Wouldn't this also cause assertion error assert cv.imread('dummy.jpg').tobytes() == img.tobytes() ? Imwrite saves in RGB format while internally opencv uses BGR?

@TemugeB commented on GitHub (Dec 27, 2024): Wouldn't this also cause assertion error ```assert cv.imread('dummy.jpg').tobytes() == img.tobytes()``` ? Imwrite saves in RGB format while internally opencv uses BGR?

yindo commented

2026-02-15 16:28:36 -05:00

@TemugeB commented on GitHub (Dec 27, 2024):

Nevermind. The problem is you're using JPEG to save, which compresses the image. So the decompression is not exactly the same as the original image. Try using PNG to save.

@TemugeB commented on GitHub (Dec 27, 2024): Nevermind. The problem is you're using JPEG to save, which compresses the image. So the decompression is not exactly the same as the original image. Try using PNG to save.

yindo referenced this issue

2026-02-15 16:30:17 -05:00

[PR #195] [CLOSED] Update _client.py - always include path in create #405

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: ollama/ollama-python#192