langchain-ai/langgraph
1
0
Fork 0
mirror of https://github.com/langchain-ai/langgraph.git synced 2026-07-15 12:35:43 -04:00
Code Issues 314 Packages Projects Releases 17 Wiki Activity

[ BUG ] Image not being sent to the model using langgraph and gpt-4o #165

New Issue
Closed
opened 2026-02-20 17:30:01 -05:00 by yindo · 17 comments
yindo commented 2026-02-20 17:30:01 -05:00
Owner
Copy Link
Copy Source

Originally created by @HELIOPOTELICKI on GitHub (Jul 29, 2024).

Checked other resources

  • I added a very descriptive title to this issue.
  • I searched the LangGraph/LangChain documentation with the integrated search.
  • I used the GitHub search to find a similar question and didn't find it.
  • I am sure that this is a bug in LangGraph/LangChain rather than my code.
  • I am sure this is better as an issue rather than a GitHub discussion, since this is a LangGraph bug and not a design question.

Example Code

** Here is not all the code, only the main parts. **

self.llm = ChatOpenAI(temperature=0, model="gpt-4o").bind_tools(self.tools)

def should_continue(self, state: MessagesState) -> Literal["tools", '__end__']:
        messages = state['messages']
        last_message = messages[-1]
        if last_message.tool_calls:
            return "tools"
        return END


def call_model(self, state: MessagesState):
    messages = state['messages']
    agent_scratchpad = state.get("agent_scratchpad", [])
    prompt = self.prompt.format(
        chat_history=messages,
        input=messages[-1].content,
        agent_scratchpad=agent_scratchpad
    )
    response = self.llm.invoke(prompt)
    return {"messages": [response]}


workflow = StateGraph(MessagesState)
tool_node = ToolNode(tools)

workflow.add_node("agent", self.call_model)
workflow.add_node("tools", tool_node)
workflow.set_entry_point("agent")
workflow.add_conditional_edges(
    "agent",
    self.should_continue,
)
workflow.add_edge("tools", 'agent')

pool = ConnectionPool(
    conninfo=f"{os.getenv('DATABASE_URL')}",
    max_size=50,
)

checkpointer = PostgresSaver(sync_connection=pool)
checkpointer.create_tables(pool)

graph = workflow.compile(checkpointer=checkpointer)

graph.invoke({
        "messages": [HumanMessage(
            content=[
                {"type": "text", "text": "Describe the image below."},
                {"type": "image_url", "image_url": {"url": "https://s3-prod.cogmo.com.br/shared/cat.jpg"}},
            ]
        )]
    },
    config={
        "configurable": {
            "thread_id": session_id,
            "recursion_limit": 50,
        }
    }
)

Error Message and Stack Trace (if applicable)

No response

Description

When using langgraph to send an image to the model, the image is not being sent correctly. The model responds with the message:
"I currently don't have the capability to view or describe images. However, if you provide me with some details about the image, I'd be happy to help you with any information or tasks related to it!"

Through langsmith, it is possible to see that the image is being sent, but the model does not respond based on it.
image

System Info

Plataform: Windows 11

langchain==0.2.11
langgraph==0.1.14
langsmith=0.1.93

Originally created by @HELIOPOTELICKI on GitHub (Jul 29, 2024). ### Checked other resources - [X] I added a very descriptive title to this issue. - [X] I searched the [LangGraph](https://langchain-ai.github.io/langgraph/)/LangChain documentation with the integrated search. - [X] I used the GitHub search to find a similar question and didn't find it. - [ ] I am sure that this is a bug in LangGraph/LangChain rather than my code. - [x] I am sure this is better as an issue [rather than a GitHub discussion](https://github.com/langchain-ai/langgraph/discussions/new/choose), since this is a LangGraph bug and not a design question. ### Example Code ```python ** Here is not all the code, only the main parts. ** self.llm = ChatOpenAI(temperature=0, model="gpt-4o").bind_tools(self.tools) def should_continue(self, state: MessagesState) -> Literal["tools", '__end__']: messages = state['messages'] last_message = messages[-1] if last_message.tool_calls: return "tools" return END def call_model(self, state: MessagesState): messages = state['messages'] agent_scratchpad = state.get("agent_scratchpad", []) prompt = self.prompt.format( chat_history=messages, input=messages[-1].content, agent_scratchpad=agent_scratchpad ) response = self.llm.invoke(prompt) return {"messages": [response]} workflow = StateGraph(MessagesState) tool_node = ToolNode(tools) workflow.add_node("agent", self.call_model) workflow.add_node("tools", tool_node) workflow.set_entry_point("agent") workflow.add_conditional_edges( "agent", self.should_continue, ) workflow.add_edge("tools", 'agent') pool = ConnectionPool( conninfo=f"{os.getenv('DATABASE_URL')}", max_size=50, ) checkpointer = PostgresSaver(sync_connection=pool) checkpointer.create_tables(pool) graph = workflow.compile(checkpointer=checkpointer) graph.invoke({ "messages": [HumanMessage( content=[ {"type": "text", "text": "Describe the image below."}, {"type": "image_url", "image_url": {"url": "https://s3-prod.cogmo.com.br/shared/cat.jpg"}}, ] )] }, config={ "configurable": { "thread_id": session_id, "recursion_limit": 50, } } ) ``` ### Error Message and Stack Trace (if applicable) _No response_ ### Description When using `langgraph` to send an image to the model, the image is not being sent correctly. The model responds with the message: "I currently don't have the capability to view or describe images. However, if you provide me with some details about the image, I'd be happy to help you with any information or tasks related to it!" Through langsmith, it is possible to see that the image is being sent, but the model does not respond based on it. ![image](https://github.com/user-attachments/assets/fe350f35-5e6d-4178-8e21-78cff57c0962) ### System Info Plataform: Windows 11 langchain==0.2.11 langgraph==0.1.14 langsmith=0.1.93
yindo closed this issue 2026-02-20 17:30:01 -05:00
yindo commented 2026-02-20 17:30:01 -05:00
Author
Owner
Copy Link
Copy Source

@hwchase17 commented on GitHub (Jul 29, 2024):

is it possible to share a public langsmith trace?

@hwchase17 commented on GitHub (Jul 29, 2024): is it possible to share a public langsmith trace?
yindo commented 2026-02-20 17:30:01 -05:00
Author
Owner
Copy Link
Copy Source

@HELIOPOTELICKI commented on GitHub (Jul 29, 2024):

is it possible to share a public langsmith trace?

https://smith.langchain.com/public/73fa25b1-4221-4993-8a1a-b2a6f5f9fb63/r

@HELIOPOTELICKI commented on GitHub (Jul 29, 2024): > is it possible to share a public langsmith trace? https://smith.langchain.com/public/73fa25b1-4221-4993-8a1a-b2a6f5f9fb63/r
yindo commented 2026-02-20 17:30:01 -05:00
Author
Owner
Copy Link
Copy Source

@HELIOPOTELICKI commented on GitHub (Jul 29, 2024):

I tried passing both the URL and the base64 of the image.
Using base64, I get the error
RateLimitError: Error code: 429 - {'error': {'message': 'Request too large for gpt-4o'}}

image

@HELIOPOTELICKI commented on GitHub (Jul 29, 2024): I tried passing both the URL and the base64 of the image. Using base64, I get the error RateLimitError: Error code: 429 - {'error': {'message': 'Request too large for gpt-4o'}} ![image](https://github.com/user-attachments/assets/ad7e7832-0a36-458e-bea9-47e6e815e18e)
yindo commented 2026-02-20 17:30:01 -05:00
Author
Owner
Copy Link
Copy Source

@hwchase17 commented on GitHub (Jul 29, 2024):

got it. I think you are just formatting the prompt incorrectly. What is your prompt template?
It looks like you chat_history=messages may be getting formatted into a string?

@hwchase17 commented on GitHub (Jul 29, 2024): got it. I think you are just formatting the prompt incorrectly. What is your prompt template? It looks like you `chat_history=messages` may be getting formatted into a string?
yindo commented 2026-02-20 17:30:01 -05:00
Author
Owner
Copy Link
Copy Source

@HELIOPOTELICKI commented on GitHub (Jul 30, 2024):

Peguei. Acho que você está apenas formatando o prompt incorretamente. Qual é o seu modelo de prompt? Parece que você pode estar sendo formatado em uma string?chat_history=messages

image

This is my Template

@HELIOPOTELICKI commented on GitHub (Jul 30, 2024): > Peguei. Acho que você está apenas formatando o prompt incorretamente. Qual é o seu modelo de prompt? Parece que você pode estar sendo formatado em uma string?`chat_history=messages` ![image](https://github.com/user-attachments/assets/d3d3038c-1ae4-4e15-a9a1-cebafce2df58) This is my Template
yindo commented 2026-02-20 17:30:01 -05:00
Author
Owner
Copy Link
Copy Source

@hwchase17 commented on GitHub (Jul 30, 2024):

so i think the issue is you are doing input=messages[-1].content

This grabs the content from the last message (a dictionary) and puts it into a single string human message. ("human", "{input}")

I think you probably don't need the input parameter at all? since you have messages[-1] already in the chat_history variable. But if you do, you should insert the whole message, not just the content as a string (because if the content is not a string then it will get messed up)

@hwchase17 commented on GitHub (Jul 30, 2024): so i think the issue is you are doing `input=messages[-1].content` This grabs the content from the last message (a dictionary) and puts it into a single string human message. ("human", "{input}") I think you probably don't need the `input` parameter at all? since you have `messages[-1]` already in the chat_history variable. But if you do, you should insert the whole message, not just the content as a string (because if the content is not a string then it will get messed up)
yindo commented 2026-02-20 17:30:01 -05:00
Author
Owner
Copy Link
Copy Source

@HELIOPOTELICKI commented on GitHub (Jul 30, 2024):

então eu acho que o problema é que você está fazendo input=messages[-1].content

Isso pega o conteúdo da última mensagem (um dicionário) e o coloca em uma única mensagem humana de string. ("humano", "{input}")

Acho que você provavelmente não precisa do parâmetro? já que você já tem na variável chat_history. Mas se você fizer isso, você deve inserir a mensagem inteira, não apenas o conteúdo como uma string (porque se o conteúdo não for uma string, ele ficará confuso)input``messages[-1]

I removed the .content, but it didn't solve it.

image

image

RUN: "https://smith.langchain.com/public/64dfa1b3-882b-468c-8908-11932b5577ad/r"

@HELIOPOTELICKI commented on GitHub (Jul 30, 2024): > então eu acho que o problema é que você está fazendo `input=messages[-1].content` > > Isso pega o conteúdo da última mensagem (um dicionário) e o coloca em uma única mensagem humana de string. ("humano", "{input}") > > Acho que você provavelmente não precisa do parâmetro? já que você já tem na variável chat_history. Mas se você fizer isso, você deve inserir a mensagem inteira, não apenas o conteúdo como uma string (porque se o conteúdo não for uma string, ele ficará confuso)`input``messages[-1]` I removed the .content, but it didn't solve it. ![image](https://github.com/user-attachments/assets/5ad505ce-f139-4408-8f96-3b69b33c144d) ![image](https://github.com/user-attachments/assets/7de2e7fc-4e2d-4950-b642-e1eef9c276da) RUN: "https://smith.langchain.com/public/64dfa1b3-882b-468c-8908-11932b5577ad/r"
yindo commented 2026-02-20 17:30:01 -05:00
Author
Owner
Copy Link
Copy Source

@hwchase17 commented on GitHub (Jul 31, 2024):

hmm does something like this work for you? (trying to debug if youre running into something with the prompt template or the model)

from langchain_openai import ChatOpenAI
from langchain_core.messages import HumanMessage
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder

model = ChatOpenAI(model="gpt-4o")

messages = [HumanMessage(
            content=[
                {"type": "text", "text": "Describe the image below."},
                {"type": "image_url", "image_url": {"url": "https://s3-prod.cogmo.com.br/shared/cat.jpg"}},
            ]
        )]

prompt = ChatPromptTemplate.from_messages([
    ("system", "be helpful"),
    ("placeholder", "{m}"),
])

chain = prompt | model

chain.invoke({"m": messages})
@hwchase17 commented on GitHub (Jul 31, 2024): hmm does something like this work for you? (trying to debug if youre running into something with the prompt template or the model) ``` from langchain_openai import ChatOpenAI from langchain_core.messages import HumanMessage from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder model = ChatOpenAI(model="gpt-4o") messages = [HumanMessage( content=[ {"type": "text", "text": "Describe the image below."}, {"type": "image_url", "image_url": {"url": "https://s3-prod.cogmo.com.br/shared/cat.jpg"}}, ] )] prompt = ChatPromptTemplate.from_messages([ ("system", "be helpful"), ("placeholder", "{m}"), ]) chain = prompt | model chain.invoke({"m": messages}) ```
yindo commented 2026-02-20 17:30:01 -05:00
Author
Owner
Copy Link
Copy Source

@HELIOPOTELICKI commented on GitHub (Jul 31, 2024):

If I send it directly to the model, everything works fine, so much so that this is how I 'worked around' the problem temporarily, I built a vision tool for the agent. But I would like to get rid of this workaround lol

gpt_vision_tool:

from typing import Any, Type
from langchain_core.pydantic_v1 import BaseModel, Field
from langchain_core.tools import BaseTool, ToolException
from langchain_core.messages.human import HumanMessage
from langchain_openai import ChatOpenAI
from app.services.bucket_storage import MinioManager


class VisionSchema(BaseModel):
    query: str = Field(
        description="Question asked by the user about an image to be analyzed and answered."
    )
    image_id: str = Field(
        description="Image ID to be analyzed."
    )


class Vision(BaseTool):
    name: str = "gpt_vision_tool"
    description: str = (
        "This tool uses GPT-4o to allow the agent to interact with images."
    )
    args_schema: Type[BaseModel] = VisionSchema
    llm: ChatOpenAI = None
    s3_bucket: MinioManager = None

    def __init__(self, **data: Any):
        super().__init__(**data)
        self.llm = ChatOpenAI(temperature=0, model="gpt-4o")
        self.s3_bucket = MinioManager()

    def get_message_schema(self, query: str, image_id: str) -> HumanMessage:
        image_url = self.s3_bucket.get_object_url(image_id)

        message = HumanMessage(
            content=[
                {"type": "text", "text": query},
                {
                    "type": "image_url",
                    "image_url": {"url": f"{image_url}"},
                },
            ],
        )
        return message

    def _run(
        self,
        query: str,
        image_id: str
    ) -> str:
        try:
            return self.llm.invoke([self.get_message_schema(query, image_id)]).content
        except Exception as e:
            raise ToolException(f"Error: {e}")

    async def _arun(
        self,
        query: str,
        image_id: str
    ) -> str:
        raise NotImplementedError(
            "Async version of this tool is not implemented.")```
@HELIOPOTELICKI commented on GitHub (Jul 31, 2024): If I send it directly to the model, everything works fine, so much so that this is how I 'worked around' the problem temporarily, I built a vision tool for the agent. But I would like to get rid of this workaround lol gpt_vision_tool: ```python from typing import Any, Type from langchain_core.pydantic_v1 import BaseModel, Field from langchain_core.tools import BaseTool, ToolException from langchain_core.messages.human import HumanMessage from langchain_openai import ChatOpenAI from app.services.bucket_storage import MinioManager class VisionSchema(BaseModel): query: str = Field( description="Question asked by the user about an image to be analyzed and answered." ) image_id: str = Field( description="Image ID to be analyzed." ) class Vision(BaseTool): name: str = "gpt_vision_tool" description: str = ( "This tool uses GPT-4o to allow the agent to interact with images." ) args_schema: Type[BaseModel] = VisionSchema llm: ChatOpenAI = None s3_bucket: MinioManager = None def __init__(self, **data: Any): super().__init__(**data) self.llm = ChatOpenAI(temperature=0, model="gpt-4o") self.s3_bucket = MinioManager() def get_message_schema(self, query: str, image_id: str) -> HumanMessage: image_url = self.s3_bucket.get_object_url(image_id) message = HumanMessage( content=[ {"type": "text", "text": query}, { "type": "image_url", "image_url": {"url": f"{image_url}"}, }, ], ) return message def _run( self, query: str, image_id: str ) -> str: try: return self.llm.invoke([self.get_message_schema(query, image_id)]).content except Exception as e: raise ToolException(f"Error: {e}") async def _arun( self, query: str, image_id: str ) -> str: raise NotImplementedError( "Async version of this tool is not implemented.")```
yindo commented 2026-02-20 17:30:01 -05:00
Author
Owner
Copy Link
Copy Source

@hwchase17 commented on GitHub (Aug 5, 2024):

I think the issue is that you have ("human", "{input}") - this is saying to take the input, format it as a string inside a human message. what you want to do is pass it more directly as the entire message.

you are passing all messages in already (in chat_history) - so why are you passing in messages[-1] in input? that should already be passed in. I would probably just remove the ("human", "{input}") from the prompt template entirely?

@hwchase17 commented on GitHub (Aug 5, 2024): I think the issue is that you have `("human", "{input}")` - this is saying to take the input, format it as a string inside a human message. what you want to do is pass it more directly as the entire message. you are passing all messages in already (in chat_history) - so why are you passing in `messages[-1]` in input? that should already be passed in. I would probably just remove the `("human", "{input}")` from the prompt template entirely?
yindo commented 2026-02-20 17:30:01 -05:00
Author
Owner
Copy Link
Copy Source

@HELIOPOTELICKI commented on GitHub (Aug 7, 2024):

Acho que o problema é que você tem - isso está dizendo para pegar a entrada, formatá-la como uma string dentro de uma mensagem humana. O que você quer fazer é passá-lo mais diretamente como a mensagem inteira.("human", "{input}")

Você já está passando todas as mensagens (em chat_history) - então por que você está passando a entrada? isso já deve ter sido transmitido. Eu provavelmente removeria totalmente o modelo de prompt?messages[-1]``("human", "{input}")

I tried removing the input as well, but even so, the model responds that it cannot analyze images.

RUN: "https://smith.langchain.com/public/2b1f7334-86e2-49aa-95b0-7c8d3e2faa4f/r"

@HELIOPOTELICKI commented on GitHub (Aug 7, 2024): > Acho que o problema é que você tem - isso está dizendo para pegar a entrada, formatá-la como uma string dentro de uma mensagem humana. O que você quer fazer é passá-lo mais diretamente como a mensagem inteira.`("human", "{input}")` > > Você já está passando todas as mensagens (em chat_history) - então por que você está passando a entrada? isso já deve ter sido transmitido. Eu provavelmente removeria totalmente o modelo de prompt?`messages[-1]``("human", "{input}")` I tried removing the input as well, but even so, the model responds that it cannot analyze images. RUN: "https://smith.langchain.com/public/2b1f7334-86e2-49aa-95b0-7c8d3e2faa4f/r"
yindo commented 2026-02-20 17:30:01 -05:00
Author
Owner
Copy Link
Copy Source

@HELIOPOTELICKI commented on GitHub (Aug 7, 2024):

Prompt Template:
image

Agent Node:
image

Test:
image

@HELIOPOTELICKI commented on GitHub (Aug 7, 2024): Prompt Template: ![image](https://github.com/user-attachments/assets/07d0af7b-ad15-46da-beab-251cc318c2c6) Agent Node: ![image](https://github.com/user-attachments/assets/2f1d6070-8019-4553-b0b7-9d798b1f1185) Test: ![image](https://github.com/user-attachments/assets/0e740d95-d7e0-4fa8-adae-c8f83198c9a3)
yindo commented 2026-02-20 17:30:01 -05:00
Author
Owner
Copy Link
Copy Source

@HELIOPOTELICKI commented on GitHub (Aug 7, 2024):

Even completely simplifying the prompt

RUN: "https://smith.langchain.com/public/3d0ce4ef-9316-405c-94b8-741a0a766970/r"

@HELIOPOTELICKI commented on GitHub (Aug 7, 2024): Even completely simplifying the prompt RUN: "https://smith.langchain.com/public/3d0ce4ef-9316-405c-94b8-741a0a766970/r"
yindo commented 2026-02-20 17:30:03 -05:00
Author
Owner
Copy Link
Copy Source

@HELIOPOTELICKI commented on GitHub (Aug 7, 2024):

When sending the image directly to the model, it receives the list of messages correctly.
image

But when I send it with the langgraph, everything is passed within the content.
image

@HELIOPOTELICKI commented on GitHub (Aug 7, 2024): When sending the image directly to the model, it receives the list of messages correctly. ![image](https://github.com/user-attachments/assets/2d1b86dc-9c74-4432-b749-227d641b22ab) But when I send it with the langgraph, everything is passed within the content. ![image](https://github.com/user-attachments/assets/ba138b3f-7927-41b4-94ba-58adffc4bd08)
yindo commented 2026-02-20 17:30:04 -05:00
Author
Owner
Copy Link
Copy Source

@hwchase17 commented on GitHub (Aug 7, 2024):

ah - i see. when you call prompt.format it formats it to a string. you want to call prompt.format_messages

@hwchase17 commented on GitHub (Aug 7, 2024): ah - i see. when you call `prompt.format` it formats it to a string. you want to call `prompt.format_messages`
yindo commented 2026-02-20 17:30:06 -05:00
Author
Owner
Copy Link
Copy Source

@HELIOPOTELICKI commented on GitHub (Aug 7, 2024):

Ok, I am an idiot.
It's working, that was it, thank you very much for the help and understanding <3

@HELIOPOTELICKI commented on GitHub (Aug 7, 2024): Ok, I am an idiot. It's working, that was it, thank you very much for the help and understanding <3
yindo commented 2026-02-20 17:30:08 -05:00
Author
Owner
Copy Link
Copy Source

@hwchase17 commented on GitHub (Aug 7, 2024):

Ok, I am an idiot. It's working, that was it, thank you very much for the help and understanding <3

thats for the patience, took me a while to spot 😅

@hwchase17 commented on GitHub (Aug 7, 2024): > Ok, I am an idiot. It's working, that was it, thank you very much for the help and understanding <3 thats for the patience, took me a while to spot 😅
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agents
awaiting author revision
bug
change
chat models
checkpointer
ci
code quality
code quality
dependencies
documentation
drawing
duplicate
enhancement
good first issue
help wanted
invalid
maintainer
meta
missing information
not maintained by langchain
open-swe-auto
open-swe-max
open-swe-max-auto
pending
performance
pull-request
Mirrored from GitHub Pull Request
 question
requires design
sdk
server
stale
studio
tests
type checking
usage question
v0.5
v1.1
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langchain-ai/langgraph#165
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?