langchain-ai/langgraphjs
1
0
Fork 0
mirror of https://github.com/langchain-ai/langgraphjs.git synced 2026-07-21 16:45:24 -04:00
Code Issues 108 Packages Projects Releases 20 Wiki Activity

streamEvents() with v2 schema and ChatAnthropic #46

New Issue
Closed
opened 2026-02-15 17:14:58 -05:00 by yindo · 25 comments
yindo commented 2026-02-15 17:14:58 -05:00
Owner
Copy Link
Copy Source

Originally created by @lionkeng on GitHub (Jul 11, 2024).

I am trying out the streaming example provided in your documentation here with v2 schema. I got it to work based on suggestion provided by @jacoblee93 here

The code works when I use ChatOpenAI as the model. But it is not working when I swap the model with ChatAnthropic. What am I missing? Thank you.

import {
  HumanMessage,
  AIMessage,
  AIMessageChunk,
} from '@langchain/core/messages'
import { DynamicStructuredTool } from '@langchain/core/tools'
import { z } from 'zod'
import { ChatAnthropic } from '@langchain/anthropic'
import { ChatOpenAI } from '@langchain/openai'
import { END, START, StateGraph, StateGraphArgs } from '@langchain/langgraph'
import { MemorySaver } from '@langchain/langgraph'
import { ToolNode } from '@langchain/langgraph/prebuilt'
import { TavilySearchResults } from '@langchain/community/tools/tavily_search'

// Define the state interface
interface AgentState {
  messages: HumanMessage[]
}

// Define the graph state
const graphState: StateGraphArgs<AgentState>['channels'] = {
  messages: {
    value: (x: HumanMessage[], y: HumanMessage[]) => x.concat(y),
    default: () => [],
  },
}

// Define the tools for the agent to use

const searchTool = new DynamicStructuredTool({
  name: 'search',
  description: 'Call to surf the web.',
  schema: z.object({
    query: z.string().describe('The query to use in your search.'),
  }),
  func: async ({ query }: { query: string }) => {
    // This is a placeholder for the actual implementation
    if (
      query.toLowerCase().includes('sf') ||
      query.toLowerCase().includes('san francisco')
    ) {
      return "It's 60 degrees and foggy."
    }
    return "It's 90 degrees and sunny."
  },
})

const tools = [new TavilySearchResults({ maxResults: 1 })]
const toolNode = new ToolNode<AgentState>(tools)

const model = new ChatAnthropic({
  model: 'claude-3-sonnet-20240229',
  temperature: 0,
  streaming: true,
}).bindTools(tools)
// const model = new ChatOpenAI({
//   model: 'gpt-4o',
//   temperature: 0,
//   streaming: true,
// }).bindTools(tools)

// Define the function that determines whether to continue or not
function shouldContinue(state: AgentState): 'tools' | typeof END {
  const messages = state.messages
  const lastMessage = messages[messages.length - 1] as AIMessage

  // If the LLM makes a tool call, then we route to the "tools" node
  // if (lastMessage.additional_kwargs.tool_calls) {
  if (lastMessage.tool_calls?.length) {
    return 'tools'
  }
  // Otherwise, we stop (reply to the user)
  return END
}

// Define the function that calls the model
async function callModel(state: AgentState) {
  const messages = state.messages
  const response = await model.invoke(messages)

  // We return a list, because this will get added to the existing list
  return { messages: [response] }
}

// Define a new graph
const workflow = new StateGraph<AgentState>({ channels: graphState })
  .addNode('agent', callModel)
  .addNode('tools', toolNode)
  .addEdge(START, 'agent')
  .addConditionalEdges('agent', shouldContinue)
  .addEdge('tools', 'agent')

// Initialize memory to persist state between graph runs
const checkpointer = new MemorySaver()

// Finally, we compile it!
// This compiles it into a LangChain Runnable.
// Note that we're (optionally) passing the memory when compiling the graph
const app = workflow.compile({ checkpointer })

async function main() {
  let config = { configurable: { thread_id: 'conversation-num-1' } }
  let inputs = { messages: [new HumanMessage('Hello. My name is Joe')] }
  for await (const event of await app.streamEvents(inputs, {
    ...config,
    streamMode: 'values',
    version: 'v2',
  })) {
    console.log('event', event.event)
    if (event.event === 'on_chat_model_stream') {
      let msg = event.data?.chunk as AIMessageChunk
      if (msg.tool_call_chunks && msg.tool_call_chunks.length > 0) {
        console.log(msg.tool_call_chunks)
      } else {
        console.log(msg.content)
      }
    }
  }
}

main()
Originally created by @lionkeng on GitHub (Jul 11, 2024). I am trying out the streaming example provided in your documentation [here](https://langchain-ai.github.io/langgraphjs/how-tos/stream-tokens/#call-streamevents) with v2 schema. I got it to work based on suggestion provided by @jacoblee93 [here](https://github.com/langchain-ai/langgraphjs/issues/246#issuecomment-2221072836) The code works when I use ChatOpenAI as the model. But it is not working when I swap the model with ChatAnthropic. What am I missing? Thank you. ``` import { HumanMessage, AIMessage, AIMessageChunk, } from '@langchain/core/messages' import { DynamicStructuredTool } from '@langchain/core/tools' import { z } from 'zod' import { ChatAnthropic } from '@langchain/anthropic' import { ChatOpenAI } from '@langchain/openai' import { END, START, StateGraph, StateGraphArgs } from '@langchain/langgraph' import { MemorySaver } from '@langchain/langgraph' import { ToolNode } from '@langchain/langgraph/prebuilt' import { TavilySearchResults } from '@langchain/community/tools/tavily_search' // Define the state interface interface AgentState { messages: HumanMessage[] } // Define the graph state const graphState: StateGraphArgs<AgentState>['channels'] = { messages: { value: (x: HumanMessage[], y: HumanMessage[]) => x.concat(y), default: () => [], }, } // Define the tools for the agent to use const searchTool = new DynamicStructuredTool({ name: 'search', description: 'Call to surf the web.', schema: z.object({ query: z.string().describe('The query to use in your search.'), }), func: async ({ query }: { query: string }) => { // This is a placeholder for the actual implementation if ( query.toLowerCase().includes('sf') || query.toLowerCase().includes('san francisco') ) { return "It's 60 degrees and foggy." } return "It's 90 degrees and sunny." }, }) const tools = [new TavilySearchResults({ maxResults: 1 })] const toolNode = new ToolNode<AgentState>(tools) const model = new ChatAnthropic({ model: 'claude-3-sonnet-20240229', temperature: 0, streaming: true, }).bindTools(tools) // const model = new ChatOpenAI({ // model: 'gpt-4o', // temperature: 0, // streaming: true, // }).bindTools(tools) // Define the function that determines whether to continue or not function shouldContinue(state: AgentState): 'tools' | typeof END { const messages = state.messages const lastMessage = messages[messages.length - 1] as AIMessage // If the LLM makes a tool call, then we route to the "tools" node // if (lastMessage.additional_kwargs.tool_calls) { if (lastMessage.tool_calls?.length) { return 'tools' } // Otherwise, we stop (reply to the user) return END } // Define the function that calls the model async function callModel(state: AgentState) { const messages = state.messages const response = await model.invoke(messages) // We return a list, because this will get added to the existing list return { messages: [response] } } // Define a new graph const workflow = new StateGraph<AgentState>({ channels: graphState }) .addNode('agent', callModel) .addNode('tools', toolNode) .addEdge(START, 'agent') .addConditionalEdges('agent', shouldContinue) .addEdge('tools', 'agent') // Initialize memory to persist state between graph runs const checkpointer = new MemorySaver() // Finally, we compile it! // This compiles it into a LangChain Runnable. // Note that we're (optionally) passing the memory when compiling the graph const app = workflow.compile({ checkpointer }) async function main() { let config = { configurable: { thread_id: 'conversation-num-1' } } let inputs = { messages: [new HumanMessage('Hello. My name is Joe')] } for await (const event of await app.streamEvents(inputs, { ...config, streamMode: 'values', version: 'v2', })) { console.log('event', event.event) if (event.event === 'on_chat_model_stream') { let msg = event.data?.chunk as AIMessageChunk if (msg.tool_call_chunks && msg.tool_call_chunks.length > 0) { console.log(msg.tool_call_chunks) } else { console.log(msg.content) } } } } main() ```
yindo closed this issue 2026-02-15 17:14:59 -05:00
yindo commented 2026-02-15 17:14:59 -05:00
Author
Owner
Copy Link
Copy Source

@hwchase17 commented on GitHub (Jul 11, 2024):

when you say "not working" - what exactly is happening?

@hwchase17 commented on GitHub (Jul 11, 2024): when you say "not working" - what exactly is happening?
yindo commented 2026-02-15 17:14:59 -05:00
Author
Owner
Copy Link
Copy Source

@bracesproul commented on GitHub (Jul 11, 2024):

+1 to Harrison's comment.

If you're referencing not being able to access token counts when streaming tool calls, I just pushed a fix here. I also see some other issues with your code. For extracting tool calls from messages, you should be looking inside message.tool_calls not message.additional_kwargs.tool_calls. If there's another bug, please let me know what the issue is and I can work it out. Thanks!

@bracesproul commented on GitHub (Jul 11, 2024): +1 to Harrison's comment. If you're referencing not being able to access token counts when streaming tool calls, I just pushed a fix [here](https://github.com/langchain-ai/langchainjs/pull/6042). I also see some other issues with your code. For extracting tool calls from messages, you should be looking inside `message.tool_calls` not `message.additional_kwargs.tool_calls`. If there's another bug, please let me know what the issue is and I can work it out. Thanks!
yindo commented 2026-02-15 17:15:00 -05:00
Author
Owner
Copy Link
Copy Source

@lionkeng commented on GitHub (Jul 11, 2024):

when you say "not working" - what exactly is happening?
@hwchase17
When I run it with ChatAnthropic, streamEvents never seem to generate any event.event with on_chat_model_stream.

@bracesproul thank you! I see that too. But I seem to be running into something more fundamental.

@lionkeng commented on GitHub (Jul 11, 2024): > when you say "not working" - what exactly is happening? @hwchase17 When I run it with ChatAnthropic, streamEvents never seem to generate any event.event with `on_chat_model_stream`. @bracesproul thank you! I see that too. But I seem to be running into something more fundamental.
yindo commented 2026-02-15 17:15:00 -05:00
Author
Owner
Copy Link
Copy Source

@bracesproul commented on GitHub (Jul 11, 2024):

@lionkeng what exactly is the issue? If you could provide a stack trace and an in depth description of the issue I can help fix it.

@bracesproul commented on GitHub (Jul 11, 2024): @lionkeng what exactly is the issue? If you could provide a stack trace and an in depth description of the issue I can help fix it.
yindo commented 2026-02-15 17:15:00 -05:00
Author
Owner
Copy Link
Copy Source

@lionkeng commented on GitHub (Jul 11, 2024):

@lionkeng what exactly is the issue? If you could provide a stack trace and an in depth description of the issue I can help fix it.

@bracesproul
There is no stack trace per-se as the program/script just runs but doesn't encounter any event named on_chat_model_stream. So if I run the script as shown above, the console.log will output all of the following but missing on_chat_model_stream:

event on_chain_start
event on_chain_start
event on_chain_end
event on_chain_stream
event on_chain_start
event on_chain_start
event on_chat_model_start
event on_chat_model_end
event on_chain_end
event on_chain_start
event on_chain_end
event on_chain_start
event on_chain_end
event on_chain_end
event on_chain_stream
event on_chain_end
@lionkeng commented on GitHub (Jul 11, 2024): > @lionkeng what exactly is the issue? If you could provide a stack trace and an in depth description of the issue I can help fix it. @bracesproul There is no stack trace per-se as the program/script just runs but doesn't encounter any event named `on_chat_model_stream`. So if I run the script as shown above, the console.log will output all of the following but missing `on_chat_model_stream`: ``` event on_chain_start event on_chain_start event on_chain_end event on_chain_stream event on_chain_start event on_chain_start event on_chat_model_start event on_chat_model_end event on_chain_end event on_chain_start event on_chain_end event on_chain_start event on_chain_end event on_chain_end event on_chain_stream event on_chain_end
yindo commented 2026-02-15 17:15:00 -05:00
Author
Owner
Copy Link
Copy Source

@bracesproul commented on GitHub (Jul 11, 2024):

Okay just dug into this, we don't support streaming tool calls with anthropic in the way you'd think. So yes you can call model.stream and it'll work, but under the hood of the ChatAnthropic class we're calling the non streaming method if there are tool calls. So because of this, streamEvents won't yield that event.

This is a top priority for us though, and we're aiming to get this implemented by midway through next week.

@bracesproul commented on GitHub (Jul 11, 2024): Okay just dug into this, we don't support streaming tool calls with anthropic in the way you'd think. So yes you can call model.stream and it'll work, but under the hood of the `ChatAnthropic` class we're calling the non streaming method if there are tool calls. So because of this, `streamEvents` won't yield that event. This is a top priority for us though, and we're aiming to get this implemented by midway through next week.
yindo commented 2026-02-15 17:15:01 -05:00
Author
Owner
Copy Link
Copy Source

@lionkeng commented on GitHub (Jul 11, 2024):

Thanks for the response @bracesproul ! Looking forwarding to testing out the changes.

@lionkeng commented on GitHub (Jul 11, 2024): Thanks for the response @bracesproul ! Looking forwarding to testing out the changes.
yindo commented 2026-02-15 17:15:01 -05:00
Author
Owner
Copy Link
Copy Source

@danny-avila commented on GitHub (Jul 23, 2024):

Still having streaming issues with ChatAnthropic using streamEvents and v2 in particular.

Here are my langchain-specific dependencies from my project package.json:

"dependencies": {
  "@langchain/anthropic": "^0.2.8",
  "@langchain/aws": "^0.0.5",
  "@langchain/community": "^0.2.20",
  "@langchain/core": "^0.2.18",
  "@langchain/google-vertexai": "^0.0.20",
  "@langchain/langgraph": "^0.0.31",
  "@langchain/mistralai": "^0.0.26",
  "langchain": "^0.2.10",
},
"resolutions": {
  "@langchain/core": "0.2.18",
},

Code to Reproduce:

Note: this correctly streams tokens when we don't bind tools, but if we do, it fails.

import { BaseMessage, HumanMessage } from "@langchain/core/messages";
import { ChatAnthropic } from "@langchain/anthropic";
import { ChatPromptTemplate } from "@langchain/core/prompts";
import { TavilySearchResults } from '@langchain/community/tools/tavily_search';
import { RunnableConfig } from "@langchain/core/runnables";
import {
  END,
  // MemorySaver,
  START,
  StateGraph,
  StateGraphArgs,
} from "@langchain/langgraph";

interface IState {
  messages: BaseMessage[];
  userInfo: string;
}

async function main() {

const graphState: StateGraphArgs<IState>["channels"] = {
  messages: {
    value: (x: BaseMessage[], y: BaseMessage[]) => x.concat(y),
    default: () => [],
  },
  userInfo: {
    value: (x?: string, y?: string) => {
      return y ? y : x ? x : "N/A";
    },
    default: () => "N/A",
  },
};

const promptTemplate = ChatPromptTemplate.fromMessages([
  ["system", "You are a helpful assistant.\n\n## User Info:\n{userInfo}"],
  ["placeholder", "{messages}"],
]);

const initializeModel = (bindTools = false) => {
  const model = new ChatAnthropic({
    // model: "claude-3-haiku-20240307",
    model: 'claude-3-5-sonnet-20240620'
  })
  if (bindTools) {
    return model.bindTools([new TavilySearchResults()]);
  }
  return model;
}

const callModel = async (
  state: { messages: BaseMessage[]; userInfo: string },
  config?: RunnableConfig,
) => {
  const { messages, userInfo } = state;
  
  /*

  This correctly streams tokens when 
  we don't bind tools, but if we do, it fails.

  */
  const model = initializeModel(true);
  // const model = initializeModel();
  const chain = promptTemplate.pipe(model);
  const response = await chain.invoke(
    {
      messages,
      userInfo,
    },
    config,
  );
  return { messages: [response] };
};

const fetchUserInformation = async (
  _: { messages: BaseMessage[] },
  config?: RunnableConfig,
) => {
  const userDB = {
    user1: {
      name: "John Doe",
      email: "jod@langchain.ai",
      phone: "+1234567890",
    },
    user2: {
      name: "Jane Doe",
      email: "jad@langchain.ai",
      phone: "+0987654321",
    },
  };
  const userId = config?.configurable?.user;
  if (userId) {
    const user = userDB[userId as keyof typeof userDB];
    if (user) {
      return {
        userInfo:
          `Name: ${user.name}\nEmail: ${user.email}\nPhone: ${user.phone}`,
      };
    }
  }
  return { userInfo: "N/A" };
};

const workflow = new StateGraph({
  channels: graphState,
})
  .addNode("fetchUserInfo", fetchUserInformation)
  .addNode("agent", callModel)
  .addEdge(START, "fetchUserInfo")
  .addEdge("fetchUserInfo", "agent")
  .addEdge("agent", END);

// Here we only save in-memory
// let memory = new MemorySaver();
// const graph = workflow.compile({ checkpointer: memory });
const graph = workflow.compile();


const config = {
  configurable: {
    user: "user1",
  },
};
const inputs = {
  messages: [new HumanMessage("Could you remind me of my email?")],
};

  const stream = graph.streamEvents(inputs, {
    ...config,
    version: "v2",
    streamMode: "values",
  });
for await (const { event, data } of stream) {

  if (event === "on_chat_model_start") {
    console.log('=======CHAT_MODEL_START=======');
    console.dir(data, { depth: null });
  } else if (event === "on_chat_model_stream") {
    console.dir(data, { depth: null });
  } else if (event === "on_chat_model_end") {
    console.log('=======CHAT_MODEL_END=======');
    console.dir(data, { depth: null });
  } else {
    console.log(event);
  }
}
}

main().catch((err) => {
  console.error(err);
  process.exit(1);
});

Output:

on_chain_start
on_chain_start
on_chain_end
on_chain_stream
on_chain_start
on_chain_start
on_chain_end
on_chain_start
on_chain_end
on_chain_end
on_chain_stream
on_chain_start
on_chain_start
on_chain_start
on_prompt_start
on_prompt_end
=======CHAT_MODEL_START=======
{
  input: {
    messages: [
      [
        SystemMessage {
          lc_serializable: true,
          lc_kwargs: {
            content: 'You are a helpful assistant.\n' +
              '\n' +
              '## User Info:\n' +
              'Name: John Doe\n' +
              'Email: jod@langchain.ai\n' +
              'Phone: +1234567890',
            additional_kwargs: {},
            response_metadata: {}
          },
          lc_namespace: [ 'langchain_core', 'messages' ],
          content: 'You are a helpful assistant.\n' +
            '\n' +
            '## User Info:\n' +
            'Name: John Doe\n' +
            'Email: jod@langchain.ai\n' +
            'Phone: +1234567890',
          name: undefined,
          additional_kwargs: {},
          response_metadata: {},
          id: undefined
        },
        HumanMessage {
          lc_serializable: true,
          lc_kwargs: {
            content: 'Could you remind me of my email?',
            additional_kwargs: {},
            response_metadata: {}
          },
          lc_namespace: [ 'langchain_core', 'messages' ],
          content: 'Could you remind me of my email?',
          name: undefined,
          additional_kwargs: {},
          response_metadata: {},
          id: undefined
        }
      ]
    ]
  }
}
=======CHAT_MODEL_END=======
{
  output: AIMessageChunk {
    lc_serializable: true,
    lc_kwargs: {
      content: [
        {
          index: 0,
          type: 'text_delta',
          text: "Certainly! I'd be happy to remind you of your email address. Based on the user information provided, your email address is:\n" +
            '\n' +
            'jod@langchain.ai\n' +
            '\n' +
            "Is there anything else you'd like to know about your contact information?"
        }
      ],
      additional_kwargs: {
        id: 'msg_01TZ28cmm6nPk7FsmRHjEDet',
        type: 'message',
        role: 'assistant',
        model: 'claude-3-5-sonnet-20240620',
        stop_reason: 'end_turn',
        stop_sequence: null,
        usage: { input_tokens: 452, output_tokens: 55 }
      },
      response_metadata: {},
      tool_call_chunks: [],
      id: 'msg_01TZ28cmm6nPk7FsmRHjEDet',
      usage_metadata: { input_tokens: 956, output_tokens: 58, total_tokens: 1014 },
      tool_calls: [],
      invalid_tool_calls: []
    },
    lc_namespace: [ 'langchain_core', 'messages' ],
    content: [
      {
        index: 0,
        type: 'text_delta',
        text: "Certainly! I'd be happy to remind you of your email address. Based on the user information provided, your email address is:\n" +
          '\n' +
          'jod@langchain.ai\n' +
          '\n' +
          "Is there anything else you'd like to know about your contact information?"
      }
    ],
    name: undefined,
    additional_kwargs: {
      id: 'msg_01TZ28cmm6nPk7FsmRHjEDet',
      type: 'message',
      role: 'assistant',
      model: 'claude-3-5-sonnet-20240620',
      stop_reason: 'end_turn',
      stop_sequence: null,
      usage: { input_tokens: 452, output_tokens: 55 }
    },
    response_metadata: {},
    id: 'msg_01TZ28cmm6nPk7FsmRHjEDet',
    tool_calls: [],
    invalid_tool_calls: [],
    tool_call_chunks: [],
    usage_metadata: { input_tokens: 956, output_tokens: 58, total_tokens: 1014 }
  },
  input: {
    messages: [
      [
        SystemMessage {
          lc_serializable: true,
          lc_kwargs: {
            content: 'You are a helpful assistant.\n' +
              '\n' +
              '## User Info:\n' +
              'Name: John Doe\n' +
              'Email: jod@langchain.ai\n' +
              'Phone: +1234567890',
            additional_kwargs: {},
            response_metadata: {}
          },
          lc_namespace: [ 'langchain_core', 'messages' ],
          content: 'You are a helpful assistant.\n' +
            '\n' +
            '## User Info:\n' +
            'Name: John Doe\n' +
            'Email: jod@langchain.ai\n' +
            'Phone: +1234567890',
          name: undefined,
          additional_kwargs: {},
          response_metadata: {},
          id: undefined
        },
        HumanMessage {
          lc_serializable: true,
          lc_kwargs: {
            content: 'Could you remind me of my email?',
            additional_kwargs: {},
            response_metadata: {}
          },
          lc_namespace: [ 'langchain_core', 'messages' ],
          content: 'Could you remind me of my email?',
          name: undefined,
          additional_kwargs: {},
          response_metadata: {},
          id: undefined
        }
      ]
    ]
  }
}
on_chain_end
on_chain_end
on_chain_start
on_chain_end
on_chain_end
on_chain_stream
on_chain_end

Output if model is not bound

on_chain_start
on_chain_start
on_chain_end
on_chain_stream
on_chain_start
on_chain_start
on_chain_end
on_chain_start
on_chain_end
on_chain_end
on_chain_stream
on_chain_start
on_chain_start
on_chain_start
on_prompt_start
on_prompt_end
=======CHAT_MODEL_START=======
{
  input: {
    messages: [
      [
        SystemMessage {
          lc_serializable: true,
          lc_kwargs: {
            content: 'You are a helpful assistant.\n' +
              '\n' +
              '## User Info:\n' +
              'Name: John Doe\n' +
              'Email: jod@langchain.ai\n' +
              'Phone: +1234567890',
            additional_kwargs: {},
            response_metadata: {}
          },
          lc_namespace: [ 'langchain_core', 'messages' ],
          content: 'You are a helpful assistant.\n' +
            '\n' +
            '## User Info:\n' +
            'Name: John Doe\n' +
            'Email: jod@langchain.ai\n' +
            'Phone: +1234567890',
          name: undefined,
          additional_kwargs: {},
          response_metadata: {},
          id: undefined
        },
        HumanMessage {
          lc_serializable: true,
          lc_kwargs: {
            content: 'Could you remind me of my email?',
            additional_kwargs: {},
            response_metadata: {}
          },
          lc_namespace: [ 'langchain_core', 'messages' ],
          content: 'Could you remind me of my email?',
          name: undefined,
          additional_kwargs: {},
          response_metadata: {},
          id: undefined
        }
      ]
    ]
  }
}
{
  chunk: AIMessageChunk {
    lc_serializable: true,
    lc_kwargs: {
      content: 'Certainly! Your email',
      tool_calls: [],
      invalid_tool_calls: [],
      tool_call_chunks: [],
      additional_kwargs: {},
      response_metadata: {}
    },
    lc_namespace: [ 'langchain_core', 'messages' ],
    content: 'Certainly! Your email',
    name: undefined,
    additional_kwargs: {},
    response_metadata: {},
    id: undefined,
    tool_calls: [],
    invalid_tool_calls: [],
    tool_call_chunks: [],
    usage_metadata: undefined
  }
}
{
  chunk: AIMessageChunk {
    lc_serializable: true,
    lc_kwargs: {
      content: ' address is jo',
      tool_calls: [],
      invalid_tool_calls: [],
      tool_call_chunks: [],
      additional_kwargs: {},
      response_metadata: {}
    },
    lc_namespace: [ 'langchain_core', 'messages' ],
    content: ' address is jo',
    name: undefined,
    additional_kwargs: {},
    response_metadata: {},
    id: undefined,
    tool_calls: [],
    invalid_tool_calls: [],
    tool_call_chunks: [],
    usage_metadata: undefined
  }
}
{
  chunk: AIMessageChunk {
    lc_serializable: true,
    lc_kwargs: {
      content: 'd@langchain.',
      tool_calls: [],
      invalid_tool_calls: [],
      tool_call_chunks: [],
      additional_kwargs: {},
      response_metadata: {}
    },
    lc_namespace: [ 'langchain_core', 'messages' ],
    content: 'd@langchain.',
    name: undefined,
    additional_kwargs: {},
    response_metadata: {},
    id: undefined,
    tool_calls: [],
    invalid_tool_calls: [],
    tool_call_chunks: [],
    usage_metadata: undefined
  }
}
{
  chunk: AIMessageChunk {
    lc_serializable: true,
    lc_kwargs: {
      content: 'ai.',
      tool_calls: [],
      invalid_tool_calls: [],
      tool_call_chunks: [],
      additional_kwargs: {},
      response_metadata: {}
    },
    lc_namespace: [ 'langchain_core', 'messages' ],
    content: 'ai.',
    name: undefined,
    additional_kwargs: {},
    response_metadata: {},
    id: undefined,
    tool_calls: [],
    invalid_tool_calls: [],
    tool_call_chunks: [],
    usage_metadata: undefined
  }
}
=======CHAT_MODEL_END=======
{
  output: AIMessageChunk {
    lc_serializable: true,
    lc_kwargs: {
      content: 'Certainly! Your email address is jod@langchain.ai.',
      additional_kwargs: {
        id: 'msg_01Li1Z6ByM93sVQVLTJbaUdR',
        type: 'message',
        role: 'assistant',
        model: 'claude-3-5-sonnet-20240620',
        stop_reason: 'end_turn',
        stop_sequence: null,
        usage: { input_tokens: 50, output_tokens: 21 }
      },
      response_metadata: {},
      tool_call_chunks: [],
      id: 'msg_01Li1Z6ByM93sVQVLTJbaUdR',
      usage_metadata: { input_tokens: 117, output_tokens: 25, total_tokens: 142 },
      tool_calls: [],
      invalid_tool_calls: []
    },
    lc_namespace: [ 'langchain_core', 'messages' ],
    content: 'Certainly! Your email address is jod@langchain.ai.',
    name: undefined,
    additional_kwargs: {
      id: 'msg_01Li1Z6ByM93sVQVLTJbaUdR',
      type: 'message',
      role: 'assistant',
      model: 'claude-3-5-sonnet-20240620',
      stop_reason: 'end_turn',
      stop_sequence: null,
      usage: { input_tokens: 50, output_tokens: 21 }
    },
    response_metadata: {},
    id: 'msg_01Li1Z6ByM93sVQVLTJbaUdR',
    tool_calls: [],
    invalid_tool_calls: [],
    tool_call_chunks: [],
    usage_metadata: { input_tokens: 117, output_tokens: 25, total_tokens: 142 }
  },
  input: {
    messages: [
      [
        SystemMessage {
          lc_serializable: true,
          lc_kwargs: {
            content: 'You are a helpful assistant.\n' +
              '\n' +
              '## User Info:\n' +
              'Name: John Doe\n' +
              'Email: jod@langchain.ai\n' +
              'Phone: +1234567890',
            additional_kwargs: {},
            response_metadata: {}
          },
          lc_namespace: [ 'langchain_core', 'messages' ],
          content: 'You are a helpful assistant.\n' +
            '\n' +
            '## User Info:\n' +
            'Name: John Doe\n' +
            'Email: jod@langchain.ai\n' +
            'Phone: +1234567890',
          name: undefined,
          additional_kwargs: {},
          response_metadata: {},
          id: undefined
        },
        HumanMessage {
          lc_serializable: true,
          lc_kwargs: {
            content: 'Could you remind me of my email?',
            additional_kwargs: {},
            response_metadata: {}
          },
          lc_namespace: [ 'langchain_core', 'messages' ],
          content: 'Could you remind me of my email?',
          name: undefined,
          additional_kwargs: {},
          response_metadata: {},
          id: undefined
        }
      ]
    ]
  }
}
on_chain_end
on_chain_end
on_chain_start
on_chain_end
on_chain_end
on_chain_stream
on_chain_end
@danny-avila commented on GitHub (Jul 23, 2024): Still having streaming issues with ChatAnthropic using `streamEvents` and `v2` in particular. Here are my langchain-specific dependencies from my project package.json: ```json "dependencies": { "@langchain/anthropic": "^0.2.8", "@langchain/aws": "^0.0.5", "@langchain/community": "^0.2.20", "@langchain/core": "^0.2.18", "@langchain/google-vertexai": "^0.0.20", "@langchain/langgraph": "^0.0.31", "@langchain/mistralai": "^0.0.26", "langchain": "^0.2.10", }, "resolutions": { "@langchain/core": "0.2.18", }, ``` ## Code to Reproduce: Note: this correctly streams tokens when we don't bind tools, but if we do, it fails. ```ts import { BaseMessage, HumanMessage } from "@langchain/core/messages"; import { ChatAnthropic } from "@langchain/anthropic"; import { ChatPromptTemplate } from "@langchain/core/prompts"; import { TavilySearchResults } from '@langchain/community/tools/tavily_search'; import { RunnableConfig } from "@langchain/core/runnables"; import { END, // MemorySaver, START, StateGraph, StateGraphArgs, } from "@langchain/langgraph"; interface IState { messages: BaseMessage[]; userInfo: string; } async function main() { const graphState: StateGraphArgs<IState>["channels"] = { messages: { value: (x: BaseMessage[], y: BaseMessage[]) => x.concat(y), default: () => [], }, userInfo: { value: (x?: string, y?: string) => { return y ? y : x ? x : "N/A"; }, default: () => "N/A", }, }; const promptTemplate = ChatPromptTemplate.fromMessages([ ["system", "You are a helpful assistant.\n\n## User Info:\n{userInfo}"], ["placeholder", "{messages}"], ]); const initializeModel = (bindTools = false) => { const model = new ChatAnthropic({ // model: "claude-3-haiku-20240307", model: 'claude-3-5-sonnet-20240620' }) if (bindTools) { return model.bindTools([new TavilySearchResults()]); } return model; } const callModel = async ( state: { messages: BaseMessage[]; userInfo: string }, config?: RunnableConfig, ) => { const { messages, userInfo } = state; /* This correctly streams tokens when we don't bind tools, but if we do, it fails. */ const model = initializeModel(true); // const model = initializeModel(); const chain = promptTemplate.pipe(model); const response = await chain.invoke( { messages, userInfo, }, config, ); return { messages: [response] }; }; const fetchUserInformation = async ( _: { messages: BaseMessage[] }, config?: RunnableConfig, ) => { const userDB = { user1: { name: "John Doe", email: "jod@langchain.ai", phone: "+1234567890", }, user2: { name: "Jane Doe", email: "jad@langchain.ai", phone: "+0987654321", }, }; const userId = config?.configurable?.user; if (userId) { const user = userDB[userId as keyof typeof userDB]; if (user) { return { userInfo: `Name: ${user.name}\nEmail: ${user.email}\nPhone: ${user.phone}`, }; } } return { userInfo: "N/A" }; }; const workflow = new StateGraph({ channels: graphState, }) .addNode("fetchUserInfo", fetchUserInformation) .addNode("agent", callModel) .addEdge(START, "fetchUserInfo") .addEdge("fetchUserInfo", "agent") .addEdge("agent", END); // Here we only save in-memory // let memory = new MemorySaver(); // const graph = workflow.compile({ checkpointer: memory }); const graph = workflow.compile(); const config = { configurable: { user: "user1", }, }; const inputs = { messages: [new HumanMessage("Could you remind me of my email?")], }; const stream = graph.streamEvents(inputs, { ...config, version: "v2", streamMode: "values", }); for await (const { event, data } of stream) { if (event === "on_chat_model_start") { console.log('=======CHAT_MODEL_START======='); console.dir(data, { depth: null }); } else if (event === "on_chat_model_stream") { console.dir(data, { depth: null }); } else if (event === "on_chat_model_end") { console.log('=======CHAT_MODEL_END======='); console.dir(data, { depth: null }); } else { console.log(event); } } } main().catch((err) => { console.error(err); process.exit(1); }); ``` --- ## Output: ```bash on_chain_start on_chain_start on_chain_end on_chain_stream on_chain_start on_chain_start on_chain_end on_chain_start on_chain_end on_chain_end on_chain_stream on_chain_start on_chain_start on_chain_start on_prompt_start on_prompt_end =======CHAT_MODEL_START======= { input: { messages: [ [ SystemMessage { lc_serializable: true, lc_kwargs: { content: 'You are a helpful assistant.\n' + '\n' + '## User Info:\n' + 'Name: John Doe\n' + 'Email: jod@langchain.ai\n' + 'Phone: +1234567890', additional_kwargs: {}, response_metadata: {} }, lc_namespace: [ 'langchain_core', 'messages' ], content: 'You are a helpful assistant.\n' + '\n' + '## User Info:\n' + 'Name: John Doe\n' + 'Email: jod@langchain.ai\n' + 'Phone: +1234567890', name: undefined, additional_kwargs: {}, response_metadata: {}, id: undefined }, HumanMessage { lc_serializable: true, lc_kwargs: { content: 'Could you remind me of my email?', additional_kwargs: {}, response_metadata: {} }, lc_namespace: [ 'langchain_core', 'messages' ], content: 'Could you remind me of my email?', name: undefined, additional_kwargs: {}, response_metadata: {}, id: undefined } ] ] } } =======CHAT_MODEL_END======= { output: AIMessageChunk { lc_serializable: true, lc_kwargs: { content: [ { index: 0, type: 'text_delta', text: "Certainly! I'd be happy to remind you of your email address. Based on the user information provided, your email address is:\n" + '\n' + 'jod@langchain.ai\n' + '\n' + "Is there anything else you'd like to know about your contact information?" } ], additional_kwargs: { id: 'msg_01TZ28cmm6nPk7FsmRHjEDet', type: 'message', role: 'assistant', model: 'claude-3-5-sonnet-20240620', stop_reason: 'end_turn', stop_sequence: null, usage: { input_tokens: 452, output_tokens: 55 } }, response_metadata: {}, tool_call_chunks: [], id: 'msg_01TZ28cmm6nPk7FsmRHjEDet', usage_metadata: { input_tokens: 956, output_tokens: 58, total_tokens: 1014 }, tool_calls: [], invalid_tool_calls: [] }, lc_namespace: [ 'langchain_core', 'messages' ], content: [ { index: 0, type: 'text_delta', text: "Certainly! I'd be happy to remind you of your email address. Based on the user information provided, your email address is:\n" + '\n' + 'jod@langchain.ai\n' + '\n' + "Is there anything else you'd like to know about your contact information?" } ], name: undefined, additional_kwargs: { id: 'msg_01TZ28cmm6nPk7FsmRHjEDet', type: 'message', role: 'assistant', model: 'claude-3-5-sonnet-20240620', stop_reason: 'end_turn', stop_sequence: null, usage: { input_tokens: 452, output_tokens: 55 } }, response_metadata: {}, id: 'msg_01TZ28cmm6nPk7FsmRHjEDet', tool_calls: [], invalid_tool_calls: [], tool_call_chunks: [], usage_metadata: { input_tokens: 956, output_tokens: 58, total_tokens: 1014 } }, input: { messages: [ [ SystemMessage { lc_serializable: true, lc_kwargs: { content: 'You are a helpful assistant.\n' + '\n' + '## User Info:\n' + 'Name: John Doe\n' + 'Email: jod@langchain.ai\n' + 'Phone: +1234567890', additional_kwargs: {}, response_metadata: {} }, lc_namespace: [ 'langchain_core', 'messages' ], content: 'You are a helpful assistant.\n' + '\n' + '## User Info:\n' + 'Name: John Doe\n' + 'Email: jod@langchain.ai\n' + 'Phone: +1234567890', name: undefined, additional_kwargs: {}, response_metadata: {}, id: undefined }, HumanMessage { lc_serializable: true, lc_kwargs: { content: 'Could you remind me of my email?', additional_kwargs: {}, response_metadata: {} }, lc_namespace: [ 'langchain_core', 'messages' ], content: 'Could you remind me of my email?', name: undefined, additional_kwargs: {}, response_metadata: {}, id: undefined } ] ] } } on_chain_end on_chain_end on_chain_start on_chain_end on_chain_end on_chain_stream on_chain_end ``` ## Output if model is not bound ```bash on_chain_start on_chain_start on_chain_end on_chain_stream on_chain_start on_chain_start on_chain_end on_chain_start on_chain_end on_chain_end on_chain_stream on_chain_start on_chain_start on_chain_start on_prompt_start on_prompt_end =======CHAT_MODEL_START======= { input: { messages: [ [ SystemMessage { lc_serializable: true, lc_kwargs: { content: 'You are a helpful assistant.\n' + '\n' + '## User Info:\n' + 'Name: John Doe\n' + 'Email: jod@langchain.ai\n' + 'Phone: +1234567890', additional_kwargs: {}, response_metadata: {} }, lc_namespace: [ 'langchain_core', 'messages' ], content: 'You are a helpful assistant.\n' + '\n' + '## User Info:\n' + 'Name: John Doe\n' + 'Email: jod@langchain.ai\n' + 'Phone: +1234567890', name: undefined, additional_kwargs: {}, response_metadata: {}, id: undefined }, HumanMessage { lc_serializable: true, lc_kwargs: { content: 'Could you remind me of my email?', additional_kwargs: {}, response_metadata: {} }, lc_namespace: [ 'langchain_core', 'messages' ], content: 'Could you remind me of my email?', name: undefined, additional_kwargs: {}, response_metadata: {}, id: undefined } ] ] } } { chunk: AIMessageChunk { lc_serializable: true, lc_kwargs: { content: 'Certainly! Your email', tool_calls: [], invalid_tool_calls: [], tool_call_chunks: [], additional_kwargs: {}, response_metadata: {} }, lc_namespace: [ 'langchain_core', 'messages' ], content: 'Certainly! Your email', name: undefined, additional_kwargs: {}, response_metadata: {}, id: undefined, tool_calls: [], invalid_tool_calls: [], tool_call_chunks: [], usage_metadata: undefined } } { chunk: AIMessageChunk { lc_serializable: true, lc_kwargs: { content: ' address is jo', tool_calls: [], invalid_tool_calls: [], tool_call_chunks: [], additional_kwargs: {}, response_metadata: {} }, lc_namespace: [ 'langchain_core', 'messages' ], content: ' address is jo', name: undefined, additional_kwargs: {}, response_metadata: {}, id: undefined, tool_calls: [], invalid_tool_calls: [], tool_call_chunks: [], usage_metadata: undefined } } { chunk: AIMessageChunk { lc_serializable: true, lc_kwargs: { content: 'd@langchain.', tool_calls: [], invalid_tool_calls: [], tool_call_chunks: [], additional_kwargs: {}, response_metadata: {} }, lc_namespace: [ 'langchain_core', 'messages' ], content: 'd@langchain.', name: undefined, additional_kwargs: {}, response_metadata: {}, id: undefined, tool_calls: [], invalid_tool_calls: [], tool_call_chunks: [], usage_metadata: undefined } } { chunk: AIMessageChunk { lc_serializable: true, lc_kwargs: { content: 'ai.', tool_calls: [], invalid_tool_calls: [], tool_call_chunks: [], additional_kwargs: {}, response_metadata: {} }, lc_namespace: [ 'langchain_core', 'messages' ], content: 'ai.', name: undefined, additional_kwargs: {}, response_metadata: {}, id: undefined, tool_calls: [], invalid_tool_calls: [], tool_call_chunks: [], usage_metadata: undefined } } =======CHAT_MODEL_END======= { output: AIMessageChunk { lc_serializable: true, lc_kwargs: { content: 'Certainly! Your email address is jod@langchain.ai.', additional_kwargs: { id: 'msg_01Li1Z6ByM93sVQVLTJbaUdR', type: 'message', role: 'assistant', model: 'claude-3-5-sonnet-20240620', stop_reason: 'end_turn', stop_sequence: null, usage: { input_tokens: 50, output_tokens: 21 } }, response_metadata: {}, tool_call_chunks: [], id: 'msg_01Li1Z6ByM93sVQVLTJbaUdR', usage_metadata: { input_tokens: 117, output_tokens: 25, total_tokens: 142 }, tool_calls: [], invalid_tool_calls: [] }, lc_namespace: [ 'langchain_core', 'messages' ], content: 'Certainly! Your email address is jod@langchain.ai.', name: undefined, additional_kwargs: { id: 'msg_01Li1Z6ByM93sVQVLTJbaUdR', type: 'message', role: 'assistant', model: 'claude-3-5-sonnet-20240620', stop_reason: 'end_turn', stop_sequence: null, usage: { input_tokens: 50, output_tokens: 21 } }, response_metadata: {}, id: 'msg_01Li1Z6ByM93sVQVLTJbaUdR', tool_calls: [], invalid_tool_calls: [], tool_call_chunks: [], usage_metadata: { input_tokens: 117, output_tokens: 25, total_tokens: 142 } }, input: { messages: [ [ SystemMessage { lc_serializable: true, lc_kwargs: { content: 'You are a helpful assistant.\n' + '\n' + '## User Info:\n' + 'Name: John Doe\n' + 'Email: jod@langchain.ai\n' + 'Phone: +1234567890', additional_kwargs: {}, response_metadata: {} }, lc_namespace: [ 'langchain_core', 'messages' ], content: 'You are a helpful assistant.\n' + '\n' + '## User Info:\n' + 'Name: John Doe\n' + 'Email: jod@langchain.ai\n' + 'Phone: +1234567890', name: undefined, additional_kwargs: {}, response_metadata: {}, id: undefined }, HumanMessage { lc_serializable: true, lc_kwargs: { content: 'Could you remind me of my email?', additional_kwargs: {}, response_metadata: {} }, lc_namespace: [ 'langchain_core', 'messages' ], content: 'Could you remind me of my email?', name: undefined, additional_kwargs: {}, response_metadata: {}, id: undefined } ] ] } } on_chain_end on_chain_end on_chain_start on_chain_end on_chain_end on_chain_stream on_chain_end ```
yindo commented 2026-02-15 17:15:01 -05:00
Author
Owner
Copy Link
Copy Source

@jacoblee93 commented on GitHub (Jul 23, 2024):

CC @bracesproul can you have a look?

@jacoblee93 commented on GitHub (Jul 23, 2024): CC @bracesproul can you have a look?
yindo commented 2026-02-15 17:15:01 -05:00
Author
Owner
Copy Link
Copy Source

@wh1pp3rz commented on GitHub (Jul 24, 2024):

There's an issue when using ChatAnthropic with tools bound and trying to stream, whether regular stream or streamEvents. There seem to be some logic expecting the message content to be a string. The console warning The "tool_calls" field on a message is only respected if content is a string. is being shown. When I dump the content, it's not a string but an object that looks like this:

"content": [
    {
      "index": 0,
      "type": "text_delta",
      "text": "To answer your question about what 4 times 6 is, I can use the \"multiply\" function that's available to me. Let me calculate that for you."
    },
    {
      "index": 1,
      "input": "{\"a\": 4, \"b\": 6}",
      "type": "input_json_delta"
    }
  ]
@wh1pp3rz commented on GitHub (Jul 24, 2024): There's an issue when using ChatAnthropic with tools bound and trying to stream, whether regular stream or streamEvents. There seem to be some logic expecting the message content to be a string. The console warning `The "tool_calls" field on a message is only respected if content is a string.` is being shown. When I dump the content, it's not a string but an object that looks like this: ``` "content": [ { "index": 0, "type": "text_delta", "text": "To answer your question about what 4 times 6 is, I can use the \"multiply\" function that's available to me. Let me calculate that for you." }, { "index": 1, "input": "{\"a\": 4, \"b\": 6}", "type": "input_json_delta" } ] ```
yindo commented 2026-02-15 17:15:01 -05:00
Author
Owner
Copy Link
Copy Source

@jacoblee93 commented on GitHub (Jul 24, 2024):

Thanks for the heads up, will have a look!

@jacoblee93 commented on GitHub (Jul 24, 2024): Thanks for the heads up, will have a look!
yindo commented 2026-02-15 17:15:01 -05:00
Author
Owner
Copy Link
Copy Source

@danny-avila commented on GitHub (Jul 24, 2024):

There's an issue when using ChatAnthropic with tools bound and trying to stream, whether regular stream or streamEvents. There seem to be some logic expecting the message content to be a string. The console warning The "tool_calls" field on a message is only respected if content is a string. is being shown. When I dump the content, it's not a string but an object that looks like this:

"content": [
    {
      "index": 0,
      "type": "text_delta",
      "text": "To answer your question about what 4 times 6 is, I can use the \"multiply\" function that's available to me. Let me calculate that for you."
    },
    {
      "index": 1,
      "input": "{\"a\": 4, \"b\": 6}",
      "type": "input_json_delta"
    }
  ]

Same issue here thanks for sharing

@danny-avila commented on GitHub (Jul 24, 2024): > There's an issue when using ChatAnthropic with tools bound and trying to stream, whether regular stream or streamEvents. There seem to be some logic expecting the message content to be a string. The console warning `The "tool_calls" field on a message is only respected if content is a string.` is being shown. When I dump the content, it's not a string but an object that looks like this: > > ``` > "content": [ > { > "index": 0, > "type": "text_delta", > "text": "To answer your question about what 4 times 6 is, I can use the \"multiply\" function that's available to me. Let me calculate that for you." > }, > { > "index": 1, > "input": "{\"a\": 4, \"b\": 6}", > "type": "input_json_delta" > } > ] > ``` Same issue here thanks for sharing
yindo commented 2026-02-15 17:15:01 -05:00
Author
Owner
Copy Link
Copy Source

@jacoblee93 commented on GitHub (Jul 24, 2024):

Really sorry about this 😕

@jacoblee93 commented on GitHub (Jul 24, 2024): Really sorry about this 😕
yindo commented 2026-02-15 17:15:02 -05:00
Author
Owner
Copy Link
Copy Source

@bracesproul commented on GitHub (Jul 24, 2024):

@danny-avila @wh1pp3rz I just released version 0.2.9 with a fix. Please ping me if you experience any other issues.

@bracesproul commented on GitHub (Jul 24, 2024): @danny-avila @wh1pp3rz I just released version `0.2.9` with a fix. Please ping me if you experience any other issues.
yindo commented 2026-02-15 17:15:02 -05:00
Author
Owner
Copy Link
Copy Source

@wh1pp3rz commented on GitHub (Jul 24, 2024):

Thanks @bracesproul , I'm no longer getting the error, however ChatAnthropic doesn't seem to work as expected. Take this sample code for example:

const graphState = {
    messages: {
        value: (x, y) => x.concat(y),
        default: () => [],
    },
};

const multiplyTool = new DynamicStructuredTool({
    name: "multiply",
    description: "multiply two numbers together",
    schema: z.object({
        a: z.number().describe("the first number to multiply"),
        b: z.number().describe("the second number to multiply"),
    }),
    func: async ({ a, b }) => {
        return (a * b).toString();
    },
});

const tools = [multiplyTool];
const toolNode = new ToolNode(tools);

const model = new ChatOpenAI({
    model: 'gpt-4o-mini',
    temperature: 0,
    streaming: true
}).bindTools(tools);

function shouldContinue(state) {
    const messages = state.messages;
    const lastMessage = messages[messages.length - 1];

    if (lastMessage.tool_calls?.length) {
        return "tools";
    }

    return END;
}

const callModel = async (state, config) => {
    const { messages } = state;
    const prompt = ChatPromptTemplate.fromMessages([
        new SystemMessage('You are a helpful assistant'),
        new MessagesPlaceholder('messages'),
    ]);

    const response = await prompt.pipe(model).invoke({messages}, config);

    return { messages: [ response ] };
}

const workflow = new StateGraph({ channels: graphState })
    .addNode("agent", callModel)
    .addNode("tools", toolNode)
    .addEdge(START, "agent")
    .addConditionalEdges("agent", shouldContinue)
    .addEdge("tools", "agent");

const checkpointer = new MemorySaver();
const app = workflow.compile({ checkpointer });

const inputs = {
    messages: [new HumanMessage("what is 4 times 6?")],
};

const config = {
    configurable: {
       thread_id: "1",
    },
};

const eventStream = await app.streamEvents(inputs, {
    ...config,
    streamMode: "values",
    version: "v2",
});
for await (const { event, data } of eventStream) {
    if (event === "on_chat_model_stream") {
        const msg = data.chunk;
        if (!msg.tool_call_chunks?.length) {
            console.log(msg.content);
        }
    }
}

With ChatOpenAI the proper output is streamed, and looks like this:

4
 times

6
 is

24
.

However when I replace the model with ChatAnthropic and without changing anything else, the output now looks like this:

{"a": 4
, "
b"
: 6}

It seems ChatAnthropic is streaming the tool call, but not the final response.

@wh1pp3rz commented on GitHub (Jul 24, 2024): Thanks @bracesproul , I'm no longer getting the error, however `ChatAnthropic` doesn't seem to work as expected. Take this sample code for example: ``` const graphState = { messages: { value: (x, y) => x.concat(y), default: () => [], }, }; const multiplyTool = new DynamicStructuredTool({ name: "multiply", description: "multiply two numbers together", schema: z.object({ a: z.number().describe("the first number to multiply"), b: z.number().describe("the second number to multiply"), }), func: async ({ a, b }) => { return (a * b).toString(); }, }); const tools = [multiplyTool]; const toolNode = new ToolNode(tools); const model = new ChatOpenAI({ model: 'gpt-4o-mini', temperature: 0, streaming: true }).bindTools(tools); function shouldContinue(state) { const messages = state.messages; const lastMessage = messages[messages.length - 1]; if (lastMessage.tool_calls?.length) { return "tools"; } return END; } const callModel = async (state, config) => { const { messages } = state; const prompt = ChatPromptTemplate.fromMessages([ new SystemMessage('You are a helpful assistant'), new MessagesPlaceholder('messages'), ]); const response = await prompt.pipe(model).invoke({messages}, config); return { messages: [ response ] }; } const workflow = new StateGraph({ channels: graphState }) .addNode("agent", callModel) .addNode("tools", toolNode) .addEdge(START, "agent") .addConditionalEdges("agent", shouldContinue) .addEdge("tools", "agent"); const checkpointer = new MemorySaver(); const app = workflow.compile({ checkpointer }); const inputs = { messages: [new HumanMessage("what is 4 times 6?")], }; const config = { configurable: { thread_id: "1", }, }; const eventStream = await app.streamEvents(inputs, { ...config, streamMode: "values", version: "v2", }); for await (const { event, data } of eventStream) { if (event === "on_chat_model_stream") { const msg = data.chunk; if (!msg.tool_call_chunks?.length) { console.log(msg.content); } } } ``` With ChatOpenAI the proper output is streamed, and looks like this: ``` 4 times 6 is 24 . ``` However when I replace the model with `ChatAnthropic` and without changing anything else, the output now looks like this: ``` {"a": 4 , " b" : 6} ``` It seems ChatAnthropic is streaming the tool call, but not the final response.
yindo commented 2026-02-15 17:15:02 -05:00
Author
Owner
Copy Link
Copy Source

@bracesproul commented on GitHub (Jul 24, 2024):

Thanks @bracesproul , I'm no longer getting the error, however ChatAnthropic doesn't seem to work as expected. Take this sample code for example:


const graphState = {

    messages: {

        value: (x, y) => x.concat(y),

        default: () => [],

    },

};



const multiplyTool = new DynamicStructuredTool({

    name: "multiply",

    description: "multiply two numbers together",

    schema: z.object({

        a: z.number().describe("the first number to multiply"),

        b: z.number().describe("the second number to multiply"),

    }),

    func: async ({ a, b }) => {

        return (a * b).toString();

    },

});



const tools = [multiplyTool];

const toolNode = new ToolNode(tools);



const model = new ChatOpenAI({

    model: 'gpt-4o-mini',

    temperature: 0,

    streaming: true

}).bindTools(tools);



function shouldContinue(state) {

    const messages = state.messages;

    const lastMessage = messages[messages.length - 1];



    if (lastMessage.tool_calls?.length) {

        return "tools";

    }



    return END;

}



const callModel = async (state, config) => {

    const { messages } = state;

    const prompt = ChatPromptTemplate.fromMessages([

        new SystemMessage('You are a helpful assistant'),

        new MessagesPlaceholder('messages'),

    ]);



    const response = await prompt.pipe(model).invoke({messages}, config);



    return { messages: [ response ] };

}



const workflow = new StateGraph({ channels: graphState })

    .addNode("agent", callModel)

    .addNode("tools", toolNode)

    .addEdge(START, "agent")

    .addConditionalEdges("agent", shouldContinue)

    .addEdge("tools", "agent");



const checkpointer = new MemorySaver();

const app = workflow.compile({ checkpointer });



const inputs = {

    messages: [new HumanMessage("what is 4 times 6?")],

};



const config = {

    configurable: {

       thread_id: "1",

    },

};



const eventStream = await app.streamEvents(inputs, {

    ...config,

    streamMode: "values",

    version: "v2",

});

for await (const { event, data } of eventStream) {

    if (event === "on_chat_model_stream") {

        const msg = data.chunk;

        if (!msg.tool_call_chunks?.length) {

            console.log(msg.content);

        }

    }

}

With ChatOpenAI the proper output is streamed, and looks like this:


4

 times



6

 is



24

.

However when I replace the model with ChatAnthropic and without changing anything else, the output now looks like this:


{"a": 4

, "

b"

: 6}

It seems ChatAnthropic is streaming the tool call, but not the final response.

It looks like Anthropic is calling a tool here and OpenAI isn't because the LLM "thinks" it knows the answer? If you send me a LangSmith trace of both I could see what's going on

@bracesproul commented on GitHub (Jul 24, 2024): > Thanks @bracesproul , I'm no longer getting the error, however `ChatAnthropic` doesn't seem to work as expected. Take this sample code for example: > > > > ``` > > const graphState = { > > messages: { > > value: (x, y) => x.concat(y), > > default: () => [], > > }, > > }; > > > > const multiplyTool = new DynamicStructuredTool({ > > name: "multiply", > > description: "multiply two numbers together", > > schema: z.object({ > > a: z.number().describe("the first number to multiply"), > > b: z.number().describe("the second number to multiply"), > > }), > > func: async ({ a, b }) => { > > return (a * b).toString(); > > }, > > }); > > > > const tools = [multiplyTool]; > > const toolNode = new ToolNode(tools); > > > > const model = new ChatOpenAI({ > > model: 'gpt-4o-mini', > > temperature: 0, > > streaming: true > > }).bindTools(tools); > > > > function shouldContinue(state) { > > const messages = state.messages; > > const lastMessage = messages[messages.length - 1]; > > > > if (lastMessage.tool_calls?.length) { > > return "tools"; > > } > > > > return END; > > } > > > > const callModel = async (state, config) => { > > const { messages } = state; > > const prompt = ChatPromptTemplate.fromMessages([ > > new SystemMessage('You are a helpful assistant'), > > new MessagesPlaceholder('messages'), > > ]); > > > > const response = await prompt.pipe(model).invoke({messages}, config); > > > > return { messages: [ response ] }; > > } > > > > const workflow = new StateGraph({ channels: graphState }) > > .addNode("agent", callModel) > > .addNode("tools", toolNode) > > .addEdge(START, "agent") > > .addConditionalEdges("agent", shouldContinue) > > .addEdge("tools", "agent"); > > > > const checkpointer = new MemorySaver(); > > const app = workflow.compile({ checkpointer }); > > > > const inputs = { > > messages: [new HumanMessage("what is 4 times 6?")], > > }; > > > > const config = { > > configurable: { > > thread_id: "1", > > }, > > }; > > > > const eventStream = await app.streamEvents(inputs, { > > ...config, > > streamMode: "values", > > version: "v2", > > }); > > for await (const { event, data } of eventStream) { > > if (event === "on_chat_model_stream") { > > const msg = data.chunk; > > if (!msg.tool_call_chunks?.length) { > > console.log(msg.content); > > } > > } > > } > > ``` > > > > With ChatOpenAI the proper output is streamed, and looks like this: > > ``` > > 4 > > times > > > > 6 > > is > > > > 24 > > . > > ``` > > > > However when I replace the model with `ChatAnthropic` and without changing anything else, the output now looks like this: > > ``` > > {"a": 4 > > , " > > b" > > : 6} > > ``` > > > > It seems ChatAnthropic is streaming the tool call, but not the final response. > > It looks like Anthropic is calling a tool here and OpenAI isn't because the LLM "thinks" it knows the answer? If you send me a LangSmith trace of both I could see what's going on
yindo commented 2026-02-15 17:15:02 -05:00
Author
Owner
Copy Link
Copy Source

@wh1pp3rz commented on GitHub (Jul 24, 2024):

OpenAI is definitely calling a tool. Here's the LangSmith trace

And. here's the trace for ChatAnthropic

@wh1pp3rz commented on GitHub (Jul 24, 2024): OpenAI is definitely calling a tool. Here's the [LangSmith trace](https://smith.langchain.com/public/c6002e77-61dd-45e5-9ab3-2120b6a2353c/r) And. here's the [trace for ChatAnthropic](https://smith.langchain.com/public/3d20678a-f45b-4caa-a581-c99334619886/r)
yindo commented 2026-02-15 17:15:03 -05:00
Author
Owner
Copy Link
Copy Source

@lionkeng commented on GitHub (Jul 25, 2024):

@bracesproul
I have updated to 0.2.9 but it is still not working as expected.
I am looking for some guidance on whether this is a bug in my code or in the Langgraph/LangchainJS implementation.

My code example lives here, nothing fancy and actually looks a lot like the code example in the Langgraph documentation.

Here is a langsmith trace with ChatAnthropic.

Here is the langsmith tracing with the same code but I swapped out ChatAnthropic with ChatOpenAI - and it appears to work there.

If I look at the runs view on the langsmith dashboard, the ChatAnthropic call seems to be running outside of the Langgraph/RunnableLambda nesting.

Screenshot 2024-07-25 at 3 39 53 PM

With ChatOpenAI, the call is nested in the RunnableLambda.
Screenshot 2024-07-25 at 3 48 24 PM

@lionkeng commented on GitHub (Jul 25, 2024): @bracesproul I have updated to 0.2.9 but it is still not working as expected. I am looking for some guidance on whether this is a bug in my code or in the Langgraph/LangchainJS implementation. My code example lives [here](https://github.com/lionkeng/langgraph-starter/tree/main/examples/persistence), nothing fancy and actually looks a lot like the code example in the Langgraph documentation. [Here](https://smith.langchain.com/public/b3e684bc-9d6e-4829-8309-2ddcd0140fbe/r) is a langsmith trace with ChatAnthropic. [Here](https://smith.langchain.com/public/f44f243b-5d21-430d-9cba-1d3e44cb120c/r) is the langsmith tracing with the same code but I swapped out ChatAnthropic with ChatOpenAI - and it appears to work there. If I look at the `runs` view on the langsmith dashboard, the ChatAnthropic call seems to be running outside of the Langgraph/RunnableLambda nesting. <img width="1368" alt="Screenshot 2024-07-25 at 3 39 53 PM" src="https://github.com/user-attachments/assets/b6fce77e-6f5d-4717-84ad-11c32e9dd025"> With ChatOpenAI, the call is nested in the RunnableLambda. <img width="1270" alt="Screenshot 2024-07-25 at 3 48 24 PM" src="https://github.com/user-attachments/assets/f343fecd-87c9-4183-af47-f31c855ff256">
yindo commented 2026-02-15 17:15:03 -05:00
Author
Owner
Copy Link
Copy Source

@jacoblee93 commented on GitHub (Jul 25, 2024):

Ah @wh1pp3rz I think Anthropic is streaming back a chunk where content is an array instead of a string, and LangSmith is not rendering it properly. I'll flag that with the team, but this is within our spec - I can see how it is confusing but Anthropic could theoretically call another tool in a block after that call, so we can't just stream back string chunks here. See the Raw Output of your trace below.

Screenshot 2024-07-25 at 2 23 58 PM

@lionkeng can you try setting resolutions as described here?

https://langchain-ai.github.io/langgraphjs/how-tos/manage-ecosystem-dependencies/

@jacoblee93 commented on GitHub (Jul 25, 2024): Ah @wh1pp3rz I think Anthropic is streaming back a chunk where `content` is an array instead of a string, and LangSmith is not rendering it properly. I'll flag that with the team, but this is within our spec - I can see how it is confusing but Anthropic could theoretically call another tool in a block after that call, so we can't just stream back string chunks here. See the `Raw Output` of your trace below. <img width="946" alt="Screenshot 2024-07-25 at 2 23 58 PM" src="https://github.com/user-attachments/assets/db80ef63-f2cc-4f10-9342-35be25e327e8"> @lionkeng can you try setting `resolutions` as described here? https://langchain-ai.github.io/langgraphjs/how-tos/manage-ecosystem-dependencies/
yindo commented 2026-02-15 17:15:03 -05:00
Author
Owner
Copy Link
Copy Source

@wh1pp3rz commented on GitHub (Jul 25, 2024):

@jacoblee93 what would be the proper way to stream the final result of a chain when using anthropic with tools bound and streamEvents v2?

@wh1pp3rz commented on GitHub (Jul 25, 2024): @jacoblee93 what would be the proper way to stream the final result of a chain when using anthropic with tools bound and streamEvents v2?
yindo commented 2026-02-15 17:15:03 -05:00
Author
Owner
Copy Link
Copy Source

@lionkeng commented on GitHub (Jul 25, 2024):

@jacoblee93
I can try setting resolutions. But what should be the right combination of versions to use for the different packages?
Here are the relevant versions of packages from the langchain eco that I am using as shown by a pnpm list.

dependencies:
@langchain/anthropic 0.2.9
@langchain/community 0.2.20
@langchain/core 0.2.18
@langchain/langgraph 0.0.26
@langchain/openai 0.0.34
better-sqlite3 9.6.0
langchain 0.2.11
langsmith 0.1.39
@lionkeng commented on GitHub (Jul 25, 2024): @jacoblee93 I _can_ try setting resolutions. But what should be the right combination of versions to use for the different packages? Here are the relevant versions of packages from the langchain eco that I am using as shown by a `pnpm list`. ``` dependencies: @langchain/anthropic 0.2.9 @langchain/community 0.2.20 @langchain/core 0.2.18 @langchain/langgraph 0.0.26 @langchain/openai 0.0.34 better-sqlite3 9.6.0 langchain 0.2.11 langsmith 0.1.39 ```
yindo commented 2026-02-15 17:15:03 -05:00
Author
Owner
Copy Link
Copy Source

@danny-avila commented on GitHub (Jul 27, 2024):

After this update (0.2.10), anthropic will stream with bound tools but it will not stream tool chunks:

https://github.com/langchain-ai/langchainjs/pull/6206

@danny-avila commented on GitHub (Jul 27, 2024): After this update (0.2.10), anthropic will stream with bound tools but it will not stream tool chunks: https://github.com/langchain-ai/langchainjs/pull/6206
yindo commented 2026-02-15 17:15:04 -05:00
Author
Owner
Copy Link
Copy Source

@bracesproul commented on GitHub (Jul 29, 2024):

After this update (0.2.10), anthropic will stream with bound tools but it will not stream tool chunks:

Could you expand on what you mean by this? Streams will contain tool chunks, but binding tools to the model doesn't do anything to streaming.

@bracesproul commented on GitHub (Jul 29, 2024): > After this update (0.2.10), anthropic will stream with bound tools but it will not stream tool chunks: Could you expand on what you mean by this? Streams will contain tool chunks, but binding tools to the model doesn't do anything to streaming.
yindo commented 2026-02-15 17:15:04 -05:00
Author
Owner
Copy Link
Copy Source

@bracesproul commented on GitHub (Jul 29, 2024):

@jacoblee93 what would be the proper way to stream the final result of a chain when using anthropic with tools bound and streamEvents v2?

Could you open a new issue for "How to stream the final result with anthropic" in the langchain-ai/langchainjs repo

@bracesproul commented on GitHub (Jul 29, 2024): > @jacoblee93 what would be the proper way to stream the final result of a chain when using anthropic with tools bound and streamEvents v2? Could you open a new issue for "How to stream the final result with anthropic" in the langchain-ai/langchainjs repo
yindo commented 2026-02-15 17:15:04 -05:00
Author
Owner
Copy Link
Copy Source

@wh1pp3rz commented on GitHub (Jul 29, 2024):

After this update (0.2.10), anthropic will stream with bound tools but it will not stream tool chunks:

Could you expand on what you mean by this? Streams will contain tool chunks, but binding tools to the model doesn't do anything to streaming.

Yeah this is working sort of now, the graph is streaming when tools are bound, but the streamed output contains the tool input parameters appended at the end and it looks weird for an end user. As an example, a simple multiply tool called with "what is 2 * 3?" streamed the following:

To
 answer
 this question, I
'll
 use
 the multiplyer function
 to calculate
 2 *
 3.
{"a"
: 2
, "
b":
3}


The
 result of 2
 * 3
 is 6
.

Using streamEvents, the tool inputs are streamed under the on_chat_model_stream event, and I don't see a way to distinguish or filter them to not be in the final output to the user.

Here's the trace if that helps.

@wh1pp3rz commented on GitHub (Jul 29, 2024): > > After this update (0.2.10), anthropic will stream with bound tools but it will not stream tool chunks: > > Could you expand on what you mean by this? Streams will contain tool chunks, but binding tools to the model doesn't do anything to streaming. Yeah this is working sort of now, the graph is streaming when tools are bound, but the streamed output contains the tool input parameters appended at the end and it looks weird for an end user. As an example, a simple multiply tool called with "what is 2 * 3?" streamed the following: ``` To answer this question, I 'll use the multiplyer function to calculate 2 * 3. {"a" : 2 , " b": 3} The result of 2 * 3 is 6 . ``` Using `streamEvents`, the tool inputs are streamed under the `on_chat_model_stream` event, and I don't see a way to distinguish or filter them to not be in the final output to the user. Here's the [trace](https://smith.langchain.com/public/04b0ef8b-2680-4e66-85e8-68909db60e15/r) if that helps.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
checkpointer
docs-needed
documentation
enhancement
good first issue
good writeup
help wanted
invalid
langgraph-supervisor
open-swe-auto
platform
prebuilt
pull-request
Mirrored from GitHub Pull Request
 question
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langchain-ai/langgraphjs#46
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?