langchain-ai/langgraphjs
1
0
Fork 0
mirror of https://github.com/langchain-ai/langgraphjs.git synced 2026-07-22 17:15:25 -04:00
Code Issues 108 Packages Projects Releases 20 Wiki Activity

withStructuredOutput does not accept tools #149

New Issue
Closed
opened 2026-02-15 17:16:20 -05:00 by yindo · 11 comments
yindo commented 2026-02-15 17:16:20 -05:00
Owner
Copy Link
Copy Source

Originally created by @TiagoGouvea on GitHub (Dec 22, 2024).

When using withStructuredOutput, the tools bind to the llm are ignored.

If we add a tools to the structured schema as a workaround it almost never works as expected.

On the invoke method, if it's a withStructuredOutput it will declare just one tool, to force the model to call a parser method to the response.

  const llm = new ChatOpenAI({ modelName: 'gpt-4o-mini' });
  llm.bindTools(tools}; // these tools will be ignored
  const llmResult = await llm
    .withStructuredOutput(outputSchema, { name: 'Response' })
    .invoke(messages);

Could be a way to use withStructuredOutput with custom tools as well.

Originally created by @TiagoGouvea on GitHub (Dec 22, 2024). When using `withStructuredOutput`, the tools bind to the llm are ignored. If we add a `tools` to the structured schema as a workaround it almost never works as expected. On the `invoke` method, if it's a `withStructuredOutput` it will declare just one tool, to force the model to call a parser method to the response. ```typescript const llm = new ChatOpenAI({ modelName: 'gpt-4o-mini' }); llm.bindTools(tools}; // these tools will be ignored const llmResult = await llm .withStructuredOutput(outputSchema, { name: 'Response' }) .invoke(messages); ``` Could be a way to use `withStructuredOutput` with custom tools as well.
yindo closed this issue 2026-02-15 17:16:20 -05:00
yindo commented 2026-02-15 17:16:21 -05:00
Author
Owner
Copy Link
Copy Source

@jacoblee93 commented on GitHub (Dec 23, 2024):

Thanks for the issue!

What would be the expected behavior here?

@jacoblee93 commented on GitHub (Dec 23, 2024): Thanks for the issue! What would be the expected behavior here?
yindo commented 2026-02-15 17:16:21 -05:00
Author
Owner
Copy Link
Copy Source

@TiagoGouvea commented on GitHub (Dec 24, 2024):

Using withStructuredOutput, it never add the tools. It add only one tool extract (that enforces the structured output) other than the tools bind before.

  withStructuredOutput(outputSchema, config) {
   ....
               llm = this.bind({
                    tools: [
                        {
                            type: "function",
                            function: {
                                name: functionName,
                                description: asJsonSchema.description,
                                parameters: asJsonSchema,
                            },
                        },
                    ],
                    tool_choice: {
                        type: "function",
                        function: {
                            name: functionName,
                        },
                    },
                    // Do not pass `strict` argument to OpenAI if `config.strict` is undefined
                    ...(config?.strict !== undefined ? { strict: config.strict } : {}),
                });
   ...
  }

The expected behavior is allow the llm to call the bind tools.

@TiagoGouvea commented on GitHub (Dec 24, 2024): Using `withStructuredOutput`, it never add the tools. It add only one tool `extract` (that enforces the structured output) other than the tools bind before. ```typescript withStructuredOutput(outputSchema, config) { .... llm = this.bind({ tools: [ { type: "function", function: { name: functionName, description: asJsonSchema.description, parameters: asJsonSchema, }, }, ], tool_choice: { type: "function", function: { name: functionName, }, }, // Do not pass `strict` argument to OpenAI if `config.strict` is undefined ...(config?.strict !== undefined ? { strict: config.strict } : {}), }); ... } ``` The expected behavior is allow the llm to call the bind tools.
yindo commented 2026-02-15 17:16:22 -05:00
Author
Owner
Copy Link
Copy Source

@jacoblee93 commented on GitHub (Dec 24, 2024):

allow the llm to call the bind tools.

So it would either return a tool message or structured output?

@jacoblee93 commented on GitHub (Dec 24, 2024): > allow the llm to call the bind tools. So it would either return a tool message or structured output?
yindo commented 2026-02-15 17:16:22 -05:00
Author
Owner
Copy Link
Copy Source

@TiagoGouvea commented on GitHub (Dec 24, 2024):

My ideia is to have the structured output, but, if I need to call some tool receive the tool_calls as well.

@TiagoGouvea commented on GitHub (Dec 24, 2024): My ideia is to have the structured output, but, if I need to call some tool receive the `tool_calls` as well.
yindo commented 2026-02-15 17:16:22 -05:00
Author
Owner
Copy Link
Copy Source

@TiagoGouvea commented on GitHub (Dec 24, 2024):

Let me explain better. I'm implementing a multi agent experiment, using the Command and the goto structure like showed on lanchain youtube video.

I loved the Command idea with the json object as return, made the life easier for me. Thank you for that!

On my project each agent will have access to some tools.

On my tests, when I use withStructuredOutput with my desired schema to return, it doesn't include the tools.

I work arround calling OpenAi library directly, with response_format parameter and it worked like expected, but on my project each agent could use a different model, so I would need implement the call for each provider and so.

What do you think?

@TiagoGouvea commented on GitHub (Dec 24, 2024): Let me explain better. I'm implementing a multi agent experiment, using the `Command` and the `goto` structure like showed on lanchain youtube video. I loved the Command idea with the json object as return, made the life easier for me. Thank you for that! On my project each agent will have access to some tools. On my tests, when I use `withStructuredOutput` with my desired schema to return, it doesn't include the tools. I work arround calling OpenAi library directly, with `response_format` parameter and it worked like expected, but on my project each agent could use a different model, so I would need implement the call for each provider and so. What do you think?
yindo commented 2026-02-15 17:16:23 -05:00
Author
Owner
Copy Link
Copy Source

@jasontjahjono commented on GitHub (Dec 30, 2024):

allow the llm to call the bind tools.

So it would either return a tool message or structured output?

Hi Jacob, I really need this functionality! LLM should return a tool call (AI Message calling a tool) or Structured Output. Do you know if this is in progress, or planned to be implemented?

@jasontjahjono commented on GitHub (Dec 30, 2024): > > allow the llm to call the bind tools. > > So it would either return a tool message or structured output? Hi Jacob, I really need this functionality! LLM should return a tool call (AI Message calling a tool) or Structured Output. Do you know if this is in progress, or planned to be implemented?
yindo commented 2026-02-15 17:16:23 -05:00
Author
Owner
Copy Link
Copy Source

@jacoblee93 commented on GitHub (Dec 30, 2024):

It's not super straightforward/generalizable - best bet at the moment is to attach a response or default tool in addition to other tools you want the model to be able to call

@jacoblee93 commented on GitHub (Dec 30, 2024): It's not super straightforward/generalizable - best bet at the moment is to attach a `response` or `default` tool in addition to other tools you want the model to be able to call
yindo commented 2026-02-15 17:16:23 -05:00
Author
Owner
Copy Link
Copy Source

@TiagoGouvea commented on GitHub (Jan 30, 2025):

Right! I'm trying different approachiess and learning a little more. Thank you @jacoblee93 for your help.

@TiagoGouvea commented on GitHub (Jan 30, 2025): Right! I'm trying different approachiess and learning a little more. Thank you @jacoblee93 for your help.
yindo commented 2026-02-15 17:16:23 -05:00
Author
Owner
Copy Link
Copy Source

@jacoblee93 commented on GitHub (Jan 30, 2025):

We've been thinking through some ways to e.g. pass this directly into an agent, which is what this would allow, so leaving this open.

@jacoblee93 commented on GitHub (Jan 30, 2025): We've been thinking through some ways to e.g. pass this directly into an agent, which is what this would allow, so leaving this open.
yindo commented 2026-02-15 17:16:24 -05:00
Author
Owner
Copy Link
Copy Source

@msevera commented on GitHub (Feb 17, 2025):

@TiagoGouvea I think you might achieve it with this code.

const model = new ChatOpenAI({
      model: 'gpt-4o-mini',
      temperature: 0,
      modelKwargs: {
        response_format: {
          "type": "json_schema",
          "json_schema": {
            "name": "response_schema",
            "strict": true,
            "schema": {
              // your schema
            }
          }
        },
      }
    }).bindTools([toolA, toolB, toolC])
@msevera commented on GitHub (Feb 17, 2025): @TiagoGouvea I think you might achieve it with this code. ```javascript const model = new ChatOpenAI({ model: 'gpt-4o-mini', temperature: 0, modelKwargs: { response_format: { "type": "json_schema", "json_schema": { "name": "response_schema", "strict": true, "schema": { // your schema } } }, } }).bindTools([toolA, toolB, toolC]) ```
yindo commented 2026-02-15 17:16:24 -05:00
Author
Owner
Copy Link
Copy Source

@michaelKurowski commented on GitHub (Oct 10, 2025):

Looks like the topic has been closed but no PR is mentioned. Is this fixed in the new version? @dqbd

@michaelKurowski commented on GitHub (Oct 10, 2025): Looks like the topic has been closed but no PR is mentioned. Is this fixed in the new version? @dqbd
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
checkpointer
docs-needed
documentation
enhancement
good first issue
good writeup
help wanted
invalid
langgraph-supervisor
open-swe-auto
platform
prebuilt
pull-request
Mirrored from GitHub Pull Request
 question
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langchain-ai/langgraphjs#149
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?