Thinking with tool calling is problematic #279

New Issue

2026-02-15T16:29:31-05:00

yindo commented

2026-02-15 16:29:31 -05:00

Originally created by @ggozad on GitHub (Jun 19, 2025).

Hey there!
This came as an issue in oterm, but I think it's an issue with Ollama.

The setup is a "thinking" model with tools available, tested with qwen for example and a couple of tools.
Sequence:

The user submits a message, so history has the message with role "user".
We perform a streaming request.
The LLM performs some thinking first deciding it needs to call some tools. The common practice (and what oterm does) is to show the thinking but not create a message with the role "assistant".
Tool calls are performed and messages are added to the history with role "tool"
A new request to streaming is performed but the "thinking" performed in (3) is missing obviously.
The llm does some more thinking again and replies with a message with role "assistant".

Given the thinking performed in (3) is gone, if it was essential for the model eventual response things can go wrong. For simple things the LLM can figure it out given it has the tool calls available but that is not always the case.

What should be the fix here? I am reluctant to make thinking part of the message history with role "assistant". Thinking is often verbose and it's an internal process really. But clearly it should be kept somehow when generating the final response.
Should a "thought" role be introduced? Should thinking be kept with role "assistant" in tags?

Originally created by @ggozad on GitHub (Jun 19, 2025). Hey there! This came as an [issue](https://github.com/ggozad/oterm/issues/254) in oterm, but I *think* it's an issue with Ollama. The setup is a "thinking" model with tools available, tested with `qwen` for example and a couple of tools. Sequence: 1. The user submits a message, so history has the message with role "user". 2. We perform a streaming request. 3. The LLM performs some thinking first deciding it needs to call some tools. The common practice (and what oterm does) is to show the thinking but not create a message with the role "assistant". 4. Tool calls are performed and messages are added to the history with role "tool" 5. A new request to streaming is performed but the "thinking" performed in (3) is missing obviously. 6. The llm does some more thinking again and replies with a message with role "assistant". Given the thinking performed in (3) is gone, if it was essential for the model eventual response things can go wrong. For simple things the LLM can figure it out given it has the tool calls available but that is not always the case. What should be the fix here? I am reluctant to make thinking part of the message history with role "assistant". Thinking is often verbose and it's an internal process really. But clearly it should be kept somehow when generating the final response. Should a "thought" role be introduced? Should thinking be kept with role "assistant" in <think/> tags?

yindo closed this issue

2026-02-15 16:29:31 -05:00

yindo commented

2026-02-15 16:29:31 -05:00

@ParthSareen commented on GitHub (Jun 24, 2025):

I believe we are discarding messages in the history on our end and that's what is recommended by model makers. I've been prototyping in the same area and so will see what some best practices with our Ollama will look like. Do let me know if something has been working well for you!

@ParthSareen commented on GitHub (Jun 24, 2025): I believe we are discarding messages in the history on our end and that's what is recommended by model makers. I've been prototyping in the same area and so will see what some best practices with our Ollama will look like. Do let me know if something has been working well for you!

yindo commented

2026-02-15 16:29:32 -05:00

@drifkin commented on GitHub (Jun 27, 2025):

I think it'd be best to create an assistant message that contains the "thinking" field. For updated model templates that support thinking, we filter out any thinking in the history, unless it's after the last user message for exactly this case (see lines 42-44 here: https://ollama.com/library/qwen3:latest/blobs/ae370d884f10). I think the interesting part here is that the assistant message could be completely empty once its thinking is filtered out and the models might not like that empty message before the tool calls, perhaps we'll need to filter those empty assistant messages entirely.

This approach of keeping only thoughts that are relevant to the current "in progress" assistant message is followed by other model providers also, e.g. deepseek https://api-docs.deepseek.com/guides/reasoning_model#multi-round-conversation, anthropic https://docs.anthropic.com/en/docs/build-with-claude/extended-thinking#extended-thinking-with-tool-use

@drifkin commented on GitHub (Jun 27, 2025): I think it'd be best to create an assistant message that contains the "thinking" field. For updated model templates that support thinking, we filter out any thinking in the history, _unless_ it's after the last user message for exactly this case (see lines 42-44 here: https://ollama.com/library/qwen3:latest/blobs/ae370d884f10). I think the interesting part here is that the assistant message could be completely empty once its thinking is filtered out and the models might not like that empty message before the tool calls, perhaps we'll need to filter those empty assistant messages entirely. This approach of keeping only thoughts that are relevant to the current "in progress" assistant message is followed by other model providers also, e.g. deepseek <https://api-docs.deepseek.com/guides/reasoning_model#multi-round-conversation>, anthropic <https://docs.anthropic.com/en/docs/build-with-claude/extended-thinking#extended-thinking-with-tool-use>

yindo commented

2026-02-15 16:29:32 -05:00

@ggozad commented on GitHub (Jun 29, 2025):

I am thinking of doing essentially what @drifkin suggests for oterm, but it feels slightly hacky:
When thinking is returned, add it wrapped in <think/> as a message in the history with role "assistant". Add the tool responses and wait for additional thinking and final response. Then only persist the final response in the database and clear the history of the thinking parts.
It would be nicer if this was somehow done by Ollama instead.

@ggozad commented on GitHub (Jun 29, 2025): I am thinking of doing essentially what @drifkin suggests for oterm, but it feels slightly hacky: When thinking is returned, add it wrapped in `<think/>` as a message in the history with role "assistant". Add the tool responses and wait for additional thinking and final response. Then only persist the final response in the database and clear the history of the thinking parts. It would be nicer if this was somehow done by Ollama instead.

yindo commented

2026-02-15 16:29:32 -05:00

@drifkin commented on GitHub (Jun 30, 2025):

I am thinking of doing essentially what @drifkin suggests for oterm, but it feels slightly hacky: When thinking is returned, add it wrapped in <think/> as a message in the history with role "assistant". Add the tool responses and wait for additional thinking and final response. Then only persist the final response in the database and clear the history of the thinking parts. It would be nicer if this was somehow done by Ollama instead.

If I'm understanding this case correctly, Ollama should handle this automatically for thinking models (i.e., those with official support in their templates). All you need to do is store the messages the model returns to you (which will have an "assistant" role and contain a thinking field) and provide them in the history for subsequent calls. Ollama will automatically decide when to show the thinking history to the model (i.e., it'll only show thinking history that occurred after the most recent user query). The goal is that as the caller you shouldn't need to modify the messages returned to you from the model

@drifkin commented on GitHub (Jun 30, 2025): > I am thinking of doing essentially what [@drifkin](https://github.com/drifkin) suggests for oterm, but it feels slightly hacky: When thinking is returned, add it wrapped in `<think/>` as a message in the history with role "assistant". Add the tool responses and wait for additional thinking and final response. Then only persist the final response in the database and clear the history of the thinking parts. It would be nicer if this was somehow done by Ollama instead. If I'm understanding this case correctly, Ollama should handle this automatically for [thinking models](https://ollama.com/search?c=thinking) (i.e., those with official support in their templates). All you need to do is store the messages the model returns to you (which will have an "assistant" role and contain a `thinking` field) and provide them in the history for subsequent calls. Ollama will automatically decide when to show the thinking history to the model (i.e., it'll only show thinking history that occurred after the most recent user query). The goal is that as the caller you shouldn't need to modify the messages returned to you from the model

yindo commented

2026-02-15 16:29:32 -05:00

@ggozad commented on GitHub (Jul 1, 2025):

I am thinking of doing essentially what @drifkin suggests for oterm, but it feels slightly hacky: When thinking is returned, add it wrapped in <think/> as a message in the history with role "assistant". Add the tool responses and wait for additional thinking and final response. Then only persist the final response in the database and clear the history of the thinking parts. It would be nicer if this was somehow done by Ollama instead.

If I'm understanding this case correctly, Ollama should handle this automatically for thinking models (i.e., those with official support in their templates). All you need to do is store the messages the model returns to you (which will have an "assistant" role and contain a thinking field) and provide them in the history for subsequent calls. Ollama will automatically decide when to show the thinking history to the model (i.e., it'll only show thinking history that occurred after the most recent user query). The goal is that as the caller you shouldn't need to modify the messages returned to you from the model

Thank you @drifkin I hadn't actually noticed the thinking attribute. This should do the job for my use case :)

@ggozad commented on GitHub (Jul 1, 2025): > > I am thinking of doing essentially what [@drifkin](https://github.com/drifkin) suggests for oterm, but it feels slightly hacky: When thinking is returned, add it wrapped in `<think/>` as a message in the history with role "assistant". Add the tool responses and wait for additional thinking and final response. Then only persist the final response in the database and clear the history of the thinking parts. It would be nicer if this was somehow done by Ollama instead. > > If I'm understanding this case correctly, Ollama should handle this automatically for [thinking models](https://ollama.com/search?c=thinking) (i.e., those with official support in their templates). All you need to do is store the messages the model returns to you (which will have an "assistant" role and contain a `thinking` field) and provide them in the history for subsequent calls. Ollama will automatically decide when to show the thinking history to the model (i.e., it'll only show thinking history that occurred after the most recent user query). The goal is that as the caller you shouldn't need to modify the messages returned to you from the model Thank you @drifkin I hadn't actually noticed the `thinking` attribute. This should do the job for my use case :)

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: ollama/ollama-python#279