Add way to abort a streaming response from chat / generate #110

New Issue

2026-02-15T16:28:02-05:00

yindo commented

2026-02-15 16:28:02 -05:00

Originally created by @paulrobello on GitHub (Jul 15, 2024).

I don't see any way to abort a streaming response from chat / generate methods

Is this or will this be supported?

Given the following snippet how can you properly abort the stream? Simply breaking from the for chunk loop is not enough.

           stream = ollama.chat(
                model=model_name,
                messages=messages,
                stream=True,
           )
           msg_content = ""
           for chunk in stream:
               msg_content += chunk["message"]["content"]

Originally created by @paulrobello on GitHub (Jul 15, 2024). I don't see any way to abort a streaming response from chat / generate methods Is this or will this be supported? Given the following snippet how can you properly abort the stream? Simply breaking from the for chunk loop is not enough. ```python stream = ollama.chat( model=model_name, messages=messages, stream=True, ) msg_content = "" for chunk in stream: msg_content += chunk["message"]["content"] ```

yindo closed this issue

2026-02-15 16:28:02 -05:00

yindo commented

2026-02-15 16:28:03 -05:00

@antoninoLorenzo commented on GitHub (Jul 26, 2024):

Technically there is no way of doing it, as you can see in that issue on llama.cpp repository; the "brute-force" way is to add a KeyboardInterrupt during the stream, for example:

try:
    for chunk in stream:
        print(chunk["message"]["content"], end='')
except KeyboardInterrupt:
    pass

However that approach may cause problems, the main one is that probably the Ollama server will continue sending stuff, however given that the stream is closed it will send chunks to nothing. To better understand what could happen you should have either httpx knowledge (what _client.py uses to make requests) or Go knowledge (what Ollama is built in).

@antoninoLorenzo commented on GitHub (Jul 26, 2024): Technically there is no way of doing it, as you can see in that [issue](https://github.com/ggerganov/llama.cpp/discussions/6212) on `llama.cpp` repository; the "brute-force" way is to add a `KeyboardInterrupt` during the stream, for example: ```python try: for chunk in stream: print(chunk["message"]["content"], end='') except KeyboardInterrupt: pass ``` However that approach may cause problems, the main one is that probably the Ollama server will continue sending stuff, however given that the stream is closed it will send chunks to nothing. To better understand what could happen you should have either `httpx` knowledge (what `_client.py` uses to make requests) or Go knowledge (what `Ollama` is built in).

yindo commented

2026-02-15 16:28:03 -05:00

@mxyng commented on GitHub (Jul 30, 2024):

exiting the generator is sufficient to abort the request and stop generation, e.g.

           for chunk in stream:
               msg_content += chunk["message"]["content"]
               if should_break():
                   break

@mxyng commented on GitHub (Jul 30, 2024): exiting the generator is sufficient to abort the request and stop generation, e.g. ```python for chunk in stream: msg_content += chunk["message"]["content"] if should_break(): break ```

yindo commented

2026-02-15 16:28:03 -05:00

@WabaScript2 commented on GitHub (Apr 4, 2025):

Hi Mxyng, does this solution actually terminate the response generation as you would expect when using the API directly and hitting CTRL + C?
If not, we break the stream, but the model is still using resources on the machine until its response is complete. I'm not sure about the python sdk, but the JS one does offer a proper abort method: https://github.com/ollama/ollama-js/blob/main/examples/abort/abort-single-request.ts

@WabaScript2 commented on GitHub (Apr 4, 2025): Hi Mxyng, does this solution actually terminate the response generation as you would expect when using the API directly and hitting CTRL + C? If not, we break the stream, but the model is still using resources on the machine until its response is complete. I'm not sure about the python sdk, but the JS one does offer a proper abort method: https://github.com/ollama/ollama-js/blob/main/examples/abort/abort-single-request.ts

yindo commented

2026-02-15 16:28:03 -05:00

@whs commented on GitHub (Jun 22, 2025):

Checking in Wireshark, it seems that Ollama Python doesn't actually stop the stream after the iterator is aborted.

It seems that Python intentionally do not stop the stream when you break out of it - you can still start another loop to continue reading from the stream.

@whs commented on GitHub (Jun 22, 2025): Checking in Wireshark, it seems that Ollama Python doesn't actually stop the stream after the iterator is aborted. It seems that Python intentionally do not stop the stream when you break out of it - you can still start *another* loop to continue reading from the stream.

yindo commented

2026-02-15 16:28:03 -05:00

@whs commented on GitHub (Jun 22, 2025):

I think I've found a workaround.

Simply throw an exception where you'd use break, then outside the async for you catch it:

response = await ollama.AsyncClient().chat(...)
try:
    async for chunk in response:
        raise StopIteration
except StopIteration:
    pass

I can see that RST packets do get sent this way, and the server response with context canceled.

@whs commented on GitHub (Jun 22, 2025): I think I've found a workaround. Simply throw an exception where you'd use `break`, then outside the `async for` you catch it: ```py response = await ollama.AsyncClient().chat(...) try: async for chunk in response: raise StopIteration except StopIteration: pass ``` I can see that RST packets do get sent this way, and the server response with context canceled.

yindo commented

2026-02-15 16:28:03 -05:00

@ryamldess commented on GitHub (Oct 8, 2025):

I tried the StopIteration example above; does not work. Creating a task does not work. Break does not work. I need a way to kill a chat when context panics occur.

@ryamldess commented on GitHub (Oct 8, 2025): I tried the StopIteration example above; does not work. Creating a task does not work. Break does not work. I need a way to kill a chat when context panics occur.

yindo referenced this issue

2026-02-15 16:30:06 -05:00

[PR #110] [MERGED] Bump pillow from 10.2.0 to 10.3.0 #360

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: ollama/ollama-python#110