Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 22:23:50 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #998] [BUG]: Stop button does not stop Ollama backend #611

New Issue
Closed
opened 2026-02-22 18:20:27 -05:00 by yindo · 6 comments
yindo commented 2026-02-22 18:20:27 -05:00
Owner
Copy Link
Copy Source

Originally created by @sumitsodhi88 on GitHub (Mar 30, 2024).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/998

How are you running AnythingLLM?

AnythingLLM desktop app

What happened?

So when I use a stop button the inference is not reflected in the chatbox, but in the backend the Ollama is still working on the inference. Now when I ask a new question the Ollama is stil working on the old question.
The STOP button thus not implemented correctly in case of ollama. The system should send abort to the Ollama too.
Screenshot from 2024-03-30 13-55-46

Are there known steps to reproduce?

No response

Originally created by @sumitsodhi88 on GitHub (Mar 30, 2024). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/998 ### How are you running AnythingLLM? AnythingLLM desktop app ### What happened? So when I use a stop button the inference is not reflected in the chatbox, but in the backend the Ollama is still working on the inference. Now when I ask a new question the Ollama is stil working on the old question. The STOP button thus not implemented correctly in case of ollama. The system should send abort to the Ollama too. ![Screenshot from 2024-03-30 13-55-46](https://github.com/Mintplex-Labs/anything-llm/assets/149290101/93c7b342-99d1-4bc7-af5f-e3548c47900f) ### Are there known steps to reproduce? _No response_
yindo added the possible bug label 2026-02-22 18:20:27 -05:00
yindo closed this issue 2026-02-22 18:20:27 -05:00
yindo commented 2026-02-22 18:20:27 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Mar 31, 2024):

Does Ollama even have an abort API? https://github.com/ollama/ollama/blob/main/docs/api.md#generate-a-chat-completion

Most LLM providers dont and its likely not a great idea to stop/start the service every time you want to halt a chat. Is there an endpoint to do that because i would be happy to force-stop on pause.

@timothycarambat commented on GitHub (Mar 31, 2024): Does Ollama even have an abort API? https://github.com/ollama/ollama/blob/main/docs/api.md#generate-a-chat-completion Most LLM providers dont and its likely not a great idea to stop/start the service every time you want to halt a chat. Is there an endpoint to do that because i would be happy to force-stop on pause.
yindo commented 2026-02-22 18:20:27 -05:00
Author
Owner
Copy Link
Copy Source

@CharlesBdg commented on GitHub (Apr 1, 2024):

This should help : https://github.com/ollama/ollama/issues/1695

@CharlesBdg commented on GitHub (Apr 1, 2024): This should help : https://github.com/ollama/ollama/issues/1695
yindo commented 2026-02-22 18:20:27 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Apr 1, 2024):

We do call abort on the response, which does exactly as that thread perscribes, so i think the issue is related to CPU inferencing (comment)

@timothycarambat commented on GitHub (Apr 1, 2024): We do call abort on the response, which does exactly as that thread perscribes, so i think the issue is related to CPU inferencing ([comment](https://github.com/open-webui/open-webui/issues/1166#issuecomment-2016946749))
yindo commented 2026-02-22 18:20:27 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Apr 1, 2024):

Closing as it seems that the ollama thread (and other fixes like this) do not stop CPU inferencing and that seems to be tied to how llamacpp sends abort requests, which seems to only take effect with GPU inferencing

@timothycarambat commented on GitHub (Apr 1, 2024): Closing as it seems that the ollama thread (and other fixes like this) do not stop CPU inferencing and that seems to be tied to how llamacpp sends abort requests, which seems to only take effect with GPU inferencing
yindo commented 2026-02-22 18:20:27 -05:00
Author
Owner
Copy Link
Copy Source

@17Reset commented on GitHub (Jun 3, 2024):

Also encountering this kind of problem, there is a situation where this must be done: when the LLM is in error and keeps returning the wrong content, this is when the stop button should be necessary, in the case of Ollama's api calls, if this kind of problem is encountered, it is possible to locally terminate the process or thread of the api call to turn off the output of the big model on the server, and then Anythingllm doesn't seem to support.

@17Reset commented on GitHub (Jun 3, 2024): Also encountering this kind of problem, there is a situation where this must be done: when the LLM is in error and keeps returning the wrong content, this is when the stop button should be necessary, in the case of Ollama's api calls, if this kind of problem is encountered, it is possible to locally terminate the process or thread of the api call to turn off the output of the big model on the server, and then Anythingllm doesn't seem to support.
yindo changed title from [BUG]: Stop button does not stop Ollama backend to [GH-ISSUE #998] [BUG]: Stop button does not stop Ollama backend 2026-06-05 14:36:16 -04:00
yindo commented 2026-06-05 14:36:17 -04:00
Author
Owner
Copy Link
Copy Source

@workturnedplay commented on GitHub (May 18, 2026):

This isn't only due to CPU inferencing, if I use a model that fits into VRAM and uses exclusively the GPU (like https://ollama.com/library/granite4.1:3b-q2_K ) it still happens (Stop doesn't cause GPU to go down from 98% usage, and model is for example stuck in a loop and keeps outputting but anythingllm doesn't show it due to Stop, but ollama keeps running it) and the only workaround to stop it is to quit AnythingLLM (ie. from systray). How about a workaround: disconnect and reconnect to ollama? like cline is doing:

Cline does call abort() on the request when you cancel a task — this was fixed in v3.41.0. What that does is close Cline's HTTP connection to Ollama. The problem is Ollama's server doesn't stop generating when the connection drops — it queues new requests and keeps the current one running until it finishes. This is specific to Ollama's architecture; llama-server handles connection drops differently, which is why it stops cleanly.
quote src

I can confirm the connection (between anythingllm and ollama serve) is kept Established (seen via TinyWall) even though Stop was pressed. That's why Quit works because it closes the connection and then ollama serve shows only this line when that happens:
[GIN] 2026/05/18 - 21:33:18 | 200 | 2m8s | 127.0.0.1 | POST "/api/chat"

AnythingLLM v1.12.1

<!-- gh-comment-id:4481311352 --> @workturnedplay commented on GitHub (May 18, 2026): This isn't only due to CPU inferencing, if I use a model that fits into VRAM and uses exclusively the GPU (like https://ollama.com/library/granite4.1:3b-q2_K ) it still happens (Stop doesn't cause GPU to go down from 98% usage, and model is for example stuck in a loop and keeps outputting but anythingllm doesn't show it due to Stop, but ollama keeps running it) and the only workaround to stop it is to quit AnythingLLM (ie. from systray). How about a workaround: disconnect and reconnect to ollama? like cline is doing: > Cline does call abort() on the request when you cancel a task — this was fixed in v3.41.0. What that does is **close** Cline's **HTTP connection to Ollama**. The problem is Ollama's server doesn't stop generating when the connection drops — it queues new requests and keeps the current one running until it finishes. This is specific to Ollama's architecture; llama-server handles connection drops differently, which is why it stops cleanly. [quote src](https://github.com/cline/cline/issues/9442#issuecomment-4182720933) I can confirm the connection (between anythingllm and `ollama serve`) is kept `Established` (seen via TinyWall) even though `Stop` was pressed. That's why `Quit` works because it closes the connection and then `ollama serve` shows only this line when that happens: `[GIN] 2026/05/18 - 21:33:18 | 200 | 2m8s | 127.0.0.1 | POST "/api/chat"` AnythingLLM v1.12.1
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label possible bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#611
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?