Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-19 14:13:35 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #1417] [Question]: Increase completion timeout to prevent crash? #906

New Issue
Closed
opened 2026-02-22 18:22:07 -05:00 by yindo · 9 comments
yindo commented 2026-02-22 18:22:07 -05:00
Owner
Copy Link
Copy Source

Originally created by @rsdmike on GitHub (May 16, 2024).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/1417

Originally assigned to: @shatfield4 on GitHub.

How are you running AnythingLLM?

Docker (local)

What happened?

Using LocalAI for backend -- loading the llama3 70b model, Anything LLM container crashed with a socket timeout.

2024-05-15 22:24:20 [TELEMETRY SENT] {
2024-05-15 22:24:20   event: 'sent_chat',
2024-05-15 22:24:20   distinctId: 'be3ac3d9-aa83-4458-ae1a-583a3fcc909b',
2024-05-15 22:24:20   properties: {
2024-05-15 22:24:20     multiUserMode: false,
2024-05-15 22:24:20     LLMSelection: 'localai',
2024-05-15 22:24:20     Embedder: 'openai',
2024-05-15 22:24:20     VectorDbSelection: 'lancedb',
2024-05-15 22:24:20     runtime: 'docker'
2024-05-15 22:24:20   }
2024-05-15 22:24:20 }
2024-05-15 22:24:20 [Event Logged] - sent_chat
2024-05-15 23:04:01 Cannonball results 3511 -> 470 tokens.
2024-05-15 23:04:01 Cannonball results 356 -> 286 tokens.
2024-05-15 23:04:53 [TELEMETRY SENT] {
2024-05-15 23:04:53   event: 'sent_chat',
2024-05-15 23:04:53   distinctId: 'be3ac3d9-aa83-4458-ae1a-583a3fcc909b',
2024-05-15 23:04:53   properties: {
2024-05-15 23:04:53     multiUserMode: false,
2024-05-15 23:04:53     LLMSelection: 'localai',
2024-05-15 23:04:53     Embedder: 'openai',
2024-05-15 23:04:53     VectorDbSelection: 'lancedb',
2024-05-15 23:04:53     runtime: 'docker'
2024-05-15 23:04:53   }
2024-05-15 23:04:53 }
2024-05-15 23:04:53 [Event Logged] - sent_chat
2024-05-15 23:25:00 node:internal/process/promises:288
2024-05-15 23:25:00             triggerUncaughtException(err, true /* fromPromise */);
2024-05-15 23:25:00             ^
2024-05-15 23:25:00 
2024-05-15 23:25:00 Error: Socket timeout
2024-05-15 23:25:00     at Socket.onTimeout (/app/server/node_modules/agentkeepalive/lib/agent.js:350:23)
2024-05-15 23:25:00     at Socket.emit (node:events:529:35)
2024-05-15 23:25:00     at Socket._onTimeout (node:net:598:8)
2024-05-15 23:25:00     at listOnTimeout (node:internal/timers:569:17)
2024-05-15 23:25:00     at process.processTimers (node:internal/timers:512:7) {
2024-05-15 23:25:00   code: 'ERR_SOCKET_TIMEOUT',
2024-05-15 23:25:00   timeout: 601000
2024-05-15 23:25:00 }
2024-05-15 23:25:00 
2024-05-15 23:25:00 Node.js v18.19.1

A timeout I think is fine , it takes a while to load. However I didn't expect the container to crash, I kind of expected just to re-initiate the thread.

Are there known steps to reproduce?

I think this should be reproducable with any load time of greater than 10 minutes.

Originally created by @rsdmike on GitHub (May 16, 2024). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/1417 Originally assigned to: @shatfield4 on GitHub. ### How are you running AnythingLLM? Docker (local) ### What happened? Using LocalAI for backend -- loading the llama3 70b model, Anything LLM container crashed with a socket timeout. ``` 2024-05-15 22:24:20 [TELEMETRY SENT] { 2024-05-15 22:24:20 event: 'sent_chat', 2024-05-15 22:24:20 distinctId: 'be3ac3d9-aa83-4458-ae1a-583a3fcc909b', 2024-05-15 22:24:20 properties: { 2024-05-15 22:24:20 multiUserMode: false, 2024-05-15 22:24:20 LLMSelection: 'localai', 2024-05-15 22:24:20 Embedder: 'openai', 2024-05-15 22:24:20 VectorDbSelection: 'lancedb', 2024-05-15 22:24:20 runtime: 'docker' 2024-05-15 22:24:20 } 2024-05-15 22:24:20 } 2024-05-15 22:24:20 [Event Logged] - sent_chat 2024-05-15 23:04:01 Cannonball results 3511 -> 470 tokens. 2024-05-15 23:04:01 Cannonball results 356 -> 286 tokens. 2024-05-15 23:04:53 [TELEMETRY SENT] { 2024-05-15 23:04:53 event: 'sent_chat', 2024-05-15 23:04:53 distinctId: 'be3ac3d9-aa83-4458-ae1a-583a3fcc909b', 2024-05-15 23:04:53 properties: { 2024-05-15 23:04:53 multiUserMode: false, 2024-05-15 23:04:53 LLMSelection: 'localai', 2024-05-15 23:04:53 Embedder: 'openai', 2024-05-15 23:04:53 VectorDbSelection: 'lancedb', 2024-05-15 23:04:53 runtime: 'docker' 2024-05-15 23:04:53 } 2024-05-15 23:04:53 } 2024-05-15 23:04:53 [Event Logged] - sent_chat 2024-05-15 23:25:00 node:internal/process/promises:288 2024-05-15 23:25:00 triggerUncaughtException(err, true /* fromPromise */); 2024-05-15 23:25:00 ^ 2024-05-15 23:25:00 2024-05-15 23:25:00 Error: Socket timeout 2024-05-15 23:25:00 at Socket.onTimeout (/app/server/node_modules/agentkeepalive/lib/agent.js:350:23) 2024-05-15 23:25:00 at Socket.emit (node:events:529:35) 2024-05-15 23:25:00 at Socket._onTimeout (node:net:598:8) 2024-05-15 23:25:00 at listOnTimeout (node:internal/timers:569:17) 2024-05-15 23:25:00 at process.processTimers (node:internal/timers:512:7) { 2024-05-15 23:25:00 code: 'ERR_SOCKET_TIMEOUT', 2024-05-15 23:25:00 timeout: 601000 2024-05-15 23:25:00 } 2024-05-15 23:25:00 2024-05-15 23:25:00 Node.js v18.19.1 ``` A timeout I think is fine , it takes a while to load. However I didn't expect the container to crash, I kind of expected just to re-initiate the thread. ### Are there known steps to reproduce? I think this should be reproducable with any load time of greater than 10 minutes.
yindo added the needs info / can't replicateinvestigating labels 2026-02-22 18:22:07 -05:00
yindo closed this issue 2026-02-22 18:22:07 -05:00
yindo commented 2026-02-22 18:22:07 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (May 16, 2024):

Did this timeout occur while your LLM was responding or while you had a session open but had not yet sent a chat to the LLM? If you LLM is taking 10 minutes to reply that is kind of an insane latency, but yes it should not crash the server

@timothycarambat commented on GitHub (May 16, 2024): Did this timeout occur _while_ your LLM was responding or while you had a session open but had not yet sent a chat to the LLM? If you LLM is taking 10 minutes to reply that is kind of an insane latency, but yes it should not crash the server
yindo commented 2026-02-22 18:22:08 -05:00
Author
Owner
Copy Link
Copy Source

@rsdmike commented on GitHub (May 17, 2024):

Kind of in-between. Continuing a previous chat session/thread, restarted back-end (LocalAI), the first chat message to LocalAI loads the model into memory -- so its not responding yet to messages, and yes on CPU it takes a while to load into RAM for 70b - but subtle difference being that its not inferencing yet. Take a look at my screenshot here to see the events, by the time the model has loaded, the AnythingLLM server has told me it has better than things to do than wait (crashed) 😆 .

image

Should mention, after its loaded, its works good no issue.

Thanks for getting back to me 👍

@rsdmike commented on GitHub (May 17, 2024): Kind of in-between. Continuing a previous chat session/thread, restarted back-end (LocalAI), the first chat message to LocalAI loads the model into memory -- so its not responding yet to messages, and yes on CPU it takes a while to load into RAM for 70b - but subtle difference being that its not inferencing yet. Take a look at my screenshot here to see the events, by the time the model has loaded, the AnythingLLM server has told me it has better than things to do than wait (crashed) 😆 . ![image](https://github.com/Mintplex-Labs/anything-llm/assets/22204562/5c930c0d-c193-4bff-b83b-2b4cf4e204d3) Should mention, after its loaded, its works good no issue. Thanks for getting back to me 👍
yindo commented 2026-02-22 18:22:08 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (May 17, 2024):

Ah, so its just the model taking a long time to load the request moves on. The 10 minutes is no coincidence either. For LocalAI we use openai's NPM package which has a 10-minute timeout

I would be nervous to have this be infinity because then you can hang the entire call. Is is unreasonable to ask to mlock the model and basically prime i before using it to prevent this 😬 ?

I'm not super excited to accidentally lead to infinitely hanging requests for LocalAI!

@timothycarambat commented on GitHub (May 17, 2024): Ah, so its just the model taking a long time to load the request moves on. The 10 minutes is no coincidence either. For LocalAI we use `openai`'s NPM package [which has a 10-minute timeout](https://github.com/openai/openai-node?tab=readme-ov-file#timeouts) I would be nervous to have this be infinity because then you can hang the entire call. Is is unreasonable to ask to `mlock` the model and basically prime i before using it to prevent this 😬 ? I'm not super excited to accidentally lead to infinitely hanging requests for LocalAI!
yindo commented 2026-02-22 18:22:08 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (May 17, 2024):

Will add that regardless this should not exit the process - so that needs to be patched for sure

@timothycarambat commented on GitHub (May 17, 2024): Will add that regardless this should not exit the process - so that needs to be patched for sure
yindo commented 2026-02-22 18:22:08 -05:00
Author
Owner
Copy Link
Copy Source

@rsdmike commented on GitHub (May 17, 2024):

Yeah, I'd agree with that. Not too worried about the specific handling for LocalAI, just that the server doesn't crash for anythingllm. I can handle model preloading and such, but when I'm downloading various models and trying them out and loading them on the fly -- just to not have a crash would be good enough.

@rsdmike commented on GitHub (May 17, 2024): Yeah, I'd agree with that. Not too worried about the specific handling for LocalAI, just that the server doesn't crash for anythingllm. I can handle model preloading and such, but when I'm downloading various models and trying them out and loading them on the fly -- just to not have a crash would be good enough.
yindo commented 2026-02-22 18:22:08 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (May 17, 2024):

This is interesting, i am trying to replicate this right now and i cant get that exact timeout to occur. It is always handled, which have me thinking this exception is being thrown somewhere else that is not being caught. Any exception during streaming would be caught and prevent an outright crash.

Right now I'm having trouble reproducing the exact error so I can locate its full stacktrace and handle it

@timothycarambat commented on GitHub (May 17, 2024): This is interesting, i am trying to replicate this right now and i cant get that exact timeout to occur. It is always handled, which have me thinking this exception is being thrown somewhere else that is not being caught. Any exception during streaming would be caught and prevent an outright crash. Right now I'm having trouble reproducing the exact error so I can locate its full stacktrace and handle it
yindo commented 2026-02-22 18:22:08 -05:00
Author
Owner
Copy Link
Copy Source

@rsdmike commented on GitHub (May 17, 2024):

I'll see if i can run this locally, in debug mode, and give ya any more info. Also, I was 11 commits behind master, so, lemme grab latest and try again as well.

@rsdmike commented on GitHub (May 17, 2024): I'll see if i can run this locally, in debug mode, and give ya any more info. Also, I was 11 commits behind master, so, lemme grab latest and try again as well.
yindo commented 2026-02-22 18:22:08 -05:00
Author
Owner
Copy Link
Copy Source

@rsdmike commented on GitHub (May 17, 2024):

Not sure if this adds any more info, but using latest, still able to reproduce though. Looks like it originated from agentkeepalive in node_modules.
image
image

I'll keep playing around with this over the weekend. Workaround is easy enough to pre-load the model

@rsdmike commented on GitHub (May 17, 2024): Not sure if this adds any more info, but using latest, still able to reproduce though. Looks like it originated from agentkeepalive in node_modules. ![image](https://github.com/Mintplex-Labs/anything-llm/assets/22204562/6e2b4d47-fb5f-4d1f-9e90-e942ffe5b9ab) ![image](https://github.com/Mintplex-Labs/anything-llm/assets/22204562/ec15ec37-0465-4c83-bb48-436a74920e3d) I'll keep playing around with this over the weekend. Workaround is easy enough to pre-load the model
yindo commented 2026-02-22 18:22:08 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (May 17, 2024):

From what i saw in the lockfile, the openai npm module requires that sub-dependency, its just frustrating because i cant determine where we call the library before it aborts so we can handle it!

@timothycarambat commented on GitHub (May 17, 2024): From what i saw in the lockfile, the `openai` npm module requires that sub-dependency, its just frustrating because i cant determine where we call the library before it aborts so we can handle it!
yindo changed title from [Question]: Increase completion timeout to prevent crash? to [GH-ISSUE #1417] [Question]: Increase completion timeout to prevent crash? 2026-06-05 14:37:51 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label investigating needs info / can't replicate
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#906
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?