Mintplex-Labs/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-07-20 14:43:37 -04:00
Code Issues 328 Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #416] LocalAI streaming text overflow #234

New Issue
Closed
opened 2026-02-22 18:18:26 -05:00 by yindo · 13 comments
yindo commented 2026-02-22 18:18:26 -05:00
Owner
Copy Link
Copy Source

Originally created by @lunamidori5 on GitHub (Dec 8, 2023).
Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/416

There is a bug that crashes when Localai sends a chunk with streaming turned on

Cannonball results 347 -> 148 tokens.
Cannonball results 196 -> 147 tokens.
Cannonball results 309 -> 148 tokens.
/app/server/utils/chats/stream.js:256
          chunk += message;
                   ^

RangeError: Invalid string length
    at IncomingMessage.<anonymous> (/app/server/utils/chats/stream.js:256:20)
    at IncomingMessage.emit (node:events:517:28)
    at addChunk (node:internal/streams/readable:368:12)
    at readableAddChunk (node:internal/streams/readable:341:9)
    at Readable.push (node:internal/streams/readable:278:10)
    at HTTPParser.parserOnBody (node:_http_common:131:24)
    at TLSSocket.socketOnData (node:_http_client:541:22)
    at TLSSocket.emit (node:events:517:28)
    at addChunk (node:internal/streams/readable:368:12)
    at readableAddChunk (node:internal/streams/readable:341:9)
    ```
    
    I added a temp fix by just having it in a try if it fails, just set the chunk to ""
Originally created by @lunamidori5 on GitHub (Dec 8, 2023). Original GitHub issue: https://github.com/Mintplex-Labs/anything-llm/issues/416 There is a bug that crashes when Localai sends a chunk with streaming turned on ```js Cannonball results 347 -> 148 tokens. Cannonball results 196 -> 147 tokens. Cannonball results 309 -> 148 tokens. /app/server/utils/chats/stream.js:256 chunk += message; ^ RangeError: Invalid string length at IncomingMessage.<anonymous> (/app/server/utils/chats/stream.js:256:20) at IncomingMessage.emit (node:events:517:28) at addChunk (node:internal/streams/readable:368:12) at readableAddChunk (node:internal/streams/readable:341:9) at Readable.push (node:internal/streams/readable:278:10) at HTTPParser.parserOnBody (node:_http_common:131:24) at TLSSocket.socketOnData (node:_http_client:541:22) at TLSSocket.emit (node:events:517:28) at addChunk (node:internal/streams/readable:368:12) at readableAddChunk (node:internal/streams/readable:341:9) ``` I added a temp fix by just having it in a try if it fails, just set the chunk to ""
yindo added the possible bugblocked labels 2026-02-22 18:18:26 -05:00
yindo closed this issue 2026-02-22 18:18:26 -05:00
yindo commented 2026-02-22 18:18:27 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Dec 8, 2023):

Can i ask which model you are using on LocalAI so I can attempt to replicate?

@timothycarambat commented on GitHub (Dec 8, 2023): Can i ask which model you are using on LocalAI so I can attempt to replicate?
yindo commented 2026-02-22 18:18:27 -05:00
Author
Owner
Copy Link
Copy Source

@lunamidori5 commented on GitHub (Dec 8, 2023):

Yes here is a list of all the models I tested on, Carly is my own sim human model and is not open yet - @timothycarambat

Carly PhotoAI 62T
Carly PhotoAI 20T
Carly PhotoAI 30B
Mistral-7b-openorca.Q6_K
Dolphin-2.2-70b.Q5_K_M
Dolphin-2.2.1-mistral-7b.Q6_K
Wizard-Vicuna-30B-Uncensored.Q8_0

@lunamidori5 commented on GitHub (Dec 8, 2023): Yes here is a list of all the models I tested on, Carly is my own sim human model and is not open yet - @timothycarambat Carly PhotoAI 62T Carly PhotoAI 20T Carly PhotoAI 30B Mistral-7b-openorca.Q6_K Dolphin-2.2-70b.Q5_K_M Dolphin-2.2.1-mistral-7b.Q6_K Wizard-Vicuna-30B-Uncensored.Q8_0
yindo commented 2026-02-22 18:18:27 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Dec 12, 2023):

Currently unable to replicate this issue right now on a streaming model. Seems that streaming is working fine for all the models I have on my machine. The issue here is that it seems like the JSON packet for streaming does not ever close the object so the string gets too long to handle.

What was the code your wrote that handles this exception? Might just be best to have that duct-tape fix right now or wait and see if this blows up for other LocalAI users as I'm unable to replicate on my LocalAI

@timothycarambat commented on GitHub (Dec 12, 2023): Currently unable to replicate this issue right now on a streaming model. Seems that streaming is working fine for all the models I have on my machine. The issue here is that it seems like the JSON packet for streaming does not ever close the object so the string gets too long to handle. What was the code your wrote that handles this exception? Might just be best to have that duct-tape fix right now or wait and see if this blows up for other LocalAI users as I'm unable to replicate on my LocalAI
yindo commented 2026-02-22 18:18:27 -05:00
Author
Owner
Copy Link
Copy Source

@lunamidori5 commented on GitHub (Dec 12, 2023):

@timothycarambat Why am I blocked?

Here is the temp fix code I added



      for (const line of lines) {
        let validJSON = false;
        const message = chunk + line.replace(/^data: /, "");

        // JSON chunk is incomplete and has not ended yet
        // so we need to stitch it together. You would think JSON
        // chunks would only come complete - but they don't!
        try {
          JSON.parse(message);
          validJSON = true;
        } catch {}

        if (!validJSON) {
        // Check if the message is not empty before appending
        if (message.trim() !== "") {
			try {
			chunk += message;
			} catch {chunk = "";}
            
        }

        // Check if the chunk is empty after appending and continue if it is
        if (chunk.trim() === "") {
            continue;
        }
@lunamidori5 commented on GitHub (Dec 12, 2023): @timothycarambat Why am I blocked? Here is the temp fix code I added ```js for (const line of lines) { let validJSON = false; const message = chunk + line.replace(/^data: /, ""); // JSON chunk is incomplete and has not ended yet // so we need to stitch it together. You would think JSON // chunks would only come complete - but they don't! try { JSON.parse(message); validJSON = true; } catch {} if (!validJSON) { // Check if the message is not empty before appending if (message.trim() !== "") { try { chunk += message; } catch {chunk = "";} } // Check if the chunk is empty after appending and continue if it is if (chunk.trim() === "") { continue; } ```
yindo commented 2026-02-22 18:18:27 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Dec 12, 2023):

Blocked because I cant replicate it! That's all that means. It means this issue cannot be worked on further in its current state.

Just for clarification in case there is any confusion on that term, you are not blocked at all (in fact thank you for being so responsive) but this specific issue is on hold for right now 😂

@timothycarambat commented on GitHub (Dec 12, 2023): Blocked because I cant replicate it! That's all that means. It means this issue cannot be worked on further in its current state. Just for clarification in case there is any confusion on that term, **you** are not blocked at all (in fact thank you for being so responsive) but this _specific_ issue is on hold for right now 😂
yindo commented 2026-02-22 18:18:27 -05:00
Author
Owner
Copy Link
Copy Source

@lunamidori5 commented on GitHub (Dec 13, 2023):

@timothycarambat please reopen, here is the error from your debug text

Chunk appending error RangeError: Invalid string length
    at IncomingMessage.<anonymous> (/app/server/utils/chats/stream.js:261:22)
    at IncomingMessage.emit (node:events:517:28)
    at addChunk (node:internal/streams/readable:368:12)
    at readableAddChunk (node:internal/streams/readable:341:9)
    at Readable.push (node:internal/streams/readable:278:10)
    at HTTPParser.parserOnBody (node:_http_common:131:24)
    at TLSSocket.socketOnData (node:_http_client:541:22)
    at TLSSocket.emit (node:events:517:28)
    at addChunk (node:internal/streams/readable:368:12)
    at readableAddChunk (node:internal/streams/readable:341:9)
@lunamidori5 commented on GitHub (Dec 13, 2023): @timothycarambat please reopen, here is the error from your debug text ``` Chunk appending error RangeError: Invalid string length at IncomingMessage.<anonymous> (/app/server/utils/chats/stream.js:261:22) at IncomingMessage.emit (node:events:517:28) at addChunk (node:internal/streams/readable:368:12) at readableAddChunk (node:internal/streams/readable:341:9) at Readable.push (node:internal/streams/readable:278:10) at HTTPParser.parserOnBody (node:_http_common:131:24) at TLSSocket.socketOnData (node:_http_client:541:22) at TLSSocket.emit (node:events:517:28) at addChunk (node:internal/streams/readable:368:12) at readableAddChunk (node:internal/streams/readable:341:9) ```
yindo commented 2026-02-22 18:18:27 -05:00
Author
Owner
Copy Link
Copy Source

@lunamidori5 commented on GitHub (Dec 13, 2023):

heres the thing thats odd, it only happens near the start of the message

Full message, note it dropped the start of the code block

Sure, here’s a simple Python program that can help userthon
import random

def add(x, y):
return x + y

def subtract(x, y):
return x - y

def multiply(x, y):
return x * y

def divide(x, y):
return x / y

def main():
print(“Welcome to the Math Tutor!”)
print(“Please select an operation:”)
print(“1. Addition”)
print(“2. Subtraction”)
print(“3. Multiplication”)
print(“4. Division”)

while True:
try:
choice = int(input("Enter your choice (1/2/3/4): "))
if choice == 1:
num1 = float(input("Enter first number: "))
num2 = float(input("Enter second number: "))
print(f"{num1} + {num2} = {add(num1, num2)}")
elif choice == 2:
num1 = float(input("Enter first number: "))
num2 = float(input("Enter second number: "))
print(f"{num1} - {num2} = {subtract(num1, num2)}")
elif choice == 3:
num1 = float(input("Enter first number: "))
num2 = float(input("Enter second number: "))
print(f"{num1} * {num2} = {multiply(num1, num2)}")
elif choice == 4:
num1 = float(input("Enter first number: "))
num2 = float(input("Enter second number: "))
print(f"{num1} / {num2} = {divide(num1, num2)}")
else:
print("Invalid choice. Please try again.")
except ValueError:
print("Invalid input. Please enter a valid number.")
if name == “main”:
main()

The model did output the code block but my jank fix just dropped the tokens, is there a better fix?

@lunamidori5 commented on GitHub (Dec 13, 2023): heres the thing thats odd, it only happens near the start of the message Full message, note it dropped the start of the code block Sure, here’s a simple Python program that can help userthon import random def add(x, y): return x + y def subtract(x, y): return x - y def multiply(x, y): return x * y def divide(x, y): return x / y def main(): print(“Welcome to the Math Tutor!”) print(“Please select an operation:”) print(“1. Addition”) print(“2. Subtraction”) print(“3. Multiplication”) print(“4. Division”) while True: try: choice = int(input("Enter your choice (1/2/3/4): ")) if choice == 1: num1 = float(input("Enter first number: ")) num2 = float(input("Enter second number: ")) print(f"{num1} + {num2} = {add(num1, num2)}") elif choice == 2: num1 = float(input("Enter first number: ")) num2 = float(input("Enter second number: ")) print(f"{num1} - {num2} = {subtract(num1, num2)}") elif choice == 3: num1 = float(input("Enter first number: ")) num2 = float(input("Enter second number: ")) print(f"{num1} * {num2} = {multiply(num1, num2)}") elif choice == 4: num1 = float(input("Enter first number: ")) num2 = float(input("Enter second number: ")) print(f"{num1} / {num2} = {divide(num1, num2)}") else: print("Invalid choice. Please try again.") except ValueError: print("Invalid input. Please enter a valid number.") if name == “main”: main() The model did output the code block but my jank fix just dropped the tokens, is there a better fix?
yindo commented 2026-02-22 18:18:27 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Dec 13, 2023):

Unfortunately not unless we can consistently replicate the error. Its definitely because the message response chunk contains an incomplete JSON object, which is not uncommon, but then the object apparently never closes so the string adds on forever never completing the json chunk so we can insert as a message.

That fix for now is simply a patch, but for sure the chunk streaming needs to be resolved since something is clearly going on with the chunking mechanism for certain responses

@timothycarambat commented on GitHub (Dec 13, 2023): Unfortunately not unless we can consistently replicate the error. Its definitely because the message response chunk contains an incomplete JSON object, which is not uncommon, but then the object apparently never closes so the string adds on forever never completing the json chunk so we can insert as a message. That fix for now is simply a patch, but for sure the chunk streaming needs to be resolved since something is clearly going on with the chunking mechanism for certain responses
yindo commented 2026-02-22 18:18:27 -05:00
Author
Owner
Copy Link
Copy Source

@lunamidori5 commented on GitHub (Dec 13, 2023):

@timothycarambat noted! Thank you for all your hard work (I really should get around to learning node / js

@lunamidori5 commented on GitHub (Dec 13, 2023): @timothycarambat noted! Thank you for all your hard work (I really should get around to learning node / js
yindo commented 2026-02-22 18:18:27 -05:00
Author
Owner
Copy Link
Copy Source

@Claudio-C commented on GitHub (Dec 31, 2023):

anything-llm | }
anything-llm | undefined:1
anything-llm |
anything-llm | ^
anything-llm |
anything-llm | SyntaxError: Unexpected token < in JSON at position 0
anything-llm | at JSON.parse ()
anything-llm | at parseJSONFromBytes (node:internal/deps/undici/undici:4747:19)
anything-llm | at successSteps (node:internal/deps/undici/undici:4718:27)
anything-llm | at fullyReadBody (node:internal/deps/undici/undici:1433:9)
anything-llm | at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
anything-llm | at async specConsumeBody (node:internal/deps/undici/undici:4727:7)
anything-llm | at async createOllamaStream (/app/server/node_modules/langchain/dist/util/ollama.cjs:21:22)
anything-llm | at async ChatOllama._streamResponseChunks (/app/server/node_modules/langchain/dist/chat_models/ollama.cjs:337:26)
anything-llm | at async ChatOllama._streamIterator (/app/server/node_modules/@langchain/core/dist/language_models/chat_models.cjs:82:34)
anything-llm | at async ChatOllama.transform (/app/server/node_modules/@langchain/core/dist/runnables/base.cjs:307:9)
anything-llm |
anything-llm | Node.js v18.19.0
anything-llm exited with code 1

Same issue both puling latest docker or building it. using ollama and llama7b

@Claudio-C commented on GitHub (Dec 31, 2023): anything-llm | } anything-llm | undefined:1 anything-llm | <!DOCTYPE html> anything-llm | ^ anything-llm | anything-llm | SyntaxError: Unexpected token < in JSON at position 0 anything-llm | at JSON.parse (<anonymous>) anything-llm | at parseJSONFromBytes (node:internal/deps/undici/undici:4747:19) anything-llm | at successSteps (node:internal/deps/undici/undici:4718:27) anything-llm | at fullyReadBody (node:internal/deps/undici/undici:1433:9) anything-llm | at process.processTicksAndRejections (node:internal/process/task_queues:95:5) anything-llm | at async specConsumeBody (node:internal/deps/undici/undici:4727:7) anything-llm | at async createOllamaStream (/app/server/node_modules/langchain/dist/util/ollama.cjs:21:22) anything-llm | at async ChatOllama._streamResponseChunks (/app/server/node_modules/langchain/dist/chat_models/ollama.cjs:337:26) anything-llm | at async ChatOllama._streamIterator (/app/server/node_modules/@langchain/core/dist/language_models/chat_models.cjs:82:34) anything-llm | at async ChatOllama.transform (/app/server/node_modules/@langchain/core/dist/runnables/base.cjs:307:9) anything-llm | anything-llm | Node.js v18.19.0 anything-llm exited with code 1 Same issue both puling latest docker or building it. using ollama and llama7b
yindo commented 2026-02-22 18:18:27 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Jan 1, 2024):

SyntaxError: Unexpected token < in JSON at position 0 like means your Ollama endpoint is incorrect because Im willing to bet that the < is the beginning of an HTML response.

So you should ensure the ollama endpoint is correct and reachable. What is the value you are using for your Ollama connection endpoint?

@timothycarambat commented on GitHub (Jan 1, 2024): `SyntaxError: Unexpected token < in JSON at position 0` like means your Ollama endpoint is incorrect because Im willing to bet that the `<` is the beginning of an HTML response. So you should ensure the ollama endpoint is correct and reachable. What is the value you are using for your Ollama connection endpoint?
yindo commented 2026-02-22 18:18:27 -05:00
Author
Owner
Copy Link
Copy Source

@Claudio-C commented on GitHub (Jan 1, 2024):

Ollama is setup ok. After some more testing i realized that it works great with one document but fails with the same error with several documents.

@Claudio-C commented on GitHub (Jan 1, 2024): Ollama is setup ok. After some more testing i realized that it works great with one document but fails with the same error with several documents.
yindo commented 2026-02-22 18:18:27 -05:00
Author
Owner
Copy Link
Copy Source

@timothycarambat commented on GitHub (Jan 1, 2024):

Are you using the proper token context window limit? If it is larger than the model permits it may crash prematurely

@timothycarambat commented on GitHub (Jan 1, 2024): Are you using the proper token context window limit? If it is larger than the model permits it may crash prematurely
yindo changed title from LocalAI streaming text overflow to [GH-ISSUE #416] LocalAI streaming text overflow 2026-06-05 14:34:12 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Desktop
Docker
Integration Request
Integration Request
OS: Linux
OS: Mobile
OS: Windows
UI/UX
blocked
bug
bug
core-team-only
documentation
duplicate
embed-widget
enhancement
feature request
github_actions
good first issue
investigating
needs info / can't replicate
possible bug
pull-request
Mirrored from GitHub Pull Request
 question
stage: specifications
wontfix
No Label blocked possible bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Mintplex-Labs/anything-llm#234
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?