langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-08-27 11:11:21 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

openai_api_compatible stream mode delimiter error #993

New Issue
Closed
opened 2026-02-21 17:29:23 -05:00 by yindo · 2 comments
yindo commented 2026-02-21 17:29:23 -05:00
Owner
Copy Link
Copy Source

Originally created by @bowenliang123 on GitHub (Jan 29, 2024).

Originally assigned to: @takatost, @guchenhe on GitHub.

Self Checks

Dify version

0.5.2

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

  1. set the stream delimiter to '\n\r' for custom qwen 千问 72b model with openai_api_compatible provider
  2. llm chat failed with (red block in the screenshot)
    Error: can only concatenate str (not "NoneType") to str

✔️ Expected Behavior

❌ Actual Behavior

During steamlingly displaying the reply:
image

When the reply ends with error:
image

Code:

image

Originally created by @bowenliang123 on GitHub (Jan 29, 2024). Originally assigned to: @takatost, @guchenhe on GitHub. ### Self Checks - [X] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [X] I confirm that I am using English to file this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). ### Dify version 0.5.2 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce 1. set the stream delimiter to '\n\r' for custom `qwen` 千问 72b model with openai_api_compatible provider 2. llm chat failed with (red block in the screenshot) `Error: can only concatenate str (not "NoneType") to str` ### ✔️ Expected Behavior ### ❌ Actual Behavior During steamlingly displaying the reply: <img width="813" alt="image" src="https://github.com/langgenius/dify/assets/1935105/4986bfc2-0a61-45e9-bf9d-5fd00e5a08ed"> When the reply ends with error: <img width="780" alt="image" src="https://github.com/langgenius/dify/assets/1935105/047cf6c1-3958-4e61-8de0-1309e64afb58"> Code: ![image](https://github.com/langgenius/dify/assets/1935105/75afb2cf-b148-4fc8-baee-f8b84589b7f5)
yindo added the 🐞 bug label 2026-02-21 17:29:23 -05:00
yindo closed this issue 2026-02-21 17:29:23 -05:00
yindo commented 2026-02-21 17:29:23 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Jan 29, 2024):

🤖

Hey there, @bowenliang123! Great to see you back here again. Let's get this sorted out for you.

Based on the error message you're encountering, it seems like the system is trying to concatenate a string with a None value, which is not allowed in Python. This might be happening when the system is trying to concatenate the response from the model (delta.text or delta.delta.content) with the existing full text or full assistant content. If the model's response is None and it's being concatenated with the existing text, this error would occur.

Here's the relevant code:

full_text += text

full_assistant_content += delta.delta.content if delta.delta.content else ''

To fix this issue, you could modify the code to ensure that you're not trying to concatenate None with a string. You could do this by converting None to an empty string before concatenation. Here's how you could modify the code:

full_text += text if text is not None else ''

full_assistant_content += delta.delta.content if delta.delta.content is not None else ''

This is a general suggestion based on the provided code and the error message. The actual issue might be different based on the complete context and the specific conditions when the error occurs. If the error persists after trying this suggestion, please provide more details about the specific conditions when the error occurs.

Regarding your question about the function of the stream delimiter in the Dify codebase, the stream parameter in the _invoke method of the OAIAPICompatLargeLanguageModel class determines whether the response from the model should be streamed or not. If stream is set to True, the response from the model will be streamed, and the _handle_generate_stream_response method will be used to handle the response.

The stream_mode_delimiter in the credentials dictionary is used as a delimiter for the streamed response. This delimiter is used to split the streamed response into chunks. Each chunk is then processed separately.

Here is the relevant code:

# delimiter for stream response, need unicode_escape
import codecs
delimiter = credentials.get("stream_mode_delimiter", "\n\n")
delimiter = codecs.decode(delimiter, "unicode_escape")
for chunk in response.iter_lines(decode_unicode=True, delimiter=delimiter):
    if chunk:
        decoded_chunk = chunk.strip().lstrip('data: ').lstrip()
        chunk_json = None
        try:
            chunk_json = json.loads(decoded_chunk)
        # stream ended
        except json.JSONDecodeError as e:
            logger.error(f"decoded_chunk error,delimiter={delimiter},decoded_chunk={decoded_chunk}")
            yield create_final_llm_result_chunk(
                index=chunk_index + 1,
                message=AssistantPromptMessage(content=""),
                finish_reason="Non-JSON encountered."
            )
            break
        if not chunk_json or len(chunk_json['choices']) == 0:
            continue

        choice = chunk_json['choices'][0]
        finish_reason = chunk_json['choices'][0].get('finish_reason')
        chunk_index += 1

In this code, the stream_mode_delimiter is retrieved from the credentials dictionary, with a default value of "\n\n" if it is not provided. The codecs.decode function is used to decode the delimiter from a unicode escape sequence to a string. The response.iter_lines function is then used to iterate over the response in chunks, with each chunk being separated by the delimiter. Each chunk is then processed separately.

I hope this helps! If you have any other questions or need further clarification, feel free to ask.

Best,
Dosu

Sources

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

@dosubot[bot] commented on GitHub (Jan 29, 2024): _🤖_ <!-- Greeting --> Hey there, @bowenliang123! Great to see you back here again. Let's get this sorted out for you. <!-- Answer --> Based on the error message you're encountering, it seems like the system is trying to concatenate a string with a None value, which is not allowed in Python. This might be happening when the system is trying to concatenate the response from the model (delta.text or delta.delta.content) with the existing full text or full assistant content. If the model's response is None and it's being concatenated with the existing text, this error would occur. Here's the relevant code: ```py full_text += text ``` ```py full_assistant_content += delta.delta.content if delta.delta.content else '' ``` To fix this issue, you could modify the code to ensure that you're not trying to concatenate None with a string. You could do this by converting None to an empty string before concatenation. Here's how you could modify the code: ```py full_text += text if text is not None else '' ``` ```py full_assistant_content += delta.delta.content if delta.delta.content is not None else '' ``` This is a general suggestion based on the provided code and the error message. The actual issue might be different based on the complete context and the specific conditions when the error occurs. If the error persists after trying this suggestion, please provide more details about the specific conditions when the error occurs. Regarding your question about the function of the stream delimiter in the Dify codebase, the `stream` parameter in the `_invoke` method of the `OAIAPICompatLargeLanguageModel` class determines whether the response from the model should be streamed or not. If `stream` is set to `True`, the response from the model will be streamed, and the `_handle_generate_stream_response` method will be used to handle the response. The `stream_mode_delimiter` in the `credentials` dictionary is used as a delimiter for the streamed response. This delimiter is used to split the streamed response into chunks. Each chunk is then processed separately. Here is the relevant code: ```py # delimiter for stream response, need unicode_escape import codecs delimiter = credentials.get("stream_mode_delimiter", "\n\n") delimiter = codecs.decode(delimiter, "unicode_escape") for chunk in response.iter_lines(decode_unicode=True, delimiter=delimiter): if chunk: decoded_chunk = chunk.strip().lstrip('data: ').lstrip() chunk_json = None try: chunk_json = json.loads(decoded_chunk) # stream ended except json.JSONDecodeError as e: logger.error(f"decoded_chunk error,delimiter={delimiter},decoded_chunk={decoded_chunk}") yield create_final_llm_result_chunk( index=chunk_index + 1, message=AssistantPromptMessage(content=""), finish_reason="Non-JSON encountered." ) break if not chunk_json or len(chunk_json['choices']) == 0: continue choice = chunk_json['choices'][0] finish_reason = chunk_json['choices'][0].get('finish_reason') chunk_index += 1 ``` In this code, the `stream_mode_delimiter` is retrieved from the `credentials` dictionary, with a default value of "\n\n" if it is not provided. The `codecs.decode` function is used to decode the delimiter from a unicode escape sequence to a string. The `response.iter_lines` function is then used to iterate over the response in chunks, with each chunk being separated by the delimiter. Each chunk is then processed separately. I hope this helps! If you have any other questions or need further clarification, feel free to ask. Best, [Dosu](https://dosu.dev) ## Sources - [api/core/model_runtime/model_providers/openai/llm/llm.py](https://github.com/langgenius/dify/blob/main/api/core/model_runtime/model_providers/openai/llm/llm.py) - [api/core/model_runtime/model_providers/openai_api_compatible/llm/llm.py](https://github.com/langgenius/dify/blob/main/api/core/model_runtime/model_providers/openai_api_compatible/llm/llm.py) --- This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant. If you want to continue the conversation, start your reply with @dosu-bot.
yindo commented 2026-02-21 17:29:23 -05:00
Author
Owner
Copy Link
Copy Source

@bowenliang123 commented on GitHub (Jan 29, 2024):

The orginal feature and bug introduced by https://github.com/langgenius/dify/pull/2190.
I don't know the reason that the guard conditions in purple block does not work as expectedly and I suggested a temporary workaround that works for me in blue block, for reference.

cc @geosmart @takatost

@bowenliang123 commented on GitHub (Jan 29, 2024): The orginal feature and bug introduced by https://github.com/langgenius/dify/pull/2190. I don't know the reason that the guard conditions in purple block does not work as expectedly and I suggested a temporary workaround that works for me in blue block, for reference. cc @geosmart @takatost
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#993
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?