langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 03:55:25 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Not support sse(server side event) #2957

New Issue
Closed
opened 2026-02-21 17:52:03 -05:00 by yindo · 0 comments
yindo commented 2026-02-21 17:52:03 -05:00
Owner
Copy Link
Copy Source

Originally created by @zengqingfu1442 on GitHub (May 6, 2024).

Self Checks

  • This is only for bug report, if you would like to ask a quesion, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • Pleas do not modify this template :) and fill in all the required fields.

Dify version

0.6.6

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

I deployed an openai-api-compatible server which is sse mode, and add a line code print(f'decoded_chunk: {decoded_chunk}') under this line https://github.com/langgenius/dify/blob/0.6.6/api/core/model_runtime/model_providers/openai_api_compatible/llm/llm.py#L433, and then call its api with the following script:

import os
from collections.abc import Generator

import pytest

from core.model_runtime.entities.llm_entities import LLMResult, LLMResultChunk, LLMResultChunkDelta
from core.model_runtime.entities.message_entities import (
    AssistantPromptMessage,
    PromptMessageTool,
    SystemPromptMessage,
    UserPromptMessage,
)
from core.model_runtime.errors.validate import CredentialsValidateFailedError
from core.model_runtime.model_providers.openai_api_compatible.llm.llm import OAIAPICompatLargeLanguageModel

"""
Using Together.ai's OpenAI-compatible API as testing endpoint
"""


def func():
    model = OAIAPICompatLargeLanguageModel()

    response = model.invoke(
        model='qwen2_14b_kv_v3',
        credentials={
            'api_key': os.environ.get('TOGETHER_API_KEY'),
            'endpoint_url': 'http://172.16.11.242:8080/v1/',
            'mode': 'chat',
            'stream_mode_delimiter': '\\n\\n'
        },
        prompt_messages=[
            SystemPromptMessage(
                content='You are a helpful AI assistant.',
            ),
            UserPromptMessage(
                content='Who are you?'
            )
        ],
        model_parameters={
            'temperature': 1.0,
            'top_k': 2,
            'top_p': 0.5,
        },
        stop=['How'],
        stream=True,
        user="abc-123"
    )
    for chunk in response:
        print(chunk)


if __name__ == '__main__':
    func()

✔️ Expected Behavior

All the prefix data: of all chunks is stripped.

❌ Actual Behavior

Only the first chunk's data: is stripped. And the json str is failed to be decoded.

None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
/home/vscode/.local/lib/python3.10/site-packages/pydub/utils.py:170: RuntimeWarning: Couldn't find ffmpeg or avconv - defaulting to ffmpeg, but may not work
  warn("Couldn't find ffmpeg or avconv - defaulting to ffmpeg, but may not work", RuntimeWarning)
decoded_chunk: {"model":"qwen2_14b_kv_v3","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"role":"assistant"},"finish_reason":null}]}

data: {"model":"qwen2_14b_kv_v3","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"I"},"finish_reason":null}]}

data: {"model":"qwen2_14b_kv_v3","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":" am"},"finish_reason":null}]}

data: {"model":"qwen2_14b_kv_v3","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":" a"},"finish_reason":null}]}

data: {"model":"qwen2_14b_kv_v3","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":" large"},"finish_reason":null}]}

data: {"model":"qwen2_14b_kv_v3","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":" language"},"finish_reason":null}]}

data: {"model":"qwen2_14b_kv_v3","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":" model"},"finish_reason":null}]}

data: {"model":"qwen2_14b_kv_v3","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":" created"},"finish_reason":null}]}

data: {"model":"qwen2_14b_kv_v3","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":" by"},"finish_reason":null}]}

data: {"model":"qwen2_14b_kv_v3","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":" Alibaba"},"finish_reason":null}]}

data: {"model":"qwen2_14b_kv_v3","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":" Cloud"},"finish_reason":null}]}

data: {"model":"qwen2_14b_kv_v3","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"."},"finish_reason":null}]}

data: {"model":"qwen2_14b_kv_v3","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":" I"},"finish_reason":null}]}

data: {"model":"qwen2_14b_kv_v3","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":" am"},"finish_reason":null}]}

data: {"model":"qwen2_14b_kv_v3","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":" called"},"finish_reason":null}]}

data: {"model":"qwen2_14b_kv_v3","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":" Q"},"finish_reason":null}]}

data: {"model":"qwen2_14b_kv_v3","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"wen"},"finish_reason":null}]}

data: {"model":"qwen2_14b_kv_v3","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"."},"finish_reason":null}]}

data: {"model":"qwen2_14b_kv_v3","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"\",\"model_name\":\"qwen2_14b_kv_v3"},"finish_reason":null}]}

data: {"model":"qwen2_14b_kv_v3","object":"chat.completion.chunk","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: [DONE]
model='qwen2_14b_kv_v3' prompt_messages=[SystemPromptMessage(role=<PromptMessageRole.SYSTEM: 'system'>, content='You are a helpful AI assistant.', name=None), UserPromptMessage(role=<PromptMessageRole.USER: 'user'>, content='Who are you?', name=None)] system_fingerprint=None delta=LLMResultChunkDelta(index=1, message=AssistantPromptMessage(role=<PromptMessageRole.ASSISTANT: 'assistant'>, content='', name=None, tool_calls=[]), usage=LLMUsage(prompt_tokens=7, prompt_unit_price=Decimal('0'), prompt_price_unit=Decimal('0'), prompt_price=Decimal('0E-7'), completion_tokens=0, completion_unit_price=Decimal('0'), completion_price_unit=Decimal('0'), completion_price=Decimal('0E-7'), total_tokens=7, total_price=Decimal('0E-7'), currency='USD', latency=7.4800044149160385), finish_reason='Non-JSON encountered.')
model='qwen2_14b_kv_v3' prompt_messages=[SystemPromptMessage(role=<PromptMessageRole.SYSTEM: 'system'>, content='You are a helpful AI assistant.', name=None), UserPromptMessage(role=<PromptMessageRole.USER: 'user'>, content='Who are you?', name=None)] system_fingerprint=None delta=LLMResultChunkDelta(index=0, message=AssistantPromptMessage(role=<PromptMessageRole.ASSISTANT: 'assistant'>, content='', name=None, tool_calls=[]), usage=LLMUsage(prompt_tokens=7, prompt_unit_price=Decimal('0'), prompt_price_unit=Decimal('0'), prompt_price=Decimal('0E-7'), completion_tokens=0, completion_unit_price=Decimal('0'), completion_price_unit=Decimal('0'), completion_price=Decimal('0E-7'), total_tokens=7, total_price=Decimal('0E-7'), currency='USD', latency=7.480802866164595), finish_reason='Unknown')
Originally created by @zengqingfu1442 on GitHub (May 6, 2024). ### Self Checks - [X] This is only for bug report, if you would like to ask a quesion, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [X] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [X] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [X] Pleas do not modify this template :) and fill in all the required fields. ### Dify version 0.6.6 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce I deployed an openai-api-compatible server which is sse mode, and add a line code `print(f'decoded_chunk: {decoded_chunk}')` under this line https://github.com/langgenius/dify/blob/0.6.6/api/core/model_runtime/model_providers/openai_api_compatible/llm/llm.py#L433, and then call its api with the following script: ``` import os from collections.abc import Generator import pytest from core.model_runtime.entities.llm_entities import LLMResult, LLMResultChunk, LLMResultChunkDelta from core.model_runtime.entities.message_entities import ( AssistantPromptMessage, PromptMessageTool, SystemPromptMessage, UserPromptMessage, ) from core.model_runtime.errors.validate import CredentialsValidateFailedError from core.model_runtime.model_providers.openai_api_compatible.llm.llm import OAIAPICompatLargeLanguageModel """ Using Together.ai's OpenAI-compatible API as testing endpoint """ def func(): model = OAIAPICompatLargeLanguageModel() response = model.invoke( model='qwen2_14b_kv_v3', credentials={ 'api_key': os.environ.get('TOGETHER_API_KEY'), 'endpoint_url': 'http://172.16.11.242:8080/v1/', 'mode': 'chat', 'stream_mode_delimiter': '\\n\\n' }, prompt_messages=[ SystemPromptMessage( content='You are a helpful AI assistant.', ), UserPromptMessage( content='Who are you?' ) ], model_parameters={ 'temperature': 1.0, 'top_k': 2, 'top_p': 0.5, }, stop=['How'], stream=True, user="abc-123" ) for chunk in response: print(chunk) if __name__ == '__main__': func() ``` ### ✔️ Expected Behavior All the prefix `data: ` of all chunks is stripped. ### ❌ Actual Behavior Only the first chunk's `data: ` is stripped. And the json str is failed to be decoded. ``` None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used. /home/vscode/.local/lib/python3.10/site-packages/pydub/utils.py:170: RuntimeWarning: Couldn't find ffmpeg or avconv - defaulting to ffmpeg, but may not work warn("Couldn't find ffmpeg or avconv - defaulting to ffmpeg, but may not work", RuntimeWarning) decoded_chunk: {"model":"qwen2_14b_kv_v3","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"role":"assistant"},"finish_reason":null}]} data: {"model":"qwen2_14b_kv_v3","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"I"},"finish_reason":null}]} data: {"model":"qwen2_14b_kv_v3","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":" am"},"finish_reason":null}]} data: {"model":"qwen2_14b_kv_v3","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":" a"},"finish_reason":null}]} data: {"model":"qwen2_14b_kv_v3","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":" large"},"finish_reason":null}]} data: {"model":"qwen2_14b_kv_v3","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":" language"},"finish_reason":null}]} data: {"model":"qwen2_14b_kv_v3","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":" model"},"finish_reason":null}]} data: {"model":"qwen2_14b_kv_v3","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":" created"},"finish_reason":null}]} data: {"model":"qwen2_14b_kv_v3","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":" by"},"finish_reason":null}]} data: {"model":"qwen2_14b_kv_v3","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":" Alibaba"},"finish_reason":null}]} data: {"model":"qwen2_14b_kv_v3","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":" Cloud"},"finish_reason":null}]} data: {"model":"qwen2_14b_kv_v3","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"."},"finish_reason":null}]} data: {"model":"qwen2_14b_kv_v3","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":" I"},"finish_reason":null}]} data: {"model":"qwen2_14b_kv_v3","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":" am"},"finish_reason":null}]} data: {"model":"qwen2_14b_kv_v3","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":" called"},"finish_reason":null}]} data: {"model":"qwen2_14b_kv_v3","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":" Q"},"finish_reason":null}]} data: {"model":"qwen2_14b_kv_v3","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"wen"},"finish_reason":null}]} data: {"model":"qwen2_14b_kv_v3","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"."},"finish_reason":null}]} data: {"model":"qwen2_14b_kv_v3","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"\",\"model_name\":\"qwen2_14b_kv_v3"},"finish_reason":null}]} data: {"model":"qwen2_14b_kv_v3","object":"chat.completion.chunk","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]} data: [DONE] model='qwen2_14b_kv_v3' prompt_messages=[SystemPromptMessage(role=<PromptMessageRole.SYSTEM: 'system'>, content='You are a helpful AI assistant.', name=None), UserPromptMessage(role=<PromptMessageRole.USER: 'user'>, content='Who are you?', name=None)] system_fingerprint=None delta=LLMResultChunkDelta(index=1, message=AssistantPromptMessage(role=<PromptMessageRole.ASSISTANT: 'assistant'>, content='', name=None, tool_calls=[]), usage=LLMUsage(prompt_tokens=7, prompt_unit_price=Decimal('0'), prompt_price_unit=Decimal('0'), prompt_price=Decimal('0E-7'), completion_tokens=0, completion_unit_price=Decimal('0'), completion_price_unit=Decimal('0'), completion_price=Decimal('0E-7'), total_tokens=7, total_price=Decimal('0E-7'), currency='USD', latency=7.4800044149160385), finish_reason='Non-JSON encountered.') model='qwen2_14b_kv_v3' prompt_messages=[SystemPromptMessage(role=<PromptMessageRole.SYSTEM: 'system'>, content='You are a helpful AI assistant.', name=None), UserPromptMessage(role=<PromptMessageRole.USER: 'user'>, content='Who are you?', name=None)] system_fingerprint=None delta=LLMResultChunkDelta(index=0, message=AssistantPromptMessage(role=<PromptMessageRole.ASSISTANT: 'assistant'>, content='', name=None, tool_calls=[]), usage=LLMUsage(prompt_tokens=7, prompt_unit_price=Decimal('0'), prompt_price_unit=Decimal('0'), prompt_price=Decimal('0E-7'), completion_tokens=0, completion_unit_price=Decimal('0'), completion_price_unit=Decimal('0'), completion_price=Decimal('0E-7'), total_tokens=7, total_price=Decimal('0E-7'), currency='USD', latency=7.480802866164595), finish_reason='Unknown') ```
yindo added the 🐞 bug label 2026-02-21 17:52:03 -05:00
yindo closed this issue 2026-02-21 17:52:03 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#2957
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?