langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-22 04:25:29 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

[PR #31925] fix: populate files array in message_end SSE event for vision responses #33471

New Issue
Closed
opened 2026-02-21 20:53:20 -05:00 by yindo · 0 comments
yindo commented 2026-02-21 20:53:20 -05:00
Owner
Copy Link
Copy Source

Original Pull Request: https://github.com/langgenius/dify/pull/31925

State: closed
Merged: No

Important

  1. Make sure you have read our contribution guidelines
  2. Ensure there is an associated issue and you have been assigned to it
  3. Use the correct syntax to link this PR: Fixes #<issue number>.

Summary

fix https://github.com/langgenius/dify/issues/31888

Screenshots

Before After
... ...

Checklist

  • This change requires a documentation update, included: Dify Document
  • I understand that this PR may be closed in case there was no previous discussion or issues. (This doesn't apply to typos!)
  • I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
  • I've updated the documentation accordingly.
  • I ran make lint and make type-check (backend) and cd web && npx lint-staged (frontend) to appease the lint gods

Chat app vision responses not rendering - message_file event handler bug (regression from #22552)

Related Issue: #22552 (closed) - The fix in commit 084dcd1a50 has a bug that prevents vision responses from rendering in chat apps.

**Fixes #31888

Dify Version: 1.11.2, 1.11.4, hotfix-1.11.4-fix.4 (likely affects 1.12.0 as well)

Cloud or Self Hosted: Self Hosted (Kubernetes)

Description: When uploading an image to a chat app (not workflow), the AI response is completely empty - neither the text answer nor the image are rendered in the chat interface. The HTML container exists but is empty. This is a regression from the fix in #22552.

Steps to Reproduce

  1. Create a chat app with vision model enabled (e.g., GPT-4 Vision, Claude 3.5 Sonnet with vision)
  2. Upload an image file (e.g., PNG, JPG) in the chat input
  3. Send a message asking about the image (e.g., "explain this image")
  4. Wait for AI response
  5. Observe the chat interface - response container is empty

Note: Check browser DevTools → Network tab to see SSE events are being received, but nothing renders in the UI.

Expected Behavior

  • ✅ AI generates a text response describing the image
  • ✅ Image thumbnail is displayed in the chat history
  • ✅ Both text answer and image are visible in the chat interface
  • ✅ User can see the complete conversation with image context

Actual Behavior

  • ✅ AI generates a text response (confirmed via API logs and SSE events)
  • ✅ Image file is uploaded successfully (201 response from /console/api/files/upload)
  • ✅ File is stored correctly in database and filesystem
  • ✅ File preview URL works when accessed directly in browser
  • ❌ Chat interface shows completely empty div container - no text, no image
  • ❌ Frontend receives message_file SSE events but doesn't render them
  • ❌ message_end event has "files": null instead of files array
  • ❌ HTML structure exists but content is missing (empty <div class="chat-answer-container">)

Additional Information

Technical Analysis

Backend (API) Behavior

The API correctly:

  1. Receives file upload → Returns 201 with upload_file_id
  2. Processes image → Generates response with correct tokens
  3. Sends SSE events:
    • message_file events with file data ✅
    • message_end event with "files": null ❌ (should contain files array)

Frontend Behavior

The frontend (web/app/components/base/chat/chat/hooks.ts):

Problem 1: onFile handler (line 419-422)

onFile(file) {
  const lastThought = responseItem.agent_thoughts?.[responseItem.agent_thoughts?.length - 1]
  if (lastThought)  // ← BUG: Only adds files if agent_thoughts exists!
    responseItem.agent_thoughts![responseItem.agent_thoughts!.length - 1].message_files = [...(lastThought as any).message_files, file]
  // Missing: else clause to add files to responseItem.message_files for chat apps
}

Problem 2: onMessageEnd handler (line 477-478)

const processedFilesFromResponse = getProcessedFilesFromResponse(messageEnd.files || [])
responseItem.allFiles = uniqBy([...(responseItem.allFiles || []), ...(processedFilesFromResponse || [])], 'id')

But message_end event has "files": null instead of the files array.

Root Cause

The fix in commit 084dcd1a50 (issue #22552) has a bug:

File: api/core/app/task_pipeline/message_cycle_manager.py

Current (incorrect) code:

message_file = db.session.query(MessageFile).filter(MessageFile.id == message_id).first()
event_type = StreamEvent.MESSAGE_FILE if message_file else StreamEvent.MESSAGE

Problem: message_id is the message ID, not the file ID! The query should check if there are files associated with the message, not if a file is the message.

Should be:

message_file = db.session.query(MessageFile).filter(MessageFile.message_id == message_id).first()
event_type = StreamEvent.MESSAGE_FILE if message_file else StreamEvent.MESSAGE

Additionally, the message_end event should include the files array, but it's currently null.

Evidence

SSE Events Received:

data: {"event":"message_file","conversation_id":"...","message_id":"...","answer":"","from_variable_selector":null}
data: {"event":"message_file","conversation_id":"...","message_id":"...","answer":"I","from_variable_selector":null}
... (streaming text chunks)
data: {"event":"message_end","conversation_id":"...","message_id":"...","files":null}  ← Should contain files!

API Response (GET /chat-messages):

{
  "answer": "I'm sorry for the confusion...",
  "message_files": [
    {
      "id": "858106aa-d422-4922-9b7f-f52eed3f0619",
      "filename": "twilio.png",
      "url": "/files/.../file-preview?...",
      ...
    }
  ]
}

Browser DOM:

<div class="chat-answer-container">
  <div class="body-lg-regular">
    <!-- EMPTY - no text, no image -->
  </div>
</div>

Proposed Fix

Backend Fix:

  1. Fix the query in message_cycle_manager.py:

    message_file = db.session.query(MessageFile).filter(MessageFile.message_id == message_id).first()

  2. Ensure message_end event includes files array:

    # In message_end_to_stream_response method
files = db.session.query(MessageFile).filter(MessageFile.message_id == message_id).all()
return MessageEndStreamResponse(..., files=files)

Frontend Fix (if backend fix doesn't solve it):

  1. Update onFile handler to handle chat apps: 
    onFile(file) {
  const lastThought = responseItem.agent_thoughts?.[responseItem.agent_thoughts?.length - 1]
  if (lastThought) {
    responseItem.agent_thoughts![responseItem.agent_thoughts!.length - 1].message_files = [...(lastThought as any).message_files, file]
  } else {
    // FIX: Add files to message_files for chat apps
    responseItem.message_files = [...(responseItem.message_files || []), file]
  }
  updateCurrentQAOnTree({...})
}

Impact

  • Severity: High - Vision features completely broken for chat apps
  • Affected Users: All users trying to use vision/image features in chat apps
  • Workaround: None - feature is unusable

Additional Context

  • File upload works correctly
  • File storage works correctly
  • File preview URLs work when accessed directly
  • The issue is specifically in the rendering pipeline between SSE events and DOM
**Original Pull Request:** https://github.com/langgenius/dify/pull/31925 **State:** closed **Merged:** No --- > [!IMPORTANT] > > 1. Make sure you have read our [contribution guidelines](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) > 1. Ensure there is an associated issue and you have been assigned to it > 1. Use the correct syntax to link this PR: `Fixes #<issue number>`. ## Summary <!-- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. --> fix https://github.com/langgenius/dify/issues/31888 ## Screenshots | Before | After | |--------|-------| | ... | ... | ## Checklist - [ ] This change requires a documentation update, included: [Dify Document](https://github.com/langgenius/dify-docs) - [x] I understand that this PR may be closed in case there was no previous discussion or issues. (This doesn't apply to typos!) - [x] I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change. - [x] I've updated the documentation accordingly. - [x] I ran `make lint` and `make type-check` (backend) and `cd web && npx lint-staged` (frontend) to appease the lint gods # Chat app vision responses not rendering - message_file event handler bug (regression from #22552) **Related Issue:** #22552 (closed) - The fix in commit `084dcd1a50` has a bug that prevents vision responses from rendering in chat apps. **Fixes #31888 **Dify Version:** `1.11.2`, `1.11.4`, `hotfix-1.11.4-fix.4` (likely affects `1.12.0` as well) **Cloud or Self Hosted:** Self Hosted (Kubernetes) **Description:** When uploading an image to a **chat app** (not workflow), the AI response is completely empty - neither the text answer nor the image are rendered in the chat interface. The HTML container exists but is empty. This is a regression from the fix in #22552. ## Steps to Reproduce 1. Create a chat app with vision model enabled (e.g., GPT-4 Vision, Claude 3.5 Sonnet with vision) 2. Upload an image file (e.g., PNG, JPG) in the chat input 3. Send a message asking about the image (e.g., "explain this image") 4. Wait for AI response 5. Observe the chat interface - response container is empty **Note:** Check browser DevTools → Network tab to see SSE events are being received, but nothing renders in the UI. ## Expected Behavior - ✅ AI generates a text response describing the image - ✅ Image thumbnail is displayed in the chat history - ✅ Both text answer and image are visible in the chat interface - ✅ User can see the complete conversation with image context ## Actual Behavior - ✅ AI generates a text response (confirmed via API logs and SSE events) - ✅ Image file is uploaded successfully (201 response from `/console/api/files/upload`) - ✅ File is stored correctly in database and filesystem - ✅ File preview URL works when accessed directly in browser - ❌ **Chat interface shows completely empty div container** - no text, no image - ❌ Frontend receives `message_file` SSE events but doesn't render them - ❌ `message_end` event has `"files": null` instead of files array - ❌ HTML structure exists but content is missing (empty `<div class="chat-answer-container">`) --- ## Additional Information ### Technical Analysis ### Backend (API) Behavior The API correctly: 1. Receives file upload → Returns 201 with `upload_file_id` 2. Processes image → Generates response with correct tokens 3. Sends SSE events: - `message_file` events with file data ✅ - `message_end` event with `"files": null` ❌ (should contain files array) ### Frontend Behavior The frontend (`web/app/components/base/chat/chat/hooks.ts`): **Problem 1: `onFile` handler (line 419-422)** ```typescript onFile(file) { const lastThought = responseItem.agent_thoughts?.[responseItem.agent_thoughts?.length - 1] if (lastThought) // ← BUG: Only adds files if agent_thoughts exists! responseItem.agent_thoughts![responseItem.agent_thoughts!.length - 1].message_files = [...(lastThought as any).message_files, file] // Missing: else clause to add files to responseItem.message_files for chat apps } ``` **Problem 2: `onMessageEnd` handler (line 477-478)** ```typescript const processedFilesFromResponse = getProcessedFilesFromResponse(messageEnd.files || []) responseItem.allFiles = uniqBy([...(responseItem.allFiles || []), ...(processedFilesFromResponse || [])], 'id') ``` But `message_end` event has `"files": null` instead of the files array. ### Root Cause The fix in commit `084dcd1a50` (issue #22552) has a bug: **File:** `api/core/app/task_pipeline/message_cycle_manager.py` **Current (incorrect) code:** ```python message_file = db.session.query(MessageFile).filter(MessageFile.id == message_id).first() event_type = StreamEvent.MESSAGE_FILE if message_file else StreamEvent.MESSAGE ``` **Problem:** `message_id` is the **message ID**, not the file ID! The query should check if there are files **associated with** the message, not if a file **is** the message. **Should be:** ```python message_file = db.session.query(MessageFile).filter(MessageFile.message_id == message_id).first() event_type = StreamEvent.MESSAGE_FILE if message_file else StreamEvent.MESSAGE ``` Additionally, the `message_end` event should include the files array, but it's currently `null`. ## Evidence ### SSE Events Received: ```json data: {"event":"message_file","conversation_id":"...","message_id":"...","answer":"","from_variable_selector":null} data: {"event":"message_file","conversation_id":"...","message_id":"...","answer":"I","from_variable_selector":null} ... (streaming text chunks) data: {"event":"message_end","conversation_id":"...","message_id":"...","files":null} ← Should contain files! ``` ### API Response (GET /chat-messages): ```json { "answer": "I'm sorry for the confusion...", "message_files": [ { "id": "858106aa-d422-4922-9b7f-f52eed3f0619", "filename": "twilio.png", "url": "/files/.../file-preview?...", ... } ] } ``` ### Browser DOM: ```html <div class="chat-answer-container"> <div class="body-lg-regular"> <!-- EMPTY - no text, no image --> </div> </div> ``` ## Proposed Fix ### Backend Fix: 1. Fix the query in `message_cycle_manager.py`: ```python message_file = db.session.query(MessageFile).filter(MessageFile.message_id == message_id).first() ``` 2. Ensure `message_end` event includes files array: ```python # In message_end_to_stream_response method files = db.session.query(MessageFile).filter(MessageFile.message_id == message_id).all() return MessageEndStreamResponse(..., files=files) ``` ### Frontend Fix (if backend fix doesn't solve it): 1. Update `onFile` handler to handle chat apps: ```typescript onFile(file) { const lastThought = responseItem.agent_thoughts?.[responseItem.agent_thoughts?.length - 1] if (lastThought) { responseItem.agent_thoughts![responseItem.agent_thoughts!.length - 1].message_files = [...(lastThought as any).message_files, file] } else { // FIX: Add files to message_files for chat apps responseItem.message_files = [...(responseItem.message_files || []), file] } updateCurrentQAOnTree({...}) } ``` ## Impact - **Severity:** High - Vision features completely broken for chat apps - **Affected Users:** All users trying to use vision/image features in chat apps - **Workaround:** None - feature is unusable ## Additional Context - File upload works correctly - File storage works correctly - File preview URLs work when accessed directly - The issue is specifically in the rendering pipeline between SSE events and DOM
yindo added the pull-request label 2026-02-21 20:53:20 -05:00
yindo closed this issue 2026-02-21 20:53:20 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#33471
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?