langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 12:05:30 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Workflow Execution Failures After Upgrading from v1.7.2 to v2.0.0-beta.2 #17722

New Issue
Closed
opened 2026-02-21 19:38:50 -05:00 by yindo · 0 comments
yindo commented 2026-02-21 19:38:50 -05:00
Owner
Copy Link
Copy Source

Originally created by @quduc on GitHub (Sep 18, 2025).

Self Checks

  • I have read the Contributing Guide and Language Policy.
  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report, otherwise it will be closed.
  • 【中文用户 & Non English User】请使用英语提交，否则会被关闭 ：）
  • Please do not modify this template :) and fill in all the required fields.

Dify version

2.0.0-beta.2

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

Bug Description

After upgrading from Dify v1.7.2 to v2.0.0-beta.2, experiencing multiple critical workflow execution failures with high error rates (~30% of workflows failing).

Environment

  • Dify Version: v2.0.0-beta.2 (upgraded from v1.7.2)
  • Deployment: Docker Compose
  • Database: PostgreSQL
  • LLM Providers: Google Gemini, OpenAI
  • OS: Ubuntu (Docker host)

Reproduction Steps

  1. Upgrade from v1.7.2 to v2.0.0-beta.2
  2. Run datasource migration: uv run flask transform-datasource-credentials
  3. Execute any workflow with:
    • Large context inputs
    • Plugin integrations
    • File processing
    • Agent nodes

Expected Behavior

Workflows should execute successfully as they did in v1.7.2

Actual Behavior

Multiple types of workflow failures occurring frequently:

Error Categories:

1. Context Limit Exceeded (Most Common)

input length and max_tokens exceed context limit: 143956 + 64000 > 200000

2. Empty Message Content

all messages must have non-empty content except for the optional final assistant message

3. Plugin Timeouts

PluginDaemonInternalServerError: killed by timeout

4. Service Unavailability

503 UNAVAILABLE. The service is currently unavailable / The model is overloaded

5. File Not Found Errors

FileNotFoundError: File not found

6. Database UUID Errors

invalid input syntax for type uuid: ""

7. Datasource Migration Conflicts

duplicate key value violates unique constraint "datasource_provider_unique_name"

Sample Error Logs

Click to expand error logs

2025-09-18 05:48:52.476 ERROR [GraphWorker-1] [base.py:211] - Error in stream reponse for plugin {'code': -500, 'message':
'{"message":"{"args":{},"error_type":"Exception","message":"read llm model failed: request failed: [google] Error: req_id: fe9be85572 PluginInvokeError:
{"args":{},"error_type":"ServerError","message":"503 UNAVAILABLE. {error: {code: 503, message: The service is currently unavailable., status:
UNAVAILABLE}}"}"}","error_type":"PluginInvokeError","args":null}', 'data': None}

2025-09-18 06:46:18.036 ERROR [GraphWorker-2] [base.py:211] - Error in stream reponse for plugin {'code': -500, 'message':
'{"message":"{"args":{},"error_type":"Exception","message":"read llm model failed: request failed: [models] Bad Request Error, Error code: 400 - {type: error, error: {type:
invalid_request_error, message: messages.13: all messages must have non-empty content except for the optional final assistant message}, request_id:
req_011CTFR6zTVsWYET5b8Aciz3}"}","error_type":"PluginInvokeError","args":null}', 'data': None}

2025-09-18 07:00:05.775 ERROR [GraphWorker-0] [node.py:341] - error while executing llm node
FileNotFoundError: File not found

Analysis

The errors appear to be related to major architectural changes in v2.0.0-beta.1:

  1. Queue-based Graph Engine: New execution limits and queue management
  2. Knowledge Pipeline: Changed file storage and retrieval mechanisms
  3. Plugin Architecture: Refactored plugin system with different timeout handling

Workarounds Attempted

  • Ran datasource credential migration
  • Enabled DEBUG logging
  • Cleaned up duplicate datasource providers
  • Implemented context truncation
  • Adjusted plugin timeouts

Impact

  • High workflow failure rate (~30%)
  • Production systems affected
  • User experience degraded
  • Multiple app types failing (AI assistants, business plan analyzers, etc.)

Additional Context

  • Migration appeared successful initially
  • Issues manifest under load
  • Multiple tenants affected
  • Both simple and complex workflows failing

Possible Solutions

  1. Migration Scripts: Better handling of datasource credential duplicates
  2. Context Management: Automatic truncation before hitting limits
  3. Plugin Timeouts: Configurable timeout values for different node types
  4. Error Handling: Better fallback mechanisms for service unavailability
  5. File Storage: Health checks for file existence before processing

Logs Available

Can provide full logs and database dump if needed for debugging.

Priority: High - Production workflows failing
Labels: bug, v2.0.0-beta.2, workflow-execution, migration

✔️ Expected Behavior

Bug Description

After upgrading from Dify v1.7.2 to v2.0.0-beta.2, experiencing multiple critical workflow execution failures with high error rates (~30% of workflows failing).

Environment

  • Dify Version: v2.0.0-beta.2 (upgraded from v1.7.2)
  • Deployment: Docker Compose
  • Database: PostgreSQL
  • LLM Providers: Google Gemini, OpenAI
  • OS: Ubuntu (Docker host)

Reproduction Steps

  1. Upgrade from v1.7.2 to v2.0.0-beta.2
  2. Run datasource migration: uv run flask transform-datasource-credentials
  3. Execute any workflow with:
    • Large context inputs
    • Plugin integrations
    • File processing
    • Agent nodes

Expected Behavior

Workflows should execute successfully as they did in v1.7.2

Actual Behavior

Multiple types of workflow failures occurring frequently:

Error Categories:

1. Context Limit Exceeded (Most Common)

input length and max_tokens exceed context limit: 143956 + 64000 > 200000

2. Empty Message Content

all messages must have non-empty content except for the optional final assistant message

3. Plugin Timeouts

PluginDaemonInternalServerError: killed by timeout

4. Service Unavailability

503 UNAVAILABLE. The service is currently unavailable / The model is overloaded

5. File Not Found Errors

FileNotFoundError: File not found

6. Database UUID Errors

invalid input syntax for type uuid: ""

Sample Error Logs

Click to expand error logs

2025-09-18 05:48:52.476 ERROR [GraphWorker-1] [base.py:211] - Error in stream reponse for plugin {'code': -500, 'message':
'{"message":"{"args":{},"error_type":"Exception","message":"read llm model failed: request failed: [google] Error: req_id: fe9be85572 PluginInvokeError:
{"args":{},"error_type":"ServerError","message":"503 UNAVAILABLE. {error: {code: 503, message: The service is currently unavailable., status:
UNAVAILABLE}}"}"}","error_type":"PluginInvokeError","args":null}', 'data': None}

2025-09-18 06:46:18.036 ERROR [GraphWorker-2] [base.py:211] - Error in stream reponse for plugin {'code': -500, 'message':
'{"message":"{"args":{},"error_type":"Exception","message":"read llm model failed: request failed: [models] Bad Request Error, Error code: 400 - {type: error, error: {type:
invalid_request_error, message: messages.13: all messages must have non-empty content except for the optional final assistant message}, request_id:
req_011CTFR6zTVsWYET5b8Aciz3}"}","error_type":"PluginInvokeError","args":null}', 'data': None}

2025-09-18 07:00:05.775 ERROR [GraphWorker-0] [node.py:341] - error while executing llm node
FileNotFoundError: File not found

Analysis

The errors appear to be related to major architectural changes in v2.0.0-beta.1:

  1. Queue-based Graph Engine: New execution limits and queue management
  2. Knowledge Pipeline: Changed file storage and retrieval mechanisms
  3. Plugin Architecture: Refactored plugin system with different timeout handling

Workarounds Attempted

  • Ran datasource credential migration
  • Enabled DEBUG logging
  • Cleaned up duplicate datasource providers
  • Implemented context truncation
  • Adjusted plugin timeouts

Impact

  • High workflow failure rate (~30%)
  • Production systems affected
  • User experience degraded
  • Multiple app types failing (AI assistants, business plan analyzers, etc.)

Additional Context

  • Migration appeared successful initially
  • Issues manifest under load
  • Multiple tenants affected
  • Both simple and complex workflows failing

Possible Solutions

  1. Migration Scripts: Better handling of datasource credential duplicates
  2. Context Management: Automatic truncation before hitting limits
  3. Plugin Timeouts: Configurable timeout values for different node types
  4. Error Handling: Better fallback mechanisms for service unavailability
  5. File Storage: Health checks for file existence before processing

Logs Available

Can provide full logs and database dump if needed for debugging.

Priority: High - Production workflows failing
Labels: bug, v2.0.0-beta.2, workflow-execution, migration

❌ Actual Behavior

No response

Originally created by @quduc on GitHub (Sep 18, 2025). ### Self Checks - [x] I have read the [Contributing Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) and [Language Policy](https://github.com/langgenius/dify/issues/1542). - [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report, otherwise it will be closed. - [x] 【中文用户 & Non English User】请使用英语提交，否则会被关闭 ：） - [x] Please do not modify this template :) and fill in all the required fields. ### Dify version 2.0.0-beta.2 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce ## **Bug Description** After upgrading from Dify v1.7.2 to v2.0.0-beta.2, experiencing multiple critical workflow execution failures with high error rates (~30% of workflows failing). ## **Environment** - **Dify Version**: v2.0.0-beta.2 (upgraded from v1.7.2) - **Deployment**: Docker Compose - **Database**: PostgreSQL - **LLM Providers**: Google Gemini, OpenAI - **OS**: Ubuntu (Docker host) ## **Reproduction Steps** 1. Upgrade from v1.7.2 to v2.0.0-beta.2 2. Run datasource migration: `uv run flask transform-datasource-credentials` 3. Execute any workflow with: - Large context inputs - Plugin integrations - File processing - Agent nodes ## **Expected Behavior** Workflows should execute successfully as they did in v1.7.2 ## **Actual Behavior** Multiple types of workflow failures occurring frequently: ### **Error Categories:** #### 1. Context Limit Exceeded (Most Common) input length and max_tokens exceed context limit: 143956 + 64000 > 200000 #### 2. Empty Message Content all messages must have non-empty content except for the optional final assistant message #### 3. Plugin Timeouts PluginDaemonInternalServerError: killed by timeout #### 4. Service Unavailability 503 UNAVAILABLE. The service is currently unavailable / The model is overloaded #### 5. File Not Found Errors FileNotFoundError: File not found #### 6. Database UUID Errors invalid input syntax for type uuid: "" #### 7. Datasource Migration Conflicts duplicate key value violates unique constraint "datasource_provider_unique_name" ## **Sample Error Logs** <details> <summary>Click to expand error logs</summary> 2025-09-18 05:48:52.476 ERROR [GraphWorker-1] [base.py:211] - Error in stream reponse for plugin {'code': -500, 'message': '{"message":"{"args":{},"error_type":"Exception","message":"read llm model failed: request failed: [google] Error: req_id: fe9be85572 PluginInvokeError: {"args":{},"error_type":"ServerError","message":"503 UNAVAILABLE. {error: {code: 503, message: The service is currently unavailable., status: UNAVAILABLE}}"}"}","error_type":"PluginInvokeError","args":null}', 'data': None} 2025-09-18 06:46:18.036 ERROR [GraphWorker-2] [base.py:211] - Error in stream reponse for plugin {'code': -500, 'message': '{"message":"{"args":{},"error_type":"Exception","message":"read llm model failed: request failed: [models] Bad Request Error, Error code: 400 - {type: error, error: {type: invalid_request_error, message: messages.13: all messages must have non-empty content except for the optional final assistant message}, request_id: req_011CTFR6zTVsWYET5b8Aciz3}"}","error_type":"PluginInvokeError","args":null}', 'data': None} 2025-09-18 07:00:05.775 ERROR [GraphWorker-0] [node.py:341] - error while executing llm node FileNotFoundError: File not found </details> ## **Analysis** The errors appear to be related to major architectural changes in v2.0.0-beta.1: 1. **Queue-based Graph Engine**: New execution limits and queue management 2. **Knowledge Pipeline**: Changed file storage and retrieval mechanisms 3. **Plugin Architecture**: Refactored plugin system with different timeout handling ## **Workarounds Attempted** - [x] Ran datasource credential migration - [x] Enabled DEBUG logging - [x] Cleaned up duplicate datasource providers - [ ] Implemented context truncation - [ ] Adjusted plugin timeouts ## **Impact** - High workflow failure rate (~30%) - Production systems affected - User experience degraded - Multiple app types failing (AI assistants, business plan analyzers, etc.) ## **Additional Context** - Migration appeared successful initially - Issues manifest under load - Multiple tenants affected - Both simple and complex workflows failing ## **Possible Solutions** 1. **Migration Scripts**: Better handling of datasource credential duplicates 2. **Context Management**: Automatic truncation before hitting limits 3. **Plugin Timeouts**: Configurable timeout values for different node types 4. **Error Handling**: Better fallback mechanisms for service unavailability 5. **File Storage**: Health checks for file existence before processing ## **Logs Available** Can provide full logs and database dump if needed for debugging. --- **Priority**: High - Production workflows failing **Labels**: bug, v2.0.0-beta.2, workflow-execution, migration ### ✔️ Expected Behavior ## **Bug Description** After upgrading from Dify v1.7.2 to v2.0.0-beta.2, experiencing multiple critical workflow execution failures with high error rates (~30% of workflows failing). ## **Environment** - **Dify Version**: v2.0.0-beta.2 (upgraded from v1.7.2) - **Deployment**: Docker Compose - **Database**: PostgreSQL - **LLM Providers**: Google Gemini, OpenAI - **OS**: Ubuntu (Docker host) ## **Reproduction Steps** 1. Upgrade from v1.7.2 to v2.0.0-beta.2 2. Run datasource migration: `uv run flask transform-datasource-credentials` 3. Execute any workflow with: - Large context inputs - Plugin integrations - File processing - Agent nodes ## **Expected Behavior** Workflows should execute successfully as they did in v1.7.2 ## **Actual Behavior** Multiple types of workflow failures occurring frequently: ### **Error Categories:** #### 1. Context Limit Exceeded (Most Common) input length and max_tokens exceed context limit: 143956 + 64000 > 200000 #### 2. Empty Message Content all messages must have non-empty content except for the optional final assistant message #### 3. Plugin Timeouts PluginDaemonInternalServerError: killed by timeout #### 4. Service Unavailability 503 UNAVAILABLE. The service is currently unavailable / The model is overloaded #### 5. File Not Found Errors FileNotFoundError: File not found #### 6. Database UUID Errors invalid input syntax for type uuid: "" ## **Sample Error Logs** <details> <summary>Click to expand error logs</summary> 2025-09-18 05:48:52.476 ERROR [GraphWorker-1] [base.py:211] - Error in stream reponse for plugin {'code': -500, 'message': '{"message":"{"args":{},"error_type":"Exception","message":"read llm model failed: request failed: [google] Error: req_id: fe9be85572 PluginInvokeError: {"args":{},"error_type":"ServerError","message":"503 UNAVAILABLE. {error: {code: 503, message: The service is currently unavailable., status: UNAVAILABLE}}"}"}","error_type":"PluginInvokeError","args":null}', 'data': None} 2025-09-18 06:46:18.036 ERROR [GraphWorker-2] [base.py:211] - Error in stream reponse for plugin {'code': -500, 'message': '{"message":"{"args":{},"error_type":"Exception","message":"read llm model failed: request failed: [models] Bad Request Error, Error code: 400 - {type: error, error: {type: invalid_request_error, message: messages.13: all messages must have non-empty content except for the optional final assistant message}, request_id: req_011CTFR6zTVsWYET5b8Aciz3}"}","error_type":"PluginInvokeError","args":null}', 'data': None} 2025-09-18 07:00:05.775 ERROR [GraphWorker-0] [node.py:341] - error while executing llm node FileNotFoundError: File not found </details> ## **Analysis** The errors appear to be related to major architectural changes in v2.0.0-beta.1: 1. **Queue-based Graph Engine**: New execution limits and queue management 2. **Knowledge Pipeline**: Changed file storage and retrieval mechanisms 3. **Plugin Architecture**: Refactored plugin system with different timeout handling ## **Workarounds Attempted** - [x] Ran datasource credential migration - [x] Enabled DEBUG logging - [x] Cleaned up duplicate datasource providers - [ ] Implemented context truncation - [ ] Adjusted plugin timeouts ## **Impact** - High workflow failure rate (~30%) - Production systems affected - User experience degraded - Multiple app types failing (AI assistants, business plan analyzers, etc.) ## **Additional Context** - Migration appeared successful initially - Issues manifest under load - Multiple tenants affected - Both simple and complex workflows failing ## **Possible Solutions** 1. **Migration Scripts**: Better handling of datasource credential duplicates 2. **Context Management**: Automatic truncation before hitting limits 3. **Plugin Timeouts**: Configurable timeout values for different node types 4. **Error Handling**: Better fallback mechanisms for service unavailability 5. **File Storage**: Health checks for file existence before processing ## **Logs Available** Can provide full logs and database dump if needed for debugging. --- **Priority**: High - Production workflows failing **Labels**: bug, v2.0.0-beta.2, workflow-execution, migration ### ❌ Actual Behavior _No response_
yindo added the 🐞 bug label 2026-02-21 19:38:50 -05:00
yindo closed this issue 2026-02-21 19:38:50 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#17722
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?