langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-22 04:25:29 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

[PR #26209] fix: resolve race condition by adding lock to workflow execution and node execution in-memory cache #31359

New Issue
Closed
opened 2026-02-21 20:49:19 -05:00 by yindo · 0 comments
yindo commented 2026-02-21 20:49:19 -05:00
Owner
Copy Link
Copy Source

Original Pull Request: https://github.com/langgenius/dify/pull/26209

State: closed
Merged: No

Fixes #25244

Summary

During a load testing, we've discovered a high-frequency database error using celery workflow repositories:

repositories:
    coreWorkflowExecutionRepository: core.repositories.celery_workflow_execution_repository.CeleryWorkflowExecutionRepository
    coreWorkflowNodeExecutionRepository: core.repositories.celery_workflow_node_execution_repository.CeleryWorkflowNodeExecutionRepository
image

this is caused by the in-memory dict caches, which are not greenlet safe, in WorkflowCycleManager
image

so the solution came clear: add locks to read and write operations targeting these caches

Before-Afters

Errors

Before

20 samples yielded many errors
image

After

No more errors in 23 samples collected
image

RPS

Before

around 210-ish
image

After

220-ish, the fix did not decrease RPS at all
image

Checklist

  • This change requires a documentation update, included: Dify Document
  • I understand that this PR may be closed in case there was no previous discussion or issues. (This doesn't apply to typos!)
  • I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
  • I've updated the documentation accordingly.
  • I ran dev/reformat(backend) and cd web && npx lint-staged(frontend) to appease the lint gods
**Original Pull Request:** https://github.com/langgenius/dify/pull/26209 **State:** closed **Merged:** No --- Fixes #25244 ## Summary During a load testing, we've discovered a high-frequency database error using celery workflow repositories: ``` repositories: coreWorkflowExecutionRepository: core.repositories.celery_workflow_execution_repository.CeleryWorkflowExecutionRepository coreWorkflowNodeExecutionRepository: core.repositories.celery_workflow_node_execution_repository.CeleryWorkflowNodeExecutionRepository ``` <img width="1920" height="919" alt="image" src="https://github.com/user-attachments/assets/22d2b454-1485-4ba1-81f5-52aba16ff9ac" /> this is caused by the in-memory dict caches, which are not greenlet safe, in WorkflowCycleManager <img width="656" height="492" alt="image" src="https://github.com/user-attachments/assets/e32e8c55-3cf5-49c8-a30f-6798397ce9a0" /> so the solution came clear: add locks to read and write operations targeting these caches ## Before-Afters ### Errors #### Before 20 samples yielded many errors <img width="565" height="486" alt="image" src="https://github.com/user-attachments/assets/441fe105-1cb3-41f4-98b8-2116122db8f2" /> #### After No more errors in 23 samples collected <img width="986" height="573" alt="image" src="https://github.com/user-attachments/assets/bfa74a71-7900-49d5-a75a-81325561d348" /> ### RPS #### Before around 210-ish <img width="1304" height="738" alt="image" src="https://github.com/user-attachments/assets/fae2b38a-1f39-4f93-933f-d8f02b198080" /> #### After 220-ish, the fix did not decrease RPS at all <img width="1424" height="681" alt="image" src="https://github.com/user-attachments/assets/e7cc8519-8c6b-4c32-9078-16712067af1e" /> ## Checklist - [X] This change requires a documentation update, included: [Dify Document](https://github.com/langgenius/dify-docs) - [x] I understand that this PR may be closed in case there was no previous discussion or issues. (This doesn't apply to typos!) - [x] I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change. - [x] I've updated the documentation accordingly. - [x] I ran `dev/reformat`(backend) and `cd web && npx lint-staged`(frontend) to appease the lint gods
yindo added the pull-request label 2026-02-21 20:49:19 -05:00
yindo closed this issue 2026-02-21 20:49:19 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#31359
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?