langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 12:05:30 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

[PR #30236] fix: release graph_runtime_state reference to prevent memory leak under high load #32737

New Issue
Closed
opened 2026-02-21 20:51:58 -05:00 by yindo · 0 comments
yindo commented 2026-02-21 20:51:58 -05:00
Owner
Copy Link
Copy Source

Original Pull Request: https://github.com/langgenius/dify/pull/30236

State: closed
Merged: Yes

Problem Description

Fixes the memory leak issue reported in #30183, where memory usage continuously increases under high load conditions.

Root Cause Analysis

After detailed code analysis, we identified that the memory leak originates from the _graph_runtime_state reference in AppQueueManager not being cleared after request completion.

Reference Chain Causing the Leak

  1. AppQueueManager holds a strong reference to GraphRuntimeState via _graph_runtime_state
  2. GraphRuntimeState contains the complete workflow runtime state (variable pool, outputs, LLM usage stats, etc.)
  3. Circular Reference: Each Node instance holds a reference back to graph_runtime_state (see api/core/workflow/nodes/base/node.py line 220)
  4. When stop_listen() is called, although the request has ended, the entire object graph cannot be garbage collected because AppQueueManager still holds the reference
  5. During the TaskPipeline lifecycle, this memory cannot be reclaimed

Key Code Locations

  • api/core/app/apps/base_app_queue_manager.py: _graph_runtime_state attribute
  • api/core/workflow/nodes/base/node.py: Node holds graph_runtime_state reference (circular reference)
  • api/core/workflow/graph_engine/graph_engine.py: EventManager accumulates events

Fix

Explicitly set _graph_runtime_state to None in the stop_listen() method, breaking the reference chain and allowing Python GC to immediately reclaim the entire workflow object graph.

def stop_listen(self):
    self._clear_task_belong_cache()
    self._q.put(None)
    self._graph_runtime_state = None  # Release reference to allow GC to reclaim memory

Testing Methodology

Approach

Since directly testing memory leaks in a production-like Docker environment is complex and time-consuming, we created a Python simulation script that accurately replicates the object relationships and lifecycle of the real Dify workflow execution.

What the Simulation Script Does

  1. Recreates the object graph structure:

    • MockGraphRuntimeState: Simulates the runtime state with variable pool, outputs, and LLM usage data
    • MockNode: Simulates workflow nodes with the exact circular reference pattern (self.graph_runtime_state = graph_runtime_state)
    • MockGraph: Holds references to all nodes
    • MockEventManager: Accumulates 100 events per workflow (simulating real execution)
    • MockAppQueueManager: Two versions - one WITHOUT the fix, one WITH the fix

  2. Simulates realistic data sizes:

    • Each node contains config data (~2KB prompt templates, variables)
    • Each event contains inputs/outputs (~4KB per event)
    • Variable pool contains user inputs, file data, and intermediate results

  3. Tests the critical lifecycle:

    # Step 1: Create workflow request
pipeline, state_ref = simulate_workflow_request(queue_manager_class)

# Step 2: Request ends, stop_listen() is called
pipeline.queue_manager.stop_listen()

# Step 3: Force garbage collection
gc.collect()

# Step 4: Check if GraphRuntimeState is still alive using weakref
if state_ref() is not None:
    print("MEMORY LEAK: Object still alive!")

  4. Uses Python weakref to verify GC behavior:

    • We store weakref.ref(graph_runtime_state) before the test
    • After stop_listen() and gc.collect(), we check if the weak reference is still valid
    • If valid → object was NOT garbage collected → memory leak
    • If None → object WAS garbage collected → no leak

Test Scale

  • 500 concurrent workflow requests simulated
  • Each request contains:
    • 20 nodes (LLM, Code, HTTP, Knowledge Retrieval, etc.)
    • 100 execution events
    • Complete variable pool and runtime state
    • ~111 KB of data per request

Test Results

Metric Before Fix After Fix
Memory held after stop_listen() 54.3 MB 0.2 MB
Surviving GraphRuntimeState objects 500/500 0/500
Memory leak risk ❌ High ✅ None

Key Findings

  • Before fix: All 500 GraphRuntimeState objects survive after stop_listen(), holding 54.3 MB of memory that cannot be reclaimed
  • After fix: All GraphRuntimeState objects are immediately eligible for garbage collection after stop_listen()

Why This Simulation is Valid

The simulation accurately captures the exact cause of the memory leak:

  1. The circular reference pattern between GraphRuntimeState and Node is replicated exactly as in the real code
  2. The external reference from AppQueueManager._graph_runtime_state that prevents GC is replicated
  3. The lifecycle where stop_listen() is called but the TaskPipeline (and thus AppQueueManager) is still alive matches real execution

Python's garbage collector can handle circular references, but only if there are no external references to the cycle. The AppQueueManager._graph_runtime_state reference is that external reference that keeps the entire object graph alive.

Impact in Production

In real high-load scenarios (as described in Issue #30183 with 100-thread stress tests):

  • Before fix: Each request holds ~111 KB of workflow state that cannot be released promptly. With 100 threads × multiple test runs → memory accumulates continuously
  • After fix: Memory is released immediately after each request ends, keeping memory usage stable

Type

  • Bug Fix

Closes #30183

**Original Pull Request:** https://github.com/langgenius/dify/pull/30236 **State:** closed **Merged:** Yes --- ## Problem Description Fixes the memory leak issue reported in #30183, where memory usage continuously increases under high load conditions. ## Root Cause Analysis After detailed code analysis, we identified that the memory leak originates from the `_graph_runtime_state` reference in `AppQueueManager` not being cleared after request completion. ### Reference Chain Causing the Leak 1. **`AppQueueManager`** holds a strong reference to `GraphRuntimeState` via `_graph_runtime_state` 2. **`GraphRuntimeState`** contains the complete workflow runtime state (variable pool, outputs, LLM usage stats, etc.) 3. **Circular Reference**: Each `Node` instance holds a reference back to `graph_runtime_state` (see `api/core/workflow/nodes/base/node.py` line 220) 4. When `stop_listen()` is called, although the request has ended, the entire object graph cannot be garbage collected because `AppQueueManager` still holds the reference 5. During the `TaskPipeline` lifecycle, this memory cannot be reclaimed ### Key Code Locations - `api/core/app/apps/base_app_queue_manager.py`: `_graph_runtime_state` attribute - `api/core/workflow/nodes/base/node.py`: Node holds `graph_runtime_state` reference (circular reference) - `api/core/workflow/graph_engine/graph_engine.py`: `EventManager` accumulates events ## Fix Explicitly set `_graph_runtime_state` to `None` in the `stop_listen()` method, breaking the reference chain and allowing Python GC to immediately reclaim the entire workflow object graph. ```python def stop_listen(self): self._clear_task_belong_cache() self._q.put(None) self._graph_runtime_state = None # Release reference to allow GC to reclaim memory ``` ## Testing Methodology ### Approach Since directly testing memory leaks in a production-like Docker environment is complex and time-consuming, we created a **Python simulation script** that accurately replicates the object relationships and lifecycle of the real Dify workflow execution. ### What the Simulation Script Does 1. **Recreates the object graph structure**: - `MockGraphRuntimeState`: Simulates the runtime state with variable pool, outputs, and LLM usage data - `MockNode`: Simulates workflow nodes with the **exact circular reference** pattern (`self.graph_runtime_state = graph_runtime_state`) - `MockGraph`: Holds references to all nodes - `MockEventManager`: Accumulates 100 events per workflow (simulating real execution) - `MockAppQueueManager`: Two versions - one WITHOUT the fix, one WITH the fix 2. **Simulates realistic data sizes**: - Each node contains config data (~2KB prompt templates, variables) - Each event contains inputs/outputs (~4KB per event) - Variable pool contains user inputs, file data, and intermediate results 3. **Tests the critical lifecycle**: ```python # Step 1: Create workflow request pipeline, state_ref = simulate_workflow_request(queue_manager_class) # Step 2: Request ends, stop_listen() is called pipeline.queue_manager.stop_listen() # Step 3: Force garbage collection gc.collect() # Step 4: Check if GraphRuntimeState is still alive using weakref if state_ref() is not None: print("MEMORY LEAK: Object still alive!") ``` 4. **Uses Python weakref to verify GC behavior**: - We store `weakref.ref(graph_runtime_state)` before the test - After `stop_listen()` and `gc.collect()`, we check if the weak reference is still valid - If valid → object was NOT garbage collected → **memory leak** - If `None` → object WAS garbage collected → **no leak** ### Test Scale - **500 concurrent workflow requests** simulated - Each request contains: - 20 nodes (LLM, Code, HTTP, Knowledge Retrieval, etc.) - 100 execution events - Complete variable pool and runtime state - ~111 KB of data per request ### Test Results | Metric | Before Fix | After Fix | |--------|------------|-----------| | Memory held after `stop_listen()` | **54.3 MB** | **0.2 MB** | | Surviving `GraphRuntimeState` objects | 500/500 | 0/500 | | Memory leak risk | ❌ High | ✅ None | ### Key Findings - **Before fix**: All 500 `GraphRuntimeState` objects survive after `stop_listen()`, holding 54.3 MB of memory that cannot be reclaimed - **After fix**: All `GraphRuntimeState` objects are immediately eligible for garbage collection after `stop_listen()` ### Why This Simulation is Valid The simulation accurately captures the **exact cause** of the memory leak: 1. **The circular reference pattern** between `GraphRuntimeState` and `Node` is replicated exactly as in the real code 2. **The external reference** from `AppQueueManager._graph_runtime_state` that prevents GC is replicated 3. **The lifecycle** where `stop_listen()` is called but the `TaskPipeline` (and thus `AppQueueManager`) is still alive matches real execution Python's garbage collector can handle circular references, but only if there are **no external references** to the cycle. The `AppQueueManager._graph_runtime_state` reference is that external reference that keeps the entire object graph alive. ## Impact in Production In real high-load scenarios (as described in Issue #30183 with 100-thread stress tests): - **Before fix**: Each request holds ~111 KB of workflow state that cannot be released promptly. With 100 threads × multiple test runs → memory accumulates continuously - **After fix**: Memory is released immediately after each request ends, keeping memory usage stable ## Type - [x] Bug Fix Closes #30183
yindo added the pull-request label 2026-02-21 20:51:58 -05:00
yindo closed this issue 2026-02-21 20:51:58 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#32737
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?