[PR #481] [CLOSED] Llama Deploy Race Condition #494

New Issue

2026-02-16T02:15:14-05:00

yindo commented

2026-02-16 02:15:14 -05:00

📋 Pull Request Information

Original PR: https://github.com/run-llama/llama_deploy/pull/481
Author: @eyao27
Created: 3/7/2025
Status: ❌ Closed

Base: main ← Head: bug_fix

📝 Commits (3)

f19c48b race condition fix
d39817f Update workflow.py
05c094a Merge branch 'main' into bug_fix

📊 Changes

1 file changed (+4 additions, -4 deletions)

View changed files

📝 llama_deploy/services/workflow.py (+4 -4)

📄 Description

Problem

At our company, we use LlamaIndex Workflows for chat bot services. We are encountering a very subtle race condition bug where certain events in the Workflow's event stream are missing.

Root Cause

After some debugging, we've narrowed the root cause down to the set_workflow_state function of the Workflow Service in Llama Deploy. This function is run when the workflow has completed. It's purpose is to update the state of the workflow on the control plane. However, a subtle bug can occur when the workflow is finished, yet there are still streaming messages that the control plane hasn't processed.

Details

In the function set_workflow_state, the variable session_state contains numerous keys, one for the session ID, but also many for the tasks, results and streams:

For example, there could be keys for:

01d59890-f639-448b-984b-b97e113d0d41 (session ID)
504b8a6d-b846-4bbb-b684-a2b2051fcd81 (task ID)
stream_504b8a6d-b846-4bbb-b684-a2b2051fcd81
result_504b8a6d-b846-4bbb-b684-a2b2051fcd81

The current code works by:

Getting this dictionary with many keys
- session_state = await self.get_session_state(current_state.session_id)
Changing the value for the key 01d59890-f639-448b-984b-b97e113d0d41 (i.e., current_state.session_id).
- session_state[current_state.session_id] = workflow_state.model_dump_json()
Setting the entire dictionary.
- await self.update_session_state(current_state.session_id, session_state)

This introduces a possible race condition where between steps 1 and 3, the value for other keys, namely the stream_504b8a6d-b846-4bbb-b684-a2b2051fcd81 can change. In particular, additional messages may have been received, but are not processed because they are replaced in step 3.

Solution

The modified code works by only updating the value for the key 01d59890-f639-448b-984b-b97e113d0d41 (the session ID), without changing the other values for the other keys.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/run-llama/llama_deploy/pull/481 **Author:** [@eyao27](https://github.com/eyao27) **Created:** 3/7/2025 **Status:** ❌ Closed **Base:** `main` ← **Head:** `bug_fix` --- ### 📝 Commits (3) - [`f19c48b`](https://github.com/run-llama/llama_deploy/commit/f19c48b8ce8ea4082e4248153b144841f0ec5635) race condition fix - [`d39817f`](https://github.com/run-llama/llama_deploy/commit/d39817fbcd532b465139e009ff91c013445d36f9) Update workflow.py - [`05c094a`](https://github.com/run-llama/llama_deploy/commit/05c094ac9e094852371ee09410cb8c3340f690c8) Merge branch 'main' into bug_fix ### 📊 Changes **1 file changed** (+4 additions, -4 deletions) <details> <summary>View changed files</summary> 📝 `llama_deploy/services/workflow.py` (+4 -4) </details> ### 📄 Description ## Problem At our company, we use LlamaIndex Workflows for chat bot services. We are encountering a very subtle race condition bug where certain events in the Workflow's event stream are missing. ## Root Cause After some debugging, we've narrowed the root cause down to the `set_workflow_state` function of the Workflow Service in Llama Deploy. This function is run when the workflow has completed. It's purpose is to update the state of the workflow on the control plane. However, a subtle bug can occur when the workflow is finished, yet there are still streaming messages that the control plane hasn't processed. ## Details In the function `set_workflow_state`, the variable `session_state` contains numerous keys, one for the session ID, but also many for the tasks, results and streams: For example, there could be keys for: * `01d59890-f639-448b-984b-b97e113d0d41` (session ID) * `504b8a6d-b846-4bbb-b684-a2b2051fcd81` (task ID) * `stream_504b8a6d-b846-4bbb-b684-a2b2051fcd81` * `result_504b8a6d-b846-4bbb-b684-a2b2051fcd81` The current code works by: 1. Getting this dictionary with many keys * `session_state = await self.get_session_state(current_state.session_id)` 2. Changing the value for the key `01d59890-f639-448b-984b-b97e113d0d41` (i.e., `current_state.session_id`). * `session_state[current_state.session_id] = workflow_state.model_dump_json()` 3. Setting the entire dictionary. * `await self.update_session_state(current_state.session_id, session_state)` This introduces a possible race condition where between steps 1 and 3, the value for other keys, namely the `stream_504b8a6d-b846-4bbb-b684-a2b2051fcd81` can change. In particular, additional messages may have been received, but are not processed because they are replaced in step 3. ## Solution The modified code works by only updating the value for the key `01d59890-f639-448b-984b-b97e113d0d41` (the session ID), without changing the other values for the other keys. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

yindo added the pull-request label 2026-02-16 02:15:14 -05:00

yindo closed this issue

2026-02-16 02:15:14 -05:00

yindo referenced this issue

2026-02-16 02:15:22 -05:00

[PR #497] [MERGED] fix: actually use Redis as message queue #499

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: run-llama/llama_deploy#494