run-llama/workflows-py
1
0
Fork 0
mirror of https://github.com/run-llama/workflows-py.git synced 2026-07-19 16:53:35 -04:00
Code Issues 18 Packages Projects Releases 10 Wiki Activity

ctx.to_dict() fails with 'Worker X not found in in_progress' when checkpointing resumed workflows #28

New Issue
Closed
opened 2026-02-16 02:16:11 -05:00 by yindo · 0 comments
yindo commented 2026-02-16 02:16:11 -05:00
Owner
Copy Link
Copy Source

Originally created by @OS-ramamurtisubramanian on GitHub (Jan 15, 2026).

Bug Description

When resuming a workflow from a checkpoint and then checkpointing again during the resumed run, ctx.to_dict() fails with:

ValueError: Worker 0 not found in in_progress

Steps to Reproduce

  1. Run a workflow with parallel workers (e.g., num_workers > 1)
  2. Checkpoint while workers are in progress
  3. Crash/stop the workflow
  4. Resume from checkpoint
  5. Let the resumed workflow continue
  6. Try to checkpoint again (call ctx.to_dict())
  7. Error: "Worker X not found in in_progress"

Root Cause

When resuming:

  1. BrokerState.from_serialized() moves in_progress events to queue
  2. rewind_in_progress() clears in_progress and assigns new worker IDs

When checkpointing the resumed workflow:

  1. ctx.to_dict() calls rebuild_state_from_ticks(init_state, tick_log)
  2. init_state still has original in_progress worker IDs from checkpoint
  3. tick_log has new worker IDs assigned after rewind
  4. Mismatch causes the error

Expected Behavior

ctx.to_dict() should work correctly on resumed workflows, enabling crash recovery scenarios where you need to checkpoint multiple times.

Environment

  • workflows version: 2.x
  • Python: 3.10+
Originally created by @OS-ramamurtisubramanian on GitHub (Jan 15, 2026). ## Bug Description When resuming a workflow from a checkpoint and then checkpointing again during the resumed run, `ctx.to_dict()` fails with: ``` ValueError: Worker 0 not found in in_progress ``` ## Steps to Reproduce 1. Run a workflow with parallel workers (e.g., `num_workers > 1`) 2. Checkpoint while workers are in progress 3. Crash/stop the workflow 4. Resume from checkpoint 5. Let the resumed workflow continue 6. Try to checkpoint again (call `ctx.to_dict()`) 7. **Error:** "Worker X not found in in_progress" ## Root Cause When resuming: 1. `BrokerState.from_serialized()` moves `in_progress` events to queue 2. `rewind_in_progress()` clears `in_progress` and assigns **new** worker IDs When checkpointing the resumed workflow: 1. `ctx.to_dict()` calls `rebuild_state_from_ticks(init_state, tick_log)` 2. `init_state` still has original `in_progress` worker IDs from checkpoint 3. `tick_log` has new worker IDs assigned after rewind 4. **Mismatch causes the error** ## Expected Behavior `ctx.to_dict()` should work correctly on resumed workflows, enabling crash recovery scenarios where you need to checkpoint multiple times. ## Environment - workflows version: 2.x - Python: 3.10+
yindo closed this issue 2026-02-16 02:16:11 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
enhancement
enhancement
good first issue
pull-request
Mirrored from GitHub Pull Request
 question
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: run-llama/workflows-py#28
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?