run-llama/workflows-py
1
0
Fork 0
mirror of https://github.com/run-llama/workflows-py.git synced 2026-07-20 19:57:02 -04:00
Code Issues 18 Packages Projects Releases 10 Wiki Activity

[PR #143] [MERGED] Refactor to support runtime plugins #160

New Issue
Closed
opened 2026-02-16 02:16:46 -05:00 by yindo · 0 comments
yindo commented 2026-02-16 02:16:46 -05:00
Owner
Copy Link
Copy Source

📋 Pull Request Information

Original PR: https://github.com/run-llama/workflows-py/pull/143
Author: @adrianlyjak
Created: 10/15/2025
Status: ✅ Merged
Merged: 10/21/2025
Merged by: @adrianlyjak

Base: adrian/context-refact ← Head: adrian/runtime-plugins

📝 Commits (5)

  • 4b232d9 add some refactoring notes
  • 0ed4f54 Clarify / document plugin interface better
  • ffc9e5c debug mem leak
  • 4f77e10 ugly things to workaround leaks
  • 8d83314 clean up from leak fix

📊 Changes

36 files changed (+3763 additions, -1243 deletions)

View changed files

📝 docs/src/content/docs/workflows/managing_events.md (+3 -7)
📝 examples/streaming_internal_events.ipynb (+42 -149)
📝 src/workflows/context/context.py (+79 -18)
📝 src/workflows/context/context_types.py (+144 -3)
📝 src/workflows/context/state_store.py (+0 -40)
📝 src/workflows/decorators.py (+4 -3)
📝 src/workflows/events.py (+3 -16)
📝 src/workflows/handler.py (+1 -3)
➕ src/workflows/runtime/basic_runtime.py (+94 -0)
📝 src/workflows/runtime/broker.py (+179 -561)
➕ src/workflows/runtime/control_loop.py (+686 -0)
➖ src/workflows/runtime/state.py (+0 -140)
➕ src/workflows/runtime/types/commands.py (+77 -0)
➕ src/workflows/runtime/types/internal_state.py (+307 -0)
➕ src/workflows/runtime/types/plugin.py (+137 -0)
➕ src/workflows/runtime/types/results.py (+172 -0)
➕ src/workflows/runtime/types/step_function.py (+152 -0)
➕ src/workflows/runtime/types/ticks.py (+70 -0)
➕ src/workflows/runtime/workflow_registry.py (+56 -0)
📝 src/workflows/server/server.py (+3 -2)

...and 16 more files

📄 Description

decouple workflow execution logic from runtime

This is the main change. Decouples our runtime logic (previously in the context and then temporarily moved into the broker), from the asyncio in-memory implementation. The core runtime logic is implemented as events that are serially reduced into state, and subsequent "commands" that should be executed. This approach is verbose, but it offers a lot of granular control, ability to reason about state, and easy unit testing. Adds a plugin based runtime to execute the workflow "somewhere"

  • Adjusted to_dict serialization to support newer internal state
  • Add runtime plugin abstraction with a default asyncio-based implementation.
  • Add Context.stream_events() and a deprecated queue bridge via Context.streaming_queue.

Simplify internal events;

remove EventsQueueChanged and simplify StepStateChanged

  • Remove excessive StepStateChanged status. Many statuses were firing effectively simultaneously. now just 3: preparing, in_progress, not_in_progress.
  • Update docs and examples to reflect new APIs and internal events.

"Breaking" changes

(these are largely debug and internal apis, so I'm calling it ok)

  • Remove EventsQueueChanged.
  • Replace Context.streaming_queue usage with Context.stream_events(); a deprecation bridge exists but consumers should migrate. This one is not documented so should be minimal
  • Simplify StepState (no IN_PROGRESS/EXITED) and worker_id may be None for PREPARING.
  • Remove InMemoryStateStore.to_dict_snapshot (public API behavior replaced by Context.to_dict()).

LLM Generated detailed summary of changes

  • New runtime/control loop:
    • src/workflows/runtime/control_loop.py: Core reducer and runner.
    • src/workflows/runtime/types/{commands,internal_state,results,step_function,ticks}.py: Typed building blocks for control loop.
    • src/workflows/runtime/workflow_registry.py: Single-registration for plugins per workflow instance.
    • src/workflows/runtime/basic_runtime.py: Default Plugin + WorkflowRuntime (asyncio) with replay snapshots.
  • Context and broker:
    • src/workflows/context/context.py: Plugin injection, V0→V1 validation on load, new stream_events() and deprecated streaming_queue bridging queue per-call.
    • src/workflows/runtime/broker.py: Broker delegates to control loop; state derived from ticks via BrokerState.
  • Events and internal events:
    • src/workflows/events.py: Simplify StepState to PREPARING/RUNNING/NOT_RUNNING; remove EventsQueueChanged.
    • Emit consistent StepStateChanged.input_event_name and output_event_name.
  • Serialization:
    • src/workflows/context/context_types.py: Add SerializedContextV0, SerializedContext (V1), conversion helpers; capture per-step queues, in-progress, collected events, waiters.
    • Remove InMemoryStateStore.to_dict_snapshot (unused; snapshot is now covered by Context.to_dict() + BrokerState).
  • Handler/Workflow:
    • src/workflows/handler.py: Switch to ctx.stream_events().
    • src/workflows/workflow.py: Localized import for Context to avoid import cycles.
  • Server:
    • src/workflows/server/server.py: Shut down handlers concurrently with asyncio.gather.
  • Docs and examples:
    • docs/.../managing_events.md: Remove EventsQueueChanged references; clarify StepStateChanged semantics and examples.
    • examples/streaming_internal_events.ipynb: Update code, remove queue-size logging, refresh outputs; pin kernelspec.
  • Tests:
    • Add runtime tests (tests/runtime): control loop integration and transformation unit tests.
    • Update existing tests to reflect new state/ticks-based behavior and removed EventsQueueChanged.
    • Update tests to use TickAddEvent expectations and V1 serialized context.

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/run-llama/workflows-py/pull/143 **Author:** [@adrianlyjak](https://github.com/adrianlyjak) **Created:** 10/15/2025 **Status:** ✅ Merged **Merged:** 10/21/2025 **Merged by:** [@adrianlyjak](https://github.com/adrianlyjak) **Base:** `adrian/context-refact` ← **Head:** `adrian/runtime-plugins` --- ### 📝 Commits (5) - [`4b232d9`](https://github.com/run-llama/workflows-py/commit/4b232d9ba9ab63051678afc603d8ebab5c1938bd) add some refactoring notes - [`0ed4f54`](https://github.com/run-llama/workflows-py/commit/0ed4f5479cf56babc39adeadbb89549d09a8d588) Clarify / document plugin interface better - [`ffc9e5c`](https://github.com/run-llama/workflows-py/commit/ffc9e5c1d6240828a5309738482fa482b11edbc3) debug mem leak - [`4f77e10`](https://github.com/run-llama/workflows-py/commit/4f77e107ed2ef1e0704d46ddbd0f0095097aeb49) ugly things to workaround leaks - [`8d83314`](https://github.com/run-llama/workflows-py/commit/8d8331411622de523c9b8033ea8923ea3f1a7a84) clean up from leak fix ### 📊 Changes **36 files changed** (+3763 additions, -1243 deletions) <details> <summary>View changed files</summary> 📝 `docs/src/content/docs/workflows/managing_events.md` (+3 -7) 📝 `examples/streaming_internal_events.ipynb` (+42 -149) 📝 `src/workflows/context/context.py` (+79 -18) 📝 `src/workflows/context/context_types.py` (+144 -3) 📝 `src/workflows/context/state_store.py` (+0 -40) 📝 `src/workflows/decorators.py` (+4 -3) 📝 `src/workflows/events.py` (+3 -16) 📝 `src/workflows/handler.py` (+1 -3) ➕ `src/workflows/runtime/basic_runtime.py` (+94 -0) 📝 `src/workflows/runtime/broker.py` (+179 -561) ➕ `src/workflows/runtime/control_loop.py` (+686 -0) ➖ `src/workflows/runtime/state.py` (+0 -140) ➕ `src/workflows/runtime/types/commands.py` (+77 -0) ➕ `src/workflows/runtime/types/internal_state.py` (+307 -0) ➕ `src/workflows/runtime/types/plugin.py` (+137 -0) ➕ `src/workflows/runtime/types/results.py` (+172 -0) ➕ `src/workflows/runtime/types/step_function.py` (+152 -0) ➕ `src/workflows/runtime/types/ticks.py` (+70 -0) ➕ `src/workflows/runtime/workflow_registry.py` (+56 -0) 📝 `src/workflows/server/server.py` (+3 -2) _...and 16 more files_ </details> ### 📄 Description ## decouple workflow execution logic from runtime This is the main change. Decouples our runtime logic (previously in the context and then temporarily moved into the broker), from the asyncio in-memory implementation. The core runtime logic is implemented as events that are serially reduced into state, and subsequent "commands" that should be executed. This approach is verbose, but it offers a lot of granular control, ability to reason about state, and easy unit testing. Adds a plugin based runtime to execute the workflow "somewhere" - Adjusted to_dict serialization to support newer internal state - Add runtime plugin abstraction with a default asyncio-based implementation. - Add Context.stream_events() and a deprecated queue bridge via Context.streaming_queue. ## Simplify internal events; remove `EventsQueueChanged` and simplify `StepStateChanged` - Remove excessive `StepStateChanged` status. Many statuses were firing effectively simultaneously. now just 3: preparing, in_progress, not_in_progress. - Update docs and examples to reflect new APIs and internal events. ## "Breaking" changes (these are largely debug and internal apis, so I'm calling it ok) - Remove `EventsQueueChanged`. - Replace `Context.streaming_queue` usage with `Context.stream_events()`; a deprecation bridge exists but consumers should migrate. This one is not documented so should be minimal - Simplify `StepState` (no IN_PROGRESS/EXITED) and `worker_id` may be `None` for PREPARING. - Remove `InMemoryStateStore.to_dict_snapshot` (public API behavior replaced by `Context.to_dict()`). ## LLM Generated detailed summary of changes - New runtime/control loop: - `src/workflows/runtime/control_loop.py`: Core reducer and runner. - `src/workflows/runtime/types/{commands,internal_state,results,step_function,ticks}.py`: Typed building blocks for control loop. - `src/workflows/runtime/workflow_registry.py`: Single-registration for plugins per workflow instance. - `src/workflows/runtime/basic_runtime.py`: Default `Plugin` + `WorkflowRuntime` (asyncio) with replay snapshots. - Context and broker: - `src/workflows/context/context.py`: Plugin injection, V0→V1 validation on load, new `stream_events()` and deprecated `streaming_queue` bridging queue per-call. - `src/workflows/runtime/broker.py`: Broker delegates to control loop; state derived from ticks via `BrokerState`. - Events and internal events: - `src/workflows/events.py`: Simplify `StepState` to PREPARING/RUNNING/NOT_RUNNING; remove `EventsQueueChanged`. - Emit consistent `StepStateChanged.input_event_name` and `output_event_name`. - Serialization: - `src/workflows/context/context_types.py`: Add `SerializedContextV0`, `SerializedContext` (V1), conversion helpers; capture per-step queues, in-progress, collected events, waiters. - Remove `InMemoryStateStore.to_dict_snapshot` (unused; snapshot is now covered by `Context.to_dict()` + `BrokerState`). - Handler/Workflow: - `src/workflows/handler.py`: Switch to `ctx.stream_events()`. - `src/workflows/workflow.py`: Localized import for `Context` to avoid import cycles. - Server: - `src/workflows/server/server.py`: Shut down handlers concurrently with `asyncio.gather`. - Docs and examples: - `docs/.../managing_events.md`: Remove `EventsQueueChanged` references; clarify `StepStateChanged` semantics and examples. - `examples/streaming_internal_events.ipynb`: Update code, remove queue-size logging, refresh outputs; pin kernelspec. - Tests: - Add runtime tests (`tests/runtime`): control loop integration and transformation unit tests. - Update existing tests to reflect new state/ticks-based behavior and removed `EventsQueueChanged`. - Update tests to use `TickAddEvent` expectations and V1 serialized context. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
yindo added the pull-request label 2026-02-16 02:16:46 -05:00
yindo closed this issue 2026-02-16 02:16:46 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
enhancement
enhancement
good first issue
pull-request
Mirrored from GitHub Pull Request
 question
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: run-llama/workflows-py#160
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?