run-llama/workflows-py
1
0
Fork 0
mirror of https://github.com/run-llama/workflows-py.git synced 2026-07-21 04:05:25 -04:00
Code Issues 18 Packages Projects Releases 10 Wiki Activity

[PR #127] [MERGED] Runtime Plugins #145

New Issue
Closed
opened 2026-02-16 02:16:43 -05:00 by yindo · 0 comments
yindo commented 2026-02-16 02:16:43 -05:00
Owner
Copy Link
Copy Source

📋 Pull Request Information

Original PR: https://github.com/run-llama/workflows-py/pull/127
Author: @adrianlyjak
Created: 10/7/2025
Status: ✅ Merged
Merged: 10/28/2025
Merged by: @adrianlyjak

Base: main ← Head: adrian/context-refact

📝 Commits (10+)

📊 Changes

55 files changed (+7945 additions, -1404 deletions)

View changed files

📝 .github/workflows/lint.yml (+1 -1)
📝 .github/workflows/publish_release.yml (+56 -11)
📝 .github/workflows/test.yml (+20 -6)
📝 docs/api_docs/docs/api_reference/context.md (+0 -1)
📝 docs/src/content/docs/llamaagents/workflows/drawing.md (+3 -2)
📝 docs/src/content/docs/llamaagents/workflows/managing_events.md (+3 -7)
📝 examples/streaming_internal_events.ipynb (+42 -149)
➕ packages/llama-index-utils-workflow/.gitignore (+1 -0)
➕ packages/llama-index-utils-workflow/README.md (+30 -0)
➕ packages/llama-index-utils-workflow/pyproject.toml (+25 -0)
➕ packages/llama-index-utils-workflow/src/llama_index/utils/workflow/__init__.py (+816 -0)
➕ packages/llama-index-utils-workflow/tests/conftest.py (+42 -0)
➕ packages/llama-index-utils-workflow/tests/test_drawing.py (+278 -0)
📝 pyproject.toml (+10 -2)
📝 src/workflows/context/context.py (+248 -565)
➕ src/workflows/context/context_types.py (+192 -0)
📝 src/workflows/context/state_store.py (+5 -42)
📝 src/workflows/decorators.py (+78 -24)
📝 src/workflows/events.py (+3 -19)
📝 src/workflows/handler.py (+13 -7)

...and 35 more files

📄 Description

This is a feature branch for refactoring the internals of workflows to better support a pluggable runtime, and to be easier to extend with additional features. It mainly split apart a lot of responsibilities that were held by the context, and create a deeper internal structure with varied responsibilities. It maintains parity with the existing public interface with minimal changes

Related PRs:

Original PR Notes:

This is a big one 😳 sorry! It's 100% hand coded though

First step of a refactor to facilitate better plugins for managing future runtime pluggability - long term goals are better support for distributed and/or persistent workflows by extending to external coordinators

This refactor focuses on giving more discrete responsibilities to a few new components, and narrow the responsibility of existing ones (namely Context):

  • Adds a WorkflowBroker class that by and large lifts methods from Context that are related to task, queue, and lock management. The long term goal is to define and break this up even further. There was a small amount of runtime/starting logic in the Workflow that was also moved here
  • Adds a SerializedContext typed intermediary pydantic model to validate and document the current serialized state, (rather than passing around a plain dict). The dict interface remains unchanged, to maintain compatibility
  • Adds a related WorkflowBrokerState, which contains the mutable/asyncio python state that parallels most of a SerializedContext

Context now contains a reference to the broker. Note, it also still contains the reference to the store (this was not moved to the Broker). My perspective is that the state and the runtime durability will have separate needs, and shouldn't be closely coupled. For this reason, I also removed the state snapshot on NOT_IN_PROGRESS internal events--I think it may be better to rethink that while it is unused so we can focus on figuring out making the state store and runtime more configurable/extendable.

The intialization of the workflow run was a little distributed before, so this part of the code is "new" (as opposed to copy pasted).

Sidequest 1: Adds better types to the @step decorator return value, such that the _step_config attribute is typed into the returned step function (this removes various get_attr calls in the code). Renames it to _step_config so python doesn't do name mangling

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/run-llama/workflows-py/pull/127 **Author:** [@adrianlyjak](https://github.com/adrianlyjak) **Created:** 10/7/2025 **Status:** ✅ Merged **Merged:** 10/28/2025 **Merged by:** [@adrianlyjak](https://github.com/adrianlyjak) **Base:** `main` ← **Head:** `adrian/context-refact` --- ### 📝 Commits (10+) - [`122e754`](https://github.com/run-llama/workflows-py/commit/122e754ae909b6aba2495e9008d3f0f8be6148bb) refactor: split context into separate components, add stronger typing - [`cbbf71d`](https://github.com/run-llama/workflows-py/commit/cbbf71d28f51bdd039a6ff7e9f7227418985f5db) rm stray print - [`886903f`](https://github.com/run-llama/workflows-py/commit/886903fddbbdd497cf34bc6267813e8e6b4f7997) Minor clean up from code review - [`8294f0e`](https://github.com/run-llama/workflows-py/commit/8294f0e7915b6bd076d6e52df90e031ded53628b) Refactor to support runtime plugins (#143) - [`b7744a5`](https://github.com/run-llama/workflows-py/commit/b7744a5122426958361488dbbea991f07d9f1d88) ugly leak fixes (#156) - [`d84314c`](https://github.com/run-llama/workflows-py/commit/d84314c42d98c77c6621cadb8088df2bdbf57abd) Add memory leak test - [`22a4c78`](https://github.com/run-llama/workflows-py/commit/22a4c788419fa2fdcf9c4469e88902c2fac36e10) Add runs to workflow_registry to help keep run context at arm's distance (#158) - [`47e3978`](https://github.com/run-llama/workflows-py/commit/47e39786673119b45e06930514b06896957dbbd2) Adrian/viz (#163) - [`862274b`](https://github.com/run-llama/workflows-py/commit/862274b7fa6edb1cc4f963709c8b17a736c452da) version bump - [`28a9397`](https://github.com/run-llama/workflows-py/commit/28a9397145bf16dc7d4566f062dc7c6f00b509ac) Clean up names ### 📊 Changes **55 files changed** (+7945 additions, -1404 deletions) <details> <summary>View changed files</summary> 📝 `.github/workflows/lint.yml` (+1 -1) 📝 `.github/workflows/publish_release.yml` (+56 -11) 📝 `.github/workflows/test.yml` (+20 -6) 📝 `docs/api_docs/docs/api_reference/context.md` (+0 -1) 📝 `docs/src/content/docs/llamaagents/workflows/drawing.md` (+3 -2) 📝 `docs/src/content/docs/llamaagents/workflows/managing_events.md` (+3 -7) 📝 `examples/streaming_internal_events.ipynb` (+42 -149) ➕ `packages/llama-index-utils-workflow/.gitignore` (+1 -0) ➕ `packages/llama-index-utils-workflow/README.md` (+30 -0) ➕ `packages/llama-index-utils-workflow/pyproject.toml` (+25 -0) ➕ `packages/llama-index-utils-workflow/src/llama_index/utils/workflow/__init__.py` (+816 -0) ➕ `packages/llama-index-utils-workflow/tests/conftest.py` (+42 -0) ➕ `packages/llama-index-utils-workflow/tests/test_drawing.py` (+278 -0) 📝 `pyproject.toml` (+10 -2) 📝 `src/workflows/context/context.py` (+248 -565) ➕ `src/workflows/context/context_types.py` (+192 -0) 📝 `src/workflows/context/state_store.py` (+5 -42) 📝 `src/workflows/decorators.py` (+78 -24) 📝 `src/workflows/events.py` (+3 -19) 📝 `src/workflows/handler.py` (+13 -7) _...and 35 more files_ </details> ### 📄 Description This is a feature branch for refactoring the internals of workflows to better support a pluggable runtime, and to be easier to extend with additional features. It mainly split apart a lot of responsibilities that were held by the context, and create a deeper internal structure with varied responsibilities. It maintains parity with the existing public interface with minimal changes Related PRs: - https://github.com/run-llama/workflows-py/pull/143 --- Original PR Notes: This is a big one 😳 sorry! It's 100% hand coded though First step of a refactor to facilitate better plugins for managing future runtime pluggability - long term goals are better support for distributed and/or persistent workflows by extending to external coordinators This refactor focuses on giving more discrete responsibilities to a few new components, and narrow the responsibility of existing ones (namely `Context`): - Adds a `WorkflowBroker` class that by and large lifts methods from `Context` that are related to task, queue, and lock management. The long term goal is to define and break this up even further. There was a small amount of runtime/starting logic in the Workflow that was also moved here - Adds a `SerializedContext` typed intermediary pydantic model to validate and document the current serialized state, (rather than passing around a plain dict). The dict interface remains unchanged, to maintain compatibility - Adds a related `WorkflowBrokerState`, which contains the mutable/asyncio python state that parallels most of a `SerializedContext` Context now contains a reference to the broker. Note, it also still contains the reference to the store (this was not moved to the Broker). My perspective is that the state and the runtime durability will have separate needs, and shouldn't be closely coupled. For this reason, I also removed the state snapshot on NOT_IN_PROGRESS internal events--I think it may be better to rethink that while it is unused so we can focus on figuring out making the state store and runtime more configurable/extendable. The intialization of the workflow run was a little distributed before, so this part of the code is "new" (as opposed to copy pasted). Sidequest 1: Adds better types to the `@step` decorator return value, such that the `_step_config` attribute is typed into the returned step function (this removes various `get_attr` calls in the code). Renames it to _step_config so python doesn't do name mangling --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
yindo added the pull-request label 2026-02-16 02:16:43 -05:00
yindo closed this issue 2026-02-16 02:16:43 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
enhancement
enhancement
good first issue
pull-request
Mirrored from GitHub Pull Request
 question
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: run-llama/workflows-py#145
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?