[Chore/Refactor] Complete decoupling of Domain Models (WorkflowExecution & WorkflowNodeExecution) from the persistence layer #16172

Open
opened 2026-02-21 19:24:56 -05:00 by yindo · 7 comments
Owner

Originally created by @hieheihei on GitHub (Aug 14, 2025).

Originally assigned to: @laipz8200, @hieheihei on GitHub.

Self Checks

  • I have read the Contributing Guide and Language Policy.
  • This is only for refactoring, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report, otherwise it will be closed.
  • 【中文用户 & Non English User】请使用英语提交,否则会被关闭 :)
  • Please do not modify this template :) and fill in all the required fields.

Description

issue https://github.com/langgenius/dify/issues/19850 and https://github.com/langgenius/dify/issues/19428 have completed the extraction of the Domain Model Architecture, yet some read operations still depend directly on the persistence layer. This prevents WorkflowNodeExecutionRepository and WorkflowExecutionRepository from being implemented outside of the database.

Motivation

Currently, execution detail records are stored in the database, consuming a large amount of space and creating stability and performance risks that necessitate periodic archiving. We do not regard this as an ideal storage strategy for execution detail records and would like to introduce a log-style storage implementation. Achieving this first requires decoupling the Domain Model from the persistence layer.

Additional Context

🔴 Direct Database Access (10 locations)

WorkflowRun Direct Access (4 locations)

File Location Line Access Method Purpose
api/services/rag_pipeline/rag_pipeline.py 1019-1045 db.session.query(WorkflowRun) RAG pagination query
api/services/rag_pipeline/rag_pipeline.py 1067-1075 db.session.query(WorkflowRun) RAG single query
api/core/ops/ops_trace_manager.py 490-491 select(WorkflowRun).where(...) Trace tracking query
api/core/memory/token_buffer_memory.py 53 select(WorkflowRun).where(...) Memory management query

WorkflowNodeExecutionModel Direct Access (1 location)

File Location Line Access Method Purpose
api/core/ops/arize_phoenix_trace/arize_phoenix_trace.py 703-718 db.session.scalars(select(...).where(...)) Phoenix Trace node execution query

Raw SQL Queries (4 statistical interfaces)

File Location Interface Purpose
api/controllers/console/app/workflow_statistic.py WorkflowDailyRunsStatistic Daily runs statistics
api/controllers/console/app/workflow_statistic.py WorkflowDailyTerminalsStatistic Daily terminals statistics
api/controllers/console/app/workflow_statistic.py WorkflowDailyTokenCostStatistic Daily token cost statistics
api/controllers/console/app/workflow_statistic.py WorkflowAverageAppInteractionStatistic Average interaction statistics

Cross-table JOIN Queries (1 location)

File Location Line JOIN Method Purpose
api/services/workflow_app_service.py 47 stmt.join(WorkflowRun, ...) WorkflowAppLog query filtering

🔴 Association Query Properties (10 locations)

WorkflowRun Association Properties (4 locations)

File Location Line Property Name Association Target
api/models/workflow.py 577-582 WorkflowRun.message → Message table
api/models/workflow.py 585-586 WorkflowRun.workflow → Workflow table
api/models/workflow.py 553-555 WorkflowRun.created_by_account → Account table
api/models/workflow.py 558-561 WorkflowRun.created_by_end_user → EndUser table

WorkflowNodeExecutionModel Association Properties (2 locations)

File Location Line Property Name Association Target
api/models/workflow.py 792-795 WorkflowNodeExecutionModel.created_by_account → Account table
api/models/workflow.py 798-803 WorkflowNodeExecutionModel.created_by_end_user → EndUser table

Message Association Properties (1 location)

File Location Line Property Name Association Target
api/models/model.py 1213-1219 Message.workflow_run → WorkflowRun table

WorkflowAppLog Association Properties (3 locations)

File Location Line Property Name Association Target
api/models/workflow.py 1036-1037 WorkflowAppLog.workflow_run → WorkflowRun table
api/models/workflow.py 1040-1042 WorkflowAppLog.created_by_account → Account table
api/models/workflow.py 1045-1048 WorkflowAppLog.created_by_end_user → EndUser table

🟡 ORM Association Relationships (2 locations, functional dependency)

Big Data Offloading Related

File Location Line Relationship Name Association Target
api/models/workflow.py 765-770 WorkflowNodeExecutionModel.offload_data → WorkflowNodeExecutionOffload
api/models/workflow.py 951-956 WorkflowNodeExecutionOffload.execution → WorkflowNodeExecutionModel

🟢 Periodic Cleanup Tasks (No modification needed)

File Location Line Access Method Purpose Note
api/schedule/clean_workflow_runlogs_precise.py 45 session.query(WorkflowRun).where(...).count() Cleanup expired log count Periodic cleanup, no modification needed
api/schedule/clean_workflow_runlogs_precise.py 57-61 select(WorkflowRun.id).where(...) Cleanup expired log query IDs Periodic cleanup, no modification needed
api/schedule/clean_workflow_runlogs_precise.py 129-131 session.query(WorkflowNodeExecutionModel).where(...).delete() Cleanup expired node execution logs Periodic cleanup, no modification needed
Originally created by @hieheihei on GitHub (Aug 14, 2025). Originally assigned to: @laipz8200, @hieheihei on GitHub. ### Self Checks - [x] I have read the [Contributing Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) and [Language Policy](https://github.com/langgenius/dify/issues/1542). - [x] This is only for refactoring, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report, otherwise it will be closed. - [x] 【中文用户 & Non English User】请使用英语提交,否则会被关闭 :) - [x] Please do not modify this template :) and fill in all the required fields. ### Description issue https://github.com/langgenius/dify/issues/19850 and https://github.com/langgenius/dify/issues/19428 have completed the extraction of the Domain Model Architecture, yet some read operations still depend directly on the persistence layer. This prevents WorkflowNodeExecutionRepository and WorkflowExecutionRepository from being implemented outside of the database. ### Motivation Currently, execution detail records are stored in the database, consuming a large amount of space and creating stability and performance risks that necessitate periodic archiving. We do not regard this as an ideal storage strategy for execution detail records and would like to introduce a log-style storage implementation. Achieving this first requires decoupling the Domain Model from the persistence layer. ### Additional Context ## 🔴 Direct Database Access (10 locations) ### WorkflowRun Direct Access (4 locations) | File Location | Line | Access Method | Purpose | |---------------|------|---------------|---------| | `api/services/rag_pipeline/rag_pipeline.py` | 1019-1045 | `db.session.query(WorkflowRun)` | RAG pagination query | | `api/services/rag_pipeline/rag_pipeline.py` | 1067-1075 | `db.session.query(WorkflowRun)` | RAG single query | | `api/core/ops/ops_trace_manager.py` | 490-491 | `select(WorkflowRun).where(...)` | Trace tracking query | | `api/core/memory/token_buffer_memory.py` | 53 | `select(WorkflowRun).where(...)` | Memory management query | ### WorkflowNodeExecutionModel Direct Access (1 location) | File Location | Line | Access Method | Purpose | |---------------|------|---------------|---------| | `api/core/ops/arize_phoenix_trace/arize_phoenix_trace.py` | 703-718 | `db.session.scalars(select(...).where(...))` | Phoenix Trace node execution query | ### Raw SQL Queries (4 statistical interfaces) | File Location | Interface | Purpose | |---------------|-----------|---------| | `api/controllers/console/app/workflow_statistic.py` | `WorkflowDailyRunsStatistic` | Daily runs statistics | | `api/controllers/console/app/workflow_statistic.py` | `WorkflowDailyTerminalsStatistic` | Daily terminals statistics | | `api/controllers/console/app/workflow_statistic.py` | `WorkflowDailyTokenCostStatistic` | Daily token cost statistics | | `api/controllers/console/app/workflow_statistic.py` | `WorkflowAverageAppInteractionStatistic` | Average interaction statistics | ### Cross-table JOIN Queries (1 location) | File Location | Line | JOIN Method | Purpose | |---------------|------|-------------|---------| | `api/services/workflow_app_service.py` | 47 | `stmt.join(WorkflowRun, ...)` | WorkflowAppLog query filtering | ## 🔴 Association Query Properties (10 locations) ### WorkflowRun Association Properties (4 locations) | File Location | Line | Property Name | Association Target | |---------------|------|---------------|-------------------| | `api/models/workflow.py` | 577-582 | `WorkflowRun.message` | → Message table | | `api/models/workflow.py` | 585-586 | `WorkflowRun.workflow` | → Workflow table | | `api/models/workflow.py` | 553-555 | `WorkflowRun.created_by_account` | → Account table | | `api/models/workflow.py` | 558-561 | `WorkflowRun.created_by_end_user` | → EndUser table | ### WorkflowNodeExecutionModel Association Properties (2 locations) | File Location | Line | Property Name | Association Target | |---------------|------|---------------|-------------------| | `api/models/workflow.py` | 792-795 | `WorkflowNodeExecutionModel.created_by_account` | → Account table | | `api/models/workflow.py` | 798-803 | `WorkflowNodeExecutionModel.created_by_end_user` | → EndUser table | ### Message Association Properties (1 location) | File Location | Line | Property Name | Association Target | |---------------|------|---------------|-------------------| | `api/models/model.py` | 1213-1219 | `Message.workflow_run` | → WorkflowRun table | ### WorkflowAppLog Association Properties (3 locations) | File Location | Line | Property Name | Association Target | |---------------|------|---------------|-------------------| | `api/models/workflow.py` | 1036-1037 | `WorkflowAppLog.workflow_run` | → WorkflowRun table | | `api/models/workflow.py` | 1040-1042 | `WorkflowAppLog.created_by_account` | → Account table | | `api/models/workflow.py` | 1045-1048 | `WorkflowAppLog.created_by_end_user` | → EndUser table | ## 🟡 ORM Association Relationships (2 locations, functional dependency) ### Big Data Offloading Related | File Location | Line | Relationship Name | Association Target | |---------------|------|-------------------|-------------------| | `api/models/workflow.py` | 765-770 | `WorkflowNodeExecutionModel.offload_data` | → WorkflowNodeExecutionOffload | | `api/models/workflow.py` | 951-956 | `WorkflowNodeExecutionOffload.execution` | → WorkflowNodeExecutionModel | ## 🟢 Periodic Cleanup Tasks (No modification needed) | File Location | Line | Access Method | Purpose | Note | |---------------|------|---------------|---------|------| | `api/schedule/clean_workflow_runlogs_precise.py` | 45 | `session.query(WorkflowRun).where(...).count()` | Cleanup expired log count | ✅ Periodic cleanup, no modification needed | | `api/schedule/clean_workflow_runlogs_precise.py` | 57-61 | `select(WorkflowRun.id).where(...)` | Cleanup expired log query IDs | ✅ Periodic cleanup, no modification needed | | `api/schedule/clean_workflow_runlogs_precise.py` | 129-131 | `session.query(WorkflowNodeExecutionModel).where(...).delete()` | Cleanup expired node execution logs | ✅ Periodic cleanup, no modification needed |
yindo added the stale label 2026-02-21 19:24:56 -05:00
Author
Owner

@hieheihei commented on GitHub (Aug 14, 2025):

For example, WorkflowRun is referenced in multiple places beyond SQLAlchemyWorkflowExecutionRepository and DifyAPISQLAlchemyWorkflowRunRepository.

Image

core.app.apps.common.workflow_response_converter.WorkflowResponseConverter.workflow_finish_to_stream_response
Image

@hieheihei commented on GitHub (Aug 14, 2025): For example, WorkflowRun is referenced in multiple places beyond SQLAlchemyWorkflowExecutionRepository and DifyAPISQLAlchemyWorkflowRunRepository. <img alt="Image" src="https://github.com/user-attachments/assets/8483b1be-12b1-4693-a9f2-634cd87c463a" /> core.app.apps.common.workflow_response_converter.WorkflowResponseConverter.workflow_finish_to_stream_response <img alt="Image" src="https://github.com/user-attachments/assets/e9e477d6-8bc9-41d0-9ace-303d37a609da" />
Author
Owner

@hieheihei commented on GitHub (Aug 14, 2025):

Hi @laipz8200 , Is there already a plan, or could we contribute to it?

@hieheihei commented on GitHub (Aug 14, 2025): Hi @laipz8200 , Is there already a plan, or could we contribute to it?
Author
Owner

@dosubot[bot] commented on GitHub (Aug 14, 2025):

There is an active plan to separate Dify's core workflow executor into an independent, stateless service without database dependencies, and the team is currently working on this effort (source). Recent refactoring—like the merged PR by @laipz8200 that removed redundant repository methods and improved caching—shows ongoing work to decouple domain logic from direct database queries (source). The repository interfaces are already designed for storage abstraction, but the implementation is still evolving.

There aren't explicit contribution guidelines or next steps for this refactor in the codebase or docs, so if you're interested in contributing, it's best to review the related issues and recent PRs, and coordinate directly with the maintainers (like @laipz8200) to align on priorities and approach.

To reply, just mention @dosu.


How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other  Chat with Dosu Join Discord Share on X

@dosubot[bot] commented on GitHub (Aug 14, 2025): <!-- Answer --> There is an active plan to separate Dify's core workflow executor into an independent, stateless service without database dependencies, and the team is currently working on this effort ([source](https://github.com/langgenius/dify/issues/21073)). Recent refactoring—like the merged PR by @laipz8200 that removed redundant repository methods and improved caching—shows ongoing work to decouple domain logic from direct database queries ([source](https://github.com/langgenius/dify/pull/22597)). The repository interfaces are already designed for storage abstraction, but the implementation is still evolving. There aren't explicit contribution guidelines or next steps for this refactor in the codebase or docs, so if you're interested in contributing, it's best to review the related issues and recent PRs, and coordinate directly with the maintainers (like @laipz8200) to align on priorities and approach. <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot).* --- <sup>How did I do? [Good](https://app.dosu.dev/response-feedback/3c66ca62-0f18-4f04-9962-d132d994d296?feedback_type=great_response) | [Irrelevant](https://app.dosu.dev/response-feedback/3c66ca62-0f18-4f04-9962-d132d994d296?feedback_type=irrelevant_answer) | [Incorrect](https://app.dosu.dev/response-feedback/3c66ca62-0f18-4f04-9962-d132d994d296?feedback_type=incorrect_sources) | [Verbose](https://app.dosu.dev/response-feedback/3c66ca62-0f18-4f04-9962-d132d994d296?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/3c66ca62-0f18-4f04-9962-d132d994d296?feedback_type=hallucination) | [Report 🐛](https://app.dosu.dev/response-feedback/3c66ca62-0f18-4f04-9962-d132d994d296?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/3c66ca62-0f18-4f04-9962-d132d994d296?feedback_type=other)</sup>&nbsp;&nbsp;[![Chat with Dosu](https://dosu.dev/dosu-chat-badge.svg)](https://app.dosu.dev/6088a4f4-073a-43f3-b3d4-4f06fb0c05a0/ask?utm_source=github)&nbsp;[![Join Discord](https://img.shields.io/badge/join-5865F2?logo=discord&logoColor=white&label=)](https://go.dosu.dev/discord-bot)&nbsp;[![Share on X](https://img.shields.io/badge/X-share-black)](https://twitter.com/intent/tweet?text=%40dosu_ai%20helped%20me%20solve%20this%20issue!&url=https%3A//github.com/langgenius/dify/issues/23966)
Author
Owner

@laipz8200 commented on GitHub (Aug 15, 2025):

Hi @hieheihei,

Thanks for your proposal. Our current milestone is to move the database models WorkflowRun and WorkflowNodeExecution out of the core.workflow package. Since Dify is an LLMOps platform, we can’t remove them from the system entirely.

Right now, we’ve extended the Workflow-related repositories in api/repositories to support the features that need to query these records. However, some packages that depend on core.workflow still directly query these records, which prevents us from fully moving away from the RDBMS.

Completely decoupling them would be a complex refactor with a large impact, so it will need to be implemented in multiple stages. I’ll share some initial proposals in the near future.

@laipz8200 commented on GitHub (Aug 15, 2025): Hi @hieheihei, Thanks for your proposal. Our current milestone is to move the database models WorkflowRun and WorkflowNodeExecution out of the `core.workflow` package. Since Dify is an LLMOps platform, we can’t remove them from the system entirely. Right now, we’ve extended the Workflow-related repositories in `api/repositories` to support the features that need to query these records. However, some packages that depend on `core.workflow` still directly query these records, which prevents us from fully moving away from the RDBMS. Completely decoupling them would be a complex refactor with a large impact, so it will need to be implemented in multiple stages. I’ll share some initial proposals in the near future.
Author
Owner

@dosubot[bot] commented on GitHub (Oct 17, 2025):

Hi, @hieheihei. I'm Dosu, and I'm helping the Dify team manage their backlog and am marking this issue as stale.

Issue Summary:

  • You proposed fully decoupling Domain Models (WorkflowExecution & WorkflowNodeExecution) from the persistence layer to allow alternative storage solutions and enhance performance.
  • Maintainer laipz8200 acknowledged the complexity and impact of this refactor and mentioned ongoing efforts to move database models out of core.workflow and extend repositories.
  • Some direct database queries still exist, indicating partial decoupling.
  • I noted ongoing work to separate the workflow executor into a stateless service and recent refactors improving decoupling.
  • Contributors are encouraged to coordinate with maintainers to align on this significant architectural change.

Next Steps:

  • Please let us know if this issue is still relevant to the latest version of Dify by commenting here to keep the discussion open.
  • Otherwise, this issue will be automatically closed in 15 days.

Thank you for your understanding and contribution!

@dosubot[bot] commented on GitHub (Oct 17, 2025): Hi, @hieheihei. I'm [Dosu](https://dosu.dev), and I'm helping the Dify team manage their backlog and am marking this issue as stale. **Issue Summary:** - You proposed fully decoupling Domain Models (WorkflowExecution & WorkflowNodeExecution) from the persistence layer to allow alternative storage solutions and enhance performance. - Maintainer laipz8200 acknowledged the complexity and impact of this refactor and mentioned ongoing efforts to move database models out of core.workflow and extend repositories. - Some direct database queries still exist, indicating partial decoupling. - I noted ongoing work to separate the workflow executor into a stateless service and recent refactors improving decoupling. - Contributors are encouraged to coordinate with maintainers to align on this significant architectural change. **Next Steps:** - Please let us know if this issue is still relevant to the latest version of Dify by commenting here to keep the discussion open. - Otherwise, this issue will be automatically closed in 15 days. Thank you for your understanding and contribution!
Author
Owner

@hieheihei commented on GitHub (Oct 19, 2025):

Hi, @hieheihei. I'm Dosu, and I'm helping the Dify team manage their backlog and am marking this issue as stale.

Issue Summary:

  • You proposed fully decoupling Domain Models (WorkflowExecution & WorkflowNodeExecution) from the persistence layer to allow alternative storage solutions and enhance performance.
  • Maintainer laipz8200 acknowledged the complexity and impact of this refactor and mentioned ongoing efforts to move database models out of core.workflow and extend repositories.
  • Some direct database queries still exist, indicating partial decoupling.
  • I noted ongoing work to separate the workflow executor into a stateless service and recent refactors improving decoupling.
  • Contributors are encouraged to coordinate with maintainers to align on this significant architectural change.

Next Steps:

  • Please let us know if this issue is still relevant to the latest version of Dify by commenting here to keep the discussion open.
  • Otherwise, this issue will be automatically closed in 15 days.

Thank you for your understanding and contribution!

I’m still working on this.

@hieheihei commented on GitHub (Oct 19, 2025): > Hi, [@hieheihei](https://github.com/hieheihei). I'm [Dosu](https://dosu.dev), and I'm helping the Dify team manage their backlog and am marking this issue as stale. > > **Issue Summary:** > > * You proposed fully decoupling Domain Models (WorkflowExecution & WorkflowNodeExecution) from the persistence layer to allow alternative storage solutions and enhance performance. > * Maintainer laipz8200 acknowledged the complexity and impact of this refactor and mentioned ongoing efforts to move database models out of core.workflow and extend repositories. > * Some direct database queries still exist, indicating partial decoupling. > * I noted ongoing work to separate the workflow executor into a stateless service and recent refactors improving decoupling. > * Contributors are encouraged to coordinate with maintainers to align on this significant architectural change. > > **Next Steps:** > > * Please let us know if this issue is still relevant to the latest version of Dify by commenting here to keep the discussion open. > * Otherwise, this issue will be automatically closed in 15 days. > > Thank you for your understanding and contribution! I’m still working on this.
Author
Owner

@dosubot[bot] commented on GitHub (Dec 31, 2025):

Hi, @hieheihei. I'm Dosu, and I'm helping the Dify team manage their backlog and am marking this issue as stale.

Issue Summary:

  • You proposed fully decoupling Domain Models WorkflowExecution and WorkflowNodeExecution from direct database access to allow alternative storage and improve performance.
  • The maintainer acknowledged the complexity and mentioned ongoing staged efforts to move database models out of core.workflow and extend repositories.
  • Recent progress includes separating the workflow executor into a stateless service and refactoring to reduce direct database queries.
  • You confirmed you are still actively working on this issue.

Next Steps:

  • Please let me know if this issue is still relevant to the latest version of Dify by commenting here to keep the discussion open.
  • If I do not hear back within 15 days, this issue will be automatically closed.

Thank you for your understanding and contribution!

@dosubot[bot] commented on GitHub (Dec 31, 2025): Hi, @hieheihei. I'm [Dosu](https://dosu.dev), and I'm helping the Dify team manage their backlog and am marking this issue as stale. **Issue Summary:** - You proposed fully decoupling Domain Models WorkflowExecution and WorkflowNodeExecution from direct database access to allow alternative storage and improve performance. - The maintainer acknowledged the complexity and mentioned ongoing staged efforts to move database models out of core.workflow and extend repositories. - Recent progress includes separating the workflow executor into a stateless service and refactoring to reduce direct database queries. - You confirmed you are still actively working on this issue. **Next Steps:** - Please let me know if this issue is still relevant to the latest version of Dify by commenting here to keep the discussion open. - If I do not hear back within 15 days, this issue will be automatically closed. Thank you for your understanding and contribution!
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#16172