langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 12:05:30 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

[Proposal] Research-Oriented Runner Framework for Agentic Workflow Generation and Evaluation #20242

New Issue
Closed
opened 2026-02-21 20:06:26 -05:00 by yindo · 3 comments
yindo commented 2026-02-21 20:06:26 -05:00
Owner
Copy Link
Copy Source

Originally created by @fhr2022 on GitHub (Nov 10, 2025).

Self Checks

  • I have read the Contributing Guide and Language Policy.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report, otherwise it will be closed.
  • Please do not modify this template :) and fill in all the required fields.

1. Is this request related to a challenge you're experiencing? Tell me about your story.

I am currently conducting research on methods for the automated generation of Agentic Workflows. I aim to integrate these methods with the Dify platform, creating an experimental framework that can be directly used on Dify, enabling rapid testing and comparison of different automated Agentic Workflow generation approaches.

2. Additional context or comments

Motivation and Objectives

Many existing Agentic Workflow generation methods are still defined at the code level (e.g., Python scripts describing workflow logic) rather than in a structured DSL form.
This leads to several challenges for researchers:

  • The generated workflows are not visualized clearly and vary greatly in code style;
  • There is no unified evaluation pipeline, making cross-method comparison difficult;
  • There lacks an end-to-end tool covering the full research loop:
    task input → workflow generation → deployment → execution → evaluation.

To address these issues, I’m developing a lightweight automatic generation and evaluation Runner framework, designed to:

  • Provide a unified interface: automatically convert task inputs into Dify-compatible Workflow DSLs;
  • Enable automated execution: wrap Dify APIs for workflow deployment and execution;
  • Support evaluation: run multiple datasets and collect metrics automatically;
  • Offer extensibility: allow researchers to plug in their own workflow generation strategies or evaluation logic.

The framework aims to serve as a research base for quickly testing and benchmarking different Agentic Workflow generation approaches directly within the Dify ecosystem.

Current Prototype Progress

The prototype currently supports:

  • Task input → Workflow DSL generation (including start, end, llm, and if-else nodes);
  • Automated deployment and execution via the Dify API;
  • Basic result recording and metric evaluation.

It’s still in the early prototype stage, and future work will focus on expanding node types, improving error handling, and enhancing visualization.

Discussion Points with the Dify Team

  1. Does Dify currently have, or plan to develop, similar automation or benchmarking workflows?
  2. If not, would the team consider including such a Runner framework as a community example or extension?
  3. If contribution is possible, could you please share the recommended directory structure, documentation format, or testing conventions?

Next Steps

If the feedback from the Dify team is positive, I plan to further improve the framework by:

  • Supporting more DSL node types (loop, parallel, tool call, etc.);
  • Adding a UI/Dashboard for visualizing results and performance metrics;
  • Providing documentation and example datasets for researchers to get started quickly.

Thank you for Dify’s continued contributions to the Agentic AI and LLM Workflow ecosystem.
I’d love to hear your thoughts on this idea, and I’m excited about the possibility of contributing this direction to the Dify community.

3. Can you help us with this feature?

  • I am interested in contributing to this feature.
Originally created by @fhr2022 on GitHub (Nov 10, 2025). ### Self Checks - [x] I have read the [Contributing Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) and [Language Policy](https://github.com/langgenius/dify/issues/1542). - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report, otherwise it will be closed. - [x] Please do not modify this template :) and fill in all the required fields. ### 1. Is this request related to a challenge you're experiencing? Tell me about your story. I am currently conducting research on methods for the automated generation of **Agentic Workflows**. I aim to integrate these methods with the **Dify** platform, creating an experimental framework that can be directly used on Dify, enabling rapid testing and comparison of different automated Agentic Workflow generation approaches. ### 2. Additional context or comments **Motivation and Objectives** Many existing Agentic Workflow generation methods are still defined at the code level (e.g., Python scripts describing workflow logic) rather than in a **structured DSL form**. This leads to several challenges for researchers: - The generated workflows are not visualized clearly and vary greatly in code style; - There is no unified evaluation pipeline, making cross-method comparison difficult; - There lacks an end-to-end tool covering the full research loop: **task input → workflow generation → deployment → execution → evaluation**. To address these issues, I’m developing a lightweight **automatic generation and evaluation Runner framework**, designed to: - **Provide a unified interface**: automatically convert task inputs into Dify-compatible Workflow DSLs; - **Enable automated execution**: wrap Dify APIs for workflow deployment and execution; - **Support evaluation**: run multiple datasets and collect metrics automatically; - **Offer extensibility**: allow researchers to plug in their own workflow generation strategies or evaluation logic. The framework aims to serve as a **research base** for quickly testing and benchmarking different Agentic Workflow generation approaches directly within the Dify ecosystem. **Current Prototype Progress** The prototype currently supports: - Task input → Workflow DSL generation (including `start`, `end`, `llm`, and `if-else` nodes); - Automated deployment and execution via the Dify API; - Basic result recording and metric evaluation. It’s still in the **early prototype stage**, and future work will focus on expanding node types, improving error handling, and enhancing visualization. **Discussion Points with the Dify Team** 1. Does Dify currently have, or plan to develop, similar automation or benchmarking workflows? 2. If not, would the team consider including such a Runner framework as a community example or extension? 3. If contribution is possible, could you please share the recommended directory structure, documentation format, or testing conventions? **Next Steps** If the feedback from the Dify team is positive, I plan to further improve the framework by: - Supporting more DSL node types (loop, parallel, tool call, etc.); - Adding a UI/Dashboard for visualizing results and performance metrics; - Providing documentation and example datasets for researchers to get started quickly. Thank you for Dify’s continued contributions to the **Agentic AI** and **LLM Workflow** ecosystem. I’d love to hear your thoughts on this idea, and I’m excited about the possibility of contributing this direction to the Dify community. ### 3. Can you help us with this feature? - [x] I am interested in contributing to this feature.
yindo added the 💪 enhancement🌊 feat:workflow labels 2026-02-21 20:06:26 -05:00
yindo closed this issue 2026-02-21 20:06:26 -05:00
yindo commented 2026-02-21 20:06:26 -05:00
Author
Owner
Copy Link
Copy Source

@13326a commented on GitHub (Nov 10, 2025):

Very much looking forward to it

@13326a commented on GitHub (Nov 10, 2025): Very much looking forward to it
yindo commented 2026-02-21 20:06:26 -05:00
Author
Owner
Copy Link
Copy Source

@crazywoola commented on GitHub (Nov 10, 2025):

@fhr2022 We do have similar plans currently, though they are still very early ideas. We are not particularly satisfied with some of the existing automatic workflow generation methods on the market. We don’t always chase after new things; providing the optimal solution is our goal. However, we warmly welcome you to connect with us—via Discord or email at banana@dify.ai.

@crazywoola commented on GitHub (Nov 10, 2025): @fhr2022 We do have similar plans currently, though they are still very early ideas. We are not particularly satisfied with some of the existing automatic workflow generation methods on the market. We don’t always chase after new things; providing the optimal solution is our goal. However, we warmly welcome you to connect with us—via Discord or email at banana@dify.ai.
yindo commented 2026-02-21 20:06:27 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Dec 17, 2025):

Hi, @fhr2022. I'm Dosu, and I'm helping the Dify team manage their backlog and am marking this issue as stale.

Issue Summary:

  • You proposed a research-focused Runner framework integrated with Dify to automate Agentic Workflows using a structured DSL.
  • A prototype with core functionalities has already been developed by you.
  • The maintainer, crazywoola, acknowledged similar plans and expressed interest in collaborating.
  • Another user, 13326a, also showed enthusiasm for this proposal.
  • The issue remains unresolved with no recent updates.

What's Next:

  • Please let me know if this issue is still relevant to the latest version of Dify by commenting here to keep the discussion open.
  • Otherwise, I will automatically close this issue in 15 days.

Thank you for your understanding and contribution!

@dosubot[bot] commented on GitHub (Dec 17, 2025): Hi, @fhr2022. I'm [Dosu](https://dosu.dev), and I'm helping the Dify team manage their backlog and am marking this issue as stale. **Issue Summary:** - You proposed a research-focused Runner framework integrated with Dify to automate Agentic Workflows using a structured DSL. - A prototype with core functionalities has already been developed by you. - The maintainer, crazywoola, acknowledged similar plans and expressed interest in collaborating. - Another user, 13326a, also showed enthusiasm for this proposal. - The issue remains unresolved with no recent updates. **What's Next:** - Please let me know if this issue is still relevant to the latest version of Dify by commenting here to keep the discussion open. - Otherwise, I will automatically close this issue in 15 days. Thank you for your understanding and contribution!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🌊 feat:workflow 💪 enhancement
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#20242
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?