[PR #306] [MERGED] docs(rfc): propose persistent flow queue and completion webhooks #311

New Issue

2026-06-06T22:10:10-04:00

yindo commented

2026-06-06 22:10:10 -04:00

📋 Pull Request Information

Original PR: https://github.com/vxcontrol/pentagi/pull/306
Author: @mason5052
Created: 5/7/2026
Status: ✅ Merged
Merged: 5/23/2026
Merged by: @asdek

Base: feature/next-release ← Head: codex/issue-298-flow-concurrency-rfc

📝 Commits (2)

295e7d3 docs(rfc): propose persistent flow queue and completion webhooks
2d407b9 docs(rfc): clarify lifecycle, terminal semantics, and webhook event names

📊 Changes

1 file changed (+187 additions, -0 deletions)

View changed files

➕ examples/proposals/flow_concurrency.md (+187 -0)

📄 Description

Summary

Adds a design RFC at examples/proposals/flow_concurrency.md describing a possible direction for native flow concurrency control and completion webhooks, as raised in issue #298. The RFC is documentation only; no runtime, schema, GraphQL, REST, or UI behavior changes here.

Problem

Issue #298 covers two real pain points for external orchestrators that drive PentAGI from a pipeline:

createFlow accepts unlimited concurrent calls, so calling it N times immediately spins up N Kali containers and saturates the host.
There is no native completion notification, so external schedulers maintain a polling loop on GET /api/v1/flows/{id}.

A naive fix can easily slip back into the anti-pattern PR #268 was rejected for: a hidden in-memory queue with no persistence, no UI/API/DB visibility, and no cancelability. PentAGI maintainers have asked, both in PR #268 and in the assistant flow management direction shipped in PR #292, for explicit, inspectable, manageable flow control instead of implicit background lifecycle.

Solution

The RFC follows the maintainer's relocated proposal pattern at examples/proposals/<topic>.md (commit 47de4e4 moved backend/docs/evidence_chain_rfc.md to examples/proposals/evidence_chain.md). It deliberately mirrors the structure of examples/proposals/evidence_chain.md (PR #277) and includes:

Goals limited to capping concurrent flows, making queued flows first-class, replacing external polling with at-least-once webhooks, and preserving the existing createFlow contract.
Non-Goals that forbid hidden in-memory queues, multi-tenant scheduling/SLA, a generic event bus, and any change to what finished means for tasks/subtasks/toolcalls.
Design Principles centered on persistence, visibility, manageability, explicit promotion, clear finished semantics, and at-least-once delivery.
Proposed concurrency model: a new persisted queued status, a single MAX_CONCURRENT_FLOWS env var, an explicit promoter, full UI/API visibility (listings, filters, detail, assistant view), and a user-driven cancel path with a documented terminal status.
Proposed completion webhook model: per-flow webhookUrl plus a global FLOW_WEBHOOK_URL fallback, HMAC-SHA256 signed payloads via X-PentAGI-Signature, stable delivery_id for receiver dedup, persisted flow_webhook_deliveries rows, bounded exponential backoff, and SSRF mitigations (config-time and delivery-time DNS validation, redacted secrets in logs, no retry on most 4xx).
Storage and API surface sketches that intentionally do not commit to a final migration shape.
Open Questions covering per-user limits, createFlow blocking semantics, alignment of the signature scheme with the issue #235 receipt direction, and queued-flow handling for resources/uploads/messages.
Suggested First Milestone that lands the queue end-to-end before webhooks, so each lifecycle change can be reviewed in isolation and PR sizes stay small.

The RFC explicitly rejects in-memory queues, hidden lifecycle state, and implicit promotion semantics, and references the PR #268 lesson directly.

User Impact

Maintainers and contributors get a written design surface to push back on before any code lands. The RFC is opinionated about what is out of scope so a future implementation PR cannot quietly grow into hidden lifecycle work.
Operators evaluating PentAGI for batch pipelines see the intended direction (a single MAX_CONCURRENT_FLOWS knob plus signed completion webhooks) and the staged milestones, so they can plan integration work even before any feature ships.
External orchestrators learn what to expect long-term: a queueable flow status, signed webhooks with a documented payload, and idempotent delivery semantics, instead of building yet another polling loop.

Test Plan

git diff --stat shows only examples/proposals/flow_concurrency.md changed.
Markdown renders cleanly (no broken anchor, list, table, or code-fence syntax).
Section structure mirrors examples/proposals/evidence_chain.md (Goals / Non-Goals / Proposed model / Open Questions / Suggested First Milestone) and adds an explicit Safety and Security section.
No code, schema, GraphQL, REST, or UI changes.

Refs #298

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/vxcontrol/pentagi/pull/306 **Author:** [@mason5052](https://github.com/mason5052) **Created:** 5/7/2026 **Status:** ✅ Merged **Merged:** 5/23/2026 **Merged by:** [@asdek](https://github.com/asdek) **Base:** `feature/next-release` ← **Head:** `codex/issue-298-flow-concurrency-rfc` --- ### 📝 Commits (2) - [`295e7d3`](https://github.com/vxcontrol/pentagi/commit/295e7d3b0100b25e4235f5fc5caa8f1d1d7e1f3a) docs(rfc): propose persistent flow queue and completion webhooks - [`2d407b9`](https://github.com/vxcontrol/pentagi/commit/2d407b928e58b4948818797ea49112e30b561b5b) docs(rfc): clarify lifecycle, terminal semantics, and webhook event names ### 📊 Changes **1 file changed** (+187 additions, -0 deletions) <details> <summary>View changed files</summary> ➕ `examples/proposals/flow_concurrency.md` (+187 -0) </details> ### 📄 Description ## Summary Adds a design RFC at ``examples/proposals/flow_concurrency.md`` describing a possible direction for native flow concurrency control and completion webhooks, as raised in issue #298. The RFC is documentation only; no runtime, schema, GraphQL, REST, or UI behavior changes here. ## Problem Issue #298 covers two real pain points for external orchestrators that drive PentAGI from a pipeline: - ``createFlow`` accepts unlimited concurrent calls, so calling it N times immediately spins up N Kali containers and saturates the host. - There is no native completion notification, so external schedulers maintain a polling loop on ``GET /api/v1/flows/{id}``. A naive fix can easily slip back into the anti-pattern PR #268 was rejected for: a hidden in-memory queue with no persistence, no UI/API/DB visibility, and no cancelability. PentAGI maintainers have asked, both in PR #268 and in the assistant flow management direction shipped in PR #292, for explicit, inspectable, manageable flow control instead of implicit background lifecycle. ## Solution The RFC follows the maintainer's relocated proposal pattern at ``examples/proposals/<topic>.md`` (commit 47de4e4 moved ``backend/docs/evidence_chain_rfc.md`` to ``examples/proposals/evidence_chain.md``). It deliberately mirrors the structure of ``examples/proposals/evidence_chain.md`` (PR #277) and includes: - **Goals** limited to capping concurrent flows, making queued flows first-class, replacing external polling with at-least-once webhooks, and preserving the existing ``createFlow`` contract. - **Non-Goals** that forbid hidden in-memory queues, multi-tenant scheduling/SLA, a generic event bus, and any change to what ``finished`` means for tasks/subtasks/toolcalls. - **Design Principles** centered on persistence, visibility, manageability, explicit promotion, clear ``finished`` semantics, and at-least-once delivery. - **Proposed concurrency model**: a new persisted ``queued`` status, a single ``MAX_CONCURRENT_FLOWS`` env var, an explicit promoter, full UI/API visibility (listings, filters, detail, assistant view), and a user-driven cancel path with a documented terminal status. - **Proposed completion webhook model**: per-flow ``webhookUrl`` plus a global ``FLOW_WEBHOOK_URL`` fallback, HMAC-SHA256 signed payloads via ``X-PentAGI-Signature``, stable ``delivery_id`` for receiver dedup, persisted ``flow_webhook_deliveries`` rows, bounded exponential backoff, and SSRF mitigations (config-time and delivery-time DNS validation, redacted secrets in logs, no retry on most 4xx). - **Storage and API surface sketches** that intentionally do not commit to a final migration shape. - **Open Questions** covering per-user limits, ``createFlow`` blocking semantics, alignment of the signature scheme with the issue #235 receipt direction, and queued-flow handling for resources/uploads/messages. - **Suggested First Milestone** that lands the queue end-to-end before webhooks, so each lifecycle change can be reviewed in isolation and PR sizes stay small. The RFC explicitly rejects in-memory queues, hidden lifecycle state, and implicit promotion semantics, and references the PR #268 lesson directly. ## User Impact - Maintainers and contributors get a written design surface to push back on before any code lands. The RFC is opinionated about what is **out of scope** so a future implementation PR cannot quietly grow into hidden lifecycle work. - Operators evaluating PentAGI for batch pipelines see the intended direction (a single ``MAX_CONCURRENT_FLOWS`` knob plus signed completion webhooks) and the staged milestones, so they can plan integration work even before any feature ships. - External orchestrators learn what to expect long-term: a queueable flow status, signed webhooks with a documented payload, and idempotent delivery semantics, instead of building yet another polling loop. ## Test Plan - [x] ``git diff --stat`` shows only ``examples/proposals/flow_concurrency.md`` changed. - [x] Markdown renders cleanly (no broken anchor, list, table, or code-fence syntax). - [x] Section structure mirrors ``examples/proposals/evidence_chain.md`` (Goals / Non-Goals / Proposed model / Open Questions / Suggested First Milestone) and adds an explicit Safety and Security section. - [x] No code, schema, GraphQL, REST, or UI changes. Refs #298 --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

yindo added the pull-request label 2026-06-06 22:10:10 -04:00

yindo closed this issue

2026-06-06 22:10:10 -04:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: vxcontrol/pentagi#311