[GH-ISSUE #298] Native flow queue/batch execution with configurable concurrency limit #95

New Issue

2026-06-06T22:09:06-04:00

yindo commented

2026-06-06 22:09:06 -04:00

Originally created by @JAPACX on GitHub (May 4, 2026).
Original GitHub issue: https://github.com/vxcontrol/pentagi/issues/298

Originally assigned to: @asdek on GitHub.

Target Component

Core Services (Frontend UI/Backend API)

Enhancement Description

Problem
When integrating PentAGI into automated pipelines (e.g. weekly security reviews across multiple repositories), there is no way to control how many flows run concurrently. Calling createFlow N times in sequence starts all N flows immediately, spinning up N Kali Linux containers simultaneously and saturating host resources.
Currently the only workaround is implementing an external polling loop:

Call createFlow for a batch of N
Poll GET /api/v1/flows/{id} until status == completed | failed
Launch next batch manually
This adds significant complexity to any external orchestration layer.

Proposed Solution
Two complementary additions:

MAX_CONCURRENT_FLOWS environment variable
A simple cap on how many flows can be in active status simultaneously. Flows created beyond the cap would enter a queued status at the Flow level and be promoted automatically as slots free up.
Webhook / callback on flow completion
A configurable WEBHOOK_URL that PentAGI calls with a POST when a flow transitions to completed or failed. This eliminates the need for external polling loops entirely.

Current Workaround
External batch orchestration via polling — functional but requires building and maintaining a custom polling loop outside PentAGI.

Use Case
Automated weekly security pipeline across 30+ repositories:

Each repo triggers a PentAGI flow with its findings
Without concurrency control, 30 Kali containers start simultaneously
With MAX_CONCURRENT_FLOWS=5, PentAGI would self-manage the queue and process them in controlled batches

Technical Details

No response

Designs and Mockups

No response

Alternative Solutions

No response

Verification

I have checked that this enhancement hasn't been already proposed
This enhancement aligns with PentAGI's goal of autonomous penetration testing
I have considered the security implications of this enhancement
I have provided clear use cases and benefits

Originally created by @JAPACX on GitHub (May 4, 2026). Original GitHub issue: https://github.com/vxcontrol/pentagi/issues/298 Originally assigned to: @asdek on GitHub. ### Target Component Core Services (Frontend UI/Backend API) ### Enhancement Description Problem When integrating PentAGI into automated pipelines (e.g. weekly security reviews across multiple repositories), there is no way to control how many flows run concurrently. Calling createFlow N times in sequence starts all N flows immediately, spinning up N Kali Linux containers simultaneously and saturating host resources. Currently the only workaround is implementing an external polling loop: 1. Call createFlow for a batch of N 2. Poll GET /api/v1/flows/{id} until status == completed | failed 3. Launch next batch manually This adds significant complexity to any external orchestration layer. Proposed Solution Two complementary additions: 1. MAX_CONCURRENT_FLOWS environment variable A simple cap on how many flows can be in active status simultaneously. Flows created beyond the cap would enter a queued status at the Flow level and be promoted automatically as slots free up. 2. Webhook / callback on flow completion A configurable WEBHOOK_URL that PentAGI calls with a POST when a flow transitions to completed or failed. This eliminates the need for external polling loops entirely. Current Workaround External batch orchestration via polling — functional but requires building and maintaining a custom polling loop outside PentAGI. Use Case Automated weekly security pipeline across 30+ repositories: - Each repo triggers a PentAGI flow with its findings - Without concurrency control, 30 Kali containers start simultaneously - With MAX_CONCURRENT_FLOWS=5, PentAGI would self-manage the queue and process them in controlled batches ### Technical Details _No response_ ### Designs and Mockups _No response_ ### Alternative Solutions _No response_ ### Verification - [x] I have checked that this enhancement hasn't been already proposed - [x] This enhancement aligns with PentAGI's goal of autonomous penetration testing - [x] I have considered the security implications of this enhancement - [x] I have provided clear use cases and benefits

yindo added the enhancement label 2026-06-06 22:09:06 -04:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: vxcontrol/pentagi#95