[PR #206] [MERGED] Agent Supervision System and Open Source Model Enhancements #235

New Issue

2026-06-06T22:09:49-04:00

yindo commented

2026-06-06 22:09:49 -04:00

📋 Pull Request Information

Original PR: https://github.com/vxcontrol/pentagi/pull/206
Author: @asdek
Created: 3/15/2026
Status: ✅ Merged
Merged: 3/15/2026
Merged by: @asdek

Base: master ← Head: feature/next_release

📝 Commits (10+)

4cddef4 chore(deps): bump the npm_and_yarn group across 1 directory with 2 updates
1f896a9 fix: add iteration cap and repeating escalation to performAgentChain
536ea86 fix: address review feedback on iteration cap and repeating escalation
8701f15 fix: isolate detached command context from parent cancellation
55aef6a test: add context cancellation tests for detached command isolation
1e2cf11 Rename mock type to avoid cross-PR symbol collision
fc90e9d test: add unit tests for repeatingDetector and clearCallArguments
0ffe6a4 fix: add named test constant and same-name-different-args edge case
afef83f test: add unit tests for DeepSeek, GLM, Kimi, and Qwen providers
7c3d6d6 Merge pull request #174 from vxcontrol/dependabot/npm_and_yarn/frontend/npm_and_yarn-8a6d6a6aaf

📊 Changes

75 files changed (+6871 additions, -1588 deletions)

View changed files

📝 .env.example (+12 -0)
📝 .vscode/launch.json (+1 -1)
📝 Dockerfile (+1 -0)
📝 README.md (+101 -22)
📝 backend/cmd/ftester/mocks/tools.go (+32 -12)
📝 backend/cmd/ftester/worker/executor.go (+22 -2)
📝 backend/cmd/ftester/worker/tester.go (+4 -1)
📝 backend/cmd/installer/wizard/controller/controller.go (+47 -3)
📝 backend/cmd/installer/wizard/locale/locale.go (+63 -12)
📝 backend/cmd/installer/wizard/models/ai_agents_settings_form.go (+268 -57)
📝 backend/docs/config.md (+164 -1)
📝 backend/docs/flow_execution.md (+241 -10)
📝 backend/go.mod (+5 -2)
📝 backend/go.sum (+10 -4)
➕ backend/migrations/sql/20260310_153000_agent_supervision.sql (+114 -0)
📝 backend/pkg/config/config.go (+72 -0)
➕ backend/pkg/config/config_test.go (+259 -0)
📝 backend/pkg/database/converter/converter.go (+3 -0)
📝 backend/pkg/database/models.go (+39 -36)
📝 backend/pkg/graph/generated.go (+217 -16)

...and 55 more files

📄 Description

Description of the Change

Problem

Smaller open source models (< 32B parameters) required additional supervision to match cloud API quality. Agents could get stuck in loops or miss optimal attack paths. Vector store searches returned insufficient results due to single-query limitation. Target information leakage in vector memory could cause agents to switch targets mid-attack. No production-grade guide existed for air-gapped deployments with local LLM inference.

Solution

Implemented multi-layered agent supervision system with:

Execution Monitoring (beta): Automatic mentor (adviser) intervention analyzes execution patterns, detects loops, suggests alternative strategies. Testing with Qwen3.5-27B-FP8 shows 2x improvement in result quality vs. baseline
Task Planning (beta): Planner generates 3-7 step execution plans before specialist agents begin work, preventing scope creep
Tool Call Limits: Hard caps prevent infinite loops (general agents: 100, limited agents: 20) with reflector-guided graceful termination
Multi-Query Vector Search: Agents can submit up to 5 search queries simultaneously for better knowledge retrieval
Target Anonymization: Search/store operations anonymize IP addresses and domains to prevent target switching
vLLM Deployment Guide: Comprehensive documentation for running Qwen3.5-27B-FP8 locally (13K TPS prompt processing on 4× RTX 5090)
Installer Configuration: AI Agents Settings form expanded with supervision controls
Prompt Improvements: Enhanced stability and OOB attack coverage across all agent prompts
langchaingo Update: v0.1.14-update.5 with Bedrock multi-tool fixes and OpenAI streaming robustness
Ollama Cloud Support: Added API key support and enhanced configuration

Closes #175, #176

Type of Change

🐛 Bug fix (non-breaking change which fixes an issue)
🚀 New feature (non-breaking change which adds functionality)
💥 Breaking change (fix or feature that would cause existing functionality to not work as expected)
📚 Documentation update
🔧 Configuration change
🧪 Test update
🛡️ Security update

Areas Affected

Core Services (Frontend UI/Backend API)
AI Agents (Researcher/Developer/Executor)
Security Tools Integration
Memory System (Vector Store/Knowledge Base)
Monitoring Stack (Grafana/OpenTelemetry)
Analytics Platform (Langfuse)
External Integrations (LLM/Search APIs)
Documentation
Infrastructure/DevOps

Testing and Verification

Test Configuration

PentAGI Version: Latest development (feature/next_release)
Docker Version: 24.0.x+
Host OS: Linux (Ubuntu 22.04)
LLM Provider: vLLM (Qwen3.5-27B-FP8), OpenAI, Anthropic
Enabled Features: Core + supervision features (beta)
Hardware: 4× RTX 5090 (128GB VRAM total)

Test Steps

Run all backend unit tests (go test ./...)
Test execution monitoring with Qwen3.5-27B-FP8 (complex penetration testing flows)
Test task planning with enhanced adviser configuration
Verify multi-query vector search (up to 5 simultaneous queries)
Test target anonymization in vector store operations
Test infinite loop prevention with tool call limits
Test detached command context isolation
Verify Chinese provider configurations (DeepSeek, GLM, Kimi, Qwen)
Test AI Agents Settings installer form with all 8 fields
Verify vLLM deployment guide with real hardware

Test Results

✅ All existing and new unit tests pass
✅ Quality improvement: 2x better results with supervision enabled on Qwen3.5-27B-FP8
✅ Performance verified: 13K TPS prompt processing, 650 TPS completion on 4× RTX 5090
✅ Infinite loop prevention working (7 consecutive calls → abort)
✅ Multi-query vector search returns diverse results
✅ Target anonymization prevents context leakage
✅ Detached commands survive parent context cancellation
✅ Chinese provider tests added and passing
✅ Installer form validates all supervision parameters correctly
✅ No breaking changes to existing functionality

Security Considerations

Enhanced Security:

Target Anonymization: IP addresses and domains anonymized in vector store to prevent accidental target switching during multi-phase attacks
Context Isolation: Detached terminal commands now properly isolated from parent context cancellation (prevents premature termination)
Loop Prevention: Hard tool call limits prevent resource exhaustion from runaway agents

No New Attack Vectors:

All supervision features are read-only analysis mechanisms
No new credentials or external dependencies required
Maintains existing security model
Backward compatible with all deployments

Performance Impact

With Supervision Features Enabled (for models < 32B):

Token consumption: 2-3x increase due to mentor/planner invocations
Execution time: 2-3x longer due to analysis and planning steps
Result quality: 2x improvement in completeness, accuracy, and attack coverage
Trade-off: User choice between speed vs. quality

Multi-Query Vector Search:

Up to 5 parallel queries vs. previous single query
Better knowledge retrieval diversity
Minimal latency increase (queries run in parallel)

Infinite Loop Prevention:

Eliminates unbounded executions
Reduces resource waste from repeated failed attempts

vLLM Performance (Qwen3.5-27B-FP8 on 4× RTX 5090):

Prompt processing: ~13,000 tokens/sec
Completion generation: ~650 tokens/sec
Concurrent flows: 12+ simultaneously
VRAM usage: ~30GB per GPU at 0.75 utilization

Documentation Updates

README.md updates - Advanced Agent Supervision section, vLLM guide link, feature descriptions
API documentation updates
Configuration documentation updates - backend/docs/config.md (supervision settings, defaults)
GraphQL schema updates - Multi-query search support
Other: New comprehensive vLLM deployment guide (examples/guides/vllm-qwen35-27b-fp8.md), backend/docs/flow_execution.md (supervision system), provider configs with/without thinking mode

Deployment Notes

New Environment Variables (Optional - Beta Features Disabled by Default):

# Execution monitoring (beta)
EXECUTION_MONITOR_ENABLED=false              # Default: false
EXECUTION_MONITOR_SAME_TOOL_LIMIT=5          # Default: 5
EXECUTION_MONITOR_TOTAL_TOOL_LIMIT=10        # Default: 10

# Tool call limits (always active)
MAX_GENERAL_AGENT_TOOL_CALLS=100             # Default: 100
MAX_LIMITED_AGENT_TOOL_CALLS=20              # Default: 20

# Task planning (beta)
AGENT_PLANNING_STEP_ENABLED=false            # Default: false

# Ollama Cloud support (optional)
OLLAMA_SERVER_API_KEY=                       # For Ollama Cloud
PENTAGI_OLLAMA_DIR=                          # Custom Ollama volume path

Database Migrations:

20260310_153000_agent_supervision.sql - Adds supervision-related columns and indexes
Migrations run automatically on startup

Recommended Configuration for Open Source Models < 32B:
Enable supervision features for 2x quality improvement:

EXECUTION_MONITOR_ENABLED=true
AGENT_PLANNING_STEP_ENABLED=true

Configure adviser with enhanced settings (see examples/configs/vllm-qwen3.5-27b-fp8.provider.yml)

Compatibility:

✅ Fully backward compatible
✅ Beta features disabled by default (no impact on existing deployments)
✅ No breaking API changes
✅ Existing flows work without modification

Checklist

Code Quality

My code follows the project's coding standards
I have added/updated necessary documentation
I have added tests to cover my changes
All new and existing tests pass
I have run go fmt and go vet (for Go code)
I have run npm run lint (for TypeScript/JavaScript code)

Security

I have considered security implications
Changes maintain or improve the security model
Sensitive information has been properly handled

Compatibility

Changes are backward compatible
Breaking changes are clearly marked and documented
Dependencies are properly updated

Documentation

Documentation is clear and complete
Comments are added for non-obvious code
API changes are documented

Additional Notes

Key Changes by Category

🤖 Agent Supervision System (New - Beta)

Execution Monitoring (PRs #178, #179, #180):

executionMonitorDetector tracks tool call patterns (same tool: 5, total: 10 thresholds)
performMentor invokes adviser agent for execution analysis
Enhanced tool responses with <mentor_analysis> and <original_result> sections
Prevents loops, suggests alternatives, guides toward completion
Configurable via EXECUTION_MONITOR_ENABLED, EXECUTION_MONITOR_SAME_TOOL_LIMIT, EXECUTION_MONITOR_TOTAL_TOOL_LIMIT
Testing shows 2x quality improvement on Qwen3.5-27B-FP8
Contributors: @asdek

Task Planning:

performPlanner generates 3-7 step execution plans via adviser in planning mode
question_task_planner.tmpl prompt template for structured planning
task_assignment_wrapper.tmpl wraps requests with execution plans
Prevents scope creep, improves completion rates for specialist agents
Works best with enhanced adviser configuration (maximum reasoning mode)
Configurable via AGENT_PLANNING_STEP_ENABLED
Contributors: @asdek

Tool Call Limits:

Differentiated limits by agent type (general: 100, limited: 20)
Reflector-guided graceful termination when approaching limits
Prevents infinite loops independently from beta features
Configurable via MAX_GENERAL_AGENT_TOOL_CALLS, MAX_LIMITED_AGENT_TOOL_CALLS
Contributors: @mason5052, @asdek

Prompt Templates:

New: question_execution_monitor.tmpl for mentor invocations
New: question_task_planner.tmpl for plan generation
New: task_assignment_wrapper.tmpl for wrapping specialist requests
Updated: All agent prompts (primary, assistant, coder, pentester, installer, adviser, searcher, enricher, memorist) with supervision awareness and OOB attack improvements
Contributors: @asdek

🧠 Memory System Enhancements

Multi-Query Vector Search:

search_in_memory, search_guide, search_answer, search_code now support up to 5 simultaneous queries
queries array parameter replaces single query string
Backward compatible: single query still supported
Returns diverse results from multiple search angles
Contributors: @asdek

Target Anonymization:

New memory_utils.go with IP/domain anonymization functions
Replaces actual targets with placeholders (e.g., 203.0.113.1 → TARGET_IP_1, example.com → TARGET_DOMAIN_1)
Applied to all search/store vector operations
Prevents agents from switching targets based on historical context
Unit tests in memory_utils_test.go
Contributors: @asdek

Enhanced Storage:

Tool results include parent agent context for better traceability
Improved metadata filtering and relevance scoring
Contributors: @asdek

🐛 Critical Bug Fixes

Infinite Loop Prevention (PR #178, Closes #175):

Added hard iteration cap (100) to performAgentChain main loop
Repeating detector escalation: 4 soft warnings → abort on 7th consecutive identical call
New constant maxSoftDetectionsBeforeAbort with sync comments
Contributors: @mason5052

Context Isolation for Detached Commands (PR #179, Closes #176):

Fixed detached terminal commands inheriting parent context cancellation
Used context.WithoutCancel() for detached goroutines
Preserves tracing/logging context values while preventing cancellation propagation
Non-detached commands unchanged (still respect parent cancellation)
Unit tests: TestExecCommandDetachSurvivesParentCancel, TestExecCommandNonDetachRespectsParentCancel
Contributors: @mason5052

🧪 Test Coverage

Chinese Provider Tests (PR #189):

Added comprehensive tests for DeepSeek, GLM, Kimi, Qwen providers
Config loading, provider type, model loading, model prefix, missing API key, usage tests
GLM tests use non-negative price assertions for free-tier models
Contributors: @mason5052

Repeating Detector Tests (PR #180):

TestRepeatingDetector: 9 cases covering threshold triggering, reset logic, escalation
TestRepeatingDetectorEscalationThreshold: validates abort math (threshold + 4 = 7)
TestClearCallArguments: message field stripping, key sorting, invalid JSON passthrough
Contributors: @mason5052

Config Tests:

New backend/pkg/config/config_test.go with supervision settings validation
Contributors: @asdek

📚 Documentation & Guides

vLLM + Qwen3.5-27B-FP8 Guide (New):

Comprehensive deployment guide: examples/guides/vllm-qwen35-27b-fp8.md
Hardware requirements, installation steps, server configuration
Two provider configs: with/without thinking mode
Performance benchmarks: 13K TPS prompt, 650 TPS completion on 4× RTX 5090
Troubleshooting section for common issues
Contributors: @asdek

Provider Configurations:

Updated vllm-qwen3.5-27b-fp8.provider.yml with optimal sampling parameters
New vllm-qwen3.5-27b-fp8-no-think.provider.yml for faster inference without thinking
Enhanced adviser configuration examples (maximum reasoning mode)
Contributors: @asdek

Configuration Documentation:

backend/docs/config.md: New Agent Supervision Settings section with usage details, recommended settings, supervision system integration
backend/docs/flow_execution.md: Expanded Advanced Agent Supervision section with execution monitoring, task planning, integration diagrams
Updated defaults to false for beta features
Contributors: @asdek

README Updates:

New Advanced Agent Supervision collapsible section with beta features, recommendations for open source models, trade-offs, configuration strategies
Updated feature descriptions for autonomous execution and specialist delegation
Added vLLM guide reference in multiple locations
Contributors: @asdek

🔧 Installer Enhancements

AI Agents Settings Form Expansion:

Expanded from 2 to 8 configuration fields
Added integer input fields with validation (ranges: 1-50, 1-100, 10-500, 5-200)
Helper methods: createBooleanField, createIntegerField, validateBooleanField, validateIntegerField, formatNumber
Updated controller: GetAIAgentsConfig, UpdateAIAgentsConfig, ResetAIAgentsConfig
Beta warnings with ⚠️ symbols in UI
Comprehensive locale strings with recommendations for open source models
Contributors: @asdek

Ollama Enhancements:

Added OLLAMA_SERVER_API_KEY for Ollama Cloud support
New PENTAGI_OLLAMA_DIR volume configuration
SSH key generation in entrypoint for Ollama
Added min_p parameter support across all providers
Contributors: @asdek

📦 Dependencies

langchaingo v0.1.14-update.5:

Fixed Bedrock ValidationException with multiple tool calls (6 message chain variants supported)
Fixed OpenAI streaming tool call errors (function name caching for incomplete chunks)
Updated Ollama SDK to v0.18.0
Contributors: @asdek

🎨 Code Quality

ftester Improvements:

Updated mocks for new multi-query vector search API
Enhanced test executor and worker logic
Contributors: @asdek

Provider Enhancements:

Pointer-based model options for memory efficiency
Enhanced error handling in tool call retry logic
Improved reflector invocation on generation failures
Contributors: @asdek

Merged Pull Requests

#174 - chore(deps): bump npm_and_yarn group
#178 - fix: prevent infinite loop in performAgentChain on repeating tool calls
#179 - fix: isolate detached command context from parent cancellation
#180 - test: add unit tests for repeatingDetector and clearCallArguments
#189 - test: add unit tests for DeepSeek, GLM, Kimi, and Qwen providers

Issues Addressed

#175 - Infinite loop in agent chain execution
#176 - Detached commands terminated by parent context cancellation

Migration Path

For existing deployments:

Pull latest changes: git pull origin feature/next_release
Rebuild containers: docker compose build
Restart services: docker compose up -d
Migrations apply automatically
Beta features disabled by default - no impact on existing workflows
To enable supervision for open source models < 32B: set EXECUTION_MONITOR_ENABLED=true and AGENT_PLANNING_STEP_ENABLED=true in .env
For local vLLM deployment: follow examples/guides/vllm-qwen35-27b-fp8.md

No manual intervention required. Existing deployments continue working without changes.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/vxcontrol/pentagi/pull/206 **Author:** [@asdek](https://github.com/asdek) **Created:** 3/15/2026 **Status:** ✅ Merged **Merged:** 3/15/2026 **Merged by:** [@asdek](https://github.com/asdek) **Base:** `master` ← **Head:** `feature/next_release` --- ### 📝 Commits (10+) - [`4cddef4`](https://github.com/vxcontrol/pentagi/commit/4cddef477295821e050b45ece455e13d9304363f) chore(deps): bump the npm_and_yarn group across 1 directory with 2 updates - [`1f896a9`](https://github.com/vxcontrol/pentagi/commit/1f896a9aa48b4882ab52b342395a8ea6e10c1af5) fix: add iteration cap and repeating escalation to performAgentChain - [`536ea86`](https://github.com/vxcontrol/pentagi/commit/536ea86a55491ccfaae7a455729906e8e503221d) fix: address review feedback on iteration cap and repeating escalation - [`8701f15`](https://github.com/vxcontrol/pentagi/commit/8701f1573dceb4955835234c4394cac1713b8e62) fix: isolate detached command context from parent cancellation - [`55aef6a`](https://github.com/vxcontrol/pentagi/commit/55aef6aa951830e7cacd4c5514fe702a088834fd) test: add context cancellation tests for detached command isolation - [`1e2cf11`](https://github.com/vxcontrol/pentagi/commit/1e2cf11c79bfda676f4e20a5bc063db1d41c4d7d) Rename mock type to avoid cross-PR symbol collision - [`fc90e9d`](https://github.com/vxcontrol/pentagi/commit/fc90e9d2081c81e2f3227d2a59806f90cf003be0) test: add unit tests for repeatingDetector and clearCallArguments - [`0ffe6a4`](https://github.com/vxcontrol/pentagi/commit/0ffe6a4639c5d7cadc2a6e7af3071a1036be5f99) fix: add named test constant and same-name-different-args edge case - [`afef83f`](https://github.com/vxcontrol/pentagi/commit/afef83f076bbf51f29ed0aec88cae951694290bd) test: add unit tests for DeepSeek, GLM, Kimi, and Qwen providers - [`7c3d6d6`](https://github.com/vxcontrol/pentagi/commit/7c3d6d6ed652caed380a2cd37f9abf8b7138c8f6) Merge pull request #174 from vxcontrol/dependabot/npm_and_yarn/frontend/npm_and_yarn-8a6d6a6aaf ### 📊 Changes **75 files changed** (+6871 additions, -1588 deletions) <details> <summary>View changed files</summary> 📝 `.env.example` (+12 -0) 📝 `.vscode/launch.json` (+1 -1) 📝 `Dockerfile` (+1 -0) 📝 `README.md` (+101 -22) 📝 `backend/cmd/ftester/mocks/tools.go` (+32 -12) 📝 `backend/cmd/ftester/worker/executor.go` (+22 -2) 📝 `backend/cmd/ftester/worker/tester.go` (+4 -1) 📝 `backend/cmd/installer/wizard/controller/controller.go` (+47 -3) 📝 `backend/cmd/installer/wizard/locale/locale.go` (+63 -12) 📝 `backend/cmd/installer/wizard/models/ai_agents_settings_form.go` (+268 -57) 📝 `backend/docs/config.md` (+164 -1) 📝 `backend/docs/flow_execution.md` (+241 -10) 📝 `backend/go.mod` (+5 -2) 📝 `backend/go.sum` (+10 -4) ➕ `backend/migrations/sql/20260310_153000_agent_supervision.sql` (+114 -0) 📝 `backend/pkg/config/config.go` (+72 -0) ➕ `backend/pkg/config/config_test.go` (+259 -0) 📝 `backend/pkg/database/converter/converter.go` (+3 -0) 📝 `backend/pkg/database/models.go` (+39 -36) 📝 `backend/pkg/graph/generated.go` (+217 -16) _...and 55 more files_ </details> ### 📄 Description  ### Description of the Change #### Problem Smaller open source models (< 32B parameters) required additional supervision to match cloud API quality. Agents could get stuck in loops or miss optimal attack paths. Vector store searches returned insufficient results due to single-query limitation. Target information leakage in vector memory could cause agents to switch targets mid-attack. No production-grade guide existed for air-gapped deployments with local LLM inference. #### Solution Implemented multi-layered agent supervision system with: - **Execution Monitoring (beta)**: Automatic mentor (adviser) intervention analyzes execution patterns, detects loops, suggests alternative strategies. Testing with Qwen3.5-27B-FP8 shows **2x improvement in result quality** vs. baseline - **Task Planning (beta)**: Planner generates 3-7 step execution plans before specialist agents begin work, preventing scope creep - **Tool Call Limits**: Hard caps prevent infinite loops (general agents: 100, limited agents: 20) with reflector-guided graceful termination - **Multi-Query Vector Search**: Agents can submit up to 5 search queries simultaneously for better knowledge retrieval - **Target Anonymization**: Search/store operations anonymize IP addresses and domains to prevent target switching - **vLLM Deployment Guide**: Comprehensive documentation for running Qwen3.5-27B-FP8 locally (13K TPS prompt processing on 4× RTX 5090) - **Installer Configuration**: AI Agents Settings form expanded with supervision controls - **Prompt Improvements**: Enhanced stability and OOB attack coverage across all agent prompts - **langchaingo Update**: v0.1.14-update.5 with Bedrock multi-tool fixes and OpenAI streaming robustness - **Ollama Cloud Support**: Added API key support and enhanced configuration Closes #175, #176 ### Type of Change  - [x] 🐛 Bug fix (non-breaking change which fixes an issue) - [x] 🚀 New feature (non-breaking change which adds functionality) - [ ] 💥 Breaking change (fix or feature that would cause existing functionality to not work as expected) - [x] 📚 Documentation update - [x] 🔧 Configuration change - [x] 🧪 Test update - [ ] 🛡️ Security update ### Areas Affected  - [x] Core Services (Frontend UI/Backend API) - [x] AI Agents (Researcher/Developer/Executor) - [ ] Security Tools Integration - [x] Memory System (Vector Store/Knowledge Base) - [ ] Monitoring Stack (Grafana/OpenTelemetry) - [ ] Analytics Platform (Langfuse) - [x] External Integrations (LLM/Search APIs) - [x] Documentation - [x] Infrastructure/DevOps ### Testing and Verification #### Test Configuration ```yaml PentAGI Version: Latest development (feature/next_release) Docker Version: 24.0.x+ Host OS: Linux (Ubuntu 22.04) LLM Provider: vLLM (Qwen3.5-27B-FP8), OpenAI, Anthropic Enabled Features: Core + supervision features (beta) Hardware: 4× RTX 5090 (128GB VRAM total) ``` #### Test Steps 1. Run all backend unit tests (`go test ./...`) 2. Test execution monitoring with Qwen3.5-27B-FP8 (complex penetration testing flows) 3. Test task planning with enhanced adviser configuration 4. Verify multi-query vector search (up to 5 simultaneous queries) 5. Test target anonymization in vector store operations 6. Test infinite loop prevention with tool call limits 7. Test detached command context isolation 8. Verify Chinese provider configurations (DeepSeek, GLM, Kimi, Qwen) 9. Test AI Agents Settings installer form with all 8 fields 10. Verify vLLM deployment guide with real hardware #### Test Results - ✅ All existing and new unit tests pass - ✅ **Quality improvement**: 2x better results with supervision enabled on Qwen3.5-27B-FP8 - ✅ **Performance verified**: 13K TPS prompt processing, 650 TPS completion on 4× RTX 5090 - ✅ Infinite loop prevention working (7 consecutive calls → abort) - ✅ Multi-query vector search returns diverse results - ✅ Target anonymization prevents context leakage - ✅ Detached commands survive parent context cancellation - ✅ Chinese provider tests added and passing - ✅ Installer form validates all supervision parameters correctly - ✅ No breaking changes to existing functionality ### Security Considerations **Enhanced Security:** - **Target Anonymization**: IP addresses and domains anonymized in vector store to prevent accidental target switching during multi-phase attacks - **Context Isolation**: Detached terminal commands now properly isolated from parent context cancellation (prevents premature termination) - **Loop Prevention**: Hard tool call limits prevent resource exhaustion from runaway agents **No New Attack Vectors:** - All supervision features are read-only analysis mechanisms - No new credentials or external dependencies required - Maintains existing security model - Backward compatible with all deployments ### Performance Impact **With Supervision Features Enabled** (for models < 32B): - Token consumption: 2-3x increase due to mentor/planner invocations - Execution time: 2-3x longer due to analysis and planning steps - Result quality: **2x improvement** in completeness, accuracy, and attack coverage - Trade-off: User choice between speed vs. quality **Multi-Query Vector Search**: - Up to 5 parallel queries vs. previous single query - Better knowledge retrieval diversity - Minimal latency increase (queries run in parallel) **Infinite Loop Prevention**: - Eliminates unbounded executions - Reduces resource waste from repeated failed attempts **vLLM Performance** (Qwen3.5-27B-FP8 on 4× RTX 5090): - Prompt processing: ~13,000 tokens/sec - Completion generation: ~650 tokens/sec - Concurrent flows: 12+ simultaneously - VRAM usage: ~30GB per GPU at 0.75 utilization ### Documentation Updates  - [x] README.md updates - Advanced Agent Supervision section, vLLM guide link, feature descriptions - [ ] API documentation updates - [x] Configuration documentation updates - backend/docs/config.md (supervision settings, defaults) - [x] GraphQL schema updates - Multi-query search support - [x] Other: New comprehensive vLLM deployment guide (examples/guides/vllm-qwen35-27b-fp8.md), backend/docs/flow_execution.md (supervision system), provider configs with/without thinking mode ### Deployment Notes **New Environment Variables (Optional - Beta Features Disabled by Default):** ```bash # Execution monitoring (beta) EXECUTION_MONITOR_ENABLED=false # Default: false EXECUTION_MONITOR_SAME_TOOL_LIMIT=5 # Default: 5 EXECUTION_MONITOR_TOTAL_TOOL_LIMIT=10 # Default: 10 # Tool call limits (always active) MAX_GENERAL_AGENT_TOOL_CALLS=100 # Default: 100 MAX_LIMITED_AGENT_TOOL_CALLS=20 # Default: 20 # Task planning (beta) AGENT_PLANNING_STEP_ENABLED=false # Default: false # Ollama Cloud support (optional) OLLAMA_SERVER_API_KEY= # For Ollama Cloud PENTAGI_OLLAMA_DIR= # Custom Ollama volume path ``` **Database Migrations:** - `20260310_153000_agent_supervision.sql` - Adds supervision-related columns and indexes - Migrations run automatically on startup **Recommended Configuration for Open Source Models < 32B:** Enable supervision features for 2x quality improvement: ```bash EXECUTION_MONITOR_ENABLED=true AGENT_PLANNING_STEP_ENABLED=true ``` Configure adviser with enhanced settings (see examples/configs/vllm-qwen3.5-27b-fp8.provider.yml) **Compatibility:** - ✅ Fully backward compatible - ✅ Beta features disabled by default (no impact on existing deployments) - ✅ No breaking API changes - ✅ Existing flows work without modification ### Checklist  #### Code Quality - [x] My code follows the project's coding standards - [x] I have added/updated necessary documentation - [x] I have added tests to cover my changes - [x] All new and existing tests pass - [x] I have run `go fmt` and `go vet` (for Go code) - [x] I have run `npm run lint` (for TypeScript/JavaScript code) #### Security - [x] I have considered security implications - [x] Changes maintain or improve the security model - [x] Sensitive information has been properly handled #### Compatibility - [x] Changes are backward compatible - [ ] Breaking changes are clearly marked and documented - [x] Dependencies are properly updated #### Documentation - [x] Documentation is clear and complete - [x] Comments are added for non-obvious code - [x] API changes are documented ### Additional Notes ## Key Changes by Category ### 🤖 Agent Supervision System (New - Beta) **Execution Monitoring** (PRs #178, #179, #180): - `executionMonitorDetector` tracks tool call patterns (same tool: 5, total: 10 thresholds) - `performMentor` invokes adviser agent for execution analysis - Enhanced tool responses with `<mentor_analysis>` and `<original_result>` sections - Prevents loops, suggests alternatives, guides toward completion - Configurable via `EXECUTION_MONITOR_ENABLED`, `EXECUTION_MONITOR_SAME_TOOL_LIMIT`, `EXECUTION_MONITOR_TOTAL_TOOL_LIMIT` - Testing shows **2x quality improvement** on Qwen3.5-27B-FP8 - Contributors: @asdek **Task Planning**: - `performPlanner` generates 3-7 step execution plans via adviser in planning mode - `question_task_planner.tmpl` prompt template for structured planning - `task_assignment_wrapper.tmpl` wraps requests with execution plans - Prevents scope creep, improves completion rates for specialist agents - Works best with enhanced adviser configuration (maximum reasoning mode) - Configurable via `AGENT_PLANNING_STEP_ENABLED` - Contributors: @asdek **Tool Call Limits**: - Differentiated limits by agent type (general: 100, limited: 20) - Reflector-guided graceful termination when approaching limits - Prevents infinite loops independently from beta features - Configurable via `MAX_GENERAL_AGENT_TOOL_CALLS`, `MAX_LIMITED_AGENT_TOOL_CALLS` - Contributors: @mason5052, @asdek **Prompt Templates**: - New: `question_execution_monitor.tmpl` for mentor invocations - New: `question_task_planner.tmpl` for plan generation - New: `task_assignment_wrapper.tmpl` for wrapping specialist requests - Updated: All agent prompts (primary, assistant, coder, pentester, installer, adviser, searcher, enricher, memorist) with supervision awareness and OOB attack improvements - Contributors: @asdek ### 🧠 Memory System Enhancements **Multi-Query Vector Search**: - `search_in_memory`, `search_guide`, `search_answer`, `search_code` now support up to 5 simultaneous queries - `queries` array parameter replaces single `query` string - Backward compatible: single query still supported - Returns diverse results from multiple search angles - Contributors: @asdek **Target Anonymization**: - New `memory_utils.go` with IP/domain anonymization functions - Replaces actual targets with placeholders (e.g., `203.0.113.1` → `TARGET_IP_1`, `example.com` → `TARGET_DOMAIN_1`) - Applied to all search/store vector operations - Prevents agents from switching targets based on historical context - Unit tests in `memory_utils_test.go` - Contributors: @asdek **Enhanced Storage**: - Tool results include parent agent context for better traceability - Improved metadata filtering and relevance scoring - Contributors: @asdek ### 🐛 Critical Bug Fixes **Infinite Loop Prevention** (PR #178, Closes #175): - Added hard iteration cap (100) to `performAgentChain` main loop - Repeating detector escalation: 4 soft warnings → abort on 7th consecutive identical call - New constant `maxSoftDetectionsBeforeAbort` with sync comments - Contributors: @mason5052 **Context Isolation for Detached Commands** (PR #179, Closes #176): - Fixed detached terminal commands inheriting parent context cancellation - Used `context.WithoutCancel()` for detached goroutines - Preserves tracing/logging context values while preventing cancellation propagation - Non-detached commands unchanged (still respect parent cancellation) - Unit tests: `TestExecCommandDetachSurvivesParentCancel`, `TestExecCommandNonDetachRespectsParentCancel` - Contributors: @mason5052 ### 🧪 Test Coverage **Chinese Provider Tests** (PR #189): - Added comprehensive tests for DeepSeek, GLM, Kimi, Qwen providers - Config loading, provider type, model loading, model prefix, missing API key, usage tests - GLM tests use non-negative price assertions for free-tier models - Contributors: @mason5052 **Repeating Detector Tests** (PR #180): - `TestRepeatingDetector`: 9 cases covering threshold triggering, reset logic, escalation - `TestRepeatingDetectorEscalationThreshold`: validates abort math (threshold + 4 = 7) - `TestClearCallArguments`: message field stripping, key sorting, invalid JSON passthrough - Contributors: @mason5052 **Config Tests**: - New `backend/pkg/config/config_test.go` with supervision settings validation - Contributors: @asdek ### 📚 Documentation & Guides **vLLM + Qwen3.5-27B-FP8 Guide** (New): - Comprehensive deployment guide: `examples/guides/vllm-qwen35-27b-fp8.md` - Hardware requirements, installation steps, server configuration - Two provider configs: with/without thinking mode - Performance benchmarks: 13K TPS prompt, 650 TPS completion on 4× RTX 5090 - Troubleshooting section for common issues - Contributors: @asdek **Provider Configurations**: - Updated `vllm-qwen3.5-27b-fp8.provider.yml` with optimal sampling parameters - New `vllm-qwen3.5-27b-fp8-no-think.provider.yml` for faster inference without thinking - Enhanced adviser configuration examples (maximum reasoning mode) - Contributors: @asdek **Configuration Documentation**: - `backend/docs/config.md`: New Agent Supervision Settings section with usage details, recommended settings, supervision system integration - `backend/docs/flow_execution.md`: Expanded Advanced Agent Supervision section with execution monitoring, task planning, integration diagrams - Updated defaults to `false` for beta features - Contributors: @asdek **README Updates**: - New Advanced Agent Supervision collapsible section with beta features, recommendations for open source models, trade-offs, configuration strategies - Updated feature descriptions for autonomous execution and specialist delegation - Added vLLM guide reference in multiple locations - Contributors: @asdek ### 🔧 Installer Enhancements **AI Agents Settings Form Expansion**: - Expanded from 2 to 8 configuration fields - Added integer input fields with validation (ranges: 1-50, 1-100, 10-500, 5-200) - Helper methods: `createBooleanField`, `createIntegerField`, `validateBooleanField`, `validateIntegerField`, `formatNumber` - Updated controller: `GetAIAgentsConfig`, `UpdateAIAgentsConfig`, `ResetAIAgentsConfig` - Beta warnings with ⚠️ symbols in UI - Comprehensive locale strings with recommendations for open source models - Contributors: @asdek **Ollama Enhancements**: - Added `OLLAMA_SERVER_API_KEY` for Ollama Cloud support - New `PENTAGI_OLLAMA_DIR` volume configuration - SSH key generation in entrypoint for Ollama - Added `min_p` parameter support across all providers - Contributors: @asdek ### 📦 Dependencies **langchaingo v0.1.14-update.5**: - Fixed Bedrock ValidationException with multiple tool calls (6 message chain variants supported) - Fixed OpenAI streaming tool call errors (function name caching for incomplete chunks) - Updated Ollama SDK to v0.18.0 - Contributors: @asdek ### 🎨 Code Quality **ftester Improvements**: - Updated mocks for new multi-query vector search API - Enhanced test executor and worker logic - Contributors: @asdek **Provider Enhancements**: - Pointer-based model options for memory efficiency - Enhanced error handling in tool call retry logic - Improved reflector invocation on generation failures - Contributors: @asdek ## Merged Pull Requests - #174 - chore(deps): bump npm_and_yarn group - #178 - fix: prevent infinite loop in performAgentChain on repeating tool calls - #179 - fix: isolate detached command context from parent cancellation - #180 - test: add unit tests for repeatingDetector and clearCallArguments - #189 - test: add unit tests for DeepSeek, GLM, Kimi, and Qwen providers ## Issues Addressed - #175 - Infinite loop in agent chain execution - #176 - Detached commands terminated by parent context cancellation ## Migration Path For existing deployments: 1. Pull latest changes: `git pull origin feature/next_release` 2. Rebuild containers: `docker compose build` 3. Restart services: `docker compose up -d` 4. Migrations apply automatically 5. Beta features disabled by default - no impact on existing workflows 6. To enable supervision for open source models < 32B: set `EXECUTION_MONITOR_ENABLED=true` and `AGENT_PLANNING_STEP_ENABLED=true` in `.env` 7. For local vLLM deployment: follow `examples/guides/vllm-qwen35-27b-fp8.md` No manual intervention required. Existing deployments continue working without changes. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

yindo added the pull-request label 2026-06-06 22:09:49 -04:00

yindo closed this issue

2026-06-06 22:09:49 -04:00

yindo referenced this issue

2026-06-06 22:10:04 -04:00

[PR #277] [MERGED] docs: add evidence chain RFC #290

yindo referenced this issue

2026-06-06 22:10:05 -04:00

[PR #279] feat: add evidence receipt hash chain prototype #293

yindo referenced this issue

2026-06-06 22:10:10 -04:00

[PR #306] [MERGED] docs(rfc): propose persistent flow queue and completion webhooks #311