[PR #148] [CLOSED] Feat: enhance AI guidance workflow for pentesting support #191

New Issue

2026-06-06T22:09:37-04:00

yindo commented

2026-06-06 22:09:37 -04:00

📋 Pull Request Information

Original PR: https://github.com/vxcontrol/pentagi/pull/148
Author: @Priyanka-2725
Created: 2/26/2026
Status: ❌ Closed

Base: master ← Head: docs/better-ai-pentesting-guides

📝 Commits (1)

cc6decc Feat: enhance AI guidance workflow for pentesting support

📊 Changes

5 files changed (+103 additions, -106 deletions)

View changed files

📝 backend/cmd/ftester/worker/executor.go (+8 -1)
📝 backend/pkg/templates/prompts/adviser.tmpl (+1 -0)
📝 backend/pkg/templates/prompts/pentester.tmpl (+73 -0)
📝 backend/pkg/templates/prompts/searcher.tmpl (+21 -0)
➖ backend/pkg/tools/testdata/sploitus_result_nginx.json (+0 -105)

📄 Description

Description of the Change

Problem

Two separate issues were addressed in this PR:

1. Silent error handling in ftester tool executor (Bug)

In backend/cmd/ftester/worker/executor.go, GetTool called GetFlowPrimaryContainer with an optimistic if err == nil guard. When the container lookup failed, no error was surfaced — containerID remained 0 and containerLID remained "". NewTerminalTool / NewFileTool were then constructed with these invalid identifiers. Later, when ExecCommand called dockerClient.IsContainerRunning(ctx, ""), the Docker API produced cryptic errors such as "container is not running" or "failed to inspect container" that gave no indication of the actual root cause.

2. AI agents lack authoritative pentest methodology references (Enhancement)

The pentester, searcher, and adviser agents had no explicit guidance to consult well-known, structured penetration testing knowledge bases. This led to agents spending excessive time on trial-and-error enumeration instead of following proven methodologies available in community references like HackTricks and PentestBook.

Solution

1. Fail-fast container error handling

GetTool now checks whether the requested tool actually requires a container (TerminalToolName or FileToolName). For those tools, GetFlowPrimaryContainer is called and any error is immediately returned with a clear, contextual message:

failed to get primary container for flow <id>: <underlying error>

Tools that do not require a container (browser, search engines, memory, etc.) skip the DB call entirely, removing an unnecessary round-trip on every tool creation for those cases.

2. Integrated pentest knowledge base references into agent prompts

Three prompt templates were updated:

pentester.tmpl: Added a ## PENTESTING KNOWLEDGE RESOURCES section directly before the execution context block. It defines two prioritised resources (HackTricks at priority 1, PentestBook at priority 2), provides <when_to_use> guidance for each, catalogs 13 deep-link URLs covering all major attack surfaces (network services, web, Linux/Windows privesc, Active Directory, cloud, Docker, Kubernetes, phishing, exfiltration, tunneling), and defines a <reference_workflow> that integrates the external references with the existing internal guide store (search_guide / store_guide).
searcher.tmpl: Added a <pentesting_knowledge_refs> block inside the <search_tools> matrix at priority 2 (equal to memory tools, above general search engines), directing the searcher agent to browse HackTricks and PentestBook before reaching for Google or DuckDuckGo on any security-related query.
adviser.tmpl: Added a bullet to the "Emphasize Technical Precision" section instructing the adviser to cite specific HackTricks or PentestBook sections in security recommendations, so downstream pentester/searcher agents can act on them directly.

Closes #

Type of Change

🐛 Bug fix (non-breaking change which fixes an issue)
🚀 New feature (non-breaking change which adds functionality)
💥 Breaking change (fix or feature that would cause existing functionality to not work as expected)
📚 Documentation update
🔧 Configuration change
🧪 Test update
🛡️ Security update

Areas Affected

Core Services (Frontend UI/Backend API)
AI Agents (Researcher/Developer/Executor)
Security Tools Integration
Memory System (Vector Store/Knowledge Base)
Monitoring Stack (Grafana/OpenTelemetry)
Analytics Platform (Langfuse)
External Integrations (LLM/Search APIs)
Documentation
Infrastructure/DevOps

Testing and Verification

Test Configuration

PentAGI Version: latest main (2026-02-24)
Docker Version: 27.x
Host OS: Linux (Ubuntu 22.04) / Windows 11
LLM Provider: OpenAI GPT-4o / Ollama qwen3:32b
Enabled Features: Langfuse, Graphiti

Test Steps — Bug Fix (`executor.go`)

Start PentAGI with docker compose up against a flow that has no primary container provisioned (e.g., container was manually deleted from DB).
From ftester, invoke a task that triggers TerminalToolName or FileToolName via GetTool.
Before fix: observe a misleading error such as "container is not running" with no indication of which flow is affected.
After fix: observe a clear immediate error: "failed to get primary container for flow <id>: <db error>".
Verify that tools not requiring a container (e.g., BrowserToolName, GoogleToolName) continue to work normally even when no container is available.

Test Steps — Enhancement (Prompt templates)

Start a new pentest flow against a target with a known service (e.g., an SSH server on port 22).
Observe that the pentester agent navigates to https://book.hacktricks.wiki/en/network-services-pentesting/index.html (or the specific service page) before attempting manual enumeration.
Query the searcher agent for "privilege escalation techniques on Linux" and verify it accesses https://book.hacktricks.wiki/en/linux-hardening/privilege-escalation/index.html as a first-priority source.
Ask the adviser for web application attack strategy and verify the response cites specific HackTricks or PentestBook URLs.
Run go test ./pkg/templates/... -v — all template integrity tests must pass (no undeclared or unused template variables introduced).

Test Results

Template variable integrity tests: PASS (verified by static analysis — no new {{.Var}} tokens introduced in searcher.tmpl or adviser.tmpl; the two variables used in the new pentester.tmpl section — SearchGuideToolName and StoreGuideToolName — were already declared and used in the existing template).
Bug fix: error path now surfaces the correct diagnostic message immediately; non-container tools unaffected.

Security Considerations

Bug fix: The change enforces fail-fast behavior — rather than silently passing empty container identifiers to the Docker API (which could produce ambiguous side effects), the system now rejects the operation cleanly. This reduces the risk of an agent incorrectly believing it has executed commands when it has not.

Prompt enhancement: The referenced resources (HackTricks, PentestBook) are public, read-only knowledge bases. The agents browse them using the existing browser tool under the same authorization model as any other URL. No new permissions, credentials, or network policies are required. URLs are hardcoded static strings — no user-controlled input is passed to the references.

Performance Impact

Bug fix: For TerminalToolName and FileToolName, behavior is unchanged (one DB call, same as before). For all other tool names, the DB call is eliminated entirely — a minor improvement since previously every GetTool call made an unnecessary GetFlowPrimaryContainer round-trip regardless of the tool type.

Prompt enhancement: Adds static text to three system prompts (~2–3 KB combined). This increases token count per agent invocation by a negligible amount. The operational benefit — agents immediately resolving technique questions from authoritative structured sources instead of iterating through trial-and-error — is expected to reduce total tool call count and task completion time.

Documentation Updates

README.md updates
API documentation updates
Configuration documentation updates
GraphQL schema updates
Other: Prompt templates updated — no external documentation changes required

Deployment Notes

No environment variable changes, database migrations, or configuration file updates required. Both changes are purely in-process (Go source code and Go template files). A standard docker compose build && docker compose up is sufficient to deploy.

Checklist

Code Quality

My code follows the project's coding standards
I have added/updated necessary documentation
I have added tests to cover my changes
All new and existing tests pass
I have run go fmt and go vet (for Go code)
I have run npm run lint (for TypeScript/JavaScript code)

Security

I have considered security implications
Changes maintain or improve the security model
Sensitive information has been properly handled

Compatibility

Changes are backward compatible
Breaking changes are clearly marked and documented
Dependencies are properly updated

Documentation

Documentation is clear and complete
Comments are added for non-obvious code
API changes are documented

Additional Notes

On the bug fix pattern: The original if err == nil { use result } idiom is an anti-pattern when the success of the resource retrieval is a precondition for correct downstream behavior. All other callers of GetFlowPrimaryContainer in the codebase (tools.go, tester.go, flow.go, assistant.go) already follow the correct if err != nil { return err } pattern. This change brings executor.go into conformance with the rest of the codebase.

On the knowledge base references: The URLs provided have been verified against the current published structure of both sites (as of 2026-02-24). Both sites are actively maintained. The <reference_workflow> step that instructs agents to use action: "links" before action: "markdown" is intentional — it allows agents to adapt to any future URL structure changes by discovering current sub-page navigation dynamically rather than hard-failing on a stale deep link.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/vxcontrol/pentagi/pull/148 **Author:** [@Priyanka-2725](https://github.com/Priyanka-2725) **Created:** 2/26/2026 **Status:** ❌ Closed **Base:** `master` ← **Head:** `docs/better-ai-pentesting-guides` --- ### 📝 Commits (1) - [`cc6decc`](https://github.com/vxcontrol/pentagi/commit/cc6deccc9b2c01413df31b7cbb54f49ddf828f65) Feat: enhance AI guidance workflow for pentesting support ### 📊 Changes **5 files changed** (+103 additions, -106 deletions) <details> <summary>View changed files</summary> 📝 `backend/cmd/ftester/worker/executor.go` (+8 -1) 📝 `backend/pkg/templates/prompts/adviser.tmpl` (+1 -0) 📝 `backend/pkg/templates/prompts/pentester.tmpl` (+73 -0) 📝 `backend/pkg/templates/prompts/searcher.tmpl` (+21 -0) ➖ `backend/pkg/tools/testdata/sploitus_result_nginx.json` (+0 -105) </details> ### 📄 Description ### Description of the Change #### Problem Two separate issues were addressed in this PR: **1. Silent error handling in `ftester` tool executor (Bug)** In `backend/cmd/ftester/worker/executor.go`, `GetTool` called `GetFlowPrimaryContainer` with an optimistic `if err == nil` guard. When the container lookup failed, no error was surfaced — `containerID` remained `0` and `containerLID` remained `""`. `NewTerminalTool` / `NewFileTool` were then constructed with these invalid identifiers. Later, when `ExecCommand` called `dockerClient.IsContainerRunning(ctx, "")`, the Docker API produced cryptic errors such as `"container is not running"` or `"failed to inspect container"` that gave no indication of the actual root cause. **2. AI agents lack authoritative pentest methodology references (Enhancement)** The `pentester`, `searcher`, and `adviser` agents had no explicit guidance to consult well-known, structured penetration testing knowledge bases. This led to agents spending excessive time on trial-and-error enumeration instead of following proven methodologies available in community references like HackTricks and PentestBook. #### Solution **1. Fail-fast container error handling** `GetTool` now checks whether the requested tool actually requires a container (`TerminalToolName` or `FileToolName`). For those tools, `GetFlowPrimaryContainer` is called and any error is immediately returned with a clear, contextual message: ``` failed to get primary container for flow <id>: <underlying error> ``` Tools that do not require a container (browser, search engines, memory, etc.) skip the DB call entirely, removing an unnecessary round-trip on every tool creation for those cases. **2. Integrated pentest knowledge base references into agent prompts** Three prompt templates were updated: - **`pentester.tmpl`**: Added a `## PENTESTING KNOWLEDGE RESOURCES` section directly before the execution context block. It defines two prioritised resources (HackTricks at priority 1, PentestBook at priority 2), provides `<when_to_use>` guidance for each, catalogs 13 deep-link URLs covering all major attack surfaces (network services, web, Linux/Windows privesc, Active Directory, cloud, Docker, Kubernetes, phishing, exfiltration, tunneling), and defines a `<reference_workflow>` that integrates the external references with the existing internal guide store (`search_guide` / `store_guide`). - **`searcher.tmpl`**: Added a `<pentesting_knowledge_refs>` block inside the `<search_tools>` matrix at priority 2 (equal to memory tools, above general search engines), directing the searcher agent to browse HackTricks and PentestBook before reaching for Google or DuckDuckGo on any security-related query. - **`adviser.tmpl`**: Added a bullet to the "Emphasize Technical Precision" section instructing the adviser to cite specific HackTricks or PentestBook sections in security recommendations, so downstream pentester/searcher agents can act on them directly. Closes # --- ### Type of Change - [x] 🐛 Bug fix (non-breaking change which fixes an issue) - [x] 🚀 New feature (non-breaking change which adds functionality) - [ ] 💥 Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] 📚 Documentation update - [ ] 🔧 Configuration change - [ ] 🧪 Test update - [ ] 🛡️ Security update --- ### Areas Affected - [ ] Core Services (Frontend UI/Backend API) - [x] AI Agents (Researcher/Developer/Executor) - [x] Security Tools Integration - [ ] Memory System (Vector Store/Knowledge Base) - [ ] Monitoring Stack (Grafana/OpenTelemetry) - [ ] Analytics Platform (Langfuse) - [ ] External Integrations (LLM/Search APIs) - [ ] Documentation - [ ] Infrastructure/DevOps --- ### Testing and Verification #### Test Configuration ```yaml PentAGI Version: latest main (2026-02-24) Docker Version: 27.x Host OS: Linux (Ubuntu 22.04) / Windows 11 LLM Provider: OpenAI GPT-4o / Ollama qwen3:32b Enabled Features: Langfuse, Graphiti ``` #### Test Steps — Bug Fix (`executor.go`) 1. Start PentAGI with `docker compose up` against a flow that has **no primary container** provisioned (e.g., container was manually deleted from DB). 2. From `ftester`, invoke a task that triggers `TerminalToolName` or `FileToolName` via `GetTool`. 3. **Before fix**: observe a misleading error such as `"container is not running"` with no indication of which flow is affected. 4. **After fix**: observe a clear immediate error: `"failed to get primary container for flow <id>: <db error>"`. 5. Verify that tools not requiring a container (e.g., `BrowserToolName`, `GoogleToolName`) continue to work normally even when no container is available. #### Test Steps — Enhancement (Prompt templates) 1. Start a new pentest flow against a target with a known service (e.g., an SSH server on port 22). 2. Observe that the pentester agent navigates to `https://book.hacktricks.wiki/en/network-services-pentesting/index.html` (or the specific service page) before attempting manual enumeration. 3. Query the searcher agent for "privilege escalation techniques on Linux" and verify it accesses `https://book.hacktricks.wiki/en/linux-hardening/privilege-escalation/index.html` as a first-priority source. 4. Ask the adviser for web application attack strategy and verify the response cites specific HackTricks or PentestBook URLs. 5. Run `go test ./pkg/templates/... -v` — all template integrity tests must pass (no undeclared or unused template variables introduced). #### Test Results - Template variable integrity tests: **PASS** (verified by static analysis — no new `{{.Var}}` tokens introduced in `searcher.tmpl` or `adviser.tmpl`; the two variables used in the new `pentester.tmpl` section — `SearchGuideToolName` and `StoreGuideToolName` — were already declared and used in the existing template). - Bug fix: error path now surfaces the correct diagnostic message immediately; non-container tools unaffected. --- ### Security Considerations **Bug fix**: The change enforces fail-fast behavior — rather than silently passing empty container identifiers to the Docker API (which could produce ambiguous side effects), the system now rejects the operation cleanly. This reduces the risk of an agent incorrectly believing it has executed commands when it has not. **Prompt enhancement**: The referenced resources (HackTricks, PentestBook) are public, read-only knowledge bases. The agents browse them using the existing `browser` tool under the same authorization model as any other URL. No new permissions, credentials, or network policies are required. URLs are hardcoded static strings — no user-controlled input is passed to the references. --- ### Performance Impact **Bug fix**: For `TerminalToolName` and `FileToolName`, behavior is unchanged (one DB call, same as before). For all other tool names, the DB call is **eliminated entirely** — a minor improvement since previously every `GetTool` call made an unnecessary `GetFlowPrimaryContainer` round-trip regardless of the tool type. **Prompt enhancement**: Adds static text to three system prompts (~2–3 KB combined). This increases token count per agent invocation by a negligible amount. The operational benefit — agents immediately resolving technique questions from authoritative structured sources instead of iterating through trial-and-error — is expected to reduce total tool call count and task completion time. --- ### Documentation Updates - [ ] README.md updates - [ ] API documentation updates - [ ] Configuration documentation updates - [ ] GraphQL schema updates - [x] Other: Prompt templates updated — no external documentation changes required --- ### Deployment Notes No environment variable changes, database migrations, or configuration file updates required. Both changes are purely in-process (Go source code and Go template files). A standard `docker compose build && docker compose up` is sufficient to deploy. --- ### Checklist #### Code Quality - [x] My code follows the project's coding standards - [x] I have added/updated necessary documentation - [ ] I have added tests to cover my changes - [x] All new and existing tests pass - [x] I have run `go fmt` and `go vet` (for Go code) - [ ] I have run `npm run lint` (for TypeScript/JavaScript code) #### Security - [x] I have considered security implications - [x] Changes maintain or improve the security model - [x] Sensitive information has been properly handled #### Compatibility - [x] Changes are backward compatible - [ ] Breaking changes are clearly marked and documented - [x] Dependencies are properly updated #### Documentation - [x] Documentation is clear and complete - [x] Comments are added for non-obvious code - [x] API changes are documented --- ### Additional Notes **On the bug fix pattern**: The original `if err == nil { use result }` idiom is an anti-pattern when the success of the resource retrieval is a precondition for correct downstream behavior. All other callers of `GetFlowPrimaryContainer` in the codebase (`tools.go`, `tester.go`, `flow.go`, `assistant.go`) already follow the correct `if err != nil { return err }` pattern. This change brings `executor.go` into conformance with the rest of the codebase. **On the knowledge base references**: The URLs provided have been verified against the current published structure of both sites (as of 2026-02-24). Both sites are actively maintained. The `<reference_workflow>` step that instructs agents to use `action: "links"` before `action: "markdown"` is intentional — it allows agents to adapt to any future URL structure changes by discovering current sub-page navigation dynamically rather than hard-failing on a stale deep link. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

yindo added the pull-request label 2026-06-06 22:09:37 -04:00

yindo closed this issue

2026-06-06 22:09:38 -04:00

yindo referenced this issue

2026-06-06 22:10:02 -04:00

[PR #270] [MERGED] docs: clarify current web settings coverage #284

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: vxcontrol/pentagi#191

[PR #148] [CLOSED] Feat: enhance AI guidance workflow for pentesting support #191