[GH-ISSUE #341] Feature Request: Configurable fallback when tools or models fail, including optional safe tool installation #110

New Issue

2026-06-06T22:09:11-04:00

yindo commented

2026-06-06 22:09:11 -04:00

Originally created by @Akalanka1337 on GitHub (Jun 6, 2026).
Original GitHub issue: https://github.com/vxcontrol/pentagi/issues/341

Originally assigned to: @asdek on GitHub.

Target Component

AI Agents (Researcher/Developer/Executor)

Enhancement Description

Summary

I would like to request a new fallback feature for PentAGI.

Currently, when an agent fails to execute a tool, repeats the same tool, reaches retry limits, or the selected model cannot produce a valid tool call, PentAGI retries and may use the reflector/mentor logic. But after the limit is reached, the flow can still fail or stop.

Can we add an optional fallback mode where the agent tries another safe path before stopping?

Current behavior I noticed

From the source/docs, PentAGI already has:

retry logic for agent calls and tool execution
repeated tool-call detection
reflector agent for fixing bad/no tool calls
execution monitor / mentor logic
Installer Agent for environment/tool setup
multiple search tools like browser, Google, DuckDuckGo, Tavily, Perplexity, Searxng, etc.
multiple LLM provider support

This means the system already has most building blocks needed for fallback.

Requested behavior

Add a configurable fallback layer that runs before the task/subtask is marked failed.

When enabled, if a tool or model fails repeatedly, the agent should be able to:

Detect the failure reason.
Try a similar already-available tool.
Ask the Installer Agent to check whether an alternative tool can be installed.
Ask the user for approval before installing, unless trusted auto-install is enabled.
Retry the original task with the alternative tool.
Optionally retry the same task with a fallback model/provider.
Log all fallback actions clearly.

Example cases

Search tool failure

If Tavily fails, the agent could try:

DuckDuckGo
Google
Searxng
Perplexity
browser

Instead of stopping immediately.

Missing CLI tool

If the agent tries to use a command but it is missing, for example:

subfinder: command not found

Then the agent could:

Detect that the tool is missing.
Ask the Installer Agent for an alternative or install plan.
Ask the user:

The required tool `subfinder` is not installed. I can install it or try an alternative such as amass. Do you want me to continue?

Install only if allowed.
Retry the original task.

Model/tool-call failure

If the current model repeatedly fails to produce valid tool calls, PentAGI could retry the same chain with a configured fallback model.

Example:

primary model: local Qwen / Ollama
fallback model: OpenAI / Anthropic / Gemini / Custom provider

Suggested config

fallback:
  enabled: true

  tool_fallback:
    enabled: true
    max_attempts: 3
    try_similar_existing_tools: true

  install_fallback:
    enabled: true
    require_user_approval: true
    allow_auto_install: false
    trusted_sources_only: true
    allowlist:
      - apt
      - pip
      - go install
      - official_release
    denylist:
      - unknown_curl_pipe_shell
      - untrusted_repositories

  model_fallback:
    enabled: true
    max_attempts: 2
    fallback_order:
      - primary_agent
      - custom
      - openai
      - anthropic
      - gemini

  logging:
    log_fallback_reason: true
    log_selected_alternative: true
    log_install_commands: true

Possible implementation idea

This could be added around the current retry/error handling path.

Possible places:

tool execution retry path
repeated tool-call detection
reflector / caller-reflector path
execution monitor / mentor path
Installer Agent workflow
provider/model selection logic

A possible flow:

tool execution fails
  -> normal retry
  -> tool-call argument fixer
  -> if still failing, fallback manager checks:
       - is there another existing tool with same capability?
       - can Installer Agent safely install a missing tool?
       - should another model/provider retry this chain?
  -> retry task with fallback
  -> only then fail/stop if fallback also fails

Safety requirements

Because this is a pentesting platform, auto-install should be safe and controlled.

Recommended rules:

disabled by default
require user approval by default
only install from trusted/allowlisted sources
log exact commands before/after install
never run random install commands from untrusted web pages
allow admins to disable install fallback completely
keep fallback attempts bounded to avoid infinite loops

Why this would help

This would make PentAGI more resilient for real autonomous workflows.

Benefits:

fewer stopped flows due to one broken tool
better support for local/open-source models
better recovery when a CLI tool is missing
better use of existing Installer Agent
better use of existing multiple search tools
better use of existing multi-provider model support
less manual intervention from the user

Acceptance criteria

A failed tool can trigger a similar-tool fallback when enabled.
A missing command can trigger Installer Agent suggestion/approval flow.
A model failure can retry with configured fallback model/provider.
Fallback attempts are limited.
All fallback actions are visible in logs/UI.
Auto-install is disabled unless explicitly enabled.
Untrusted tool installation is blocked by default.

Technical Details

No response

Designs and Mockups

No response

Alternative Solutions

No response

Verification

I have checked that this enhancement hasn't been already proposed
This enhancement aligns with PentAGI's goal of autonomous penetration testing
I have considered the security implications of this enhancement
I have provided clear use cases and benefits

Originally created by @Akalanka1337 on GitHub (Jun 6, 2026). Original GitHub issue: https://github.com/vxcontrol/pentagi/issues/341 Originally assigned to: @asdek on GitHub. ### Target Component AI Agents (Researcher/Developer/Executor) ### Enhancement Description ### Summary I would like to request a new fallback feature for PentAGI. Currently, when an agent fails to execute a tool, repeats the same tool, reaches retry limits, or the selected model cannot produce a valid tool call, PentAGI retries and may use the reflector/mentor logic. But after the limit is reached, the flow can still fail or stop. Can we add an optional fallback mode where the agent tries another safe path before stopping? ### Current behavior I noticed From the source/docs, PentAGI already has: * retry logic for agent calls and tool execution * repeated tool-call detection * reflector agent for fixing bad/no tool calls * execution monitor / mentor logic * Installer Agent for environment/tool setup * multiple search tools like browser, Google, DuckDuckGo, Tavily, Perplexity, Searxng, etc. * multiple LLM provider support This means the system already has most building blocks needed for fallback. ### Requested behavior Add a configurable fallback layer that runs before the task/subtask is marked failed. When enabled, if a tool or model fails repeatedly, the agent should be able to: 1. Detect the failure reason. 2. Try a similar already-available tool. 3. Ask the Installer Agent to check whether an alternative tool can be installed. 4. Ask the user for approval before installing, unless trusted auto-install is enabled. 5. Retry the original task with the alternative tool. 6. Optionally retry the same task with a fallback model/provider. 7. Log all fallback actions clearly. ### Example cases #### Search tool failure If Tavily fails, the agent could try: * DuckDuckGo * Google * Searxng * Perplexity * browser Instead of stopping immediately. #### Missing CLI tool If the agent tries to use a command but it is missing, for example: ```bash subfinder: command not found ``` Then the agent could: 1. Detect that the tool is missing. 2. Ask the Installer Agent for an alternative or install plan. 3. Ask the user: ```text The required tool `subfinder` is not installed. I can install it or try an alternative such as amass. Do you want me to continue? ``` 4. Install only if allowed. 5. Retry the original task. #### Model/tool-call failure If the current model repeatedly fails to produce valid tool calls, PentAGI could retry the same chain with a configured fallback model. Example: * primary model: local Qwen / Ollama * fallback model: OpenAI / Anthropic / Gemini / Custom provider ### Suggested config ```yaml fallback: enabled: true tool_fallback: enabled: true max_attempts: 3 try_similar_existing_tools: true install_fallback: enabled: true require_user_approval: true allow_auto_install: false trusted_sources_only: true allowlist: - apt - pip - go install - official_release denylist: - unknown_curl_pipe_shell - untrusted_repositories model_fallback: enabled: true max_attempts: 2 fallback_order: - primary_agent - custom - openai - anthropic - gemini logging: log_fallback_reason: true log_selected_alternative: true log_install_commands: true ``` ### Possible implementation idea This could be added around the current retry/error handling path. Possible places: * tool execution retry path * repeated tool-call detection * reflector / caller-reflector path * execution monitor / mentor path * Installer Agent workflow * provider/model selection logic A possible flow: ```text tool execution fails -> normal retry -> tool-call argument fixer -> if still failing, fallback manager checks: - is there another existing tool with same capability? - can Installer Agent safely install a missing tool? - should another model/provider retry this chain? -> retry task with fallback -> only then fail/stop if fallback also fails ``` ### Safety requirements Because this is a pentesting platform, auto-install should be safe and controlled. Recommended rules: * disabled by default * require user approval by default * only install from trusted/allowlisted sources * log exact commands before/after install * never run random install commands from untrusted web pages * allow admins to disable install fallback completely * keep fallback attempts bounded to avoid infinite loops ### Why this would help This would make PentAGI more resilient for real autonomous workflows. Benefits: * fewer stopped flows due to one broken tool * better support for local/open-source models * better recovery when a CLI tool is missing * better use of existing Installer Agent * better use of existing multiple search tools * better use of existing multi-provider model support * less manual intervention from the user ### Acceptance criteria * A failed tool can trigger a similar-tool fallback when enabled. * A missing command can trigger Installer Agent suggestion/approval flow. * A model failure can retry with configured fallback model/provider. * Fallback attempts are limited. * All fallback actions are visible in logs/UI. * Auto-install is disabled unless explicitly enabled. * Untrusted tool installation is blocked by default. ### Technical Details _No response_ ### Designs and Mockups _No response_ ### Alternative Solutions _No response_ ### Verification - [ ] I have checked that this enhancement hasn't been already proposed - [x] This enhancement aligns with PentAGI's goal of autonomous penetration testing - [x] I have considered the security implications of this enhancement - [ ] I have provided clear use cases and benefits

yindo added the enhancement label 2026-06-06 22:09:11 -04:00

yindo commented

2026-06-06 22:09:11 -04:00

@kehansama commented on GitHub (Jun 6, 2026):

Excellent analysis of the fallback architecture. The multi-layered approach you've outlined — tool fallback → installer fallback → model fallback — maps exactly to what we're seeing as a critical gap in the AI agent ecosystem right now.

One observation from our work on AgentRelay (an agent orchestration & tool-routing middleware): the hardest part isn't the fallback logic, it's the capability discovery layer. Before an agent can fall back to "a similar tool," it needs a unified capability registry that maps abstract capabilities (e.g., "web_search", "code_execution", "file_read") to concrete tool implementations across different providers.

Your config proposal is very close to what we've found works in practice. One addition I'd suggest: a capability similarity graph. For example, if Tavily fails, the system should know that DuckDuckGo and Searxng are "high similarity" (both web search) while Google Search API is "medium similarity" (different rate limits, different auth model). This prevents the fallback from picking a tool that's nominally available but semantically mismatched.

Also worth noting: the model fallback layer you describe is where cross-platform agent relay becomes critical. If the primary model fails tool-call generation, falling back to a different provider requires maintaining session state and tool schema compatibility — something we've solved with an adapter layer in AgentRelay.

Happy to share more implementation details if useful. This is a well-specified enhancement and would make PentAGI significantly more resilient in production.

@kehansama commented on GitHub (Jun 6, 2026): Excellent analysis of the fallback architecture. The multi-layered approach you've outlined — tool fallback → installer fallback → model fallback — maps exactly to what we're seeing as a critical gap in the AI agent ecosystem right now. One observation from our work on **AgentRelay** (an agent orchestration & tool-routing middleware): the hardest part isn't the fallback *logic*, it's the **capability discovery** layer. Before an agent can fall back to "a similar tool," it needs a unified capability registry that maps abstract capabilities (e.g., "web_search", "code_execution", "file_read") to concrete tool implementations across different providers. Your config proposal is very close to what we've found works in practice. One addition I'd suggest: a **capability similarity graph**. For example, if Tavily fails, the system should know that DuckDuckGo and Searxng are "high similarity" (both web search) while Google Search API is "medium similarity" (different rate limits, different auth model). This prevents the fallback from picking a tool that's nominally available but semantically mismatched. Also worth noting: the model fallback layer you describe is where cross-platform agent relay becomes critical. If the primary model fails tool-call generation, falling back to a different provider requires maintaining session state and tool schema compatibility — something we've solved with an adapter layer in AgentRelay. Happy to share more implementation details if useful. This is a well-specified enhancement and would make PentAGI significantly more resilient in production.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: vxcontrol/pentagi#110