[GH-ISSUE #46] [Enhancement] Switch from OpenAI to Claude for More Reliable Model Access and Stronger Reasoning #22

New Issue

2026-06-06T22:08:03-04:00

yindo commented

2026-06-06 22:08:03 -04:00

Originally created by @iain0901 on GitHub (Jun 21, 2025).
Original GitHub issue: https://github.com/vxcontrol/pentagi/issues/46

Originally assigned to: @asdek on GitHub.

Target Component

External Integrations (LLM/Search APIs)

Enhancement Description

I've been genuinely impressed by the continued progress of this project. However, it's unfortunate that I'm unable to access the latest features due to persistent issues with OpenAI's identity verification system, which has prevented me from using the o3 model. This appears to be a growing issue, as more users have reported similar problems online.

Given this, I highly recommend considering a switch to Claude as the default or an alternative provider for all AI model services. Claude has demonstrated stronger capabilities in both programming and reasoning tasks, and may offer a more reliable and powerful experience for users of PentAGI.

Technical Details

Implementation Approach:

Architecture Changes: Abstract the model provider layer to support pluggable backends (e.g., Claude, OpenAI, Cohere).

New Components: Integration with Anthropic Claude API (token management, response parsing, error handling).

Dependencies: Anthropic Claude SDK or custom API wrapper; new environment variables for key management.

Integration Points:

AI Agents: Claude models can replace or complement existing agents, with slight adjustments to prompt tuning or system messages.

Memory System: No changes required unless Claude’s token/response format introduces constraints.

Monitoring: Add metrics to distinguish Claude-based inference latency, error rates, and output quality.

Security Considerations:

Secure storage of Claude API keys (e.g., Vault, environment variables).

Ensure compliance with Claude’s usage policy for sensitive data.

Log filtering and redaction if using Claude for audit-critical tasks.

Designs and Mockups

sequenceDiagram
User->>PentAGI: Request Model Inference
PentAGI->>Model Router: Select Claude as Provider
Model Router->>Claude API: Send Prompt
Claude API-->>Model Router: Return Output
Model Router-->>PentAGI: Return Claude-Enhanced Response

Alternative Solutions

Retry OpenAI Integration Fixes

Pros: Minimal change to current setup; no external migration.

Cons: Issue appears widespread and persistent; not within PentAGI’s control.

Add Claude as a Secondary Fallback

Pros: Increased redundancy and reliability; allows performance comparison.
Cons: More complex routing logic; increased cost and maintenance burden.
Reason for Preferred Solution:
Using Claude as the primary model provider offers immediate reliability improvements and performance benefits, especially in programming and reasoning, with minimal impact on existing system design if structured properly.

I know I can use custom LLM, but the underlying layer of this project seems to use openai's models.

Thank you for your contribution

Verification

I have checked that this enhancement hasn't been already proposed
This enhancement aligns with PentAGI's goal of autonomous penetration testing
I have considered the security implications of this enhancement
I have provided clear use cases and benefits

Originally created by @iain0901 on GitHub (Jun 21, 2025). Original GitHub issue: https://github.com/vxcontrol/pentagi/issues/46 Originally assigned to: @asdek on GitHub. ### Target Component External Integrations (LLM/Search APIs) ### Enhancement Description I've been genuinely impressed by the continued progress of this project. However, it's unfortunate that I'm unable to access the latest features due to persistent issues with OpenAI's identity verification system, which has prevented me from using the o3 model. This appears to be a growing issue, as more users have reported similar problems online. Given this, I highly recommend considering a switch to Claude as the default or an alternative provider for all AI model services. Claude has demonstrated stronger capabilities in both programming and reasoning tasks, and may offer a more reliable and powerful experience for users of PentAGI. ### Technical Details Implementation Approach: Architecture Changes: Abstract the model provider layer to support pluggable backends (e.g., Claude, OpenAI, Cohere). New Components: Integration with Anthropic Claude API (token management, response parsing, error handling). Dependencies: Anthropic Claude SDK or custom API wrapper; new environment variables for key management. Integration Points: AI Agents: Claude models can replace or complement existing agents, with slight adjustments to prompt tuning or system messages. Memory System: No changes required unless Claude’s token/response format introduces constraints. Monitoring: Add metrics to distinguish Claude-based inference latency, error rates, and output quality. Security Considerations: Secure storage of Claude API keys (e.g., Vault, environment variables). Ensure compliance with Claude’s usage policy for sensitive data. Log filtering and redaction if using Claude for audit-critical tasks. ### Designs and Mockups sequenceDiagram User->>PentAGI: Request Model Inference PentAGI->>Model Router: Select Claude as Provider Model Router->>Claude API: Send Prompt Claude API-->>Model Router: Return Output Model Router-->>PentAGI: Return Claude-Enhanced Response ### Alternative Solutions Retry OpenAI Integration Fixes Pros: Minimal change to current setup; no external migration. Cons: Issue appears widespread and persistent; not within PentAGI’s control. Add Claude as a Secondary Fallback Pros: Increased redundancy and reliability; allows performance comparison. Cons: More complex routing logic; increased cost and maintenance burden. Reason for Preferred Solution: Using Claude as the primary model provider offers immediate reliability improvements and performance benefits, especially in programming and reasoning, with minimal impact on existing system design if structured properly. I know I can use custom LLM, but the underlying layer of this project seems to use openai's models. Thank you for your contribution ### Verification - [x] I have checked that this enhancement hasn't been already proposed - [x] This enhancement aligns with PentAGI's goal of autonomous penetration testing - [x] I have considered the security implications of this enhancement - [x] I have provided clear use cases and benefits

yindo added the enhancement label 2026-06-06 22:08:03 -04:00

yindo closed this issue

2026-06-06 22:08:03 -04:00

yindo commented

2026-06-06 22:08:04 -04:00

@asdek commented on GitHub (Jun 21, 2025):

Hello @iain0901

Thanks for the detailed feature request and the technical breakdown. Really appreciate the thorough analysis.

Claude/Anthropic is actually already supported in PentAGI - you just need to set your ANTHROPIC_API_KEY in the env file and you're good to go (see anthropic.go).

However, from my testing experience, only 2 Anthropic models are really suitable: claude-sonnet-4 as the workhorse for most tasks, and claude-3-5-haiku mainly for summarization/data structuring. The challenge is that sonnet-4 ends up being used for almost everything, which gets expensive fast - I couldn't solve medium complexity tasks for less than $10 in my tests. So you really need to understand why you want Claude specifically - it's excellent for the most complex scenarios where price doesn't matter, but for simpler tasks you'd want something cheaper.

For mid-tier tasks, I've put together tested configs for alternative providers - personally love OpenRouter because of their huge model variety where you can build custom configs for simple tasks (~~$1-2) and medium ones (~~$3-5). Important thing is to use the ctester utility when creating your own config to make sure the models actually work properly with PentAGI - costs about 10 cents or less but gives you confidence in the results.

Regarding the architectural changes you mentioned - I've been thinking about that too and plan to eventually build an AI agent framework for any tasks, but that's future work. Right now expansion happens through alternative custom configurations for OpenAI-compatible backends.

In the near term roadmap:

UI for customizing prompts
MCP support for extending agent capabilities

You can check out the pre-configured setups and test reports here:

Also worth checking issue #19 for more details on alternative providers.

@asdek commented on GitHub (Jun 21, 2025): Hello @iain0901 Thanks for the detailed feature request and the technical breakdown. Really appreciate the thorough analysis. Claude/Anthropic is actually already supported in PentAGI - you just need to set your `ANTHROPIC_API_KEY` in the env file and you're good to go (see [anthropic.go](https://github.com/vxcontrol/pentagi/blob/master/backend/pkg/providers/anthropic/anthropic.go)). However, from my testing experience, only 2 Anthropic models are really suitable: claude-sonnet-4 as the workhorse for most tasks, and claude-3-5-haiku mainly for summarization/data structuring. The challenge is that sonnet-4 ends up being used for almost everything, which gets expensive fast - I couldn't solve medium complexity tasks for less than $10 in my tests. So you really need to understand *why* you want Claude specifically - it's excellent for the most complex scenarios where price doesn't matter, but for simpler tasks you'd want something cheaper. For mid-tier tasks, I've put together tested configs for alternative providers - personally love OpenRouter because of their huge model variety where you can build custom configs for simple tasks (~$1-2) and medium ones (~$3-5). Important thing is to use the `ctester` utility when creating your own config to make sure the models actually work properly with PentAGI - costs about 10 cents or less but gives you confidence in the results. Regarding the architectural changes you mentioned - I've been thinking about that too and plan to eventually build an AI agent framework for any tasks, but that's future work. Right now expansion happens through alternative custom configurations for OpenAI-compatible backends. In the near term roadmap: - UI for customizing prompts - MCP support for extending agent capabilities You can check out the pre-configured setups and test reports here: - [OpenRouter config](https://github.com/vxcontrol/pentagi/blob/master/examples/configs/openrouter.provider.yml) - [Test reports](https://github.com/vxcontrol/pentagi/blob/master/examples/tests/) Also worth checking [issue #19](https://github.com/vxcontrol/pentagi/issues/19#issuecomment-2755840895) for more details on alternative providers.

yindo commented

2026-06-06 22:08:04 -04:00

@asdek commented on GitHub (Jul 1, 2025):

Hello @iain0901

As an interim solution while we work on implementing full UI functionality for custom provider configurations, I've created a custom OpenAI configuration specifically designed to address the verification issues you've mentioned, config.

This configuration maps agent types to models that should be accessible to unverified organizations, using o3-mini instead of the restricted models like o1, o3, and o4-mini, while maintaining optimal performance for each agent type.

To use this:

LLM_SERVER_URL=https://api.openai.com/v1
LLM_SERVER_KEY=your_openai_api_key
LLM_SERVER_MODEL=                                # Leave empty, models are specified in config
LLM_SERVER_CONFIG_PATH=/opt/pentagi/conf/custom-openai.provider.yml
LLM_SERVER_LEGACY_REASONING=true                 # Required for OpenAI reasoning format

Also, you can find instructions here

To get the updated configuration, pull the latest image:

docker compose pull && docker compose up -d

@asdek commented on GitHub (Jul 1, 2025): Hello @iain0901 As an interim solution while we work on implementing full UI functionality for custom provider configurations, I've created a custom OpenAI configuration specifically designed to address the verification issues you've mentioned, [config](https://github.com/vxcontrol/pentagi/blob/master/examples/configs/custom-openai.provider.yml). This configuration maps agent types to models that should be accessible to unverified organizations, using `o3-mini` instead of the restricted models like `o1`, `o3`, and `o4-mini`, while maintaining optimal performance for each agent type. **To use this:** ```bash LLM_SERVER_URL=https://api.openai.com/v1 LLM_SERVER_KEY=your_openai_api_key LLM_SERVER_MODEL= # Leave empty, models are specified in config LLM_SERVER_CONFIG_PATH=/opt/pentagi/conf/custom-openai.provider.yml LLM_SERVER_LEGACY_REASONING=true # Required for OpenAI reasoning format ``` Also, you can find instructions [here](https://github.com/vxcontrol/pentagi/?tab=readme-ov-file#using-openai-with-unverified-organizations) To get the updated configuration, pull the latest image: ```bash docker compose pull && docker compose up -d ```

yindo referenced this issue

2026-06-06 22:09:17 -04:00

[PR #22] [MERGED] Update entrypoint in docker-compose for Langfuse MinIO #123

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: vxcontrol/pentagi#22