vxcontrol/pentagi
1
0
Fork 0
mirror of https://github.com/vxcontrol/pentagi.git synced 2026-07-20 23:57:11 -04:00
Code Issues 45 Packages Projects Releases 9 Wiki Activity

[PR #17] [MERGED] Add ctester utility for testing LLM agents #122

New Issue
Closed
opened 2026-06-06 22:09:16 -04:00 by yindo · 0 comments
yindo commented 2026-06-06 22:09:16 -04:00
Owner
Copy Link
Copy Source

📋 Pull Request Information

Original PR: https://github.com/vxcontrol/pentagi/pull/17
Author: @asdek
Created: 3/24/2025
Status: ✅ Merged
Merged: 3/24/2025
Merged by: @asdek

Base: master ← Head: 14-enhancement-openrouter-configuration

📝 Commits (1)

  • 7da46ef feat: added ctester utility for testing LLM agents, updated Dockerfile to build ctester and added provider configuration files, updated README.md with instructions for using ctester

📊 Changes

17 files changed (+2978 additions, -37 deletions)

View changed files

📝 .vscode/launch.json (+18 -0)
📝 Dockerfile (+10 -1)
📝 README.md (+142 -0)
➕ backend/cmd/ctester/main.go (+248 -0)
➕ backend/cmd/ctester/models.go (+109 -0)
➕ backend/cmd/ctester/report.go (+208 -0)
➕ backend/cmd/ctester/tests.go (+833 -0)
➕ backend/cmd/ctester/utils.go (+120 -0)
📝 backend/go.mod (+1 -1)
📝 backend/go.sum (+2 -2)
📝 backend/pkg/providers/provider/wrapper.go (+48 -5)
➕ backend/pkg/queue/queue.go (+209 -0)
➕ backend/pkg/queue/queue_test.go (+228 -0)
➕ examples/configs/deepinfra.provider.yml (+84 -0)
📝 examples/configs/openrouter.provider.yml (+26 -28)
➕ examples/tests/deepinfra-report.md (+346 -0)
➕ examples/tests/openrouter-report.md (+346 -0)

📄 Description

Description of the Change

This PR introduces the ctester utility for testing LLM agents. The utility enables developers to validate and optimize LLM agent performance through automated testing.

Problem

The project lacked a dedicated utility for testing LLM agents, making it difficult to validate their functionality and optimize model selection.

Solution

  • Added the ctester utility with comprehensive test suites for various agent types
  • Implemented parallel testing capabilities for improved efficiency
  • Added detailed reporting with success rates and latency metrics
  • Created flexible configuration options for testing different providers

Closes #14

Type of Change

  • 🚀 New feature (non-breaking change which adds functionality)
  • 📚 Documentation update
  • 🔧 Configuration change

Areas Affected

  • Core Services (Frontend UI/Backend API)
  • AI Agents (Researcher/Developer/Executor)
  • External Integrations (LLM/Search APIs)
  • Documentation
  • Infrastructure/DevOps

Testing and Verification

Test Steps

  1. Build the Docker image with the updated Dockerfile
  2. Run ctester with various agent types
  3. Review generated reports for performance metrics

Documentation Updates

  • README.md updates with detailed instructions for:
    • Using the utility as a developer (local Go environment)
    • Using the utility as an external user (Docker)
    • Available command-line options
    • Example provider configurations

Deployment Notes

No special considerations for deployment; the utility is built during Docker image creation.

Additional Notes

This utility will help improve LLM agent reliability by identifying optimal model configurations for different agent types.

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/vxcontrol/pentagi/pull/17 **Author:** [@asdek](https://github.com/asdek) **Created:** 3/24/2025 **Status:** ✅ Merged **Merged:** 3/24/2025 **Merged by:** [@asdek](https://github.com/asdek) **Base:** `master` ← **Head:** `14-enhancement-openrouter-configuration` --- ### 📝 Commits (1) - [`7da46ef`](https://github.com/vxcontrol/pentagi/commit/7da46efb18bedc5b01fc1b70e48d955c82f28085) feat: added ctester utility for testing LLM agents, updated Dockerfile to build ctester and added provider configuration files, updated README.md with instructions for using ctester ### 📊 Changes **17 files changed** (+2978 additions, -37 deletions) <details> <summary>View changed files</summary> 📝 `.vscode/launch.json` (+18 -0) 📝 `Dockerfile` (+10 -1) 📝 `README.md` (+142 -0) ➕ `backend/cmd/ctester/main.go` (+248 -0) ➕ `backend/cmd/ctester/models.go` (+109 -0) ➕ `backend/cmd/ctester/report.go` (+208 -0) ➕ `backend/cmd/ctester/tests.go` (+833 -0) ➕ `backend/cmd/ctester/utils.go` (+120 -0) 📝 `backend/go.mod` (+1 -1) 📝 `backend/go.sum` (+2 -2) 📝 `backend/pkg/providers/provider/wrapper.go` (+48 -5) ➕ `backend/pkg/queue/queue.go` (+209 -0) ➕ `backend/pkg/queue/queue_test.go` (+228 -0) ➕ `examples/configs/deepinfra.provider.yml` (+84 -0) 📝 `examples/configs/openrouter.provider.yml` (+26 -28) ➕ `examples/tests/deepinfra-report.md` (+346 -0) ➕ `examples/tests/openrouter-report.md` (+346 -0) </details> ### 📄 Description ### Description of the Change This PR introduces the `ctester` utility for testing LLM agents. The utility enables developers to validate and optimize LLM agent performance through automated testing. #### Problem The project lacked a dedicated utility for testing LLM agents, making it difficult to validate their functionality and optimize model selection. #### Solution - Added the `ctester` utility with comprehensive test suites for various agent types - Implemented parallel testing capabilities for improved efficiency - Added detailed reporting with success rates and latency metrics - Created flexible configuration options for testing different providers Closes #14 ### Type of Change - [x] 🚀 New feature (non-breaking change which adds functionality) - [x] 📚 Documentation update - [x] 🔧 Configuration change ### Areas Affected - [x] Core Services (Frontend UI/Backend API) - [x] AI Agents (Researcher/Developer/Executor) - [x] External Integrations (LLM/Search APIs) - [x] Documentation - [x] Infrastructure/DevOps ### Testing and Verification #### Test Steps 1. Build the Docker image with the updated Dockerfile 2. Run `ctester` with various agent types 3. Review generated reports for performance metrics ### Documentation Updates - [x] README.md updates with detailed instructions for: - Using the utility as a developer (local Go environment) - Using the utility as an external user (Docker) - Available command-line options - Example provider configurations ### Deployment Notes No special considerations for deployment; the utility is built during Docker image creation. ### Additional Notes This utility will help improve LLM agent reliability by identifying optimal model configurations for different agent types. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
yindo added the pull-request label 2026-06-06 22:09:16 -04:00
yindo closed this issue 2026-06-06 22:09:17 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
enhancement
help wanted
pull-request
Mirrored from GitHub Pull Request
 question
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: vxcontrol/pentagi#122
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?