vxcontrol/pentagi
1
0
Fork 0
mirror of https://github.com/vxcontrol/pentagi.git synced 2026-07-20 12:37:04 -04:00
Code Issues 45 Packages Projects Releases 9 Wiki Activity

[PR #53] [MERGED] feat: add SearXNG meta search engine tool integration #137

New Issue
Closed
opened 2026-06-06 22:09:21 -04:00 by yindo · 0 comments
yindo commented 2026-06-06 22:09:21 -04:00
Owner
Copy Link
Copy Source

📋 Pull Request Information

Original PR: https://github.com/vxcontrol/pentagi/pull/53
Author: @stoykovstoyk
Created: 8/21/2025
Status: ✅ Merged
Merged: 8/24/2025
Merged by: @asdek

Base: master ← Head: master

📝 Commits (4)

  • 9c259c3 feat: add SearXNG meta search engine tool integration
  • 19eaffa fix: add missing SEARXNG environment variables to docker-compose.yml
  • a23d881 fix(searxng): disable SEARXNG_URL by default in .env.example
  • 36c7fcc fix(searxng): update logging to use agent context and follow naming conventions

📊 Changes

14 files changed (+811 additions, -4 deletions)

View changed files

📝 .env.example (+8 -0)
📝 EULA.md (+1 -1)
📝 README.md (+4 -2)
📝 backend/cmd/ftester/mocks/tools.go (+33 -0)
📝 backend/cmd/ftester/worker/args.go (+1 -0)
📝 backend/cmd/ftester/worker/executor.go (+16 -0)
➕ backend/migrations/sql/20250821_123456_add_searxng_search_type.sql (+50 -0)
📝 backend/pkg/config/config.go (+8 -0)
📝 backend/pkg/database/models.go (+1 -0)
📝 backend/pkg/tools/registry.go (+11 -1)
➕ backend/pkg/tools/searxng.go (+304 -0)
➕ backend/pkg/tools/searxng_test.go (+329 -0)
📝 backend/pkg/tools/tools.go (+39 -0)
📝 docker-compose.yml (+6 -0)

📄 Description

Description of the Change

Problem

PentAGI lacked integration with Searxng, a popular privacy-focused meta search engine that aggregates results from multiple search engines. Users requested the ability to add Searxng as an additional search option alongside existing providers like Google, DuckDuckGo, Tavily, and Perplexity.

Solution

Implemented comprehensive Searxng search tool integration with the following key components:

  • Created backend/pkg/tools/searxng.go with full tool implementation supporting configurable search parameters
  • Added Searxng environment variables to the configuration system for flexible deployment
  • Implemented proper HTTP client with timeout and proxy support for reliable API communication
  • Added comprehensive error handling and logging for debugging and monitoring
  • Created database integration for search result tracking and analytics
  • Developed extensive unit tests with mock servers to ensure reliability
  • Updated documentation to reflect the new search capability

Closes #

Type of Change

  • 🐛 Bug fix (non-breaking change which fixes an issue)
  • 🚀 New feature (non-breaking change which adds functionality)
  • 💥 Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • 📚 Documentation update
  • 🔧 Configuration change
  • 🧪 Test update
  • 🛡️ Security update

Areas Affected

  • Core Services (Frontend UI/Backend API)
  • AI Agents (Researcher/Developer/Executor)
  • Security Tools Integration
  • Memory System (Vector Store/Knowledge Base)
  • Monitoring Stack (Grafana/OpenTelemetry)
  • Analytics Platform (Langfuse)
  • External Integrations (LLM/Search APIs)
  • Documentation
  • Infrastructure/DevOps

Testing and Verification

Test Configuration

PentAGI Version: Latest master branch
Docker Version: 24.0.0+
Host OS: Linux
LLM Provider: Not required for this feature
Enabled Features: [ftester for testing, mock mode]

Test Steps

  1. Build the backend: cd backend && go build ./...
  2. Run unit tests: cd backend && go test -v ./pkg/tools/
  3. Verify all tests pass including edge cases and error scenarios

Test Results

All unit tests pass successfully:

  • TestNewSearxngTool: Tool creation and initialization
  • TestSearxngToolIsAvailable: Availability checking logic
  • TestSearxngToolHandleWithInvalidArgs: Invalid argument handling
  • TestSearxngToolHandleWithValidArgs: Valid search execution with mock server
  • TestSearxngToolHandleWithServerError: Server error handling
  • TestSearxngToolHandleWithInvalidURL: Invalid URL handling
  • TestSearxngToolHandleWithNoResults: No results scenarios
  • Build verification shows no compilation errors and all existing functionality remains intact.

Security Considerations

The Searxng integration enhances security by providing users with a privacy-focused search option that can be self-hosted. Key security considerations:

  • Network Security: The tool supports proxy configuration for network isolation
  • Input Validation: All search queries are validated before API calls
  • Error Handling: Comprehensive error handling prevents information leakage
  • No New Dependencies: Uses existing HTTP client and database libraries
  • Self-Hostable Option: Users can run their own Searxng instance for maximum privacy
  • Configuration Security: Environment variables follow existing security patterns

Performance Impact

The Searxng tool is designed with performance in mind:

  • Configurable Timeouts: Default 30-second timeout prevents hanging requests
  • Connection Reuse: Uses standard HTTP client for efficient connection management
  • Proxy Support: Optional proxy configuration for enterprise environments
  • Result Caching: Database integration allows for result tracking and potential caching
  • Lightweight Implementation: Minimal overhead, follows existing search tool patterns
  • Async Processing: Non-blocking design that doesn't impact other system components
  • Performance testing with mock servers confirmed sub-second response times for typical search queries.

Documentation Updates

  • README.md updates
  • API documentation updates
  • Configuration documentation updates
  • GraphQL schema updates
  • Other: EULA.md updates

Deployment Notes

New Environment Variables

# Required: Searxng instance URL
SEARXNG_URL=http://localhost:8080

# Optional: Search configuration
SEARXNG_CATEGORIES=general          # Default: "general"
SEARXNG_LANGUAGE=en                # Default: "en" 
SEARXNG_SAFESearch=0               # Default: "0"
SEARXNG_TIME_RANGE=                # Optional: time range filter
SEARXNG_PROXY_URL=                 # Optional: proxy URL

Checklist

Code Quality

  • My code follows the project's coding standards
  • I have added/updated necessary documentation
  • I have added tests to cover my changes
  • All new and existing tests pass
  • I have run go fmt and go vet (for Go code)
  • I have run npm run lint (for TypeScript/JavaScript code)

Security

  • I have considered security implications
  • Changes maintain or improve the security model
  • Sensitive information has been properly handled

Compatibility

  • Changes are backward compatible
  • Breaking changes are clearly marked and documented
  • Dependencies are properly updated

Documentation

  • Documentation is clear and complete
  • Comments are added for non-obvious code
  • API changes are documented

Additional Notes

The Searxng tool implementation follows the exact same patterns as existing search tools (Google, DuckDuckGo, Tavily, etc.) ensuring consistency in the codebase. The feature is optional and only activates when SEARXNG_URL is configured, making it safe for production deployment.

Key implementation highlights:

  • Uses existing SearchLogProvider interface for database integration
  • Follows established error handling and logging patterns
  • Includes comprehensive mock support for testing
  • Supports all Searxng API parameters with sensible defaults
  • Maintains compatibility with existing tool registry system

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/vxcontrol/pentagi/pull/53 **Author:** [@stoykovstoyk](https://github.com/stoykovstoyk) **Created:** 8/21/2025 **Status:** ✅ Merged **Merged:** 8/24/2025 **Merged by:** [@asdek](https://github.com/asdek) **Base:** `master` ← **Head:** `master` --- ### 📝 Commits (4) - [`9c259c3`](https://github.com/vxcontrol/pentagi/commit/9c259c399fd24ea5d3f4e6036b5332643edcf5b9) feat: add SearXNG meta search engine tool integration - [`19eaffa`](https://github.com/vxcontrol/pentagi/commit/19eaffaae2b75faf94fbe65dd5dd354ab0dd89d3) fix: add missing SEARXNG environment variables to docker-compose.yml - [`a23d881`](https://github.com/vxcontrol/pentagi/commit/a23d881d46d3b39f85587409e7117331cafaf265) fix(searxng): disable SEARXNG_URL by default in .env.example - [`36c7fcc`](https://github.com/vxcontrol/pentagi/commit/36c7fcca1a6d66572da5722c161c01bf244b85dd) fix(searxng): update logging to use agent context and follow naming conventions ### 📊 Changes **14 files changed** (+811 additions, -4 deletions) <details> <summary>View changed files</summary> 📝 `.env.example` (+8 -0) 📝 `EULA.md` (+1 -1) 📝 `README.md` (+4 -2) 📝 `backend/cmd/ftester/mocks/tools.go` (+33 -0) 📝 `backend/cmd/ftester/worker/args.go` (+1 -0) 📝 `backend/cmd/ftester/worker/executor.go` (+16 -0) ➕ `backend/migrations/sql/20250821_123456_add_searxng_search_type.sql` (+50 -0) 📝 `backend/pkg/config/config.go` (+8 -0) 📝 `backend/pkg/database/models.go` (+1 -0) 📝 `backend/pkg/tools/registry.go` (+11 -1) ➕ `backend/pkg/tools/searxng.go` (+304 -0) ➕ `backend/pkg/tools/searxng_test.go` (+329 -0) 📝 `backend/pkg/tools/tools.go` (+39 -0) 📝 `docker-compose.yml` (+6 -0) </details> ### 📄 Description <!-- Thank you for your contribution to PentAGI! Please fill out this template completely to help us review your changes effectively. Any PR that does not include enough information may be closed at maintainers' discretion. --> ### Description of the Change <!-- We must be able to understand the design of your change from this description. Please provide as much detail as possible. --> #### Problem PentAGI lacked integration with Searxng, a popular privacy-focused meta search engine that aggregates results from multiple search engines. Users requested the ability to add Searxng as an additional search option alongside existing providers like Google, DuckDuckGo, Tavily, and Perplexity. #### Solution Implemented comprehensive Searxng search tool integration with the following key components: - Created backend/pkg/tools/searxng.go with full tool implementation supporting configurable search parameters - Added Searxng environment variables to the configuration system for flexible deployment - Implemented proper HTTP client with timeout and proxy support for reliable API communication - Added comprehensive error handling and logging for debugging and monitoring - Created database integration for search result tracking and analytics - Developed extensive unit tests with mock servers to ensure reliability - Updated documentation to reflect the new search capability <!-- Enter any applicable Issue number(s) here that will be closed/resolved by this PR. --> Closes # ### Type of Change <!-- Mark with an `x` all options that apply --> - [ ] 🐛 Bug fix (non-breaking change which fixes an issue) - [x] 🚀 New feature (non-breaking change which adds functionality) - [ ] 💥 Breaking change (fix or feature that would cause existing functionality to not work as expected) - [x] 📚 Documentation update - [ ] 🔧 Configuration change - [x] 🧪 Test update - [ ] 🛡️ Security update ### Areas Affected <!-- Mark with an `x` all components that are affected --> - [ ] Core Services (Frontend UI/Backend API) - [ ] AI Agents (Researcher/Developer/Executor) - [ ] Security Tools Integration - [ ] Memory System (Vector Store/Knowledge Base) - [ ] Monitoring Stack (Grafana/OpenTelemetry) - [ ] Analytics Platform (Langfuse) - [x] External Integrations (LLM/Search APIs) - [x] Documentation - [ ] Infrastructure/DevOps ### Testing and Verification <!-- Please describe the tests that you ran to verify your changes and provide instructions so we can reproduce. --> #### Test Configuration ```yaml PentAGI Version: Latest master branch Docker Version: 24.0.0+ Host OS: Linux LLM Provider: Not required for this feature Enabled Features: [ftester for testing, mock mode] ``` #### Test Steps 1. Build the backend: cd backend && go build ./... 2. Run unit tests: cd backend && go test -v ./pkg/tools/ 3. Verify all tests pass including edge cases and error scenarios #### Test Results All unit tests pass successfully: - [x] TestNewSearxngTool: Tool creation and initialization - [x] TestSearxngToolIsAvailable: Availability checking logic - [x] TestSearxngToolHandleWithInvalidArgs: Invalid argument handling - [x] TestSearxngToolHandleWithValidArgs: Valid search execution with mock server - [x] TestSearxngToolHandleWithServerError: Server error handling - [x] TestSearxngToolHandleWithInvalidURL: Invalid URL handling - [x] TestSearxngToolHandleWithNoResults: No results scenarios - [x] Build verification shows no compilation errors and all existing functionality remains intact. ### Security Considerations The Searxng integration enhances security by providing users with a privacy-focused search option that can be self-hosted. Key security considerations: - Network Security: The tool supports proxy configuration for network isolation - Input Validation: All search queries are validated before API calls - Error Handling: Comprehensive error handling prevents information leakage - No New Dependencies: Uses existing HTTP client and database libraries - Self-Hostable Option: Users can run their own Searxng instance for maximum privacy - Configuration Security: Environment variables follow existing security patterns ### Performance Impact The Searxng tool is designed with performance in mind: - Configurable Timeouts: Default 30-second timeout prevents hanging requests - Connection Reuse: Uses standard HTTP client for efficient connection management - Proxy Support: Optional proxy configuration for enterprise environments - Result Caching: Database integration allows for result tracking and potential caching - Lightweight Implementation: Minimal overhead, follows existing search tool patterns - Async Processing: Non-blocking design that doesn't impact other system components - Performance testing with mock servers confirmed sub-second response times for typical search queries. ### Documentation Updates <!-- Note any documentation changes required by this PR --> - [x] README.md updates - [ ] API documentation updates - [x] Configuration documentation updates - [ ] GraphQL schema updates - [x] Other: <!-- specify --> EULA.md updates ### Deployment Notes New Environment Variables ``` # Required: Searxng instance URL SEARXNG_URL=http://localhost:8080 # Optional: Search configuration SEARXNG_CATEGORIES=general # Default: "general" SEARXNG_LANGUAGE=en # Default: "en" SEARXNG_SAFESearch=0 # Default: "0" SEARXNG_TIME_RANGE= # Optional: time range filter SEARXNG_PROXY_URL= # Optional: proxy URL ``` ### Checklist <!--- Go over all the following points, and put an `x` in all the boxes that apply. --> #### Code Quality - [x] My code follows the project's coding standards - [x] I have added/updated necessary documentation - [x] I have added tests to cover my changes - [x] All new and existing tests pass - [x] I have run `go fmt` and `go vet` (for Go code) - [ ] I have run `npm run lint` (for TypeScript/JavaScript code) #### Security - [x] I have considered security implications - [x] Changes maintain or improve the security model - [x] Sensitive information has been properly handled #### Compatibility - [x] Changes are backward compatible - [ ] Breaking changes are clearly marked and documented - [x] Dependencies are properly updated #### Documentation - [x] Documentation is clear and complete - [x] Comments are added for non-obvious code - [x] API changes are documented ### Additional Notes The Searxng tool implementation follows the exact same patterns as existing search tools (Google, DuckDuckGo, Tavily, etc.) ensuring consistency in the codebase. The feature is optional and only activates when SEARXNG_URL is configured, making it safe for production deployment. Key implementation highlights: - Uses existing SearchLogProvider interface for database integration - Follows established error handling and logging patterns - Includes comprehensive mock support for testing - Supports all Searxng API parameters with sensible defaults - Maintains compatibility with existing tool registry system --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
yindo added the pull-request label 2026-06-06 22:09:21 -04:00
yindo closed this issue 2026-06-06 22:09:21 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
enhancement
help wanted
pull-request
Mirrored from GitHub Pull Request
 question
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: vxcontrol/pentagi#137
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?