vxcontrol/pentagi
1
0
Fork 0
mirror of https://github.com/vxcontrol/pentagi.git synced 2026-07-20 23:57:11 -04:00
Code Issues 45 Packages Projects Releases 9 Wiki Activity

[PR #26] [MERGED] Embedding Providers Implementation #127

New Issue
Closed
opened 2026-06-06 22:09:18 -04:00 by yindo · 0 comments
yindo commented 2026-06-06 22:09:18 -04:00
Owner
Copy Link
Copy Source

📋 Pull Request Information

Original PR: https://github.com/vxcontrol/pentagi/pull/26
Author: @asdek
Created: 3/29/2025
Status: ✅ Merged
Merged: 3/29/2025
Merged by: @asdek

Base: master ← Head: 21-new-embedding-providers

📝 Commits (4)

  • dd753f4 fix: update prompts to clarify memory usage for tools
  • e919884 refactor: replace terminal package imports with pkg/terminal
  • aa230e1 feat: add embedding configuration and provider integration
  • c33d88d feat: add etester utility for embedding management

📊 Changes

34 files changed (+1548 additions, -110 deletions)

View changed files

📝 .env.example (+7 -0)
📝 .vscode/launch.json (+16 -0)
📝 Dockerfile (+4 -0)
📝 README.md (+133 -0)
➕ backend/cmd/etester/flush.go (+42 -0)
➕ backend/cmd/etester/info.go (+203 -0)
➕ backend/cmd/etester/main.go (+131 -0)
➕ backend/cmd/etester/reindex.go (+122 -0)
➕ backend/cmd/etester/search.go (+289 -0)
➕ backend/cmd/etester/test.go (+106 -0)
➕ backend/cmd/etester/tester.go (+76 -0)
📝 backend/cmd/ftester/main.go (+1 -1)
📝 backend/cmd/ftester/mocks/logs.go (+1 -1)
📝 backend/cmd/ftester/mocks/tools.go (+1 -1)
📝 backend/cmd/ftester/worker/executor.go (+23 -13)
📝 backend/cmd/ftester/worker/interactive.go (+1 -1)
📝 backend/cmd/ftester/worker/tester.go (+2 -2)
📝 backend/go.mod (+14 -2)
📝 backend/go.sum (+2 -0)
📝 backend/pkg/config/config.go (+8 -0)

...and 14 more files

📄 Description

Description of the Change

This PR implements comprehensive support for embedding providers and introduces an etester utility for testing and managing embeddings. The implementation adds constructors for Cybertron, Huggingface, Bedrock, and VoyageAI embedding providers, updates the embedding configuration in various files, and provides a new command-line tool for embedding testing and management.

Problem

The project lacked proper support for various embedding providers and tools for testing and debugging embedding functionality. This made it difficult to work with different embedding models and to effectively test and manage embeddings in the system.

Solution

The solution consists of two main components:

  1. Implementation of missing embedding provider constructors in the embedding.go file with support for OpenAI, Ollama, Mistral, Jina, Hugging Face, GoogleAI, VoyageAI
  2. Creation of an etester utility with commands for testing, displaying statistics, flushing documents, and reindexing embeddings

Type of Change

  • 🐛 Bug fix (non-breaking change which fixes an issue)
  • 🚀 New feature (non-breaking change which adds functionality)
  • 💥 Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • 📚 Documentation update
  • 🔧 Configuration change
  • 🧪 Test update
  • 🛡️ Security update

Areas Affected

  • Core Services (Frontend UI/Backend API)
  • AI Agents (Researcher/Developer/Executor)
  • Security Tools Integration
  • Memory System (Vector Store/Knowledge Base)
  • Monitoring Stack (Grafana/OpenTelemetry)
  • Analytics Platform (Langfuse)
  • External Integrations (LLM/Search APIs)
  • Documentation
  • Infrastructure/DevOps

Testing and Verification

Test Configuration

PentAGI Version: latest
Docker Version: 28.0.1
Host OS: Linux/macOS
LLM Provider: [all]
Enabled Features: [Embedding Provider]

Test Steps

  1. Build the etester utility with go build -o etester cmd/etester/main.go
  2. Run ./etester test to verify embedding provider connectivity
  3. Test embedding functionality with various providers (Cybertron, Huggingface, Bedrock, VoyageAI)
  4. Use ./etester info to check embedding statistics
  5. Verify management commands with ./etester flush and ./etester reindex

Test Results

The embedding providers correctly initialize and connect to the respective services. The etester utility successfully performs all operations including testing, statistics gathering, flushing, and reindexing.

Security Considerations

The implementation adds support for external embedding providers which require API keys and credentials. These are handled securely through environment variables and configuration files. No plaintext credentials are stored in the code.

Performance Impact

The new embedding providers offer various performance characteristics depending on the model and service used. Performance testing shows acceptable response times for embedding generation. The etester utility includes verbose output options that can help diagnose performance issues.

Documentation Updates

  • README.md updates
  • API documentation updates
  • Configuration documentation updates
  • GraphQL schema updates
  • Other: .env.example updates with embedding configuration options

Deployment Notes

To use the new embedding providers, users need to:

  1. Set appropriate environment variables for the desired embedding provider
  2. Update configuration files to specify the provider and model
  3. For Docker deployments, the updated Dockerfile includes the etester utility

New environment variables:

  • EMBEDDING_PROVIDER: Type of embedding provider to use
  • EMBEDDING_MODEL: Specific model for the selected provider
  • EMBEDDING_API_KEY: API key for the embedding provider
  • Various provider-specific configuration options

Checklist

Code Quality

  • My code follows the project's coding standards
  • I have added/updated necessary documentation
  • I have added tests to cover my changes
  • All new and existing tests pass
  • I have run go fmt and go vet (for Go code)
  • I have run npm run lint (for TypeScript/JavaScript code)

Security

  • I have considered security implications
  • Changes maintain or improve the security model
  • Sensitive information has been properly handled

Compatibility

  • Changes are backward compatible
  • Breaking changes are clearly marked and documented
  • Dependencies are properly updated

Documentation

  • Documentation is clear and complete
  • Comments are added for non-obvious code
  • API changes are documented

Additional Notes

The etester utility provides a flexible framework for testing and managing embeddings that can be extended with additional functionality in the future. The implementation of multiple embedding providers allows users to choose the most appropriate model for their specific use case and performance requirements.

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/vxcontrol/pentagi/pull/26 **Author:** [@asdek](https://github.com/asdek) **Created:** 3/29/2025 **Status:** ✅ Merged **Merged:** 3/29/2025 **Merged by:** [@asdek](https://github.com/asdek) **Base:** `master` ← **Head:** `21-new-embedding-providers` --- ### 📝 Commits (4) - [`dd753f4`](https://github.com/vxcontrol/pentagi/commit/dd753f4dae76d20348f2af37813f27d74105feba) fix: update prompts to clarify memory usage for tools - [`e919884`](https://github.com/vxcontrol/pentagi/commit/e9198841edd2194b2cc68bc95357c642a24e5c87) refactor: replace terminal package imports with pkg/terminal - [`aa230e1`](https://github.com/vxcontrol/pentagi/commit/aa230e1eb09169e2731242861b26b2654698d35a) feat: add embedding configuration and provider integration - [`c33d88d`](https://github.com/vxcontrol/pentagi/commit/c33d88d275ad399482f1766c09e2d6df1128562c) feat: add etester utility for embedding management ### 📊 Changes **34 files changed** (+1548 additions, -110 deletions) <details> <summary>View changed files</summary> 📝 `.env.example` (+7 -0) 📝 `.vscode/launch.json` (+16 -0) 📝 `Dockerfile` (+4 -0) 📝 `README.md` (+133 -0) ➕ `backend/cmd/etester/flush.go` (+42 -0) ➕ `backend/cmd/etester/info.go` (+203 -0) ➕ `backend/cmd/etester/main.go` (+131 -0) ➕ `backend/cmd/etester/reindex.go` (+122 -0) ➕ `backend/cmd/etester/search.go` (+289 -0) ➕ `backend/cmd/etester/test.go` (+106 -0) ➕ `backend/cmd/etester/tester.go` (+76 -0) 📝 `backend/cmd/ftester/main.go` (+1 -1) 📝 `backend/cmd/ftester/mocks/logs.go` (+1 -1) 📝 `backend/cmd/ftester/mocks/tools.go` (+1 -1) 📝 `backend/cmd/ftester/worker/executor.go` (+23 -13) 📝 `backend/cmd/ftester/worker/interactive.go` (+1 -1) 📝 `backend/cmd/ftester/worker/tester.go` (+2 -2) 📝 `backend/go.mod` (+14 -2) 📝 `backend/go.sum` (+2 -0) 📝 `backend/pkg/config/config.go` (+8 -0) _...and 14 more files_ </details> ### 📄 Description <!-- Thank you for your contribution to PentAGI! Please fill out this template completely to help us review your changes effectively. Any PR that does not include enough information may be closed at maintainers' discretion. --> ### Description of the Change This PR implements comprehensive support for embedding providers and introduces an `etester` utility for testing and managing embeddings. The implementation adds constructors for Cybertron, Huggingface, Bedrock, and VoyageAI embedding providers, updates the embedding configuration in various files, and provides a new command-line tool for embedding testing and management. #### Problem The project lacked proper support for various embedding providers and tools for testing and debugging embedding functionality. This made it difficult to work with different embedding models and to effectively test and manage embeddings in the system. #### Solution The solution consists of two main components: 1. Implementation of missing embedding provider constructors in the `embedding.go` file with support for OpenAI, Ollama, Mistral, Jina, Hugging Face, GoogleAI, VoyageAI 2. Creation of an `etester` utility with commands for testing, displaying statistics, flushing documents, and reindexing embeddings ### Type of Change <!-- Mark with an `x` all options that apply --> - [ ] 🐛 Bug fix (non-breaking change which fixes an issue) - [x] 🚀 New feature (non-breaking change which adds functionality) - [ ] 💥 Breaking change (fix or feature that would cause existing functionality to not work as expected) - [x] 📚 Documentation update - [x] 🔧 Configuration change - [x] 🧪 Test update - [ ] 🛡️ Security update ### Areas Affected <!-- Mark with an `x` all components that are affected --> - [x] Core Services (Frontend UI/Backend API) - [ ] AI Agents (Researcher/Developer/Executor) - [ ] Security Tools Integration - [x] Memory System (Vector Store/Knowledge Base) - [ ] Monitoring Stack (Grafana/OpenTelemetry) - [ ] Analytics Platform (Langfuse) - [x] External Integrations (LLM/Search APIs) - [x] Documentation - [x] Infrastructure/DevOps ### Testing and Verification <!-- Please describe the tests that you ran to verify your changes and provide instructions so we can reproduce. --> #### Test Configuration ```yaml PentAGI Version: latest Docker Version: 28.0.1 Host OS: Linux/macOS LLM Provider: [all] Enabled Features: [Embedding Provider] ``` #### Test Steps 1. Build the `etester` utility with `go build -o etester cmd/etester/main.go` 2. Run `./etester test` to verify embedding provider connectivity 3. Test embedding functionality with various providers (Cybertron, Huggingface, Bedrock, VoyageAI) 4. Use `./etester info` to check embedding statistics 5. Verify management commands with `./etester flush` and `./etester reindex` #### Test Results The embedding providers correctly initialize and connect to the respective services. The `etester` utility successfully performs all operations including testing, statistics gathering, flushing, and reindexing. ### Security Considerations The implementation adds support for external embedding providers which require API keys and credentials. These are handled securely through environment variables and configuration files. No plaintext credentials are stored in the code. ### Performance Impact The new embedding providers offer various performance characteristics depending on the model and service used. Performance testing shows acceptable response times for embedding generation. The `etester` utility includes verbose output options that can help diagnose performance issues. ### Documentation Updates <!-- Note any documentation changes required by this PR --> - [x] README.md updates - [ ] API documentation updates - [x] Configuration documentation updates - [ ] GraphQL schema updates - [x] Other: `.env.example` updates with embedding configuration options ### Deployment Notes To use the new embedding providers, users need to: 1. Set appropriate environment variables for the desired embedding provider 2. Update configuration files to specify the provider and model 3. For Docker deployments, the updated Dockerfile includes the `etester` utility New environment variables: - `EMBEDDING_PROVIDER`: Type of embedding provider to use - `EMBEDDING_MODEL`: Specific model for the selected provider - `EMBEDDING_API_KEY`: API key for the embedding provider - Various provider-specific configuration options ### Checklist <!--- Go over all the following points, and put an `x` in all the boxes that apply. --> #### Code Quality - [x] My code follows the project's coding standards - [x] I have added/updated necessary documentation - [x] I have added tests to cover my changes - [x] All new and existing tests pass - [x] I have run `go fmt` and `go vet` (for Go code) - [ ] I have run `npm run lint` (for TypeScript/JavaScript code) #### Security - [x] I have considered security implications - [x] Changes maintain or improve the security model - [x] Sensitive information has been properly handled #### Compatibility - [x] Changes are backward compatible - [ ] Breaking changes are clearly marked and documented - [x] Dependencies are properly updated #### Documentation - [x] Documentation is clear and complete - [x] Comments are added for non-obvious code - [x] API changes are documented ### Additional Notes The `etester` utility provides a flexible framework for testing and managing embeddings that can be extended with additional functionality in the future. The implementation of multiple embedding providers allows users to choose the most appropriate model for their specific use case and performance requirements. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
yindo added the pull-request label 2026-06-06 22:09:18 -04:00
yindo closed this issue 2026-06-06 22:09:18 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
enhancement
help wanted
pull-request
Mirrored from GitHub Pull Request
 question
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: vxcontrol/pentagi#127
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?