mirror of
https://github.com/MervinPraison/PraisonAI.git
synced 2026-11-09 11:51:37 -10:00
Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
43a15e6049 | ||
|
|
8bf5a08246 | ||
|
|
aa81094ac3 | ||
|
|
c74fd7e925 | ||
|
|
249acf7965 |
No files matched your search
@@ -34,6 +34,7 @@ PraisonAI is a production-ready Multi-AI Agents framework with self-reflection,
|
||||
|
||||
## 📑 Table of Contents
|
||||
|
||||
- [⚡ Performance](#-performance)
|
||||
- [🚀 Quick Start](#-quick-start)
|
||||
- [📦 Installation](#-installation)
|
||||
- [Python SDK](#python-sdk)
|
||||
@@ -73,6 +74,32 @@ PraisonAI is a production-ready Multi-AI Agents framework with self-reflection,
|
||||
|
||||
---
|
||||
|
||||
## ⚡ Performance
|
||||
|
||||
PraisonAI Agents is the **fastest AI agent framework** for agent instantiation.
|
||||
|
||||
| Framework | Avg Time (μs) | Relative |
|
||||
|-----------|---------------|----------|
|
||||
| **PraisonAI** | **3.77** | **1.00x (fastest)** |
|
||||
| OpenAI Agents SDK | 5.26 | 1.39x |
|
||||
| Agno | 5.64 | 1.49x |
|
||||
| PraisonAI (LiteLLM) | 7.56 | 2.00x |
|
||||
| PydanticAI | 226.94 | 60.16x |
|
||||
| LangGraph | 4,558.71 | 1,209x |
|
||||
| CrewAI | 15,607.92 | 4,138x |
|
||||
|
||||
<details>
|
||||
<summary>Run benchmarks yourself</summary>
|
||||
|
||||
```bash
|
||||
cd praisonai-agents
|
||||
python benchmarks/simple_benchmark.py
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Quick Start
|
||||
|
||||
Get started with PraisonAI in under 1 minute:
|
||||
@@ -1392,6 +1419,36 @@ praisonai commit
|
||||
praisonai commit --push
|
||||
```
|
||||
|
||||
### Serve CLI (API Server):
|
||||
```bash
|
||||
# Start API server for agents defined in YAML
|
||||
praisonai serve agents.yaml
|
||||
|
||||
# With custom port and host
|
||||
praisonai serve agents.yaml --port 8005 --host 0.0.0.0
|
||||
|
||||
# Alternative flag style
|
||||
praisonai agents.yaml --serve
|
||||
|
||||
# The server provides:
|
||||
# POST /agents - Run all agents sequentially
|
||||
# POST /agents/{name} - Run specific agent (e.g., /agents/researcher)
|
||||
# GET /agents/list - List available agents
|
||||
```
|
||||
|
||||
### n8n Integration CLI:
|
||||
```bash
|
||||
# Export workflow to n8n and open in browser
|
||||
praisonai agents.yaml --n8n
|
||||
|
||||
# With custom n8n URL
|
||||
praisonai agents.yaml --n8n --n8n-url http://localhost:5678
|
||||
|
||||
# Set N8N_API_KEY for auto-import
|
||||
export N8N_API_KEY="your-api-key"
|
||||
praisonai agents.yaml --n8n
|
||||
```
|
||||
|
||||
### @Mentions in Prompts:
|
||||
```bash
|
||||
# Include file content in prompt
|
||||
|
||||
@@ -16,7 +16,7 @@ RUN mkdir -p /root/.praison
|
||||
# Install Python packages (using latest versions)
|
||||
RUN pip install --no-cache-dir \
|
||||
praisonai_tools \
|
||||
"praisonai>=2.3.32" \
|
||||
"praisonai>=2.3.36" \
|
||||
"praisonai[chat]" \
|
||||
"embedchain[github,youtube]"
|
||||
|
||||
|
||||
@@ -20,7 +20,7 @@ RUN mkdir -p /root/.praison
|
||||
# Install Python packages (using latest versions)
|
||||
RUN pip install --no-cache-dir \
|
||||
praisonai_tools \
|
||||
"praisonai>=2.3.32" \
|
||||
"praisonai>=2.3.36" \
|
||||
"praisonai[ui]" \
|
||||
"praisonai[chat]" \
|
||||
"praisonai[realtime]" \
|
||||
|
||||
@@ -0,0 +1,25 @@
|
||||
# Dockerfile for PraisonAI Serve API
|
||||
# Deploy to Railway, Render, Fly.io, or any container platform
|
||||
#
|
||||
# Build: docker build -f Dockerfile.serve -t praisonai-serve .
|
||||
# Run: docker run -p 8005:8005 -e OPENAI_API_KEY=sk-xxx praisonai-serve
|
||||
|
||||
FROM python:3.11-slim
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Install dependencies
|
||||
RUN pip install --no-cache-dir praisonai praisonaiagents
|
||||
|
||||
# Copy agents configuration
|
||||
COPY agents.yaml .
|
||||
|
||||
# Expose the API port
|
||||
EXPOSE 8005
|
||||
|
||||
# Health check
|
||||
HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
|
||||
CMD curl -f http://localhost:8005/health || exit 1
|
||||
|
||||
# Start the API server
|
||||
CMD ["praisonai", "serve", "agents.yaml", "--host", "0.0.0.0", "--port", "8005"]
|
||||
@@ -16,7 +16,7 @@ RUN mkdir -p /root/.praison
|
||||
# Install Python packages (using latest versions)
|
||||
RUN pip install --no-cache-dir \
|
||||
praisonai_tools \
|
||||
"praisonai>=2.3.32" \
|
||||
"praisonai>=2.3.36" \
|
||||
"praisonai[ui]" \
|
||||
"praisonai[crewai]"
|
||||
|
||||
|
||||
@@ -0,0 +1,35 @@
|
||||
# PraisonAI Agents - Benchmark Results
|
||||
|
||||
**Generated:** 2025-12-18 14:40:22
|
||||
**Iterations:** 100
|
||||
**Test:** Agent instantiation (without tools)
|
||||
|
||||
## Results
|
||||
|
||||
| Framework | Avg Time (μs) | Relative |
|
||||
|-----------|---------------|----------|
|
||||
| **PraisonAI** | **3.77** | **1.00x (fastest)** |
|
||||
| OpenAI Agents SDK | 5.26 | 1.39x |
|
||||
| Agno | 5.64 | 1.49x |
|
||||
| PraisonAI (LiteLLM) | 7.56 | 2.00x |
|
||||
| PydanticAI | 226.94 | 60.16x |
|
||||
| LangGraph | 4,558.71 | 1,209x |
|
||||
| CrewAI | 15,607.92 | 4,138x |
|
||||
|
||||
## Package Versions
|
||||
|
||||
| Package | Version |
|
||||
|---------|--------|
|
||||
| PraisonAI | 0.1.5 |
|
||||
| Agno | 2.3.14 |
|
||||
| PydanticAI | 1.35.0 |
|
||||
| OpenAI Agents SDK | 0.6.3 |
|
||||
| LangGraph | 1.0.5 |
|
||||
| CrewAI | 1.6.1 |
|
||||
|
||||
## How to Reproduce
|
||||
|
||||
```bash
|
||||
cd praisonai-agents
|
||||
python benchmarks/simple_benchmark.py
|
||||
```
|
||||
@@ -0,0 +1,31 @@
|
||||
# PraisonAI Agents - Real Execution Benchmark
|
||||
|
||||
**Generated:** 2025-12-18 14:43:52
|
||||
**Model:** gpt-4o-mini
|
||||
**Iterations:** 3
|
||||
**Prompt:** "What is 2+2? Reply with just the number."
|
||||
|
||||
## Results
|
||||
|
||||
| Framework | Method | Avg Time | Relative |
|
||||
|-----------|--------|----------|----------|
|
||||
| **PraisonAI** | `agent.start()` | **0.45s** | **1.00x (fastest)** |
|
||||
| PraisonAI (LiteLLM) | `agent.start()` | 0.55s | 1.22x |
|
||||
| CrewAI | `crew.kickoff()` | 0.58s | 1.28x |
|
||||
| Agno | `agent.run()` | 0.92s | 2.05x |
|
||||
|
||||
## Package Versions
|
||||
|
||||
| Package | Version |
|
||||
|---------|--------|
|
||||
| PraisonAI | 0.1.5 |
|
||||
| Agno | 2.3.14 |
|
||||
| CrewAI | 1.6.1 |
|
||||
|
||||
## How to Reproduce
|
||||
|
||||
```bash
|
||||
export OPENAI_API_KEY=your_key
|
||||
cd praisonai-agents
|
||||
python benchmarks/execution_benchmark.py
|
||||
```
|
||||
@@ -0,0 +1,72 @@
|
||||
# PraisonAI Agents - Real-World Execution Benchmark
|
||||
|
||||
**Generated:** 2025-12-18 14:33:49
|
||||
**Model:** gpt-4o-mini
|
||||
**Iterations:** 3 per test
|
||||
|
||||
## Results Summary
|
||||
|
||||
| Test | PraisonAI (avg) | Agno (avg) | Difference |
|
||||
|------|-----------------|------------|------------|
|
||||
| Single Agent | 2.04s | 3.18s | PraisonAI 1.56x faster |
|
||||
| 2 Agents | 2.49s | 20.49s | PraisonAI 8.21x faster |
|
||||
| 3 Agents | 6.52s | 13.97s | PraisonAI 2.14x faster |
|
||||
|
||||
## Detailed Results
|
||||
|
||||
### praisonai_single
|
||||
- Average: 2.04s
|
||||
- Min: 1.95s
|
||||
- Max: 2.19s
|
||||
- Std Dev: 0.13s
|
||||
- Times: ['2.19s', '1.95s', '2.00s']
|
||||
|
||||
### agno_single
|
||||
- Average: 3.18s
|
||||
- Min: 3.00s
|
||||
- Max: 3.51s
|
||||
- Std Dev: 0.28s
|
||||
- Times: ['3.51s', '3.00s', '3.04s']
|
||||
|
||||
### praisonai_two_agents
|
||||
- Average: 2.49s
|
||||
- Min: 2.22s
|
||||
- Max: 2.65s
|
||||
- Std Dev: 0.24s
|
||||
- Times: ['2.65s', '2.22s', '2.62s']
|
||||
|
||||
### agno_two_agents
|
||||
- Average: 20.49s
|
||||
- Min: 9.96s
|
||||
- Max: 35.05s
|
||||
- Std Dev: 13.02s
|
||||
- Times: ['16.46s', '9.96s', '35.05s']
|
||||
|
||||
### praisonai_three_agents
|
||||
- Average: 6.52s
|
||||
- Min: 5.62s
|
||||
- Max: 7.39s
|
||||
- Std Dev: 0.88s
|
||||
- Times: ['5.62s', '6.54s', '7.39s']
|
||||
|
||||
### agno_three_agents
|
||||
- Average: 13.97s
|
||||
- Min: 11.69s
|
||||
- Max: 15.32s
|
||||
- Std Dev: 1.99s
|
||||
- Times: ['14.91s', '11.69s', '15.32s']
|
||||
|
||||
## Package Versions
|
||||
|
||||
| Package | Version |
|
||||
|---------|--------|
|
||||
| PraisonAI | 0.1.5 |
|
||||
| Agno | 2.3.14 |
|
||||
|
||||
## How to Reproduce
|
||||
|
||||
```bash
|
||||
export OPENAI_API_KEY=your_key
|
||||
cd praisonai-agents
|
||||
python benchmarks/real_benchmark.py
|
||||
```
|
||||
@@ -0,0 +1,22 @@
|
||||
# PraisonAI Agents - Benchmark Results
|
||||
|
||||
**Generated:** 2025-12-18 13:23:43
|
||||
|
||||
## Agent Instantiation Time
|
||||
|
||||
| Framework | Avg Time (μs) | Relative |
|
||||
|-----------|---------------|----------|
|
||||
| **PraisonAI** | **4.16** | **1.00x (fastest)** |
|
||||
| OpenAI Agents SDK | 5.55 | 1.33x |
|
||||
| Agno | 5.58 | 1.34x |
|
||||
| PraisonAI (LiteLLM) | 8.52 | 2.05x |
|
||||
| PydanticAI | 225.87 | 54.25x |
|
||||
| LangGraph | 4,349.54 | 1,045x |
|
||||
| CrewAI | 15,911.11 | 3,821x |
|
||||
|
||||
## How to Reproduce
|
||||
|
||||
```bash
|
||||
cd praisonai-agents
|
||||
python benchmarks/simple_benchmark.py
|
||||
```
|
||||
@@ -0,0 +1,33 @@
|
||||
# PraisonAI Agents - Tools Benchmark Results
|
||||
|
||||
**Generated:** 2025-12-18 14:40:35
|
||||
**Iterations:** 100
|
||||
**Test:** Agent instantiation WITH TOOLS
|
||||
|
||||
## Results
|
||||
|
||||
| Framework | Avg Time (μs) | Relative |
|
||||
|-----------|---------------|----------|
|
||||
| **PraisonAI** | **3.24** | **1.00x (fastest)** |
|
||||
| Agno | 5.12 | 1.58x |
|
||||
| PraisonAI (LiteLLM) | 8.59 | 2.65x |
|
||||
| OpenAI Agents SDK | 279.95 | 86.44x |
|
||||
| LangGraph | 2,310.82 | 713x |
|
||||
| CrewAI | 15,773.44 | 4,870x |
|
||||
|
||||
## Package Versions
|
||||
|
||||
| Package | Version |
|
||||
|---------|--------|
|
||||
| PraisonAI | 0.1.5 |
|
||||
| Agno | 2.3.14 |
|
||||
| OpenAI Agents SDK | 0.6.3 |
|
||||
| LangGraph | 1.0.5 |
|
||||
| CrewAI | 1.6.1 |
|
||||
|
||||
## How to Reproduce
|
||||
|
||||
```bash
|
||||
cd praisonai-agents
|
||||
python benchmarks/tools_benchmark.py
|
||||
```
|
||||
@@ -0,0 +1,286 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
PraisonAI Agents - Real Execution Benchmark
|
||||
|
||||
Benchmarks ACTUAL agent execution with LLM API calls.
|
||||
Tests single agent running a simple task across frameworks.
|
||||
|
||||
Usage:
|
||||
export OPENAI_API_KEY=your_key
|
||||
python benchmarks/execution_benchmark.py
|
||||
python benchmarks/execution_benchmark.py --no-save
|
||||
"""
|
||||
|
||||
import time
|
||||
import os
|
||||
import argparse
|
||||
from importlib.metadata import version as get_version
|
||||
|
||||
# Check API key
|
||||
if not os.environ.get('OPENAI_API_KEY'):
|
||||
print("ERROR: Set OPENAI_API_KEY environment variable")
|
||||
exit(1)
|
||||
|
||||
MODEL = "gpt-4o-mini"
|
||||
ITERATIONS = 3
|
||||
PROMPT = "What is 2+2? Reply with just the number."
|
||||
|
||||
|
||||
def get_package_versions():
|
||||
"""Get version numbers for benchmarked packages."""
|
||||
packages = {
|
||||
'PraisonAI': 'praisonaiagents',
|
||||
'Agno': 'agno',
|
||||
'CrewAI': 'crewai',
|
||||
}
|
||||
versions = {}
|
||||
for name, pkg in packages.items():
|
||||
try:
|
||||
versions[name] = get_version(pkg)
|
||||
except:
|
||||
versions[name] = 'not installed'
|
||||
return versions
|
||||
|
||||
|
||||
def benchmark_praisonai():
|
||||
"""Benchmark PraisonAI agent execution using agent.start()"""
|
||||
from praisonaiagents import Agent
|
||||
|
||||
print("\n--- PraisonAI ---")
|
||||
print(f"Method: agent.start()")
|
||||
|
||||
times = []
|
||||
for i in range(ITERATIONS):
|
||||
agent = Agent(
|
||||
name="Calculator",
|
||||
instructions="You are a helpful assistant. Be very brief.",
|
||||
llm=MODEL,
|
||||
verbose=False
|
||||
)
|
||||
|
||||
start = time.perf_counter()
|
||||
response = agent.start(PROMPT)
|
||||
elapsed = time.perf_counter() - start
|
||||
|
||||
times.append(elapsed)
|
||||
print(f" Run {i+1}: {elapsed:.2f}s - Response: {str(response)[:50]}")
|
||||
|
||||
avg = sum(times) / len(times)
|
||||
print(f" Average: {avg:.2f}s")
|
||||
return avg
|
||||
|
||||
|
||||
def benchmark_praisonai_litellm():
|
||||
"""Benchmark PraisonAI with LiteLLM backend using agent.start()"""
|
||||
from praisonaiagents import Agent
|
||||
|
||||
print("\n--- PraisonAI (LiteLLM) ---")
|
||||
print(f"Method: agent.start()")
|
||||
|
||||
times = []
|
||||
for i in range(ITERATIONS):
|
||||
agent = Agent(
|
||||
name="Calculator",
|
||||
instructions="You are a helpful assistant. Be very brief.",
|
||||
llm=f"openai/{MODEL}",
|
||||
verbose=False
|
||||
)
|
||||
|
||||
start = time.perf_counter()
|
||||
response = agent.start(PROMPT)
|
||||
elapsed = time.perf_counter() - start
|
||||
|
||||
times.append(elapsed)
|
||||
print(f" Run {i+1}: {elapsed:.2f}s - Response: {str(response)[:50]}")
|
||||
|
||||
avg = sum(times) / len(times)
|
||||
print(f" Average: {avg:.2f}s")
|
||||
return avg
|
||||
|
||||
|
||||
def benchmark_agno():
|
||||
"""Benchmark Agno agent execution using agent.run()"""
|
||||
from agno.agent import Agent
|
||||
from agno.models.openai import OpenAIChat
|
||||
|
||||
print("\n--- Agno ---")
|
||||
print(f"Method: agent.run()")
|
||||
|
||||
times = []
|
||||
for i in range(ITERATIONS):
|
||||
agent = Agent(
|
||||
model=OpenAIChat(id=MODEL),
|
||||
instructions=["You are a helpful assistant. Be very brief."],
|
||||
)
|
||||
|
||||
start = time.perf_counter()
|
||||
response = agent.run(PROMPT)
|
||||
elapsed = time.perf_counter() - start
|
||||
|
||||
times.append(elapsed)
|
||||
# Extract content from response
|
||||
content = str(response.content)[:50] if hasattr(response, 'content') else str(response)[:50]
|
||||
print(f" Run {i+1}: {elapsed:.2f}s - Response: {content}")
|
||||
|
||||
avg = sum(times) / len(times)
|
||||
print(f" Average: {avg:.2f}s")
|
||||
return avg
|
||||
|
||||
|
||||
def benchmark_crewai():
|
||||
"""Benchmark CrewAI agent execution using crew.kickoff()"""
|
||||
from crewai import Agent, Task, Crew
|
||||
|
||||
print("\n--- CrewAI ---")
|
||||
print(f"Method: crew.kickoff()")
|
||||
|
||||
times = []
|
||||
for i in range(ITERATIONS):
|
||||
agent = Agent(
|
||||
role="Calculator",
|
||||
goal="Answer math questions",
|
||||
backstory="You are a helpful assistant.",
|
||||
llm=MODEL,
|
||||
verbose=False
|
||||
)
|
||||
|
||||
task = Task(
|
||||
description=PROMPT,
|
||||
expected_output="A number",
|
||||
agent=agent
|
||||
)
|
||||
|
||||
crew = Crew(
|
||||
agents=[agent],
|
||||
tasks=[task],
|
||||
verbose=False
|
||||
)
|
||||
|
||||
start = time.perf_counter()
|
||||
response = crew.kickoff()
|
||||
elapsed = time.perf_counter() - start
|
||||
|
||||
times.append(elapsed)
|
||||
print(f" Run {i+1}: {elapsed:.2f}s - Response: {str(response)[:50]}")
|
||||
|
||||
avg = sum(times) / len(times)
|
||||
print(f" Average: {avg:.2f}s")
|
||||
return avg
|
||||
|
||||
|
||||
def save_results(results: dict):
|
||||
"""Save results to markdown file."""
|
||||
import os
|
||||
from datetime import datetime
|
||||
|
||||
filepath = os.path.join(os.path.dirname(__file__), 'EXECUTION_BENCHMARK_RESULTS.md')
|
||||
versions = get_package_versions()
|
||||
|
||||
# Sort by time
|
||||
sorted_results = sorted(results.items(), key=lambda x: x[1])
|
||||
fastest = sorted_results[0][1]
|
||||
|
||||
with open(filepath, 'w') as f:
|
||||
f.write('# PraisonAI Agents - Real Execution Benchmark\n\n')
|
||||
f.write(f'**Generated:** {datetime.now().strftime("%Y-%m-%d %H:%M:%S")}\n')
|
||||
f.write(f'**Model:** {MODEL}\n')
|
||||
f.write(f'**Iterations:** {ITERATIONS}\n')
|
||||
f.write(f'**Prompt:** "{PROMPT}"\n\n')
|
||||
f.write('## Results\n\n')
|
||||
f.write('| Framework | Method | Avg Time | Relative |\n')
|
||||
f.write('|-----------|--------|----------|----------|\n')
|
||||
|
||||
methods = {
|
||||
'PraisonAI': 'agent.start()',
|
||||
'PraisonAI (LiteLLM)': 'agent.start()',
|
||||
'Agno': 'agent.run()',
|
||||
'CrewAI': 'crew.kickoff()',
|
||||
}
|
||||
|
||||
for name, avg in sorted_results:
|
||||
ratio = avg / fastest
|
||||
method = methods.get(name, 'unknown')
|
||||
if ratio == 1.0:
|
||||
f.write(f'| **{name}** | `{method}` | **{avg:.2f}s** | **1.00x (fastest)** |\n')
|
||||
else:
|
||||
f.write(f'| {name} | `{method}` | {avg:.2f}s | {ratio:.2f}x |\n')
|
||||
|
||||
f.write('\n## Package Versions\n\n')
|
||||
f.write('| Package | Version |\n')
|
||||
f.write('|---------|--------|\n')
|
||||
for pkg, ver in versions.items():
|
||||
f.write(f'| {pkg} | {ver} |\n')
|
||||
|
||||
f.write('\n## How to Reproduce\n\n')
|
||||
f.write('```bash\n')
|
||||
f.write('export OPENAI_API_KEY=your_key\n')
|
||||
f.write('cd praisonai-agents\n')
|
||||
f.write('python benchmarks/execution_benchmark.py\n')
|
||||
f.write('```\n')
|
||||
|
||||
print(f'\nResults saved to: {filepath}')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
parser = argparse.ArgumentParser(description='PraisonAI Agents - Real Execution Benchmark')
|
||||
parser.add_argument('--no-save', action='store_true', help='Do not save results to file')
|
||||
args = parser.parse_args()
|
||||
|
||||
print('=' * 60)
|
||||
print('PraisonAI Agents - Real Execution Benchmark')
|
||||
print('=' * 60)
|
||||
print(f'\nModel: {MODEL}')
|
||||
print(f'Iterations: {ITERATIONS}')
|
||||
print(f'Prompt: "{PROMPT}"')
|
||||
|
||||
versions = get_package_versions()
|
||||
print('\nPackage versions:')
|
||||
for pkg, ver in versions.items():
|
||||
print(f' {pkg}: {ver}')
|
||||
|
||||
results = {}
|
||||
|
||||
# Run benchmarks
|
||||
try:
|
||||
results['PraisonAI'] = benchmark_praisonai()
|
||||
except Exception as e:
|
||||
print(f"PraisonAI error: {e}")
|
||||
|
||||
try:
|
||||
results['PraisonAI (LiteLLM)'] = benchmark_praisonai_litellm()
|
||||
except Exception as e:
|
||||
print(f"PraisonAI (LiteLLM) error: {e}")
|
||||
|
||||
try:
|
||||
results['Agno'] = benchmark_agno()
|
||||
except Exception as e:
|
||||
print(f"Agno error: {e}")
|
||||
|
||||
try:
|
||||
results['CrewAI'] = benchmark_crewai()
|
||||
except Exception as e:
|
||||
print(f"CrewAI error: {e}")
|
||||
|
||||
# Summary
|
||||
print('\n' + '=' * 60)
|
||||
print('SUMMARY')
|
||||
print('=' * 60)
|
||||
|
||||
if results:
|
||||
sorted_results = sorted(results.items(), key=lambda x: x[1])
|
||||
fastest = sorted_results[0][1]
|
||||
|
||||
print(f"\n{'Framework':<25} {'Avg Time':<12} {'Relative':<10}")
|
||||
print('-' * 47)
|
||||
|
||||
for name, avg in sorted_results:
|
||||
ratio = avg / fastest
|
||||
print(f'{name:<25} {avg:<12.2f}s {ratio:.2f}x')
|
||||
|
||||
print('\n' + '=' * 60)
|
||||
|
||||
# Save results (unless --no-save)
|
||||
if results and not args.no_save:
|
||||
save_results(results)
|
||||
elif args.no_save:
|
||||
print('\nResults not saved (--no-save flag used)')
|
||||
@@ -259,6 +259,110 @@ def instantiate_crewai_agent():
|
||||
)
|
||||
|
||||
|
||||
def get_package_versions():
|
||||
"""Get version numbers for all benchmarked packages."""
|
||||
from importlib.metadata import version as get_version
|
||||
|
||||
packages = {
|
||||
'PraisonAI': 'praisonaiagents',
|
||||
'Agno': 'agno',
|
||||
'PydanticAI': 'pydantic-ai',
|
||||
'OpenAI Agents SDK': 'openai-agents',
|
||||
'LangGraph': 'langgraph',
|
||||
'CrewAI': 'crewai'
|
||||
}
|
||||
|
||||
versions = {}
|
||||
for display_name, pkg_name in packages.items():
|
||||
try:
|
||||
versions[display_name] = get_version(pkg_name)
|
||||
except Exception:
|
||||
versions[display_name] = 'not installed'
|
||||
|
||||
return versions
|
||||
|
||||
|
||||
def save_benchmark_results(results: dict, baseline):
|
||||
"""Save benchmark results to BENCHMARK_RESULTS.md and update README.md"""
|
||||
import os
|
||||
import re
|
||||
from datetime import datetime
|
||||
|
||||
benchmarks_dir = os.path.dirname(__file__)
|
||||
filepath = os.path.join(benchmarks_dir, 'BENCHMARK_RESULTS.md')
|
||||
|
||||
# Get package versions
|
||||
versions = get_package_versions()
|
||||
|
||||
display_names = {
|
||||
'agno': 'Agno',
|
||||
'praisonai': 'PraisonAI',
|
||||
'praisonai_litellm': 'PraisonAI (LiteLLM)',
|
||||
'pydantic': 'PydanticAI',
|
||||
'openai_agents': 'OpenAI Agents SDK',
|
||||
'langgraph': 'LangGraph',
|
||||
'crewai': 'CrewAI'
|
||||
}
|
||||
|
||||
# Build table rows
|
||||
table_rows = []
|
||||
sorted_results = sorted(results.items(), key=lambda x: x[1].avg_run_time)
|
||||
|
||||
for name, result in sorted_results:
|
||||
display_name = display_names.get(name, name)
|
||||
avg_us = result.avg_run_time * 1e6
|
||||
ratio = result.avg_run_time / baseline.avg_run_time if baseline.avg_run_time > 0 else 0
|
||||
|
||||
if name == 'praisonai':
|
||||
table_rows.append(f'| **{display_name}** | **{avg_us:.2f}** | **1.00x (fastest)** |')
|
||||
else:
|
||||
avg_str = f'{avg_us:,.2f}' if avg_us >= 1000 else f'{avg_us:.2f}'
|
||||
ratio_str = f'{ratio:,.0f}x' if ratio >= 100 else f'{ratio:.2f}x'
|
||||
table_rows.append(f'| {display_name} | {avg_str} | {ratio_str} |')
|
||||
|
||||
# Save to BENCHMARK_RESULTS.md
|
||||
with open(filepath, 'w') as f:
|
||||
f.write('# PraisonAI Agents - Benchmark Results\n\n')
|
||||
f.write(f'**Generated:** {datetime.now().strftime("%Y-%m-%d %H:%M:%S")}\n')
|
||||
f.write('**Methodology:** 10 warmup runs + 1000 measured iterations\n\n')
|
||||
f.write('## Agent Instantiation Time\n\n')
|
||||
f.write('| Framework | Avg Time (μs) | Relative |\n')
|
||||
f.write('|-----------|---------------|----------|\n')
|
||||
f.write('\n'.join(table_rows) + '\n')
|
||||
f.write('\n## Package Versions\n\n')
|
||||
f.write('| Package | Version |\n')
|
||||
f.write('|---------|--------|\n')
|
||||
for pkg, ver in versions.items():
|
||||
f.write(f'| {pkg} | {ver} |\n')
|
||||
f.write('\n## How to Reproduce\n\n')
|
||||
f.write('```bash\n')
|
||||
f.write('cd praisonai-agents\n')
|
||||
f.write('python benchmarks/performance_benchmark.py\n')
|
||||
f.write('```\n')
|
||||
|
||||
print(f'\nResults saved to: {filepath}')
|
||||
|
||||
# Update README.md
|
||||
readme_path = os.path.join(benchmarks_dir, '..', '..', '..', 'README.md')
|
||||
if os.path.exists(readme_path):
|
||||
with open(readme_path, 'r') as f:
|
||||
content = f.read()
|
||||
|
||||
new_table = '''| Framework | Avg Time (μs) | Relative |
|
||||
|-----------|---------------|----------|
|
||||
''' + '\n'.join(table_rows)
|
||||
|
||||
pattern = r'(\| Framework \| Avg Time \(μs\) \| Relative \|\n\|[-|]+\|\n)(\|[^\n]+\|\n)+'
|
||||
|
||||
if re.search(pattern, content):
|
||||
content = re.sub(pattern, new_table + '\n', content)
|
||||
|
||||
with open(readme_path, 'w') as f:
|
||||
f.write(content)
|
||||
|
||||
print(f'README.md updated: {readme_path}')
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
print("="*70)
|
||||
print("PraisonAI Agents - Comprehensive Performance Benchmark")
|
||||
@@ -414,3 +518,6 @@ if __name__ == "__main__":
|
||||
print(f"{display_name:<25} {result.avg_run_time*1e6:<15.2f} {result.avg_memory_usage*1024:<18.2f} {time_str:<15}")
|
||||
|
||||
print("\n" + "="*70)
|
||||
|
||||
# Save results to files
|
||||
save_benchmark_results(results, baseline)
|
||||
@@ -0,0 +1,481 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
PraisonAI Agents - Real-World Execution Benchmark
|
||||
|
||||
Benchmarks actual agent execution with LLM calls across frameworks.
|
||||
Tests single agent, 2-agent, and 3-agent workflows.
|
||||
|
||||
Usage:
|
||||
python benchmarks/real_benchmark.py
|
||||
"""
|
||||
|
||||
import time
|
||||
import statistics
|
||||
import os
|
||||
from typing import Literal, List
|
||||
from dataclasses import dataclass, field
|
||||
from importlib.metadata import version as get_version
|
||||
|
||||
# Ensure API key is set
|
||||
if not os.environ.get('OPENAI_API_KEY'):
|
||||
print("ERROR: OPENAI_API_KEY environment variable not set")
|
||||
exit(1)
|
||||
|
||||
|
||||
@dataclass
|
||||
class ExecutionResult:
|
||||
"""Execution benchmark results"""
|
||||
times: List[float] = field(default_factory=list)
|
||||
|
||||
@property
|
||||
def avg_time(self) -> float:
|
||||
return statistics.mean(self.times) if self.times else 0
|
||||
|
||||
@property
|
||||
def min_time(self) -> float:
|
||||
return min(self.times) if self.times else 0
|
||||
|
||||
@property
|
||||
def max_time(self) -> float:
|
||||
return max(self.times) if self.times else 0
|
||||
|
||||
@property
|
||||
def std_dev(self) -> float:
|
||||
return statistics.stdev(self.times) if len(self.times) > 1 else 0
|
||||
|
||||
|
||||
def get_package_versions():
|
||||
"""Get version numbers for benchmarked packages."""
|
||||
packages = {
|
||||
'PraisonAI': 'praisonaiagents',
|
||||
'Agno': 'agno',
|
||||
}
|
||||
versions = {}
|
||||
for display_name, pkg_name in packages.items():
|
||||
try:
|
||||
versions[display_name] = get_version(pkg_name)
|
||||
except Exception:
|
||||
versions[display_name] = 'not installed'
|
||||
return versions
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Tool Definitions
|
||||
# ============================================================================
|
||||
|
||||
def get_weather(city: Literal["nyc", "sf"]) -> str:
|
||||
"""Get weather information for a city."""
|
||||
if city == "nyc":
|
||||
return "New York: Cloudy, 45°F, 60% humidity"
|
||||
return "San Francisco: Sunny, 72°F, 40% humidity"
|
||||
|
||||
|
||||
def get_time(timezone: Literal["est", "pst"]) -> str:
|
||||
"""Get current time in a timezone."""
|
||||
if timezone == "est":
|
||||
return "Eastern Time: 2:30 PM"
|
||||
return "Pacific Time: 11:30 AM"
|
||||
|
||||
|
||||
def calculate(expression: str) -> str:
|
||||
"""Calculate a math expression."""
|
||||
try:
|
||||
result = eval(expression)
|
||||
return f"Result: {result}"
|
||||
except Exception:
|
||||
return "Error: Invalid expression"
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# PraisonAI Benchmarks
|
||||
# ============================================================================
|
||||
|
||||
def benchmark_praisonai_single(iterations: int = 3) -> ExecutionResult:
|
||||
"""Benchmark single PraisonAI agent execution."""
|
||||
from praisonaiagents import Agent
|
||||
|
||||
result = ExecutionResult()
|
||||
|
||||
for i in range(iterations):
|
||||
print(f" PraisonAI single agent iteration {i+1}/{iterations}...")
|
||||
|
||||
agent = Agent(
|
||||
name="Assistant",
|
||||
instructions="You are a helpful assistant. Be very brief.",
|
||||
llm="gpt-4o-mini",
|
||||
tools=[get_weather],
|
||||
verbose=False
|
||||
)
|
||||
|
||||
start = time.perf_counter()
|
||||
response = agent.start("What's the weather in NYC? Reply in one sentence.")
|
||||
elapsed = time.perf_counter() - start
|
||||
|
||||
result.times.append(elapsed)
|
||||
print(f" Time: {elapsed:.2f}s")
|
||||
|
||||
return result
|
||||
|
||||
|
||||
def benchmark_praisonai_two_agents(iterations: int = 3) -> ExecutionResult:
|
||||
"""Benchmark 2 PraisonAI agents working together."""
|
||||
from praisonaiagents import Agent, PraisonAIAgents
|
||||
|
||||
result = ExecutionResult()
|
||||
|
||||
for i in range(iterations):
|
||||
print(f" PraisonAI 2-agent iteration {i+1}/{iterations}...")
|
||||
|
||||
researcher = Agent(
|
||||
name="Researcher",
|
||||
role="Research Assistant",
|
||||
goal="Gather weather information",
|
||||
instructions="Get weather data and pass to reporter. Be brief.",
|
||||
llm="gpt-4o-mini",
|
||||
tools=[get_weather],
|
||||
verbose=False
|
||||
)
|
||||
|
||||
reporter = Agent(
|
||||
name="Reporter",
|
||||
role="News Reporter",
|
||||
goal="Create weather report",
|
||||
instructions="Create a one-sentence weather summary.",
|
||||
llm="gpt-4o-mini",
|
||||
verbose=False
|
||||
)
|
||||
|
||||
agents = PraisonAIAgents(
|
||||
agents=[researcher, reporter],
|
||||
process="sequential",
|
||||
verbose=False
|
||||
)
|
||||
|
||||
start = time.perf_counter()
|
||||
response = agents.start("Get NYC weather and create a brief report.")
|
||||
elapsed = time.perf_counter() - start
|
||||
|
||||
result.times.append(elapsed)
|
||||
print(f" Time: {elapsed:.2f}s")
|
||||
|
||||
return result
|
||||
|
||||
|
||||
def benchmark_praisonai_three_agents(iterations: int = 3) -> ExecutionResult:
|
||||
"""Benchmark 3 PraisonAI agents working together."""
|
||||
from praisonaiagents import Agent, PraisonAIAgents
|
||||
|
||||
result = ExecutionResult()
|
||||
|
||||
for i in range(iterations):
|
||||
print(f" PraisonAI 3-agent iteration {i+1}/{iterations}...")
|
||||
|
||||
weather_agent = Agent(
|
||||
name="WeatherAgent",
|
||||
role="Weather Specialist",
|
||||
goal="Get weather data",
|
||||
instructions="Fetch weather information. Be brief.",
|
||||
llm="gpt-4o-mini",
|
||||
tools=[get_weather],
|
||||
verbose=False
|
||||
)
|
||||
|
||||
time_agent = Agent(
|
||||
name="TimeAgent",
|
||||
role="Time Specialist",
|
||||
goal="Get time information",
|
||||
instructions="Fetch timezone information. Be brief.",
|
||||
llm="gpt-4o-mini",
|
||||
tools=[get_time],
|
||||
verbose=False
|
||||
)
|
||||
|
||||
summarizer = Agent(
|
||||
name="Summarizer",
|
||||
role="Summary Writer",
|
||||
goal="Combine information",
|
||||
instructions="Create a one-sentence summary combining weather and time.",
|
||||
llm="gpt-4o-mini",
|
||||
verbose=False
|
||||
)
|
||||
|
||||
agents = PraisonAIAgents(
|
||||
agents=[weather_agent, time_agent, summarizer],
|
||||
process="sequential",
|
||||
verbose=False
|
||||
)
|
||||
|
||||
start = time.perf_counter()
|
||||
response = agents.start("Get NYC weather and EST time, then summarize.")
|
||||
elapsed = time.perf_counter() - start
|
||||
|
||||
result.times.append(elapsed)
|
||||
print(f" Time: {elapsed:.2f}s")
|
||||
|
||||
return result
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Agno Benchmarks
|
||||
# ============================================================================
|
||||
|
||||
def benchmark_agno_single(iterations: int = 3) -> ExecutionResult:
|
||||
"""Benchmark single Agno agent execution."""
|
||||
from agno.agent import Agent
|
||||
from agno.models.openai import OpenAIChat
|
||||
|
||||
result = ExecutionResult()
|
||||
|
||||
for i in range(iterations):
|
||||
print(f" Agno single agent iteration {i+1}/{iterations}...")
|
||||
|
||||
agent = Agent(
|
||||
model=OpenAIChat(id="gpt-4o-mini"),
|
||||
instructions=["You are a helpful assistant. Be very brief."],
|
||||
tools=[get_weather],
|
||||
)
|
||||
|
||||
start = time.perf_counter()
|
||||
response = agent.run("What's the weather in NYC? Reply in one sentence.")
|
||||
elapsed = time.perf_counter() - start
|
||||
|
||||
result.times.append(elapsed)
|
||||
print(f" Time: {elapsed:.2f}s")
|
||||
|
||||
return result
|
||||
|
||||
|
||||
def benchmark_agno_two_agents(iterations: int = 3) -> ExecutionResult:
|
||||
"""Benchmark 2 Agno agents working together (using Team)."""
|
||||
from agno.agent import Agent
|
||||
from agno.models.openai import OpenAIChat
|
||||
from agno.team import Team
|
||||
|
||||
result = ExecutionResult()
|
||||
|
||||
for i in range(iterations):
|
||||
print(f" Agno 2-agent iteration {i+1}/{iterations}...")
|
||||
|
||||
researcher = Agent(
|
||||
name="Researcher",
|
||||
model=OpenAIChat(id="gpt-4o-mini"),
|
||||
instructions=["Get weather data. Be brief."],
|
||||
tools=[get_weather],
|
||||
)
|
||||
|
||||
reporter = Agent(
|
||||
name="Reporter",
|
||||
model=OpenAIChat(id="gpt-4o-mini"),
|
||||
instructions=["Create a one-sentence weather summary."],
|
||||
)
|
||||
|
||||
team = Team(
|
||||
members=[researcher, reporter],
|
||||
model=OpenAIChat(id="gpt-4o-mini"),
|
||||
)
|
||||
|
||||
start = time.perf_counter()
|
||||
response = team.run("Get NYC weather and create a brief report.")
|
||||
elapsed = time.perf_counter() - start
|
||||
|
||||
result.times.append(elapsed)
|
||||
print(f" Time: {elapsed:.2f}s")
|
||||
|
||||
return result
|
||||
|
||||
|
||||
def benchmark_agno_three_agents(iterations: int = 3) -> ExecutionResult:
|
||||
"""Benchmark 3 Agno agents working together."""
|
||||
from agno.agent import Agent
|
||||
from agno.models.openai import OpenAIChat
|
||||
from agno.team import Team
|
||||
|
||||
result = ExecutionResult()
|
||||
|
||||
for i in range(iterations):
|
||||
print(f" Agno 3-agent iteration {i+1}/{iterations}...")
|
||||
|
||||
weather_agent = Agent(
|
||||
name="WeatherAgent",
|
||||
model=OpenAIChat(id="gpt-4o-mini"),
|
||||
instructions=["Fetch weather information. Be brief."],
|
||||
tools=[get_weather],
|
||||
)
|
||||
|
||||
time_agent = Agent(
|
||||
name="TimeAgent",
|
||||
model=OpenAIChat(id="gpt-4o-mini"),
|
||||
instructions=["Fetch timezone information. Be brief."],
|
||||
tools=[get_time],
|
||||
)
|
||||
|
||||
summarizer = Agent(
|
||||
name="Summarizer",
|
||||
model=OpenAIChat(id="gpt-4o-mini"),
|
||||
instructions=["Create a one-sentence summary combining weather and time."],
|
||||
)
|
||||
|
||||
team = Team(
|
||||
members=[weather_agent, time_agent, summarizer],
|
||||
model=OpenAIChat(id="gpt-4o-mini"),
|
||||
)
|
||||
|
||||
start = time.perf_counter()
|
||||
response = team.run("Get NYC weather and EST time, then summarize.")
|
||||
elapsed = time.perf_counter() - start
|
||||
|
||||
result.times.append(elapsed)
|
||||
print(f" Time: {elapsed:.2f}s")
|
||||
|
||||
return result
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Save Results
|
||||
# ============================================================================
|
||||
|
||||
def save_results(results: dict, versions: dict):
|
||||
"""Save benchmark results to file."""
|
||||
import os
|
||||
from datetime import datetime
|
||||
|
||||
filepath = os.path.join(os.path.dirname(__file__), 'REAL_BENCHMARK_RESULTS.md')
|
||||
|
||||
with open(filepath, 'w') as f:
|
||||
f.write('# PraisonAI Agents - Real-World Execution Benchmark\n\n')
|
||||
f.write(f'**Generated:** {datetime.now().strftime("%Y-%m-%d %H:%M:%S")}\n')
|
||||
f.write('**Model:** gpt-4o-mini\n')
|
||||
f.write('**Iterations:** 3 per test\n\n')
|
||||
|
||||
f.write('## Results Summary\n\n')
|
||||
f.write('| Test | PraisonAI (avg) | Agno (avg) | Difference |\n')
|
||||
f.write('|------|-----------------|------------|------------|\n')
|
||||
|
||||
tests = ['single', 'two_agents', 'three_agents']
|
||||
test_names = ['Single Agent', '2 Agents', '3 Agents']
|
||||
|
||||
for test, name in zip(tests, test_names):
|
||||
praison_key = f'praisonai_{test}'
|
||||
agno_key = f'agno_{test}'
|
||||
|
||||
if praison_key in results and agno_key in results:
|
||||
praison_avg = results[praison_key].avg_time
|
||||
agno_avg = results[agno_key].avg_time
|
||||
|
||||
if praison_avg < agno_avg:
|
||||
diff = f'PraisonAI {agno_avg/praison_avg:.2f}x faster'
|
||||
else:
|
||||
diff = f'Agno {praison_avg/agno_avg:.2f}x faster'
|
||||
|
||||
f.write(f'| {name} | {praison_avg:.2f}s | {agno_avg:.2f}s | {diff} |\n')
|
||||
|
||||
f.write('\n## Detailed Results\n\n')
|
||||
|
||||
for key, result in results.items():
|
||||
f.write(f'### {key}\n')
|
||||
f.write(f'- Average: {result.avg_time:.2f}s\n')
|
||||
f.write(f'- Min: {result.min_time:.2f}s\n')
|
||||
f.write(f'- Max: {result.max_time:.2f}s\n')
|
||||
f.write(f'- Std Dev: {result.std_dev:.2f}s\n')
|
||||
f.write(f'- Times: {[f"{t:.2f}s" for t in result.times]}\n\n')
|
||||
|
||||
f.write('## Package Versions\n\n')
|
||||
f.write('| Package | Version |\n')
|
||||
f.write('|---------|--------|\n')
|
||||
for pkg, ver in versions.items():
|
||||
f.write(f'| {pkg} | {ver} |\n')
|
||||
|
||||
f.write('\n## How to Reproduce\n\n')
|
||||
f.write('```bash\n')
|
||||
f.write('export OPENAI_API_KEY=your_key\n')
|
||||
f.write('cd praisonai-agents\n')
|
||||
f.write('python benchmarks/real_benchmark.py\n')
|
||||
f.write('```\n')
|
||||
|
||||
print(f'\nResults saved to: {filepath}')
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Main
|
||||
# ============================================================================
|
||||
|
||||
if __name__ == "__main__":
|
||||
print("="*70)
|
||||
print("PraisonAI Agents - Real-World Execution Benchmark")
|
||||
print("="*70)
|
||||
print("\nThis benchmark measures actual agent execution time including:")
|
||||
print(" - LLM API calls")
|
||||
print(" - Tool execution")
|
||||
print(" - Multi-agent coordination")
|
||||
print("\nModel: gpt-4o-mini")
|
||||
print("Iterations: 3 per test")
|
||||
|
||||
versions = get_package_versions()
|
||||
print(f"\nPackage versions:")
|
||||
for pkg, ver in versions.items():
|
||||
print(f" {pkg}: {ver}")
|
||||
|
||||
results = {}
|
||||
|
||||
# Single Agent Tests
|
||||
print("\n" + "="*70)
|
||||
print("SINGLE AGENT BENCHMARK")
|
||||
print("="*70)
|
||||
|
||||
print("\nTesting PraisonAI single agent...")
|
||||
results['praisonai_single'] = benchmark_praisonai_single()
|
||||
|
||||
print("\nTesting Agno single agent...")
|
||||
results['agno_single'] = benchmark_agno_single()
|
||||
|
||||
# 2-Agent Tests
|
||||
print("\n" + "="*70)
|
||||
print("2-AGENT BENCHMARK")
|
||||
print("="*70)
|
||||
|
||||
print("\nTesting PraisonAI 2 agents...")
|
||||
results['praisonai_two_agents'] = benchmark_praisonai_two_agents()
|
||||
|
||||
print("\nTesting Agno 2 agents (Team)...")
|
||||
results['agno_two_agents'] = benchmark_agno_two_agents()
|
||||
|
||||
# 3-Agent Tests
|
||||
print("\n" + "="*70)
|
||||
print("3-AGENT BENCHMARK")
|
||||
print("="*70)
|
||||
|
||||
print("\nTesting PraisonAI 3 agents...")
|
||||
results['praisonai_three_agents'] = benchmark_praisonai_three_agents()
|
||||
|
||||
print("\nTesting Agno 3 agents (Team)...")
|
||||
results['agno_three_agents'] = benchmark_agno_three_agents()
|
||||
|
||||
# Summary
|
||||
print("\n" + "="*70)
|
||||
print("RESULTS SUMMARY")
|
||||
print("="*70)
|
||||
|
||||
print(f"\n{'Test':<20} {'PraisonAI (avg)':<18} {'Agno (avg)':<15} {'Difference':<20}")
|
||||
print("-" * 73)
|
||||
|
||||
tests = [
|
||||
('Single Agent', 'praisonai_single', 'agno_single'),
|
||||
('2 Agents', 'praisonai_two_agents', 'agno_two_agents'),
|
||||
('3 Agents', 'praisonai_three_agents', 'agno_three_agents'),
|
||||
]
|
||||
|
||||
for name, praison_key, agno_key in tests:
|
||||
praison_avg = results[praison_key].avg_time
|
||||
agno_avg = results[agno_key].avg_time
|
||||
|
||||
if praison_avg < agno_avg:
|
||||
diff = f'PraisonAI {agno_avg/praison_avg:.2f}x faster'
|
||||
else:
|
||||
diff = f'Agno {praison_avg/agno_avg:.2f}x faster'
|
||||
|
||||
print(f"{name:<20} {praison_avg:<18.2f}s {agno_avg:<15.2f}s {diff:<20}")
|
||||
|
||||
print("\n" + "="*70)
|
||||
|
||||
# Save results
|
||||
save_results(results, versions)
|
||||
@@ -0,0 +1,194 @@
|
||||
❯ python benchmarks/performance_benchmark.py
|
||||
/Users/praison/praisonai-package/src/praisonai-agents/praisonaiagents/_warning_patch.py:47: UserWarning: Mixing V1 models and V2 models (or constructs, like `TypeAdapter`) is not supported. Please upgrade `Settings` to V2.
|
||||
_original_warn(message, category, stacklevel, source)
|
||||
[13:04:15] INFO [13:04:15] config.py:50
|
||||
config.py:50 INFO
|
||||
Using config path:
|
||||
/Users/praison/.con
|
||||
fig/crewai/settings
|
||||
.json
|
||||
======================================================================
|
||||
PraisonAI Agents - Comprehensive Performance Benchmark
|
||||
======================================================================
|
||||
|
||||
This benchmark measures:
|
||||
1. Agent instantiation time (how fast an agent object is created)
|
||||
2. Memory footprint (how much memory an agent uses)
|
||||
|
||||
Methodology:
|
||||
- 10 warmup runs (not measured)
|
||||
- 1000 measured iterations
|
||||
- tracemalloc for memory measurement
|
||||
- Baseline subtraction for accuracy
|
||||
|
||||
Frameworks available:
|
||||
- PraisonAI: ✅
|
||||
- Agno: ✅
|
||||
- PydanticAI: ✅
|
||||
- OpenAI Agents SDK: ✅
|
||||
- LangGraph: ✅
|
||||
- CrewAI: ✅
|
||||
|
||||
======================================================================
|
||||
Testing PraisonAI Agents
|
||||
======================================================================
|
||||
|
||||
============================================================
|
||||
Benchmarking: PraisonAI
|
||||
============================================================
|
||||
Warmup runs: 10
|
||||
Measured iterations: 1000
|
||||
|
||||
Measuring runtime...
|
||||
Measuring memory...
|
||||
|
||||
Results for PraisonAI:
|
||||
Average Time: 0.000003 seconds (3.20 μs)
|
||||
Min Time: 0.000003 seconds
|
||||
Max Time: 0.000025 seconds
|
||||
Std Dev: 0.000001 seconds
|
||||
Median Time: 0.000003 seconds
|
||||
Avg Memory: 0.003678 MiB (3.77 KiB)
|
||||
|
||||
======================================================================
|
||||
Testing PraisonAI Agents (LiteLLM)
|
||||
======================================================================
|
||||
|
||||
============================================================
|
||||
Benchmarking: PraisonAI (LiteLLM)
|
||||
============================================================
|
||||
Warmup runs: 10
|
||||
Measured iterations: 1000
|
||||
|
||||
Measuring runtime...
|
||||
Measuring memory...
|
||||
|
||||
Results for PraisonAI (LiteLLM):
|
||||
Average Time: 0.000007 seconds (7.31 μs)
|
||||
Min Time: 0.000007 seconds
|
||||
Max Time: 0.000060 seconds
|
||||
Std Dev: 0.000002 seconds
|
||||
Median Time: 0.000007 seconds
|
||||
Avg Memory: 0.006419 MiB (6.57 KiB)
|
||||
|
||||
======================================================================
|
||||
Testing Agno
|
||||
======================================================================
|
||||
|
||||
============================================================
|
||||
Benchmarking: Agno
|
||||
============================================================
|
||||
Warmup runs: 10
|
||||
Measured iterations: 1000
|
||||
|
||||
Measuring runtime...
|
||||
Measuring memory...
|
||||
|
||||
Results for Agno:
|
||||
Average Time: 0.000005 seconds (5.30 μs)
|
||||
Min Time: 0.000005 seconds
|
||||
Max Time: 0.000039 seconds
|
||||
Std Dev: 0.000002 seconds
|
||||
Median Time: 0.000005 seconds
|
||||
Avg Memory: 0.006599 MiB (6.76 KiB)
|
||||
|
||||
======================================================================
|
||||
Testing PydanticAI
|
||||
======================================================================
|
||||
|
||||
============================================================
|
||||
Benchmarking: PydanticAI Agent Instantiation
|
||||
============================================================
|
||||
Warmup runs: 10
|
||||
Measured iterations: 1000
|
||||
|
||||
Measuring runtime...
|
||||
Measuring memory...
|
||||
|
||||
Results for PydanticAI Agent Instantiation:
|
||||
Average Time: 0.000247 seconds (247.34 μs)
|
||||
Min Time: 0.000218 seconds
|
||||
Max Time: 0.000552 seconds
|
||||
Std Dev: 0.000031 seconds
|
||||
Median Time: 0.000237 seconds
|
||||
Avg Memory: 0.028486 MiB (29.17 KiB)
|
||||
|
||||
======================================================================
|
||||
Testing OpenAI Agents SDK
|
||||
======================================================================
|
||||
|
||||
============================================================
|
||||
Benchmarking: OpenAI Agents SDK Instantiation
|
||||
============================================================
|
||||
Warmup runs: 10
|
||||
Measured iterations: 1000
|
||||
|
||||
Measuring runtime...
|
||||
Measuring memory...
|
||||
|
||||
Results for OpenAI Agents SDK Instantiation:
|
||||
Average Time: 0.000282 seconds (282.25 μs)
|
||||
Min Time: 0.000247 seconds
|
||||
Max Time: 0.001110 seconds
|
||||
Std Dev: 0.000055 seconds
|
||||
Median Time: 0.000263 seconds
|
||||
Avg Memory: 0.029032 MiB (29.73 KiB)
|
||||
|
||||
======================================================================
|
||||
Testing LangGraph
|
||||
======================================================================
|
||||
|
||||
============================================================
|
||||
Benchmarking: LangGraph Agent Instantiation
|
||||
============================================================
|
||||
Warmup runs: 10
|
||||
Measured iterations: 1000
|
||||
|
||||
Measuring runtime...
|
||||
Measuring memory...
|
||||
|
||||
Results for LangGraph Agent Instantiation:
|
||||
Average Time: 0.003200 seconds (3200.15 μs)
|
||||
Min Time: 0.002808 seconds
|
||||
Max Time: 0.116363 seconds
|
||||
Std Dev: 0.003628 seconds
|
||||
Median Time: 0.002972 seconds
|
||||
Avg Memory: 0.207509 MiB (212.49 KiB)
|
||||
|
||||
======================================================================
|
||||
Testing CrewAI
|
||||
======================================================================
|
||||
|
||||
============================================================
|
||||
Benchmarking: CrewAI Agent Instantiation
|
||||
============================================================
|
||||
Warmup runs: 10
|
||||
Measured iterations: 1000
|
||||
|
||||
Measuring runtime...
|
||||
Measuring memory...
|
||||
|
||||
Results for CrewAI Agent Instantiation:
|
||||
Average Time: 0.016772 seconds (16771.69 μs)
|
||||
Min Time: 0.014404 seconds
|
||||
Max Time: 0.091841 seconds
|
||||
Std Dev: 0.003445 seconds
|
||||
Median Time: 0.015816 seconds
|
||||
Avg Memory: 0.065050 MiB (66.61 KiB)
|
||||
|
||||
======================================================================
|
||||
RESULTS SUMMARY
|
||||
======================================================================
|
||||
|
||||
Framework Avg Time (μs) Avg Memory (KiB) Relative
|
||||
-------------------------------------------------------------------------
|
||||
PraisonAI 3.20 3.77 1.00x
|
||||
Agno 5.30 6.76 1.66x
|
||||
PraisonAI (LiteLLM) 7.31 6.57 2.28x
|
||||
PydanticAI 247.34 29.17 77.25x
|
||||
OpenAI Agents SDK 282.25 29.73 88.15x
|
||||
LangGraph 3200.15 212.49 999.46x
|
||||
CrewAI 16771.69 66.61 5238.06x
|
||||
|
||||
======================================================================
|
||||
❯
|
||||
@@ -33,8 +33,7 @@ def measure_instantiation(create_fn, iterations=ITERATIONS):
|
||||
|
||||
|
||||
def run_benchmark():
|
||||
"""Run the benchmark across all available frameworks."""
|
||||
tools = [sample_tool]
|
||||
"""Run the benchmark across all available frameworks (without tools)."""
|
||||
results = {}
|
||||
|
||||
print('=' * 60)
|
||||
@@ -43,15 +42,16 @@ def run_benchmark():
|
||||
print(f'\nIterations: {ITERATIONS}')
|
||||
print('Metric: Agent instantiation time (microseconds)\n')
|
||||
|
||||
# PraisonAI
|
||||
# PraisonAI (without tools)
|
||||
print("Testing PraisonAI...")
|
||||
from praisonaiagents import Agent as PraisonAgent
|
||||
|
||||
results['PraisonAI'] = measure_instantiation(
|
||||
lambda: PraisonAgent(name='Test', llm='gpt-4o-mini', tools=tools, verbose=False)
|
||||
lambda: PraisonAgent(name='Test', llm='gpt-4o-mini', verbose=False)
|
||||
)
|
||||
|
||||
results['PraisonAI (LiteLLM)'] = measure_instantiation(
|
||||
lambda: PraisonAgent(name='Test', llm='openai/gpt-4o-mini', tools=tools, verbose=False)
|
||||
lambda: PraisonAgent(name='Test', llm='openai/gpt-4o-mini', verbose=False)
|
||||
)
|
||||
|
||||
# Other frameworks for comparison
|
||||
@@ -61,7 +61,7 @@ def run_benchmark():
|
||||
from agno.agent import Agent as AgnoAgent
|
||||
from agno.models.openai import OpenAIChat
|
||||
results['Agno'] = measure_instantiation(
|
||||
lambda: AgnoAgent(model=OpenAIChat(id='gpt-4o-mini'), tools=tools)
|
||||
lambda: AgnoAgent(model=OpenAIChat(id='gpt-4o-mini'))
|
||||
)
|
||||
except ImportError:
|
||||
pass
|
||||
@@ -82,6 +82,43 @@ def run_benchmark():
|
||||
except ImportError:
|
||||
pass
|
||||
|
||||
try:
|
||||
from langchain_core.tools import tool as langchain_tool
|
||||
from langchain_openai import ChatOpenAI
|
||||
from langgraph.prebuilt import create_react_agent
|
||||
|
||||
@langchain_tool
|
||||
def get_weather_lg(city: Literal['nyc', 'sf']):
|
||||
"""Get weather info."""
|
||||
return 'sunny' if city == 'sf' else 'cloudy'
|
||||
|
||||
results['LangGraph'] = measure_instantiation(
|
||||
lambda: create_react_agent(model=ChatOpenAI(model='gpt-4o-mini'), tools=[get_weather_lg])
|
||||
)
|
||||
except ImportError:
|
||||
pass
|
||||
|
||||
try:
|
||||
from crewai.agent import Agent as CrewAgent
|
||||
from crewai.tools import tool as crewai_tool
|
||||
|
||||
@crewai_tool("Weather Tool")
|
||||
def get_weather_crew(city: Literal['nyc', 'sf']):
|
||||
"""Get weather info."""
|
||||
return 'sunny' if city == 'sf' else 'cloudy'
|
||||
|
||||
results['CrewAI'] = measure_instantiation(
|
||||
lambda: CrewAgent(
|
||||
role='Weather Agent',
|
||||
goal='Provide weather info',
|
||||
backstory='A weather expert',
|
||||
tools=[get_weather_crew],
|
||||
verbose=False
|
||||
)
|
||||
)
|
||||
except ImportError:
|
||||
pass
|
||||
|
||||
# Print results
|
||||
print('\n' + '=' * 60)
|
||||
print('RESULTS')
|
||||
@@ -99,5 +136,108 @@ def run_benchmark():
|
||||
return results
|
||||
|
||||
|
||||
def get_package_versions():
|
||||
"""Get version numbers for benchmarked packages."""
|
||||
from importlib.metadata import version as get_version
|
||||
|
||||
packages = {
|
||||
'PraisonAI': 'praisonaiagents',
|
||||
'Agno': 'agno',
|
||||
'PydanticAI': 'pydantic-ai',
|
||||
'OpenAI Agents SDK': 'openai-agents',
|
||||
'LangGraph': 'langgraph',
|
||||
'CrewAI': 'crewai'
|
||||
}
|
||||
|
||||
versions = {}
|
||||
for display_name, pkg_name in packages.items():
|
||||
try:
|
||||
versions[display_name] = get_version(pkg_name)
|
||||
except Exception:
|
||||
versions[display_name] = 'not installed'
|
||||
|
||||
return versions
|
||||
|
||||
|
||||
def save_results(results: dict, filename: str = 'BENCHMARK_RESULTS.md'):
|
||||
"""Save benchmark results to a markdown file and update README."""
|
||||
import os
|
||||
import re
|
||||
from datetime import datetime
|
||||
|
||||
filepath = os.path.join(os.path.dirname(__file__), filename)
|
||||
versions = get_package_versions()
|
||||
|
||||
# Find the fastest result (baseline)
|
||||
sorted_results = sorted(results.items(), key=lambda x: x[1])
|
||||
fastest_time = sorted_results[0][1] if sorted_results else 1
|
||||
|
||||
# Build the table rows
|
||||
table_rows = []
|
||||
for name, avg in sorted_results:
|
||||
ratio = avg / fastest_time
|
||||
if ratio == 1.0:
|
||||
table_rows.append(f'| **{name}** | **{avg:.2f}** | **1.00x (fastest)** |')
|
||||
else:
|
||||
avg_str = f'{avg:,.2f}' if avg >= 1000 else f'{avg:.2f}'
|
||||
ratio_str = f'{ratio:,.0f}x' if ratio >= 100 else f'{ratio:.2f}x'
|
||||
table_rows.append(f'| {name} | {avg_str} | {ratio_str} |')
|
||||
|
||||
# Save to BENCHMARK_RESULTS.md
|
||||
with open(filepath, 'w') as f:
|
||||
f.write('# PraisonAI Agents - Benchmark Results\n\n')
|
||||
f.write(f'**Generated:** {datetime.now().strftime("%Y-%m-%d %H:%M:%S")}\n')
|
||||
f.write(f'**Iterations:** {ITERATIONS}\n')
|
||||
f.write('**Test:** Agent instantiation (without tools)\n\n')
|
||||
f.write('## Results\n\n')
|
||||
f.write('| Framework | Avg Time (μs) | Relative |\n')
|
||||
f.write('|-----------|---------------|----------|\n')
|
||||
f.write('\n'.join(table_rows) + '\n')
|
||||
f.write('\n## Package Versions\n\n')
|
||||
f.write('| Package | Version |\n')
|
||||
f.write('|---------|--------|\n')
|
||||
for pkg, ver in versions.items():
|
||||
f.write(f'| {pkg} | {ver} |\n')
|
||||
f.write('\n## How to Reproduce\n\n')
|
||||
f.write('```bash\n')
|
||||
f.write('cd praisonai-agents\n')
|
||||
f.write('python benchmarks/simple_benchmark.py\n')
|
||||
f.write('```\n')
|
||||
|
||||
print(f'\nResults saved to: {filepath}')
|
||||
|
||||
# Also update the main README.md
|
||||
readme_path = os.path.join(os.path.dirname(__file__), '..', '..', '..', 'README.md')
|
||||
if os.path.exists(readme_path):
|
||||
update_readme(readme_path, table_rows)
|
||||
|
||||
return filepath
|
||||
|
||||
|
||||
def update_readme(readme_path: str, table_rows: list):
|
||||
"""Update the performance section in README.md with latest results."""
|
||||
import re
|
||||
|
||||
with open(readme_path, 'r') as f:
|
||||
content = f.read()
|
||||
|
||||
# Build the new table
|
||||
new_table = '''| Framework | Avg Time (μs) | Relative |
|
||||
|-----------|---------------|----------|
|
||||
''' + '\n'.join(table_rows)
|
||||
|
||||
# Pattern to match the performance table
|
||||
pattern = r'(\| Framework \| Avg Time \(μs\) \| Relative \|\n\|[-|]+\|\n)(\|[^\n]+\|\n)+'
|
||||
|
||||
if re.search(pattern, content):
|
||||
content = re.sub(pattern, new_table + '\n', content)
|
||||
|
||||
with open(readme_path, 'w') as f:
|
||||
f.write(content)
|
||||
|
||||
print(f'README.md updated: {readme_path}')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
run_benchmark()
|
||||
results = run_benchmark()
|
||||
save_results(results)
|
||||
@@ -0,0 +1,213 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
PraisonAI Agents - Tools Benchmark
|
||||
|
||||
Compares agent instantiation times WITH TOOLS across frameworks.
|
||||
|
||||
Usage:
|
||||
python benchmarks/tools_benchmark.py
|
||||
"""
|
||||
|
||||
import time
|
||||
from typing import Literal
|
||||
from importlib.metadata import version as get_version
|
||||
|
||||
|
||||
ITERATIONS = 100
|
||||
|
||||
|
||||
def sample_tool(city: Literal['nyc', 'sf']):
|
||||
"""Sample tool for benchmark testing."""
|
||||
if city == 'nyc':
|
||||
return 'cloudy'
|
||||
return 'sunny'
|
||||
|
||||
|
||||
def measure_instantiation(create_fn, iterations=ITERATIONS):
|
||||
"""Measure average instantiation time in microseconds."""
|
||||
times = []
|
||||
|
||||
# Warmup
|
||||
for _ in range(10):
|
||||
create_fn()
|
||||
|
||||
# Measure
|
||||
for _ in range(iterations):
|
||||
start = time.perf_counter()
|
||||
create_fn()
|
||||
elapsed = time.perf_counter() - start
|
||||
times.append(elapsed * 1_000_000) # Convert to microseconds
|
||||
|
||||
return sum(times) / len(times)
|
||||
|
||||
|
||||
def run_benchmark():
|
||||
"""Run the benchmark across all available frameworks WITH TOOLS."""
|
||||
tools = [sample_tool]
|
||||
results = {}
|
||||
|
||||
print('=' * 60)
|
||||
print('PraisonAI Agents - Tools Benchmark')
|
||||
print('=' * 60)
|
||||
print(f'\nIterations: {ITERATIONS}')
|
||||
print('Metric: Agent instantiation time WITH TOOLS (microseconds)\n')
|
||||
|
||||
# PraisonAI with tools
|
||||
print("Testing PraisonAI...")
|
||||
from praisonaiagents import Agent as PraisonAgent
|
||||
|
||||
results['PraisonAI'] = measure_instantiation(
|
||||
lambda: PraisonAgent(name='Test', llm='gpt-4o-mini', tools=tools, verbose=False)
|
||||
)
|
||||
|
||||
results['PraisonAI (LiteLLM)'] = measure_instantiation(
|
||||
lambda: PraisonAgent(name='Test', llm='openai/gpt-4o-mini', tools=tools, verbose=False)
|
||||
)
|
||||
|
||||
# Other frameworks with tools
|
||||
print("Testing other frameworks...")
|
||||
|
||||
try:
|
||||
from agno.agent import Agent as AgnoAgent
|
||||
from agno.models.openai import OpenAIChat
|
||||
results['Agno'] = measure_instantiation(
|
||||
lambda: AgnoAgent(model=OpenAIChat(id='gpt-4o-mini'), tools=tools)
|
||||
)
|
||||
except ImportError:
|
||||
pass
|
||||
|
||||
try:
|
||||
from agents import Agent as OpenAIAgent, function_tool
|
||||
results['OpenAI Agents SDK'] = measure_instantiation(
|
||||
lambda: OpenAIAgent(name='Test', model='gpt-4o-mini', tools=[function_tool(sample_tool)])
|
||||
)
|
||||
except ImportError:
|
||||
pass
|
||||
|
||||
try:
|
||||
from langchain_core.tools import tool as langchain_tool
|
||||
from langchain_openai import ChatOpenAI
|
||||
from langgraph.prebuilt import create_react_agent
|
||||
|
||||
@langchain_tool
|
||||
def get_weather_lg(city: Literal['nyc', 'sf']):
|
||||
"""Get weather info."""
|
||||
return 'sunny' if city == 'sf' else 'cloudy'
|
||||
|
||||
results['LangGraph'] = measure_instantiation(
|
||||
lambda: create_react_agent(model=ChatOpenAI(model='gpt-4o-mini'), tools=[get_weather_lg])
|
||||
)
|
||||
except ImportError:
|
||||
pass
|
||||
|
||||
try:
|
||||
from crewai.agent import Agent as CrewAgent
|
||||
from crewai.tools import tool as crewai_tool
|
||||
|
||||
@crewai_tool("Weather Tool")
|
||||
def get_weather_crew(city: Literal['nyc', 'sf']):
|
||||
"""Get weather info."""
|
||||
return 'sunny' if city == 'sf' else 'cloudy'
|
||||
|
||||
results['CrewAI'] = measure_instantiation(
|
||||
lambda: CrewAgent(
|
||||
role='Weather Agent',
|
||||
goal='Provide weather info',
|
||||
backstory='A weather expert',
|
||||
tools=[get_weather_crew],
|
||||
verbose=False
|
||||
)
|
||||
)
|
||||
except ImportError:
|
||||
pass
|
||||
|
||||
# Print results
|
||||
print('\n' + '=' * 60)
|
||||
print('RESULTS (WITH TOOLS)')
|
||||
print('=' * 60)
|
||||
|
||||
sorted_results = sorted(results.items(), key=lambda x: x[1])
|
||||
baseline = sorted_results[0][1] if sorted_results else 1
|
||||
|
||||
print(f"\n{'Framework':<25} {'Avg Time (μs)':<15} {'Relative':<10}")
|
||||
print('-' * 50)
|
||||
|
||||
for name, avg in sorted_results:
|
||||
ratio = avg / baseline
|
||||
print(f'{name:<25} {avg:<15.2f} {ratio:.2f}x')
|
||||
|
||||
print('\n' + '=' * 60)
|
||||
return results
|
||||
|
||||
|
||||
def get_package_versions():
|
||||
"""Get version numbers for benchmarked packages."""
|
||||
packages = {
|
||||
'PraisonAI': 'praisonaiagents',
|
||||
'Agno': 'agno',
|
||||
'OpenAI Agents SDK': 'openai-agents',
|
||||
'LangGraph': 'langgraph',
|
||||
'CrewAI': 'crewai'
|
||||
}
|
||||
|
||||
versions = {}
|
||||
for display_name, pkg_name in packages.items():
|
||||
try:
|
||||
versions[display_name] = get_version(pkg_name)
|
||||
except Exception:
|
||||
versions[display_name] = 'not installed'
|
||||
|
||||
return versions
|
||||
|
||||
|
||||
def save_results(results: dict, filename: str = 'TOOLS_BENCHMARK_RESULTS.md'):
|
||||
"""Save benchmark results to a markdown file."""
|
||||
import os
|
||||
from datetime import datetime
|
||||
|
||||
filepath = os.path.join(os.path.dirname(__file__), filename)
|
||||
versions = get_package_versions()
|
||||
|
||||
# Find the fastest result (baseline)
|
||||
sorted_results = sorted(results.items(), key=lambda x: x[1])
|
||||
fastest_time = sorted_results[0][1] if sorted_results else 1
|
||||
|
||||
# Build the table rows
|
||||
table_rows = []
|
||||
for name, avg in sorted_results:
|
||||
ratio = avg / fastest_time
|
||||
if ratio == 1.0:
|
||||
table_rows.append(f'| **{name}** | **{avg:.2f}** | **1.00x (fastest)** |')
|
||||
else:
|
||||
avg_str = f'{avg:,.2f}' if avg >= 1000 else f'{avg:.2f}'
|
||||
ratio_str = f'{ratio:,.0f}x' if ratio >= 100 else f'{ratio:.2f}x'
|
||||
table_rows.append(f'| {name} | {avg_str} | {ratio_str} |')
|
||||
|
||||
# Save to file
|
||||
with open(filepath, 'w') as f:
|
||||
f.write('# PraisonAI Agents - Tools Benchmark Results\n\n')
|
||||
f.write(f'**Generated:** {datetime.now().strftime("%Y-%m-%d %H:%M:%S")}\n')
|
||||
f.write(f'**Iterations:** {ITERATIONS}\n')
|
||||
f.write('**Test:** Agent instantiation WITH TOOLS\n\n')
|
||||
f.write('## Results\n\n')
|
||||
f.write('| Framework | Avg Time (μs) | Relative |\n')
|
||||
f.write('|-----------|---------------|----------|\n')
|
||||
f.write('\n'.join(table_rows) + '\n')
|
||||
f.write('\n## Package Versions\n\n')
|
||||
f.write('| Package | Version |\n')
|
||||
f.write('|---------|--------|\n')
|
||||
for pkg, ver in versions.items():
|
||||
f.write(f'| {pkg} | {ver} |\n')
|
||||
f.write('\n## How to Reproduce\n\n')
|
||||
f.write('```bash\n')
|
||||
f.write('cd praisonai-agents\n')
|
||||
f.write('python benchmarks/tools_benchmark.py\n')
|
||||
f.write('```\n')
|
||||
|
||||
print(f'\nResults saved to: {filepath}')
|
||||
return filepath
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
results = run_benchmark()
|
||||
save_results(results)
|
||||
@@ -1370,6 +1370,61 @@ Context:
|
||||
agent_names = ", ".join([agent.name for agent in self.agents])
|
||||
print(f"📊 Available agents for this endpoint ({len(self.agents)}): {agent_names}")
|
||||
|
||||
# Create per-agent endpoints for individual agent access
|
||||
# This allows n8n and other tools to call specific agents
|
||||
agents_dict = {agent.name.lower().replace(' ', '_'): agent for agent in self.agents}
|
||||
|
||||
# Add GET endpoint to list available agents
|
||||
@_agents_shared_apps[port].get(f"{path}/list")
|
||||
async def list_agents():
|
||||
return {
|
||||
"agents": [
|
||||
{"name": agent.name, "id": agent.name.lower().replace(' ', '_')}
|
||||
for agent in self.agents
|
||||
]
|
||||
}
|
||||
|
||||
# Add per-agent POST endpoints
|
||||
for agent_id, agent_instance in agents_dict.items():
|
||||
agent_path = f"{path}/{agent_id}"
|
||||
|
||||
# Create a closure to capture the agent instance
|
||||
def create_agent_handler(agent):
|
||||
async def handle_single_agent(request: Request):
|
||||
try:
|
||||
request_data = await request.json()
|
||||
query = request_data.get("query", "")
|
||||
if not query:
|
||||
raise HTTPException(status_code=400, detail="Missing 'query' field")
|
||||
except:
|
||||
raise HTTPException(status_code=400, detail="Invalid JSON body")
|
||||
|
||||
try:
|
||||
if asyncio.iscoroutinefunction(agent.chat):
|
||||
response = await agent.achat(query)
|
||||
else:
|
||||
loop = asyncio.get_running_loop()
|
||||
response = await loop.run_in_executor(None, lambda q=query: agent.chat(q))
|
||||
|
||||
return {
|
||||
"agent": agent.name,
|
||||
"query": query,
|
||||
"response": response
|
||||
}
|
||||
except Exception as e:
|
||||
logging.error(f"Error with agent {agent.name}: {str(e)}", exc_info=True)
|
||||
return JSONResponse(
|
||||
status_code=500,
|
||||
content={"error": f"Agent error: {str(e)}"}
|
||||
)
|
||||
return handle_single_agent
|
||||
|
||||
# Register the endpoint
|
||||
_agents_shared_apps[port].post(agent_path)(create_agent_handler(agent_instance))
|
||||
_agents_registered_endpoints[port][agent_path] = f"{endpoint_id}_{agent_id}"
|
||||
|
||||
print(f"🔗 Per-agent endpoints: {', '.join([f'{path}/{aid}' for aid in agents_dict.keys()])}")
|
||||
|
||||
# Start the server if it's not already running for this port
|
||||
if not _agents_server_started.get(port, False):
|
||||
# Mark the server as started first to prevent duplicate starts
|
||||
|
||||
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "praisonaiagents"
|
||||
version = "0.1.13"
|
||||
version = "0.1.14"
|
||||
description = "Praison AI agents for completing complex tasks with Self Reflection Agents"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.10"
|
||||
|
||||
Generated
+1
-1
@@ -2422,7 +2422,7 @@ wheels = [
|
||||
|
||||
[[package]]
|
||||
name = "praisonaiagents"
|
||||
version = "0.1.13"
|
||||
version = "0.1.14"
|
||||
source = { editable = "." }
|
||||
dependencies = [
|
||||
{ name = "aiohttp" },
|
||||
|
||||
@@ -34,6 +34,7 @@ PraisonAI is a production-ready Multi-AI Agents framework with self-reflection,
|
||||
|
||||
## 📑 Table of Contents
|
||||
|
||||
- [⚡ Performance](#-performance)
|
||||
- [🚀 Quick Start](#-quick-start)
|
||||
- [📦 Installation](#-installation)
|
||||
- [Python SDK](#python-sdk)
|
||||
@@ -73,6 +74,32 @@ PraisonAI is a production-ready Multi-AI Agents framework with self-reflection,
|
||||
|
||||
---
|
||||
|
||||
## ⚡ Performance
|
||||
|
||||
PraisonAI Agents is the **fastest AI agent framework** for agent instantiation.
|
||||
|
||||
| Framework | Avg Time (μs) | Relative |
|
||||
|-----------|---------------|----------|
|
||||
| **PraisonAI** | **3.77** | **1.00x (fastest)** |
|
||||
| OpenAI Agents SDK | 5.26 | 1.39x |
|
||||
| Agno | 5.64 | 1.49x |
|
||||
| PraisonAI (LiteLLM) | 7.56 | 2.00x |
|
||||
| PydanticAI | 226.94 | 60.16x |
|
||||
| LangGraph | 4,558.71 | 1,209x |
|
||||
| CrewAI | 15,607.92 | 4,138x |
|
||||
|
||||
<details>
|
||||
<summary>Run benchmarks yourself</summary>
|
||||
|
||||
```bash
|
||||
cd praisonai-agents
|
||||
python benchmarks/simple_benchmark.py
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
---
|
||||
|
||||
## 🚀 Quick Start
|
||||
|
||||
Get started with PraisonAI in under 1 minute:
|
||||
@@ -1392,6 +1419,36 @@ praisonai commit
|
||||
praisonai commit --push
|
||||
```
|
||||
|
||||
### Serve CLI (API Server):
|
||||
```bash
|
||||
# Start API server for agents defined in YAML
|
||||
praisonai serve agents.yaml
|
||||
|
||||
# With custom port and host
|
||||
praisonai serve agents.yaml --port 8005 --host 0.0.0.0
|
||||
|
||||
# Alternative flag style
|
||||
praisonai agents.yaml --serve
|
||||
|
||||
# The server provides:
|
||||
# POST /agents - Run all agents sequentially
|
||||
# POST /agents/{name} - Run specific agent (e.g., /agents/researcher)
|
||||
# GET /agents/list - List available agents
|
||||
```
|
||||
|
||||
### n8n Integration CLI:
|
||||
```bash
|
||||
# Export workflow to n8n and open in browser
|
||||
praisonai agents.yaml --n8n
|
||||
|
||||
# With custom n8n URL
|
||||
praisonai agents.yaml --n8n --n8n-url http://localhost:5678
|
||||
|
||||
# Set N8N_API_KEY for auto-import
|
||||
export N8N_API_KEY="your-api-key"
|
||||
praisonai agents.yaml --n8n
|
||||
```
|
||||
|
||||
### @Mentions in Prompts:
|
||||
```bash
|
||||
# Include file content in prompt
|
||||
|
||||
@@ -4,60 +4,87 @@
|
||||
"nodes": [
|
||||
{
|
||||
"id": "trigger",
|
||||
"name": "Manual Trigger",
|
||||
"type": "n8n-nodes-base.manualTrigger",
|
||||
"typeVersion": 1,
|
||||
"name": "Webhook",
|
||||
"type": "n8n-nodes-base.webhook",
|
||||
"typeVersion": 2,
|
||||
"position": [
|
||||
250,
|
||||
300
|
||||
],
|
||||
"parameters": {}
|
||||
"webhookId": "dd7cf5cb72d19eae",
|
||||
"parameters": {
|
||||
"path": "create-movie-script-about-cat-in-mars",
|
||||
"httpMethod": "POST",
|
||||
"responseMode": "lastNode",
|
||||
"options": {}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "agent_0",
|
||||
"name": "Researcher",
|
||||
"type": "n8n-nodes-base.executeCommand",
|
||||
"typeVersion": 1,
|
||||
"type": "n8n-nodes-base.httpRequest",
|
||||
"typeVersion": 4.2,
|
||||
"position": [
|
||||
450,
|
||||
300
|
||||
],
|
||||
"parameters": {
|
||||
"command": "praisonai \"Research about Mars environment and cat behavior for the movie concept\"",
|
||||
"executeOnce": true
|
||||
}
|
||||
"method": "POST",
|
||||
"url": "http://127.0.0.1:8005/agents/researcher",
|
||||
"sendBody": true,
|
||||
"specifyBody": "json",
|
||||
"jsonBody": "={{ JSON.stringify({ query: $json.body?.query || $json.query || 'Research about Mars environment and cat behavior for the movie concept' }) }}",
|
||||
"options": {
|
||||
"timeout": 300000
|
||||
}
|
||||
},
|
||||
"notes": "Agent: Researcher\nTask: Research about Mars environment and cat behavior for the movie concept"
|
||||
},
|
||||
{
|
||||
"id": "agent_1",
|
||||
"name": "Narrative Designer",
|
||||
"type": "n8n-nodes-base.executeCommand",
|
||||
"typeVersion": 1,
|
||||
"type": "n8n-nodes-base.httpRequest",
|
||||
"typeVersion": 4.2,
|
||||
"position": [
|
||||
650,
|
||||
700,
|
||||
300
|
||||
],
|
||||
"parameters": {
|
||||
"command": "praisonai \"Create story concept based on research: {{previous_output}}\"",
|
||||
"executeOnce": true
|
||||
}
|
||||
"method": "POST",
|
||||
"url": "http://127.0.0.1:8005/agents/narrative_designer",
|
||||
"sendBody": true,
|
||||
"specifyBody": "json",
|
||||
"jsonBody": "={{ JSON.stringify({ query: $json.response || 'Create story concept based on research: previous_output' }) }}",
|
||||
"options": {
|
||||
"timeout": 300000
|
||||
}
|
||||
},
|
||||
"notes": "Agent: Narrative Designer\nTask: Create story concept based on research: {{previous_output}}"
|
||||
},
|
||||
{
|
||||
"id": "agent_2",
|
||||
"name": "Scriptwriter",
|
||||
"type": "n8n-nodes-base.executeCommand",
|
||||
"typeVersion": 1,
|
||||
"type": "n8n-nodes-base.httpRequest",
|
||||
"typeVersion": 4.2,
|
||||
"position": [
|
||||
850,
|
||||
950,
|
||||
300
|
||||
],
|
||||
"parameters": {
|
||||
"command": "praisonai \"Write the final movie script based on the story concept: {{previous_output}}\"",
|
||||
"executeOnce": true
|
||||
}
|
||||
"method": "POST",
|
||||
"url": "http://127.0.0.1:8005/agents/scriptwriter",
|
||||
"sendBody": true,
|
||||
"specifyBody": "json",
|
||||
"jsonBody": "={{ JSON.stringify({ query: $json.response || 'Write the final movie script based on the story concept: previous_output' }) }}",
|
||||
"options": {
|
||||
"timeout": 300000
|
||||
}
|
||||
},
|
||||
"notes": "Agent: Scriptwriter\nTask: Write the final movie script based on the story concept: {{previous_output}}"
|
||||
}
|
||||
],
|
||||
"connections": {
|
||||
"Manual Trigger": {
|
||||
"Webhook": {
|
||||
"main": [
|
||||
[
|
||||
{
|
||||
|
||||
@@ -3,8 +3,8 @@ class Praisonai < Formula
|
||||
|
||||
desc "AI tools for various AI applications"
|
||||
homepage "https://github.com/MervinPraison/PraisonAI"
|
||||
url "https://github.com/MervinPraison/PraisonAI/archive/refs/tags/v2.3.32.tar.gz"
|
||||
sha256 `curl -sL https://github.com/MervinPraison/PraisonAI/archive/refs/tags/v2.3.32.tar.gz | shasum -a 256`.split.first
|
||||
url "https://github.com/MervinPraison/PraisonAI/archive/refs/tags/v2.3.36.tar.gz"
|
||||
sha256 `curl -sL https://github.com/MervinPraison/PraisonAI/archive/refs/tags/v2.3.36.tar.gz | shasum -a 256`.split.first
|
||||
license "MIT"
|
||||
|
||||
depends_on "[email protected]"
|
||||
|
||||
@@ -35,7 +35,7 @@ class N8nHandler(FlagHandler):
|
||||
"""
|
||||
|
||||
def __init__(self, verbose: bool = False, n8n_url: str = "http://localhost:5678",
|
||||
use_execute_command: bool = True):
|
||||
use_execute_command: bool = False, api_url: str = "http://127.0.0.1:8005"):
|
||||
"""
|
||||
Initialize the n8n handler.
|
||||
|
||||
@@ -44,10 +44,11 @@ class N8nHandler(FlagHandler):
|
||||
n8n_url: Base URL of the n8n instance
|
||||
use_execute_command: Use Execute Command nodes (runs praisonai directly)
|
||||
instead of HTTP Request nodes
|
||||
api_url: PraisonAI API URL that n8n will call (for tunnel/cloud deployments)
|
||||
"""
|
||||
super().__init__(verbose=verbose)
|
||||
self.n8n_url = n8n_url
|
||||
self.praisonai_api_url = "http://localhost:8000"
|
||||
self.praisonai_api_url = api_url
|
||||
self.use_execute_command = use_execute_command
|
||||
|
||||
@property
|
||||
@@ -94,11 +95,28 @@ class N8nHandler(FlagHandler):
|
||||
if api_key and REQUESTS_AVAILABLE:
|
||||
workflow_id = self._create_workflow_via_api(workflow_json, api_key)
|
||||
|
||||
webhook_url = None
|
||||
if workflow_id:
|
||||
# Successfully created via API - open directly
|
||||
url = f"{self.n8n_url}/workflow/{workflow_id}"
|
||||
self.print_status("🚀 Workflow created in n8n!", "success")
|
||||
|
||||
# Activate the workflow so webhook is available
|
||||
if self._activate_workflow(workflow_id, api_key):
|
||||
self.print_status("✅ Workflow activated!", "success")
|
||||
|
||||
# Get webhook path from the workflow
|
||||
trigger_node = next((n for n in workflow_json["nodes"]
|
||||
if n.get("type") == "n8n-nodes-base.webhook"), None)
|
||||
if trigger_node:
|
||||
webhook_path = trigger_node["parameters"].get("path", "praisonai")
|
||||
webhook_url = f"{self.n8n_url}/webhook/{webhook_path}"
|
||||
self.print_status("", "info")
|
||||
self.print_status("🔗 Webhook URL (to trigger workflow):", "info")
|
||||
self.print_status(f" POST {webhook_url}", "info")
|
||||
else:
|
||||
self.print_status("⚠️ Could not activate workflow (activate manually in n8n)", "warning")
|
||||
|
||||
if open_browser:
|
||||
self.open_in_browser(url)
|
||||
self.print_status(f"🌐 Opening: {url}", "info")
|
||||
@@ -126,7 +144,8 @@ class N8nHandler(FlagHandler):
|
||||
"workflow": workflow_json,
|
||||
"url": url,
|
||||
"output_path": output_path,
|
||||
"workflow_id": workflow_id
|
||||
"workflow_id": workflow_id,
|
||||
"webhook_url": webhook_url
|
||||
}
|
||||
|
||||
def _create_workflow_via_api(self, workflow_json: Dict[str, Any],
|
||||
@@ -179,12 +198,52 @@ class N8nHandler(FlagHandler):
|
||||
self.log(f"Failed to create workflow via API: {e}", "warning")
|
||||
return None
|
||||
|
||||
def convert_yaml_to_n8n(self, yaml_path: str) -> Dict[str, Any]:
|
||||
def _activate_workflow(self, workflow_id: str, api_key: str) -> bool:
|
||||
"""
|
||||
Activate a workflow via n8n API.
|
||||
|
||||
Args:
|
||||
workflow_id: The workflow ID to activate
|
||||
api_key: n8n API key
|
||||
|
||||
Returns:
|
||||
True if successful, False otherwise
|
||||
"""
|
||||
if not REQUESTS_AVAILABLE:
|
||||
return False
|
||||
|
||||
try:
|
||||
response = requests.post(
|
||||
f"{self.n8n_url}/api/v1/workflows/{workflow_id}/activate",
|
||||
headers={
|
||||
"X-N8N-API-KEY": api_key,
|
||||
"Content-Type": "application/json"
|
||||
},
|
||||
timeout=30
|
||||
)
|
||||
|
||||
if response.status_code == 200:
|
||||
return True
|
||||
else:
|
||||
self.log(f"Activation error: {response.status_code} - {response.text}", "warning")
|
||||
return False
|
||||
|
||||
except Exception as e:
|
||||
self.log(f"Failed to activate workflow: {e}", "warning")
|
||||
return False
|
||||
|
||||
def convert_yaml_to_n8n(self, yaml_path: str, use_webhook: bool = True) -> Dict[str, Any]:
|
||||
"""
|
||||
Convert a PraisonAI YAML workflow to n8n JSON format.
|
||||
|
||||
The n8n workflow will have:
|
||||
- A webhook trigger to receive requests
|
||||
- One HTTP Request node per agent (calls /agents/{agent_name})
|
||||
- Each agent node passes its output to the next agent
|
||||
|
||||
Args:
|
||||
yaml_path: Path to the agents.yaml file
|
||||
use_webhook: Use webhook trigger for programmatic execution (default: True)
|
||||
|
||||
Returns:
|
||||
n8n workflow JSON as dictionary
|
||||
@@ -197,6 +256,9 @@ class N8nHandler(FlagHandler):
|
||||
workflow_name = yaml_content.get('name', 'PraisonAI Workflow')
|
||||
description = yaml_content.get('description', '')
|
||||
|
||||
# Generate a unique webhook path from workflow name
|
||||
webhook_path = workflow_name.lower().replace(' ', '-').replace('/', '-')[:50]
|
||||
|
||||
# Get agents and steps
|
||||
agents = yaml_content.get('agents', yaml_content.get('roles', {}))
|
||||
steps = yaml_content.get('steps', [])
|
||||
@@ -209,82 +271,43 @@ class N8nHandler(FlagHandler):
|
||||
nodes = []
|
||||
connections = {}
|
||||
|
||||
# Add manual trigger node
|
||||
trigger_node = self._create_trigger_node()
|
||||
# Add trigger node (webhook for programmatic execution, manual otherwise)
|
||||
trigger_node = self._create_trigger_node(use_webhook=use_webhook, webhook_path=webhook_path)
|
||||
nodes.append(trigger_node)
|
||||
|
||||
# Track previous node for connections
|
||||
prev_node_name = trigger_node["name"]
|
||||
x_position = 450
|
||||
|
||||
# Process each step
|
||||
x_position = 450 # Start position after trigger
|
||||
|
||||
# Create one HTTP node per agent step
|
||||
for i, step in enumerate(steps):
|
||||
if isinstance(step, dict):
|
||||
if 'parallel' in step:
|
||||
# Handle parallel steps
|
||||
parallel_nodes, parallel_connections = self._create_parallel_nodes(
|
||||
step['parallel'], x_position, agents
|
||||
)
|
||||
nodes.extend(parallel_nodes)
|
||||
connections.update(parallel_connections)
|
||||
|
||||
# Connect previous node to first parallel node
|
||||
if prev_node_name not in connections:
|
||||
connections[prev_node_name] = {"main": [[]]}
|
||||
for pnode in parallel_nodes:
|
||||
connections[prev_node_name]["main"][0].append({
|
||||
"node": pnode["name"],
|
||||
"type": "main",
|
||||
"index": 0
|
||||
})
|
||||
|
||||
x_position += 200 * len(parallel_nodes)
|
||||
prev_node_name = parallel_nodes[-1]["name"] if parallel_nodes else prev_node_name
|
||||
|
||||
elif 'route' in step:
|
||||
# Handle route/decision steps
|
||||
route_node = self._create_route_node(step, x_position, i)
|
||||
nodes.append(route_node)
|
||||
|
||||
# Connect previous node
|
||||
if prev_node_name not in connections:
|
||||
connections[prev_node_name] = {"main": [[]]}
|
||||
connections[prev_node_name]["main"][0].append({
|
||||
"node": route_node["name"],
|
||||
"type": "main",
|
||||
"index": 0
|
||||
})
|
||||
|
||||
prev_node_name = route_node["name"]
|
||||
x_position += 200
|
||||
|
||||
else:
|
||||
# Regular agent step
|
||||
agent_id = step.get('agent', '')
|
||||
action = step.get('action', step.get('description', ''))
|
||||
agent_config = agents.get(agent_id, {})
|
||||
|
||||
agent_node = self._create_agent_node(
|
||||
agent_id=agent_id,
|
||||
agent_config=agent_config,
|
||||
action=action,
|
||||
position=[x_position, 300],
|
||||
index=i
|
||||
)
|
||||
nodes.append(agent_node)
|
||||
|
||||
# Connect to previous node
|
||||
if prev_node_name not in connections:
|
||||
connections[prev_node_name] = {"main": [[]]}
|
||||
connections[prev_node_name]["main"][0].append({
|
||||
"node": agent_node["name"],
|
||||
"type": "main",
|
||||
"index": 0
|
||||
})
|
||||
|
||||
prev_node_name = agent_node["name"]
|
||||
x_position += 200
|
||||
agent_id = step.get('agent', '')
|
||||
action = step.get('action', step.get('description', ''))
|
||||
agent_config = agents.get(agent_id, {})
|
||||
|
||||
# Create HTTP node that calls the specific agent endpoint
|
||||
agent_node = self._create_per_agent_node(
|
||||
agent_id=agent_id,
|
||||
agent_config=agent_config,
|
||||
action=action,
|
||||
position=[x_position, 300],
|
||||
index=i,
|
||||
is_first=(i == 0)
|
||||
)
|
||||
nodes.append(agent_node)
|
||||
|
||||
# Connect to previous node
|
||||
if prev_node_name not in connections:
|
||||
connections[prev_node_name] = {"main": [[]]}
|
||||
connections[prev_node_name]["main"][0].append({
|
||||
"node": agent_node["name"],
|
||||
"type": "main",
|
||||
"index": 0
|
||||
})
|
||||
|
||||
prev_node_name = agent_node["name"]
|
||||
x_position += 250
|
||||
|
||||
# Build the complete workflow
|
||||
# Generate a unique ID based on workflow name
|
||||
@@ -334,8 +357,31 @@ class N8nHandler(FlagHandler):
|
||||
})
|
||||
return steps
|
||||
|
||||
def _create_trigger_node(self) -> Dict[str, Any]:
|
||||
"""Create a manual trigger node."""
|
||||
def _create_trigger_node(self, use_webhook: bool = False,
|
||||
webhook_path: str = "praisonai") -> Dict[str, Any]:
|
||||
"""Create a trigger node (webhook or manual).
|
||||
|
||||
Args:
|
||||
use_webhook: If True, creates a webhook trigger for programmatic execution
|
||||
webhook_path: Path for the webhook URL (only used if use_webhook=True)
|
||||
"""
|
||||
if use_webhook:
|
||||
# Generate a unique webhook ID
|
||||
webhook_id = hashlib.md5(webhook_path.encode()).hexdigest()[:16]
|
||||
return {
|
||||
"id": "trigger",
|
||||
"name": "Webhook",
|
||||
"type": "n8n-nodes-base.webhook",
|
||||
"typeVersion": 2,
|
||||
"position": [250, 300],
|
||||
"webhookId": webhook_id,
|
||||
"parameters": {
|
||||
"path": webhook_path,
|
||||
"httpMethod": "POST",
|
||||
"responseMode": "lastNode",
|
||||
"options": {}
|
||||
}
|
||||
}
|
||||
return {
|
||||
"id": "trigger",
|
||||
"name": "Manual Trigger",
|
||||
@@ -345,6 +391,94 @@ class N8nHandler(FlagHandler):
|
||||
"parameters": {}
|
||||
}
|
||||
|
||||
def _create_per_agent_node(self, agent_id: str, agent_config: Dict,
|
||||
action: str, position: List[int],
|
||||
index: int, is_first: bool = False) -> Dict[str, Any]:
|
||||
"""
|
||||
Create an HTTP Request node that calls a specific agent endpoint.
|
||||
|
||||
Args:
|
||||
agent_id: Agent identifier (used in URL path)
|
||||
agent_config: Agent configuration from YAML
|
||||
action: The action/task description for this agent
|
||||
position: Node position [x, y]
|
||||
index: Node index
|
||||
is_first: Whether this is the first agent (uses webhook input)
|
||||
|
||||
Returns:
|
||||
n8n HTTP Request node configuration
|
||||
"""
|
||||
agent_name = agent_config.get('name', agent_id.title())
|
||||
# Convert agent_id to URL-safe format (lowercase, underscores)
|
||||
agent_url_id = agent_id.lower().replace(' ', '_')
|
||||
|
||||
# Build the query - first agent uses webhook input, others use previous response
|
||||
# Escape single quotes and double braces (n8n expression syntax)
|
||||
escaped_action = action.replace("'", "\\'").replace("{{", "").replace("}}", "")
|
||||
if is_first:
|
||||
# First agent: use query from webhook body
|
||||
query_expr = "$json.body?.query || $json.query || '" + escaped_action + "'"
|
||||
else:
|
||||
# Subsequent agents: use previous agent's response + original action context
|
||||
query_expr = "$json.response || '" + escaped_action + "'"
|
||||
|
||||
return {
|
||||
"id": f"agent_{index}",
|
||||
"name": agent_name,
|
||||
"type": "n8n-nodes-base.httpRequest",
|
||||
"typeVersion": 4.2,
|
||||
"position": position,
|
||||
"parameters": {
|
||||
"method": "POST",
|
||||
"url": f"{self.praisonai_api_url}/agents/{agent_url_id}",
|
||||
"sendBody": True,
|
||||
"specifyBody": "json",
|
||||
"jsonBody": f"={{{{ JSON.stringify({{ query: {query_expr} }}) }}}}",
|
||||
"options": {
|
||||
"timeout": 300000 # 5 minute timeout per agent
|
||||
}
|
||||
},
|
||||
"notes": f"Agent: {agent_name}\nTask: {action[:100]}"
|
||||
}
|
||||
|
||||
def _create_praisonai_workflow_node(self, workflow_name: str,
|
||||
agent_names: List[str],
|
||||
position: List[int]) -> Dict[str, Any]:
|
||||
"""
|
||||
Create a single HTTP Request node that triggers the entire PraisonAI workflow.
|
||||
|
||||
The PraisonAI API will run all agents sequentially when called.
|
||||
|
||||
Args:
|
||||
workflow_name: Name of the workflow
|
||||
agent_names: List of agent names (for documentation)
|
||||
position: Node position [x, y]
|
||||
|
||||
Returns:
|
||||
n8n HTTP Request node configuration
|
||||
"""
|
||||
# The query will be passed from the webhook trigger
|
||||
# Using n8n expression to get the body from the webhook
|
||||
return {
|
||||
"id": "praisonai_workflow",
|
||||
"name": f"PraisonAI: {workflow_name}",
|
||||
"type": "n8n-nodes-base.httpRequest",
|
||||
"typeVersion": 4.2,
|
||||
"position": position,
|
||||
"parameters": {
|
||||
"method": "POST",
|
||||
"url": f"{self.praisonai_api_url}/agents",
|
||||
"sendBody": True,
|
||||
"specifyBody": "json",
|
||||
# Pass the query from webhook body, or use a default
|
||||
"jsonBody": "={{ JSON.stringify({ query: $json.body?.query || $json.query || 'Start workflow' }) }}",
|
||||
"options": {
|
||||
"timeout": 600000 # 10 minute timeout for full workflow
|
||||
}
|
||||
},
|
||||
"notes": f"Runs agents: {', '.join(agent_names)}"
|
||||
}
|
||||
|
||||
def _create_agent_node(self, agent_id: str, agent_config: Dict,
|
||||
action: str, position: List[int],
|
||||
index: int) -> Dict[str, Any]:
|
||||
|
||||
@@ -504,7 +504,8 @@ class PraisonAI:
|
||||
from .features.n8n import N8nHandler
|
||||
n8n_handler = N8nHandler(
|
||||
verbose=getattr(args, 'verbose', False),
|
||||
n8n_url=getattr(args, 'n8n_url', 'http://localhost:5678')
|
||||
n8n_url=getattr(args, 'n8n_url', 'http://localhost:5678'),
|
||||
api_url=getattr(args, 'api_url', 'http://127.0.0.1:8005')
|
||||
)
|
||||
result = n8n_handler.execute(self.agent_file)
|
||||
return result
|
||||
@@ -578,7 +579,7 @@ class PraisonAI:
|
||||
return default_args
|
||||
|
||||
# Define special commands
|
||||
special_commands = ['chat', 'code', 'call', 'realtime', 'train', 'ui', 'context', 'research', 'memory', 'rules', 'workflow', 'hooks', 'knowledge', 'session', 'tools', 'todo', 'docs', 'mcp', 'commit']
|
||||
special_commands = ['chat', 'code', 'call', 'realtime', 'train', 'ui', 'context', 'research', 'memory', 'rules', 'workflow', 'hooks', 'knowledge', 'session', 'tools', 'todo', 'docs', 'mcp', 'commit', 'serve']
|
||||
|
||||
parser = argparse.ArgumentParser(prog="praisonai", description="praisonAI command-line interface")
|
||||
parser.add_argument("--framework", choices=["crewai", "autogen", "praisonai"], help="Specify the framework")
|
||||
@@ -682,6 +683,12 @@ class PraisonAI:
|
||||
# n8n Integration - export workflow to n8n
|
||||
parser.add_argument("--n8n", action="store_true", help="Export workflow to n8n and open in browser")
|
||||
parser.add_argument("--n8n-url", type=str, default="http://localhost:5678", help="n8n instance URL (default: http://localhost:5678)")
|
||||
parser.add_argument("--api-url", type=str, default="http://127.0.0.1:8005", help="PraisonAI API URL for n8n to call (default: http://127.0.0.1:8005)")
|
||||
|
||||
# Serve - start API server for agents
|
||||
parser.add_argument("--serve", action="store_true", help="Start API server for agents (use with agents.yaml)")
|
||||
parser.add_argument("--port", type=int, default=8005, help="Server port (default: 8005)")
|
||||
parser.add_argument("--host", type=str, default="127.0.0.1", help="Server host (default: 127.0.0.1)")
|
||||
|
||||
# If we're in a test environment, parse with empty args to avoid pytest interference
|
||||
if in_test_env:
|
||||
@@ -710,6 +717,13 @@ class PraisonAI:
|
||||
args.realtime = True
|
||||
if args.command == 'call':
|
||||
args.call = True
|
||||
if args.command == 'serve':
|
||||
args.serve = True
|
||||
|
||||
# Handle serve command - start API server for agents
|
||||
if args.command == 'serve' or args.serve:
|
||||
self._handle_serve_command(args, unknown_args)
|
||||
sys.exit(0)
|
||||
|
||||
# Handle both command and flag versions for call
|
||||
if args.command == 'call' or args.call:
|
||||
@@ -3044,6 +3058,219 @@ Now, {final_instruction.lower()}:"""
|
||||
print("pip install praisonaiagents # For PraisonAIAgents\n")
|
||||
sys.exit(1)
|
||||
|
||||
def _handle_serve_command(self, args, unknown_args):
|
||||
"""
|
||||
Handle the serve command - start API server for agents.
|
||||
|
||||
Supports:
|
||||
- Sequential workflows with agents and steps
|
||||
- Workflow patterns: route, parallel, loop, repeat
|
||||
- All process types: sequential, workflow, hierarchical
|
||||
|
||||
Usage:
|
||||
praisonai serve agents.yaml
|
||||
praisonai serve agents.yaml --port 8005 --host 0.0.0.0
|
||||
praisonai agents.yaml --serve
|
||||
"""
|
||||
import yaml
|
||||
from praisonaiagents import Agent, PraisonAIAgents
|
||||
|
||||
# Determine the YAML file path
|
||||
yaml_file = None
|
||||
|
||||
# Check if command is 'serve' and there's a file in unknown_args
|
||||
if args.command == 'serve' and unknown_args:
|
||||
yaml_file = unknown_args[0]
|
||||
# Check if command itself is a yaml file (praisonai agents.yaml --serve)
|
||||
elif args.command and args.command.endswith('.yaml'):
|
||||
yaml_file = args.command
|
||||
# Default to agents.yaml
|
||||
else:
|
||||
yaml_file = 'agents.yaml'
|
||||
|
||||
# Check if file exists
|
||||
if not os.path.exists(yaml_file):
|
||||
print(f"[red]ERROR: File not found: {yaml_file}[/red]")
|
||||
print("\nUsage:")
|
||||
print(" praisonai serve agents.yaml")
|
||||
print(" praisonai serve agents.yaml --port 8005")
|
||||
print(" praisonai agents.yaml --serve")
|
||||
sys.exit(1)
|
||||
|
||||
print(f"📄 Loading workflow from: {yaml_file}")
|
||||
|
||||
# Load YAML file
|
||||
with open(yaml_file, 'r') as f:
|
||||
config = yaml.safe_load(f)
|
||||
|
||||
# Extract agents
|
||||
agents_config = config.get('agents', config.get('roles', {}))
|
||||
agents_dict = {} # Map agent_id to Agent instance
|
||||
agents_list = []
|
||||
|
||||
for agent_id, agent_config in agents_config.items():
|
||||
if isinstance(agent_config, dict):
|
||||
agent = Agent(
|
||||
name=agent_config.get('name', agent_id.title()),
|
||||
role=agent_config.get('role', ''),
|
||||
goal=agent_config.get('goal', ''),
|
||||
backstory=agent_config.get('backstory', ''),
|
||||
instructions=agent_config.get('instructions', ''),
|
||||
llm=agent_config.get('llm', 'gpt-4o-mini'),
|
||||
verbose=False
|
||||
)
|
||||
agents_dict[agent_id] = agent
|
||||
agents_list.append(agent)
|
||||
print(f" ✓ Loaded agent: {agent.name}")
|
||||
|
||||
if not agents_list:
|
||||
print("[red]ERROR: No agents found in YAML file[/red]")
|
||||
sys.exit(1)
|
||||
|
||||
# Extract tasks from steps (if defined)
|
||||
steps = config.get('steps', [])
|
||||
tasks_list = []
|
||||
|
||||
# Detect advanced workflow patterns
|
||||
has_route = any(isinstance(s, dict) and 'route' in s for s in steps)
|
||||
has_parallel = any(isinstance(s, dict) and 'parallel' in s for s in steps)
|
||||
has_loop = any(isinstance(s, dict) and 'loop' in s for s in steps)
|
||||
has_repeat = any(isinstance(s, dict) and 'repeat' in s for s in steps)
|
||||
|
||||
if has_route or has_parallel or has_loop or has_repeat:
|
||||
patterns = []
|
||||
if has_route: patterns.append("route")
|
||||
if has_parallel: patterns.append("parallel")
|
||||
if has_loop: patterns.append("loop")
|
||||
if has_repeat: patterns.append("repeat")
|
||||
print(f"\n🔀 Detected workflow patterns: {', '.join(patterns)}")
|
||||
|
||||
if steps:
|
||||
from praisonaiagents import Task
|
||||
print(f"\n📋 Loading workflow steps...")
|
||||
|
||||
# First pass: create all tasks
|
||||
task_name_map = {} # Map step names to Task objects for context/next_tasks
|
||||
for i, step in enumerate(steps):
|
||||
if isinstance(step, dict):
|
||||
# Handle advanced patterns
|
||||
if 'route' in step:
|
||||
print(f" ✓ Loaded routing step: {step.get('name', f'route_{i+1}')}")
|
||||
continue # Route steps are handled by workflow process
|
||||
if 'parallel' in step:
|
||||
parallel_tasks = step.get('parallel', [])
|
||||
print(f" ✓ Loaded parallel step: {len(parallel_tasks)} concurrent tasks")
|
||||
for j, pt in enumerate(parallel_tasks):
|
||||
if isinstance(pt, dict):
|
||||
agent_id = pt.get('agent', '')
|
||||
action = pt.get('action', pt.get('description', ''))
|
||||
if agent_id in agents_dict:
|
||||
task = Task(
|
||||
name=f"parallel_{i}_{j}",
|
||||
description=action,
|
||||
agent=agents_dict[agent_id],
|
||||
expected_output=pt.get('expected_output', 'Completed task output'),
|
||||
async_execution=True, # Parallel tasks run async
|
||||
)
|
||||
tasks_list.append(task)
|
||||
task_name_map[f"parallel_{i}_{j}"] = task
|
||||
continue
|
||||
if 'loop' in step:
|
||||
loop_config = step.get('loop', {})
|
||||
print(f" ✓ Loaded loop step: over '{loop_config.get('over', 'items')}'")
|
||||
continue # Loop steps are handled by workflow process
|
||||
if 'repeat' in step:
|
||||
repeat_config = step.get('repeat', {})
|
||||
print(f" ✓ Loaded repeat step: until '{repeat_config.get('until', 'done')}'")
|
||||
continue # Repeat steps are handled by workflow process
|
||||
|
||||
# Standard step
|
||||
agent_id = step.get('agent', '')
|
||||
action = step.get('action', step.get('description', ''))
|
||||
task_name = step.get('name', f"step_{i+1}")
|
||||
|
||||
if agent_id in agents_dict:
|
||||
task = Task(
|
||||
name=task_name,
|
||||
description=action,
|
||||
agent=agents_dict[agent_id],
|
||||
expected_output=step.get('expected_output', 'Completed task output'),
|
||||
task_type=step.get('task_type', 'task'),
|
||||
is_start=step.get('is_start', i == 0),
|
||||
async_execution=step.get('async_execution', False),
|
||||
output_file=step.get('output_file'),
|
||||
)
|
||||
tasks_list.append(task)
|
||||
task_name_map[task_name] = task
|
||||
print(f" ✓ Loaded step {i+1}: {agent_id} → {action[:50]}...")
|
||||
|
||||
# Second pass: set up next_tasks and context relationships
|
||||
for i, step in enumerate(steps):
|
||||
if isinstance(step, dict) and 'agent' in step:
|
||||
task_name = step.get('name', f"step_{i+1}")
|
||||
if task_name in task_name_map:
|
||||
task = task_name_map[task_name]
|
||||
|
||||
# Set next_tasks if specified
|
||||
next_tasks = step.get('next_tasks', [])
|
||||
if next_tasks:
|
||||
task.next_tasks = next_tasks
|
||||
|
||||
# Set context if specified
|
||||
context_refs = step.get('context', [])
|
||||
if context_refs:
|
||||
task.context = [task_name_map[ref] for ref in context_refs if ref in task_name_map]
|
||||
|
||||
# Set condition for workflow branching
|
||||
condition = step.get('condition', {})
|
||||
if condition:
|
||||
task.condition = condition
|
||||
|
||||
port = args.port
|
||||
host = args.host
|
||||
|
||||
# Extract process type from workflow config (default: sequential)
|
||||
workflow_config = config.get('workflow', {})
|
||||
process_type = workflow_config.get('process', 'sequential')
|
||||
verbose = workflow_config.get('verbose', False)
|
||||
|
||||
# Also check top-level process key
|
||||
if 'process' in config:
|
||||
process_type = config['process']
|
||||
|
||||
print(f"\n🚀 Starting PraisonAI API server...")
|
||||
print(f" Host: {host}")
|
||||
print(f" Port: {port}")
|
||||
print(f" Agents: {len(agents_list)}")
|
||||
if tasks_list:
|
||||
print(f" Tasks: {len(tasks_list)}")
|
||||
print(f" Process: {process_type}")
|
||||
|
||||
# Create and launch - with tasks if defined
|
||||
if tasks_list:
|
||||
praison = PraisonAIAgents(
|
||||
agents=agents_list,
|
||||
tasks=tasks_list,
|
||||
process=process_type,
|
||||
verbose=1 if verbose else 0
|
||||
)
|
||||
else:
|
||||
praison = PraisonAIAgents(
|
||||
agents=agents_list,
|
||||
process=process_type,
|
||||
verbose=1 if verbose else 0
|
||||
)
|
||||
praison.launch(port=port, host=host)
|
||||
|
||||
# Keep the main thread alive to prevent exit
|
||||
print("\n✅ Server running. Press Ctrl+C to stop.")
|
||||
try:
|
||||
import time
|
||||
while True:
|
||||
time.sleep(1)
|
||||
except KeyboardInterrupt:
|
||||
print("\n👋 Server stopped.")
|
||||
|
||||
def create_chainlit_chat_interface(self):
|
||||
"""
|
||||
Create a Chainlit interface for the chat application.
|
||||
|
||||
@@ -57,7 +57,7 @@ class CloudDeployer:
|
||||
file.write("FROM python:3.11-slim\n")
|
||||
file.write("WORKDIR /app\n")
|
||||
file.write("COPY . .\n")
|
||||
file.write("RUN pip install flask praisonai==2.3.32 gunicorn markdown\n")
|
||||
file.write("RUN pip install flask praisonai==2.3.36 gunicorn markdown\n")
|
||||
file.write("EXPOSE 8080\n")
|
||||
file.write('CMD ["gunicorn", "-b", "0.0.0.0:8080", "api:app"]\n')
|
||||
|
||||
|
||||
@@ -1 +1 @@
|
||||
__version__ = "2.3.32"
|
||||
__version__ = "2.3.36"
|
||||
@@ -12,7 +12,7 @@ dependencies = [
|
||||
"rich>=13.7",
|
||||
"markdown>=3.5",
|
||||
"pyparsing>=3.0.0",
|
||||
"praisonaiagents>=0.1.13",
|
||||
"praisonaiagents>=0.1.14",
|
||||
"python-dotenv>=0.19.0",
|
||||
"instructor>=1.3.3",
|
||||
"PyYAML>=6.0",
|
||||
|
||||
@@ -81,26 +81,33 @@ steps:
|
||||
assert "connections" in result
|
||||
|
||||
def test_n8n_workflow_has_trigger_node(self, sample_yaml_file):
|
||||
"""Test that n8n workflow has a manual trigger node."""
|
||||
"""Test that n8n workflow has a trigger node (webhook or manual)."""
|
||||
from praisonai.cli.features.n8n import N8nHandler
|
||||
handler = N8nHandler()
|
||||
result = handler.convert_yaml_to_n8n(sample_yaml_file)
|
||||
|
||||
nodes = result["nodes"]
|
||||
trigger_nodes = [n for n in nodes if "trigger" in n.get("type", "").lower() or "trigger" in n.get("name", "").lower()]
|
||||
# Check for webhook or manual trigger nodes
|
||||
trigger_nodes = [n for n in nodes if "trigger" in n.get("type", "").lower()
|
||||
or "webhook" in n.get("type", "").lower()
|
||||
or "trigger" in n.get("name", "").lower()
|
||||
or "webhook" in n.get("name", "").lower()]
|
||||
assert len(trigger_nodes) >= 1
|
||||
|
||||
def test_n8n_workflow_has_agent_nodes(self, sample_yaml_file):
|
||||
"""Test that n8n workflow has agent nodes (Execute Command or HTTP Request)."""
|
||||
def test_n8n_workflow_has_per_agent_nodes(self, sample_yaml_file):
|
||||
"""Test that n8n workflow has HTTP Request nodes for each agent."""
|
||||
from praisonai.cli.features.n8n import N8nHandler
|
||||
handler = N8nHandler()
|
||||
result = handler.convert_yaml_to_n8n(sample_yaml_file)
|
||||
|
||||
nodes = result["nodes"]
|
||||
# Agent nodes can be either executeCommand or httpRequest
|
||||
agent_nodes = [n for n in nodes if "executeCommand" in n.get("type", "") or "httpRequest" in n.get("type", "")]
|
||||
# Should have at least 2 agent nodes (one per agent step)
|
||||
assert len(agent_nodes) >= 2
|
||||
# Should have HTTP Request nodes for each agent
|
||||
http_nodes = [n for n in nodes if "httpRequest" in n.get("type", "")]
|
||||
assert len(http_nodes) >= 2 # At least 2 agents
|
||||
|
||||
# Each node should call a specific agent endpoint
|
||||
for node in http_nodes:
|
||||
assert "/agents/" in node["parameters"]["url"]
|
||||
|
||||
def test_n8n_workflow_nodes_are_connected(self, sample_yaml_file):
|
||||
"""Test that nodes are properly connected."""
|
||||
|
||||
Generated
+4
-4
@@ -4197,7 +4197,7 @@ requires-dist = [
|
||||
{ name = "praisonai-tools", marker = "extra == 'autogen'", specifier = ">=0.0.22" },
|
||||
{ name = "praisonai-tools", marker = "extra == 'autogen-v4'", specifier = ">=0.0.22" },
|
||||
{ name = "praisonai-tools", marker = "extra == 'crewai'", specifier = ">=0.0.22" },
|
||||
{ name = "praisonaiagents", specifier = ">=0.1.13" },
|
||||
{ name = "praisonaiagents", specifier = ">=0.1.14" },
|
||||
{ name = "pyautogen", marker = "extra == 'autogen'", specifier = "==0.2.29" },
|
||||
{ name = "pydantic", marker = "extra == 'chat'", specifier = "<=2.10.1" },
|
||||
{ name = "pydantic", marker = "extra == 'code'", specifier = "<=2.10.1" },
|
||||
@@ -4252,7 +4252,7 @@ wheels = [
|
||||
|
||||
[[package]]
|
||||
name = "praisonaiagents"
|
||||
version = "0.1.13"
|
||||
version = "0.1.14"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "aiohttp" },
|
||||
@@ -4261,9 +4261,9 @@ dependencies = [
|
||||
{ name = "pydantic" },
|
||||
{ name = "rich" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/79/9e/076a981f4d53b0fa58888748971797ab4ab968bf1d183e846e2b76653b19/praisonaiagents-0.1.13.tar.gz", hash = "sha256:1547fd38eef596c5b9bfffec86ce425935990e951605b143c17950ade23e2452", size = 620335, upload-time = "2025-12-18T13:15:34.77Z" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/aa/19/3493e61f00afc8f2dc6beed5407dd5d18e36f70c96107aa0d8f23c136f26/praisonaiagents-0.1.14.tar.gz", hash = "sha256:3e86b548fa6a44f37b84fc9588d275ede34a303d2f5ac3d0c596f4e2464131b9", size = 621584, upload-time = "2025-12-18T14:39:37.194Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/35/50/968fde58d1e02a41411c357251ab6f9f3f07a95167081c71a0a8ee873594/praisonaiagents-0.1.13-py3-none-any.whl", hash = "sha256:4d63873db8fcfaddffec35006864bd73aa55021a4a5e99b6ce9ba818b2ac0311", size = 547445, upload-time = "2025-12-18T13:15:32.935Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/0b/f7/618f7278370091eb62c118331646010d139dd88a8477c8bea91e18ac274e/praisonaiagents-0.1.14-py3-none-any.whl", hash = "sha256:034c862bb293248c29f8bf6ce4a910cb17bc8b980dd5cc38d790a4cb16a08729", size = 548184, upload-time = "2025-12-18T14:39:34.989Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
|
||||
Reference in new issue
Block a user