mirror of
https://github.com/vxcontrol/pentagi.git
synced 2026-07-20 23:57:11 -04:00
feat(tools): add Firecrawl web search provider
Add Firecrawl as a network search provider alongside Tavily, Perplexity, Traversaal, SearXNG and the others. Firecrawl's /v2/search endpoint combines web search with page scraping, returning full-page markdown content for each result, which the searcher agent summarizes like the other content-rich engines. - New firecrawl tool mirrors the Tavily provider layer-for-layer: registry definition, tool-type mapping, factory wiring in both tool builders, and ftester executor/args/mocks support. - Configurable endpoint: FIRECRAWL_API_KEY plus an optional FIRECRAWL_API_URL (default https://api.firecrawl.dev) so self-hosted Firecrawl instances work. - Adds 'firecrawl' to the searchengine_type enum (goose migration + models) and the search-log server model so searches are recorded per engine. - Installer wizard, .env.example, docker-compose, prompts and docs updated. The tool is only registered when FIRECRAWL_API_KEY is set, so existing deployments are unaffected.
This commit is contained in:
rramprakash
@@ -185,6 +185,11 @@ TRAVERSAAL_API_KEY=
|
||||
## Tavily search engine API
|
||||
TAVILY_API_KEY=
|
||||
|
||||
## Firecrawl search engine API
|
||||
## FIRECRAWL_API_URL is optional; leave the default for the cloud API or point it at a self-hosted instance
|
||||
FIRECRAWL_API_KEY=
|
||||
FIRECRAWL_API_URL=https://api.firecrawl.dev
|
||||
|
||||
## Perplexity search engine API
|
||||
PERPLEXITY_API_KEY=
|
||||
PERPLEXITY_MODEL=
|
||||
|
||||
@@ -122,7 +122,7 @@ State is managed primarily through Apollo Client (GraphQL) with real-time update
|
||||
### Key Integrations
|
||||
|
||||
- **LLM Providers**: OpenAI, Anthropic, Gemini, AWS Bedrock, Ollama, DeepSeek, GLM, Kimi, Qwen, and custom HTTP endpoints — configured via environment variables or the Settings UI
|
||||
- **Search**: DuckDuckGo, Google, Tavily, Traversaal, Perplexity, Searxng
|
||||
- **Search**: DuckDuckGo, Google, Tavily, Firecrawl, Traversaal, Perplexity, Searxng
|
||||
- **Databases**: PostgreSQL + pgvector (required), Neo4j (optional, for knowledge graph)
|
||||
- **Observability**: OpenTelemetry → VictoriaMetrics + Loki + Jaeger → Grafana; Langfuse for LLM analytics
|
||||
|
||||
|
||||
@@ -62,7 +62,7 @@ You can watch the video **PentAGI overview**:
|
||||
- Smart Memory System. Long-term storage of research results and successful approaches for future use.
|
||||
- Knowledge Graph Integration. Graphiti-powered knowledge graph using Neo4j for semantic relationship tracking and advanced context understanding.
|
||||
- Web Intelligence. Built-in browser via [scraper](https://hub.docker.com/r/vxcontrol/scraper) for gathering latest information from web sources.
|
||||
- External Search Systems. Integration with advanced search APIs including [Tavily](https://tavily.com), [Traversaal](https://traversaal.ai), [Perplexity](https://www.perplexity.ai), [DuckDuckGo](https://duckduckgo.com/), [Google Custom Search](https://programmablesearchengine.google.com/), [Sploitus Search](https://sploitus.com) and [Searxng](https://searxng.org) for comprehensive information gathering.
|
||||
- External Search Systems. Integration with advanced search APIs including [Tavily](https://tavily.com), [Firecrawl](https://www.firecrawl.dev), [Traversaal](https://traversaal.ai), [Perplexity](https://www.perplexity.ai), [DuckDuckGo](https://duckduckgo.com/), [Google Custom Search](https://programmablesearchengine.google.com/), [Sploitus Search](https://sploitus.com) and [Searxng](https://searxng.org) for comprehensive information gathering.
|
||||
- Team of Specialists. Delegation system with specialized AI agents for research, development, and infrastructure tasks, enhanced with optional execution monitoring and intelligent task planning for optimal performance with smaller models.
|
||||
- Comprehensive Monitoring. Detailed logging and integration with Grafana/Prometheus for real-time system observation.
|
||||
- Detailed Reporting. Generation of thorough vulnerability reports with exploitation guides.
|
||||
@@ -629,7 +629,7 @@ The installer will:
|
||||
1. **System Checks**: Verify Docker, network connectivity, and system requirements
|
||||
2. **Environment Setup**: Create and configure `.env` file with optimal defaults
|
||||
3. **Provider Configuration**: Set up LLM providers (OpenAI, Anthropic, Gemini, Bedrock, Ollama, Custom)
|
||||
4. **Search Engines**: Configure DuckDuckGo, Google, Tavily, Traversaal, Perplexity, Sploitus, Searxng
|
||||
4. **Search Engines**: Configure DuckDuckGo, Google, Tavily, Firecrawl, Traversaal, Perplexity, Sploitus, Searxng
|
||||
5. **Security Hardening**: Generate secure credentials and configure SSL certificates
|
||||
6. **Deployment**: Start PentAGI with docker-compose
|
||||
|
||||
@@ -647,7 +647,7 @@ The PentAGI web console already manages several settings areas after the server
|
||||
The following configuration areas still need to be set on the server through environment variables, compose files, or mounted config files:
|
||||
|
||||
- **LLM credentials and connection details**: API keys, endpoints, auth modes, and provider-specific connection settings for OpenAI, Anthropic, Bedrock, Ollama, custom providers, and similar backends; config-path settings apply only where supported, such as `OLLAMA_SERVER_CONFIG_PATH` and `LLM_SERVER_CONFIG_PATH`.
|
||||
- **Search provider credentials and options**: Settings such as `DUCKDUCKGO_*`, `GOOGLE_*`, `TAVILY_API_KEY`, `TRAVERSAAL_API_KEY`, `PERPLEXITY_*`, `SEARXNG_*`, and `SPLOITUS_ENABLED`.
|
||||
- **Search provider credentials and options**: Settings such as `DUCKDUCKGO_*`, `GOOGLE_*`, `TAVILY_API_KEY`, `FIRECRAWL_API_KEY`, `FIRECRAWL_API_URL`, `TRAVERSAAL_API_KEY`, `PERPLEXITY_*`, `SEARXNG_*`, and `SPLOITUS_ENABLED`.
|
||||
- **Third-party integrations**: Langfuse, Graphiti, and similar external services remain server-side configuration.
|
||||
- **MCP server management**: MCP settings pages are not currently exposed as a live web-console feature.
|
||||
|
||||
@@ -724,6 +724,7 @@ SPLOITUS_ENABLED=true
|
||||
GOOGLE_API_KEY=your_google_key
|
||||
GOOGLE_CX_KEY=your_google_cx
|
||||
TAVILY_API_KEY=your_tavily_key
|
||||
FIRECRAWL_API_KEY=your_firecrawl_key
|
||||
TRAVERSAAL_API_KEY=your_traversaal_key
|
||||
PERPLEXITY_API_KEY=your_perplexity_key
|
||||
PERPLEXITY_MODEL=sonar-pro
|
||||
@@ -3447,6 +3448,7 @@ go run cmd/ftester/main.go browser
|
||||
- **google**: Search the web using Google Custom Search
|
||||
- **duckduckgo**: Search the web using DuckDuckGo
|
||||
- **tavily**: Search using Tavily AI search engine
|
||||
- **firecrawl**: Search using Firecrawl with full-page content scraping
|
||||
- **traversaal**: Search using Traversaal AI search engine
|
||||
- **perplexity**: Search using Perplexity AI
|
||||
- **sploitus**: Search for security exploits, vulnerabilities (CVEs), and pentesting tools
|
||||
|
||||
@@ -132,6 +132,28 @@ func MockResponse(funcName string, args json.RawMessage) (string, error) {
|
||||
|
||||
resultObj = builder.String()
|
||||
|
||||
case tools.FirecrawlToolName:
|
||||
var searchArgs tools.SearchAction
|
||||
if err := json.Unmarshal(args, &searchArgs); err != nil {
|
||||
return "", fmt.Errorf("error unmarshaling search arguments: %w", err)
|
||||
}
|
||||
|
||||
terminal.PrintMock("Firecrawl search:")
|
||||
terminal.PrintKeyValue("Query", searchArgs.Query)
|
||||
terminal.PrintKeyValueFormat("Max results", "%d", searchArgs.MaxResults.Int())
|
||||
|
||||
var builder strings.Builder
|
||||
builder.WriteString("# Links\n\n")
|
||||
|
||||
for i := 1; i <= min(searchArgs.MaxResults.Int(), 3); i++ {
|
||||
builder.WriteString(fmt.Sprintf("## %d. Mock Firecrawl Result %d\n\n", i, i))
|
||||
builder.WriteString(fmt.Sprintf("* URL https://example.com/firecrawl/result%d\n\n", i))
|
||||
builder.WriteString(fmt.Sprintf("### Short content\n\nHere is a brief description of the content from this search result related to '%s'.\n\n", searchArgs.Query))
|
||||
builder.WriteString(fmt.Sprintf("### Raw content for %d. Mock Firecrawl Result %d\n\nThis is the full scraped markdown content that Firecrawl would retrieve from the URL. It contains comprehensive information about '%s' that helps answer your query with specific facts and data points relevant to your search.\n\n", i, i, searchArgs.Query))
|
||||
}
|
||||
|
||||
resultObj = builder.String()
|
||||
|
||||
case tools.TraversaalToolName:
|
||||
var searchArgs tools.SearchAction
|
||||
if err := json.Unmarshal(args, &searchArgs); err != nil {
|
||||
|
||||
@@ -293,6 +293,7 @@ func getStructTypeForFunction(funcName string) (reflect.Type, error) {
|
||||
tools.GoogleToolName: &tools.SearchAction{},
|
||||
tools.DuckDuckGoToolName: &tools.SearchAction{},
|
||||
tools.TavilyToolName: &tools.SearchAction{},
|
||||
tools.FirecrawlToolName: &tools.SearchAction{},
|
||||
tools.TraversaalToolName: &tools.SearchAction{},
|
||||
tools.PerplexityToolName: &tools.SearchAction{},
|
||||
tools.SearxngToolName: &tools.SearchAction{},
|
||||
|
||||
@@ -196,6 +196,16 @@ func (te *toolExecutor) GetTool(ctx context.Context, funcName string) (tools.Too
|
||||
te.GetSummarizer(),
|
||||
), nil
|
||||
|
||||
case tools.FirecrawlToolName:
|
||||
return tools.NewFirecrawlTool(
|
||||
te.cfg,
|
||||
te.flowID,
|
||||
te.taskID,
|
||||
te.subtaskID,
|
||||
te.proxies.GetSearchLogProvider(),
|
||||
te.GetSummarizer(),
|
||||
), nil
|
||||
|
||||
case tools.TraversaalToolName:
|
||||
return tools.NewTraversaalTool(
|
||||
te.cfg,
|
||||
|
||||
@@ -1575,6 +1575,8 @@ type SearchEnginesConfig struct {
|
||||
SploitusEnabled loader.EnvVar // SPLOITUS_ENABLED
|
||||
PerplexityAPIKey loader.EnvVar // PERPLEXITY_API_KEY
|
||||
TavilyAPIKey loader.EnvVar // TAVILY_API_KEY
|
||||
FirecrawlAPIKey loader.EnvVar // FIRECRAWL_API_KEY
|
||||
FirecrawlAPIURL loader.EnvVar // FIRECRAWL_API_URL
|
||||
TraversaalAPIKey loader.EnvVar // TRAVERSAAL_API_KEY
|
||||
GoogleAPIKey loader.EnvVar // GOOGLE_API_KEY
|
||||
GoogleCXKey loader.EnvVar // GOOGLE_CX_KEY
|
||||
@@ -1611,6 +1613,8 @@ func (c *controller) GetSearchEnginesConfig() *SearchEnginesConfig {
|
||||
sploitusEnabled, _ := c.GetVar("SPLOITUS_ENABLED")
|
||||
perplexityAPIKey, _ := c.GetVar("PERPLEXITY_API_KEY")
|
||||
tavilyAPIKey, _ := c.GetVar("TAVILY_API_KEY")
|
||||
firecrawlAPIKey, _ := c.GetVar("FIRECRAWL_API_KEY")
|
||||
firecrawlAPIURL, _ := c.GetVar("FIRECRAWL_API_URL")
|
||||
traversaalAPIKey, _ := c.GetVar("TRAVERSAAL_API_KEY")
|
||||
googleAPIKey, _ := c.GetVar("GOOGLE_API_KEY")
|
||||
googleCXKey, _ := c.GetVar("GOOGLE_CX_KEY")
|
||||
@@ -1634,6 +1638,8 @@ func (c *controller) GetSearchEnginesConfig() *SearchEnginesConfig {
|
||||
PerplexityModel: perplexityModel,
|
||||
PerplexityContextSize: perplexityContextSize,
|
||||
TavilyAPIKey: tavilyAPIKey,
|
||||
FirecrawlAPIKey: firecrawlAPIKey,
|
||||
FirecrawlAPIURL: firecrawlAPIURL,
|
||||
TraversaalAPIKey: traversaalAPIKey,
|
||||
GoogleAPIKey: googleAPIKey,
|
||||
GoogleCXKey: googleCXKey,
|
||||
@@ -1664,6 +1670,9 @@ func (c *controller) GetSearchEnginesConfig() *SearchEnginesConfig {
|
||||
if tavilyAPIKey.Value != "" {
|
||||
configuredCount++
|
||||
}
|
||||
if firecrawlAPIKey.Value != "" {
|
||||
configuredCount++
|
||||
}
|
||||
if traversaalAPIKey.Value != "" {
|
||||
configuredCount++
|
||||
}
|
||||
@@ -1712,6 +1721,12 @@ func (c *controller) UpdateSearchEnginesConfig(config *SearchEnginesConfig) erro
|
||||
if err := c.SetVar("TAVILY_API_KEY", config.TavilyAPIKey.Value); err != nil {
|
||||
return fmt.Errorf("failed to set TAVILY_API_KEY: %w", err)
|
||||
}
|
||||
if err := c.SetVar("FIRECRAWL_API_KEY", config.FirecrawlAPIKey.Value); err != nil {
|
||||
return fmt.Errorf("failed to set FIRECRAWL_API_KEY: %w", err)
|
||||
}
|
||||
if err := c.SetVar("FIRECRAWL_API_URL", config.FirecrawlAPIURL.Value); err != nil {
|
||||
return fmt.Errorf("failed to set FIRECRAWL_API_URL: %w", err)
|
||||
}
|
||||
if err := c.SetVar("TRAVERSAAL_API_KEY", config.TraversaalAPIKey.Value); err != nil {
|
||||
return fmt.Errorf("failed to set TRAVERSAAL_API_KEY: %w", err)
|
||||
}
|
||||
@@ -1759,6 +1774,8 @@ func (c *controller) ResetSearchEnginesConfig() *SearchEnginesConfig {
|
||||
"PERPLEXITY_MODEL",
|
||||
"PERPLEXITY_CONTEXT_SIZE",
|
||||
"TAVILY_API_KEY",
|
||||
"FIRECRAWL_API_KEY",
|
||||
"FIRECRAWL_API_URL",
|
||||
"TRAVERSAAL_API_KEY",
|
||||
"GOOGLE_API_KEY",
|
||||
"GOOGLE_CX_KEY",
|
||||
@@ -2247,6 +2264,8 @@ func (c *controller) getVariableDescription(varName string) string {
|
||||
"SPLOITUS_ENABLED": locale.EnvDesc_SPLOITUS_ENABLED,
|
||||
"PERPLEXITY_API_KEY": locale.EnvDesc_PERPLEXITY_API_KEY,
|
||||
"TAVILY_API_KEY": locale.EnvDesc_TAVILY_API_KEY,
|
||||
"FIRECRAWL_API_KEY": locale.EnvDesc_FIRECRAWL_API_KEY,
|
||||
"FIRECRAWL_API_URL": locale.EnvDesc_FIRECRAWL_API_URL,
|
||||
"TRAVERSAAL_API_KEY": locale.EnvDesc_TRAVERSAAL_API_KEY,
|
||||
"GOOGLE_API_KEY": locale.EnvDesc_GOOGLE_API_KEY,
|
||||
"GOOGLE_CX_KEY": locale.EnvDesc_GOOGLE_CX_KEY,
|
||||
@@ -2342,6 +2361,7 @@ var maskedVariables = map[string]bool{
|
||||
"LOCAL_SCRAPER_PASSWORD": true,
|
||||
"PERPLEXITY_API_KEY": true,
|
||||
"TAVILY_API_KEY": true,
|
||||
"FIRECRAWL_API_KEY": true,
|
||||
"TRAVERSAAL_API_KEY": true,
|
||||
"GOOGLE_API_KEY": true,
|
||||
"GOOGLE_CX_KEY": true,
|
||||
@@ -2440,6 +2460,8 @@ var criticalVariables = map[string]bool{
|
||||
"PERPLEXITY_MODEL": true,
|
||||
"PERPLEXITY_CONTEXT_SIZE": true,
|
||||
"TAVILY_API_KEY": true,
|
||||
"FIRECRAWL_API_KEY": true,
|
||||
"FIRECRAWL_API_URL": true,
|
||||
"TRAVERSAAL_API_KEY": true,
|
||||
"GOOGLE_API_KEY": true,
|
||||
"GOOGLE_CX_KEY": true,
|
||||
|
||||
@@ -1391,6 +1391,7 @@ const (
|
||||
• Sploitus - Security exploits and vulnerabilities database (no API key required)
|
||||
• Perplexity - AI-powered search with reasoning
|
||||
• Tavily - Search API for AI applications
|
||||
• Firecrawl - Web search with full-page scraping (self-hosting supported)
|
||||
• Traversaal - Web scraping and search
|
||||
• Google Search - Requires API key and Custom Search Engine ID
|
||||
• Searxng - Internet metasearch engine
|
||||
@@ -1398,6 +1399,7 @@ const (
|
||||
Get API keys from:
|
||||
• Perplexity: https://www.perplexity.ai/
|
||||
• Tavily: https://tavily.com/
|
||||
• Firecrawl: https://www.firecrawl.dev/
|
||||
• Traversaal: https://traversaal.ai/
|
||||
• Google: https://developers.google.com/custom-search/v1/introduction`
|
||||
|
||||
@@ -1415,6 +1417,10 @@ Get API keys from:
|
||||
ToolsSearchEnginesPerplexityKeyDesc = "API key for Perplexity AI search"
|
||||
ToolsSearchEnginesTavilyKey = "Tavily API Key"
|
||||
ToolsSearchEnginesTavilyKeyDesc = "API key for Tavily search service"
|
||||
ToolsSearchEnginesFirecrawlKey = "Firecrawl API Key"
|
||||
ToolsSearchEnginesFirecrawlKeyDesc = "API key for Firecrawl search service"
|
||||
ToolsSearchEnginesFirecrawlURL = "Firecrawl API URL"
|
||||
ToolsSearchEnginesFirecrawlURLDesc = "Firecrawl API base URL (leave empty for cloud; set for self-hosted)"
|
||||
ToolsSearchEnginesTraversaalKey = "Traversaal API Key"
|
||||
ToolsSearchEnginesTraversaalKeyDesc = "API key for Traversaal web scraping"
|
||||
ToolsSearchEnginesGoogleKey = "Google Search API Key"
|
||||
@@ -2271,6 +2277,8 @@ const (
|
||||
EnvDesc_SPLOITUS_ENABLED = "Sploitus Search"
|
||||
EnvDesc_PERPLEXITY_API_KEY = "Perplexity API Key"
|
||||
EnvDesc_TAVILY_API_KEY = "Tavily API Key"
|
||||
EnvDesc_FIRECRAWL_API_KEY = "Firecrawl API Key"
|
||||
EnvDesc_FIRECRAWL_API_URL = "Firecrawl API URL"
|
||||
EnvDesc_TRAVERSAAL_API_KEY = "Traversaal API Key"
|
||||
EnvDesc_GOOGLE_API_KEY = "Google Search API Key"
|
||||
EnvDesc_GOOGLE_CX_KEY = "Google Search CX Key"
|
||||
|
||||
@@ -113,6 +113,21 @@ func (m *SearchEnginesFormModel) BuildForm() tea.Cmd {
|
||||
config.TavilyAPIKey,
|
||||
))
|
||||
|
||||
// Firecrawl API Key
|
||||
fields = append(fields, m.createAPIKeyField("firecrawl_api_key",
|
||||
locale.ToolsSearchEnginesFirecrawlKey,
|
||||
locale.ToolsSearchEnginesFirecrawlKeyDesc,
|
||||
config.FirecrawlAPIKey,
|
||||
))
|
||||
|
||||
// Firecrawl API URL (optional, for self-hosted deployments)
|
||||
fields = append(fields, m.createTextField("firecrawl_api_url",
|
||||
locale.ToolsSearchEnginesFirecrawlURL,
|
||||
locale.ToolsSearchEnginesFirecrawlURLDesc,
|
||||
config.FirecrawlAPIURL,
|
||||
false,
|
||||
))
|
||||
|
||||
// Traversaal API Key
|
||||
fields = append(fields, m.createAPIKeyField("traversaal_api_key",
|
||||
locale.ToolsSearchEnginesTraversaalKey,
|
||||
@@ -325,6 +340,15 @@ func (m *SearchEnginesFormModel) GetCurrentConfiguration() string {
|
||||
m.GetStyles().Warning.Render(locale.StatusNotConfigured)))
|
||||
}
|
||||
|
||||
// Firecrawl
|
||||
if config.FirecrawlAPIKey.Value != "" {
|
||||
sections = append(sections, fmt.Sprintf("• Firecrawl: %s",
|
||||
m.GetStyles().Success.Render(locale.StatusConfigured)))
|
||||
} else {
|
||||
sections = append(sections, fmt.Sprintf("• Firecrawl: %s",
|
||||
m.GetStyles().Warning.Render(locale.StatusNotConfigured)))
|
||||
}
|
||||
|
||||
// Traversaal
|
||||
if config.TraversaalAPIKey.Value != "" {
|
||||
sections = append(sections, fmt.Sprintf("• Traversaal: %s",
|
||||
@@ -393,6 +417,8 @@ func (m *SearchEnginesFormModel) HandleSave() error {
|
||||
PerplexityModel: config.PerplexityModel,
|
||||
PerplexityContextSize: config.PerplexityContextSize,
|
||||
TavilyAPIKey: config.TavilyAPIKey,
|
||||
FirecrawlAPIKey: config.FirecrawlAPIKey,
|
||||
FirecrawlAPIURL: config.FirecrawlAPIURL,
|
||||
TraversaalAPIKey: config.TraversaalAPIKey,
|
||||
GoogleAPIKey: config.GoogleAPIKey,
|
||||
GoogleCXKey: config.GoogleCXKey,
|
||||
@@ -436,6 +462,10 @@ func (m *SearchEnginesFormModel) HandleSave() error {
|
||||
newConfig.PerplexityContextSize.Value = value
|
||||
case "tavily_api_key":
|
||||
newConfig.TavilyAPIKey.Value = value
|
||||
case "firecrawl_api_key":
|
||||
newConfig.FirecrawlAPIKey.Value = value
|
||||
case "firecrawl_api_url":
|
||||
newConfig.FirecrawlAPIURL.Value = value
|
||||
case "traversaal_api_key":
|
||||
newConfig.TraversaalAPIKey.Value = value
|
||||
case "google_api_key":
|
||||
|
||||
@@ -107,7 +107,7 @@ The `isAgent` field in `FunctionToolcallsStats` categorizes tools into two types
|
||||
|
||||
**Non-Agent Tools (`isAgent: false`):**
|
||||
- Direct tool execution (e.g., `terminal`, `browser`, `file`)
|
||||
- Search engines (e.g., `google`, `duckduckgo`, `tavily`, `sploitus`, `searxng`)
|
||||
- Search engines (e.g., `google`, `duckduckgo`, `tavily`, `firecrawl`, `sploitus`, `searxng`)
|
||||
- Vector database operations (e.g., `search_in_memory`, `store_guide`)
|
||||
- Environment operations (e.g., `terminal`, `file`)
|
||||
- Execution time represents actual operation duration
|
||||
|
||||
@@ -58,6 +58,7 @@ This document serves as a comprehensive guide to the configuration system in Pen
|
||||
- [Google Search](#google-search)
|
||||
- [Traversaal Search](#traversaal-search)
|
||||
- [Tavily Search](#tavily-search)
|
||||
- [Firecrawl Search](#firecrawl-search)
|
||||
- [Perplexity Search](#perplexity-search)
|
||||
- [Searxng Search](#searxng-search)
|
||||
- [Usage Details](#usage-details-10)
|
||||
@@ -1400,6 +1401,7 @@ Common built-in functions that can be disabled:
|
||||
- `google` - Google Search
|
||||
- `duckduckgo` - DuckDuckGo Search
|
||||
- `tavily` - Tavily Search
|
||||
- `firecrawl` - Firecrawl Search
|
||||
- `traversaal` - Traversaal Search
|
||||
- `perplexity` - Perplexity Search
|
||||
- `searxng` - SearXNG Search
|
||||
@@ -1447,6 +1449,13 @@ These settings control the integration with various search engines used for web
|
||||
| ------------ | -------------------- | ------------- | -------------------------------- |
|
||||
| TavilyAPIKey | `TAVILY_API_KEY` | *(none)* | API key for Tavily search engine |
|
||||
|
||||
### Firecrawl Search
|
||||
|
||||
| Option | Environment Variable | Default Value | Description |
|
||||
| --------------- | -------------------- | --------------------------- | ------------------------------------------------------------------------------ |
|
||||
| FirecrawlAPIKey | `FIRECRAWL_API_KEY` | *(none)* | API key for Firecrawl search engine |
|
||||
| FirecrawlAPIURL | `FIRECRAWL_API_URL` | `https://api.firecrawl.dev` | Base URL for the Firecrawl API (override to point at a self-hosted deployment) |
|
||||
|
||||
### Perplexity Search
|
||||
|
||||
| Option | Environment Variable | Default Value | Description |
|
||||
@@ -1492,6 +1501,14 @@ tavilySearch: &functions.TavilySearchFunc{
|
||||
summarizer: cfg.Summarizer,
|
||||
},
|
||||
|
||||
// Firecrawl Search configuration
|
||||
firecrawlSearch: &functions.FirecrawlSearchFunc{
|
||||
apiKey: fte.cfg.FirecrawlAPIKey,
|
||||
apiURL: fte.cfg.FirecrawlAPIURL,
|
||||
proxyURL: fte.cfg.ProxyURL,
|
||||
summarizer: cfg.Summarizer,
|
||||
},
|
||||
|
||||
// Perplexity Search configuration
|
||||
perplexitySearch: &functions.PerplexitySearchFunc{
|
||||
apiKey: fte.cfg.PerplexityAPIKey,
|
||||
|
||||
@@ -458,7 +458,7 @@ The package provides comprehensive logging for all system operations:
|
||||
- **AgentLog**: Inter-agent communication and delegation
|
||||
- **AssistantLog**: Human-assistant interactions
|
||||
- **MsgLog**: General message logging (thoughts/browser/terminal/file/search/advice/ask/input/done)
|
||||
- **SearchLog**: External search operations (google/tavily/traversaal/browser/duckduckgo/perplexity/sploitus/searxng)
|
||||
- **SearchLog**: External search operations (google/tavily/firecrawl/traversaal/browser/duckduckgo/perplexity/sploitus/searxng)
|
||||
- **TermLog**: Terminal command execution (stdin/stdout/stderr)
|
||||
- **ToolCall**: AI function calling with duration tracking
|
||||
- `duration_seconds` - pre-calculated execution duration (DOUBLE PRECISION, NOT NULL, DEFAULT 0.0)
|
||||
|
||||
@@ -47,6 +47,7 @@ This document describes the internal architecture and execution workflow of Flow
|
||||
- `google` - Google Custom Search API integration
|
||||
- `duckduckgo` - Anonymous search engine
|
||||
- `tavily` - Advanced research with citations
|
||||
- `firecrawl` - Web search with full-page content scraping
|
||||
- `traversaal` - Structured Q&A search
|
||||
- `perplexity` - AI-powered comprehensive research
|
||||
- `sploitus` - Search for security exploits and pentest tools
|
||||
@@ -760,14 +761,16 @@ The Searcher Agent follows a strict hierarchy for information retrieval:
|
||||
3. **Priority 5: Deep Analysis Tools** - Complex research synthesis
|
||||
- `traversaal` - Structured answers for common questions
|
||||
- `tavily` - Research-grade exploration of technical topics
|
||||
- `firecrawl` - Deep research with full-page content scraped from discovered sites
|
||||
- `perplexity` - Comprehensive analysis with advanced reasoning
|
||||
|
||||
**Available Search Engines**: Google, DuckDuckGo, Tavily, Traversaal, Perplexity, Sploitus, Searxng
|
||||
**Available Search Engines**: Google, DuckDuckGo, Tavily, Firecrawl, Traversaal, Perplexity, Sploitus, Searxng
|
||||
|
||||
**Search Engine Configurations**:
|
||||
- **Google** - Custom Search API with CX key and language restrictions
|
||||
- **DuckDuckGo** - Anonymous search with VQD token authentication
|
||||
- **Tavily** - Advanced research with raw content and citations
|
||||
- **Firecrawl** - Web search with full-page markdown content scraped from each result
|
||||
- **Perplexity** - AI-powered synthesis with configurable context size
|
||||
- **Traversaal** - Structured Q&A responses with web links
|
||||
- **Sploitus** - Search for security exploits and pentest tools
|
||||
|
||||
@@ -0,0 +1,54 @@
|
||||
-- +goose Up
|
||||
-- +goose StatementBegin
|
||||
-- Add firecrawl to the searchengine_type enum
|
||||
CREATE TYPE SEARCHENGINE_TYPE_NEW AS ENUM (
|
||||
'google',
|
||||
'tavily',
|
||||
'firecrawl',
|
||||
'traversaal',
|
||||
'browser',
|
||||
'duckduckgo',
|
||||
'perplexity',
|
||||
'searxng',
|
||||
'sploitus'
|
||||
);
|
||||
|
||||
-- Update the searchlogs table to use the new enum type
|
||||
ALTER TABLE searchlogs
|
||||
ALTER COLUMN engine TYPE SEARCHENGINE_TYPE_NEW USING engine::text::SEARCHENGINE_TYPE_NEW;
|
||||
|
||||
-- Drop the old type and rename the new one
|
||||
DROP TYPE SEARCHENGINE_TYPE;
|
||||
ALTER TYPE SEARCHENGINE_TYPE_NEW RENAME TO SEARCHENGINE_TYPE;
|
||||
|
||||
-- Ensure NOT NULL constraint is preserved
|
||||
ALTER TABLE searchlogs
|
||||
ALTER COLUMN engine SET NOT NULL;
|
||||
-- +goose StatementEnd
|
||||
|
||||
-- +goose Down
|
||||
-- +goose StatementBegin
|
||||
-- Revert the changes by removing firecrawl from the enum
|
||||
CREATE TYPE SEARCHENGINE_TYPE_NEW AS ENUM (
|
||||
'google',
|
||||
'tavily',
|
||||
'traversaal',
|
||||
'browser',
|
||||
'duckduckgo',
|
||||
'perplexity',
|
||||
'searxng',
|
||||
'sploitus'
|
||||
);
|
||||
|
||||
-- Update the searchlogs table to use the reverted enum type
|
||||
ALTER TABLE searchlogs
|
||||
ALTER COLUMN engine TYPE SEARCHENGINE_TYPE_NEW USING engine::text::SEARCHENGINE_TYPE_NEW;
|
||||
|
||||
-- Drop the new type and rename the reverted one
|
||||
DROP TYPE SEARCHENGINE_TYPE;
|
||||
ALTER TYPE SEARCHENGINE_TYPE_NEW RENAME TO SEARCHENGINE_TYPE;
|
||||
|
||||
-- Ensure NOT NULL constraint is preserved
|
||||
ALTER TABLE searchlogs
|
||||
ALTER COLUMN engine SET NOT NULL;
|
||||
-- +goose StatementEnd
|
||||
@@ -167,6 +167,10 @@ type Config struct {
|
||||
// === Search Engine: Tavily AI ===
|
||||
TavilyAPIKey string `env:"TAVILY_API_KEY"`
|
||||
|
||||
// === Search Engine: Firecrawl ===
|
||||
FirecrawlAPIKey string `env:"FIRECRAWL_API_KEY"`
|
||||
FirecrawlAPIURL string `env:"FIRECRAWL_API_URL" envDefault:"https://api.firecrawl.dev"`
|
||||
|
||||
// === Search Engine: Perplexity AI ===
|
||||
PerplexityAPIKey string `env:"PERPLEXITY_API_KEY"`
|
||||
PerplexityModel string `env:"PERPLEXITY_MODEL" envDefault:"sonar"`
|
||||
@@ -330,6 +334,7 @@ func (c *Config) GetSecretPatterns() []patterns.Pattern {
|
||||
{c.OAuthGithubClientSecret, "Github Client Secret"},
|
||||
{c.TraversaalAPIKey, "Traversaal Key"},
|
||||
{c.TavilyAPIKey, "Tavily Key"},
|
||||
{c.FirecrawlAPIKey, "Firecrawl Key"},
|
||||
{c.PerplexityAPIKey, "Perplexity Key"},
|
||||
{c.ProxyURL, "Proxy URL"},
|
||||
{c.LangfusePublicKey, "Langfuse Public Key"},
|
||||
|
||||
@@ -474,6 +474,7 @@ type SearchengineType string
|
||||
const (
|
||||
SearchengineTypeGoogle SearchengineType = "google"
|
||||
SearchengineTypeTavily SearchengineType = "tavily"
|
||||
SearchengineTypeFirecrawl SearchengineType = "firecrawl"
|
||||
SearchengineTypeTraversaal SearchengineType = "traversaal"
|
||||
SearchengineTypeBrowser SearchengineType = "browser"
|
||||
SearchengineTypeDuckduckgo SearchengineType = "duckduckgo"
|
||||
|
||||
@@ -13,6 +13,7 @@ const (
|
||||
SearchEngineTypeGoogle SearchEngineType = "google"
|
||||
SearchEngineTypeDuckduckgo SearchEngineType = "duckduckgo"
|
||||
SearchEngineTypeTavily SearchEngineType = "tavily"
|
||||
SearchEngineTypeFirecrawl SearchEngineType = "firecrawl"
|
||||
SearchEngineTypeTraversaal SearchEngineType = "traversaal"
|
||||
SearchEngineTypePerplexity SearchEngineType = "perplexity"
|
||||
SearchEngineTypeBrowser SearchEngineType = "browser"
|
||||
@@ -29,6 +30,7 @@ func (s SearchEngineType) Valid() error {
|
||||
case SearchEngineTypeGoogle,
|
||||
SearchEngineTypeDuckduckgo,
|
||||
SearchEngineTypeTavily,
|
||||
SearchEngineTypeFirecrawl,
|
||||
SearchEngineTypeTraversaal,
|
||||
SearchEngineTypePerplexity,
|
||||
SearchEngineTypeBrowser,
|
||||
|
||||
@@ -162,7 +162,7 @@ External search engines and the vector store are indexed in English and shared a
|
||||
{{- if .UseAgents}}
|
||||
- Web research and memory: delegated to specialists (see TEAM COLLABORATION & DELEGATION above) — you do NOT invoke search engines or vector-store searches directly in this mode
|
||||
{{- else}}
|
||||
- Web search engines: `{{.GoogleToolName}}`, `{{.DuckDuckGoToolName}}`, `{{.TavilyToolName}}`, `{{.TraversaalToolName}}`, `{{.PerplexityToolName}}` (and Searxng/Sploitus when configured) — pick the engine matching the query complexity
|
||||
- Web search engines: `{{.GoogleToolName}}`, `{{.DuckDuckGoToolName}}`, `{{.TavilyToolName}}`, `{{.TraversaalToolName}}`, `{{.PerplexityToolName}}` (and Firecrawl/Searxng/Sploitus when configured) — pick the engine matching the query complexity
|
||||
- Memory search (read-only): `{{.SearchInMemoryToolName}}`, `{{.SearchGuideToolName}}`, `{{.SearchAnswerToolName}}`, `{{.SearchCodeToolName}}` for retrieving stored knowledge from past engagements
|
||||
{{- end}}
|
||||
</available_tools>
|
||||
|
||||
@@ -20,7 +20,7 @@ You operate on two parallel channels. The channel of each tool argument is fixed
|
||||
1. **Engagement log — engagement language `{{.Lang}}`.** Your running commentary on this engagement. Entries are every `message` field of every tool call you make (external search engines, browser navigation, the answer vector-store search and store, the memorist delegation, the closing call) and the `message` of your closing `{{.SearchResultToolName}}` call. The engagement coordination team reads the log in `{{.Lang}}`. Keep `message` log entries to 1-2 short sentences narrating what you are about to do or what you just produced.
|
||||
|
||||
2. **Technical channel — English.** The wire between you, your team, search engines, the vector store, and the web. Outgoing entries are:
|
||||
- search-engine queries you send to whichever providers are available in this engagement (Google, DuckDuckGo, Tavily, Traversaal, Perplexity, Searxng, Sploitus, etc.) — use exact technical terms, identifiers, and error codes
|
||||
- search-engine queries you send to whichever providers are available in this engagement (Google, DuckDuckGo, Tavily, Firecrawl, Traversaal, Perplexity, Searxng, Sploitus, etc.) — use exact technical terms, identifiers, and error codes
|
||||
- browser `url` for direct retrieval from known sources
|
||||
- answer vector-store search queries: `{{.SearchAnswerToolName}}.questions`
|
||||
- answer vector-store write payloads with `{{.StoreAnswerToolName}}` (`answer`, `question`)
|
||||
@@ -106,6 +106,7 @@ External search engines and the answer vector store are indexed in English and s
|
||||
|
||||
<deep_analysis_tools>
|
||||
<tool name="tavily" priority="5">For research-grade exploration of complex technical topics</tool>
|
||||
<tool name="firecrawl" priority="5">For deep research with full-page content scraped directly from the discovered web sites</tool>
|
||||
<tool name="perplexity" priority="5">For comprehensive analysis with advanced reasoning</tool>
|
||||
<tool name="traversaal" priority="4">For discovering structured answers to common questions</tool>
|
||||
</deep_analysis_tools>
|
||||
|
||||
@@ -0,0 +1,369 @@
|
||||
package tools
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"strings"
|
||||
"text/template"
|
||||
|
||||
"pentagi/pkg/config"
|
||||
"pentagi/pkg/database"
|
||||
obs "pentagi/pkg/observability"
|
||||
"pentagi/pkg/observability/langfuse"
|
||||
"pentagi/pkg/system"
|
||||
|
||||
"github.com/sirupsen/logrus"
|
||||
)
|
||||
|
||||
const (
|
||||
firecrawlDefaultURL = "https://api.firecrawl.dev"
|
||||
firecrawlSearchPath = "/v2/search"
|
||||
)
|
||||
|
||||
type firecrawlRequest struct {
|
||||
Query string `json:"query"`
|
||||
Limit int `json:"limit,omitempty"`
|
||||
Sources []string `json:"sources,omitempty"`
|
||||
ScrapeOptions *firecrawlScrapeOptions `json:"scrapeOptions,omitempty"`
|
||||
}
|
||||
|
||||
type firecrawlScrapeOptions struct {
|
||||
Formats []string `json:"formats"`
|
||||
OnlyMainContent bool `json:"onlyMainContent"`
|
||||
}
|
||||
|
||||
type firecrawlSearchResult struct {
|
||||
Success bool `json:"success"`
|
||||
Warning string `json:"warning"`
|
||||
Error string `json:"error"`
|
||||
Data firecrawlData `json:"data"`
|
||||
}
|
||||
|
||||
type firecrawlData struct {
|
||||
Web []firecrawlResult `json:"web"`
|
||||
}
|
||||
|
||||
type firecrawlResult struct {
|
||||
Title string `json:"title"`
|
||||
Description string `json:"description"`
|
||||
URL string `json:"url"`
|
||||
Markdown string `json:"markdown"`
|
||||
Metadata *firecrawlMetadata `json:"metadata"`
|
||||
}
|
||||
|
||||
type firecrawlMetadata struct {
|
||||
Title string `json:"title"`
|
||||
SourceURL string `json:"sourceURL"`
|
||||
}
|
||||
|
||||
// resolvedURL prefers the top-level URL and falls back to the metadata source
|
||||
// URL, which is where it lands when scrapeOptions turn a result into a document.
|
||||
func (r firecrawlResult) resolvedURL() string {
|
||||
if r.URL != "" {
|
||||
return r.URL
|
||||
}
|
||||
if r.Metadata != nil {
|
||||
return r.Metadata.SourceURL
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// resolvedTitle prefers the top-level title and falls back to the metadata title.
|
||||
func (r firecrawlResult) resolvedTitle() string {
|
||||
if r.Title != "" {
|
||||
return r.Title
|
||||
}
|
||||
if r.Metadata != nil {
|
||||
return r.Metadata.Title
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
type firecrawl struct {
|
||||
cfg *config.Config
|
||||
flowID int64
|
||||
taskID *int64
|
||||
subtaskID *int64
|
||||
slp SearchLogProvider
|
||||
summarizer SummarizeHandler
|
||||
}
|
||||
|
||||
func NewFirecrawlTool(
|
||||
cfg *config.Config,
|
||||
flowID int64,
|
||||
taskID, subtaskID *int64,
|
||||
slp SearchLogProvider,
|
||||
summarizer SummarizeHandler,
|
||||
) Tool {
|
||||
return &firecrawl{
|
||||
cfg: cfg,
|
||||
flowID: flowID,
|
||||
taskID: taskID,
|
||||
subtaskID: subtaskID,
|
||||
slp: slp,
|
||||
summarizer: summarizer,
|
||||
}
|
||||
}
|
||||
|
||||
func (f *firecrawl) Handle(ctx context.Context, name string, args json.RawMessage) (string, error) {
|
||||
if !f.IsAvailable() {
|
||||
return "", fmt.Errorf("firecrawl is not available")
|
||||
}
|
||||
|
||||
var action SearchAction
|
||||
ctx, observation := obs.Observer.NewObservation(ctx)
|
||||
logger := logrus.WithContext(ctx).WithFields(enrichLogrusFields(f.flowID, f.taskID, f.subtaskID, logrus.Fields{
|
||||
"tool": name,
|
||||
"args": string(args),
|
||||
}))
|
||||
|
||||
if err := json.Unmarshal(args, &action); err != nil {
|
||||
logger.WithError(err).Error("failed to unmarshal firecrawl search action")
|
||||
return "", fmt.Errorf("failed to unmarshal %s search action arguments: %w", name, err)
|
||||
}
|
||||
|
||||
logger = logger.WithFields(logrus.Fields{
|
||||
"query": action.Query[:min(len(action.Query), 1000)],
|
||||
"max_results": action.MaxResults,
|
||||
})
|
||||
|
||||
result, err := f.search(ctx, action.Query, action.MaxResults.Int())
|
||||
if err != nil {
|
||||
observation.Event(
|
||||
langfuse.WithEventName("search engine error swallowed"),
|
||||
langfuse.WithEventInput(action.Query),
|
||||
langfuse.WithEventStatus(err.Error()),
|
||||
langfuse.WithEventLevel(langfuse.ObservationLevelWarning),
|
||||
langfuse.WithEventMetadata(langfuse.Metadata{
|
||||
"tool_name": FirecrawlToolName,
|
||||
"engine": "firecrawl",
|
||||
"query": action.Query,
|
||||
"max_results": action.MaxResults.Int(),
|
||||
"error": err.Error(),
|
||||
}),
|
||||
)
|
||||
|
||||
logger.WithError(err).Error("failed to search in firecrawl")
|
||||
return fmt.Sprintf("failed to search in firecrawl: %v", err), nil
|
||||
}
|
||||
|
||||
if agentCtx, ok := GetAgentContext(ctx); ok {
|
||||
_, _ = f.slp.PutLog(
|
||||
ctx,
|
||||
agentCtx.ParentAgentType,
|
||||
agentCtx.CurrentAgentType,
|
||||
database.SearchengineTypeFirecrawl,
|
||||
action.Query,
|
||||
result,
|
||||
f.taskID,
|
||||
f.subtaskID,
|
||||
)
|
||||
}
|
||||
|
||||
return result, nil
|
||||
}
|
||||
|
||||
func (f *firecrawl) search(ctx context.Context, query string, maxResults int) (string, error) {
|
||||
client, err := system.GetHTTPClient(f.cfg)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("failed to create http client: %w", err)
|
||||
}
|
||||
|
||||
reqPayload := firecrawlRequest{
|
||||
Query: query,
|
||||
Limit: maxResults,
|
||||
Sources: []string{"web"},
|
||||
ScrapeOptions: &firecrawlScrapeOptions{
|
||||
Formats: []string{"markdown"},
|
||||
OnlyMainContent: true,
|
||||
},
|
||||
}
|
||||
reqBody, err := json.Marshal(reqPayload)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("failed to marshal request body: %v", err)
|
||||
}
|
||||
|
||||
req, err := http.NewRequest(http.MethodPost, f.searchURL(), bytes.NewBuffer(reqBody))
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("failed to build request: %v", err)
|
||||
}
|
||||
|
||||
req = req.WithContext(ctx)
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
req.Header.Set("Authorization", "Bearer "+f.apiKey())
|
||||
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("failed to do request: %v", err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
return f.parseHTTPResponse(ctx, resp)
|
||||
}
|
||||
|
||||
func (f *firecrawl) parseHTTPResponse(ctx context.Context, resp *http.Response) (string, error) {
|
||||
switch resp.StatusCode {
|
||||
case http.StatusOK:
|
||||
var respBody firecrawlSearchResult
|
||||
if err := json.NewDecoder(resp.Body).Decode(&respBody); err != nil {
|
||||
return "", fmt.Errorf("failed to decode response body: %v", err)
|
||||
}
|
||||
if !respBody.Success {
|
||||
if respBody.Error != "" {
|
||||
return "", fmt.Errorf("request failed: %s", respBody.Error)
|
||||
}
|
||||
return "", fmt.Errorf("request failed")
|
||||
}
|
||||
return f.buildFirecrawlResult(ctx, &respBody), nil
|
||||
case http.StatusBadRequest:
|
||||
return "", fmt.Errorf("request is invalid")
|
||||
case http.StatusUnauthorized:
|
||||
return "", fmt.Errorf("API key is wrong")
|
||||
case http.StatusPaymentRequired:
|
||||
return "", fmt.Errorf("insufficient credits to perform this request")
|
||||
case http.StatusForbidden:
|
||||
return "", fmt.Errorf("the endpoint requested is hidden for administrators only")
|
||||
case http.StatusNotFound:
|
||||
return "", fmt.Errorf("the specified endpoint could not be found")
|
||||
case http.StatusMethodNotAllowed:
|
||||
return "", fmt.Errorf("there need to try to access an endpoint with an invalid method")
|
||||
case http.StatusRequestTimeout:
|
||||
return "", fmt.Errorf("the request timed out. try again later")
|
||||
case http.StatusTooManyRequests:
|
||||
return "", fmt.Errorf("there are requesting too many results")
|
||||
case http.StatusInternalServerError:
|
||||
return "", fmt.Errorf("there had a problem with our server. try again later")
|
||||
case http.StatusBadGateway:
|
||||
return "", fmt.Errorf("there was a problem with the server. Please try again later")
|
||||
case http.StatusServiceUnavailable:
|
||||
return "", fmt.Errorf("there are temporarily offline for maintenance. please try again later")
|
||||
case http.StatusGatewayTimeout:
|
||||
return "", fmt.Errorf("there are temporarily offline for maintenance. please try again later")
|
||||
default:
|
||||
return "", fmt.Errorf("unexpected status code: %d", resp.StatusCode)
|
||||
}
|
||||
}
|
||||
|
||||
func (f *firecrawl) buildFirecrawlResult(ctx context.Context, result *firecrawlSearchResult) string {
|
||||
var writer strings.Builder
|
||||
writer.WriteString("# Links\n\n")
|
||||
|
||||
isMarkdownExists := false
|
||||
for i, res := range result.Data.Web {
|
||||
writer.WriteString(fmt.Sprintf("## %d. %s\n\n", i+1, res.resolvedTitle()))
|
||||
writer.WriteString(fmt.Sprintf("* URL %s\n\n", res.resolvedURL()))
|
||||
if res.Description != "" {
|
||||
writer.WriteString(fmt.Sprintf("### Short content\n\n%s\n\n", res.Description))
|
||||
}
|
||||
if res.Markdown != "" {
|
||||
isMarkdownExists = true
|
||||
}
|
||||
}
|
||||
|
||||
if isMarkdownExists && f.summarizer != nil {
|
||||
summarizePrompt, err := f.getSummarizePrompt(result)
|
||||
if err != nil {
|
||||
writer.WriteString(f.getContentFromResults(result.Data.Web))
|
||||
} else {
|
||||
summarizedContents, err := f.summarizer(ctx, summarizePrompt)
|
||||
if err != nil {
|
||||
writer.WriteString(f.getContentFromResults(result.Data.Web))
|
||||
} else {
|
||||
writer.WriteString(fmt.Sprintf("### Summarized Content\n\n%s\n\n", summarizedContents))
|
||||
}
|
||||
}
|
||||
} else {
|
||||
writer.WriteString(f.getContentFromResults(result.Data.Web))
|
||||
}
|
||||
|
||||
return writer.String()
|
||||
}
|
||||
|
||||
func (f *firecrawl) getContentFromResults(results []firecrawlResult) string {
|
||||
var writer strings.Builder
|
||||
for i, res := range results {
|
||||
if res.Markdown != "" {
|
||||
markdown := res.Markdown
|
||||
markdown = markdown[:min(len(markdown), maxRawContentLength)]
|
||||
writer.WriteString(fmt.Sprintf("### Raw content for %d. %s\n\n%s\n\n", i+1, res.resolvedTitle(), markdown))
|
||||
}
|
||||
}
|
||||
return writer.String()
|
||||
}
|
||||
|
||||
func (f *firecrawl) getSummarizePrompt(result *firecrawlSearchResult) (string, error) {
|
||||
templateText := `<instructions>
|
||||
TASK: Summarize web search results for the following user query:
|
||||
|
||||
DATA:
|
||||
- <raw_content> tags contain web page content with attributes: id, title, url
|
||||
- Content may include HTML, structured data, tables, or plain text
|
||||
|
||||
REQUIREMENTS:
|
||||
1. Create concise summary (max {{.MaxLength}} chars) that DIRECTLY answers the user query
|
||||
2. Preserve ALL critical facts, statistics, technical details, and numerical data
|
||||
3. Maintain all actionable insights, procedures, or code examples exactly as presented
|
||||
4. Keep ALL query-relevant information even if reducing overall length
|
||||
5. Highlight authoritative information and note contradictions between sources
|
||||
6. Cite sources using [Source #] format when presenting specific claims
|
||||
7. Ensure the user query is fully addressed in the summary
|
||||
8. NEVER remove information that answers the user's original question
|
||||
|
||||
FORMAT:
|
||||
- Begin with a direct answer to the user query
|
||||
- Organize thematically with clear structure using headings
|
||||
- Keep bullet points and numbered lists for clarity and steps
|
||||
- Include brief "Sources Overview" section identifying key references
|
||||
|
||||
The summary MUST provide complete answers to the user's query, preserving all relevant information.
|
||||
</instructions>
|
||||
|
||||
{{range $index, $result := .Results}}
|
||||
{{if $result.Markdown}}
|
||||
<raw_content id="{{$index}}" title="{{$result.Title}}" url="{{$result.URL}}">
|
||||
{{$result.Markdown}}
|
||||
</raw_content>
|
||||
{{end}}
|
||||
{{end}}`
|
||||
|
||||
templateContext := map[string]any{
|
||||
"MaxLength": maxRawContentLength,
|
||||
"Results": result.Data.Web,
|
||||
}
|
||||
|
||||
tmpl, err := template.New("summarize").Parse(templateText)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("error creating template: %v", err)
|
||||
}
|
||||
|
||||
var buf bytes.Buffer
|
||||
if err := tmpl.Execute(&buf, templateContext); err != nil {
|
||||
return "", fmt.Errorf("error executing template: %v", err)
|
||||
}
|
||||
|
||||
return buf.String(), nil
|
||||
}
|
||||
|
||||
func (f *firecrawl) IsAvailable() bool {
|
||||
return f.apiKey() != ""
|
||||
}
|
||||
|
||||
func (f *firecrawl) apiKey() string {
|
||||
if f.cfg == nil {
|
||||
return ""
|
||||
}
|
||||
|
||||
return f.cfg.FirecrawlAPIKey
|
||||
}
|
||||
|
||||
func (f *firecrawl) searchURL() string {
|
||||
baseURL := firecrawlDefaultURL
|
||||
if f.cfg != nil && f.cfg.FirecrawlAPIURL != "" {
|
||||
baseURL = f.cfg.FirecrawlAPIURL
|
||||
}
|
||||
|
||||
return strings.TrimRight(baseURL, "/") + firecrawlSearchPath
|
||||
}
|
||||
@@ -0,0 +1,515 @@
|
||||
package tools
|
||||
|
||||
import (
|
||||
"context"
|
||||
"io"
|
||||
"net/http"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"pentagi/pkg/config"
|
||||
"pentagi/pkg/database"
|
||||
)
|
||||
|
||||
const testFirecrawlAPIKey = "test-key"
|
||||
|
||||
func testFirecrawlConfig() *config.Config {
|
||||
return &config.Config{FirecrawlAPIKey: testFirecrawlAPIKey}
|
||||
}
|
||||
|
||||
func TestFirecrawlHandle(t *testing.T) {
|
||||
var seenRequest bool
|
||||
var receivedMethod string
|
||||
var receivedContentType string
|
||||
var receivedAuthorization string
|
||||
var receivedBody []byte
|
||||
|
||||
mockMux := http.NewServeMux()
|
||||
mockMux.HandleFunc("/v2/search", func(w http.ResponseWriter, r *http.Request) {
|
||||
seenRequest = true
|
||||
receivedMethod = r.Method
|
||||
receivedContentType = r.Header.Get("Content-Type")
|
||||
receivedAuthorization = r.Header.Get("Authorization")
|
||||
|
||||
var err error
|
||||
receivedBody, err = io.ReadAll(r.Body)
|
||||
if err != nil {
|
||||
t.Errorf("failed to read request body: %v", err)
|
||||
}
|
||||
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.WriteHeader(http.StatusOK)
|
||||
w.Write([]byte(`{"success":true,"data":{"web":[{"title":"Doc","description":"short","url":"https://example.com","markdown":"long markdown content","metadata":{"title":"Doc","sourceURL":"https://example.com"}}]}}`))
|
||||
})
|
||||
|
||||
proxy, err := newTestProxy("api.firecrawl.dev", mockMux)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to create proxy: %v", err)
|
||||
}
|
||||
defer proxy.Close()
|
||||
|
||||
flowID := int64(1)
|
||||
taskID := int64(10)
|
||||
subtaskID := int64(20)
|
||||
slp := &searchLogProviderMock{}
|
||||
|
||||
cfg := &config.Config{
|
||||
FirecrawlAPIKey: testFirecrawlAPIKey,
|
||||
ProxyURL: proxy.URL(),
|
||||
ExternalSSLCAPath: proxy.CACertPath(),
|
||||
ExternalSSLInsecure: false,
|
||||
}
|
||||
|
||||
fc := NewFirecrawlTool(cfg, flowID, &taskID, &subtaskID, slp, nil)
|
||||
|
||||
ctx := PutAgentContext(t.Context(), database.MsgchainTypeSearcher)
|
||||
got, err := fc.Handle(
|
||||
ctx,
|
||||
FirecrawlToolName,
|
||||
[]byte(`{"query":"test query","max_results":5,"message":"m"}`),
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("Handle() unexpected error: %v", err)
|
||||
}
|
||||
|
||||
// Verify mock handler was called
|
||||
if !seenRequest {
|
||||
t.Fatal("request was not intercepted by proxy - mock handler was not called")
|
||||
}
|
||||
|
||||
// Verify request was built correctly
|
||||
if receivedMethod != http.MethodPost {
|
||||
t.Errorf("request method = %q, want POST", receivedMethod)
|
||||
}
|
||||
if receivedContentType != "application/json" {
|
||||
t.Errorf("Content-Type = %q, want application/json", receivedContentType)
|
||||
}
|
||||
if receivedAuthorization != "Bearer "+testFirecrawlAPIKey {
|
||||
t.Errorf("Authorization = %q, want %q", receivedAuthorization, "Bearer "+testFirecrawlAPIKey)
|
||||
}
|
||||
if !strings.Contains(string(receivedBody), `"query":"test query"`) {
|
||||
t.Errorf("request body = %q, expected to contain query", string(receivedBody))
|
||||
}
|
||||
if !strings.Contains(string(receivedBody), `"limit":5`) {
|
||||
t.Errorf("request body = %q, expected to contain limit", string(receivedBody))
|
||||
}
|
||||
if !strings.Contains(string(receivedBody), `"markdown"`) {
|
||||
t.Errorf("request body = %q, expected to contain markdown format", string(receivedBody))
|
||||
}
|
||||
|
||||
// Verify response was parsed correctly
|
||||
if !strings.Contains(got, "# Links") {
|
||||
t.Errorf("result missing '# Links' section: %q", got)
|
||||
}
|
||||
if !strings.Contains(got, "https://example.com") {
|
||||
t.Errorf("result missing expected URL 'https://example.com': %q", got)
|
||||
}
|
||||
|
||||
// Verify search log was written with agent context
|
||||
if slp.calls != 1 {
|
||||
t.Errorf("PutLog() calls = %d, want 1", slp.calls)
|
||||
}
|
||||
if slp.engine != database.SearchengineTypeFirecrawl {
|
||||
t.Errorf("engine = %q, want %q", slp.engine, database.SearchengineTypeFirecrawl)
|
||||
}
|
||||
if slp.query != "test query" {
|
||||
t.Errorf("logged query = %q, want %q", slp.query, "test query")
|
||||
}
|
||||
if slp.parentType != database.MsgchainTypeSearcher {
|
||||
t.Errorf("parent agent type = %q, want %q", slp.parentType, database.MsgchainTypeSearcher)
|
||||
}
|
||||
if slp.currType != database.MsgchainTypeSearcher {
|
||||
t.Errorf("current agent type = %q, want %q", slp.currType, database.MsgchainTypeSearcher)
|
||||
}
|
||||
if slp.taskID == nil || *slp.taskID != taskID {
|
||||
t.Errorf("task ID = %v, want %d", slp.taskID, taskID)
|
||||
}
|
||||
if slp.subtaskID == nil || *slp.subtaskID != subtaskID {
|
||||
t.Errorf("subtask ID = %v, want %d", slp.subtaskID, subtaskID)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFirecrawlIsAvailable(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
cfg *config.Config
|
||||
want bool
|
||||
}{
|
||||
{
|
||||
name: "available when API key is set",
|
||||
cfg: testFirecrawlConfig(),
|
||||
want: true,
|
||||
},
|
||||
{
|
||||
name: "unavailable when API key is empty",
|
||||
cfg: &config.Config{},
|
||||
want: false,
|
||||
},
|
||||
{
|
||||
name: "unavailable when nil config",
|
||||
cfg: nil,
|
||||
want: false,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
fc := &firecrawl{cfg: tt.cfg}
|
||||
if got := fc.IsAvailable(); got != tt.want {
|
||||
t.Errorf("IsAvailable() = %v, want %v", got, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestFirecrawlSearchURL(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
cfg *config.Config
|
||||
want string
|
||||
}{
|
||||
{
|
||||
name: "default when URL is empty",
|
||||
cfg: &config.Config{},
|
||||
want: "https://api.firecrawl.dev/v2/search",
|
||||
},
|
||||
{
|
||||
name: "custom self-hosted URL",
|
||||
cfg: &config.Config{FirecrawlAPIURL: "https://firecrawl.internal:3002"},
|
||||
want: "https://firecrawl.internal:3002/v2/search",
|
||||
},
|
||||
{
|
||||
name: "custom URL with trailing slash",
|
||||
cfg: &config.Config{FirecrawlAPIURL: "https://firecrawl.internal/"},
|
||||
want: "https://firecrawl.internal/v2/search",
|
||||
},
|
||||
{
|
||||
name: "nil config falls back to default",
|
||||
cfg: nil,
|
||||
want: "https://api.firecrawl.dev/v2/search",
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
fc := &firecrawl{cfg: tt.cfg}
|
||||
if got := fc.searchURL(); got != tt.want {
|
||||
t.Errorf("searchURL() = %q, want %q", got, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestFirecrawlParseHTTPResponse_StatusAndDecodeErrors(t *testing.T) {
|
||||
fc := &firecrawl{flowID: 1}
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
statusCode int
|
||||
body string
|
||||
wantErr bool
|
||||
errContain string
|
||||
}{
|
||||
{
|
||||
name: "successful response",
|
||||
statusCode: http.StatusOK,
|
||||
body: `{"success":true,"data":{"web":[{"title":"A","description":"c","url":"https://a.com"}]}}`,
|
||||
wantErr: false,
|
||||
},
|
||||
{
|
||||
name: "success false with error message",
|
||||
statusCode: http.StatusOK,
|
||||
body: `{"success":false,"error":"Request timed out"}`,
|
||||
wantErr: true,
|
||||
errContain: "Request timed out",
|
||||
},
|
||||
{
|
||||
name: "decode error",
|
||||
statusCode: http.StatusOK,
|
||||
body: "{invalid json",
|
||||
wantErr: true,
|
||||
errContain: "failed to decode response body",
|
||||
},
|
||||
{
|
||||
name: "bad request",
|
||||
statusCode: http.StatusBadRequest,
|
||||
body: "",
|
||||
wantErr: true,
|
||||
errContain: "invalid",
|
||||
},
|
||||
{
|
||||
name: "unauthorized",
|
||||
statusCode: http.StatusUnauthorized,
|
||||
body: "",
|
||||
wantErr: true,
|
||||
errContain: "API key",
|
||||
},
|
||||
{
|
||||
name: "payment required",
|
||||
statusCode: http.StatusPaymentRequired,
|
||||
body: "",
|
||||
wantErr: true,
|
||||
errContain: "credits",
|
||||
},
|
||||
{
|
||||
name: "forbidden",
|
||||
statusCode: http.StatusForbidden,
|
||||
body: "",
|
||||
wantErr: true,
|
||||
errContain: "administrators only",
|
||||
},
|
||||
{
|
||||
name: "not found",
|
||||
statusCode: http.StatusNotFound,
|
||||
body: "",
|
||||
wantErr: true,
|
||||
errContain: "could not be found",
|
||||
},
|
||||
{
|
||||
name: "method not allowed",
|
||||
statusCode: http.StatusMethodNotAllowed,
|
||||
body: "",
|
||||
wantErr: true,
|
||||
errContain: "invalid method",
|
||||
},
|
||||
{
|
||||
name: "request timeout",
|
||||
statusCode: http.StatusRequestTimeout,
|
||||
body: "",
|
||||
wantErr: true,
|
||||
errContain: "timed out",
|
||||
},
|
||||
{
|
||||
name: "too many requests",
|
||||
statusCode: http.StatusTooManyRequests,
|
||||
body: "",
|
||||
wantErr: true,
|
||||
errContain: "too many",
|
||||
},
|
||||
{
|
||||
name: "internal server error",
|
||||
statusCode: http.StatusInternalServerError,
|
||||
body: "",
|
||||
wantErr: true,
|
||||
errContain: "server",
|
||||
},
|
||||
{
|
||||
name: "bad gateway",
|
||||
statusCode: http.StatusBadGateway,
|
||||
body: "",
|
||||
wantErr: true,
|
||||
errContain: "server",
|
||||
},
|
||||
{
|
||||
name: "service unavailable",
|
||||
statusCode: http.StatusServiceUnavailable,
|
||||
body: "",
|
||||
wantErr: true,
|
||||
errContain: "offline",
|
||||
},
|
||||
{
|
||||
name: "gateway timeout",
|
||||
statusCode: http.StatusGatewayTimeout,
|
||||
body: "",
|
||||
wantErr: true,
|
||||
errContain: "offline",
|
||||
},
|
||||
{
|
||||
name: "unknown status code",
|
||||
statusCode: 418,
|
||||
body: "",
|
||||
wantErr: true,
|
||||
errContain: "unexpected status code",
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
resp := &http.Response{
|
||||
StatusCode: tt.statusCode,
|
||||
Body: io.NopCloser(strings.NewReader(tt.body)),
|
||||
}
|
||||
result, err := fc.parseHTTPResponse(t.Context(), resp)
|
||||
|
||||
if !tt.wantErr {
|
||||
if err != nil {
|
||||
t.Errorf("parseHTTPResponse() unexpected error: %v", err)
|
||||
}
|
||||
if !strings.Contains(result, "# Links") {
|
||||
t.Errorf("parseHTTPResponse() result missing '# Links': %q", result)
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
if err == nil {
|
||||
t.Fatal("parseHTTPResponse() expected error, got nil")
|
||||
}
|
||||
if !strings.Contains(err.Error(), tt.errContain) {
|
||||
t.Errorf("parseHTTPResponse() error = %q, want to contain %q", err.Error(), tt.errContain)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestFirecrawlBuildResult_WithSummarizer(t *testing.T) {
|
||||
t.Run("uses summarizer when markdown content exists", func(t *testing.T) {
|
||||
fc := &firecrawl{
|
||||
summarizer: func(ctx context.Context, prompt string) (string, error) {
|
||||
if !strings.Contains(prompt, "<raw_content") {
|
||||
t.Fatalf("summarizer prompt must include raw content, got: %q", prompt)
|
||||
}
|
||||
return "short summary", nil
|
||||
},
|
||||
}
|
||||
|
||||
out := fc.buildFirecrawlResult(t.Context(), &firecrawlSearchResult{
|
||||
Success: true,
|
||||
Data: firecrawlData{
|
||||
Web: []firecrawlResult{
|
||||
{
|
||||
Title: "Title",
|
||||
Description: "content",
|
||||
URL: "https://example.com",
|
||||
Markdown: "very long markdown content",
|
||||
},
|
||||
},
|
||||
},
|
||||
})
|
||||
|
||||
if !strings.Contains(out, "### Summarized Content") {
|
||||
t.Errorf("buildFirecrawlResult() missing '### Summarized Content', got: %q", out)
|
||||
}
|
||||
if !strings.Contains(out, "short summary") {
|
||||
t.Errorf("buildFirecrawlResult() missing 'short summary', got: %q", out)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("falls back to markdown content when no summarizer", func(t *testing.T) {
|
||||
fc := &firecrawl{}
|
||||
|
||||
out := fc.buildFirecrawlResult(t.Context(), &firecrawlSearchResult{
|
||||
Success: true,
|
||||
Data: firecrawlData{
|
||||
Web: []firecrawlResult{
|
||||
{
|
||||
Title: "Title",
|
||||
Description: "content",
|
||||
URL: "https://example.com",
|
||||
Markdown: "very long markdown content",
|
||||
},
|
||||
},
|
||||
},
|
||||
})
|
||||
|
||||
if !strings.Contains(out, "### Raw content for") {
|
||||
t.Errorf("buildFirecrawlResult() missing '### Raw content for', got: %q", out)
|
||||
}
|
||||
if !strings.Contains(out, "very long markdown content") {
|
||||
t.Errorf("buildFirecrawlResult() missing markdown content, got: %q", out)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("no content sections when markdown is empty", func(t *testing.T) {
|
||||
fc := &firecrawl{}
|
||||
|
||||
out := fc.buildFirecrawlResult(t.Context(), &firecrawlSearchResult{
|
||||
Success: true,
|
||||
Data: firecrawlData{
|
||||
Web: []firecrawlResult{
|
||||
{
|
||||
Title: "Title",
|
||||
Description: "content",
|
||||
URL: "https://example.com",
|
||||
},
|
||||
},
|
||||
},
|
||||
})
|
||||
|
||||
if strings.Contains(out, "### Raw content for") {
|
||||
t.Errorf("buildFirecrawlResult() should not have raw content section, got: %q", out)
|
||||
}
|
||||
if strings.Contains(out, "### Summarized Content") {
|
||||
t.Errorf("buildFirecrawlResult() should not have summarized content section, got: %q", out)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("falls back to metadata url and title", func(t *testing.T) {
|
||||
fc := &firecrawl{}
|
||||
|
||||
out := fc.buildFirecrawlResult(t.Context(), &firecrawlSearchResult{
|
||||
Success: true,
|
||||
Data: firecrawlData{
|
||||
Web: []firecrawlResult{
|
||||
{
|
||||
Description: "content",
|
||||
Metadata: &firecrawlMetadata{
|
||||
Title: "Meta Title",
|
||||
SourceURL: "https://meta.example.com",
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
})
|
||||
|
||||
if !strings.Contains(out, "Meta Title") {
|
||||
t.Errorf("buildFirecrawlResult() missing metadata title, got: %q", out)
|
||||
}
|
||||
if !strings.Contains(out, "https://meta.example.com") {
|
||||
t.Errorf("buildFirecrawlResult() missing metadata source URL, got: %q", out)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
func TestFirecrawlHandle_ValidationAndSwallowedError(t *testing.T) {
|
||||
t.Run("invalid json", func(t *testing.T) {
|
||||
fc := &firecrawl{cfg: testFirecrawlConfig()}
|
||||
_, err := fc.Handle(t.Context(), FirecrawlToolName, []byte("{"))
|
||||
if err == nil || !strings.Contains(err.Error(), "failed to unmarshal") {
|
||||
t.Fatalf("expected unmarshal error, got: %v", err)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("search error swallowed", func(t *testing.T) {
|
||||
var seenRequest bool
|
||||
mockMux := http.NewServeMux()
|
||||
mockMux.HandleFunc("/v2/search", func(w http.ResponseWriter, r *http.Request) {
|
||||
seenRequest = true
|
||||
w.WriteHeader(http.StatusBadGateway)
|
||||
})
|
||||
|
||||
proxy, err := newTestProxy("api.firecrawl.dev", mockMux)
|
||||
if err != nil {
|
||||
t.Fatalf("failed to create proxy: %v", err)
|
||||
}
|
||||
defer proxy.Close()
|
||||
|
||||
fc := &firecrawl{
|
||||
flowID: 1,
|
||||
cfg: &config.Config{
|
||||
FirecrawlAPIKey: testFirecrawlAPIKey,
|
||||
ProxyURL: proxy.URL(),
|
||||
ExternalSSLCAPath: proxy.CACertPath(),
|
||||
ExternalSSLInsecure: false,
|
||||
},
|
||||
}
|
||||
|
||||
result, err := fc.Handle(
|
||||
t.Context(),
|
||||
FirecrawlToolName,
|
||||
[]byte(`{"query":"q","max_results":5,"message":"m"}`),
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("Handle() unexpected error: %v", err)
|
||||
}
|
||||
|
||||
// Verify mock handler was called (request was intercepted)
|
||||
if !seenRequest {
|
||||
t.Error("request was not intercepted by proxy - mock handler was not called")
|
||||
}
|
||||
|
||||
// Verify error was swallowed and returned as string
|
||||
if !strings.Contains(result, "failed to search in firecrawl") {
|
||||
t.Errorf("Handle() = %q, expected swallowed error message", result)
|
||||
}
|
||||
})
|
||||
}
|
||||
@@ -24,6 +24,7 @@ const (
|
||||
GoogleToolName = "google"
|
||||
DuckDuckGoToolName = "duckduckgo"
|
||||
TavilyToolName = "tavily"
|
||||
FirecrawlToolName = "firecrawl"
|
||||
TraversaalToolName = "traversaal"
|
||||
PerplexityToolName = "perplexity"
|
||||
SearxngToolName = "searxng"
|
||||
@@ -109,6 +110,7 @@ var toolsTypeMapping = map[string]ToolType{
|
||||
GoogleToolName: SearchNetworkToolType,
|
||||
DuckDuckGoToolName: SearchNetworkToolType,
|
||||
TavilyToolName: SearchNetworkToolType,
|
||||
FirecrawlToolName: SearchNetworkToolType,
|
||||
TraversaalToolName: SearchNetworkToolType,
|
||||
PerplexityToolName: SearchNetworkToolType,
|
||||
SearxngToolName: SearchNetworkToolType,
|
||||
@@ -153,6 +155,7 @@ var allowedStoringInMemoryTools = []string{
|
||||
GoogleToolName,
|
||||
DuckDuckGoToolName,
|
||||
TavilyToolName,
|
||||
FirecrawlToolName,
|
||||
TraversaalToolName,
|
||||
PerplexityToolName,
|
||||
SearxngToolName,
|
||||
@@ -230,6 +233,13 @@ var registryDefinitions = map[string]llms.FunctionDefinition{
|
||||
"with answer by query and detailed information from the web sites",
|
||||
Parameters: reflector.Reflect(&SearchAction{}),
|
||||
},
|
||||
FirecrawlToolName: {
|
||||
Name: FirecrawlToolName,
|
||||
Description: "Search in the firecrawl search engine, it combines web search with page scraping to return " +
|
||||
"full-page markdown content for each result, ideal for deep research on complex technical topics " +
|
||||
"and reading documentation directly from the discovered web sites",
|
||||
Parameters: reflector.Reflect(&SearchAction{}),
|
||||
},
|
||||
TraversaalToolName: {
|
||||
Name: TraversaalToolName,
|
||||
Description: "Search in the traversaal search engine, presents you answer and web-links " +
|
||||
@@ -439,8 +449,8 @@ func getMessageType(name string) database.MsglogType {
|
||||
return database.MsglogTypeFile
|
||||
case BrowserToolName:
|
||||
return database.MsglogTypeBrowser
|
||||
case MemoristToolName, SearchToolName, GoogleToolName, DuckDuckGoToolName, TavilyToolName, TraversaalToolName,
|
||||
PerplexityToolName, SearxngToolName, SploitusToolName,
|
||||
case MemoristToolName, SearchToolName, GoogleToolName, DuckDuckGoToolName, TavilyToolName, FirecrawlToolName,
|
||||
TraversaalToolName, PerplexityToolName, SearxngToolName, SploitusToolName,
|
||||
SearchGuideToolName, SearchAnswerToolName, SearchCodeToolName, SearchInMemoryToolName, GraphitiSearchToolName:
|
||||
return database.MsglogTypeSearch
|
||||
case AdviceToolName:
|
||||
|
||||
@@ -939,6 +939,17 @@ func (fte *flowToolsExecutor) GetAssistantExecutor(cfg AssistantExecutorConfig)
|
||||
handlers[TavilyToolName] = tavily.Handle
|
||||
}
|
||||
|
||||
firecrawl := NewFirecrawlTool(
|
||||
fte.cfg,
|
||||
fte.flowID, nil, nil,
|
||||
fte.slp,
|
||||
cfg.Summarizer,
|
||||
)
|
||||
if firecrawl.IsAvailable() {
|
||||
definitions = append(definitions, registryDefinitions[FirecrawlToolName])
|
||||
handlers[FirecrawlToolName] = firecrawl.Handle
|
||||
}
|
||||
|
||||
traversaal := NewTraversaalTool(
|
||||
fte.cfg,
|
||||
fte.flowID, nil, nil,
|
||||
@@ -1541,6 +1552,19 @@ func (fte *flowToolsExecutor) GetSearcherExecutor(cfg SearcherExecutorConfig) (C
|
||||
ce.handlers[TavilyToolName] = tavily.Handle
|
||||
}
|
||||
|
||||
firecrawl := NewFirecrawlTool(
|
||||
fte.cfg,
|
||||
fte.flowID,
|
||||
cfg.TaskID,
|
||||
cfg.SubtaskID,
|
||||
fte.slp,
|
||||
cfg.Summarizer,
|
||||
)
|
||||
if firecrawl.IsAvailable() {
|
||||
ce.definitions = append(ce.definitions, registryDefinitions[FirecrawlToolName])
|
||||
ce.handlers[FirecrawlToolName] = firecrawl.Handle
|
||||
}
|
||||
|
||||
traversaal := NewTraversaalTool(
|
||||
fte.cfg,
|
||||
fte.flowID,
|
||||
|
||||
@@ -151,6 +151,8 @@ services:
|
||||
- GOOGLE_LR_KEY=${GOOGLE_LR_KEY:-}
|
||||
- TRAVERSAAL_API_KEY=${TRAVERSAAL_API_KEY:-}
|
||||
- TAVILY_API_KEY=${TAVILY_API_KEY:-}
|
||||
- FIRECRAWL_API_KEY=${FIRECRAWL_API_KEY:-}
|
||||
- FIRECRAWL_API_URL=${FIRECRAWL_API_URL:-https://api.firecrawl.dev}
|
||||
- PERPLEXITY_API_KEY=${PERPLEXITY_API_KEY:-}
|
||||
- PERPLEXITY_MODEL=${PERPLEXITY_MODEL:-sonar}
|
||||
- PERPLEXITY_CONTEXT_SIZE=${PERPLEXITY_CONTEXT_SIZE:-low}
|
||||
|
||||
Reference in New Issue
Block a user