mirror of
https://github.com/codewhale-hq/Codewhale.git
synced 2026-11-11 10:01:30 -10:00
feat(search): SearXNG results rank by score; self-host docs for the JSON requirement (web-search slice B)
Per codewhale-ops/WEB-SEARCH-20260917.md slice B, scoped to `parse_searxng_results`: - New `searxng_score(&Value)`: f64 / i64 / trimmed numeric string -> f64; anything else or non-finite -> 0.0. Rows are collected with their score and stable-sorted descending (`sort_by(|a, b| b.0.total_cmp(&a.0))` — total order, so a NaN cannot sneak past a partial_cmp fallback; Rust's `sort_by` is the stable one, so equal scores keep instance order), then truncated to `max_results`. Empty title/url skip rules and the 403/429 copy are unchanged; `preflight_search_provider` / `run_tavily_search` (slice A's surfaces) have no hunk in this diff. - Tests: helper arm coverage incl. "NaN"/"inf" clamps, higher-score-later promotion, equal-score insertion order, missing/invalid score last (still included), cap applied after the sort. Existing `searxng_parser_normalizes_results` untouched and green (no scores -> 0.0 -> original order, so the old visible behavior is the degenerate case). - Docs: the existing SearXNG subsection (EN + zh_hans pair) now says how to self-host the separate process: `settings.yml` `search.formats` must include `json` (HTML-only instances answer 403, which Codewhale already explains), bind to loopback or a network-policy-allowed host, `provider = "searxng"` with required `base_url` (no default/discovered instance), doctor `--probe-search` is a transport-only HEAD with no `q=`, and a `curl … | jq '.results[0]'` smoke. No public instance is named. - `config.example.toml`: one added line — the JSON requirement, next to the `searxng:` entry; slice A's Tavily env list untouched. Evidence (local, macOS arm64): `cargo test -p codewhale-tui --lib --locked searxng` => "15 passed; 0 failed" (re-run by the parent); `--all-targets` check clean (32.6s); `cargo fmt --all -- --check` clean. Doc claims cross-checked against docs.searxng.org search-API and settings pages.
This commit is contained in:
CodeWhale Bot committed
1 parent
95bc5c008f
commit
66022cf60b
4 files changed
+199
-10
No files matched your search
@@ -1034,6 +1034,7 @@ max_subagents = 10 # optional (default 64, clamped to 1-128)
|
||||
# # 设置 METASO_API_KEY 或 [search] api_key
|
||||
# # searxng: https://docs.searxng.org — trusted/self-hosted JSON API,
|
||||
# # set base_url; no public instance is used by default
|
||||
# # instance must enable JSON (settings.yml search.formats); HTML-only instances 403
|
||||
# # baidu: 百度 AI Search via qianfan.baidubce.com，需 api_key
|
||||
# # volcengine: 火山引擎 Ark web_search (免费 2 万次/月), 需 api_key
|
||||
# # 也回退到 VOLCENGINE_API_KEY / VOLCENGINE_ARK_API_KEY / ARK_API_KEY 环境变量
|
||||
|
||||
@@ -1805,8 +1805,34 @@ fn parse_baidu_results(parsed: &Value, max_results: usize) -> Vec<WebSearchEntry
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// Read a SearXNG result `score`.
|
||||
///
|
||||
/// SearXNG emits a float, but instances and versions vary: a JSON integer, a
|
||||
/// numeric string, or no `score` at all are all tolerated. Unusable or
|
||||
/// non-finite values (`"not-a-number"`, `"NaN"`, `"inf"`, missing) read as
|
||||
/// `0.0`, so such rows keep their input order behind scored rows instead of
|
||||
/// being dropped or sorted by NaN.
|
||||
fn searxng_score(item: &Value) -> f64 {
|
||||
let raw = item.get("score");
|
||||
let n = raw
|
||||
.and_then(Value::as_f64)
|
||||
.or_else(|| raw.and_then(Value::as_i64).map(|i| i as f64))
|
||||
.or_else(|| {
|
||||
raw.and_then(Value::as_str)
|
||||
.and_then(|s| s.trim().parse().ok())
|
||||
})
|
||||
.unwrap_or(0.0);
|
||||
if n.is_finite() { n } else { 0.0 }
|
||||
}
|
||||
|
||||
/// Normalize a SearXNG JSON response into the engine-agnostic result shape.
|
||||
///
|
||||
/// Rows without a non-empty `title` or `url` are skipped. Everything else is
|
||||
/// ordered by descending `score` with a stable sort (equal scores keep the
|
||||
/// instance's order) and only then capped, so a strong late row is not lost to
|
||||
/// an earlier `take` over the raw instance order.
|
||||
fn parse_searxng_results(parsed: &Value, max_results: usize) -> Vec<WebSearchEntry> {
|
||||
parsed
|
||||
let mut scored: Vec<(f64, WebSearchEntry)> = parsed
|
||||
.get("results")
|
||||
.and_then(|v| v.as_array())
|
||||
.into_iter()
|
||||
@@ -1818,14 +1844,21 @@ fn parse_searxng_results(parsed: &Value, max_results: usize) -> Vec<WebSearchEnt
|
||||
return None;
|
||||
}
|
||||
let snippet = first_non_empty_string(item, &["content", "snippet"]);
|
||||
Some(WebSearchEntry {
|
||||
title: title.to_string(),
|
||||
url: url.to_string(),
|
||||
snippet,
|
||||
})
|
||||
Some((
|
||||
searxng_score(item),
|
||||
WebSearchEntry {
|
||||
title: title.to_string(),
|
||||
url: url.to_string(),
|
||||
snippet,
|
||||
},
|
||||
))
|
||||
})
|
||||
.take(max_results)
|
||||
.collect()
|
||||
.collect();
|
||||
|
||||
scored.sort_by(|a, b| b.0.total_cmp(&a.0));
|
||||
scored.truncate(max_results);
|
||||
|
||||
scored.into_iter().map(|(_, entry)| entry).collect()
|
||||
}
|
||||
|
||||
fn baidu_error_message(parsed: &Value) -> Option<String> {
|
||||
@@ -2280,8 +2313,8 @@ mod tests {
|
||||
parse_baidu_results, parse_bocha_results, parse_metaso_results, parse_searxng_results,
|
||||
parse_serply_results, parse_sofya_results, parse_tavily_results, parse_volcengine_results,
|
||||
register_search_citations, rerank, run_scrape_search_with_endpoints, sanitize_error_body,
|
||||
search_probe_target, search_timeout_budgets, searxng_search_url, serply_search_url,
|
||||
truncate_error_body, volcengine_extract_text,
|
||||
search_probe_target, search_timeout_budgets, searxng_score, searxng_search_url,
|
||||
serply_search_url, truncate_error_body, volcengine_extract_text,
|
||||
};
|
||||
use crate::config::SearchProvider;
|
||||
use crate::tools::web::contract::{
|
||||
@@ -3260,6 +3293,120 @@ mod tests {
|
||||
assert_eq!(results[1].snippet.as_deref(), Some("Fallback snippet"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn searxng_score_reads_floats_integers_strings_and_clamps_junk() {
|
||||
assert_eq!(searxng_score(&json!({"score": 0.75})), 0.75);
|
||||
assert_eq!(searxng_score(&json!({"score": 1})), 1.0);
|
||||
assert_eq!(searxng_score(&json!({"score": " 2.5 "})), 2.5);
|
||||
assert_eq!(searxng_score(&json!({"score": "-1.5"})), -1.5);
|
||||
assert_eq!(searxng_score(&json!({})), 0.0);
|
||||
assert_eq!(searxng_score(&json!({"score": null})), 0.0);
|
||||
assert_eq!(searxng_score(&json!({"score": true})), 0.0);
|
||||
assert_eq!(searxng_score(&json!({"score": ""})), 0.0);
|
||||
assert_eq!(searxng_score(&json!({"score": "not-a-number"})), 0.0);
|
||||
assert_eq!(searxng_score(&json!({"score": {"nested": 1.0}})), 0.0);
|
||||
assert_eq!(
|
||||
searxng_score(&json!({"score": "NaN"})),
|
||||
0.0,
|
||||
"a non-finite score must not reach the sort"
|
||||
);
|
||||
assert_eq!(
|
||||
searxng_score(&json!({"score": "inf"})),
|
||||
0.0,
|
||||
"an infinite score must not outrank every finite row"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn searxng_parser_sorts_by_descending_score() {
|
||||
// The strongest row is last in the instance's own order; only the
|
||||
// score sort can promote it.
|
||||
let parsed = json!({
|
||||
"results": [
|
||||
{"title": "Low", "url": "https://example.com/low", "score": 0.25},
|
||||
{"title": "Middle", "url": "https://example.com/mid", "score": 1},
|
||||
{"title": "High", "url": "https://example.com/high", "score": "4.5"},
|
||||
{"title": "Zero", "url": "https://example.com/zero", "score": 0.0}
|
||||
]
|
||||
});
|
||||
|
||||
let titles: Vec<String> = parse_searxng_results(&parsed, 10)
|
||||
.into_iter()
|
||||
.map(|entry| entry.title)
|
||||
.collect();
|
||||
assert_eq!(titles, ["High", "Middle", "Low", "Zero"]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn searxng_parser_keeps_input_order_for_equal_scores() {
|
||||
let parsed = json!({
|
||||
"results": [
|
||||
{"title": "First", "url": "https://example.com/1", "score": 1.5},
|
||||
{"title": "Second", "url": "https://example.com/2", "score": 1.5},
|
||||
{"title": "Third", "url": "https://example.com/3", "score": 1.5},
|
||||
{"title": "Lower", "url": "https://example.com/4", "score": 1.4}
|
||||
]
|
||||
});
|
||||
|
||||
let titles: Vec<String> = parse_searxng_results(&parsed, 10)
|
||||
.into_iter()
|
||||
.map(|entry| entry.title)
|
||||
.collect();
|
||||
assert_eq!(titles, ["First", "Second", "Third", "Lower"]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn searxng_parser_sorts_missing_or_invalid_scores_last() {
|
||||
let parsed = json!({
|
||||
"results": [
|
||||
{"title": "No score", "url": "https://example.com/none"},
|
||||
{
|
||||
"title": "Garbage",
|
||||
"url": "https://example.com/garbage",
|
||||
"score": "not-a-number"
|
||||
},
|
||||
{"title": "NaN string", "url": "https://example.com/nan", "score": "NaN"},
|
||||
{"title": "Infinite string", "url": "https://example.com/inf", "score": "inf"},
|
||||
{"title": "Boolean", "url": "https://example.com/bool", "score": true},
|
||||
{"title": "Scored", "url": "https://example.com/scored", "score": 0.5}
|
||||
]
|
||||
});
|
||||
|
||||
let results = parse_searxng_results(&parsed, 10);
|
||||
let titles: Vec<&str> = results.iter().map(|entry| entry.title.as_str()).collect();
|
||||
// Every row with a title and a URL survives. Unusable scores read as
|
||||
// 0.0 and keep their input order behind the one scored row.
|
||||
assert_eq!(
|
||||
titles,
|
||||
[
|
||||
"Scored",
|
||||
"No score",
|
||||
"Garbage",
|
||||
"NaN string",
|
||||
"Infinite string",
|
||||
"Boolean"
|
||||
]
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn searxng_parser_caps_after_score_sort() {
|
||||
// A `take` before the sort would drop "Strong"; the cap must apply to
|
||||
// the ranked list instead.
|
||||
let parsed = json!({
|
||||
"results": [
|
||||
{"title": "Weak one", "url": "https://example.com/1", "score": 0.1},
|
||||
{"title": "Weak two", "url": "https://example.com/2", "score": 0.2},
|
||||
{"title": "Strong", "url": "https://example.com/3", "score": 9.0}
|
||||
]
|
||||
});
|
||||
|
||||
let results = parse_searxng_results(&parsed, 2);
|
||||
assert_eq!(results.len(), 2, "max_results caps the ranked list");
|
||||
assert_eq!(results[0].title, "Strong");
|
||||
assert_eq!(results[1].title, "Weak two");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn searxng_provider_requires_base_url() {
|
||||
use crate::config::SearchProvider;
|
||||
|
||||
@@ -2909,6 +2909,32 @@ configured instance's JSON API. Set `provider = "searxng"` and
|
||||
by default because public instances often disable JSON output or rate-limit API
|
||||
traffic.
|
||||
|
||||
Self-host it as a separate process (Docker is fine); Codewhale never bundles or
|
||||
manages the search engine itself:
|
||||
|
||||
- Enable JSON on the instance (`settings.yml`, `search.formats` must include
|
||||
`json`) and restart it. An HTML-only instance answers the API with HTTP 403;
|
||||
Codewhale reports that as a JSON/API-access problem on the SearXNG hop rather
|
||||
than silently returning no results.
|
||||
- Bind it to loopback, or to a host and port your network policy allows. The
|
||||
instance keeps its own engine list, limiter, and limits.
|
||||
- Point Codewhale at it with `[search] provider = "searxng"` and `base_url`
|
||||
(required; either the root URL or the `/search` endpoint). No instance ships
|
||||
as a default, and none is discovered automatically.
|
||||
- `codewhale doctor --probe-search` sends a transport-only `HEAD` to that
|
||||
origin — no `q=`, no credentials, no redirects, no audit receipt — so a green
|
||||
probe proves reachability and network-policy admission, not that JSON is on.
|
||||
|
||||
Confirm the JSON API itself before assuming a Codewhale bug:
|
||||
|
||||
```sh
|
||||
curl -sS "$BASE/search?q=codewhale&format=json" | jq '.results[0] | {title,url,score}'
|
||||
```
|
||||
|
||||
Codewhale ranks the returned rows by `score`, highest first, and applies
|
||||
`max_results` to that ranking; rows an instance reports without a usable score
|
||||
keep their original relative order.
|
||||
|
||||
**Metaso** ([metaso.cn](https://metaso.cn)) requires a user-supplied key. Set
|
||||
`METASO_API_KEY` or `[search] api_key`; Codewhale does not ship a shared key.
|
||||
|
||||
|
||||
@@ -1331,6 +1331,21 @@ exec_policy = true
|
||||
|
||||
**SearXNG**([docs](https://docs.searxng.org/dev/search_api.html))使用配置实例的 JSON API。设置 `provider = "searxng"` 和 `base_url = "https://your-searxng.example"`；Codewhale 调用 `/search?q=...&format=json`。Codewhale 默认不使用公共 SearXNG 实例，因为公共实例常禁用 JSON 输出或对 API 流量限速。
|
||||
|
||||
把它作为独立进程自托管（Docker 即可）；Codewhale 不内嵌也不管理搜索引擎本身：
|
||||
|
||||
- 在实例上启用 JSON（`settings.yml` 的 `search.formats` 必须包含 `json`）并重启。仅 HTML 的实例会对 API 返回 HTTP 403；Codewhale 会把它作为 SearXNG 这一跳的 JSON/API 访问问题报告出来，而不是静默返回空结果。
|
||||
- 绑定到 loopback，或绑定到你的网络策略允许的主机和端口。实例自带引擎列表、限流器和限额。
|
||||
- 用 `[search] provider = "searxng"` 和 `base_url` 指向它（必填；根 URL 或 `/search` 端点均可）。不随任何默认实例发布，也不会自动发现实例。
|
||||
- `codewhale doctor --probe-search` 只对该 origin 发送一个传输层 `HEAD`——不带 `q=`、不带凭据、不跟随重定向、不写审计回执——所以探测通过只证明可达性和网络策略放行，不证明 JSON 已开启。
|
||||
|
||||
在假定是 Codewhale 的 bug 之前，先用直接请求确认 JSON API 本身：
|
||||
|
||||
```sh
|
||||
curl -sS "$BASE/search?q=codewhale&format=json" | jq '.results[0] | {title,url,score}'
|
||||
```
|
||||
|
||||
Codewhale 按 `score` 从高到低排序返回行，再对排序结果应用 `max_results`；实例未给出可用分数的行保持原始相对顺序。
|
||||
|
||||
**秘塔(Metaso)**([metaso.cn](https://metaso.cn))需要用户提供的 key。设置 `METASO_API_KEY` 或 `[search] api_key`；Codewhale 不提供共享 key。
|
||||
|
||||
**Firecrawl**([docs](https://docs.firecrawl.dev/sdks/cli))用其有界的按 IP 每日配额无 key 搜索 Firecrawl Cloud。设置 `FIRECRAWL_API_KEY` 或 `[search] api_key` 用于认证限额。Codewhale 在无 key 模式不发送 `Authorization` 头。
|
||||
|
||||
Reference in new issue
Block a user