From a9cd8aceea89cde601ac3e46c8861c3b4fbbfa63 Mon Sep 17 00:00:00 2001 From: pomelo Date: Wed, 14 Oct 2026 02:33:54 +0000 Subject: [PATCH] docs(readme): simplify evaluation overview and refine trend chart (#14093) Co-authored-by: Qwen-Coder --- .github/assets/qwen-code-avg-score-trend.svg | 89 +++++++++++-------- README.md | 28 +----- docs/design/2026-10-14-readme-evaluation.md | 41 +++++++++ .../2026-10-14-readme-evaluation.zh-CN.md | 34 +++++++ docs/evaluation.md | 39 ++++++++ docs/evaluation.zh-CN.md | 37 ++++++++ 6 files changed, 208 insertions(+), 60 deletions(-) create mode 100644 docs/design/2026-10-14-readme-evaluation.md create mode 100644 docs/design/2026-10-14-readme-evaluation.zh-CN.md create mode 100644 docs/evaluation.md create mode 100644 docs/evaluation.zh-CN.md diff --git a/.github/assets/qwen-code-avg-score-trend.svg b/.github/assets/qwen-code-avg-score-trend.svg index 5987ec318bb..1408c0361df 100644 --- a/.github/assets/qwen-code-avg-score-trend.svg +++ b/.github/assets/qwen-code-avg-score-trend.svg @@ -1,37 +1,56 @@ - - - - - 76.0% - - 76.5% - - 77.0% - - 77.5% - - 78.0% - - - 77.80% - 0.0.14 - - 76.73% - 0.1.0 - - 76.60% - 0.5.0 - - 76.47% - 0.10.0 - - 77.67% - 0.15.0 - - 76.40% - 0.19.0 - - 77.33% - 0.22.0 + + Average SWE-bench Verified score by Qwen Code version + Qwen 3.7 Max, 500 tasks, three trials per version. Average scores: 0.0.14, 77.80%; 0.1.0, 76.73%; 0.5.0, 76.60%; 0.10.0, 76.47%; 0.15.0, 77.67%; 0.19.0, 76.40%; 0.22.0, 77.33%. The line connects evaluated versions in release order. The vertical axis spans 70% to 80%. + + + + Average score by version + SWE-bench Verified · Qwen 3.7 Max · y-axis: 70–80% + + 80% + + 75% + + 70% + + + + 77.80% + 0.0.14 + + 76.73% + 0.1.0 + + 76.60% + 0.5.0 + + 76.47% + 0.10.0 + + 77.67% + 0.15.0 + + 76.40% + 0.19.0 + + 77.33% + 0.22.0 diff --git a/README.md b/README.md index 46ca52863f2..09942c1edd7 100644 --- a/README.md +++ b/README.md @@ -157,33 +157,11 @@ If you know Claude Code, you already know Qwen Code — and then some. We've put ## Qwen Code Evaluation -### Evaluation Configuration +On **SWE-bench Verified**, seven Qwen Code versions scored **76.40%–77.80%** on average using the same Qwen 3.7 Max model (500 tasks, three trials per version). -| Configuration | Value | -| ----------------------- | --------------------------------------------------------------------------------------------------- | -| Dataset | `princeton-nlp/SWE-bench_Verified`, 500 cases | -| Runs | 3 trials per version, 1,500 jobs per version; 7 Qwen Code versions | -| Model | `Qwen 3.7 Max` | -| Sampling | `temperature=1`, `top_p=0.95` | -| Thinking / Reasoning | `interleaved_thinking=true`, `thinking_type=adaptive`, `reasoning_effort=high` | -| Context / Output Limits | `context_1m=true`, `model_context_length=1,000,000`, `max_tokens=64,000`, `model_max_tokens=65,535` | -| Harness Limits | `max_iterations=500`, `runtime_timeout_sec=7,200`, `agent_timeout_multiplier=3.0` | +![Average SWE-bench Verified scores by Qwen Code version: 0.0.14, 77.80%; 0.1.0, 76.73%; 0.5.0, 76.60%; 0.10.0, 76.47%; 0.15.0, 77.67%; 0.19.0, 76.40%; 0.22.0, 77.33%.](./.github/assets/qwen-code-avg-score-trend.svg) -### Scores by Qwen Code Version - -| Version | Avg Score | Pass Rate | pass@3 | pass^3 | -| ------- | ---------: | --------: | -----: | -----: | -| 0.0.14 | **77.80%** | 84.6% | 84.6% | 69.2% | -| 0.1.0 | **76.73%** | 84.6% | 84.6% | 68.0% | -| 0.5.0 | **76.60%** | 83.0% | 83.0% | 68.4% | -| 0.10.0 | **76.47%** | 84.4% | 84.4% | 67.4% | -| 0.15.0 | **77.67%** | 84.6% | 84.6% | 68.6% | -| 0.19.0 | **76.40%** | 85.0% | 85.0% | 65.8% | -| 0.22.0 | **77.33%** | 84.4% | 84.4% | 68.0% | - -### Avg Score Trend - -![Qwen Code version Avg Score trend](./.github/assets/qwen-code-avg-score-trend.svg) +[Full results and evaluation configuration →](./docs/evaluation.md) ## Ecosystem diff --git a/docs/design/2026-10-14-readme-evaluation.md b/docs/design/2026-10-14-readme-evaluation.md new file mode 100644 index 00000000000..40592e03298 --- /dev/null +++ b/docs/design/2026-10-14-readme-evaluation.md @@ -0,0 +1,41 @@ +# README Evaluation Presentation + +[English](2026-10-14-readme-evaluation.md) | [简体中文](2026-10-14-readme-evaluation.zh-CN.md) + +## Problem + +The README expands evaluation configuration, a results table, and a large chart +before readers reach the ecosystem section. The chart's 76%–78% axis visually +amplifies the 1.40 percentage point spread across versions. + +## Design + +- Keep the existing evaluation heading and anchor. Show one sentence of benchmark + context, a compact chart, and a link to the full results. +- Move all existing configuration and result values to linked English and Chinese + evaluation documents under `docs/`. +- Use a compact line chart to show the version sequence. Widen the vertical axis + to 70%–80% and label that range in the subtitle and ticks. Preserve release + order and exact score labels; connect observed points with straight segments + without smoothing. Use one Qwen purple accent, quiet grid lines, and readable + text. Support light and dark backgrounds. +- Retain the existing SVG path so incoming image references continue to work. + Include accessible title, description, and Markdown alternative text. +- Change presentation only. Do not update benchmark results or claim statistical + significance, improvement, or results for untested releases. + +## Validation and Acceptance Criteria + +- The README evaluation section has no expanded tables or third-level headings. +- Both evaluation documents preserve every score and configuration value from + the original README and have reciprocal language links. +- SVG point positions agree with the reported scores and labeled 70%–80% scale. +- Inspect rendered desktop and narrow views in light and dark themes; check + labels, contrast, spacing, links, and an accessible table fallback. +- Check Markdown formatting, SVG syntax, and the full diff in two clean passes. + Attempt the repository build and typecheck; report any environment limitation. + +## Scope + +This change affects repository documentation and its existing chart asset only. +It does not change CLI behavior and needs no CLI E2E plan or unit test. diff --git a/docs/design/2026-10-14-readme-evaluation.zh-CN.md b/docs/design/2026-10-14-readme-evaluation.zh-CN.md new file mode 100644 index 00000000000..47d58e1fcda --- /dev/null +++ b/docs/design/2026-10-14-readme-evaluation.zh-CN.md @@ -0,0 +1,34 @@ +# README 评测展示 + +[English](2026-10-14-readme-evaluation.md) | [简体中文](2026-10-14-readme-evaluation.zh-CN.md) + +## 问题 + +README 在生态章节之前完整展开评测配置、成绩表和大幅图表。原图使用 76%–78% +的纵轴，把版本间 1.40 个百分点的差异在视觉上放大。 + +## 设计 + +- 保留既有评测标题和锚点。正文只展示一句评测背景、紧凑图表和完整结果链接。 +- 将现有全部配置和成绩移到 `docs/` 下互相链接的英文、中文评测文档。 +- 使用紧凑折线图展示版本序列。将纵轴扩大到 70%–80%，并在副标题和刻度中明确 + 标示。保留版本顺序和精确得分标签，以直线连接观测点，不做平滑处理。使用一种 + Qwen 紫色、浅色网格和清晰文字，支持浅色与深色背景。 +- 保留现有 SVG 路径，确保外部图片引用仍然有效。补充无障碍标题、描述和 + Markdown 图片替代文本。 +- 只调整展示，不更新评测结果，不宣称统计显著性、能力提升或未测试版本的成绩。 + +## 验证与验收标准 + +- README 评测部分没有展开的表格或三级标题。 +- 两份评测文档保留原 README 的所有成绩和配置值，并包含双向语言链接。 +- SVG 数据点位置与已报告得分和标明的 70%–80% 刻度一致。 +- 检查桌面与窄屏的浅色、深色实际渲染，确认标签、对比度、间距、链接和可访问的 + 表格替代形式。 +- 检查 Markdown 格式、SVG 语法和完整 diff，完成两轮无问题审阅。尝试仓库构建 + 和类型检查，如受环境限制则如实记录。 + +## 范围 + +本次变更只涉及仓库文档及现有图表资源，不改变 CLI 行为，无需 CLI E2E 计划或 +单元测试。 diff --git a/docs/evaluation.md b/docs/evaluation.md new file mode 100644 index 00000000000..d875a7068d4 --- /dev/null +++ b/docs/evaluation.md @@ -0,0 +1,39 @@ +# Qwen Code Evaluation + +[English](evaluation.md) | [简体中文](evaluation.zh-CN.md) + +This evaluation compares seven Qwen Code versions on SWE-bench Verified using +the same Qwen 3.7 Max model and configuration. Average scores range from +**76.40% to 77.80%**, a spread of **1.40 percentage points**. + +## Scores by Qwen Code Version + +![Average SWE-bench Verified scores by Qwen Code version, ranging from 76.40% to 77.80%.](../.github/assets/qwen-code-avg-score-trend.svg) + +| Version | Avg Score | Pass Rate | pass@3 | pass^3 | +| ------- | --------: | --------: | -----: | -----: | +| 0.0.14 | 77.80% | 84.6% | 84.6% | 69.2% | +| 0.1.0 | 76.73% | 84.6% | 84.6% | 68.0% | +| 0.5.0 | 76.60% | 83.0% | 83.0% | 68.4% | +| 0.10.0 | 76.47% | 84.4% | 84.4% | 67.4% | +| 0.15.0 | 77.67% | 84.6% | 84.6% | 68.6% | +| 0.19.0 | 76.40% | 85.0% | 85.0% | 65.8% | +| 0.22.0 | 77.33% | 84.4% | 84.4% | 68.0% | + +The line chart connects versions in release order and uses an explicitly labeled +70%–80% scale. The table retains all reported metrics; +Pass Rate and pass@3 have identical values in these results. + +## Evaluation Configuration + +| Configuration | Value | +| ----------------------- | --------------------------------------------------------------------------------------------------- | +| Dataset | `princeton-nlp/SWE-bench_Verified`, 500 cases | +| Runs | 3 trials per version, 1,500 jobs per version; 7 Qwen Code versions | +| Model | `Qwen 3.7 Max` | +| Sampling | `temperature=1`, `top_p=0.95` | +| Thinking / Reasoning | `interleaved_thinking=true`, `thinking_type=adaptive`, `reasoning_effort=high` | +| Context / Output Limits | `context_1m=true`, `model_context_length=1,000,000`, `max_tokens=64,000`, `model_max_tokens=65,535` | +| Harness Limits | `max_iterations=500`, `runtime_timeout_sec=7,200`, `agent_timeout_multiplier=3.0` | + +[Back to README](../README.md#qwen-code-evaluation) diff --git a/docs/evaluation.zh-CN.md b/docs/evaluation.zh-CN.md new file mode 100644 index 00000000000..ed45aac27a3 --- /dev/null +++ b/docs/evaluation.zh-CN.md @@ -0,0 +1,37 @@ +# Qwen Code 评测 + +[English](evaluation.md) | [简体中文](evaluation.zh-CN.md) + +本次评测使用相同的 Qwen 3.7 Max 模型和配置，在 SWE-bench Verified 上比较七个 +Qwen Code 版本。平均得分为 **76.40%–77.80%**，最大差值为 **1.40 个百分点**。 + +## 各版本得分 + +![各 Qwen Code 版本的 SWE-bench Verified 平均得分，范围为 76.40%–77.80%。](../.github/assets/qwen-code-avg-score-trend.svg) + +| 版本 | Avg Score | Pass Rate | pass@3 | pass^3 | +| ------ | --------: | --------: | -----: | -----: | +| 0.0.14 | 77.80% | 84.6% | 84.6% | 69.2% | +| 0.1.0 | 76.73% | 84.6% | 84.6% | 68.0% | +| 0.5.0 | 76.60% | 83.0% | 83.0% | 68.4% | +| 0.10.0 | 76.47% | 84.4% | 84.4% | 67.4% | +| 0.15.0 | 77.67% | 84.6% | 84.6% | 68.6% | +| 0.19.0 | 76.40% | 85.0% | 85.0% | 65.8% | +| 0.22.0 | 77.33% | 84.4% | 84.4% | 68.0% | + +折线图按发布顺序连接各版本，明确标示 70%–80% 的纵轴范围。表格保留全部已报告指标；本组结果中 Pass Rate 与 +pass@3 的数值相同。 + +## 评测配置 + +| 配置项 | 值 | +| ------------- | --------------------------------------------------------------------------------------------------- | +| 数据集 | `princeton-nlp/SWE-bench_Verified`，500 个任务 | +| 运行次数 | 每个版本 3 次，共 1,500 个作业；覆盖 7 个 Qwen Code 版本 | +| 模型 | `Qwen 3.7 Max` | +| 采样 | `temperature=1`, `top_p=0.95` | +| 思考 / 推理 | `interleaved_thinking=true`, `thinking_type=adaptive`, `reasoning_effort=high` | +| 上下文 / 输出 | `context_1m=true`, `model_context_length=1,000,000`, `max_tokens=64,000`, `model_max_tokens=65,535` | +| Harness 限制 | `max_iterations=500`, `runtime_timeout_sec=7,200`, `agent_timeout_multiplier=3.0` | + +[返回 README](../README.md#qwen-code-evaluation)