mirror of
https://github.com/QwenLM/qwen-code.git
synced 2026-11-07 06:19:43 -10:00
40 lines
2.4 KiB
Markdown
40 lines
2.4 KiB
Markdown
# Qwen Code Evaluation
|
||
|
||
[English](evaluation.md) | [简体中文](evaluation.zh-CN.md)
|
||
|
||
This evaluation compares seven Qwen Code versions on SWE-bench Verified using
|
||
the same Qwen 3.7 Max model and configuration. Average scores range from
|
||
**76.40% to 77.80%**, a spread of **1.40 percentage points**.
|
||
|
||
## Scores by Qwen Code Version
|
||
|
||
|
||
|
||
| Version | Avg Score | Pass Rate | pass@3 | pass^3 |
|
||
| ------- | --------: | --------: | -----: | -----: |
|
||
| 0.0.14 | 77.80% | 84.6% | 84.6% | 69.2% |
|
||
| 0.1.0 | 76.73% | 84.6% | 84.6% | 68.0% |
|
||
| 0.5.0 | 76.60% | 83.0% | 83.0% | 68.4% |
|
||
| 0.10.0 | 76.47% | 84.4% | 84.4% | 67.4% |
|
||
| 0.15.0 | 77.67% | 84.6% | 84.6% | 68.6% |
|
||
| 0.19.0 | 76.40% | 85.0% | 85.0% | 65.8% |
|
||
| 0.22.0 | 77.33% | 84.4% | 84.4% | 68.0% |
|
||
|
||
The line chart connects versions in release order and uses an explicitly labeled
|
||
70%–80% scale. The table retains all reported metrics;
|
||
Pass Rate and pass@3 have identical values in these results.
|
||
|
||
## Evaluation Configuration
|
||
|
||
| Configuration | Value |
|
||
| ----------------------- | --------------------------------------------------------------------------------------------------- |
|
||
| Dataset | `princeton-nlp/SWE-bench_Verified`, 500 cases |
|
||
| Runs | 3 trials per version, 1,500 jobs per version; 7 Qwen Code versions |
|
||
| Model | `Qwen 3.7 Max` |
|
||
| Sampling | `temperature=1`, `top_p=0.95` |
|
||
| Thinking / Reasoning | `interleaved_thinking=true`, `thinking_type=adaptive`, `reasoning_effort=high` |
|
||
| Context / Output Limits | `context_1m=true`, `model_context_length=1,000,000`, `max_tokens=64,000`, `model_max_tokens=65,535` |
|
||
| Harness Limits | `max_iterations=500`, `runtime_timeout_sec=7,200`, `agent_timeout_multiplier=3.0` |
|
||
|
||
[Back to README](../README.md#qwen-code-evaluation)
|