3 Commits

Author SHA1 Message Date
Mason Daugherty 758487a454 chore(oss): add Overall score column and suppress zero scores in eval matrix (#4066)
Two improvements to the Deep Agents eval matrix generator. Zero scores
(a known CI artifact for models that fail entirely, like
`openrouter:deepseek/deepseek-v4-pro`) now render as `—` instead of
`0%`. The top-level `correctness` field from `evals_summary.json` — the
aggregate score across all eval tasks — is extracted as a new
**Overall** column, inserted immediately after the Model column so it's
the first thing readers see.

## Changes

- `_fmt_pct` now treats `v <= 0` as missing, rendering `—` instead of
`0%` for zero scores
- `_merge_rows` captures `rep["correctness"]` (a 0..1 float) per model
under an `OVERALL_KEY` sentinel, using the same first-seen-wins logic as
category scores
- `build_fragment` prepends `OVERALL_KEY`/`"Overall"` to the
`cat_keys`/`display_headers` passed to `_table_markdown`, making it
participate in column-max bolding automatically
- Regenerated snippet reflects both changes:
`openrouter:deepseek/deepseek-v4-pro` row removed (all-zero scores now
suppressed), Overall column added to all remaining rows
2026-05-19 00:10:26 -05:00
Mason Daugherty 540540ad4e chore(oss): add INCLUDED_MODELS allowlist to Deep Agents eval matrix script (#4065) 2026-05-18 23:10:41 -05:00
Naomi Pentrel 1e25f1ee27 get model leaderboard info into docs (#3715) 2026-04-24 21:32:54 +02:00