langchain-ai/docs
1
0
Fork 0
mirror of https://github.com/langchain-ai/docs.git synced 2026-07-21 11:55:23 -04:00
Code Issues 490 Packages Projects Releases Wiki Activity

[PR #4066] [MERGED] chore(oss): add Overall score column and suppress zero scores in eval matrix #4069

New Issue
Closed
opened 2026-06-05 19:12:39 -04:00 by yindo · 0 comments
yindo commented 2026-06-05 19:12:39 -04:00
Owner
Copy Link
Copy Source

📋 Pull Request Information

Original PR: https://github.com/langchain-ai/docs/pull/4066
Author: @mdrxy
Created: 5/19/2026
Status: ✅ Merged
Merged: 5/19/2026
Merged by: @mdrxy

Base: main ← Head: mdrxy/scores-no-0

📝 Commits (2)

  • 3b2dbe8 chore: render 0 score as em-dash
  • 756a92b chore(oss): add Overall score column and suppress zero scores in eval matrix

📊 Changes

2 files changed (+26 additions, -19 deletions)

View changed files

📝 scripts/refresh_deepagents_category_matrix.py (+11 -3)
📝 src/snippets/deepagents-eval-category-matrix.mdx (+15 -16)

📄 Description

Two improvements to the Deep Agents eval matrix generator. Zero scores (a known CI artifact for models that fail entirely, like openrouter:deepseek/deepseek-v4-pro) now render as — instead of 0%. The top-level correctness field from evals_summary.json — the aggregate score across all eval tasks — is extracted as a new Overall column, inserted immediately after the Model column so it's the first thing readers see.

Changes

  • _fmt_pct now treats v <= 0 as missing, rendering — instead of 0% for zero scores
  • _merge_rows captures rep["correctness"] (a 0..1 float) per model under an OVERALL_KEY sentinel, using the same first-seen-wins logic as category scores
  • build_fragment prepends OVERALL_KEY/"Overall" to the cat_keys/display_headers passed to _table_markdown, making it participate in column-max bolding automatically
  • Regenerated snippet reflects both changes: openrouter:deepseek/deepseek-v4-pro row removed (all-zero scores now suppressed), Overall column added to all remaining rows

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/langchain-ai/docs/pull/4066 **Author:** [@mdrxy](https://github.com/mdrxy) **Created:** 5/19/2026 **Status:** ✅ Merged **Merged:** 5/19/2026 **Merged by:** [@mdrxy](https://github.com/mdrxy) **Base:** `main` ← **Head:** `mdrxy/scores-no-0` --- ### 📝 Commits (2) - [`3b2dbe8`](https://github.com/langchain-ai/docs/commit/3b2dbe8dd2aec792f9f51d4200e5a34f5217bfcb) chore: render 0 score as em-dash - [`756a92b`](https://github.com/langchain-ai/docs/commit/756a92bf06dedbaf134ccc675afc3ef2bf0b5545) chore(oss): add Overall score column and suppress zero scores in eval matrix ### 📊 Changes **2 files changed** (+26 additions, -19 deletions) <details> <summary>View changed files</summary> 📝 `scripts/refresh_deepagents_category_matrix.py` (+11 -3) 📝 `src/snippets/deepagents-eval-category-matrix.mdx` (+15 -16) </details> ### 📄 Description Two improvements to the Deep Agents eval matrix generator. Zero scores (a known CI artifact for models that fail entirely, like `openrouter:deepseek/deepseek-v4-pro`) now render as `—` instead of `0%`. The top-level `correctness` field from `evals_summary.json` — the aggregate score across all eval tasks — is extracted as a new **Overall** column, inserted immediately after the Model column so it's the first thing readers see. ## Changes - `_fmt_pct` now treats `v <= 0` as missing, rendering `—` instead of `0%` for zero scores - `_merge_rows` captures `rep["correctness"]` (a 0..1 float) per model under an `OVERALL_KEY` sentinel, using the same first-seen-wins logic as category scores - `build_fragment` prepends `OVERALL_KEY`/`"Overall"` to the `cat_keys`/`display_headers` passed to `_table_markdown`, making it participate in column-max bolding automatically - Regenerated snippet reflects both changes: `openrouter:deepseek/deepseek-v4-pro` row removed (all-zero scores now suppressed), Overall column added to all remaining rows --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
yindo added the pull-request label 2026-06-05 19:12:39 -04:00
yindo closed this issue 2026-06-05 19:12:39 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
deepagents
docs-infra
external
external
help wanted
integration
internal
internal
javascript
js/ts
langchain
langgraph
langsmith
needs-title
pull-request
Mirrored from GitHub Pull Request
 python
reference
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langchain-ai/docs#4069
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?