langchain-ai/docs
1
0
Fork 0
mirror of https://github.com/langchain-ai/docs.git synced 2026-07-23 04:45:23 -04:00
Code Issues 490 Packages Projects Releases Wiki Activity

[PR #3923] [MERGED] docs(langsmith): add Author assertions section to annotation queues #3934

New Issue
Closed
opened 2026-06-05 18:53:38 -04:00 by yindo · 0 comments
yindo commented 2026-06-05 18:53:38 -04:00
Owner
Copy Link
Copy Source

📋 Pull Request Information

Original PR: https://github.com/langchain-ai/docs/pull/3923
Author: @ayoung19
Created: 5/8/2026
Status: ✅ Merged
Merged: 5/12/2026
Merged by: @fjmorris

Base: main ← Head: docs/aq-assertions

📝 Commits (7)

  • d9d6a6b docs(langsmith): add Author assertions section to annotation queues page
  • 7b39a45 Moved Assertions section to separate page. Modified other pages to link to assertions.mdx.
  • 46cd038 Merge remote-tracking branch 'origin/main' into docs/aq-assertions
  • 8d36ef8 Merge remote-tracking branch 'origin/main' into docs/aq-assertions
  • afa98fc Incorporated feedback from Ben and Lauren.
  • ac8650b Merge remote-tracking branch 'origin/main' into docs/aq-assertions
  • a90e797 Merge remote-tracking branch 'origin/main' into docs/aq-assertions

📊 Changes

12 files changed (+111 additions, -5 deletions)

View changed files

📝 src/docs.json (+1 -0)
📝 src/langsmith/annotation-queues-sdk.mdx (+4 -0)
📝 src/langsmith/annotation-queues.mdx (+4 -2)
➕ src/langsmith/assertions.mdx (+87 -0)
📝 src/langsmith/code-evaluator-sdk.mdx (+1 -1)
📝 src/langsmith/code-evaluator-ui.mdx (+1 -1)
📝 src/langsmith/evaluation-concepts.mdx (+1 -1)
📝 src/langsmith/example-data-format.mdx (+4 -0)
➕ src/langsmith/images/assertions-example-dark.png (+0 -0)
➕ src/langsmith/images/assertions-example-light.png (+0 -0)
📝 src/langsmith/llm-as-judge.mdx (+4 -0)
📝 src/langsmith/set-up-feedback-criteria.mdx (+4 -0)

📄 Description

Draft. Rough first pass at documenting the new free-form assertions feature in single-run annotation queues. PM can enrich.

What this adds

A new ## Author assertions while reviewing section in src/langsmith/annotation-queues.mdx, slotted between "Review runs in an annotation queue" and "Video guide". It covers:

  • What assertions are (free-form {key, comment} claims) and when to use them
  • How to add them in the side panel (with the new "+ Add" button)
  • The readonly outputs preview that activates once an assertion is staged
  • The saved example shape (outputs.assertions: [...])
  • How they're consumed by offline evaluators (kept unopinionated — points to llm-as-judge.mdx and evaluation-concepts)

Open items for PM / SME review

  • Screenshots — two placeholder image refs are included with {/* TODO(screenshot): ... */} comments describing what each shot should show:
    • annotation-queue-assertions-side-panel.png — side panel with Feedback + Assertions sections, two filled rows, "+ Add" button visible
    • annotation-queue-assertions-output-preview.png — Outputs panel rendered as readonly assertion blocks
    • An optional third (dataset example JSON) is flagged inline; skippable if the JSON code block is enough
  • Cross-link from llm-as-judge.mdx — not done in this PR. Worth adding a small mention there ("examples saved from annotation queues may use the assertion format — outputs.assertions: [...]") so the LLM-judge audience finds the feature?
  • Issues-agent / proposed-example flow intentionally not mentioned in this draft (per scoping discussion)
  • Tone / phrasing — the "constraints on the right answer" framing is mine; PM may want to phrase the value prop differently

Release Note

Documentation: new section covering free-form assertion authoring in annotation queues.

Test Plan

  • Mintlify build succeeds locally / in preview
  • Real screenshots replace the placeholder image refs before merge
  • PM reviews tone/scoping

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/langchain-ai/docs/pull/3923 **Author:** [@ayoung19](https://github.com/ayoung19) **Created:** 5/8/2026 **Status:** ✅ Merged **Merged:** 5/12/2026 **Merged by:** [@fjmorris](https://github.com/fjmorris) **Base:** `main` ← **Head:** `docs/aq-assertions` --- ### 📝 Commits (7) - [`d9d6a6b`](https://github.com/langchain-ai/docs/commit/d9d6a6bda8af692942ef0d8dd51eba8e5e2b7861) docs(langsmith): add Author assertions section to annotation queues page - [`7b39a45`](https://github.com/langchain-ai/docs/commit/7b39a4594f458fee22c6dd640c35e35058d2289f) Moved Assertions section to separate page. Modified other pages to link to assertions.mdx. - [`46cd038`](https://github.com/langchain-ai/docs/commit/46cd038d266d468bf92851bbd1a6a7416d6a2a6a) Merge remote-tracking branch 'origin/main' into docs/aq-assertions - [`8d36ef8`](https://github.com/langchain-ai/docs/commit/8d36ef8e6627c90b9a9b95d3048222e35c498a2d) Merge remote-tracking branch 'origin/main' into docs/aq-assertions - [`afa98fc`](https://github.com/langchain-ai/docs/commit/afa98fce38e481c380a300f81196df3f82db0376) Incorporated feedback from Ben and Lauren. - [`ac8650b`](https://github.com/langchain-ai/docs/commit/ac8650bbf295df2c3768e98455090db3a7eed8e5) Merge remote-tracking branch 'origin/main' into docs/aq-assertions - [`a90e797`](https://github.com/langchain-ai/docs/commit/a90e797e5baaec21574cb05c3571ac7fb4db5de1) Merge remote-tracking branch 'origin/main' into docs/aq-assertions ### 📊 Changes **12 files changed** (+111 additions, -5 deletions) <details> <summary>View changed files</summary> 📝 `src/docs.json` (+1 -0) 📝 `src/langsmith/annotation-queues-sdk.mdx` (+4 -0) 📝 `src/langsmith/annotation-queues.mdx` (+4 -2) ➕ `src/langsmith/assertions.mdx` (+87 -0) 📝 `src/langsmith/code-evaluator-sdk.mdx` (+1 -1) 📝 `src/langsmith/code-evaluator-ui.mdx` (+1 -1) 📝 `src/langsmith/evaluation-concepts.mdx` (+1 -1) 📝 `src/langsmith/example-data-format.mdx` (+4 -0) ➕ `src/langsmith/images/assertions-example-dark.png` (+0 -0) ➕ `src/langsmith/images/assertions-example-light.png` (+0 -0) 📝 `src/langsmith/llm-as-judge.mdx` (+4 -0) 📝 `src/langsmith/set-up-feedback-criteria.mdx` (+4 -0) </details> ### 📄 Description **Draft.** Rough first pass at documenting the new free-form assertions feature in single-run annotation queues. PM can enrich. ## What this adds A new `## Author assertions while reviewing` section in `src/langsmith/annotation-queues.mdx`, slotted between "Review runs in an annotation queue" and "Video guide". It covers: - What assertions are (free-form `{key, comment}` claims) and when to use them - How to add them in the side panel (with the new "+ Add" button) - The readonly outputs preview that activates once an assertion is staged - The saved example shape (`outputs.assertions: [...]`) - How they're consumed by offline evaluators (kept unopinionated — points to llm-as-judge.mdx and evaluation-concepts) ## Open items for PM / SME review - **Screenshots** — two placeholder image refs are included with `{/* TODO(screenshot): ... */}` comments describing what each shot should show: - `annotation-queue-assertions-side-panel.png` — side panel with Feedback + Assertions sections, two filled rows, "+ Add" button visible - `annotation-queue-assertions-output-preview.png` — Outputs panel rendered as readonly assertion blocks - An optional third (dataset example JSON) is flagged inline; skippable if the JSON code block is enough - **Cross-link from `llm-as-judge.mdx`** — not done in this PR. Worth adding a small mention there ("examples saved from annotation queues may use the assertion format — `outputs.assertions: [...]`") so the LLM-judge audience finds the feature? - **Issues-agent / proposed-example flow** intentionally not mentioned in this draft (per scoping discussion) - **Tone / phrasing** — the "constraints on the right answer" framing is mine; PM may want to phrase the value prop differently ## Release Note Documentation: new section covering free-form assertion authoring in annotation queues. ## Test Plan - [ ] Mintlify build succeeds locally / in preview - [ ] Real screenshots replace the placeholder image refs before merge - [ ] PM reviews tone/scoping --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
yindo added the pull-request label 2026-06-05 18:53:38 -04:00
yindo closed this issue 2026-06-05 18:53:38 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
deepagents
docs-infra
external
external
help wanted
integration
internal
internal
javascript
js/ts
langchain
langgraph
langsmith
needs-title
pull-request
Mirrored from GitHub Pull Request
 python
reference
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langchain-ai/docs#3934
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?