docs(langsmith): document adding threads to a dataset

This commit is contained in:
Winston-Huynh-LangChain
2026-08-20 17:01:30 -07:00
parent 4a7bd195ba
commit cbddcc9c2e
6 changed files with 28 additions and 7 deletions
+4 -4
View File
@@ -31,7 +31,7 @@ Run items and thread items support different capabilities:
| Rubric feedback | Yes | Yes |
| Reviewer notes | Yes | No |
| Assertions | Yes | No |
| Add to Dataset | Yes | No |
| Add to Dataset | Yes | Yes |
| Default dataset | Yes | No |
| Automation rules | Yes | Yes |
@@ -44,7 +44,7 @@ Run items and thread items support different capabilities:
1. Fill in the **Name** and **Description** of the queue.
1. Optionally select an **Application**.
1. Optionally **Select a default dataset** to streamline exporting reviewed runs into a dataset in your LangSmith [workspace](/langsmith/administration-overview#workspaces). Default datasets apply when you use **Add to Dataset** on run items; thread items do not support adding to a dataset.
1. Optionally **Select a default dataset** to streamline exporting reviewed runs into a dataset in your LangSmith [workspace](/langsmith/administration-overview#workspaces). The default dataset backs the **Add to Dataset & Next** action on run items: when one is set, that action adds the run to it directly instead of opening the dataset picker. Thread items do not offer **Add to Dataset & Next**, so a default dataset does not apply to them.
#### Annotation rubric
@@ -187,12 +187,12 @@ Manually adding runs or threads to an annotation queue does not change trace ret
1. Review the current item:
- **Run items**: Inspect inputs and outputs in the center pane. Add **Reviewer Notes**, score [**Feedback**](/langsmith/observability-concepts#feedback) criteria, or mark the item as reviewed. To build a dataset, edit the run's input and output to create a corrected reference example and click **Add to Dataset**. Instead of crafting a corrected reference output by hand, you can [write **Assertions**](/langsmith/assertions) directly in the review side panel and save them as the example's expected output.
- **Thread items**: The center pane shows the conversation transcript for that thread. Read the transcript and score the same rubric **Feedback** keys. Use **View item** to open the thread in the conversation peek.
- **Thread items**: The center pane shows the conversation transcript for that thread. Read the transcript and score the same rubric **Feedback** keys. Use **View item** to open the thread in the conversation peek. To build a dataset, click **Add to Dataset** and choose a dataset, which adds the whole conversation as one example. For what a thread example contains, refer to [Create and manage datasets in the UI](/langsmith/manage-datasets-in-application#manually-from-a-tracing-project).
Click **Delete** to remove the item from the queue for all users, regardless of any current reservations or queue settings.
<Note>
Thread items support rubric feedback only. See the [capability table](#single-run-annotation-queues) for what differs between run and thread items.
Thread items support rubric feedback and **Add to Dataset**. They do not support reviewer notes or assertions. See the [capability table](#single-run-annotation-queues) for what differs between run and thread items.
</Note>
<img
+2
View File
@@ -159,6 +159,8 @@ Learn more about [runs and traces in the Observability concepts](/langsmith/obse
_Threads_ are collections of related runs representing multi-turn conversations. Online evaluators can run at the thread level to evaluate entire conversations rather than individual turns. This enables assessment of conversation-level properties like coherence across turns, topic maintenance, and user satisfaction throughout an interaction.
Threads are also an offline evaluation target. Adding a thread to a dataset stores the whole conversation as a single example, which you can then evaluate in an experiment. For more information, refer to [Create and manage datasets in the UI](/langsmith/manage-datasets-in-application#manually-from-a-tracing-project).
## Evaluators
_Evaluators_ are workspace-level resources that score application performance. They provide the measurement layer for both offline and online evaluation, adapting their inputs based on what data is available. Because evaluators are scoped to the workspace, you can attach a single evaluator to multiple tracing projects and datasets without recreating it each time.
+1
View File
@@ -21,6 +21,7 @@ LangSmith stores examples in datasets as follows:
| **outputs** | object | A map or set of outputs generated by the run. |
| **dataset\_id** | UUID | The dataset the example belongs to |
| **source\_run\_id** | UUID | If this example was created from a LangSmith [`Run`](/langsmith/run-data-format), the ID of said run |
| **source\_thread\_id** | string | If this example was created from a [thread](/langsmith/observability-concepts#threads), the ID of that thread |
| **metadata** | object | A map of additional, user or SDK defined information that can be stored on an example. |
To learn more about how examples are used in evaluation, read our how-to guide on [evaluating LLM applications](/langsmith/evaluate-llm-application).
@@ -28,7 +28,7 @@ A common pattern for constructing datasets is to convert notable traces from you
A technique to build datasets is to filter the most interesting traces, such as traces that were tagged with poor user feedback, and add them to a dataset. For tips on how to filter traces, refer to the [Filter traces](/langsmith/filter-traces-in-application) guide.
</Check>
There are two ways to add data manually from a tracing project to datasets. Navigate to **Tracing Projects** and select a project.
There are three ways to add data manually from a tracing project to datasets. Navigate to **Tracing Projects** and select a project.
1. Multi-select runs from the runs table. On the **Runs** tab, multi-select runs. At the bottom of the page, click <Icon icon="database" /> **Add to Dataset**.
1. On the **Runs** tab, select a run from the table. On the individual run details page, select **Add to** -> **Dataset** in the top right corner.
@@ -37,6 +37,23 @@ There are two ways to add data manually from a tracing project to datasets. Navi
You can then optionally edit the run before adding it to the dataset.
1. Multi-select threads from the threads table. On the **Threads** tab, multi-select [threads](/langsmith/observability-concepts#threads). At the bottom of the page, click <Icon icon="database" /> **Add to Dataset**, then choose a dataset. You can add at most **100** threads in a single action.
<Note>
If the **Threads** tab is empty, the traces in the project are not instrumented with `thread_id` or `session_id` metadata. For more information, refer to [Configure threads](/langsmith/threads).
</Note>
Adding threads differs from adding runs in the following ways:
- **One example per thread**: Each thread becomes a single example holding the full conversation, rather than one example per turn.
- **No reference outputs**: A thread example records the conversation only, so it is reference-free.
- **No preview**: The confirmation modal described above does not appear for threads. Dataset [transformations](/langsmith/dataset-transformations) still apply.
- **No deduplication**: Adding the same thread twice creates two examples.
<Note>
The dataset picker lists every dataset in the workspace, including datasets whose [schema](#create-a-dataset-schema) a conversation does not satisfy. Incompatible datasets are not filtered out, so the import reports an error instead. Add threads to a dataset that has no schema, or one whose schema accepts a conversation.
</Note>
### Automatically from a tracing project
You can use [run rules](/langsmith/rules) to add traces automatically to a dataset based on certain conditions. For example, you could add all traces that are [tagged](/langsmith/observability-concepts#tags) with a specific use case or have a [low feedback score](/langsmith/observability-concepts#feedback).
@@ -52,7 +69,7 @@ You can optionally configure annotation queues with a default dataset, though yo
Any modifications you make to the run in your annotation queue will carry over to the dataset, and all metadata associated with the run will also be copied.
<Note>
**Add to Dataset** is available for **run** queue items only. [Thread](/langsmith/observability-concepts#threads) items in an annotation queue support rubric feedback, but not dataset export.
**Add to Dataset** works on both run items and [thread](/langsmith/observability-concepts#threads) items. A thread item adds the whole conversation as one example, with the same behavior described in [Manually from a tracing project](#manually-from-a-tracing-project). Thread items do not offer **Add to Dataset & Next**, so add the item to a dataset and advance the queue separately.
</Note>
<Tip>
+1
View File
@@ -297,4 +297,5 @@ On the **Threads** tab of a project, you can save commonly used filters: [Set a
- [Add metadata and tags to traces](/langsmith/add-metadata-tags): how to pass `thread_id` and other metadata keys.
- [Filter traces](/langsmith/filter-traces-in-application): filter by thread metadata in the tracing UI.
- [Set up multi-turn online evaluators](/langsmith/online-evaluations-multi-turn): evaluate threads rather than individual runs.
- [Create and manage datasets in the UI](/langsmith/manage-datasets-in-application#manually-from-a-tracing-project): add threads to a dataset for offline evaluation.
- [Log user feedback using the SDK](/langsmith/attach-user-feedback): attach feedback to runs within a thread.
+1 -1
View File
@@ -18,7 +18,7 @@ Use Perceived Error feedback to:
- Find failures that did not produce a system error or explicit user rating.
- Filter for conversations that need investigation and review the evaluator's explanation with the original interaction.
- Compare flagged threads to identify recurring failure modes.
- Add useful threads to evaluation datasets or send ambiguous cases for human review.
- Add useful threads to [evaluation datasets](/langsmith/manage-datasets-in-application#manually-from-a-tracing-project) or send ambiguous cases for human review.
- Test agent changes against examples drawn from traced failures.
## Understand how LangChain Tuned Evaluators work