langchain-ai/docs
1
0
Fork 0
mirror of https://github.com/langchain-ai/docs.git synced 2026-07-22 12:25:55 -04:00
Code Issues 490 Packages Projects Releases Wiki Activity

[GH-ISSUE #1499] json_schema evaluator docs are misleading #19

New Issue
Open
opened 2026-02-17 17:19:00 -05:00 by yindo · 0 comments
yindo commented 2026-02-17 17:19:00 -05:00
Owner
Copy Link
Copy Source

Originally created by @mshavliuk on GitHub (Feb 17, 2025).
Original GitHub issue: https://github.com/langchain-ai/docs/issues/1499

Originally assigned to: @jacoblee93 on GitHub.

Description

Following the instructions in docs/reference/sdk_reference/langchain_evaluators.mdx I tried to create 'json_schema' evaluator:

from langsmith.evaluation import LangChainStringEvaluator

LangChainStringEvaluator("json_schema")

And got the following error:

Traceback (most recent call last):
  File "/Users/user/PycharmProjects/ml-sandbox/src/bug.py", line 3, in <module>
    LangChainStringEvaluator("json_schema")
  File "/Users/user/PycharmProjects/ml-sandbox/.venv/lib/python3.12/site-packages/langsmith/evaluation/integrations/_langchain.py", line 165, in __init__
    self.evaluator = load_evaluator(evaluator, **(config or {}))  # type: ignore[assignment, arg-type]
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/PycharmProjects/ml-sandbox/.venv/lib/python3.12/site-packages/langchain/evaluation/loading.py", line 127, in load_evaluator
    raise ValueError(
ValueError: Unknown evaluator type: json_schema
Valid types are: [<EvaluatorType.QA: 'qa'>, <EvaluatorType.COT_QA: 'cot_qa'>, <EvaluatorType.CONTEXT_QA: 'context_qa'>, <EvaluatorType.PAIRWISE_STRING: 'pairwise_string'>, <EvaluatorType.SCORE_STRING: 'score_string'>, <EvaluatorType.LABELED_PAIRWISE_STRING: 'labeled_pairwise_string'>, <EvaluatorType.LABELED_SCORE_STRING: 'labeled_score_string'>, <EvaluatorType.AGENT_TRAJECTORY: 'trajectory'>, <EvaluatorType.CRITERIA: 'criteria'>, <EvaluatorType.LABELED_CRITERIA: 'labeled_criteria'>, <EvaluatorType.STRING_DISTANCE: 'string_distance'>, <EvaluatorType.PAIRWISE_STRING_DISTANCE: 'pairwise_string_distance'>, <EvaluatorType.EMBEDDING_DISTANCE: 'embedding_distance'>, <EvaluatorType.PAIRWISE_EMBEDDING_DISTANCE: 'pairwise_embedding_distance'>, <EvaluatorType.JSON_VALIDITY: 'json_validity'>, <EvaluatorType.JSON_EQUALITY: 'json_equality'>, <EvaluatorType.JSON_EDIT_DISTANCE: 'json_edit_distance'>, <EvaluatorType.JSON_SCHEMA_VALIDATION: 'json_schema_validation'>, <EvaluatorType.REGEX_MATCH: 'regex_match'>, <EvaluatorType.EXACT_MATCH: 'exact_match'>]

Partial solution
Looking at the error message I found that it includes 'json_schema_validation', which looks as a correct candidate, so I used the following code:


class SummaryResponse(BaseModel):
    summary: str
    reason: str


json_schema = LangChainStringEvaluator(
    "json_schema_validation",
    config={
        "schema": SummaryResponse.model_json_schema()
    }
)
experiment_results = langsmith_client.evaluate(
        simple,
        data="dataset_name",
        evaluators=[json_schema],
    )

However, this led to another error:

Error running evaluator <DynamicRunEvaluator evaluate> on run 8ba9cf6c-90bf-4c51-ad37-e2ab01290413: ValueError('JsonSchemaEvaluator requires a reference string.')
Traceback (most recent call last):
  File "/Users/user/PycharmProjects/ml-sandbox/.venv/lib/python3.12/site-packages/langsmith/evaluation/_runner.py", line 1634, in _run_evaluators
    evaluator_response = evaluator.evaluate_run(
                         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/PycharmProjects/ml-sandbox/.venv/lib/python3.12/site-packages/langsmith/evaluation/evaluator.py", line 331, in evaluate_run
    result = self.func(
             ^^^^^^^^^^
  File "/Users/user/PycharmProjects/ml-sandbox/.venv/lib/python3.12/site-packages/langsmith/run_helpers.py", line 629, in wrapper
    raise e
  File "/Users/user/PycharmProjects/ml-sandbox/.venv/lib/python3.12/site-packages/langsmith/run_helpers.py", line 626, in wrapper
    function_result = run_container["context"].run(func, *args, **kwargs)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/PycharmProjects/ml-sandbox/.venv/lib/python3.12/site-packages/langsmith/evaluation/integrations/_langchain.py", line 260, in evaluate
    results = self.evaluator.evaluate_strings(**eval_inputs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/PycharmProjects/ml-sandbox/.venv/lib/python3.12/site-packages/langchain/evaluation/schema.py", line 219, in evaluate_strings
    self._check_evaluation_args(reference=reference, input=input)
  File "/Users/user/PycharmProjects/ml-sandbox/.venv/lib/python3.12/site-packages/langchain/evaluation/schema.py", line 127, in _check_evaluation_args
    raise ValueError(f"{self.__class__.__name__} requires a reference string.")
ValueError: JsonSchemaEvaluator requires a reference string.

Looking deeper at the implementation of LangChainStringEvaluator.evaluate I can see that the second argument given to the StringEvaluator is an example field, taken from the dataset, which is None in my case. Instead, I expect it to use the given JSON schema, but I didn't manage to figure out how to make it work.

Context
LangSmith version: 0.3.8

Originally created by @mshavliuk on GitHub (Feb 17, 2025). Original GitHub issue: https://github.com/langchain-ai/docs/issues/1499 Originally assigned to: @jacoblee93 on GitHub. **Description** Following the instructions in [docs/reference/sdk_reference/langchain_evaluators.mdx](https://github.com/langchain-ai/langsmith-docs/blob/25f33508ce42656a17ca39e79510a033f4983858/docs/reference/sdk_reference/langchain_evaluators.mdx) I tried to create 'json_schema' evaluator: ```python from langsmith.evaluation import LangChainStringEvaluator LangChainStringEvaluator("json_schema") ``` And got the following error: ``` Traceback (most recent call last): File "/Users/user/PycharmProjects/ml-sandbox/src/bug.py", line 3, in <module> LangChainStringEvaluator("json_schema") File "/Users/user/PycharmProjects/ml-sandbox/.venv/lib/python3.12/site-packages/langsmith/evaluation/integrations/_langchain.py", line 165, in __init__ self.evaluator = load_evaluator(evaluator, **(config or {})) # type: ignore[assignment, arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/user/PycharmProjects/ml-sandbox/.venv/lib/python3.12/site-packages/langchain/evaluation/loading.py", line 127, in load_evaluator raise ValueError( ValueError: Unknown evaluator type: json_schema Valid types are: [<EvaluatorType.QA: 'qa'>, <EvaluatorType.COT_QA: 'cot_qa'>, <EvaluatorType.CONTEXT_QA: 'context_qa'>, <EvaluatorType.PAIRWISE_STRING: 'pairwise_string'>, <EvaluatorType.SCORE_STRING: 'score_string'>, <EvaluatorType.LABELED_PAIRWISE_STRING: 'labeled_pairwise_string'>, <EvaluatorType.LABELED_SCORE_STRING: 'labeled_score_string'>, <EvaluatorType.AGENT_TRAJECTORY: 'trajectory'>, <EvaluatorType.CRITERIA: 'criteria'>, <EvaluatorType.LABELED_CRITERIA: 'labeled_criteria'>, <EvaluatorType.STRING_DISTANCE: 'string_distance'>, <EvaluatorType.PAIRWISE_STRING_DISTANCE: 'pairwise_string_distance'>, <EvaluatorType.EMBEDDING_DISTANCE: 'embedding_distance'>, <EvaluatorType.PAIRWISE_EMBEDDING_DISTANCE: 'pairwise_embedding_distance'>, <EvaluatorType.JSON_VALIDITY: 'json_validity'>, <EvaluatorType.JSON_EQUALITY: 'json_equality'>, <EvaluatorType.JSON_EDIT_DISTANCE: 'json_edit_distance'>, <EvaluatorType.JSON_SCHEMA_VALIDATION: 'json_schema_validation'>, <EvaluatorType.REGEX_MATCH: 'regex_match'>, <EvaluatorType.EXACT_MATCH: 'exact_match'>] ``` **Partial solution** Looking at the error message I found that it includes `'json_schema_validation'`, which looks as a correct candidate, so I used the following code: ``` class SummaryResponse(BaseModel): summary: str reason: str json_schema = LangChainStringEvaluator( "json_schema_validation", config={ "schema": SummaryResponse.model_json_schema() } ) experiment_results = langsmith_client.evaluate( simple, data="dataset_name", evaluators=[json_schema], ) ``` However, this led to another error: ``` Error running evaluator <DynamicRunEvaluator evaluate> on run 8ba9cf6c-90bf-4c51-ad37-e2ab01290413: ValueError('JsonSchemaEvaluator requires a reference string.') Traceback (most recent call last): File "/Users/user/PycharmProjects/ml-sandbox/.venv/lib/python3.12/site-packages/langsmith/evaluation/_runner.py", line 1634, in _run_evaluators evaluator_response = evaluator.evaluate_run( ^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/user/PycharmProjects/ml-sandbox/.venv/lib/python3.12/site-packages/langsmith/evaluation/evaluator.py", line 331, in evaluate_run result = self.func( ^^^^^^^^^^ File "/Users/user/PycharmProjects/ml-sandbox/.venv/lib/python3.12/site-packages/langsmith/run_helpers.py", line 629, in wrapper raise e File "/Users/user/PycharmProjects/ml-sandbox/.venv/lib/python3.12/site-packages/langsmith/run_helpers.py", line 626, in wrapper function_result = run_container["context"].run(func, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/user/PycharmProjects/ml-sandbox/.venv/lib/python3.12/site-packages/langsmith/evaluation/integrations/_langchain.py", line 260, in evaluate results = self.evaluator.evaluate_strings(**eval_inputs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/user/PycharmProjects/ml-sandbox/.venv/lib/python3.12/site-packages/langchain/evaluation/schema.py", line 219, in evaluate_strings self._check_evaluation_args(reference=reference, input=input) File "/Users/user/PycharmProjects/ml-sandbox/.venv/lib/python3.12/site-packages/langchain/evaluation/schema.py", line 127, in _check_evaluation_args raise ValueError(f"{self.__class__.__name__} requires a reference string.") ValueError: JsonSchemaEvaluator requires a reference string. ``` Looking deeper at the implementation of [`LangChainStringEvaluator.evaluate`](https://github.com/langchain-ai/langsmith-sdk/blob/8f682e1abe246b99ae263d5e3f625eb3969c267f/python/langsmith/evaluation/integrations/_langchain.py#L253-L261) I can see that the second argument given to the `StringEvaluator` is an example field, taken from the dataset, which is `None` in my case. Instead, I expect it to use the given JSON schema, but I didn't manage to figure out how to make it work. **Context** LangSmith version: 0.3.8
yindo added the langsmithexternal labels 2026-02-17 17:19:00 -05:00
yindo changed title from `json_schema` evaluator docs are misleading to [GH-ISSUE #1499] `json_schema` evaluator docs are misleading 2026-06-05 17:24:36 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
deepagents
docs-infra
external
external
help wanted
integration
internal
internal
javascript
js/ts
langchain
langgraph
langsmith
needs-title
pull-request
Mirrored from GitHub Pull Request
 python
reference
No Label external langsmith
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langchain-ai/docs#19
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?