langchain-ai/docs
1
0
Fork 0
mirror of https://github.com/langchain-ai/docs.git synced 2026-07-23 04:45:23 -04:00
Code Issues 490 Packages Projects Releases Wiki Activity

[GH-ISSUE #1512] DOC: <Issue related to /tutorials/Developers/backtesting> #4

New Issue
Open
opened 2026-02-17 17:18:58 -05:00 by yindo · 0 comments
yindo commented 2026-02-17 17:18:58 -05:00
Owner
Copy Link
Copy Source

Originally created by @labdmitriy on GitHub (Sep 17, 2024).
Original GitHub issue: https://github.com/langchain-ai/docs/issues/1512

Hi,

I tried to reproduce the example from the documentation for backtesting and also Backtesting | LangSmith Evaluations - Part 19, but each time I have the following error during convert_runs_to_test invocation:

AttributeError                            Traceback (most recent call last)
Cell In[18], line 27
     24 dataset_name = f'{project_name}-backtesting {start_time.strftime("%Y-%m-%d")}-{end_time.strftime("%Y-%m-%d")}'
     25 # This converts the runs to a dataset + experiment
     26 # It does not actually invoke your model
---> 27 convert_runs_to_test(
     28     prod_runs,
     29     # Name of the resulting dataset
     30     dataset_name=dataset_name,
     31     # Whether to include the run outputs as reference[/ground](http://localhost:8888/ground) truth
     32     include_outputs=False,
     33     # Whether to include the full traces in the resulting experiment
     34     # (default is to just include the root run)
     35 
     36     # AttributeError: 'NoneType' object has no attribute 'items'
     37     load_child_runs=True,
     38 )

File [.../lib/python3.11/site-packages/langsmith/_internal/_beta_decorator.py:19](.../lib/python3.11/site-packages/langsmith/_internal/_beta_decorator.py#line=18), in warn_beta.<locals>.wrapper(*args, **kwargs)
     16 @functools.wraps(func)
     17 def wrapper(*args, **kwargs):
     18     _warn_once(f"Function {func.__name__} is in beta.")
---> 19     return func(*args, **kwargs)

File [.../lib/python3.11/site-packages/langsmith/beta/_evals.py:165](.../lib/python3.11/site-packages/langsmith/beta/_evals.py#line=164), in convert_runs_to_test(runs, dataset_name, test_project_name, client, load_child_runs, include_outputs)
    155 project = client.create_project(
    156     project_name=test_project_name,
    157     reference_dataset_id=ds.id,
   (...)
    161     },
    162 )
    164 for new_run in to_create:
--> 165     client.create_run(**new_run, project_name=test_project_name)
    167 _ = client.update_project(
    168     project.id, end_time=datetime.datetime.now(tz=datetime.timezone.utc)
    169 )
    170 return project

File [.../lib/python3.11/site-packages/langsmith/client.py:1277](.../lib/python3.11/site-packages/langsmith/client.py#line=1276), in Client.create_run(self, name, inputs, run_type, project_name, revision_id, **kwargs)
   1275 if not self._filter_for_sampling([run_create]):
   1276     return
-> 1277 run_create = self._run_transform(run_create, copy=True)
   1278 if revision_id is not None:
   1279     run_create["extra"]["metadata"]["revision_id"] = revision_id

File [.../lib/python3.11/site-packages/langsmith/client.py:1181](.../lib/python3.11/site-packages/langsmith/client.py#line=1180), in Client._run_transform(self, run, update, copy)
   1177         run_create = {k: v for k, v in run_create.items() if k != "serialized"}
   1178     else:
   1179         # Drop graph
   1180         serialized = {
-> 1181             k: v for k, v in run_create["serialized"].items() if k != "graph"
   1182         }
   1183         run_create = {**run_create, "serialized": serialized}
   1185 return run_create

AttributeError: 'NoneType' object has no attribute 'items'

When I set load_child_runs=False then there are no errors, but the dataset will be created empty without any runs.

Thank you.

Originally created by @labdmitriy on GitHub (Sep 17, 2024). Original GitHub issue: https://github.com/langchain-ai/docs/issues/1512 Hi, I tried to reproduce the example from the documentation for backtesting and also [Backtesting | LangSmith Evaluations - Part 19](https://www.youtube.com/watch?v=3cDtDI2W-xA), but each time I have the following error during `convert_runs_to_test` invocation: ``` AttributeError Traceback (most recent call last) Cell In[18], line 27 24 dataset_name = f'{project_name}-backtesting {start_time.strftime("%Y-%m-%d")}-{end_time.strftime("%Y-%m-%d")}' 25 # This converts the runs to a dataset + experiment 26 # It does not actually invoke your model ---> 27 convert_runs_to_test( 28 prod_runs, 29 # Name of the resulting dataset 30 dataset_name=dataset_name, 31 # Whether to include the run outputs as reference[/ground](http://localhost:8888/ground) truth 32 include_outputs=False, 33 # Whether to include the full traces in the resulting experiment 34 # (default is to just include the root run) 35 36 # AttributeError: 'NoneType' object has no attribute 'items' 37 load_child_runs=True, 38 ) File [.../lib/python3.11/site-packages/langsmith/_internal/_beta_decorator.py:19](.../lib/python3.11/site-packages/langsmith/_internal/_beta_decorator.py#line=18), in warn_beta.<locals>.wrapper(*args, **kwargs) 16 @functools.wraps(func) 17 def wrapper(*args, **kwargs): 18 _warn_once(f"Function {func.__name__} is in beta.") ---> 19 return func(*args, **kwargs) File [.../lib/python3.11/site-packages/langsmith/beta/_evals.py:165](.../lib/python3.11/site-packages/langsmith/beta/_evals.py#line=164), in convert_runs_to_test(runs, dataset_name, test_project_name, client, load_child_runs, include_outputs) 155 project = client.create_project( 156 project_name=test_project_name, 157 reference_dataset_id=ds.id, (...) 161 }, 162 ) 164 for new_run in to_create: --> 165 client.create_run(**new_run, project_name=test_project_name) 167 _ = client.update_project( 168 project.id, end_time=datetime.datetime.now(tz=datetime.timezone.utc) 169 ) 170 return project File [.../lib/python3.11/site-packages/langsmith/client.py:1277](.../lib/python3.11/site-packages/langsmith/client.py#line=1276), in Client.create_run(self, name, inputs, run_type, project_name, revision_id, **kwargs) 1275 if not self._filter_for_sampling([run_create]): 1276 return -> 1277 run_create = self._run_transform(run_create, copy=True) 1278 if revision_id is not None: 1279 run_create["extra"]["metadata"]["revision_id"] = revision_id File [.../lib/python3.11/site-packages/langsmith/client.py:1181](.../lib/python3.11/site-packages/langsmith/client.py#line=1180), in Client._run_transform(self, run, update, copy) 1177 run_create = {k: v for k, v in run_create.items() if k != "serialized"} 1178 else: 1179 # Drop graph 1180 serialized = { -> 1181 k: v for k, v in run_create["serialized"].items() if k != "graph" 1182 } 1183 run_create = {**run_create, "serialized": serialized} 1185 return run_create AttributeError: 'NoneType' object has no attribute 'items' ``` When I set `load_child_runs=False` then there are no errors, but the dataset will be created empty without any runs. Thank you.
yindo added the langsmithexternal labels 2026-02-17 17:18:58 -05:00
yindo changed title from DOC: <Issue related to /tutorials/Developers/backtesting> to [GH-ISSUE #1512] DOC: <Issue related to /tutorials/Developers/backtesting> 2026-06-05 17:24:31 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
deepagents
docs-infra
external
external
help wanted
integration
internal
internal
javascript
js/ts
langchain
langgraph
langsmith
needs-title
pull-request
Mirrored from GitHub Pull Request
 python
reference
No Label external langsmith
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langchain-ai/docs#4
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?