ccurme
34cd281494
- Drop support for python 3.8 - Bump langchain-core to 0.3 - Update pydantic objects to v2
🦜💯 LangChain Benchmarks
A package to help benchmark various LLM related tasks.
The benchmarks are organized by end-to-end use cases, and utilize LangSmith heavily.
We have several goals in open sourcing this:
- Showing how we collect our benchmark datasets for each task
- Showing what the benchmark datasets we use for each task is
- Showing how we evaluate each task
- Encouraging others to benchmark their solutions on these tasks (we are always looking for better ways of doing things!)
Benchmarking Results
Read some of the articles about benchmarking results on our blog.
Tool Usage (2024-04-18)
See tool usage docs to recreate!
Explore Agent Traces on LangSmith:
Installation
To install the packages, run the following command:
pip install -U langchain-benchmarks
All the benchmarks come with an associated benchmark dataset stored in LangSmith. To take advantage of the eval and debugging experience, sign up, and set your API key in your environment:
export LANGCHAIN_API_KEY=ls-...
Repo Structure
The package is located within langchain_benchmarks. Check out the docs for information on how to get starte.
The other directories are legacy and may be moved in the future.
Archived
Below are archived benchmarks that require cloning this repo to run.
- CSV Question Answering
- Extraction
- Q&A over the LangChain docs
- Meta-evaluation of 'correctness' evaluators
Related
- For cookbooks on other ways to test, debug, monitor, and improve your LLM applications, check out the LangSmith docs
- For information on building with LangChain, check out the python documentation or JS documentation