Does LangChain-Benchmarks only support models from the model registry? #5

New Issue

2026-02-16T00:17:53-05:00

yindo commented

2026-02-16 00:17:53 -05:00

Originally created by @harpreetsahota204 on GitHub (Dec 21, 2023).

I'd like to run some benchmarks against models from Hugging Face. The tutorials seem tailored for models from the registry or OpenAI.

Before I go down the rabbit hole and try to use it myself, I thought I'd see if it was possible or if anyone has done this before and has examples I can look at.

Thanks

Originally created by @harpreetsahota204 on GitHub (Dec 21, 2023). I'd like to run some benchmarks against models from Hugging Face. The tutorials seem tailored for models from the registry or OpenAI. Before I go down the rabbit hole and try to use it myself, I thought I'd see if it was possible or if anyone has done this before and has examples I can look at. Thanks

yindo closed this issue

2026-02-16 00:17:53 -05:00

yindo commented

2026-02-16 00:17:53 -05:00

@hinthornw commented on GitHub (Dec 21, 2023):

Nope! The registry is a convenience wrapper that makes it easier for us to do a sweep over a bunch of llms and architectures we've already defined (and make it easier for you to re-run our things if you want to reproduce them), but you can run it on anything. Here's an example that's more manual: https://langchain-ai.github.io/langchain-benchmarks/notebooks/tool_usage/intro.html#custom-agent-factory

Or you can pass in a langchain LLM to the other agent "factories"

@hinthornw commented on GitHub (Dec 21, 2023): Nope! The registry is a convenience wrapper that makes it easier for us to do a sweep over a bunch of llms and architectures we've already defined (and make it easier for you to re-run our things if you want to reproduce them), but you can run it on anything. Here's an example that's more manual: https://langchain-ai.github.io/langchain-benchmarks/notebooks/tool_usage/intro.html#custom-agent-factory Or you can pass in a langchain LLM to the other agent "factories"

yindo referenced this issue

2026-02-16 00:17:57 -05:00

[PR #5] [MERGED] Add Criteria Evals #16

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: langchain-ai/langchain-benchmarks#5