[PR #111] [CLOSED] Add support for Cohere's generative model and embeddings #111

New Issue

2026-02-15T19:15:03-05:00

yindo commented

2026-02-15 19:15:03 -05:00

📋 Pull Request Information

Original PR: https://github.com/langchain-ai/auto-evaluator/pull/111
Author: @samuelpath
Created: 11/19/2023
Status: ❌ Closed

Base: main ← Head: add-cohere-support

📝 Commits (1)

cbaeedf Add support for Cohere model and embeddings

📊 Changes

5 files changed (+44 additions, -5 deletions)

View changed files

📝 README.md (+1 -0)
📝 api/README.md (+2 -2)
📝 api/evaluator_app.py (+27 -2)
📝 api/requirements.txt (+1 -1)
📝 nextjs/components/Sidebar.tsx (+13 -0)

📄 Description

Here's the list of the main changes in this PR:

1. Add the Cohere model (as LLM for summarization of retrieved chunks into the answer)

This is what it would look like in the UI:

The front-end sends the value cohere to the API through the model_version parameter thanks to this additional mapping in the front-end code:

{ label: "Cohere", value: "cohere" }

And in the API, we simply need to import Cohere from langchain.llms, as well as add a conditional to handle the model_version=cohere parameter:

from langchain.llms import Cohere

# ...
elif model == "cohere":
    llm = Cohere(model="command", temperature=0)
# ...

Under the hood, we call Cohere's Co.Generate endpoint with their default command model.

Also as part of this addition, we need to tell users in the README to add Cohere's API key:

export COHERE_API_KEY=

2. Add 3 embeddings models from Cohere (used to embed chunks)

Previously, the evaluator had only access to OpenAI's API to embed chunks. We now add the following 3 embeddings models from Cohere's Co.Embed endpoint:

Cohere English v2.0
Cohere English v3.0
Cohere Multilingual v3.0

This is what it would look like in the UI:

These are the 3 additional mappings added to the front-end code:

{
  label: "Cohere English v2.0",
  value: "cohere-english-v2.0",
},
{
  label: "Cohere English v3.0",
  value: "cohere-english-v3.0",
},
{
  label: "Cohere Multilingual v3.0",
  value: "cohere-multilingual-v3.0",
},

And this is how the API parses these new parameters sent to the embeddings parameter:

elif embeddings == "cohere-english-v2.0":
    embd = CohereEmbeddings(model='embed-english-v2.0')
elif embeddings == "cohere-english-v3.0":
    embd = CohereEmbeddings(model='embed-english-v3.0')
elif embeddings == "cohere-multilingual-v3.0":
    embd = CohereEmbeddings(model='embed-multilingual-v3.0')

3. Bump the `langchain` package dependency from `0.0.181` to `0.0.338`

This was necessary because this recent commit (https://github.com/langchain-ai/langchain/commit/52d0055a915e9d87f0175c0052a7754f1b73ccf8) added support for Cohere Embed v3 in langchain's Cohere embeddings adapter (the one we import using from langchain.embeddings.cohere import CohereEmbeddings).

Basically, an input_type parameter now needs to be passed to the embed endpoint:

embeddings = await self.async_client.embed(
-    model=self.model, texts=texts, truncate=self.truncate
+    model=self.model,
+    texts=texts,
+    input_type="search_document",
+    truncate=self.truncate,
)

As we see in Cohere's embed docs, this is only required starting at v3, but it is also compatible with previous versions:

Following this langchain version bump, we had to replace text by results in the graded_answers and the graded_retrieval:

- d['answerScore'] = [g['text'] for g in graded_answers]
- d['retrievalScore'] = [g['text'] for g in graded_retrieval]
+ d['answerScore'] = [g['results'] for g in graded_answers]
+ d['retrievalScore'] = [g['results'] for g in graded_retrieval]

This is due to the line below added in QAEvalChain 4 months ago as part of this breaking commit (https://github.com/langchain-ai/langchain/commit/a673a51efa3e03aaa7c8c7e0004dc5ff9c536f2e):

output_key: str = "results"  #: :meta private:

4. Add a log at the beginning of each call to the API with the params passed

This has been very useful for me as I was implementing this PR, and I think it would benefit all those who work on this repo:

logger.info(
    f"Calling evaluator with params:\n"
    f"  files={[file.filename for file in files]},\n"
    f"  num_eval_questions={num_eval_questions},\n"
    f"  chunk_chars={chunk_chars},\n"
    f"  overlap={overlap},\n"
    f"  split_method={split_method},\n"
    f"  retriever_type={retriever_type},\n"
    f"  embeddings={embeddings},\n"
    f"  model_version={model_version},\n"
    f"  grade_prompt={grade_prompt},\n"
    f"  num_neighbors={num_neighbors},\n"
    f"  test_dataset={test_dataset},\n"
)

This is what it looks like in the server logs:

2023-11-19 12:32:38,522 loglevel=INFO   logger=evaluator_app run_evaluator() L359  Calling run_evaluator with params:
  files=['karpathy-pod.txt'],
  num_eval_questions=1,
  chunk_chars=1000,
  overlap=100,
  split_method=RecursiveTextSplitter,
  retriever_type=similarity-search,
  embeddings=cohere-english-v3.0,
  model_version=cohere,
  grade_prompt=Fast,
  num_neighbors=3,
  test_dataset=[],

I welcome any feedback on this PR and would happily implement all required changes :).

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/langchain-ai/auto-evaluator/pull/111 **Author:** [@samuelpath](https://github.com/samuelpath) **Created:** 11/19/2023 **Status:** ❌ Closed **Base:** `main` ← **Head:** `add-cohere-support` --- ### 📝 Commits (1) - [`cbaeedf`](https://github.com/langchain-ai/auto-evaluator/commit/cbaeedf23c737fb90503b9e2212844803bd18c92) Add support for Cohere model and embeddings ### 📊 Changes **5 files changed** (+44 additions, -5 deletions) <details> <summary>View changed files</summary> 📝 `README.md` (+1 -0) 📝 `api/README.md` (+2 -2) 📝 `api/evaluator_app.py` (+27 -2) 📝 `api/requirements.txt` (+1 -1) 📝 `nextjs/components/Sidebar.tsx` (+13 -0) </details> ### 📄 Description Here's the list of the main changes in this PR: ## 1. Add the Cohere model (as LLM for summarization of retrieved chunks into the answer) This is what it would look like in the UI: <img width="441" alt="image" src="https://github.com/langchain-ai/auto-evaluator/assets/6030745/85521182-d06d-48cd-9149-e69ff5766309"> The front-end sends the value `cohere` to the API through the `model_version` parameter thanks to this additional mapping in the front-end code: ```ts { label: "Cohere", value: "cohere" } ``` And in the API, we simply need to import `Cohere` from `langchain.llms`, as well as add a conditional to handle the `model_version=cohere` parameter: ```python from langchain.llms import Cohere # ... elif model == "cohere": llm = Cohere(model="command", temperature=0) # ... ``` Under the hood, we call [Cohere's Co.Generate endpoint](https://docs.cohere.com/reference/generate) with their default `command` model. Also as part of this addition, we need to tell users in the `README` to add Cohere's API key: ``` export COHERE_API_KEY= ``` ## 2. Add 3 embeddings models from Cohere (used to embed chunks) Previously, the evaluator had only access to `OpenAI`'s API to embed chunks. We now add the following 3 embeddings models from [Cohere's Co.Embed endpoint](https://docs.cohere.com/reference/embed): - Cohere English v2.0 - Cohere English v3.0 - Cohere Multilingual v3.0 This is what it would look like in the UI: <img width="441" alt="image" src="https://github.com/langchain-ai/auto-evaluator/assets/6030745/3a8f2444-96fc-48ee-b9fc-4c8ac634706a"> These are the 3 additional mappings added to the front-end code: ```ts { label: "Cohere English v2.0", value: "cohere-english-v2.0", }, { label: "Cohere English v3.0", value: "cohere-english-v3.0", }, { label: "Cohere Multilingual v3.0", value: "cohere-multilingual-v3.0", }, ``` And this is how the API parses these new parameters sent to the `embeddings` parameter: ```python elif embeddings == "cohere-english-v2.0": embd = CohereEmbeddings(model='embed-english-v2.0') elif embeddings == "cohere-english-v3.0": embd = CohereEmbeddings(model='embed-english-v3.0') elif embeddings == "cohere-multilingual-v3.0": embd = CohereEmbeddings(model='embed-multilingual-v3.0') ``` ## 3. Bump the `langchain` package dependency from `0.0.181` to `0.0.338` This was necessary because this recent commit (https://github.com/langchain-ai/langchain/commit/52d0055a915e9d87f0175c0052a7754f1b73ccf8) added support for Cohere Embed v3 in langchain's Cohere embeddings adapter (the one we import using `from langchain.embeddings.cohere import CohereEmbeddings`). Basically, an `input_type` parameter now needs to be passed to the `embed` endpoint: ```diff embeddings = await self.async_client.embed( - model=self.model, texts=texts, truncate=self.truncate + model=self.model, + texts=texts, + input_type="search_document", + truncate=self.truncate, ) ``` As we see in [Cohere's embed docs](https://docs.cohere.com/reference/embed), this is only required starting at v3, but it is also compatible with previous versions: <img width="455" alt="image" src="https://github.com/langchain-ai/auto-evaluator/assets/6030745/1ca60055-f1fa-4617-a3f2-85956202788a"> Following this langchain version bump, we had to replace `text` by `results` in the `graded_answers` and the `graded_retrieval`: ```diff - d['answerScore'] = [g['text'] for g in graded_answers] - d['retrievalScore'] = [g['text'] for g in graded_retrieval] + d['answerScore'] = [g['results'] for g in graded_answers] + d['retrievalScore'] = [g['results'] for g in graded_retrieval] ``` This is due to the line below added in [QAEvalChain](https://github.com/langchain-ai/langchain/blob/78a1f4b264fbdca263a4f8873b980eaadb8912a7/libs/langchain/langchain/evaluation/qa/eval_chain.py#L72) 4 months ago as part of this breaking commit (https://github.com/langchain-ai/langchain/commit/a673a51efa3e03aaa7c8c7e0004dc5ff9c536f2e): ```python output_key: str = "results" #: :meta private: ``` ## 4. Add a log at the beginning of each call to the API with the params passed This has been very useful for me as I was implementing this PR, and I think it would benefit all those who work on this repo: ```python logger.info( f"Calling evaluator with params:\n" f" files={[file.filename for file in files]},\n" f" num_eval_questions={num_eval_questions},\n" f" chunk_chars={chunk_chars},\n" f" overlap={overlap},\n" f" split_method={split_method},\n" f" retriever_type={retriever_type},\n" f" embeddings={embeddings},\n" f" model_version={model_version},\n" f" grade_prompt={grade_prompt},\n" f" num_neighbors={num_neighbors},\n" f" test_dataset={test_dataset},\n" ) ``` This is what it looks like in the server logs: ``` 2023-11-19 12:32:38,522 loglevel=INFO logger=evaluator_app run_evaluator() L359 Calling run_evaluator with params: files=['karpathy-pod.txt'], num_eval_questions=1, chunk_chars=1000, overlap=100, split_method=RecursiveTextSplitter, retriever_type=similarity-search, embeddings=cohere-english-v3.0, model_version=cohere, grade_prompt=Fast, num_neighbors=3, test_dataset=[], ``` I welcome any feedback on this PR and would happily implement all required changes :). --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

yindo added the pull-request label 2026-02-15 19:15:03 -05:00

yindo closed this issue

2026-02-15 19:15:03 -05:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: langchain-ai/auto-evaluator#111

[PR #111] [CLOSED] Add support for Cohere's generative model and embeddings #111