langchain-ai/auto-evaluator
1
0
Fork 0
mirror of https://github.com/langchain-ai/auto-evaluator.git synced 2026-07-19 22:53:35 -04:00
Code Issues 21 Packages Projects Releases Wiki Activity

[PR #92] [MERGED] Add Vicuna 13b along w/ Transformer test set #105

New Issue
Closed
opened 2026-02-15 19:15:02 -05:00 by yindo · 0 comments
yindo commented 2026-02-15 19:15:02 -05:00
Owner
Copy Link
Copy Source

📋 Pull Request Information

Original PR: https://github.com/langchain-ai/auto-evaluator/pull/92
Author: @rlancemartin
Created: 5/5/2023
Status: ✅ Merged
Merged: 5/29/2023
Merged by: @rlancemartin

Base: main ← Head: local_many_file_upload

📝 Commits (10+)

📊 Changes

14 files changed (+502 additions, -18 deletions)

View changed files

➕ api/Test_Vicuna_Locally.ipynb (+442 -0)
📝 api/docs/gpt3/gpt3-eval.csv (+1 -1)
➕ api/docs/transformers-challenge/2005.14165.pdf (+0 -0)
➕ api/docs/transformers-challenge/2112.04426.pdf (+0 -0)
➕ api/docs/transformers-challenge/2203.15556.pdf (+0 -0)
➕ api/docs/transformers-challenge/2205.06175.pdf (+0 -0)
➕ api/docs/transformers-challenge/2302.13971.pdf (+0 -0)
➕ api/docs/transformers-challenge/transformers-eval.csv (+11 -0)
📝 api/evaluator_app.py (+27 -12)
📝 api/requirements.txt (+4 -3)
📝 api/text_utils.py (+13 -0)
📝 nextjs/components/Playground.tsx (+2 -1)
📝 nextjs/components/Sidebar.tsx (+1 -0)
📝 nextjs/pages/playground/index.tsx (+1 -1)

📄 Description

Blog fro running Vicuna locally here.

Using Replicate to host model:

  • Fixed bug w/ API here
  • Tested prompting tips here

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/langchain-ai/auto-evaluator/pull/92 **Author:** [@rlancemartin](https://github.com/rlancemartin) **Created:** 5/5/2023 **Status:** ✅ Merged **Merged:** 5/29/2023 **Merged by:** [@rlancemartin](https://github.com/rlancemartin) **Base:** `main` ← **Head:** `local_many_file_upload` --- ### 📝 Commits (10+) - [`2c0936e`](https://github.com/langchain-ai/auto-evaluator/commit/2c0936ea916502533e4bc996bc79bd56c0330a0f) Add Vicuna - [`7cd7a50`](https://github.com/langchain-ai/auto-evaluator/commit/7cd7a50a94b8c96f091e084a776704e00a4d9b7a) Add challenge set - [`b22d6b9`](https://github.com/langchain-ai/auto-evaluator/commit/b22d6b96872cfea8bd0351e73d15019bf1187595) Update Vicuna params and prompt - [`11c9d56`](https://github.com/langchain-ai/auto-evaluator/commit/11c9d5649a6941e4da983766ff1ddb97fbdf6e05) Notebook for local testing - [`43304fe`](https://github.com/langchain-ai/auto-evaluator/commit/43304fe64d01328d2ce4887938203c515fc64382) Add hosted model - [`87b64b2`](https://github.com/langchain-ai/auto-evaluator/commit/87b64b26b8ec95d4faff019dda990a5c64940cba) Only allow a single file upload for deployed all - [`0cbb3e4`](https://github.com/langchain-ai/auto-evaluator/commit/0cbb3e4974e919511f56c35a9b2091ca6baa4939) update notebook - [`2534ca4`](https://github.com/langchain-ai/auto-evaluator/commit/2534ca4534a25401226b9052d425fab1911c5d9d) fix Replicate in notebook - [`c966923`](https://github.com/langchain-ai/auto-evaluator/commit/c966923dc10ada413afd09ed628d5d94c46ef7f8) Add Vicuna13b - [`9b1db27`](https://github.com/langchain-ai/auto-evaluator/commit/9b1db275323e6b37f64b04a3f5feca81c68908ba) Update prompt and model params ### 📊 Changes **14 files changed** (+502 additions, -18 deletions) <details> <summary>View changed files</summary> ➕ `api/Test_Vicuna_Locally.ipynb` (+442 -0) 📝 `api/docs/gpt3/gpt3-eval.csv` (+1 -1) ➕ `api/docs/transformers-challenge/2005.14165.pdf` (+0 -0) ➕ `api/docs/transformers-challenge/2112.04426.pdf` (+0 -0) ➕ `api/docs/transformers-challenge/2203.15556.pdf` (+0 -0) ➕ `api/docs/transformers-challenge/2205.06175.pdf` (+0 -0) ➕ `api/docs/transformers-challenge/2302.13971.pdf` (+0 -0) ➕ `api/docs/transformers-challenge/transformers-eval.csv` (+11 -0) 📝 `api/evaluator_app.py` (+27 -12) 📝 `api/requirements.txt` (+4 -3) 📝 `api/text_utils.py` (+13 -0) 📝 `nextjs/components/Playground.tsx` (+2 -1) 📝 `nextjs/components/Sidebar.tsx` (+1 -0) 📝 `nextjs/pages/playground/index.tsx` (+1 -1) </details> ### 📄 Description Blog fro running Vicuna locally [here](https://rlancemartin.notion.site/Benchmarking-Vicuna-in-LangChain-with-Auto-Evaluator-5b698a56cb3e45b89f82586408883316). Using Replicate to host model: * Fixed bug w/ API [here](https://github.com/replicate/cog-vicuna-13b/issues/4) * Tested prompting tips [here](https://github.com/replicate/cog-vicuna-13b/issues/5) --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
yindo added the pull-request label 2026-02-15 19:15:02 -05:00
yindo closed this issue 2026-02-15 19:15:02 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
bug
good first issue
pull-request
Mirrored from GitHub Pull Request
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langchain-ai/auto-evaluator#105
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?