langchain-ai/auto-evaluator
1
0
Fork 0
mirror of https://github.com/langchain-ai/auto-evaluator.git synced 2026-07-18 12:46:06 -04:00
Code Issues 21 Packages Projects Releases Wiki Activity

[GH-ISSUE #97] How generalizable is this framework? #57

New Issue
Open
opened 2026-02-15 18:17:11 -05:00 by yindo · 0 comments
yindo commented 2026-02-15 18:17:11 -05:00
Owner
Copy Link
Copy Source

Originally created by @samching on GitHub (May 7, 2023).
Original GitHub issue: https://github.com/langchain-ai/auto-evaluator/issues/97

Great work on this! I created an issue because there isn't a discussion tab in this repo. Feel free to close if irrelevant.

It seems like the framework described here can be generalized to the following steps:

  1. You define a task with the prompt(s) for it, and a test set with a few "gold standard positive and negative examples".
  2. You run n generations
  3. You score the generations (either human in the loop or grader prompt or a combination of both)
  4. Tweak the task models / prompts / hyperparams and repeat

Right now, auto-evaluator is focused on qa with retrieval, but I can imagine that you could run the same process with:

  • Classification tasks (i.e. grader prompt: how well is the classifier doing?)
  • Summarization tasks (i.e. grader prompt: how accurately does the summary capture the target text)
  • Style transfer tasks (i.e. grader prompt: how much does this sound like X?)
  • Ranking tasks (i.e. grader prompt: how much does this order capture Y?)
  • ...

Maybe I'm thinking about it too broadly, but curious if that's the vision of where y'all see this project headed!

Originally created by @samching on GitHub (May 7, 2023). Original GitHub issue: https://github.com/langchain-ai/auto-evaluator/issues/97 Great work on this! I created an issue because there isn't a discussion tab in this repo. Feel free to close if irrelevant. It seems like the framework described here can be generalized to the following steps: 1. You define a task with the prompt(s) for it, and a test set with a few "gold standard positive and negative examples". 2. You run _n_ generations 3. You score the generations (either human in the loop or grader prompt or a combination of both) 4. Tweak the task models / prompts / hyperparams and repeat Right now, auto-evaluator is focused on qa with retrieval, but I can imagine that you could run the same process with: - Classification tasks (i.e. grader prompt: how well is the classifier doing?) - Summarization tasks (i.e. grader prompt: how accurately does the summary capture the target text) - Style transfer tasks (i.e. grader prompt: how much does this sound like X?) - Ranking tasks (i.e. grader prompt: how much does this order capture Y?) - ... Maybe I'm thinking about it too broadly, but curious if that's the vision of where y'all see this project headed!
yindo changed title from How generalizable is this framework? to [GH-ISSUE #97] How generalizable is this framework? 2026-06-05 17:14:16 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
bug
good first issue
pull-request
Mirrored from GitHub Pull Request
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langchain-ai/auto-evaluator#57
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?