lm-evaluation-harness/examples/visualize-zeno.ipynb

Visualizing Results in Zeno

Benchmarking your models is the first step towards making sure your model performs well. However, looking at the data behind the benchmark, slicing the data into subsets, and comparing models on individual instances can help you even more in evaluating and quantifying the behavior of your AI system.

All of this can be done in Zeno! Zeno is super easy to use with the eval harness, let's explore how you can easily upload and visualize your eval results.

In [ ]:

# Install this project if you did not already do that. This is all that needs to be installed for you to be able to visualize your data in Zeno!
!pip install -e ..
!pip install -e ..[zeno]

Run the Eval Harness

To visualize the results, run the eval harness with the log_samples and output_path flags. We expect output_path to contain multiple folders that represent individual model names. You can thus run your evaluation on any number of tasks and models and upload all of the results as projects on Zeno.

In [ ]:

!lm_eval \
    --model hf \
    --model_args pretrained=EleutherAI/gpt-neo-2.7B \
    --tasks hellaswag,wikitext \
    --batch_size 8 \
    --device mps \
    --log_samples \
    --output_path output/gpt-neo-2.7B \
    --limit 10

Set your API Key

This is so you can be authenticated with Zeno. If you don't already have a Zeno account, first create an account on Zeno Hub. After logging in to Zeno Hub, generate your API key by clicking on your profile at the bottom left to navigate to your account page.

In [ ]:

%env ZENO_API_KEY=YOUR_API_KEY

Visualize Eval Results

You can now use the zeno_visualize script to upload the results to Zeno.

This will use all subfolders in data_path as different models and upload all tasks within these model folders to Zeno. If you run the eval harness on multiple tasks, the project_name will be used as a prefix and one project will be created per task.

In [ ]:

!python ../scripts/zeno_visualize.py --data_path output --project_name "Zeno Upload Test"