EleutherAI/lm-evaluation-harness
1
0
Fork 0
mirror of https://github.com/EleutherAI/lm-evaluation-harness.git synced 2026-10-12 02:02:05 +00:00
Code Issues Packages Projects Releases Wiki Activity

add docstyle rule; update readme; linting

Browse Source
This commit is contained in:
Baber
2025-12-02 15:19:25 +05:00
parent 33fedbfa9f
commit 69741cfd8d
11 changed files with 308 additions and 191 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/workflows/unit_tests.yml
+4
View File
@@ -22,6 +22,8 @@ jobs:
steps:
- name: Checkout Code
uses: actions/checkout@v6
with:
fetch-depth: 0
- name: Install uv
uses: astral-sh/setup-uv@v7
with:
@@ -34,6 +36,8 @@ jobs:
env:
SKIP: "no-commit-to-branch,mypy"
uses: pre-commit/[email protected]
with:
extra_args: --from-ref ${{ github.event.pull_request.base.sha || 'HEAD~1' }} --to-ref HEAD
# Job 2
testcpu:
name: CPU Tests
README.md
+83 -23
View File
@@ -63,7 +63,35 @@ cd lm-evaluation-harness
pip install -e .
```
We also provide a number of optional dependencies for extended functionality. A detailed table is available at the end of this document.
### Installing Model Backends
The base installation provides the core evaluation framework. **Model backends must be installed separately** using optional extras:
For HuggingFace transformers models:
```bash
pip install "lm_eval[hf]"
```
For vLLM inference:
```bash
pip install "lm_eval[vllm]"
```
For API-based models (OpenAI, Anthropic, etc.):
```bash
pip install "lm_eval[api]"
```
Multiple backends can be installed together:
```bash
pip install "lm_eval[hf,vllm,api]"
```
A detailed table of all optional extras is available at the end of this document.
## Basic Usage
@@ -75,6 +103,9 @@ A list of supported tasks (or groupings of tasks) can be viewed with `lm-eval --
### Hugging Face `transformers`
> [!Important]
> To use the HuggingFace backend, first install: `pip install "lm_eval[hf]"`
To evaluate a model hosted on the [HuggingFace Hub](https://huggingface.co/models) (e.g. GPT-J-6B) on `hellaswag` you can use the following command (this assumes you are using a CUDA-compatible GPU):
```bash
@@ -307,9 +338,9 @@ lm_eval --model vllm \
--batch_size auto
```
To use vllm, do `pip install lm_eval[vllm]`. For a full list of supported vLLM configurations, please reference our [vLLM integration](https://github.com/EleutherAI/lm-evaluation-harness/blob/e74ec966556253fbe3d8ecba9de675c77c075bce/lm_eval/models/vllm_causallms.py) and the vLLM documentation.
To use vllm, do `pip install "lm_eval[vllm]"`. For a full list of supported vLLM configurations, please reference our [vLLM integration](https://github.com/EleutherAI/lm-evaluation-harness/blob/e74ec966556253fbe3d8ecba9de675c77c075bce/lm_eval/models/vllm_causallms.py) and the vLLM documentation.
vLLM occasionally differs in output from Huggingface. We treat Huggingface as the reference implementation, and provide a [script](./scripts/model_comparator.py) for checking the validity of vllm results against HF.
vLLM occasionally differs in output from Huggingface. We treat Huggingface as the reference implementation and provide a [script](./scripts/model_comparator.py) for checking the validity of vllm results against HF.
> [!Tip]
> For fastest performance, we recommend using `--batch_size auto` for vLLM whenever possible, to leverage its continuous batching functionality!
@@ -336,7 +367,7 @@ lm_eval --model sglang \
```
> [!Tip]
> When encountering out of memory (OOM) errors (especially for multiple-choice tasks), try these solutions:
> When encountering out-of-memory (OOM) errors (especially for multiple-choice tasks), try these solutions:
>
> 1. Use a manual `batch_size`, rather than `auto`.
> 2. Lower KV cache pool memory usage by adjusting `mem_fraction_static` - Add to your model arguments for example `--model_args pretrained=...,mem_fraction_static=0.7`.
@@ -344,6 +375,9 @@ lm_eval --model sglang \
### Model APIs and Inference Servers
> [!Important]
> To use API-based models, first install: `pip install "lm_eval[api]"`
Our library also supports the evaluation of models served via several commercial APIs, and we hope to implement support for the most commonly used performant local/self-hosted inference servers.
To call a hosted model, use:
@@ -581,7 +615,7 @@ To get started with development, first clone the repository and install the dev
```bash
git clone https://github.com/EleutherAI/lm-evaluation-harness
cd lm-evaluation-harness
pip install -e ".[dev]"
pip install -e ".[dev,hf]"
````
### Implementing new tasks
@@ -607,24 +641,50 @@ The best way to get support is to open an issue on this repo or join the [Eleuth
Extras dependencies can be installed via `pip install -e ".[NAME]"`
| NAME | Description | NAME | Description |
|----------------------|--------------------------------|----------------|---------------------------------------|
| tasks | All task-specific dependencies | api | API models (Anthropic, OpenAI, local) |
| acpbench | ACP Bench tasks | audiolm_qwen | Qwen2 audio models |
| ifeval | IFEval task | | |
| japanese_leaderboard | Japanese LLM tasks | gptq | AutoGPTQ models |
| longbench | LongBench tasks | gptqmodel | GPTQModel models |
| math | Math answer checking | hf_transfer | Speed up HF downloads |
| multilingual | Multilingual tokenizers | ibm_watsonx_ai | IBM watsonx.ai models |
| ruler | RULER tasks | ipex | Intel IPEX backend |
| | | | |
| dev | Linting & contributions | mamba | Mamba SSM models |
| promptsource | PromptSource prompts | neuronx | AWS inf2 instances |
| sentencepiece | Sentencepiece tokenizer | optimum | Intel OpenVINO models |
| testing | Run test suite | sae_lens | SAELens model steering |
| unitxt | Run unitxt tasks | | |
| wandb | Weights & Biases | sparsify | Sparsify model steering |
| zeno | Result visualization | vllm | vLLM models |
### Model Backends
These extras install dependencies required to run specific model backends:
| NAME | Description |
|----------------|--------------------------------------------------|
| hf | HuggingFace Transformers (torch, transformers, accelerate, peft) |
| vllm | vLLM fast inference |
| api | API models (OpenAI, Anthropic, local servers) |
| gptq | AutoGPTQ quantized models |
| gptqmodel | GPTQModel quantized models |
| ibm_watsonx_ai | IBM watsonx.ai models |
| ipex | Intel IPEX backend |
| optimum | Intel OpenVINO models |
| neuronx | AWS Inferentia2 instances |
| sparsify | Sparsify model steering |
| sae_lens | SAELens model steering |
### Task Dependencies
These extras install dependencies required for specific evaluation tasks:
| NAME | Description |
|----------------------|--------------------------------|
| tasks | All task-specific dependencies |
| acpbench | ACP Bench tasks |
| audiolm_qwen | Qwen2 audio models |
| ifeval | IFEval task |
| japanese_leaderboard | Japanese LLM tasks |
| longbench | LongBench tasks |
| math | Math answer checking |
| multilingual | Multilingual tokenizers |
| ruler | RULER tasks |
### Development & Utilities
| NAME | Description |
|---------------|--------------------------------|
| dev | Linting & contributions |
| hf_transfer | Speed up HF downloads |
| sentencepiece | Sentencepiece tokenizer |
| unitxt | Unitxt tasks |
| wandb | Weights & Biases logging |
| zeno | Zeno result visualization |
## Cite as
lm_eval/__init__.py
+9
View File
@@ -4,6 +4,15 @@ import os
__version__ = "0.4.9.2"
# Enable hf_transfer if available
try:
import hf_transfer # type: ignore
import huggingface_hub.constants # type: ignore
huggingface_hub.constants.HF_HUB_ENABLE_HF_TRANSFER = True
except ImportError:
pass
# Lazy-load .evaluator module to improve CLI startup
def __getattr__(name):
lm_eval/api/registry.py
+11 -9
View File
@@ -19,7 +19,7 @@ class MyModel(LM):
### Registering with Lazy Loading
```python
# Register without importing the actual implementation
model_registry.register("lazy-model", target="my_package.models:LazyModel")
model_registry.register("lazy-model", target="my_package.models: LazyModel")
```
### Looking up Components
@@ -39,9 +39,10 @@ import importlib.metadata as md
import inspect
import logging
import threading
from collections.abc import Callable
from functools import lru_cache
from types import MappingProxyType
from typing import TYPE_CHECKING, Any, Generic, TypeVar, Union, cast, overload
from typing import TYPE_CHECKING, Any, Generic, TypeVar, cast, overload
eval_logger = logging.getLogger(__name__)
@@ -53,6 +54,7 @@ if TYPE_CHECKING:
from lm_eval.api.filter import Filter
from lm_eval.api.model import LM
__all__ = [
# Core registry class
"Registry",
@@ -201,7 +203,7 @@ class Registry(Generic[T]):
>>> class MyModel(LM):
... pass
>>>
>>> # Direct registration with lazy placeholder
>>> # Direct registration with a lazy placeholder
>>> model_registry.register("lazy-name", target="mymodule:MyModel")
Raises:
@@ -401,7 +403,7 @@ class Registry(Generic[T]):
"""Erase registry (for isolated tests).
Clears both the registry contents and the materialization cache.
Only use this in test code to ensure clean state between tests.
Only use this in test code to ensure a clean state between tests.
"""
if isinstance(self._objs, MappingProxyType):
self._objs = dict(self._objs) # type: ignore[assignment]
@@ -520,7 +522,7 @@ MODEL_REGISTRY = model_registry
# =============================================================================
def register_filter(name):
def register_filter(name: str):
"""Decorator to register a filter class.
Args:
@@ -550,12 +552,12 @@ def get_filter(filter_name: str | Callable) -> Callable:
The filter class/function
Raises:
KeyError: If filter name is not found and is not callable
KeyError: If a filter name is not found and is not callable
"""
if callable(filter_name):
return filter_name
try:
return filter_registry.get(filter_name)
return filter_registry.get(cast("str", filter_name))
except KeyError as e:
eval_logger.warning(f"filter `{filter_name}` is not registered!")
raise e
@@ -574,7 +576,7 @@ def register_metric(**args):
"""Decorator to register a metric function.
Args:
**args: Keyword arguments including:
**args: Keyword arguments including
- metric: Name to register the metric under (required)
- higher_is_better: Whether higher scores are better
- aggregation: Name of aggregation function to use
@@ -609,7 +611,7 @@ def get_metric(name: str, hf_evaluate_metric: bool = False) -> Callable | None:
Args:
name: The metric name
hf_evaluate_metric: If True, skip local registry and use HF evaluate
hf_evaluate_metric: If True, skip the local registry and use HF evaluate
Returns:
The metric compute function, or None if not found
lm_eval/evaluator.py
+39 -43
View File
@@ -5,7 +5,7 @@ import os
import random
import time
from collections import defaultdict
from typing import TYPE_CHECKING, List, Optional, Union
from typing import TYPE_CHECKING
import numpy as np
@@ -49,28 +49,28 @@ eval_logger = logging.getLogger(__name__)
@positional_deprecated
def simple_evaluate(
model,
model_args: Optional[Union[str, dict]] = None,
tasks: Optional[List[Union[str, dict, object]]] = None,
num_fewshot: Optional[int] = None,
batch_size: Optional[Union[int, str]] = None,
max_batch_size: Optional[int] = None,
device: Optional[str] = None,
use_cache: Optional[str] = None,
model_args: str | dict | None = None,
tasks: list[str | dict | object] | None = None,
num_fewshot: int | None = None,
batch_size: int | str | None = None,
max_batch_size: int | None = None,
device: str | None = None,
use_cache: str | None = None,
cache_requests: bool = False,
rewrite_requests_cache: bool = False,
delete_requests_cache: bool = False,
limit: Optional[Union[int, float]] = None,
samples: Optional[dict] = None,
limit: int | float | None = None,
samples: dict | None = None,
bootstrap_iters: int = 100000,
check_integrity: bool = False,
write_out: bool = False,
log_samples: bool = True,
evaluation_tracker: Optional[EvaluationTracker] = None,
system_instruction: Optional[str] = None,
apply_chat_template: Union[bool, str] = False,
evaluation_tracker: EvaluationTracker | None = None,
system_instruction: str | None = None,
apply_chat_template: bool | str = False,
fewshot_as_multiturn: bool = False,
gen_kwargs: Union[str, dict, None] = None,
task_manager: Optional[TaskManager] = None,
gen_kwargs: str | dict | None = None,
task_manager: TaskManager | None = None,
verbosity=None,
predict_only: bool = False,
random_seed: int = 0,
@@ -78,7 +78,7 @@ def simple_evaluate(
torch_random_seed: int = 1234,
fewshot_random_seed: int = 1234,
confirm_run_unsafe_code: bool = False,
metadata: Optional[dict] = None,
metadata: dict | None = None,
):
"""Instantiate and evaluate a model on a list of tasks.
@@ -143,7 +143,8 @@ def simple_evaluate(
Random seed for fewshot sampler random generator. If set to None, the seed of generator will be set to None.
:param metadata: dict
Additional metadata to be added to the task manager. Will get passed to the download function of the task.
return
Return:
Dictionary of results
"""
if verbosity is not None:
@@ -420,15 +421,15 @@ def simple_evaluate(
def evaluate(
lm: "LM",
task_dict,
limit: Optional[int] = None,
samples: Optional[dict] = None,
limit: int | None = None,
samples: dict | None = None,
cache_requests: bool = False,
rewrite_requests_cache: bool = False,
bootstrap_iters: Optional[int] = 100000,
bootstrap_iters: int | None = 100000,
write_out: bool = False,
log_samples: bool = True,
system_instruction: Optional[str] = None,
apply_chat_template: Union[bool, str] = False,
system_instruction: str | None = None,
apply_chat_template: bool | str = False,
fewshot_as_multiturn: bool = False,
verbosity: str = "INFO",
confirm_run_unsafe_code: bool = False,
@@ -488,12 +489,11 @@ def evaluate(
# get lists of group hierarchy and each type of request
eval_tasks = get_task_list(task_dict)
if not log_samples:
if not all(
"bypass" not in getattr(task_output.task, "_metric_fn_list", {}).keys()
for task_output in eval_tasks
):
raise ValueError("log_samples must be True for 'bypass' metric-only tasks")
if not log_samples and not all(
"bypass" not in getattr(task_output.task, "_metric_fn_list", {})
for task_output in eval_tasks
):
raise ValueError("log_samples must be True for 'bypass' metric-only tasks")
# validation checks:
# 1.are we running multimodal task <-> non-multimodal model class, or vice-versa.
@@ -508,11 +508,10 @@ def evaluate(
raise ValueError(
f"Attempted to run task: {task_output.task_name} which is marked as unsafe. Set confirm_run_unsafe_code=True to run this task."
)
if len(incompatible_tasks) > 0:
if not getattr(lm, "MULTIMODAL", False):
raise ValueError(
f"Attempted to run tasks: {incompatible_tasks} which require multimodal input, but the selected model type does not currently implement this. Multimodal support is currently restricted to the ['hf-multimodal', 'vllm-vlm'] model type."
)
if len(incompatible_tasks) > 0 and not getattr(lm, "MULTIMODAL", False):
raise ValueError(
f"Attempted to run tasks: {incompatible_tasks} which require multimodal input, but the selected model type does not currently implement this. Multimodal support is currently restricted to the ['hf-multimodal', 'vllm-vlm'] model type."
)
# end validation check
# Cache the limit arg.
@@ -535,7 +534,7 @@ def evaluate(
system_instruction=system_instruction,
apply_chat_template=bool(apply_chat_template),
fewshot_as_multiturn=fewshot_as_multiturn,
chat_template=getattr(lm, "apply_chat_template")
chat_template=getattr(lm, "apply_chat_template", None)
if apply_chat_template
else None,
tokenizer_name=getattr(lm, "tokenizer_name", "")
@@ -587,7 +586,7 @@ def evaluate(
resps = getattr(lm, reqtype)(cloned_reqs)
# put responses from model into a list of length K for each request.
for x, req in zip(resps, cloned_reqs):
for x, req in zip(resps, cloned_reqs, strict=True):
req.resps.append(x)
if lm.world_size > 1:
@@ -597,7 +596,7 @@ def evaluate(
WORLD_SIZE = lm.world_size
### Postprocess outputs ###
# TODO: del model here, maybe (idea: allow user to specify device of e.g. reward model separately)
for task_output, limit in zip(eval_tasks, limits):
for task_output, limit in zip(eval_tasks, limits, strict=True):
task = task_output.task
task.apply_filters()
@@ -612,7 +611,7 @@ def evaluate(
for instances in instances_by_doc_id.values():
instances.sort(key=lambda x: x.idx)
# iterate over different filters used
for filter_key in task.instances[0].filtered_resps.keys():
for filter_key in task.instances[0].filtered_resps:
indices = (
samples.get(task_output.task_name, None)
if samples is not None
@@ -625,10 +624,7 @@ def evaluate(
samples=indices,
)
for doc_id, doc in doc_iterator:
if indices:
doc_id_true = indices[doc_id]
else:
doc_id_true = doc_id
doc_id_true = indices[doc_id] if indices else doc_id
requests = instances_by_doc_id[doc_id]
metrics = task.process_results(
doc, [req.filtered_resps[filter_key] for req in requests]
@@ -728,7 +724,7 @@ def evaluate(
): # subtask list will list "task_name": [] for solo tasks
for task in task_list:
for m, h in higher_is_better[task].items():
if m not in _higher_is_better.keys():
if m not in _higher_is_better:
_higher_is_better[m] = h
if (
@@ -762,7 +758,7 @@ def evaluate(
len(task_output.task.eval_docs),
),
}
for task_output, limit in zip(eval_tasks, limits)
for task_output, limit in zip(eval_tasks, limits, strict=True)
},
}
if log_samples:
lm_eval/loggers/utils.py
+25 -15
View File
@@ -4,17 +4,15 @@ import re
import subprocess
from importlib.metadata import version
from pathlib import Path
from typing import Any, Dict, Optional, Tuple, Union
from typing import Any
import numpy as np
from torch.utils.collect_env import get_pretty_env_info
from transformers import __version__ as trans_version
logger = logging.getLogger(__name__)
def remove_none_pattern(input_string: str) -> Tuple[str, bool]:
def remove_none_pattern(input_string: str) -> tuple[str, bool]:
"""Remove the ',none' substring from the input_string if it exists at the end.
Args:
@@ -36,7 +34,7 @@ def remove_none_pattern(input_string: str) -> Tuple[str, bool]:
return result, removed
def _handle_non_serializable(o: Any) -> Union[int, str, list]:
def _handle_non_serializable(o: Any) -> int | str | list:
"""Handle non-serializable objects by converting them to serializable types.
Args:
@@ -47,7 +45,7 @@ def _handle_non_serializable(o: Any) -> Union[int, str, list]:
it will be converted to int. If the object is of type set, it will be converted
to a list. Otherwise, it will be converted to str.
"""
if isinstance(o, np.int64) or isinstance(o, np.int32):
if isinstance(o, (np.int64, np.int32)):
return int(o)
elif isinstance(o, set):
return list(o)
@@ -55,7 +53,7 @@ def _handle_non_serializable(o: Any) -> Union[int, str, list]:
return str(o)
def get_commit_from_path(repo_path: Union[Path, str]) -> Optional[str]:
def get_commit_from_path(repo_path: Path | str) -> str | None:
try:
git_folder = Path(repo_path, ".git")
if git_folder.is_file():
@@ -96,16 +94,28 @@ def get_git_commit_hash():
return git_hash
def add_env_info(storage: Dict[str, Any]):
try:
pretty_env_info = get_pretty_env_info()
except Exception as err:
pretty_env_info = str(err)
def add_env_info(storage: dict[str, Any]):
from lm_eval.utils import is_torch_available, is_transformers_available
if is_torch_available():
try:
from torch.utils.collect_env import get_pretty_env_info
pretty_env_info = get_pretty_env_info()
except Exception as err:
pretty_env_info = str(err)
else:
pretty_env_info = "N/A (torch not installed)"
try:
lm_eval_version = version("lm_eval")
except Exception as err:
lm_eval_version = str(err)
transformers_version = trans_version
if is_transformers_available():
from transformers import __version__ as transformers_version
else:
transformers_version = "N/A"
upper_dir_commit = get_commit_from_path(
Path(os.getcwd(), "..")
) # git hash of upper repo if exists
@@ -118,7 +128,7 @@ def add_env_info(storage: Dict[str, Any]):
storage.update(added_info)
def add_tokenizer_info(storage: Dict[str, Any], lm):
def add_tokenizer_info(storage: dict[str, Any], lm):
if getattr(lm, "tokenizer", False):
try:
tokenizer_info = {
@@ -142,7 +152,7 @@ def add_tokenizer_info(storage: Dict[str, Any], lm):
logger.debug(
f"Logging detailed tokenizer info failed with {err}, skipping..."
)
# seems gguf and textsynth do not have tokenizer
# seems gguf and textsynth do not have a tokenizer
else:
logger.debug(
"LM does not have a 'tokenizer' attribute, not logging tokenizer metadata to results."
lm_eval/models/__init__.py
+35 -25
View File
@@ -1,35 +1,55 @@
# Models are now lazily loaded via the registry system
# No need to import them all at once - they're loaded on demand
"""Model implementations for lm_eval.
Models are lazily loaded via the registry system to improve startup performance.
Usage
-----
For programmatic access, use the registry:
from lm_eval.api.registry import get_model
model_cls = get_model("hf")
model = model_cls(pretrained="gpt2")
For direct imports (e.g., subclassing), use explicit module paths:
from lm_eval.models.huggingface import HFLM
from lm_eval.models.vllm_causallms import VLLM
Adding New Models
-----------------
1. Create your model class in a new file under lm_eval/models/
2. Use the @register_model decorator on your class
3. Add an entry to MODEL_MAPPING below for lazy discovery
"""
# Define model mappings for lazy registration
MODEL_MAPPING = {
"anthropic-completions": "lm_eval.models.anthropic_llms:AnthropicLM",
"anthropic-chat": "lm_eval.models.anthropic_llms:AnthropicChatLM",
"anthropic-chat-completions": "lm_eval.models.anthropic_llms:AnthropicCompletionsLM",
"local-completions": "lm_eval.models.openai_completions:LocalCompletionsAPI",
"local-chat-completions": "lm_eval.models.openai_completions:LocalChatCompletion",
"openai-completions": "lm_eval.models.openai_completions:OpenAICompletionsAPI",
"openai-chat-completions": "lm_eval.models.openai_completions:OpenAIChatCompletion",
"anthropic-completions": "lm_eval.models.anthropic_llms:AnthropicLM",
"dummy": "lm_eval.models.dummy:DummyLM",
"gguf": "lm_eval.models.gguf:GGUFLM",
"ggml": "lm_eval.models.gguf:GGUFLM",
"hf-audiolm-qwen": "lm_eval.models.hf_audiolm:HFAudioLM",
"steered": "lm_eval.models.hf_steered:SteeredHF",
"hf-multimodal": "lm_eval.models.hf_vlms:HFMultimodalLM",
"hf-auto": "lm_eval.models.huggingface:HFLM",
"gguf": "lm_eval.models.gguf:GGUFLM",
"hf": "lm_eval.models.huggingface:HFLM",
"hf-audiolm-qwen": "lm_eval.models.hf_audiolm:HFAudioLM",
"hf-auto": "lm_eval.models.huggingface:HFLM",
"hf-multimodal": "lm_eval.models.hf_vlms:HFMultimodalLM",
"huggingface": "lm_eval.models.huggingface:HFLM",
"watsonx_llm": "lm_eval.models.ibm_watsonx_ai:IBMWatsonxAI",
"ipex": "lm_eval.models.optimum_ipex:IPEXForCausalLM",
"local-chat-completions": "lm_eval.models.openai_completions:LocalChatCompletion",
"local-completions": "lm_eval.models.openai_completions:LocalCompletionsAPI",
"mamba_ssm": "lm_eval.models.mamba_lm:MambaLMWrapper",
"nemo_lm": "lm_eval.models.nemo_lm:NeMoLM",
"neuronx": "lm_eval.models.neuron_optimum:NeuronModelForCausalLM",
"ipex": "lm_eval.models.optimum_ipex:IPEXForCausalLM",
"openai-chat-completions": "lm_eval.models.openai_completions:OpenAIChatCompletion",
"openai-completions": "lm_eval.models.openai_completions:OpenAICompletionsAPI",
"openvino": "lm_eval.models.optimum_lm:OptimumLM",
"sglang": "lm_eval.models.sglang_causallms:SGLANG",
"sglang-generate": "lm_eval.models.sglang_generate_API:SGAPI",
"steered": "lm_eval.models.hf_steered:SteeredHF",
"textsynth": "lm_eval.models.textsynth:TextSynthLM",
"vllm": "lm_eval.models.vllm_causallms:VLLM",
"vllm-vlm": "lm_eval.models.vllm_vlms:VLLM_VLM",
"watsonx_llm": "lm_eval.models.ibm_watsonx_ai:IBMWatsonxAI",
}
@@ -48,13 +68,3 @@ def _register_all_models():
_register_all_models()
__all__ = ["MODEL_MAPPING"]
try:
# enable hf hub transfer if available
import hf_transfer # type: ignore # noqa
import huggingface_hub.constants # type: ignore
huggingface_hub.constants.HF_HUB_ENABLE_HF_TRANSFER = True
except ImportError:
pass
lm_eval/tasks/__init__.py
+18 -16
View File
@@ -2,9 +2,10 @@ import collections
import inspect
import logging
import os
from collections.abc import Mapping
from functools import partial
from pathlib import Path
from typing import Dict, List, Mapping, Optional, Union
from typing import Dict, List, Optional, Union
from lm_eval import utils
from lm_eval.api.group import ConfigurableGroup, GroupConfig
@@ -25,10 +26,10 @@ class TaskManager:
def __init__(
self,
verbosity: Optional[str] = None,
include_path: Optional[Union[str, List]] = None,
verbosity: str | None = None,
include_path: str | list | None = None,
include_defaults: bool = True,
metadata: Optional[dict] = None,
metadata: dict | None = None,
) -> None:
if verbosity is not None:
utils.setup_logging(verbosity)
@@ -57,7 +58,7 @@ class TaskManager:
def initialize_tasks(
self,
include_path: Optional[Union[str, List]] = None,
include_path: str | list | None = None,
include_defaults: bool = True,
) -> dict[str, dict]:
"""Creates a dictionary of tasks indexes.
@@ -257,9 +258,9 @@ class TaskManager:
def _load_individual_task_or_group(
self,
name_or_config: Optional[Union[str, dict]] = None,
parent_name: Optional[str] = None,
update_config: Optional[dict] = None,
name_or_config: str | dict | None = None,
parent_name: str | None = None,
update_config: dict | None = None,
) -> Mapping:
def _load_task(config, task):
if "include" in config:
@@ -411,7 +412,7 @@ class TaskManager:
group_name: dict(collections.ChainMap(*map(fn, reversed(subtask_list))))
}
def load_task_or_group(self, task_list: Optional[Union[str, list]] = None) -> dict:
def load_task_or_group(self, task_list: str | list | None = None) -> dict:
"""Loads a dictionary of task objects from a list
:param task_list: Union[str, list] = None
@@ -433,7 +434,7 @@ class TaskManager:
)
return all_loaded_tasks
def load_config(self, config: Dict):
def load_config(self, config: dict):
return self._load_individual_task_or_group(config)
def _get_task_and_group(self, task_dir: str):
@@ -551,7 +552,7 @@ class TaskManager:
return tasks_and_groups
def get_task_name_from_config(task_config: Dict[str, str]) -> str:
def get_task_name_from_config(task_config: dict[str, str]) -> str:
if "task" in task_config:
return task_config["task"]
if "dataset_name" in task_config:
@@ -601,8 +602,8 @@ def _check_duplicates(task_dict: dict) -> None:
def get_task_dict(
task_name_list: Union[str, List[Union[str, Dict, Task]]],
task_manager: Optional[TaskManager] = None,
task_name_list: str | list[str | dict | Task],
task_manager: TaskManager | None = None,
):
"""Creates a dictionary of task objects from either a name of task, config, or prepared Task object.
@@ -681,12 +682,13 @@ def get_task_dict(
yaml_path = Path(yaml_path)
lm_eval_tasks_path = Path(__file__).parent
try:
relative_yaml_path = yaml_path.relative_to(lm_eval_tasks_path)
display_path = yaml_path.relative_to(lm_eval_tasks_path)
except ValueError:
relative_yaml_path = yaml_path
# Path is outside lm_eval/tasks (e.g., from include_path)
display_path = yaml_path
pad = " " * indent
eval_logger.info(f"{pad}Task: {task_name} ({relative_yaml_path})")
eval_logger.info(f"{pad}Task: {task_name} ({display_path})")
# NOTE: Only nicely logs:
# 1/ group
lm_eval/utils.py
+49 -42
View File
@@ -9,10 +9,11 @@ import logging
import os
import re
import threading
from collections.abc import Callable, Generator
from dataclasses import asdict, is_dataclass
from itertools import islice
from pathlib import Path
from typing import Any, Callable, Generator, List, Optional, Tuple
from typing import Any
import numpy as np
import requests
@@ -28,7 +29,15 @@ HIGHER_IS_BETTER_SYMBOLS = {
}
def wrap_text(string: str, width: int = 140, **kwargs) -> Optional[str]:
def is_torch_available() -> bool:
return importlib.util.find_spec("torch") is not None
def is_transformers_available() -> bool:
return importlib.util.find_spec("transformers") is not None
def wrap_text(string: str, width: int = 140, **kwargs) -> str | None:
"""
Wraps the given string to the specified width.
"""
@@ -49,8 +58,7 @@ def setup_logging(verbosity=logging.INFO):
# Configure the root logger
class CustomFormatter(logging.Formatter):
def format(self, record):
if record.name.startswith("lm_eval."):
record.name = record.name[len("lm_eval.") :]
record.name = record.name.removeprefix("lm_eval.")
return super().format(record)
formatter = CustomFormatter(
@@ -110,7 +118,7 @@ def escaped_split(text, sep_char, maxsplit=-1):
return text
maxsplit = max(0, maxsplit)
return re.split(r"(?<!\\)" + sep_char, text, maxsplit)
return re.split(r"(?<!\\)" + sep_char, text, maxsplit=maxsplit)
def handle_arg_string(arg):
@@ -127,7 +135,7 @@ def handle_arg_string(arg):
def handle_non_serializable(o):
if isinstance(o, np.int64) or isinstance(o, np.int32):
if isinstance(o, (np.int64, np.int32)):
return int(o)
elif isinstance(o, set):
return list(o)
@@ -147,7 +155,7 @@ def sanitize_list(sub):
return str(sub)
def simple_parse_args_string(args_string: Optional[str]) -> dict:
def simple_parse_args_string(args_string: str | None) -> dict:
"""
Parses something like
args1=val1,arg2=val2
@@ -227,7 +235,7 @@ def sanitize_model_name(model_name: str) -> str:
"""
Given the model name, returns a sanitized version of it.
"""
return re.sub(r"[\"<>:/\|\\?\*\[\]]+", "__", model_name)
return re.sub(r"[\"<>:/|\\?*\[\]]+", "__", model_name)
def sanitize_task_name(task_name: str) -> str:
@@ -237,21 +245,21 @@ def sanitize_task_name(task_name: str) -> str:
return re.sub(r"\W", "_", task_name)
def get_latest_filename(filenames: List[str]) -> str:
def get_latest_filename(filenames: list[str]) -> str:
"""
Given a list of filenames, returns the filename with the latest datetime.
"""
return max(filenames, key=lambda f: get_file_datetime(f))
def get_results_filenames(filenames: List[str]) -> List[str]:
def get_results_filenames(filenames: list[str]) -> list[str]:
"""
Extracts filenames that correspond to aggregated results.
"""
return [f for f in filenames if "/results_" in f and ".json" in f]
def get_sample_results_filenames(filenames: List[str]) -> List[str]:
def get_sample_results_filenames(filenames: list[str]) -> list[str]:
"""
Extracts filenames that correspond to sample results.
"""
@@ -259,8 +267,8 @@ def get_sample_results_filenames(filenames: List[str]) -> List[str]:
def get_rolling_token_windows(
token_list: List[int], prefix_token: int, max_seq_len: int, context_len: int
) -> Generator[Tuple[List[int], List[int]], None, None]:
token_list: list[int], prefix_token: int, max_seq_len: int, context_len: int
) -> Generator[tuple[list[int], list[int]], None, None]:
"""
- context_len allows for a rolling window context, allowing each prediction window to potentially
condition on some context
@@ -302,8 +310,8 @@ def get_rolling_token_windows(
def make_disjoint_window(
pair: Tuple[List[int], List[int]],
) -> Tuple[List[int], List[int]]:
pair: tuple[list[int], list[int]],
) -> tuple[list[int], list[int]]:
"""Takes output from get_rolling_token_windows and makes the context not overlap with the continuation"""
a, b = pair
return a[: len(a) - (len(b) - 1)], b
@@ -322,7 +330,7 @@ class EnhancedJSONEncoder(json.JSONEncoder):
class Reorderer:
def __init__(self, arr: List[Any], fn: Callable) -> None:
def __init__(self, arr: list[Any], fn: Callable) -> None:
"""Reorder an array according to some function
Args:
@@ -359,7 +367,7 @@ class Reorderer:
res = [None] * self.size
cov = [False] * self.size
for (inds, _), v in zip(self.arr, newarr):
for (inds, _), v in zip(self.arr, newarr, strict=True):
for ind in inds:
res[ind] = v
cov[ind] = True
@@ -422,11 +430,11 @@ def make_table(result_dict, column: str = "results", sort_results: bool = False)
hib = HIGHER_IS_BETTER_SYMBOLS.get(higher_is_better.get(m), "")
v = "%.4f" % v if isinstance(v, float) else v
v = f"{v:.4f}" if isinstance(v, float) else v
if m + "_stderr" + "," + f in dic:
se = dic[m + "_stderr" + "," + f]
se = " N/A" if se == "N/A" else "%.4f" % se
se = " N/A" if se == "N/A" else f"{se:.4f}"
values.append([k, version, f, n, m, hib, v, "±", se])
else:
values.append([k, version, f, n, m, hib, v, "", ""])
@@ -564,7 +572,7 @@ def create_iterator(raw_iterator, *, rank=0, world_size=1, limit=None):
def weighted_f1_score(items):
from sklearn.metrics import f1_score
unzipped_list = list(zip(*items))
unzipped_list = list(zip(*items, strict=True))
golds = unzipped_list[0]
preds = unzipped_list[1]
fscore = f1_score(golds, preds, average="weighted")
@@ -637,8 +645,8 @@ class RemoteTokenizer:
base_url: str,
timeout: int = 30,
verify_certificate: bool = True,
ca_cert_path: Optional[str] = None,
auth_token: Optional[str] = None,
ca_cert_path: str | None = None,
auth_token: str | None = None,
max_retries: int = 3,
):
self.timeout = timeout
@@ -708,34 +716,34 @@ class RemoteTokenizer:
return self._tokenizer_info
@property
def eos_token(self) -> Optional[str]:
def eos_token(self) -> str | None:
return self.tokenizer_info.get("eos_token")
@property
def bos_token(self) -> Optional[str]:
def bos_token(self) -> str | None:
return self.tokenizer_info.get("bos_token")
@property
def pad_token(self) -> Optional[str]:
def pad_token(self) -> str | None:
return self.tokenizer_info.get("pad_token")
@property
def eos_token_id(self) -> Optional[int]:
if self.eos_token is None:
def eos_token_id(self) -> int | None:
if (eos := self.eos_token) is None:
return None
return self.encode(self.eos_token)[0]
return self.encode(eos)[0]
@property
def bos_token_id(self) -> Optional[int]:
if self.bos_token is None:
def bos_token_id(self) -> int | None:
if (bos := self.bos_token) is None:
return None
return self.encode(self.bos_token)[0]
return self.encode(bos)[0]
@property
def eot_token(self) -> Optional[int]:
def eot_token(self) -> int | None:
return self.eos_token_id
def encode(self, text: str) -> List[int]:
def encode(self, text: str) -> list[int]:
url = f"{self.base_url}/tokenize"
payload = {"prompt": text, "add_special_tokens": False}
resp = self._request_with_retries("POST", url, json=payload)
@@ -744,7 +752,7 @@ class RemoteTokenizer:
raise RuntimeError("Malformed response from /tokenize endpoint.")
return tokens
def decode(self, tokens: List[int]) -> str:
def decode(self, tokens: list[int]) -> str:
url = f"{self.base_url}/detokenize"
payload = {"tokens": tokens}
resp = self._request_with_retries("POST", url, json=payload)
@@ -753,7 +761,7 @@ class RemoteTokenizer:
raise RuntimeError("Malformed response from /detokenize endpoint.")
return prompt
def batch_decode(self, tokens_list: List[List[int]]) -> List[str]:
def batch_decode(self, tokens_list: list[list[int]]) -> list[str]:
return [self.decode(tokens) for tokens in tokens_list]
def apply_chat_template(
@@ -778,8 +786,8 @@ def check_remote_tokenizer_support(
base_url: str,
timeout: int = 5,
verify_certificate: bool = True,
ca_cert_path: Optional[str] = None,
auth_token: Optional[str] = None,
ca_cert_path: str | None = None,
auth_token: str | None = None,
max_retries: int = 3,
) -> bool:
"""
@@ -836,13 +844,12 @@ def check_remote_tokenizer_support(
if not resp:
return False
tokens = resp.json().get("tokens")
if not isinstance(tokens, list):
return False
return True
return isinstance(tokens, list)
def set_torch_seed(seed: int):
import torch
if is_torch_available():
import torch
torch.manual_seed(seed)
torch.manual_seed(seed)
pyproject.toml
+34 -17
View File
@@ -122,33 +122,46 @@ plugins.md028.enabled = false # no-blanks-blockquote
plugins.md029.allow_extended_start_values = true # ol-prefix
plugins.md034.enabled = false # no-bare-urls
[tool.ruff]
lint.extend-select = [
"I", # isort
"UP", # pyupgrade
"E", # pycodestyle errors
"C419", # unnecessary-comprehension-in-call
"F", # pyflakes
[tool.ruff.lint]
extend-select = [
"B", # flake8-bugbear
"SIM", # flake8-simplify
"RUF034", # useless-if-else
"W605", # invalid-escape-sequence
"C419", # unnecessary-comprehension-in-call
"D", # docstyle
"E", # pycodestyle errors
"F", # pyflakes
"FURB", # refurb
"I", # isort
"TC", # typecheck
"RUF034", # useless-if-else
"SIM", # flake8-simplify
"UP", # pyupgrade
"W605", # invalid-escape-sequence
]
lint.fixable = [
"I001", # unsorted-imports
fixable = [
"F401", # unused-import
"I001", # unsorted-imports
"UP", # pyupgrade fixes
]
lint.ignore = [
"E402", # module-import-not-at-top-of-file
"E731", # lambda-assignment
"E501", # line-too-long
ignore = [
# docstrings
"D10", # missing-docstring
# docstrings
"D212", # start first line
"D415", # terminal punctuation
"D205", # missing blank line
"D417", # docs: undocumented param
"D200", # unnecessary line
"D202", # blank-line-after-function
"E111", # indentation-with-invalid-multiple
"E114", # indentation-with-invalid-multiple-comment
"E117", # over-indented
"E741", # ambiguous-variable-name
"E402", # module-import-not-at-top-of-file
"E501", # line-too-long
"E701", # multiple-statements-on-one-line-colon
"E731", # lambda-assignment
"E741", # ambiguous-variable-name
"SIM118", # dict-in-keys
]
[tool.ruff.lint.extend-per-file-ignores]
@@ -163,3 +176,7 @@ lint.ignore = [
combine-as-imports = true
known-first-party = ["lm_eval"]
lines-after-imports = 2
[tool.ruff.lint.pydocstyle]
convention = "google"
ignore-var-parameters = true
tests/models/test_bos_handling.py
+1 -1
View File
@@ -580,7 +580,7 @@ class TestEdgeCases:
assert result == expected
def test_vllm_empty_input(self):
"""vLLM should handle empty input gracefully."""
"""vLLM should handle empty input gracefully.""" # noqa: D403
from lm_eval.models.vllm_causallms import VLLM
mock_vllm = Mock()