mirror of
https://github.com/EleutherAI/lm-evaluation-harness.git
synced 2026-10-11 21:59:53 +00:00
b954108c9b
* feat(registry): runtime plugin registration for model backends Add a discovery layer so external packages can contribute model backends to lm-eval without editing its source, and have them behave exactly like upstream backends. The Registry class already supported EntryPoint placeholders, but nothing ever scanned installed distributions for them. Two mechanisms: - Entry points: packages declaring an `lm_eval.models` entry point are discovered automatically (lazy, zero-config) when a model name is resolved. - Explicit import: `--plugins mod` / `LM_EVAL_PLUGINS` env / `simple_evaluate(plugins=...)` import named modules so their `@register_*` decorators run, for local or unpublished components. Also fixes Registry collision handling to upgrade an EntryPoint placeholder to its decorated class (the natural case where a plugin declares both an entry point and `@register_model`), guarding EntryPoint.__eq__ which raises on mismatched types. Co-Authored-By: Claude <[email protected]> * feat(registry): plugin discovery for filters, metrics, and aggregations Extend the entry-point plugin mechanism (added for models) to the other Registry-backed components. External packages can now contribute: - filters via the `lm_eval.filters` entry-point group - metrics via `lm_eval.metrics` - aggregations via `lm_eval.aggregations` get_filter/get_metric/get_aggregation discover installed entry points lazily. Because @register_metric populates the higher_is_better and metric-aggregation registries as a decorator side effect, is_higher_better and get_metric_aggregation now force discovery + materialization of the named metric so plugin metrics carry their metadata even when those getters are called standalone. Generalize the Registry placeholder→concrete upgrade to any object with module/name (functions, not just classes), so metric/aggregation plugins that declare both an entry point and a decorator upgrade cleanly instead of colliding. Co-Authored-By: Claude <[email protected]> * remove plugins from `simple_evaluate` * fix tests * fix docs * add to news --------- Co-authored-by: Baber Abbasi <[email protected]>
Eval Harness Documentation
Welcome to the docs for the LM Evaluation Harness!
Table of Contents
- To learn about the public interface of the library, as well as how to evaluate via the command line or as integrated into an external library, see the Interface.
- To learn how to add a new library, API, or model type to the library, as well as a quick explainer on the types of ways to evaluate an LM, see the Model Guide.
- For an extended description of how to extend the library to new model classes served over an API, see the API Guide.
- To ship a model, filter, metric or aggregation from your own package instead of the
lm-evalsource tree, see the Plugin Guide.
- For a crash course on adding new tasks to the library, see our New Task Guide.
- To learn more about pushing the limits of task configuration that the Eval Harness supports, see the Task Configuration Guide.