* feat(registry): runtime plugin registration for model backends
Add a discovery layer so external packages can contribute model backends to
lm-eval without editing its source, and have them behave exactly like upstream
backends. The Registry class already supported EntryPoint placeholders, but
nothing ever scanned installed distributions for them.
Two mechanisms:
- Entry points: packages declaring an `lm_eval.models` entry point are
discovered automatically (lazy, zero-config) when a model name is resolved.
- Explicit import: `--plugins mod` / `LM_EVAL_PLUGINS` env / `simple_evaluate(plugins=...)`
import named modules so their `@register_*` decorators run, for local or
unpublished components.
Also fixes Registry collision handling to upgrade an EntryPoint placeholder to
its decorated class (the natural case where a plugin declares both an entry
point and `@register_model`), guarding EntryPoint.__eq__ which raises on
mismatched types.
Co-Authored-By: Claude <[email protected]>
* feat(registry): plugin discovery for filters, metrics, and aggregations
Extend the entry-point plugin mechanism (added for models) to the other
Registry-backed components. External packages can now contribute:
- filters via the `lm_eval.filters` entry-point group
- metrics via `lm_eval.metrics`
- aggregations via `lm_eval.aggregations`
get_filter/get_metric/get_aggregation discover installed entry points lazily.
Because @register_metric populates the higher_is_better and metric-aggregation
registries as a decorator side effect, is_higher_better and get_metric_aggregation
now force discovery + materialization of the named metric so plugin metrics carry
their metadata even when those getters are called standalone.
Generalize the Registry placeholder→concrete upgrade to any object with
module/name (functions, not just classes), so metric/aggregation plugins that
declare both an entry point and a decorator upgrade cleanly instead of colliding.
Co-Authored-By: Claude <[email protected]>
* remove plugins from `simple_evaluate`
* fix tests
* fix docs
* add to news
---------
Co-authored-by: Baber Abbasi <[email protected]>
* feat(registry): runtime plugin registration for model backends
Add a discovery layer so external packages can contribute model backends to
lm-eval without editing its source, and have them behave exactly like upstream
backends. The Registry class already supported EntryPoint placeholders, but
nothing ever scanned installed distributions for them.
Two mechanisms:
- Entry points: packages declaring an `lm_eval.models` entry point are
discovered automatically (lazy, zero-config) when a model name is resolved.
- Explicit import: `--plugins mod` / `LM_EVAL_PLUGINS` env / `simple_evaluate(plugins=...)`
import named modules so their `@register_*` decorators run, for local or
unpublished components.
Also fixes Registry collision handling to upgrade an EntryPoint placeholder to
its decorated class (the natural case where a plugin declares both an entry
point and `@register_model`), guarding EntryPoint.__eq__ which raises on
mismatched types.
Co-Authored-By: Claude <[email protected]>
* Address review: drop plugin env var, fix dead except and EntryPoint extras
- Remove LM_EVAL_PLUGINS env var from simple_evaluate: importing third-party
code via an env var is a security risk and makes the function depend on
hidden env state. --plugins / --config already cover the use case.
- load_plugins: lazy registration validates nothing, so the try/except could
never fire. Move real error handling to the actual failure site
(_materialise_placeholder / EntryPoint.load) so a broken plugin logs the
offending name and aborts on access.
- Registry collision handling: use EntryPoint.module/.attr instead of .value
so entry points declared with extras (e.g. "my_pkg:MyLM [gpu]") upgrade
correctly instead of comparing against the bracketed value.
Co-Authored-By: Claude <[email protected]>
* style: satisfy ruff 0.16.3 in touched files (registry, evaluator)
The pre-commit ruff pin was bumped to 0.16.3 on main, which lints whole
changed files. registry.py and evaluator.py carried pre-existing preview-rule
violations that only surface now that this PR touches them:
- move Callable import into TYPE_CHECKING block (TC003)
- sort __all__ (RUF022)
- bare `raise` instead of `raise e` in get_filter (TRY201)
- `float` instead of `int | float` for simple_evaluate limit (PYI041)
- noqa: BLE001 on intentional defensive Exception catches
Co-Authored-By: Claude <[email protected]>
---------
Co-authored-by: Claude <[email protected]>
* feat(models): add cross-platform onnxruntime-genai backend + refactor winml
Add a cross-platform `onnxruntime-genai` backend that runs ONNX LLMs exported
by the ONNX Runtime GenAI Model Builder on CPU/CUDA/DirectML/WebGPU/AMD-NPU via
the `og.Config` provider API. All lm-eval logic (single-pass log-likelihood,
multiple-choice, rolling perplexity, generation) lives in
`ONNXRuntimeGenAILM(TemplateLM)`, following the `OptimumLM(HFLM)` single-file
subclass precedent.
Refactor `winml` into a thin `WindowsML(ONNXRuntimeGenAILM)` subclass that
overrides only execution-provider selection, inheriting the shared scoring
core. This removes the previous winml quality debt: the per-token
teacher-forcing loop (now a single `get_output("logits")` pass), the direct
`loglikelihood` override that bypassed `TemplateLM` token handling, and the
silent `(0.0, False)`-on-exception path that corrupted scores.
Add a shared parametrized test suite (unit tests for the scoring math with a
fake `_forward_logits`, plus CPU smoke tests over hellaswag/wikitext/gsm8k
using a tiny Model Builder fixture), packaging extra, and docs. The test is
added to the CI `--ignore` list since the ONNX runtimes are not installed in CI.
Part of QUARK-606 Phase 1. The raw `onnxruntime` backend, cross-backend parity
CI, OpenAI-compatible server, and Quark migration are follow-up stages.
Co-Authored-By: Claude <[email protected]>
* docs: keep model_guide subclassing tip generic
Trim the subclassing note to the pre-existing OptimumLM example instead of
documenting the onnxruntime-genai backend internals, which belong in the
backend's own docstrings/README rather than the general model-authoring guide.
Co-Authored-By: Claude <[email protected]>
* test: split winml tests into their own module
Move winml-specific coverage (registration, subclass relationship, non-Windows
CPU fallback) into tests/models/test_winml.py, leaving test_onnxruntime_genai.py
focused on the genai backend and its scoring math. Keeps each backend's tests
self-contained and adds the new file to the CI ignore list.
Co-Authored-By: Claude <[email protected]>
* fix(onnxruntime-genai): honor genai_config.json providers by default
execution_provider now defaults to None, meaning "use the providers the export
declares in genai_config.json" — og.Config(model_dir) seeds its provider list
from that file, and _select_ep no longer calls clear_providers() unless the
user explicitly asks for a provider. Previously the "cpu" default cleared the
list and appended nothing, silently downgrading a CUDA/DML/NPU export to CPU.
Pass execution_provider=cpu to force CPU, or a provider name to override.
winml pins its historical max_length/max_gen_toks (4096) so existing runs are
numerically unchanged, while the cross-platform backend keeps model-aware
defaults (context_length from the config; max_gen_toks=256).
Add regression tests for the three _select_ep branches and winml's defaults.
Co-Authored-By: Claude <[email protected]>
* pacify pre-commmit
---------
Co-authored-by: Claude <[email protected]>
Co-authored-by: Baber Abbasi <[email protected]>
* Add think_end_token and enable_thinking validation - auto split dev branch
Co-Authored-By: Claude Opus 4.6 <[email protected]>
* precisions and doc
* cleanup and test
* fix(huggingface): enable_thinking never set
---------
Co-authored-by: Claude Opus 4.6 <[email protected]>
Co-authored-by: Baber Abbasi <[email protected]>
* added headers and custom model name | fixed bug with trust_remote_code param
* linting
* removed custom model name | changed headers override
* add `header` to base TemplateAPI
* nit
---------
Co-authored-by: Baber <[email protected]>
* changed source of eval_logger
* allow eval_logger to be set from args
* removed verbosity arg from non-main methods
* fix logging
* pre-commit
* set verbosity in eval logger
* replace utils.eval_logger
* fix logging in main
* add logging to docs
* add logging message
* nit
* add logging to docs
* refactor setup_logging to utils
---------
Co-authored-by: Baber <[email protected]>
* add greoup_config arg
* add a group config that allows disabling table for group score and group aggregate in general
* fixed size configuration
* adjust config
* add group config
* adjust mmlu to use group_config
* fixed args input in aggregate_subtask_metrics
* fixed issues related to printing alias of group and updated yaml
* update all mmlu variants to include group_config
* edit format
* modify mmlu tasks
* adjust group to also be a configurable group
* add configurable group
* simplify get_task_list
* adjust group scoring with using ConfigurableGroup
* adjust args
* update mmlu
* update mmlu
* update to work with new group and task configuration
* readd group_agg
* readd files
* move prepare_print_tasks to evaluator_utils
* sort set to False by default, fix predict_only arg
* add version for groups
* reversed task list
* update additional condition when loading a group in a group yaml
* update truthfulqa
* add description regarding tags replacing group
* replace group to tag
* fixed conditional statement
* remove warning
* update loading of task group and newly added tags
* reformat with pre-commit
* fixed info log
* update
* fix bug
* fix bug
* use task id to differentiate tasks
* convert all groups to configurable groups
* use task_id
* reformat
* add task_id for python tasks as well
* add task_id for python tasks as well
* add task_id for python tasks as well
* revert truthfulqa
* revert mmlu tasks
* new mmlu config
* new group config parameter `tag_to_task`
* Update truthfulqa_mc2.yaml
* reformate
* add _process_group_config
* adjust task_id
* add get_subtask_list function to get proper subtask list
* group config to_dict update
* remove tag check
* update mmlu
* fix config passing issues
* add test yaml
* format fix
* add documentation
* corner case for single tag being called
* fix indentation
* formatting
* update all mmlu variants
* Update docs/task_guide.md
Co-authored-by: Hailey Schoelkopf <[email protected]>
* remove group_alias
* Update docs/task_guide.md
Co-authored-by: Hailey Schoelkopf <[email protected]>
* remove version for metadata
* Update docs/task_guide.md
Co-authored-by: Hailey Schoelkopf <[email protected]>
* update mmlu/
* removed " " in make_table
* change how aggregate_metric is loaded
* change how aggregate_metric is loaded
* update aggregate_metric arg
* update format
* update format
* some docs fixes
* add groups for agieval, aexams, aclue
* add more explicit aggregation groups
* add more groupings / tags distinctions
* add more groupings
* more groupings
* add many explicit group configs
* add many explicit group configs
* add more explicit group configs
* add more explicit group configs
* add more error msgs, agg_metric -> agg_metric_list
* some docs updates
* update task_id to be updateable and uses group:task format
* make KMMLU a tag for now
* update docs
* don't duplicate task names
* fix merge conflicts?
* giving this a try
* clean up diff
* switch mmlu variants over to using
* don't use to-be-deprecated group: config field in overview notebook
* Python tasks which subclass ConfigurableTask now run
* update mmlu
* pre-commit format
* fixed sorting for multi-level printing
* move group api to separate file
* fix bbh aggregation filter usage
* track api/group.py
* adjust group and tags loading
* make explicit group configs for leaderboard and other newer tasks
* fix arabicmmlu
* update
* change arabicmmlu template name???
* update group alias
* fix printing bugs
* check table printing is correct ; update tests
* use mmlu_stem to have a group included in print tests
---------
Co-authored-by: Hailey Schoelkopf <[email protected]>
Co-authored-by: haileyschoelkopf <[email protected]>
* init test 1
* fix
* this format seems to be working - need to update all other tasks with the new format
* bbh with few shot format
* fix fewshot bbh
* add mmlu flan cot
* samples of cot
* kmmlu
* fix gsm8k
* update keys for mmlu
* minerva math
* bbh
* fix
* fix samples
* small fixes to templates
* last prompt format change
* fixing prompt
* fixed minerva math format
* rm accidental commited file
* added doc for few shot samples
* Update lm_eval/loggers/evaluation_tracker.py
* Update lm_eval/loggers/evaluation_tracker.py
* Update docs/new_task_guide.md
Co-authored-by: Hailey Schoelkopf <[email protected]>
* added check in sampler per code review
* added the system from a function, plus an example in minerva math
* style
* Apply suggestions from code review
Co-authored-by: Hailey Schoelkopf <[email protected]>
* fix unit tests 1
* forcing use of test split
---------
Co-authored-by: Hailey Schoelkopf <[email protected]>
* Update interface.md
* fix: make caching reqs always work with accelerate launch
* remove stale task migration checklist
* remove deprecation warnings
* make informative TypeErrors for get_task_dict
* bump version metadata
* fix num_fewshot printing bug
* add fewshot value to cache key
