diff --git a/README.md b/README.md index bdf7de36c..d9d9251b0 100644 --- a/README.md +++ b/README.md @@ -5,6 +5,7 @@ --- ## Latest News 📣 +- [2026/09] **Plugins**: model backends, filters, metrics and aggregations can now be registered from your own package instead of forking — declare an `lm_eval.*` entry point for zero-config discovery, or point `--plugins` at a local module. See the [Plugin Guide](./docs/plugins.md). - [2025/12] **CLI refactored** with subcommands (`run`, `ls`, `validate`) and YAML config file support via `--config`. See the [CLI Reference](./docs/interface.md) and [Configuration Guide](./docs/config_files.md). - [2025/12] **Lighter install**: Base package no longer includes `transformers`/`torch`. Install model backends separately: `pip install lm_eval[hf]`, `lm_eval[vllm]`, etc. - [2025/07] Added `think_end_token` arg to `hf` (token/str), `vllm` and `sglang` (str) for stripping CoT reasoning traces from models that support it. diff --git a/docs/README.md b/docs/README.md index f040eabae..209d5308a 100644 --- a/docs/README.md +++ b/docs/README.md @@ -7,5 +7,6 @@ Welcome to the docs for the LM Evaluation Harness! * To learn about the public interface of the library, as well as how to evaluate via the command line or as integrated into an external library, see the [Interface](./interface.md). * To learn how to add a new library, API, or model type to the library, as well as a quick explainer on the types of ways to evaluate an LM, see the [Model Guide](./model_guide.md). * For an extended description of how to extend the library to new model classes served over an API, see the [API Guide](./API_guide.md). + * To ship a model, filter, metric or aggregation from your own package instead of the `lm-eval` source tree, see the [Plugin Guide](./plugins.md). * For a crash course on adding new tasks to the library, see our [New Task Guide](./new_task_guide.md). * To learn more about pushing the limits of task configuration that the Eval Harness supports, see the [Task Configuration Guide](./task_guide.md). diff --git a/docs/interface.md b/docs/interface.md index 9f0d63da8..c78233d3e 100644 --- a/docs/interface.md +++ b/docs/interface.md @@ -119,6 +119,7 @@ lm-eval run --config my_config.yaml --tasks mmlu | Argument | Description | |----------|-------------| | `--include_path` | Additional directory containing external task YAML files. | +| `--plugins` | Modules to import before evaluation so their `@register_*` decorators run (models, filters, metrics, aggregations). See the [Plugin Guide](./plugins.md). | ### Logging and Tracking diff --git a/docs/model_guide.md b/docs/model_guide.md index 88f886779..8c72ccf36 100644 --- a/docs/model_guide.md +++ b/docs/model_guide.md @@ -105,33 +105,23 @@ Using this decorator results in the class being added to an accounting of the us ### Registering a backend from an external package (plugins) If your backend lives in your **own** package rather than in the `lm-eval` source -tree, you do not need to fork or edit `lm-eval`. There are two ways to make -`lm-eval --model ` resolve it at runtime. +tree, you do not need to fork or edit `lm-eval`. Declare an `lm_eval.models` entry +point and `lm-eval --model ` resolves it at runtime: -**1. Entry points (zero-config, recommended for published packages).** Declare an -`lm_eval.models` entry point in your package's `pyproject.toml`, mapping the model -name to `module:Class`: +# in your_project/pyproject.toml ```toml [project.entry-points."lm_eval.models"] my-backend = "my_pkg.models:MyBackendLM" ``` -Once your package is `pip install`ed, `lm-eval --model my-backend ...` works with no -extra flags — `lm-eval` discovers installed entry points automatically. The plugin -module is imported lazily, only when the model is actually requested. (Your class can -still also carry a `@register_model("my-backend")` decorator; the two are reconciled -automatically.) - -**2. Explicit import (for local or unpublished modules).** Point `lm-eval` at one or -more modules to import before evaluation, so their `@register_model` decorators run: - ```bash -lm-eval run --model my-backend --plugins my_pkg.models --tasks hellaswag +lm-eval run --model my-backend --tasks hellaswag ``` -The same `plugins=[...]` argument is available on `lm_eval.simple_evaluate(...)` for -programmatic use. +For unpublished modules, `--plugins my_pkg.models` imports them so their +`@register_model` decorator runs. Filters, metrics and aggregations work the same +way — see the [Plugin Guide](./plugins.md) for the full picture. ## Testing diff --git a/docs/new_task_guide.md b/docs/new_task_guide.md index 66889f85f..bb3532444 100644 --- a/docs/new_task_guide.md +++ b/docs/new_task_guide.md @@ -315,7 +315,7 @@ metric_list: `aggregation` and `higher_is_better` can optionally be left out to default to the manually-set defaults if using a natively supported metric, otherwise it must be defined explicitly (for example, when using a custom metric implemented as a function). -For a full list of natively supported metrics and aggregation functions see [`docs/task_guide.md`](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/task_guide.md). All metrics supported in [HuggingFace Evaluate](https://github.com/huggingface/evaluate/tree/main/metrics) can also be used, and will be loaded if a given metric name is not one natively supported in `lm-eval` or `hf_evaluate` is set to `true`. +For a full list of natively supported metrics and aggregation functions see [`docs/task_guide.md`](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/task_guide.md). To ship a custom metric, aggregation or filter from your own package rather than adding it to the `lm-eval` source tree, see the [Plugin Guide](./plugins.md). All metrics supported in [HuggingFace Evaluate](https://github.com/huggingface/evaluate/tree/main/metrics) can also be used, and will be loaded if a given metric name is not one natively supported in `lm-eval` or `hf_evaluate` is set to `true`. ### Optional, More Advanced Setup diff --git a/docs/plugins.md b/docs/plugins.md new file mode 100644 index 000000000..7de7db200 --- /dev/null +++ b/docs/plugins.md @@ -0,0 +1,109 @@ +# Plugin Guide + +`lm-eval` resolves components — model backends, filters, metrics, aggregations — by +name against internal registries. A plugin is a component that lives in **your own +package** and gets registered into those registries at runtime, so `lm-eval` treats +it exactly like a built-in without you forking or editing the `lm-eval` source tree. + +## What can be contributed + +| Component | Entry-point group | Decorator | Resolved by | +|-------------|------------------------|-------------------------|----------------------| +| Model | `lm_eval.models` | `@register_model` | `--model ` | +| Filter | `lm_eval.filters` | `@register_filter` | `filter_list` in a task YAML | +| Metric | `lm_eval.metrics` | `@register_metric` | `metric_list` in a task YAML | +| Aggregation | `lm_eval.aggregations` | `@register_aggregation` | `aggregation` in a task YAML | + +Tasks are *not* plugins — point `--include_path` at a directory of task YAMLs +instead. See the [New Task Guide](./new_task_guide.md). + +## 1. Entry points (zero-config, recommended for published packages) + +Declare the entry point in your package's `pyproject.toml`, mapping the name +`lm-eval` should resolve to `module:object`: + +```toml +[project.entry-points."lm_eval.models"] +my-backend = "your_pkg.models:MyBackendLM" + +[project.entry-points."lm_eval.metrics"] +my-metric = "your_pkg.metrics:my_metric" +``` + +Once your package is `pip install`ed it works with no extra flags: + +```bash +lm-eval run --model my-backend --tasks hellaswag +``` + +Discovery is **lazy**: the entry point is recorded as a placeholder, and your module +is only imported when the component is actually requested. A broken plugin therefore +only breaks runs that ask for it, and is reported with the offending name. + +Your object can also carry its `@register_*` decorator — importing the module fires +the decorator, which upgrades the placeholder in place. The two are reconciled +automatically as long as the entry-point value and the decorated object agree. + +### Metrics need their decorator + +`higher_is_better` and the default aggregation are recorded as *side effects* of +`@register_metric(...)`, not as separate entry points. So a metric plugin's module +must run the decorator, and the entry point should name the decorated function: + +```python +# my_pkg/metrics.py +from lm_eval.api.registry import register_aggregation, register_metric + + +@register_aggregation("my-agg") +def my_agg(items): + return sum(items) / len(items) + + +@register_metric(metric="my-metric", higher_is_better=True, aggregation="my-agg") +def my_metric(items): + return sum(items) +``` + +If a metric's aggregation is also a plugin, register it in the **same module** — the +aggregation must already be in the registry when `@register_metric` runs. + +## 2. Explicit import (for local or unpublished modules) + +Point `lm-eval` at one or more modules to import before evaluation, so their +`@register_*` decorators run: + +```bash +lm-eval run --model my-backend --plugins my_pkg.models --tasks hellaswag +``` + +`--plugins` accepts a comma-separated or space-separated list, and is also settable +as `plugins:` in a `--config` YAML. A module that fails to import is logged and +skipped rather than aborting the run. + +## Precedence + +Built-in components always win. Discovery skips any alias that is already +registered, so a plugin declaring `acc` cannot shadow the core `acc` metric — the +plugin module is not even imported. Pick a distinctive name. + +## Programmatic use + +Entry-point plugins need no wiring at all — they are discovered on first name +resolution, including from `lm_eval.simple_evaluate(...)`. + +For a module that is not an installed entry-point plugin, import it yourself before +evaluating. + +!!! Warning +Registration mutates **process-global** registries and is not scoped to +a single evaluation: + +```python +import lm_eval +from lm_eval.api.registry import import_plugins + +import_plugins(["my_pkg.models"]) # or simply: import my_pkg.models + +results = lm_eval.simple_evaluate(model="my-backend", tasks=["hellaswag"]) +``` diff --git a/lm_eval/_cli/run.py b/lm_eval/_cli/run.py index 7398f128c..b326124c0 100644 --- a/lm_eval/_cli/run.py +++ b/lm_eval/_cli/run.py @@ -248,9 +248,9 @@ class Run(SubCommand): action=SplitArgs, metavar="", help="Comma-separated plugin modules to import before evaluation so their " - "@register_model (and other @register_*) decorators run. Use for local or " - "unpublished backends; pip-installed packages that declare an " - "'lm_eval.models' entry point are discovered automatically.", + "@register_* decorators run (models, filters, metrics, aggregations). Use " + "for local or unpublished components; pip-installed packages that declare " + "an 'lm_eval.*' entry point are discovered automatically.", ) # Logging and Tracking @@ -369,6 +369,16 @@ class Run(SubCommand): # Create and validate config (most validation now occurs in EvaluationConfig) cfg = EvaluatorConfig.from_cli(args) + # Import explicit plugin modules (--plugins) so their @register_* decorators + # run before task discovery or any model/filter/metric name is resolved. + # Registration is process-global, so this belongs here rather than inside + # simple_evaluate. Installed entry-point plugins need no wiring: they are + # discovered lazily by the registry getters. + if cfg.plugins: + from lm_eval.api.registry import import_plugins + + import_plugins(cfg.plugins) + from lm_eval import simple_evaluate from lm_eval.loggers import EvaluationTracker, TrackioLogger, WandbLogger from lm_eval.utils import handle_non_serializable, make_table @@ -438,7 +448,6 @@ class Run(SubCommand): fewshot_random_seed=cfg.seed[3] if cfg.seed else None, confirm_run_unsafe_code=cfg.confirm_run_unsafe_code, metadata=cfg.metadata, - plugins=cfg.plugins, ) # Process results diff --git a/lm_eval/api/registry.py b/lm_eval/api/registry.py index d60a45130..9edacfb12 100644 --- a/lm_eval/api/registry.py +++ b/lm_eval/api/registry.py @@ -10,10 +10,10 @@ filters, and other components in the lm_eval framework. from lm_eval.api.registry import register_model from lm_eval.api.model import LM + @register_model("my-model") class MyModel(LM): - def __init__(self, **kwargs): - ... + def __init__(self, **kwargs): ... ``` ### Registering with Lazy Loading @@ -137,6 +137,21 @@ def _safe_eq(a: Any, b: Any) -> bool: return a is b +def _placeholder_path(obj: Any) -> str | None: + """Return the "module:qualname" path of a class or function, else None. + + Used to match a materialized component back to the lazy placeholder string + it was registered under, so that a plugin declaring both an entry point and + an @register_* decorator upgrades cleanly instead of colliding. Applies to + both model classes and metric/aggregation functions. + """ + module = getattr(obj, "__module__", None) + name = getattr(obj, "__name__", None) + if module and name: + return f"{module}:{name}" + return None + + def _suggest_similar( query: str, options: Iterable[str], max_suggestions: int = 3 ) -> list[str]: @@ -316,17 +331,18 @@ class Registry(Generic[T]): if current is not None and not _safe_eq(current, target): # allow placeholder → real object upgrade. This is the common # path when a plugin advertises an EntryPoint *and* decorates its - # class with @register_model: materializing the EntryPoint imports - # the module, whose decorator then upgrades the placeholder here. - if isinstance(current, (str, md.EntryPoint)) and isinstance( - target, type + # component with @register_* (a model/filter class, or a metric or + # aggregation function): materializing the EntryPoint imports the + # module, whose decorator then upgrades the placeholder here. + if isinstance(current, (str, md.EntryPoint)) and ( + target_path := _placeholder_path(target) ): placeholder_path = ( current if isinstance(current, str) else f"{current.module}:{current.attr}" ) - if placeholder_path == f"{target.__module__}:{target.__name__}": + if placeholder_path == target_path: self._objs[alias] = target return raise ValueError( @@ -642,8 +658,13 @@ def register_filter(name: str): """ def decorate(cls): - if name in filter_registry: - eval_logger.info(f"Registering filter `{name}` that is already in Registry") + # origin() returns None only while the alias still holds a lazy placeholder: + # that is a plugin's own entry point being upgraded by its decorator on + # import, which is the sanctioned path rather than a clash worth reporting. + if name in filter_registry and filter_registry.origin(name) is not None: + eval_logger.info( + "Registering filter `%s` that is already in Registry", name + ) # Use Registry's public API for registration filter_registry.register(name)(cls) return cls @@ -665,10 +686,12 @@ def get_filter(filter_name: str | Callable) -> Callable: """ if callable(filter_name): return filter_name + # Discover external filters advertised via entry points (once). + load_plugins("lm_eval.filters", filter_registry) try: return filter_registry.get(cast("str", filter_name)) except KeyError: - eval_logger.warning(f"filter `{filter_name}` is not registered!") + eval_logger.warning("filter `%s` is not registered!", filter_name) raise @@ -729,12 +752,16 @@ def get_metric(name: str, hf_evaluate_metric: bool = False) -> Callable | None: if len(metric_registry) == 0: import lm_eval.api.metrics # noqa: F401 + # Discover external metrics advertised via entry points (once). + load_plugins("lm_eval.metrics", metric_registry) + if not hf_evaluate_metric: if name in metric_registry: return metric_registry.get(name) else: eval_logger.warning( - f"Could not find registered metric '{name}' in lm-eval, searching in HF Evaluate library..." + "Could not find registered metric '%s' in lm-eval, searching in HF Evaluate library...", + name, ) try: @@ -744,7 +771,8 @@ def get_metric(name: str, hf_evaluate_metric: bool = False) -> Callable | None: return metric_object.compute except Exception: # noqa: BLE001 - surface any load failure as a warning eval_logger.error( - f"{name} not found in the evaluate library! Please check https://huggingface.co/evaluate-metric", + "%s not found in the evaluate library! Please check https://huggingface.co/evaluate-metric", + name, ) return None @@ -779,13 +807,37 @@ def get_aggregation(name: str) -> Callable[..., float] | None: if len(aggregation_registry) == 0: import lm_eval.api.metrics # noqa: F401 + # Discover external aggregations advertised via entry points (once). + load_plugins("lm_eval.aggregations", aggregation_registry) + try: return aggregation_registry.get(name) except KeyError: - eval_logger.warning(f"{name} not a registered aggregation metric!") + eval_logger.warning("%s not a registered aggregation metric!", name) return None +def _materialise_metric_side_effects(name: str) -> None: + """Ensure a plugin metric's ``@register_metric`` side effects have run. + + ``register_metric`` populates the ``higher_is_better`` and metric-aggregation + registries as a side effect of the decorator. When a metric is contributed as + a lazy entry-point placeholder, those side effects only fire once the metric + function is materialized. ``is_higher_better`` / ``get_metric_aggregation`` + may be called for such a metric, so trigger discovery + materialization here. + """ + # Built-ins must be registered *before* discovery so that a plugin advertising + # an existing name is skipped rather than shadowing core. The callers guard on + # a different registry, so repeat the lazy import here. + if len(metric_registry) == 0: + import lm_eval.api.metrics # noqa: F401 + + load_plugins("lm_eval.metrics", metric_registry) + if name in metric_registry: + # Force materialization of the placeholder so its decorator runs. + metric_registry.get(name) + + def get_metric_aggregation(name: str) -> Callable[..., float] | None: """Get the aggregation function for a metric. @@ -799,10 +851,12 @@ def get_metric_aggregation(name: str) -> Callable[..., float] | None: if len(metric_agg_registry) == 0: import lm_eval.api.metrics # noqa: F401 + _materialise_metric_side_effects(name) + try: return metric_agg_registry.get(name) except KeyError: - eval_logger.warning(f"{name} metric is not assigned a default aggregation!") + eval_logger.warning("%s metric is not assigned a default aggregation!", name) return None @@ -819,10 +873,12 @@ def is_higher_better(metric_name: str) -> bool | None: if len(higher_is_better_registry) == 0: import lm_eval.api.metrics # noqa: F401 + _materialise_metric_side_effects(metric_name) + try: return higher_is_better_registry.get(metric_name) except KeyError: eval_logger.warning( - f"higher_is_better not specified for metric '{metric_name}'!" + "higher_is_better not specified for metric '%s'!", metric_name ) return None diff --git a/lm_eval/evaluator.py b/lm_eval/evaluator.py index 71d480515..797941933 100644 --- a/lm_eval/evaluator.py +++ b/lm_eval/evaluator.py @@ -84,7 +84,6 @@ def simple_evaluate( fewshot_random_seed: int = DEFAULT_OTHER_SEED, confirm_run_unsafe_code: bool = False, metadata: dict[str, Any] | None = None, - plugins: list[str] | None = None, ) -> EvalResults | None: """Instantiate and evaluate a model on a list of tasks. @@ -155,11 +154,6 @@ def simple_evaluate( as unsafe (e.g. code execution tasks). metadata (dict | None): Additional metadata to be added to the task manager. Will get passed to the download function of the task. - plugins (list[str] | None): Module import paths to import before - evaluation so their ``@register_*`` decorators run (for local or - unpublished components). Installed packages that declare an - ``lm_eval.models`` entry point are discovered automatically and do - not need to be listed here. Returns: dict | None: Dictionary of results, or None if not on rank 0. @@ -168,11 +162,6 @@ def simple_evaluate( eval_logger.info("Setting verbosity through simple_evaluate is deprecated.") start_date = time.time() - # Import explicit plugin modules (--plugins) so their @register_* decorators - # run before any model/task/metric name is resolved. - if plugins: - lm_eval.api.registry.import_plugins(plugins) - if limit is not None and samples is not None: raise ValueError( "Either 'limit' or 'samples' must be None, but both are not None." diff --git a/tests/test_plugins.py b/tests/test_plugins.py index 9ff6e8257..58cc17209 100644 --- a/tests/test_plugins.py +++ b/tests/test_plugins.py @@ -32,11 +32,58 @@ def _make_ep(name, obj, *, broken=False): ) +_REGISTRIES = ( + "model_registry", + "filter_registry", + "aggregation_registry", + "metric_registry", + "metric_agg_registry", + "higher_is_better_registry", +) + + @pytest.fixture(autouse=True) def _reset_plugin_state(monkeypatch): - """Ensure the once-per-group guard is clean for every test.""" + """Undo the process-global state a plugin test mutates. + + Registration is not scoped to a call: discovering a plugin writes into module + level registries, imports its module into ``sys.modules`` and marks its entry + point group as scanned. Snapshot and restore all three so tests cannot leak + into each other regardless of ordering. + """ monkeypatch.setattr(registry_mod, "_loaded_plugin_groups", set()) + snapshots = {name: dict(getattr(registry_mod, name)._objs) for name in _REGISTRIES} yield + for name, snapshot in snapshots.items(): + objs = getattr(registry_mod, name)._objs + objs.clear() + objs.update(snapshot) + sys.modules.pop(_FAKE_MODULE, None) + registry_mod._materialise_placeholder.cache_clear() + + +@pytest.fixture +def lazy_plugin_module(tmp_path, monkeypatch): + """Write a plugin module to disk that starts out *unimported*. + + A module pre-seeded into ``sys.modules`` lets a test pass without the entry + point ever being loaded, which hides exactly the lazy-discovery behaviour + these tests exist to pin down. Writing a real file and asserting it is absent + from ``sys.modules`` forces materialization to do the import. + """ + monkeypatch.syspath_prepend(str(tmp_path)) + written: list[str] = [] + + def _write(name: str, source: str) -> str: + (tmp_path / f"{name}.py").write_text(source) + assert name not in sys.modules, f"{name} must not be imported yet" + written.append(name) + return name + + yield _write + + for name in written: + sys.modules.pop(name, None) def _patch_entry_points(monkeypatch, group_name, eps): @@ -143,8 +190,9 @@ def test_import_plugins_runs_register_decorator(monkeypatch): def test_entrypoint_upgrades_to_decorated_class(monkeypatch): - """A plugin that declares BOTH an entry point and @register_model must not - collide: materializing the EntryPoint imports the module whose decorator + """A plugin that declares BOTH an entry point and @register_model must not collide. + + Materializing the EntryPoint imports the module whose decorator re-registers the same alias, and that should upgrade the placeholder. """ reg = Registry("thing") @@ -167,6 +215,158 @@ def test_entrypoint_upgrades_to_decorated_class(monkeypatch): assert reg.get("dual") is Plugin +def test_get_filter_discovers_entry_point(monkeypatch): + """get_filter resolves a filter contributed via the lm_eval.filters group.""" + from lm_eval.api.filter import Filter + from lm_eval.api.registry import get_filter + + class PluginFilter(Filter): + def apply(self, resps, docs): + return resps + + mod = sys.modules.setdefault(_FAKE_MODULE, types.ModuleType(_FAKE_MODULE)) + mod.PluginFilter = PluginFilter + PluginFilter.__module__ = _FAKE_MODULE + ep = registry_mod.md.EntryPoint( + name="plugin-filter", + value=f"{_FAKE_MODULE}:PluginFilter", + group="lm_eval.filters", + ) + _patch_entry_points(monkeypatch, "lm_eval.filters", [ep]) + + assert get_filter("plugin-filter") is PluginFilter + + +def test_get_aggregation_discovers_entry_point(monkeypatch): + """get_aggregation resolves a function from the lm_eval.aggregations group.""" + from lm_eval.api.registry import get_aggregation + + def plugin_agg(items): + return sum(items) + + mod = sys.modules.setdefault(_FAKE_MODULE, types.ModuleType(_FAKE_MODULE)) + mod.plugin_agg = plugin_agg + plugin_agg.__module__ = _FAKE_MODULE + ep = registry_mod.md.EntryPoint( + name="plugin-agg", + value=f"{_FAKE_MODULE}:plugin_agg", + group="lm_eval.aggregations", + ) + _patch_entry_points(monkeypatch, "lm_eval.aggregations", [ep]) + + # Metrics module must be importable for the len()==0 guard; it is. + assert get_aggregation("plugin-agg") is plugin_agg + + +_METRIC_PLUGIN_SOURCE = """ +from lm_eval.api.registry import register_aggregation, register_metric + + +@register_aggregation("plugin-metric-agg") +def plugin_agg(items): + return sum(items) / len(items) + + +@register_metric( + metric="plugin-metric", + higher_is_better=True, + aggregation="plugin-metric-agg", +) +def plugin_metric(items): + return sum(items) +""" + + +def test_metric_plugin_populates_side_effect_registries( + monkeypatch, lazy_plugin_module +): + """A metric contributed only as an entry point must still carry its metadata. + + ``@register_metric`` fills the higher_is_better and metric-aggregation + registries as a decorator side effect, which for a lazy plugin only happens + once its module is imported. ``is_higher_better`` / ``get_metric_aggregation`` + must therefore force discovery and materialization themselves. + """ + from lm_eval.api.registry import ( + get_metric, + get_metric_aggregation, + is_higher_better, + ) + + modname = lazy_plugin_module("lm_eval_fake_metric_plugin", _METRIC_PLUGIN_SOURCE) + ep = registry_mod.md.EntryPoint( + name="plugin-metric", + value=f"{modname}:plugin_metric", + group="lm_eval.metrics", + ) + _patch_entry_points(monkeypatch, "lm_eval.metrics", [ep]) + + # Reached without get_metric() having run first: the metadata lookups are the + # only thing that can trigger the import. + assert is_higher_better("plugin-metric") is True + assert get_metric_aggregation("plugin-metric") is not None + assert callable(get_metric("plugin-metric")) + + +def test_entrypoint_upgrades_to_decorated_function(monkeypatch, lazy_plugin_module): + """The placeholder upgrade must accept functions, not just classes. + + Loading the entry point imports the plugin module, whose ``@register_metric`` + immediately re-registers the same alias. For a metric that target is a + function, so gating the upgrade on ``isinstance(target, type)`` turns the + normal plugin path into a spurious "already registered" collision. + """ + from lm_eval.api.registry import get_metric + + modname = lazy_plugin_module("lm_eval_fake_upgrade_plugin", _METRIC_PLUGIN_SOURCE) + ep = registry_mod.md.EntryPoint( + name="plugin-metric", + value=f"{modname}:plugin_metric", + group="lm_eval.metrics", + ) + _patch_entry_points(monkeypatch, "lm_eval.metrics", [ep]) + + metric = get_metric("plugin-metric") + assert metric is sys.modules[modname].plugin_metric + # Placeholder was replaced by the concrete function, not left as an EntryPoint. + assert not isinstance( + registry_mod.metric_registry._objs["plugin-metric"], registry_mod.md.EntryPoint + ) + + +_SHADOWING_PLUGIN_SOURCE = """ +from lm_eval.api.registry import register_aggregation, register_metric + + +@register_aggregation("shadow-agg") +def shadow_agg(items): + return 0.0 + + +# Deliberately contradicts core's acc (higher_is_better=True) so that reading the +# metadata is enough to tell which registration won. +@register_metric(metric="acc", higher_is_better=False, aggregation="shadow-agg") +def shadow_metric(items): + return 0.0 +""" + + +def test_builtin_metric_wins_over_plugin_of_same_name(monkeypatch, lazy_plugin_module): + """Core components are never shadowed by a plugin claiming the same alias.""" + from lm_eval.api.registry import get_metric, is_higher_better + + modname = lazy_plugin_module("lm_eval_fake_shadow_plugin", _SHADOWING_PLUGIN_SOURCE) + ep = registry_mod.md.EntryPoint( + name="acc", value=f"{modname}:shadow_metric", group="lm_eval.metrics" + ) + _patch_entry_points(monkeypatch, "lm_eval.metrics", [ep]) + + # The plugin declares higher_is_better=False; core declares True. + assert is_higher_better("acc") is True + assert get_metric("acc").__name__ == "acc_fn" + assert modname not in sys.modules, "a shadowed plugin must never be imported" + + def test_import_plugins_tolerates_bad_module(): # Should log and continue, not raise. import_plugins(["definitely_not_a_real_module_xyz"])