Feat/plugin filter metric registration (#4014)

* feat(registry): runtime plugin registration for model backends

Add a discovery layer so external packages can contribute model backends to
lm-eval without editing its source, and have them behave exactly like upstream
backends. The Registry class already supported EntryPoint placeholders, but
nothing ever scanned installed distributions for them.

Two mechanisms:
- Entry points: packages declaring an `lm_eval.models` entry point are
  discovered automatically (lazy, zero-config) when a model name is resolved.
- Explicit import: `--plugins mod` / `LM_EVAL_PLUGINS` env / `simple_evaluate(plugins=...)`
  import named modules so their `@register_*` decorators run, for local or
  unpublished components.

Also fixes Registry collision handling to upgrade an EntryPoint placeholder to
its decorated class (the natural case where a plugin declares both an entry
point and `@register_model`), guarding EntryPoint.__eq__ which raises on
mismatched types.

Co-Authored-By: Claude <[email protected]>

* feat(registry): plugin discovery for filters, metrics, and aggregations

Extend the entry-point plugin mechanism (added for models) to the other
Registry-backed components. External packages can now contribute:
- filters via the `lm_eval.filters` entry-point group
- metrics via `lm_eval.metrics`
- aggregations via `lm_eval.aggregations`

get_filter/get_metric/get_aggregation discover installed entry points lazily.
Because @register_metric populates the higher_is_better and metric-aggregation
registries as a decorator side effect, is_higher_better and get_metric_aggregation
now force discovery + materialization of the named metric so plugin metrics carry
their metadata even when those getters are called standalone.

Generalize the Registry placeholder→concrete upgrade to any object with
module/name (functions, not just classes), so metric/aggregation plugins that
declare both an entry point and a decorator upgrade cleanly instead of colliding.

Co-Authored-By: Claude <[email protected]>

* remove plugins from `simple_evaluate`

* fix tests

* fix docs

* add to news

---------

Co-authored-by: Baber Abbasi <[email protected]>

README.md

@@ -5,6 +5,7 @@
 ---
 
 ## Latest News 📣
+- [2026/09] **Plugins**: model backends, filters, metrics and aggregations can now be registered from your own package instead of forking — declare an `lm_eval.*` entry point for zero-config discovery, or point `--plugins` at a local module. See the [Plugin Guide](./docs/plugins.md).
 - [2025/12] **CLI refactored** with subcommands (`run`, `ls`, `validate`) and YAML config file support via `--config`. See the [CLI Reference](./docs/interface.md) and [Configuration Guide](./docs/config_files.md).
 - [2025/12] **Lighter install**: Base package no longer includes `transformers`/`torch`. Install model backends separately: `pip install lm_eval[hf]`, `lm_eval[vllm]`, etc.
 - [2025/07] Added `think_end_token` arg to `hf` (token/str), `vllm` and `sglang` (str) for stripping CoT reasoning traces from models that support it.

docs/README.md

@@ -7,5 +7,6 @@ Welcome to the docs for the LM Evaluation Harness!
 * To learn about the public interface of the library, as well as how to evaluate via the command line or as integrated into an external library, see the [Interface](./interface.md).
 * To learn how to add a new library, API, or model type to the library, as well as a quick explainer on the types of ways to evaluate an LM, see the [Model Guide](./model_guide.md).
   * For an extended description of how to extend the library to new model classes served over an API, see the [API Guide](./API_guide.md).
+  * To ship a model, filter, metric or aggregation from your own package instead of the `lm-eval` source tree, see the [Plugin Guide](./plugins.md).
 * For a crash course on adding new tasks to the library, see our [New Task Guide](./new_task_guide.md).
 * To learn more about pushing the limits of task configuration that the Eval Harness supports, see the [Task Configuration Guide](./task_guide.md).

docs/interface.md

@@ -119,6 +119,7 @@ lm-eval run --config my_config.yaml --tasks mmlu
 | Argument | Description |
 |----------|-------------|
 | `--include_path` | Additional directory containing external task YAML files. |
+| `--plugins` | Modules to import before evaluation so their `@register_*` decorators run (models, filters, metrics, aggregations). See the [Plugin Guide](./plugins.md). |
 
 ### Logging and Tracking
 

docs/model_guide.md

@@ -105,33 +105,23 @@ Using this decorator results in the class being added to an accounting of the us
 ### Registering a backend from an external package (plugins)
 
 If your backend lives in your **own** package rather than in the `lm-eval` source
-tree, you do not need to fork or edit `lm-eval`. There are two ways to make
-`lm-eval --model <name>` resolve it at runtime.
+tree, you do not need to fork or edit `lm-eval`. Declare an `lm_eval.models` entry
+point and `lm-eval --model <name>` resolves it at runtime:
 
-**1. Entry points (zero-config, recommended for published packages).** Declare an
-`lm_eval.models` entry point in your package's `pyproject.toml`, mapping the model
-name to `module:Class`:
+# in your_project/pyproject.toml
 
 ```toml
 [project.entry-points."lm_eval.models"]
 my-backend = "my_pkg.models:MyBackendLM"
 ```
 
-Once your package is `pip install`ed, `lm-eval --model my-backend ...` works with no
-extra flags — `lm-eval` discovers installed entry points automatically. The plugin
-module is imported lazily, only when the model is actually requested. (Your class can
-still also carry a `@register_model("my-backend")` decorator; the two are reconciled
-automatically.)
-
-**2. Explicit import (for local or unpublished modules).** Point `lm-eval` at one or
-more modules to import before evaluation, so their `@register_model` decorators run:
-
 ```bash
-lm-eval run --model my-backend --plugins my_pkg.models --tasks hellaswag
+lm-eval run --model my-backend --tasks hellaswag
 ```
 
-The same `plugins=[...]` argument is available on `lm_eval.simple_evaluate(...)` for
-programmatic use.
+For unpublished modules, `--plugins my_pkg.models` imports them so their
+`@register_model` decorator runs. Filters, metrics and aggregations work the same
+way — see the [Plugin Guide](./plugins.md) for the full picture.
 
 ## Testing
 

docs/new_task_guide.md

@@ -315,7 +315,7 @@ metric_list:
 
 `aggregation` and `higher_is_better` can optionally be left out to default to the manually-set defaults if using a natively supported metric, otherwise it must be defined explicitly (for example, when using a custom metric implemented as a function).
 
-For a full list of natively supported metrics and aggregation functions see [`docs/task_guide.md`](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/task_guide.md). All metrics supported in [HuggingFace Evaluate](https://github.com/huggingface/evaluate/tree/main/metrics) can also be used, and will be loaded if a given metric name is not one natively supported in `lm-eval` or `hf_evaluate` is set to `true`.
+For a full list of natively supported metrics and aggregation functions see [`docs/task_guide.md`](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/task_guide.md). To ship a custom metric, aggregation or filter from your own package rather than adding it to the `lm-eval` source tree, see the [Plugin Guide](./plugins.md). All metrics supported in [HuggingFace Evaluate](https://github.com/huggingface/evaluate/tree/main/metrics) can also be used, and will be loaded if a given metric name is not one natively supported in `lm-eval` or `hf_evaluate` is set to `true`.
 
 ### Optional, More Advanced Setup
 

docs/plugins.md

@@ -0,0 +1,109 @@
+# Plugin Guide
+
+`lm-eval` resolves components — model backends, filters, metrics, aggregations — by
+name against internal registries. A plugin is a component that lives in **your own
+package** and gets registered into those registries at runtime, so `lm-eval` treats
+it exactly like a built-in without you forking or editing the `lm-eval` source tree.
+
+## What can be contributed
+
+| Component   | Entry-point group      | Decorator               | Resolved by          |
+|-------------|------------------------|-------------------------|----------------------|
+| Model       | `lm_eval.models`       | `@register_model`       | `--model <name>`     |
+| Filter      | `lm_eval.filters`      | `@register_filter`      | `filter_list` in a task YAML |
+| Metric      | `lm_eval.metrics`      | `@register_metric`      | `metric_list` in a task YAML |
+| Aggregation | `lm_eval.aggregations` | `@register_aggregation` | `aggregation` in a task YAML |
+
+Tasks are *not* plugins — point `--include_path` at a directory of task YAMLs
+instead. See the [New Task Guide](./new_task_guide.md).
+
+## 1. Entry points (zero-config, recommended for published packages)
+
+Declare the entry point in your package's `pyproject.toml`, mapping the name
+`lm-eval` should resolve to `module:object`:
+
+```toml
+[project.entry-points."lm_eval.models"]
+my-backend = "your_pkg.models:MyBackendLM"
+
+[project.entry-points."lm_eval.metrics"]
+my-metric = "your_pkg.metrics:my_metric"
+```
+
+Once your package is `pip install`ed it works with no extra flags:
+
+```bash
+lm-eval run --model my-backend --tasks hellaswag
+```
+
+Discovery is **lazy**: the entry point is recorded as a placeholder, and your module
+is only imported when the component is actually requested. A broken plugin therefore
+only breaks runs that ask for it, and is reported with the offending name.
+
+Your object can also carry its `@register_*` decorator — importing the module fires
+the decorator, which upgrades the placeholder in place. The two are reconciled
+automatically as long as the entry-point value and the decorated object agree.
+
+### Metrics need their decorator
+
+`higher_is_better` and the default aggregation are recorded as *side effects* of
+`@register_metric(...)`, not as separate entry points. So a metric plugin's module
+must run the decorator, and the entry point should name the decorated function:
+
+```python
+# my_pkg/metrics.py
+from lm_eval.api.registry import register_aggregation, register_metric
+
+
+@register_aggregation("my-agg")
+def my_agg(items):
+    return sum(items) / len(items)
+
+
+@register_metric(metric="my-metric", higher_is_better=True, aggregation="my-agg")
+def my_metric(items):
+    return sum(items)
+```
+
+If a metric's aggregation is also a plugin, register it in the **same module** — the
+aggregation must already be in the registry when `@register_metric` runs.
+
+## 2. Explicit import (for local or unpublished modules)
+
+Point `lm-eval` at one or more modules to import before evaluation, so their
+`@register_*` decorators run:
+
+```bash
+lm-eval run --model my-backend --plugins my_pkg.models --tasks hellaswag
+```
+
+`--plugins` accepts a comma-separated or space-separated list, and is also settable
+as `plugins:` in a `--config` YAML. A module that fails to import is logged and
+skipped rather than aborting the run.
+
+## Precedence
+
+Built-in components always win. Discovery skips any alias that is already
+registered, so a plugin declaring `acc` cannot shadow the core `acc` metric — the
+plugin module is not even imported. Pick a distinctive name.
+
+## Programmatic use
+
+Entry-point plugins need no wiring at all — they are discovered on first name
+resolution, including from `lm_eval.simple_evaluate(...)`.
+
+For a module that is not an installed entry-point plugin, import it yourself before
+evaluating.
+
+!!! Warning
+Registration mutates **process-global** registries and is not scoped to
+a single evaluation:
+
+```python
+import lm_eval
+from lm_eval.api.registry import import_plugins
+
+import_plugins(["my_pkg.models"])   # or simply: import my_pkg.models
+
+results = lm_eval.simple_evaluate(model="my-backend", tasks=["hellaswag"])
+```

lm_eval/_cli/run.py

@@ -248,9 +248,9 @@ class Run(SubCommand):
             action=SplitArgs,
             metavar="<module>",
             help="Comma-separated plugin modules to import before evaluation so their "
-            "@register_model (and other @register_*) decorators run. Use for local or "
-            "unpublished backends; pip-installed packages that declare an "
-            "'lm_eval.models' entry point are discovered automatically.",
+            "@register_* decorators run (models, filters, metrics, aggregations). Use "
+            "for local or unpublished components; pip-installed packages that declare "
+            "an 'lm_eval.*' entry point are discovered automatically.",
         )
 
         # Logging and Tracking
@@ -369,6 +369,16 @@ class Run(SubCommand):
         # Create and validate config (most validation now occurs in EvaluationConfig)
         cfg = EvaluatorConfig.from_cli(args)
 
+        # Import explicit plugin modules (--plugins) so their @register_* decorators
+        # run before task discovery or any model/filter/metric name is resolved.
+        # Registration is process-global, so this belongs here rather than inside
+        # simple_evaluate. Installed entry-point plugins need no wiring: they are
+        # discovered lazily by the registry getters.
+        if cfg.plugins:
+            from lm_eval.api.registry import import_plugins
+
+            import_plugins(cfg.plugins)
+
         from lm_eval import simple_evaluate
         from lm_eval.loggers import EvaluationTracker, TrackioLogger, WandbLogger
         from lm_eval.utils import handle_non_serializable, make_table
@@ -438,7 +448,6 @@ class Run(SubCommand):
             fewshot_random_seed=cfg.seed[3] if cfg.seed else None,
             confirm_run_unsafe_code=cfg.confirm_run_unsafe_code,
             metadata=cfg.metadata,
-            plugins=cfg.plugins,
         )
 
         # Process results

lm_eval/api/registry.py

@@ -10,10 +10,10 @@ filters, and other components in the lm_eval framework.
 from lm_eval.api.registry import register_model
 from lm_eval.api.model import LM
 
+
 @register_model("my-model")
 class MyModel(LM):
-    def __init__(self, **kwargs):
-        ...
+    def __init__(self, **kwargs): ...
 ```
 
 ### Registering with Lazy Loading
@@ -137,6 +137,21 @@ def _safe_eq(a: Any, b: Any) -> bool:
         return a is b
 
 
+def _placeholder_path(obj: Any) -> str | None:
+    """Return the "module:qualname" path of a class or function, else None.
+
+    Used to match a materialized component back to the lazy placeholder string
+    it was registered under, so that a plugin declaring both an entry point and
+    an @register_* decorator upgrades cleanly instead of colliding. Applies to
+    both model classes and metric/aggregation functions.
+    """
+    module = getattr(obj, "__module__", None)
+    name = getattr(obj, "__name__", None)
+    if module and name:
+        return f"{module}:{name}"
+    return None
+
+
 def _suggest_similar(
     query: str, options: Iterable[str], max_suggestions: int = 3
 ) -> list[str]:
@@ -316,17 +331,18 @@ class Registry(Generic[T]):
             if current is not None and not _safe_eq(current, target):
                 # allow placeholder → real object upgrade. This is the common
                 # path when a plugin advertises an EntryPoint *and* decorates its
-                # class with @register_model: materializing the EntryPoint imports
-                # the module, whose decorator then upgrades the placeholder here.
-                if isinstance(current, (str, md.EntryPoint)) and isinstance(
-                    target, type
+                # component with @register_* (a model/filter class, or a metric or
+                # aggregation function): materializing the EntryPoint imports the
+                # module, whose decorator then upgrades the placeholder here.
+                if isinstance(current, (str, md.EntryPoint)) and (
+                    target_path := _placeholder_path(target)
                 ):
                     placeholder_path = (
                         current
                         if isinstance(current, str)
                         else f"{current.module}:{current.attr}"
                     )
-                    if placeholder_path == f"{target.__module__}:{target.__name__}":
+                    if placeholder_path == target_path:
                         self._objs[alias] = target
                         return
                 raise ValueError(
@@ -642,8 +658,13 @@ def register_filter(name: str):
     """
 
     def decorate(cls):
-        if name in filter_registry:
-            eval_logger.info(f"Registering filter `{name}` that is already in Registry")
+        # origin() returns None only while the alias still holds a lazy placeholder:
+        # that is a plugin's own entry point being upgraded by its decorator on
+        # import, which is the sanctioned path rather than a clash worth reporting.
+        if name in filter_registry and filter_registry.origin(name) is not None:
+            eval_logger.info(
+                "Registering filter `%s` that is already in Registry", name
+            )
         # Use Registry's public API for registration
         filter_registry.register(name)(cls)
         return cls
@@ -665,10 +686,12 @@ def get_filter(filter_name: str | Callable) -> Callable:
     """
     if callable(filter_name):
         return filter_name
+    # Discover external filters advertised via entry points (once).
+    load_plugins("lm_eval.filters", filter_registry)
     try:
         return filter_registry.get(cast("str", filter_name))
     except KeyError:
-        eval_logger.warning(f"filter `{filter_name}` is not registered!")
+        eval_logger.warning("filter `%s` is not registered!", filter_name)
         raise
 
 
@@ -729,12 +752,16 @@ def get_metric(name: str, hf_evaluate_metric: bool = False) -> Callable | None:
     if len(metric_registry) == 0:
         import lm_eval.api.metrics  # noqa: F401
 
+    # Discover external metrics advertised via entry points (once).
+    load_plugins("lm_eval.metrics", metric_registry)
+
     if not hf_evaluate_metric:
         if name in metric_registry:
             return metric_registry.get(name)
         else:
             eval_logger.warning(
-                f"Could not find registered metric '{name}' in lm-eval, searching in HF Evaluate library..."
+                "Could not find registered metric '%s' in lm-eval, searching in HF Evaluate library...",
+                name,
             )
 
     try:
@@ -744,7 +771,8 @@ def get_metric(name: str, hf_evaluate_metric: bool = False) -> Callable | None:
         return metric_object.compute
     except Exception:  # noqa: BLE001 - surface any load failure as a warning
         eval_logger.error(
-            f"{name} not found in the evaluate library! Please check https://huggingface.co/evaluate-metric",
+            "%s not found in the evaluate library! Please check https://huggingface.co/evaluate-metric",
+            name,
         )
         return None
 
@@ -779,13 +807,37 @@ def get_aggregation(name: str) -> Callable[..., float] | None:
     if len(aggregation_registry) == 0:
         import lm_eval.api.metrics  # noqa: F401
 
+    # Discover external aggregations advertised via entry points (once).
+    load_plugins("lm_eval.aggregations", aggregation_registry)
+
     try:
         return aggregation_registry.get(name)
     except KeyError:
-        eval_logger.warning(f"{name} not a registered aggregation metric!")
+        eval_logger.warning("%s not a registered aggregation metric!", name)
         return None
 
 
+def _materialise_metric_side_effects(name: str) -> None:
+    """Ensure a plugin metric's ``@register_metric`` side effects have run.
+
+    ``register_metric`` populates the ``higher_is_better`` and metric-aggregation
+    registries as a side effect of the decorator. When a metric is contributed as
+    a lazy entry-point placeholder, those side effects only fire once the metric
+    function is materialized. ``is_higher_better`` / ``get_metric_aggregation``
+    may be called for such a metric, so trigger discovery + materialization here.
+    """
+    # Built-ins must be registered *before* discovery so that a plugin advertising
+    # an existing name is skipped rather than shadowing core. The callers guard on
+    # a different registry, so repeat the lazy import here.
+    if len(metric_registry) == 0:
+        import lm_eval.api.metrics  # noqa: F401
+
+    load_plugins("lm_eval.metrics", metric_registry)
+    if name in metric_registry:
+        # Force materialization of the placeholder so its decorator runs.
+        metric_registry.get(name)
+
+
 def get_metric_aggregation(name: str) -> Callable[..., float] | None:
     """Get the aggregation function for a metric.
 
@@ -799,10 +851,12 @@ def get_metric_aggregation(name: str) -> Callable[..., float] | None:
     if len(metric_agg_registry) == 0:
         import lm_eval.api.metrics  # noqa: F401
 
+    _materialise_metric_side_effects(name)
+
     try:
         return metric_agg_registry.get(name)
     except KeyError:
-        eval_logger.warning(f"{name} metric is not assigned a default aggregation!")
+        eval_logger.warning("%s metric is not assigned a default aggregation!", name)
         return None
 
 
@@ -819,10 +873,12 @@ def is_higher_better(metric_name: str) -> bool | None:
     if len(higher_is_better_registry) == 0:
         import lm_eval.api.metrics  # noqa: F401
 
+    _materialise_metric_side_effects(metric_name)
+
     try:
         return higher_is_better_registry.get(metric_name)
     except KeyError:
         eval_logger.warning(
-            f"higher_is_better not specified for metric '{metric_name}'!"
+            "higher_is_better not specified for metric '%s'!", metric_name
         )
         return None

lm_eval/evaluator.py

@@ -84,7 +84,6 @@ def simple_evaluate(
     fewshot_random_seed: int = DEFAULT_OTHER_SEED,
     confirm_run_unsafe_code: bool = False,
     metadata: dict[str, Any] | None = None,
-    plugins: list[str] | None = None,
 ) -> EvalResults | None:
     """Instantiate and evaluate a model on a list of tasks.
 
@@ -155,11 +154,6 @@ def simple_evaluate(
             as unsafe (e.g. code execution tasks).
         metadata (dict | None): Additional metadata to be added to the task
             manager. Will get passed to the download function of the task.
-        plugins (list[str] | None): Module import paths to import before
-            evaluation so their ``@register_*`` decorators run (for local or
-            unpublished components). Installed packages that declare an
-            ``lm_eval.models`` entry point are discovered automatically and do
-            not need to be listed here.
 
     Returns:
         dict | None: Dictionary of results, or None if not on rank 0.
@@ -168,11 +162,6 @@ def simple_evaluate(
         eval_logger.info("Setting verbosity through simple_evaluate is deprecated.")
     start_date = time.time()
 
-    # Import explicit plugin modules (--plugins) so their @register_* decorators
-    # run before any model/task/metric name is resolved.
-    if plugins:
-        lm_eval.api.registry.import_plugins(plugins)
-
     if limit is not None and samples is not None:
         raise ValueError(
             "Either 'limit' or 'samples' must be None, but both are not None."

tests/test_plugins.py

@@ -32,11 +32,58 @@ def _make_ep(name, obj, *, broken=False):
     )
 
 
+_REGISTRIES = (
+    "model_registry",
+    "filter_registry",
+    "aggregation_registry",
+    "metric_registry",
+    "metric_agg_registry",
+    "higher_is_better_registry",
+)
+
+
 @pytest.fixture(autouse=True)
 def _reset_plugin_state(monkeypatch):
-    """Ensure the once-per-group guard is clean for every test."""
+    """Undo the process-global state a plugin test mutates.
+
+    Registration is not scoped to a call: discovering a plugin writes into module
+    level registries, imports its module into ``sys.modules`` and marks its entry
+    point group as scanned. Snapshot and restore all three so tests cannot leak
+    into each other regardless of ordering.
+    """
     monkeypatch.setattr(registry_mod, "_loaded_plugin_groups", set())
+    snapshots = {name: dict(getattr(registry_mod, name)._objs) for name in _REGISTRIES}
     yield
+    for name, snapshot in snapshots.items():
+        objs = getattr(registry_mod, name)._objs
+        objs.clear()
+        objs.update(snapshot)
+    sys.modules.pop(_FAKE_MODULE, None)
+    registry_mod._materialise_placeholder.cache_clear()
+
+
+@pytest.fixture
+def lazy_plugin_module(tmp_path, monkeypatch):
+    """Write a plugin module to disk that starts out *unimported*.
+
+    A module pre-seeded into ``sys.modules`` lets a test pass without the entry
+    point ever being loaded, which hides exactly the lazy-discovery behaviour
+    these tests exist to pin down. Writing a real file and asserting it is absent
+    from ``sys.modules`` forces materialization to do the import.
+    """
+    monkeypatch.syspath_prepend(str(tmp_path))
+    written: list[str] = []
+
+    def _write(name: str, source: str) -> str:
+        (tmp_path / f"{name}.py").write_text(source)
+        assert name not in sys.modules, f"{name} must not be imported yet"
+        written.append(name)
+        return name
+
+    yield _write
+
+    for name in written:
+        sys.modules.pop(name, None)
 
 
 def _patch_entry_points(monkeypatch, group_name, eps):
@@ -143,8 +190,9 @@ def test_import_plugins_runs_register_decorator(monkeypatch):
 
 
 def test_entrypoint_upgrades_to_decorated_class(monkeypatch):
-    """A plugin that declares BOTH an entry point and @register_model must not
-    collide: materializing the EntryPoint imports the module whose decorator
+    """A plugin that declares BOTH an entry point and @register_model must not collide.
+
+    Materializing the EntryPoint imports the module whose decorator
     re-registers the same alias, and that should upgrade the placeholder.
     """
     reg = Registry("thing")
@@ -167,6 +215,158 @@ def test_entrypoint_upgrades_to_decorated_class(monkeypatch):
     assert reg.get("dual") is Plugin
 
 
+def test_get_filter_discovers_entry_point(monkeypatch):
+    """get_filter resolves a filter contributed via the lm_eval.filters group."""
+    from lm_eval.api.filter import Filter
+    from lm_eval.api.registry import get_filter
+
+    class PluginFilter(Filter):
+        def apply(self, resps, docs):
+            return resps
+
+    mod = sys.modules.setdefault(_FAKE_MODULE, types.ModuleType(_FAKE_MODULE))
+    mod.PluginFilter = PluginFilter
+    PluginFilter.__module__ = _FAKE_MODULE
+    ep = registry_mod.md.EntryPoint(
+        name="plugin-filter",
+        value=f"{_FAKE_MODULE}:PluginFilter",
+        group="lm_eval.filters",
+    )
+    _patch_entry_points(monkeypatch, "lm_eval.filters", [ep])
+
+    assert get_filter("plugin-filter") is PluginFilter
+
+
+def test_get_aggregation_discovers_entry_point(monkeypatch):
+    """get_aggregation resolves a function from the lm_eval.aggregations group."""
+    from lm_eval.api.registry import get_aggregation
+
+    def plugin_agg(items):
+        return sum(items)
+
+    mod = sys.modules.setdefault(_FAKE_MODULE, types.ModuleType(_FAKE_MODULE))
+    mod.plugin_agg = plugin_agg
+    plugin_agg.__module__ = _FAKE_MODULE
+    ep = registry_mod.md.EntryPoint(
+        name="plugin-agg",
+        value=f"{_FAKE_MODULE}:plugin_agg",
+        group="lm_eval.aggregations",
+    )
+    _patch_entry_points(monkeypatch, "lm_eval.aggregations", [ep])
+
+    # Metrics module must be importable for the len()==0 guard; it is.
+    assert get_aggregation("plugin-agg") is plugin_agg
+
+
+_METRIC_PLUGIN_SOURCE = """
+from lm_eval.api.registry import register_aggregation, register_metric
+
+
+@register_aggregation("plugin-metric-agg")
+def plugin_agg(items):
+    return sum(items) / len(items)
+
+
+@register_metric(
+    metric="plugin-metric",
+    higher_is_better=True,
+    aggregation="plugin-metric-agg",
+)
+def plugin_metric(items):
+    return sum(items)
+"""
+
+
+def test_metric_plugin_populates_side_effect_registries(
+    monkeypatch, lazy_plugin_module
+):
+    """A metric contributed only as an entry point must still carry its metadata.
+
+    ``@register_metric`` fills the higher_is_better and metric-aggregation
+    registries as a decorator side effect, which for a lazy plugin only happens
+    once its module is imported. ``is_higher_better`` / ``get_metric_aggregation``
+    must therefore force discovery and materialization themselves.
+    """
+    from lm_eval.api.registry import (
+        get_metric,
+        get_metric_aggregation,
+        is_higher_better,
+    )
+
+    modname = lazy_plugin_module("lm_eval_fake_metric_plugin", _METRIC_PLUGIN_SOURCE)
+    ep = registry_mod.md.EntryPoint(
+        name="plugin-metric",
+        value=f"{modname}:plugin_metric",
+        group="lm_eval.metrics",
+    )
+    _patch_entry_points(monkeypatch, "lm_eval.metrics", [ep])
+
+    # Reached without get_metric() having run first: the metadata lookups are the
+    # only thing that can trigger the import.
+    assert is_higher_better("plugin-metric") is True
+    assert get_metric_aggregation("plugin-metric") is not None
+    assert callable(get_metric("plugin-metric"))
+
+
+def test_entrypoint_upgrades_to_decorated_function(monkeypatch, lazy_plugin_module):
+    """The placeholder upgrade must accept functions, not just classes.
+
+    Loading the entry point imports the plugin module, whose ``@register_metric``
+    immediately re-registers the same alias. For a metric that target is a
+    function, so gating the upgrade on ``isinstance(target, type)`` turns the
+    normal plugin path into a spurious "already registered" collision.
+    """
+    from lm_eval.api.registry import get_metric
+
+    modname = lazy_plugin_module("lm_eval_fake_upgrade_plugin", _METRIC_PLUGIN_SOURCE)
+    ep = registry_mod.md.EntryPoint(
+        name="plugin-metric",
+        value=f"{modname}:plugin_metric",
+        group="lm_eval.metrics",
+    )
+    _patch_entry_points(monkeypatch, "lm_eval.metrics", [ep])
+
+    metric = get_metric("plugin-metric")
+    assert metric is sys.modules[modname].plugin_metric
+    # Placeholder was replaced by the concrete function, not left as an EntryPoint.
+    assert not isinstance(
+        registry_mod.metric_registry._objs["plugin-metric"], registry_mod.md.EntryPoint
+    )
+
+
+_SHADOWING_PLUGIN_SOURCE = """
+from lm_eval.api.registry import register_aggregation, register_metric
+
+
+@register_aggregation("shadow-agg")
+def shadow_agg(items):
+    return 0.0
+
+
+# Deliberately contradicts core's acc (higher_is_better=True) so that reading the
+# metadata is enough to tell which registration won.
+@register_metric(metric="acc", higher_is_better=False, aggregation="shadow-agg")
+def shadow_metric(items):
+    return 0.0
+"""
+
+
+def test_builtin_metric_wins_over_plugin_of_same_name(monkeypatch, lazy_plugin_module):
+    """Core components are never shadowed by a plugin claiming the same alias."""
+    from lm_eval.api.registry import get_metric, is_higher_better
+
+    modname = lazy_plugin_module("lm_eval_fake_shadow_plugin", _SHADOWING_PLUGIN_SOURCE)
+    ep = registry_mod.md.EntryPoint(
+        name="acc", value=f"{modname}:shadow_metric", group="lm_eval.metrics"
+    )
+    _patch_entry_points(monkeypatch, "lm_eval.metrics", [ep])
+
+    # The plugin declares higher_is_better=False; core declares True.
+    assert is_higher_better("acc") is True
+    assert get_metric("acc").__name__ == "acc_fn"
+    assert modname not in sys.modules, "a shadowed plugin must never be imported"
+
+
 def test_import_plugins_tolerates_bad_module():
     # Should log and continue, not raise.
     import_plugins(["definitely_not_a_real_module_xyz"])