lm-evaluation-harness

mirror of https://github.com/EleutherAI/lm-evaluation-harness.git synced 2026-10-11 21:59:53 +00:00

Files

T

History

Ruilin Gao d6de816439 fix(basqueglue): load the f1 metric with evaluate (#4076 )

datasets.load_metric was removed in datasets 3.0 and the unit-test job
resolves datasets==5.0.1, so the module-level

    from datasets import load_metric

makes lm_eval/tasks/basqueglue/utils.py unimportable. Five of the six
BasqueGLUE tasks resolve a !function from that module, so they now fail to
load at all, not just at scoring time:

    bec.yaml    bec2016eu        utils.micro_f1_score
    bhtc.yaml   bhtc_v2          utils.micro_f1_score
    coref.yaml  epec_koref_bin   utils.coref_doc_to_text
    vaxx.yaml   vaxx_stance      utils.vaxx_f1_score
    wic.yaml    wiceu            utils.process_wic_docs

Load the metric with evaluate instead, the same move #2351 made for
squadv2, with the import inside the function as that fix and
kobest/utils.py do. The metric and the compute() arguments are unchanged.

2026-09-14 15:51:05 +05:00

_cli

fix(cli): preserve signed integer argument types (#4136 )

2026-09-14 14:40:04 +05:00

api

Feat/plugin filter metric registration (#4014 )

2026-09-01 18:51:28 +05:00

caching

fix(cache): create missing cache parent directories (#4047 )

2026-08-27 17:02:55 +05:00

config

feat(registry): runtime plugin registration for model backends (#4013 )