datasets.load_metric was removed in datasets 3.0 and the unit-test job
resolves datasets==5.0.1, so the module-level
from datasets import load_metric
makes lm_eval/tasks/basqueglue/utils.py unimportable. Five of the six
BasqueGLUE tasks resolve a !function from that module, so they now fail to
load at all, not just at scoring time:
bec.yaml bec2016eu utils.micro_f1_score
bhtc.yaml bhtc_v2 utils.micro_f1_score
coref.yaml epec_koref_bin utils.coref_doc_to_text
vaxx.yaml vaxx_stance utils.vaxx_f1_score
wic.yaml wiceu utils.process_wic_docs
Load the metric with evaluate instead, the same move #2351 made for
squadv2, with the import inside the function as that fix and
kobest/utils.py do. The metric and the compute() arguments are unchanged.
* fix(bbh): guard find_match against an all-empty capture tuple
`ExtendedRegexFilter.find_match` narrowed a capture tuple with
`[m for m in match if m][0]`, which raises IndexError when every group in
the tuple is empty. A regex whose groups are all optional and whose matched
text lies outside them -- e.g. `x(foo)?(bar)?` against "x" -- makes
`findall` return `[("", "")]`, and the run aborts instead of falling
through to the filter's fallback.
`lm_eval/filters/extraction.py` already guards the same narrowing in
`MultiChoiceRegexFilter.find_match`; these two files are copies that
predate that fix. Adopt the upstream shape verbatim so an all-empty tuple
is treated as a non-match.
No shipped bbh task can currently reach this: every configured
`regex_pattern` has a single capture group, and the two-group regex
`MapRegexFilter` builds by joining its `regex_pattern_to_value` keys always
fills exactly one group when it matches. The crash is reachable from a
user-supplied filter config, so this is a latent fix -- no task's score
changes, and no `metadata.version` bump is warranted.
Co-Authored-By: Claude Opus 5 <[email protected]>
* chore(bbh): fix pre-existing ruff violations blocking Linters CI
The Linters job runs pre-commit with `--from-ref`/`--to-ref`, which scopes
to changed *files* rather than changed lines. Touching these two files
therefore surfaces every existing violation in them, so the preceding fix
cannot go green without clearing these first. Same situation, and same
remedy, as #3999.
Twelve violations, six mirrored in each file:
- `mutable-argument-default` (x4) -- `convert_dict={}` on `find_match` and
`regex_pattern_to_value={}` on `MapRegexFilter.__init__`. These are not
unrelated cleanup: `find_match` is the function the preceding commit
fixes, and `lm_eval/filters/extraction.py` already carries exactly this
change (`convert_dict: dict[str, str] | None = None` plus an in-body
None check). Adopting it closes the rest of the same divergence.
- `empty-docstring` (x2) -- a vestigial `""" """` on `WordSortFilter`.
Removed rather than filled; `D10` (undocumented-public-*) is ignored
repo-wide, so no docstring is the expected state.
- `zip-without-explicit-strict` (x6) -- made explicit with `strict=False`,
which is the pre-3.10 default and so preserves behaviour exactly.
`extraction.py` uses `strict=True` for the corresponding
`zip(resps, docs)`, but adopting that here would change behaviour on a
length mismatch (silent truncation becomes a raised error). That is a
real question and deserves its own change, not a smuggled one inside a
lint commit.
No behaviour changes. Both files remain byte-identical to each other.
Co-Authored-By: Claude Opus 5 <[email protected]>
* nits
---------
Co-authored-by: Claude Opus 5 <[email protected]>
Co-authored-by: Baber Abbasi <[email protected]>
arguments is built as [req.args for req in requests] (list of
per-request args, tuples in memory) and only takes the
gen_args_N/arg_M dict shape after save-path sanitization; per-sample
scores are commonly numpy.float64 in memory, plain floats after save.
Reported in #4022
Co-authored-by: omid511 <[email protected]>
* gguf: fix loglikelihood scoring with exact teacher forcing
This commit combines the following four commits by @xms991, originally
from https://github.com/xms991/lm-evaluation-harness, squashed into one.
All implementation credit goes to @xms991.
gguf: support modern llama.cpp logprobs format via grammar-forced loglikelihood
llama.cpp's /v1/completions endpoint returns logprobs in the modern
OpenAI format (logprobs.content, see llama.cpp PR #10783) and silently
ignores 'echo', so the legacy token_logprobs/echo-based scoring no
longer works. Score continuations by forcing them with a GBNF grammar
instead: llama.cpp reports pre-sampling probabilities for
grammar-constrained tokens, which are exactly the teacher-forced
loglikelihoods needed.
Also:
- add 'model' model-arg for llama-server router mode (fixes 400s)
- accept base_url with or without /v1 or /v1/completions suffix
- handle empty continuations and llama.cpp's trailing empty-token entry
- honor max_gen_toks in generate_until
- add request timeout; return (-inf, False) instead of asserting on
invalid responses
- document llama.cpp server usage and requirements in README
FixesEleutherAI/lm-evaluation-harness#1437
gguf: issue requests concurrently with a thread pool
Add a 'parallel' model arg controlling how many requests are in flight
at once. By default the degree of parallelism is auto-detected from the
server's /props endpoint (total_slots, i.e. llama-server's --parallel
setting), falling back to serial requests when /props is unavailable.
Result order is preserved regardless of completion order.
Also derive and keep the server root URL from base_url so /props can be
queried.
gguf: pin loglikelihood requests to server slots by context group
llama.cpp keeps the KV cache per slot, so parallel requests sharing a
prompt prefix were each recomputing it on different slots. Use the
documented id_slot request parameter to pin consecutive same-context
loglikelihood requests (e.g. the candidate continuations of one
multiple-choice question) to the same slot, round-robing context groups
across slots. This restores prefix-cache reuse in parallel mode and
makes results match serial runs.
generate_until is deliberately left unpinned: generation lengths vary
widely and the server's dynamic idle-slot assignment load-balances
measurably better (~30% faster on gsm8k without pinning).
Benchmarks (Qwen3.6-35B-A3B-Q8_0, 4 server slots):
- hellaswag limit=40: 67.7s serial -> 59.4s, metrics identical to serial
- gsm8k limit=16: 51.1s serial -> 28.4s
gguf: fix loglikelihood scoring with exact teacher forcing
The grammar-forcing approach was subtly but seriously wrong: llama.cpp's
grammar-constrained decoding can select a fragmented tokenization of the
continuation (e.g. forcing "'t" as "'" + "t"), and the resulting
logprobs are for that token path, not the natural tokenization. This
systematically deflated loglikelihoods (mean -32 nats vs the HF backend
on winogrande/Qwen3-0.6B) and cost ~9 points of accuracy (0.525 vs
0.615).
New method: tokenize context and continuation separately via the
server's /tokenize endpoint (same convention as the HF backend), then
score each continuation token individually with its natural prefix as a
token-id prompt and a logit_bias that forces the server to sample it.
llama.cpp reports the forced token's pre-sampling logprob and an
unbiased top_logprobs list, giving exact teacher-forced loglikelihoods.
Validated against the HF backend on identical weights (Qwen3-0.6B,
winogrande limit=200): mean per-token loglikelihood diff -0.011 nats
(stdev 0.26, i.e. quantization noise), 189/200 prediction agreement,
acc 0.610 vs 0.615. Per-position requests to the same slot also chain
off llama.cpp's prompt cache, so scoring stays fast (15s for 200 docs
on the 0.6B).
* fix tokenization: match HF backend's encode-together-then-split and whitespace migration
Baber identified that the HF backend (model.py:391-400) does two things
the gguf model was missing:
1. Migrates trailing whitespace from context into continuation before
tokenizing (n_spaces logic).
2. Encodes context+continuation as one string, then splits at
len(tok(context)) — BPE-correct: avoids boundary mismatches when
the join would merge tokens.
Applied both, and pinned the convention in tests/models/test_gguf.py
(encode-together assertions + a dedicated whitespace-migration test).
Verified against live llama-server (Qwen3.5 2B):
gsm8k: 60% exact match (generation)
mmlu_abstract_algebra: 20% acc (loglikelihood)
Validated vs HF backend on gemma-4-E2B (bf16): tinyMMLU and
tinyHellaswag loglikelihood decisions match 100%.
Tested-by: dotVansh
* gguf: default generate_until to max_gen_toks=256
Match the harness contract: every other backend caps generation at 256
tokens when the task doesn't set one. The gguf backend sent
max_tokens=None, letting llama.cpp generate unbounded, so docs that
never emit a stop string (e.g. repetition loops) produced different
text — and different filtered answers — than the HF backend.
Resolution now goes through the shared normalize_gen_kwargs helper, so
max_gen_toks, max_new_tokens, max_tokens and max_completion_tokens are
all honored (with the usual priority order) and CLI string values are
int()-converted; the default is configurable via a max_gen_toks
model-arg. This also fixes the local-completions-style case where
'--gen_kwargs max_tokens=...' was silently ignored (cf. #2081).
Validated on tinyGSM8k (gemma-4-E2B bf16, 9070 XT): scores now match
the HF backend to within cross-backend numerics (8/100 doc flips,
~0.5 IRT points) instead of diverging on uncapped generations.
* lint
---------
Co-authored-by: xms991 <[email protected]>
Co-authored-by: dotVansh <[email protected]>
Co-authored-by: Baber Abbasi <[email protected]>
* feat(registry): runtime plugin registration for model backends
Add a discovery layer so external packages can contribute model backends to
lm-eval without editing its source, and have them behave exactly like upstream
backends. The Registry class already supported EntryPoint placeholders, but
nothing ever scanned installed distributions for them.
Two mechanisms:
- Entry points: packages declaring an `lm_eval.models` entry point are
discovered automatically (lazy, zero-config) when a model name is resolved.
- Explicit import: `--plugins mod` / `LM_EVAL_PLUGINS` env / `simple_evaluate(plugins=...)`
import named modules so their `@register_*` decorators run, for local or
unpublished components.
Also fixes Registry collision handling to upgrade an EntryPoint placeholder to
its decorated class (the natural case where a plugin declares both an entry
point and `@register_model`), guarding EntryPoint.__eq__ which raises on
mismatched types.
Co-Authored-By: Claude <[email protected]>
* feat(registry): plugin discovery for filters, metrics, and aggregations
Extend the entry-point plugin mechanism (added for models) to the other
Registry-backed components. External packages can now contribute:
- filters via the `lm_eval.filters` entry-point group
- metrics via `lm_eval.metrics`
- aggregations via `lm_eval.aggregations`
get_filter/get_metric/get_aggregation discover installed entry points lazily.
Because @register_metric populates the higher_is_better and metric-aggregation
registries as a decorator side effect, is_higher_better and get_metric_aggregation
now force discovery + materialization of the named metric so plugin metrics carry
their metadata even when those getters are called standalone.
Generalize the Registry placeholder→concrete upgrade to any object with
module/name (functions, not just classes), so metric/aggregation plugins that
declare both an entry point and a decorator upgrade cleanly instead of colliding.
Co-Authored-By: Claude <[email protected]>
* remove plugins from `simple_evaluate`
* fix tests
* fix docs
* add to news
---------
Co-authored-by: Baber Abbasi <[email protected]>
* feat(registry): runtime plugin registration for model backends
Add a discovery layer so external packages can contribute model backends to
lm-eval without editing its source, and have them behave exactly like upstream
backends. The Registry class already supported EntryPoint placeholders, but
nothing ever scanned installed distributions for them.
Two mechanisms:
- Entry points: packages declaring an `lm_eval.models` entry point are
discovered automatically (lazy, zero-config) when a model name is resolved.
- Explicit import: `--plugins mod` / `LM_EVAL_PLUGINS` env / `simple_evaluate(plugins=...)`
import named modules so their `@register_*` decorators run, for local or
unpublished components.
Also fixes Registry collision handling to upgrade an EntryPoint placeholder to
its decorated class (the natural case where a plugin declares both an entry
point and `@register_model`), guarding EntryPoint.__eq__ which raises on
mismatched types.
Co-Authored-By: Claude <[email protected]>
* Address review: drop plugin env var, fix dead except and EntryPoint extras
- Remove LM_EVAL_PLUGINS env var from simple_evaluate: importing third-party
code via an env var is a security risk and makes the function depend on
hidden env state. --plugins / --config already cover the use case.
- load_plugins: lazy registration validates nothing, so the try/except could
never fire. Move real error handling to the actual failure site
(_materialise_placeholder / EntryPoint.load) so a broken plugin logs the
offending name and aborts on access.
- Registry collision handling: use EntryPoint.module/.attr instead of .value
so entry points declared with extras (e.g. "my_pkg:MyLM [gpu]") upgrade
correctly instead of comparing against the bracketed value.
Co-Authored-By: Claude <[email protected]>
* style: satisfy ruff 0.16.3 in touched files (registry, evaluator)
The pre-commit ruff pin was bumped to 0.16.3 on main, which lints whole
changed files. registry.py and evaluator.py carried pre-existing preview-rule
violations that only surface now that this PR touches them:
- move Callable import into TYPE_CHECKING block (TC003)
- sort __all__ (RUF022)
- bare `raise` instead of `raise e` in get_filter (TRY201)
- `float` instead of `int | float` for simple_evaluate limit (PYI041)
- noqa: BLE001 on intentional defensive Exception catches
Co-Authored-By: Claude <[email protected]>
---------
Co-authored-by: Claude <[email protected]>
* fix: make --use_cache keys hashable for multimodal image/byte requests
hash_args() fed request arguments straight into json.dumps(), so any
request carrying a PIL image or bytes crashed with TypeError before the
model was ever called. A JSON default handler now replaces such payloads
with type-tagged content digests (PNG-canonicalized sha256 for images,
matching logged-sample sanitization; raw sha256 for byte-likes), keeping
keys deterministic, content-sensitive, and text-only compatible while
unsupported types keep failing loudly.
Fixes#4005
* chore(dev): add pillow to deps
---------
Co-authored-by: Baber Abbasi <[email protected]>
* fix(evaluator): report a sample count that does not depend on metric order
`_compute_task_aggregations` assigned `sample_len = len(items)` inside the
loop over metrics, so the value left standing was whichever metric came last.
The existing TODO noted this.
Metrics can hold different numbers of values: a document whose
`process_results` omits a key contributes to some metrics and not others,
which is how a task reports a refusal, an unparseable answer or an empty
extraction. When that happens the counts differ, and the reported
`n_samples.effective` depends on dict ordering rather than on the run. The
same raw metrics reported 1 or 3 depending only on which key came first.
Takes the largest per-metric count instead. A document contributes at most
one value to each metric, so the largest count is the number of documents
that produced any metric at all. When every metric holds the same number of
values, which is the common case, the reported count is unchanged.
This does not address the wider question in #4062 of whether dropped
documents should be surfaced per metric; that needs a decision on the
scoring contract. It fixes only the order dependence in what is reported
today.
* feat(evaluator): add warnings for inconsistent metric document counts
---------
Co-authored-by: Baber Abbasi <[email protected]>
* feat(physics_gre): add InflectionAI Physics GRE multiple-choice task
Add the Physics GRE benchmark from InflectionAI/Inflection-Benchmarks as
multiple_choice tasks, loaded from the shayekh/physics_gre Hub mirror
(verified identical to the original). Following the original methodology,
only image-free questions are scored; items with a missing answer key are
also dropped.
Tasks: physics_gre (GR8677, main), physics_gre_additional, physics_gre_all.
Registered in lm_eval/tasks/README.md.
Closes#1554
* feat(physics_gre): add _multiple_choice suffix to task names
The original benchmark reports maj@8/maj@32 over sampled generations, while
these tasks are loglikelihood multiple-choice, so the scores are not directly
comparable. Follow the bigbench naming convention and mark the scoring setup
in the task name.
physics_gre -> physics_gre_multiple_choice
physics_gre_additional -> physics_gre_additional_multiple_choice
physics_gre_all -> physics_gre_all_multiple_choice
Document the suffix and the maj@k/loglikelihood mismatch in the README.
---------
Co-authored-by: bongho.lee <[email protected]>
The `__main__` block passed `model=` and `device=` to
`run_model_for_task_caching()`, whose signature is
`(tasks, cache_requests)` -- both are read from module constants inside
the function, so the call raised
`TypeError: run_model_for_task_caching() got an unexpected keyword
argument 'model'` before anything ran.
With that fixed the script then failed with
`NameError: name 'request_caching_arg_to_dict' is not defined`,
because that helper was defined *below* the `__main__` block and so was
not yet bound when the block executed.
Drop the two extra keyword arguments and move the helper above the
`__main__` block, next to its only caller.
Every RULER task crashes before inference when the tokenizer does not arrive
as a plain string:
lm_eval --model local-chat-completions --tasks ruler_qa_hotpot \
--model_args model=<name>,base_url=http://localhost:8080/v1/chat/completions
TypeError: unhashable type: 'dict'
get_tokenizer is decorated with functools.cache, which hashes its arguments
before the body runs. qa_utils, cwe_utils and fwe_utils resolved the tokenizer
with a `{}` default and passed it straight in, so with neither `tokenizer` nor
`pretrained` present the empty dict was hashed and raised -- leaving the
`assert pretrained, "No tokenizer or pretrained provided."` on the next line
unreachable, which is the message the user actually needs. vt_utils defaulted
to `""` and behaved correctly; this makes the other three match it.
niah_utils failed differently: it forwards every model argument into the
cached function, so an unhashable value among them raises even when a valid
tokenizer was supplied.
The missing key is that these builders are handed the parsed --model_args, and
the backends disagree on what the model is called: `pretrained=` for local
HuggingFace models, `model=` for the API backends. Only the first two were
looked for.
Add resolve_tokenizer_name, taking the first non-empty string among
`tokenizer`, `pretrained` and `model`, and route all eleven builders through
it. None is hashable, so a missing tokenizer now reaches the assertion, whose
message names the argument that fixes it. Resolution is hoisted above the
download in get_qa_dataset and out of the generator in cwe/fwe: previously the
QA builders fetched SQuAD or HotpotQA first and crashed afterwards.
This does not make RULER run against a backend that supplies no usable
tokenizer name -- that now gets the assertion -- and does not touch the
separate report that task metadata does not reach a custom_dataset task.
tests/test_ruler.py is new. cwe_utils and niah_utils need the `ruler` extra,
which the unit test workflow does not install, so those builders skip
individually rather than taking the module with them.
* fix: stop sqrt shorthand normalization from corrupting indexed roots
The \sqrta -> \sqrt{a} shorthand rule consumed the [ of an optional
root index, turning \sqrt[3]{8} into invalid \sqrt{[}3]{8} before
is_equiv ever saw it -- even though parse_latex handles the intact form
natively. Canonicalize indexed roots with an unbraced argument first
and make the shorthand rule skip indexed forms; all non-indexed inputs
are byte-identical under old and new rules.
Fixes#4036
* revert leaderboard `utils`
* move tests
* add changelog
---------
Co-authored-by: Baber Abbasi <[email protected]>
Co-authored-by: tan7vir <[email protected]>
* fix: thousands-separator comma strip fuses bare digit tuples
The digit check behind the 100,000 -> 100000 rule accepted any string
that becomes digits after a naive comma strip, so tuple answers like
"0,1" were fused into the garbage number "01". Sixteen MATH test-set
gold answers are bare tuples; the fused gold can no longer match a
model's "(0,1)" or "0, 1" spelling. Restrict stripping to true
thousands grouping (-?d{1,3}(,d{3})+), preserving all separator cases
in the test set.
* consolidate minerva tests
* revert leaderboard `utils.py` and add note
---------
Co-authored-by: Baber Abbasi <[email protected]>
`is_equiv` decides equivalence only by parsing both operands with sympy and
subtracting them. `parse_latex` raises on whole classes of valid MATH answers
-- tuples, intervals, matrices -- and the handler returns False for the pair,
so an answer that is character-for-character identical to the gold is scored
wrong for every model:
>>> is_equiv("(2,4)", "(2,4)")
False
>>> is_equiv("[2,5)", "[2,5)")
False
>>> is_equiv(r"\begin{pmatrix}1\\2\end{pmatrix}", r"\begin{pmatrix}1\\2\end{pmatrix}")
False
On that slice a correct answer and a wrong one get the same mark, so the metric
is not merely noisy there, it carries no signal at all.
Compare the strings before asking sympy. Both operands reach `is_equiv` already
passed through `normalize_final_answer`, so this compares like with like. It
can only turn a False into a True -- identical strings that sympy can parse
already compared equal further down -- so nothing that scored correct before
stops doing so.
The guards are deliberately narrow: `isinstance` because the TypeError caught
below shows non-str operands do reach this function, and a truthiness test
because two empty answers are not a match (`\text{}` is in REMOVED_EXPRESSIONS,
so `normalize_final_answer` can produce "").
Tests live in a new tests/test_minerva_math.py. It skips when the [math] extra
is absent, since minerva_math imports math_verify and antlr4 at module scope
and the unit test workflow installs neither.
* feat(tydiqa): add TyDiQA Gold Passage tasks (9 languages)
Adds the TyDi QA Gold Passage (secondary) task as 9 per-language tasks
under the `tydiqa_goldp` tag, using SQuAD 1.1 style extractive QA with
language-aware Exact Match and F1.
The `secondary_task` config contains examples for 9 of TyDi QA's 11
languages -- Japanese and Thai are absent from both the train and
validation splits upstream, so no tasks are registered for them.
* fix(tydiqa): draw few-shot examples from train, not the eval split
Without training_split/fewshot_split the harness fell back to
validation_split for few-shot context, so few-shot examples were drawn
from the same 5,077-row pool being evaluated. Point both at train
(49,881 rows), matching the setup used for the merged legalbench tasks.
* feat: add chrf++ aggregation and metric (word_order=2)
Fixes#2256. The existing chrf aggregation uses word_order=0 (plain
ChrF) but its docstring incorrectly described it as chrF++. This PR:
- Corrects the chrf docstring
- Adds a new chrf++ aggregation and metric using word_order=2,
matching the definition from Popovic (2017)
- Adds chrfpp to the bootstrappable list for stderr support
- Adds three tests covering score correctness and registry registration
* cap 'chrf++' in stderr bootstrapping.
* style: fix ruff lint violations in touched files
CI's Linters job runs ruff-check on the full content of any file
changed in a PR, which surfaced pre-existing lint debt in
lm_eval/api/metrics.py and lm_eval/evaluator_utils.py once this PR
touched them. Fixes are mechanical (explicit zip strict=, PEP 585/604
type hints, docstring capitalization, unnecessary comprehensions)
with no behavior change. All 76 metrics/evaluator_utils tests pass.
* Revert "style: fix ruff lint violations in touched files"
This reverts commit 8252012730.
* fix test
---------
Co-authored-by: Baber Abbasi <[email protected]>
Prior to this change, directly creating Unitxt objects would fail when
passed to get_task_dict since it expected a task name. The workaround
is to set .config.task after object creation.
Signed-off-by: mprahl <[email protected]>
* Fix MultiChoiceRegexFilter prefix-shadowing of choice text
The fallback regex joined choices in list order: re.compile("|".join(choices)).
Python regex alternation is leftmost-wins, so a choice whose text is a prefix
of a later choice matched inside the longer choice's text. When the model named
the longer choice (e.g. "Guilty of Romance" with choices ["Guilty",
"Guilty of Romance"]), the shorter prefix "Guilty" matched first and the
answer was mapped to the wrong letter, silently scoring a correct answer as
exact_match=0 on tasks like bbh movie_recommendation.
Sort the alternation branches longest-first so the longest matching choice wins.
choice_to_alpha is keyed by choice text, so letter labels are unaffected.
* fix: apply to bbh as well; add tests
---------
Co-authored-by: Baber Abbasi <[email protected]>
Group.aggregate computes the point estimate with aggregate_subtask_metrics
honoring weight_by_size, but always reported pooled_sample_stderr (the
size-weighted stderr) regardless. When weight_by_size=False the group score is
the simple unweighted mean of subtask means, so the size-weighted pooled stderr
is the stderr of a different estimator -- producing an overconfident error bar
(e.g. ~2.9x too small for one large + two small subtasks). Affects task configs
that set weight_by_size: false.
Add metrics.unweighted_mean_stderr (sqrt(sum(se_i**2))/k, the error-propagation
stderr of the unweighted mean of k independent subtask means) and branch on
weight_by_size in Group.aggregate. The new formula coincides exactly with
pooled_sample_stderr when all subtasks are the same size, so only the buggy
unequal-size case changes.
When few-shot examples are drawn from the same split being evaluated, the eval
doc is passed to ContextSampler.sample, which samples n + 1 docs to leave
headroom for removing a single eval_doc occurrence before truncating to n.
If the pool holds duplicate rows equal to eval_doc, more than one copy can be
drawn and removed by rm_eval_doc, leaving fewer than n examples and tripping the
`len(res) == n` assertion, which aborts the whole evaluation.
Detect this case and resample from the pool with every eval_doc copy removed so
exactly n examples are returned. The common (no-duplicate) path is unchanged, so
existing few-shot selections remain reproducible. Adds a regression test.
TER is an edit rate, so lower is better, but it was registered with
higher_is_better=True. Flip it, and fix the chrf docstring to describe
what the code computes.
TER: registry.py forwards higher_is_better into higher_is_better_registry,
and ConfigurableTask falls back to it whenever a metric_list entry omits
the key, so this boolean is what a task inherits. Resolving every task
yaml through the include rule in tasks/_yaml_loader.py gives eight tasks
that inherit it -- wmt14 en-fr/fr-en, wmt16 de-en/en-de/en-ro/ro-en and
iwslt2017 ar-en/en-ar, all via translation/wmt_common_yaml -- each
reporting the direction backwards. Eighty-four other tasks, from thirteen
config files, already write `higher_is_better: false` by hand; they are now
redundant rather than load-bearing, and are left alone.
chrF: the docstring described chrF++ and carried a "# TODO I think" beside
its direction. corpus_chrf is called with defaults, and sacrebleu defaults
to word_order=0; it names the result chrF2, and chrF2++ only at
word_order=2. Higher is better is correct either way.
handle_non_serializable (and the loggers copy _handle_non_serializable) only
special-cased np.int64/np.int32; every other numpy scalar fell through to
str(o). So np.float32/np.float16, np.int16/int8/uint8 and np.bool_ were written
into the results/samples JSON as quoted strings ("0.5", "True", "5")
instead of numbers/booleans. (np.float64 worked only because it subclasses
Python float.)
Broaden the checks to the numpy abstract base types (np.integer, np.floating,
np.bool_), which cover every width, so all numpy scalars serialize faithfully.
Adds ONNXRuntimeLM (`--model onnxruntime`), scoring Model Builder ONNX exports through onnxruntime.InferenceSession directly -- for execution providers onnxruntime-genai has no build for (ROCm, MIGraphX), and for parity with the session/EP path production serving uses.
- Engine-agnostic scoring moves verbatim into _ONNXLMBase so both backends
share one implementation. onnxruntime_genai.py keeps only engine primitives;
winml.py is untouched.
- The execution_provider=None default from 86555286 moves to the shared
signature; the raw backend reads genai_config.json's providers itself, since
InferenceSession does not.
- Pins max_length to the prompt length in genai's _forward_logits, which
otherwise sizes its KV cache from the full context_length (~34x). Logits
unchanged.
- Scope is loglikelihood and loglikelihood_rolling; generate_until raises
NotImplementedError. Requires onnxruntime>=1.23.
- Cross-backend parity suite runs on CPU, skips without the runtimes.
* fix(med_prescriptions): require both keys before combining complaints and diagnosis
`elif "chief_complaints" and "diagnosis" in results_dict["procedure"]` tests the
truthiness of the non-empty string literal, so it collapses to
`"diagnosis" in results_dict["procedure"]`. A `procedure` entry that carries
`diagnosis` but not `chief_complaints` therefore enters the branch and raises
`KeyError: 'chief_complaints'` while building the prompt.
Test both keys explicitly. Documents carrying only one of them now reach the
existing "NA" fallback instead of aborting the run.
* increment version
---------
Co-authored-by: Baber Abbasi <[email protected]>
ChatGLM maps eos_token_id to a low id that decodes to an empty string,
while tokenizer_config's eos_token (e.g. "</s>") holds the correct
value. Prefer the configured eos_token string when decode is empty so
that generate_until properly detects the end-of-sequence marker.
The afrimmlu per-prompt utils.py files lost the weighted_f1_score re-import in a lint
cleanup (#3577), so every afrimmlu task failed to load with an AttributeError on the
!function reference. The afrimgsm direct prompt_2 tasks were tagged
afrimgsm_tasks_prompt_1, leaving the afrimgsm-irokobench group pointing at an
unregistered name, and the afrixnli manual translate tasks were tagged
afrixnli_manual_direct instead of afrixnli_manual_translate. On top of that, the three
READMEs still documented the pre-AfroBench task names, which is what users hit first
(KeyError: 'afrimmlu_direct_amh'). Restore the imports, fix the two tags, rewrite the
README task lists to match the registry, and add regression tests pinning the
registered names.
Fixes#3360
Ruilin Gao