* gguf: fix loglikelihood scoring with exact teacher forcing
This commit combines the following four commits by @xms991, originally
from https://github.com/xms991/lm-evaluation-harness, squashed into one.
All implementation credit goes to @xms991.
gguf: support modern llama.cpp logprobs format via grammar-forced loglikelihood
llama.cpp's /v1/completions endpoint returns logprobs in the modern
OpenAI format (logprobs.content, see llama.cpp PR #10783) and silently
ignores 'echo', so the legacy token_logprobs/echo-based scoring no
longer works. Score continuations by forcing them with a GBNF grammar
instead: llama.cpp reports pre-sampling probabilities for
grammar-constrained tokens, which are exactly the teacher-forced
loglikelihoods needed.
Also:
- add 'model' model-arg for llama-server router mode (fixes 400s)
- accept base_url with or without /v1 or /v1/completions suffix
- handle empty continuations and llama.cpp's trailing empty-token entry
- honor max_gen_toks in generate_until
- add request timeout; return (-inf, False) instead of asserting on
invalid responses
- document llama.cpp server usage and requirements in README
FixesEleutherAI/lm-evaluation-harness#1437
gguf: issue requests concurrently with a thread pool
Add a 'parallel' model arg controlling how many requests are in flight
at once. By default the degree of parallelism is auto-detected from the
server's /props endpoint (total_slots, i.e. llama-server's --parallel
setting), falling back to serial requests when /props is unavailable.
Result order is preserved regardless of completion order.
Also derive and keep the server root URL from base_url so /props can be
queried.
gguf: pin loglikelihood requests to server slots by context group
llama.cpp keeps the KV cache per slot, so parallel requests sharing a
prompt prefix were each recomputing it on different slots. Use the
documented id_slot request parameter to pin consecutive same-context
loglikelihood requests (e.g. the candidate continuations of one
multiple-choice question) to the same slot, round-robing context groups
across slots. This restores prefix-cache reuse in parallel mode and
makes results match serial runs.
generate_until is deliberately left unpinned: generation lengths vary
widely and the server's dynamic idle-slot assignment load-balances
measurably better (~30% faster on gsm8k without pinning).
Benchmarks (Qwen3.6-35B-A3B-Q8_0, 4 server slots):
- hellaswag limit=40: 67.7s serial -> 59.4s, metrics identical to serial
- gsm8k limit=16: 51.1s serial -> 28.4s
gguf: fix loglikelihood scoring with exact teacher forcing
The grammar-forcing approach was subtly but seriously wrong: llama.cpp's
grammar-constrained decoding can select a fragmented tokenization of the
continuation (e.g. forcing "'t" as "'" + "t"), and the resulting
logprobs are for that token path, not the natural tokenization. This
systematically deflated loglikelihoods (mean -32 nats vs the HF backend
on winogrande/Qwen3-0.6B) and cost ~9 points of accuracy (0.525 vs
0.615).
New method: tokenize context and continuation separately via the
server's /tokenize endpoint (same convention as the HF backend), then
score each continuation token individually with its natural prefix as a
token-id prompt and a logit_bias that forces the server to sample it.
llama.cpp reports the forced token's pre-sampling logprob and an
unbiased top_logprobs list, giving exact teacher-forced loglikelihoods.
Validated against the HF backend on identical weights (Qwen3-0.6B,
winogrande limit=200): mean per-token loglikelihood diff -0.011 nats
(stdev 0.26, i.e. quantization noise), 189/200 prediction agreement,
acc 0.610 vs 0.615. Per-position requests to the same slot also chain
off llama.cpp's prompt cache, so scoring stays fast (15s for 200 docs
on the 0.6B).
* fix tokenization: match HF backend's encode-together-then-split and whitespace migration
Baber identified that the HF backend (model.py:391-400) does two things
the gguf model was missing:
1. Migrates trailing whitespace from context into continuation before
tokenizing (n_spaces logic).
2. Encodes context+continuation as one string, then splits at
len(tok(context)) — BPE-correct: avoids boundary mismatches when
the join would merge tokens.
Applied both, and pinned the convention in tests/models/test_gguf.py
(encode-together assertions + a dedicated whitespace-migration test).
Verified against live llama-server (Qwen3.5 2B):
gsm8k: 60% exact match (generation)
mmlu_abstract_algebra: 20% acc (loglikelihood)
Validated vs HF backend on gemma-4-E2B (bf16): tinyMMLU and
tinyHellaswag loglikelihood decisions match 100%.
Tested-by: dotVansh
* gguf: default generate_until to max_gen_toks=256
Match the harness contract: every other backend caps generation at 256
tokens when the task doesn't set one. The gguf backend sent
max_tokens=None, letting llama.cpp generate unbounded, so docs that
never emit a stop string (e.g. repetition loops) produced different
text — and different filtered answers — than the HF backend.
Resolution now goes through the shared normalize_gen_kwargs helper, so
max_gen_toks, max_new_tokens, max_tokens and max_completion_tokens are
all honored (with the usual priority order) and CLI string values are
int()-converted; the default is configurable via a max_gen_toks
model-arg. This also fixes the local-completions-style case where
'--gen_kwargs max_tokens=...' was silently ignored (cf. #2081).
Validated on tinyGSM8k (gemma-4-E2B bf16, 9070 XT): scores now match
the HF backend to within cross-backend numerics (8/100 doc flips,
~0.5 IRT points) instead of diverging on uncapped generations.
* lint
---------
Co-authored-by: xms991 <[email protected]>
Co-authored-by: dotVansh <[email protected]>
Co-authored-by: Baber Abbasi <[email protected]>
* feat(registry): runtime plugin registration for model backends
Add a discovery layer so external packages can contribute model backends to
lm-eval without editing its source, and have them behave exactly like upstream
backends. The Registry class already supported EntryPoint placeholders, but
nothing ever scanned installed distributions for them.
Two mechanisms:
- Entry points: packages declaring an `lm_eval.models` entry point are
discovered automatically (lazy, zero-config) when a model name is resolved.
- Explicit import: `--plugins mod` / `LM_EVAL_PLUGINS` env / `simple_evaluate(plugins=...)`
import named modules so their `@register_*` decorators run, for local or
unpublished components.
Also fixes Registry collision handling to upgrade an EntryPoint placeholder to
its decorated class (the natural case where a plugin declares both an entry
point and `@register_model`), guarding EntryPoint.__eq__ which raises on
mismatched types.
Co-Authored-By: Claude <[email protected]>
* feat(registry): plugin discovery for filters, metrics, and aggregations
Extend the entry-point plugin mechanism (added for models) to the other
Registry-backed components. External packages can now contribute:
- filters via the `lm_eval.filters` entry-point group
- metrics via `lm_eval.metrics`
- aggregations via `lm_eval.aggregations`
get_filter/get_metric/get_aggregation discover installed entry points lazily.
Because @register_metric populates the higher_is_better and metric-aggregation
registries as a decorator side effect, is_higher_better and get_metric_aggregation
now force discovery + materialization of the named metric so plugin metrics carry
their metadata even when those getters are called standalone.
Generalize the Registry placeholder→concrete upgrade to any object with
module/name (functions, not just classes), so metric/aggregation plugins that
declare both an entry point and a decorator upgrade cleanly instead of colliding.
Co-Authored-By: Claude <[email protected]>
* remove plugins from `simple_evaluate`
* fix tests
* fix docs
* add to news
---------
Co-authored-by: Baber Abbasi <[email protected]>
Adds ONNXRuntimeLM (`--model onnxruntime`), scoring Model Builder ONNX exports through onnxruntime.InferenceSession directly -- for execution providers onnxruntime-genai has no build for (ROCm, MIGraphX), and for parity with the session/EP path production serving uses.
- Engine-agnostic scoring moves verbatim into _ONNXLMBase so both backends
share one implementation. onnxruntime_genai.py keeps only engine primitives;
winml.py is untouched.
- The execution_provider=None default from 86555286 moves to the shared
signature; the raw backend reads genai_config.json's providers itself, since
InferenceSession does not.
- Pins max_length to the prompt length in genai's _forward_logits, which
otherwise sizes its KV cache from the full context_length (~34x). Logits
unchanged.
- Scope is loglikelihood and loglikelihood_rolling; generate_until raises
NotImplementedError. Requires onnxruntime>=1.23.
- Cross-backend parity suite runs on CPU, skips without the runtimes.
* feat(models): add cross-platform onnxruntime-genai backend + refactor winml
Add a cross-platform `onnxruntime-genai` backend that runs ONNX LLMs exported
by the ONNX Runtime GenAI Model Builder on CPU/CUDA/DirectML/WebGPU/AMD-NPU via
the `og.Config` provider API. All lm-eval logic (single-pass log-likelihood,
multiple-choice, rolling perplexity, generation) lives in
`ONNXRuntimeGenAILM(TemplateLM)`, following the `OptimumLM(HFLM)` single-file
subclass precedent.
Refactor `winml` into a thin `WindowsML(ONNXRuntimeGenAILM)` subclass that
overrides only execution-provider selection, inheriting the shared scoring
core. This removes the previous winml quality debt: the per-token
teacher-forcing loop (now a single `get_output("logits")` pass), the direct
`loglikelihood` override that bypassed `TemplateLM` token handling, and the
silent `(0.0, False)`-on-exception path that corrupted scores.
Add a shared parametrized test suite (unit tests for the scoring math with a
fake `_forward_logits`, plus CPU smoke tests over hellaswag/wikitext/gsm8k
using a tiny Model Builder fixture), packaging extra, and docs. The test is
added to the CI `--ignore` list since the ONNX runtimes are not installed in CI.
Part of QUARK-606 Phase 1. The raw `onnxruntime` backend, cross-backend parity
CI, OpenAI-compatible server, and Quark migration are follow-up stages.
Co-Authored-By: Claude <[email protected]>
* docs: keep model_guide subclassing tip generic
Trim the subclassing note to the pre-existing OptimumLM example instead of
documenting the onnxruntime-genai backend internals, which belong in the
backend's own docstrings/README rather than the general model-authoring guide.
Co-Authored-By: Claude <[email protected]>
* test: split winml tests into their own module
Move winml-specific coverage (registration, subclass relationship, non-Windows
CPU fallback) into tests/models/test_winml.py, leaving test_onnxruntime_genai.py
focused on the genai backend and its scoring math. Keeps each backend's tests
self-contained and adds the new file to the CI ignore list.
Co-Authored-By: Claude <[email protected]>
* fix(onnxruntime-genai): honor genai_config.json providers by default
execution_provider now defaults to None, meaning "use the providers the export
declares in genai_config.json" — og.Config(model_dir) seeds its provider list
from that file, and _select_ep no longer calls clear_providers() unless the
user explicitly asks for a provider. Previously the "cpu" default cleared the
list and appended nothing, silently downgrading a CUDA/DML/NPU export to CPU.
Pass execution_provider=cpu to force CPU, or a provider name to override.
winml pins its historical max_length/max_gen_toks (4096) so existing runs are
numerically unchanged, while the cross-platform backend keeps model-aware
defaults (context_length from the config; max_gen_toks=256).
Add regression tests for the three _select_ep branches and winml's defaults.
Co-Authored-By: Claude <[email protected]>
* pacify pre-commmit
---------
Co-authored-by: Claude <[email protected]>
Co-authored-by: Baber Abbasi <[email protected]>
* [Feat] Add native Tensor Parallelism support for HF backend
* Delete tp_size
* results only on main rank; add guard before model init
* Doc
---------
Co-authored-by: Baber Abbasi <[email protected]>
* support megatron-lm backend
* update megatron-lm backend code
* support data parallel/tensor parallel/pipeline parallel
* update RAADEME.md for megatron-lm backend
* remove --use-cpu-initialization
* support expert model parallel
* fix generate_until for ep
* support batch_size > 1 for generate_until
* fix tp > 1
* assert pp > 1
* trim trailing whitespace
* fix: tokenizers and gen_kwargs
* assert batch_size==1 for generate_until
* fix: Fix accuracy degradation with batch_size > 1
> This fixes batched inference accuracy issues caused by padding tokens being visible to attention when using left-padded inputs.
>
> - Force TE SelfAttention to use AttnMaskType.arbitrary so the provided 4D (causal + padding) mask is honored (instead of TE assuming causal-only masking).
> - For RoPE models, always use standard position_ids = [0..S-1] across the batch (avoid mask-derived position ids that can distort positional encoding).
> - Construct and pass a 2D attention_mask (0=pad, 1=real) in _loglikelihood_tokens, matching generate_until, so USE_PADDING_MASK consistently controls whether padding is applied.
> - Remove the batch_size == 1 restriction in generate_until and sort generation batches by token length to minimize padding.
* fix a few pre-commit hook errors
* pacify pre-commit
---------
Co-authored-by: Baber <[email protected]>
* Add Windows ML Backend Support for LM Evaluation Harness
This PR adds support for Windows ML (WinML) as a new backend for the LM
Evaluation Harness, enabling evaluation of LLM Onnx models on Windows devices including CPU, GPU, and NPU
(Neural Processing Unit).
Main Features:
- Windows ML Optimization: Leverage hardware-accelerated inference on Windows AI PC platforms
- Multiple hardware NPU/GPU/CPU evaluation Support: Enable evaluation on devices with Neural Processing Units (e.g., Intel OpenVino, Qualcomm QNN, AMD VitisAI)
, GPU (Nvidia TensorRtRtx, Direct ML GPU provider, etc)
- ONNX Runtime GenAI: Use optimized ONNX models with the GenAI API for efficient inference
- Cross-platform: Extend lm-eval-harness to Windows-specific hardware acceleration
Co-authored-by: Youngjun Lee <[email protected]>
Co-authored-by: Ashrit Shetty <[email protected]>
Co-authored-by: Swetha Joshi <[email protected]>
* Revise API for the perf optimization to get the full logit
* comment resolved
* clean up 2
* clean up 3
* Handle BOS/EOS and empty context
* Revise loglikelihood to resolve NPU performance issue
* Update GenAI set_search_options usage
* pacify pre-commit
* readme
---------
Co-authored-by: Ashrit Shetty <[email protected]>
Co-authored-by: Swetha Joshi <[email protected]>
Co-authored-by: Youngjun Lee <[email protected]>
Co-authored-by: Baber <[email protected]>
* feat: add postprocessing for generated text to strip stop sequences and thinking tokens
* nit
* fix: trim leading whitespace after stripping thinking tokens from generation
* feat: add think_end_token to model_args
* nit
* nit
* nit
* add to readme
* nit
This hasn't been a library for few shot language model evaluation in quite a while. Let's update the citation to use "the Language Model Evaluation Harness" as the title.
* initial components to support sglang
* init of class SGLangLM
* draft for generate_until of SGLang model
* mock loglikelihood
* initial loglikelihood_tokens
* todo: fix bug of sglang engine init
* implement generation tasks and test
* support output type loglikelihood and loglikelihood_rolling (#1)
* .
* loglikelihood_rolling
* /
* support dp_size>1
* typo
* add tests and clean code
* skip tests of sglang for now
* fix OOM error of sglang pytest
* finish test for sglang
* add sglang to readme
* fix OOM of tests and clean SGLang model
* update readme
* clean pyproject and add tests for evaluator
* add accuracy tests and it passed locally
* add notes for test
* Update README.md
update readme
* pre-commit
* add OOM guideline for sglang and fix readme error
* fix typo
* fix typo
* add readme
---------
Co-authored-by: Xiaotong Jiang <[email protected]>
Co-authored-by: Baber Abbasi <[email protected]>
Co-authored-by: Baber <[email protected]>
* initial components to support sglang
* init of class SGLangLM
* draft for generate_until of SGLang model
* mock loglikelihood
* initial loglikelihood_tokens
* todo: fix bug of sglang engine init
* implement generation tasks and test
* support output type loglikelihood and loglikelihood_rolling (#1)
* .
* loglikelihood_rolling
* /
* support dp_size>1
* typo
* add tests and clean code
* skip tests of sglang for now
* fix OOM error of sglang pytest
* finish test for sglang
* add sglang to readme
* fix OOM of tests and clean SGLang model
* update readme
* clean pyproject and add tests for evaluator
* add accuracy tests and it passed locally
* add notes for test
* Update README.md
update readme
* pre-commit
---------
Co-authored-by: Xiaotong Jiang <[email protected]>
Co-authored-by: Baber Abbasi <[email protected]>
Co-authored-by: Baber <[email protected]>
* initial support for optimum-intel ipex model. LM model as first step
* format
Signed-off-by: Yao Matrix <[email protected]>
* pass dtype
Signed-off-by: Yao Matrix <[email protected]>
* update README
Signed-off-by: Yao, Matrix <[email protected]>
---------
Signed-off-by: Yao Matrix <[email protected]>
* Moved to require unitxt installation and not download unitxt from HF hub.
This has performance benefits and simplifies the code.
Signed-off-by: Yoav Katz <[email protected]>
* Updated watsonx documentation
* Updated installation instructions
* Removed redundant comman
* Allowed unitxt tasks to generate chat APIs
Modified WatsonXI model to support chat apis
* Removed print
* Run precommit formatting
---------
Signed-off-by: Yoav Katz <[email protected]>
* Update README.md
I encounter some Git buffer size limits when trying to download all commits history of the repository, such as:
```error: RPC failed; curl 18 transfer closed with outstanding read data remaining
error: 5815 bytes of body are still expected
fetch-pack: unexpected disconnect while reading sideband packet
fatal: early EOF```
therefore the installation is faster and there are not errors when I download only the last version of the repository
* Fix linting issue
* Initial support for Unitxt datasets in LM Eval Harness
See https://github.com/IBM/unitxt
The script 'generate_yamls.py' creates LM Eval Harness yaml files corresponding to Unitxt datasets specified in the 'unitxt_datasets' file.
The glue code required to register Unitxt metrics is in 'unitxt_wrapper.py'.
* Added dataset loading check to generate_yaml
Improved error messages.
* Speed up generate_yaml
Added printouts and improved error message
* Added output printout
* Simplified integration of unitxt datasets
Store all the common yaml configuration in a yaml include shared by all datasets of the same task.
* Post code review comments - part 1
1. Made sure include files don't end wth 'yaml' so they won't be marked as tasks
2. Added more datasets and tasks (NER, GEC)
3. Added README
* Post code review comments - part 2
1. Added install unitxt install option in pyproject.toml:
pip install 'lm_eval[unitxt]'
2. Added a check that unitxt is installed and print a clear error message if not
* Commited missing pyproject change
* Added documentation on adding datasets
* More doc changes
* add unitxt extra to readme
* run precommit
---------
Co-authored-by: haileyschoelkopf <[email protected]>
