mirror of
https://github.com/EleutherAI/lm-evaluation-harness.git
synced 2026-10-12 02:02:05 +00:00
b0195a94f4
* feat(models): add cross-platform onnxruntime-genai backend + refactor winml
Add a cross-platform `onnxruntime-genai` backend that runs ONNX LLMs exported
by the ONNX Runtime GenAI Model Builder on CPU/CUDA/DirectML/WebGPU/AMD-NPU via
the `og.Config` provider API. All lm-eval logic (single-pass log-likelihood,
multiple-choice, rolling perplexity, generation) lives in
`ONNXRuntimeGenAILM(TemplateLM)`, following the `OptimumLM(HFLM)` single-file
subclass precedent.
Refactor `winml` into a thin `WindowsML(ONNXRuntimeGenAILM)` subclass that
overrides only execution-provider selection, inheriting the shared scoring
core. This removes the previous winml quality debt: the per-token
teacher-forcing loop (now a single `get_output("logits")` pass), the direct
`loglikelihood` override that bypassed `TemplateLM` token handling, and the
silent `(0.0, False)`-on-exception path that corrupted scores.
Add a shared parametrized test suite (unit tests for the scoring math with a
fake `_forward_logits`, plus CPU smoke tests over hellaswag/wikitext/gsm8k
using a tiny Model Builder fixture), packaging extra, and docs. The test is
added to the CI `--ignore` list since the ONNX runtimes are not installed in CI.
Part of QUARK-606 Phase 1. The raw `onnxruntime` backend, cross-backend parity
CI, OpenAI-compatible server, and Quark migration are follow-up stages.
Co-Authored-By: Claude <[email protected]>
* docs: keep model_guide subclassing tip generic
Trim the subclassing note to the pre-existing OptimumLM example instead of
documenting the onnxruntime-genai backend internals, which belong in the
backend's own docstrings/README rather than the general model-authoring guide.
Co-Authored-By: Claude <[email protected]>
* test: split winml tests into their own module
Move winml-specific coverage (registration, subclass relationship, non-Windows
CPU fallback) into tests/models/test_winml.py, leaving test_onnxruntime_genai.py
focused on the genai backend and its scoring math. Keeps each backend's tests
self-contained and adds the new file to the CI ignore list.
Co-Authored-By: Claude <[email protected]>
* fix(onnxruntime-genai): honor genai_config.json providers by default
execution_provider now defaults to None, meaning "use the providers the export
declares in genai_config.json" — og.Config(model_dir) seeds its provider list
from that file, and _select_ep no longer calls clear_providers() unless the
user explicitly asks for a provider. Previously the "cpu" default cleared the
list and appended nothing, silently downgrading a CUDA/DML/NPU export to CPU.
Pass execution_provider=cpu to force CPU, or a provider name to override.
winml pins its historical max_length/max_gen_toks (4096) so existing runs are
numerically unchanged, while the cross-platform backend keeps model-aware
defaults (context_length from the config; max_gen_toks=256).
Add regression tests for the three _select_ep branches and winml's defaults.
Co-Authored-By: Claude <[email protected]>
* pacify pre-commmit
---------
Co-authored-by: Claude <[email protected]>
Co-authored-by: Baber Abbasi <[email protected]>