mirror of
https://github.com/EleutherAI/lm-evaluation-harness.git
synced 2026-10-12 02:02:05 +00:00
e54687eac0
* fix(bbh): guard find_match against an all-empty capture tuple
`ExtendedRegexFilter.find_match` narrowed a capture tuple with
`[m for m in match if m][0]`, which raises IndexError when every group in
the tuple is empty. A regex whose groups are all optional and whose matched
text lies outside them -- e.g. `x(foo)?(bar)?` against "x" -- makes
`findall` return `[("", "")]`, and the run aborts instead of falling
through to the filter's fallback.
`lm_eval/filters/extraction.py` already guards the same narrowing in
`MultiChoiceRegexFilter.find_match`; these two files are copies that
predate that fix. Adopt the upstream shape verbatim so an all-empty tuple
is treated as a non-match.
No shipped bbh task can currently reach this: every configured
`regex_pattern` has a single capture group, and the two-group regex
`MapRegexFilter` builds by joining its `regex_pattern_to_value` keys always
fills exactly one group when it matches. The crash is reachable from a
user-supplied filter config, so this is a latent fix -- no task's score
changes, and no `metadata.version` bump is warranted.
Co-Authored-By: Claude Opus 5 <[email protected]>
* chore(bbh): fix pre-existing ruff violations blocking Linters CI
The Linters job runs pre-commit with `--from-ref`/`--to-ref`, which scopes
to changed *files* rather than changed lines. Touching these two files
therefore surfaces every existing violation in them, so the preceding fix
cannot go green without clearing these first. Same situation, and same
remedy, as #3999.
Twelve violations, six mirrored in each file:
- `mutable-argument-default` (x4) -- `convert_dict={}` on `find_match` and
`regex_pattern_to_value={}` on `MapRegexFilter.__init__`. These are not
unrelated cleanup: `find_match` is the function the preceding commit
fixes, and `lm_eval/filters/extraction.py` already carries exactly this
change (`convert_dict: dict[str, str] | None = None` plus an in-body
None check). Adopting it closes the rest of the same divergence.
- `empty-docstring` (x2) -- a vestigial `""" """` on `WordSortFilter`.
Removed rather than filled; `D10` (undocumented-public-*) is ignored
repo-wide, so no docstring is the expected state.
- `zip-without-explicit-strict` (x6) -- made explicit with `strict=False`,
which is the pre-3.10 default and so preserves behaviour exactly.
`extraction.py` uses `strict=True` for the corresponding
`zip(resps, docs)`, but adopting that here would change behaviour on a
length mismatch (silent truncation becomes a raised error). That is a
real question and deserves its own change, not a smuggled one inside a
lint commit.
No behaviour changes. Both files remain byte-identical to each other.
Co-Authored-By: Claude Opus 5 <[email protected]>
* nits
---------
Co-authored-by: Claude Opus 5 <[email protected]>
Co-authored-by: Baber Abbasi <[email protected]>