Kiersten Stokes
7158f4f4ab
Add Markdown linter ( #2818 )
...
* Add markdown linter to pre-commit hooks
* Reformat existing markdown (excluding lm_eval/tasks/*.md)
2025-03-20 06:23:27 +05:00
Leonid Sinev
d4ae963510
[Fix] Replace generic exception classes with a more specific ones ( #1989 )
...
* Replace generic exception classes with a more specific ones
* rerun pre-commit to pass linter tests
* Revert "rerun pre-commit to pass linter tests"
This reverts commit 67f88ccf14
2024-10-22 21:13:46 +05:00
Baber Abbasi and Lintang Sutawika
3850e21acd
samples is newline delimited (#1930 )
...
* `samples` is newline delimited
* updated git and pre-commit
* appease pre-commit
* nit
* Revert back for now
* Revert for now
---------
Co-authored-by: Lintang Sutawika <[email protected]
2024-06-13 16:54:06 +07:00
Baber Abbasi
e5e35fca08
Vllm update DP+TP ( #1508 )
...
* use `@ray.remote` with distributed vLLM
* update versions
* bugfix
* unpin vllm
* fix pre-commit
* added version assertion error
* Revert "added version assertion error"
This reverts commit 8041e9b78e
2024-03-03 07:35:22 -05:00
Michael Chen
4d7d2f6457
Update README.md ( #1398 )
...
Add instructions for non-MacOS users on how to compile janitor_util.cpp so that janitor.py can use it.
2024-02-05 21:00:21 -05:00
LSinev
488759d2c9
Apply some best practices and guideline recommendations to code ( #1363 )
...
* raise Exception, not a string
Additional info https://peps.python.org/pep-0352/#exception-hierarchy-changes
https://docs.python.org/3.8/tutorial/errors.html#raising-exceptions
* Apply PEP8 recommendation to prefer isinstance
"Object type comparisons should always use isinstance() instead of comparing types directly"
https://peps.python.org/pep-0008/
* Remove dangerous default mutable values in arguments
https://pylint.readthedocs.io/en/stable/user_guide/messages/warning/dangerous-default-value.html
* Format logging messages with fstring (not with format)
Additional info
https://pylint.readthedocs.io/en/stable/user_guide/messages/warning/logging-format-interpolation.html
There are also discussions about the speed of formatting while logging or some unintended code executions
https://github.com/pylint-dev/pylint/issues/2395
https://stackoverflow.com/a/54368109
but at least one format (fstring one) will be used throughout the project
* Specify utf-8 encoding for `open` explicitly
If not specified, it may be supposed differently in different environments, OSes, and Python versions. See
https://peps.python.org/pep-0597/
https://docs.python.org/3.11/library/locale.html#locale.getencoding
https://docs.python.org/3.10/library/os.html#utf8-mode
https://pylint.readthedocs.io/en/stable/user_guide/messages/warning/unspecified-encoding.html
Helps also if some code from English language tasks is taken as inspiration for tasks in non-English languages.
* Use inline-ignoring comments to pass pre-commit instead of identity process
https://flake8.pycqa.org/en/3.0.1/user/ignoring-errors.html#in-line-ignoring-errors
https://www.flake8rules.com/rules/F841.html
flake8 comments are supported by ruff: https://docs.astral.sh/ruff/linter/#error-suppression
2024-01-28 15:44:08 -05:00
Baber Abbasi
65b8761db9
Switch Linting to ruff ( #1166 )
...
* add ruff and isort. remove black and flake8
* remove unnecessary dependencies
* remove dependency from table
* change order
* ran ruff
* check 3.9
* exclude evaluator
* update CI workflow
* use ruff config in pyproject.toml
* test
* add isort rules to ruff
* sort imports
* import `make_table`
* try stages for no-commit-to-branch
* turn on mypy for pre-commit
* test
* test
* test
* change no-commit-to-branch to default
* nits
* fixed dependency
2023-12-20 09:14:46 -05:00
jon-tow
f81db68e87
Fix undeclared identifier error
2022-05-10 23:52:16 -04:00
jon-tow
8c997e532c
Revert tests/testdata changes and address flake8 issues
2022-05-03 01:07:46 -04:00
Fabrizio Milo
d95a433304
fix codespell
2022-05-02 21:39:17 -07:00
Fabrizio Milo
121b7096ab
add pre-commit
2022-05-02 18:09:18 -07:00
researcher2
f9b8115158
Update sort_13_gram_buckets.py
2022-03-14 07:06:43 +10:00
researcher2
4411e788b4
More Changes For PR
...
Add documentation for decontamination.
Improvements to 13-gram generation.
2022-03-14 02:54:39 +10:00
researcher2
f495bfb4f6
Refactor for PR
2022-03-03 16:44:56 +10:00
Leo Gao
b246009951
pull stuff into lm_eval.decontamination
2022-02-24 20:01:51 -07:00
researcher2
98d75af053
Changes for PR
...
Remove arguments from main.py.
Add "decontamination" prefix to ngrams arguments.
2022-02-21 07:43:20 +10:00
Quentin Anthony
bb0eafbb02
Added more eval tasks and fixed directory error tasks
2022-01-30 19:23:36 +00:00
researcher2
9b472bc91f
Create contamination.py
2022-01-31 02:36:05 +10:00
researcher2
6597c3479d
Implement decontamination of evals against training set 13 grams.
2021-12-27 00:02:16 +10:00
researcher2
7549befa4f
Add inits
2021-05-12 17:03:29 +10:00
researcher2
d0277a3fbb
Update janitor.py
...
Comment out cpp tests.
2021-05-12 16:55:55 +10:00
researcher2
7cb91943a6
Janitor testing and changes
...
Small fixes to python version of Janitor, cpp version named properly for import but code doesn't work - may or may not fix in the future.
Add unit tests for python version of Janitor.
2021-05-12 16:48:54 +10:00
researcher2
2a11129d18
Add pile 13gram processing.
2021-05-12 16:46:32 +10:00
&
4de29dcaa5
Moving over training data decontamination code
2021-02-11 20:05:45 -05:00
Kiersten Stokes