The `__main__` block passed `model=` and `device=` to
`run_model_for_task_caching()`, whose signature is
`(tasks, cache_requests)` -- both are read from module constants inside
the function, so the call raised
`TypeError: run_model_for_task_caching() got an unexpected keyword
argument 'model'` before anything ran.
With that fixed the script then failed with
`NameError: name 'request_caching_arg_to_dict' is not defined`,
because that helper was defined *below* the `__main__` block and so was
not yet bound when the block executed.
Drop the two extra keyword arguments and move the helper above the
`__main__` block, next to its only caller.
- Refactors TaskManager into four modules: TaskManager (public API), TaskIndex (YAML discovery), TaskFactory (object construction), and _yaml_loader (YAML parsing).
- Replaces ConfigurableGroup with Group dataclass that holds direct child references and handles metric aggregation via Group.aggregate().
* git push --force
correctly processes both formats of model_args: string and dictionary both
* exctract to function for better test
* nit
---------
Co-authored-by: Baber <[email protected]>
* changed source of eval_logger
* allow eval_logger to be set from args
* removed verbosity arg from non-main methods
* fix logging
* pre-commit
* set verbosity in eval logger
* replace utils.eval_logger
* fix logging in main
* add logging to docs
* add logging message
* nit
* add logging to docs
* refactor setup_logging to utils
---------
Co-authored-by: Baber <[email protected]>
* fix(zeno): Generate unique ids in case of multiple filters
* fix(zeno): Report even non-aggregable metrics, just not as metrics
* pre-commit
---------
Co-authored-by: Baber <[email protected]>
* Replace generic exception classes with a more specific ones
* rerun pre-commit to pass linter tests
* Revert "rerun pre-commit to pass linter tests"
This reverts commit 67f88ccf14.
* reduce repetitions in errors or so
* Replace generic exception class with a more specific one
* refactor: move scipy and sklearn module imports to func imports
Signed-off-by: Nathan Weinberg <[email protected]>
* refactor: consolidate weighted_f1_score func into lm_eval utils
Signed-off-by: Nathan Weinberg <[email protected]>
* lint: allow for utils file to have unused imports
this allows for shared functions to be defined only
once while allowing for the YAML function importing
to continue working
Signed-off-by: Nathan Weinberg <[email protected]>
---------
Signed-off-by: Nathan Weinberg <[email protected]>
* `samples` is newline delimited
* updated git and pre-commit
* appease pre-commit
* nit
* Revert back for now
* Revert for now
---------
Co-authored-by: Lintang Sutawika <[email protected]>
* results filenames handling moved to utils
* zeno results handling fix
* tasks_for_model backward compatibility
* results files logic moved to tasks_for_model
* moved sanitize_model_name to utils
* Create a means for caching task registration and request building. Add the ability to specify an args dict for simple_evaluate().
* Remove extra S in cache path in caching module
Co-authored-by: Hailey Schoelkopf <[email protected]>
* Rename requests cache args, make model_args polymorphic so that a dict can also be accepted.
* Update docs to reflect new caching behavior, add CLI args for requests caching. Create a function for deleting items in the cache.
* Update documentation, fix minor bug with arg parsing for requests caching where an undefined variable was used.
* Remove line from gitignore, add to cli for caching datasets.
* Add hashing suffix to .pickles. Update test script typo.
* Favor isinstance() over type() in evaluator.py
* Add tests for caching, gets tests working, remove unneeded arg from build_all_requests().
* Update arg description to simple_evaluate.
* Update pyproject.toml
* Fix typehint
* Remove the use of random() for creating default cache pickle hash.
* Check that cache dir exists before clearing it in request cache tests.
* Fix linting problems.
* Fix additional formatting errors.
* Remove trailing whitespace.
* Add new line to the end of .gitignore.
---------
Co-authored-by: Hailey Schoelkopf <[email protected]>
* add trust_remote_code as default
* task for testing recursive
* changed source of ALL_TASKS
* tasks should only accept TaskObjects
* initialize_tasks returns list of tasks and groups
* remove trust_remote_code for now
* moved constructor process to inside load_yaml_config
* more comprehensive way to index tasks and groups
* pre-commit format
* add exit after error
* adjust how task objects are called
* no need to use get_task_dict
* load_task_or_group works but only for tasks
* pre-commit format
* half working for nested groups
* changed variable names
* allow groups and tasks to work
* temp save
* indexing and loading are part of a task_manager object
* adapted initialize_tasks
* iron out bugs
* fixed typo
* fixed typo
* simplified code
* further tidy up
* remove lines for testing
* removed test lines
* removed unused code
* remove unused import
* fixed bug
* removed comments
* group in a list of group can accept parameter changes like `num_fewshot`
* add trust_remote_code as default
* task for testing recursive
* changed source of ALL_TASKS
* tasks should only accept TaskObjects
* initialize_tasks returns list of tasks and groups
* remove trust_remote_code for now
* moved constructor process to inside load_yaml_config
* more comprehensive way to index tasks and groups
* pre-commit format
* add exit after error
* adjust how task objects are called
* no need to use get_task_dict
* load_task_or_group works but only for tasks
* pre-commit format
* half working for nested groups
* changed variable names
* allow groups and tasks to work
* temp save
* indexing and loading are part of a task_manager object
* adapted initialize_tasks
* iron out bugs
* fixed typo
* fixed typo
* simplified code
* further tidy up
* remove lines for testing
* removed test lines
* removed unused code
* remove unused import
* fixed bug
* removed comments
* group in a list of group can accept parameter changes like `num_fewshot`
* check if config is task update
* add GroupConfig object
* edit test yaml
* remove args
* testing returning to python task list
* add weight_by_size config
* describe weight_by_size in docs
* fix weight by size potential error
* can load individual custom python class task
* moved import_function into the config loading file
* remove print lines
* add squadv2 yaml
* temporary scroll implementation
* revert back to use load_yaml_config but with modes
* fix group being loaded with a None
* reformat
* can load unregistered tasks from a group
* update scrolls
* edit scrolls multiplechoice task
* adjust class initialization
* fix initialization
* changed how to identify group and python tasks, fix logger
* allow loading "include" that is nested in a group config
* reworked flan benchmark
* allow duplicate task in the same group to co-exist
* process group_alias
* removed group_alias
* allow parameters set in group_config to apply to all tasks in tasklist
* add function, but comment for now
* reworked processing dict-base config
* fixed how configs in group are processed
* update to allow root group to have its alias used
* remove unused classes
* remove unused classes
* revert some parts to original
* forgot to change one variable
* adapt the new process to use get_task_dict
* fix for singular group call
* fix variable names
* add TaskManager into the evaluator
* format
* changed how dict tasks are loaded
* add docs
* Update docs/new_task_guide.md
Co-authored-by: Hailey Schoelkopf <[email protected]>
* Update evaluator.py
* Update evaluator.py
* remove groupconfig for now
* changed _config to config
* update interface.md to explain TaskManager
* added property functions
* adjusted logger
* update write_out.py
* updated tests
* added documentation and some modifications
* added docstring documentation
* precommit format
* updated task loading for tests
* updates tests
* changed arg order for load_yaml_config
* update to handle scrolls and edit log message
* remove unused lines
* return a list of task classes and not a dict
* Update __init__.py
* Delete lm_eval/tasks/benchmarks/test.yaml
* Update task.py
* Update lm_eval/utils.py
Co-authored-by: Hailey Schoelkopf <[email protected]>
* Update lm_eval/utils.py
Co-authored-by: Hailey Schoelkopf <[email protected]>
* Update utils.py
* re-added old functions with new log message
* Update docs/new_task_guide.md
Co-authored-by: Hailey Schoelkopf <[email protected]>
* Update new_task_guide.md
* added infor regarding `get_task_dict` and documentation
* add get_config for Task
* pre-commit formatting
---------
Co-authored-by: Hailey Schoelkopf <[email protected]>
* add ruff and isort. remove black and flake8
* remove unnecessary dependencies
* remove dependency from table
* change order
* ran ruff
* check 3.9
* exclude evaluator
* update CI workflow
* use ruff config in pyproject.toml
* test
* add isort rules to ruff
* sort imports
* import `make_table`
* try stages for no-commit-to-branch
* turn on mypy for pre-commit
* test
* test
* test
* change no-commit-to-branch to default
* nits
* fixed dependency
Tai An