EleutherAI/lm-evaluation-harness
1
0
Fork 0
mirror of https://github.com/EleutherAI/lm-evaluation-harness.git synced 2026-10-12 02:02:05 +00:00
Code Issues Packages Projects Releases Wiki Activity

Support jinja templating for task descriptions (#1553)

Browse Source 
* Support jinja templating for "description"

* Update task_guide.md

* Update lm_eval/api/task.py

* fix format?

* whitespace errors

* fix whitespace

* fix bad variable reference

---------

Co-authored-by: Hailey Schoelkopf <[email protected]>
Co-authored-by: haileyschoelkopf <[email protected]>
This commit is contained in:
Hisham Alyahya
2024-03-10 16:07:27 -04:00
committed by GitHub
co-authored by Hailey Schoelkopf haileyschoelkopf
parent f518228f25
commit 3bdf25ec00
2 changed files with 8 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/task_guide.md
+4 -3
View File
@@ -30,9 +30,10 @@ Dataset configuration options:
Prompting / in-context formatting options:
- **use_prompt** (`str`, *optional*) — Name of prompt in promptsource to use. if defined, will overwrite doc_to_text, doc_to_target, and doc_to_choice.
- **doc_to_text** (`Union[Callable, str]`, *optional*) — Jinja2, f-string, or function to process a sample into the appropriate input for the model
- **doc_to_target** (`Union[Callable, str]`, *optional*) — Jinja2, f-string, or function to process a sample into the appropriate target output for the model. For multiple choice tasks, this should return an index into
- **doc_to_choice** (`Union[Callable, str]`, *optional*) — Jinja2, f-string, or function to process a sample into a list of possible string choices for `multiple_choice` tasks. Left undefined for `generate_until` tasks.
- **description** (`str`, *optional*) — An optional prepended Jinja2 template or string which will be prepended to the few-shot examples passed into the model, often describing the task or providing instructions to a model, such as `"The following are questions (with answers) about {{subject}}.\n\n"`. No delimiters or spacing are inserted between the description and the first few-shot example.
- **doc_to_text** (`Union[Callable, str]`, *optional*) — Jinja2 template, string, or function to process a sample into the appropriate input for the model
- **doc_to_target** (`Union[Callable, str]`, *optional*) — Jinja2 template, string, or function to process a sample into the appropriate target output for the model. For multiple choice tasks, this should return an index into
- **doc_to_choice** (`Union[Callable, str]`, *optional*) — Jinja2 template, string, or function to process a sample into a list of possible string choices for `multiple_choice` tasks. Left undefined for `generate_until` tasks.
- **fewshot_delimiter** (`str`, *optional*, defaults to "\n\n") — String to insert between few-shot examples.
- **target_delimiter** (`str`, *optional*, defaults to `" "`) — String to insert between input and target output for the datapoint being tested.
lm_eval/api/task.py
+4 -4
View File
@@ -940,14 +940,14 @@ class ConfigurableTask(Task):
:returns: str
The fewshot context.
"""
if description := self.config.description:
description = utils.apply_template(self.config.description, doc)
if num_fewshot == 0:
# always prepend the (possibly empty) task description
labeled_examples = self.config.description
labeled_examples = description
else:
labeled_examples = self.config.description + self.sampler.get_context(
doc, num_fewshot
)
labeled_examples = description + self.sampler.get_context(doc, num_fewshot)
example = self.doc_to_text(doc)
if self.multiple_input: