mirror of
https://github.com/linkedin/Liger-Kernel.git
synced 2026-10-12 02:17:47 +00:00
Steven Shimizu
8ce3b5310e
## Summary
- Make convergence tests more portable and easier to run by using
pre-tokenized data. This removes internal paths and allows users to not
have to download specific model tokenizers in a certain location.
- Since we're just testing convergence on a mini model with random
weights, the specific tokenizer doesn't really matter.
- Convergence tests also finish faster: **~95 sec -> ~60 sec**
Alternatives:
- Provide users the ability to configure paths to the different models
used in convergence test or HF token to download the tokenizer
(inconvenient to configure/download things run tests, not portable,
different tokenizer versions could break tests)
- Save the tokenizers in the repo (licensing issues)
- Save a small, completely OS tokenizer in the repo and use across all
tests (could do this, but also more performant to just pre-tokenize the
data)
## Testing Done
Ran convergence tests successfully
- [x] run `make test` to ensure correctness
- [x] run `make checkstyle` to ensure code style
- [x] run `make test-convergence` to ensure convergence
```
jobuser [ ~/Liger-Kernel ]$ make checkstyle
flake8 .; flake8_status=$?; \
isort .; isort_status=$?; \
black .; black_status=$?; \
if [ $flake8_status -ne 0 ] || [ $isort_status -ne 0 ] || [ $black_status -ne 0 ]; then \
exit 1; \
fi
Fixing /home/jobuser/Liger-Kernel/test/convergence/test_mini_models.py
Skipped 1 files
All done! ✨ 🍰 ✨
45 files left unchanged.
jobuser [ ~/Liger-Kernel ]$ make test
pytest --disable-warnings test/ --ignore=test/convergence
===================================================================================================================== test session starts ======================================================================================================================
platform linux -- Python 3.10.14, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/jobuser/Liger-Kernel
plugins: lipy-config-base-30.6.1, lipy-fabric-35.2.3, lipy-test-8.0.52, datadir-1.3.1, lipy-mp-34.4.191
collected 111 items
test/transformers/test_cross_entropy.py .......................................................... [ 52%]
test/transformers/test_fused_linear_cross_entropy.py ...... [ 57%]
test/transformers/test_geglu.py ........ [ 64%]
test/transformers/test_rms_norm.py ................ [ 79%]
test/transformers/test_rope.py ............ [ 90%]
test/transformers/test_swiglu.py ........ [ 97%]
test/transformers/test_transformers_monkey_patch.py . [ 98%]
test/triton/test_triton_monkey_patch.py .. [100%]
================================================================================================================ 111 passed in 60.81s (0:01:00) ================================================================================================================
jobuser [ ~/Liger-Kernel ]$ make test-convergence
HF_DATASETS_OFFLINE=1 pytest --disable-warnings test/convergence
===================================================================================================================== test session starts ======================================================================================================================
platform linux -- Python 3.10.14, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/jobuser/Liger-Kernel
plugins: lipy-config-base-30.6.1, lipy-fabric-35.2.3, lipy-test-8.0.52, datadir-1.3.1, lipy-mp-34.4.191
collected 8 items
test/convergence/test_mini_models.py ...... [ 75%]
test/convergence/test_mini_models_no_logits.py .. [100%]
====================================================================================================================== 8 passed in 58.41s ======================================================================================================================
```
13 lines
250 B
JSON
13 lines
250 B
JSON
{
|
|
"_data_files": [
|
|
{
|
|
"filename": "data-00000-of-00001.arrow"
|
|
}
|
|
],
|
|
"_fingerprint": "9a3c013ec22d1ce0",
|
|
"_format_columns": null,
|
|
"_format_kwargs": {},
|
|
"_format_type": null,
|
|
"_output_all_columns": false,
|
|
"_split": "train"
|
|
} |