Steven Shimizu b5672a153b Refactored benchmark tests (#196) ...

## Summary
- Refactored kernel benchmarking tests to save data to a single CSV
(`benchmark/data/all_benchmark_data.csv`) that contains more complete
information on how the benchmarking test was setup:
```
kernel_name (e.g. swiglu)
kernel_provider (e.g. liger or huggingface)
kernel_operation_mode (e.g. full, forward, backward)
metric_name (e.g. speed, memory)
metric_unit (e.g. MB)
x_name
x_label (e.g. hidden size)
x_value
y_value_50 (median)
y_value_20 (20th percentile)
y_value_80 (80th percnetile)
extra_benchmark_config_str (e.g. {""B"": 32, ""T"": 512, ""D"": 768, ""dtype"": ""torch.float32""})
gpu_name (e.g. NVIDIA A100-SXM4-80GB)
timestamp
liger_version (e.g. 0.2.1)
```
- Removed existing image files to not clutter up the git history per
previous discussion (https://github.com/linkedin/Liger-Kernel/pull/54)
- Added sample jupyter notebook that can be used to filter and plot the
benchmark data
- Added makefile command to run all benchmarks: `make run-benchmarks`
- Don't overwrite existing data by default if same benchmark. Same
benchmark means same kernel_name, kernel_provider,
kernel_operation_mode, metric_name, x_label, gpu_name, and
extra_benchmark_config_str

## Testing Done
- Ran all benchmarks and spot-checked values with existing benchmark
data


![image](https://github.com/user-attachments/assets/886a2fb1-72ae-4eab-a486-a1c9fc7a1957)

- Hardware Type: A100
- [x] run `make test` to ensure correctness
- [x] run `make checkstyle` to ensure code style
- [x] run `make test-convergence` to ensure convergence

2024-09-03 22:29:00 -07:00

0 lines

0 B

Python

Raw Permalink Blame History

The file is empty.