mirror of
https://github.com/linkedin/Liger-Kernel.git
synced 2026-10-12 02:17:47 +00:00
Parag Ekbote
38aac93ea6
## Summary Fixes #64 Instead of using Sphinx which I found to be cumbersome to set-up and iterate upon, I have created the docs using Material for Markdown, which uses markdown files for pages and does not change the original markdown files to a greater extent. To preview in your local system: run make serve in the terminal. To build pages: run make build in the terminal. Index page:  Examples page:  Please let me know if any styling or further corrections are needed and I will make the necessary changes. cc: @ByronHsu
2.0 KiB
2.0 KiB
There are a couple of ways to apply Liger kernels, depending on the level of customization required.
1. Use AutoLigerKernelForCausalLM
Using the AutoLigerKernelForCausalLM is the simplest approach, as you don't have to import a model-specific patching API. If the model type is supported, the modeling code will be automatically patched using the default settings.
!!! Example
from liger_kernel.transformers import AutoLigerKernelForCausalLM
# This AutoModel wrapper class automatically monkey-patches the
# model with the optimized Liger kernels if the model is supported.
model = AutoLigerKernelForCausalLM.from_pretrained("path/to/some/model")
2. Apply Model-Specific Patching APIs
Using the patching APIs, you can swap Hugging Face models with optimized Liger Kernels.
!!! Example
import transformers
from liger_kernel.transformers import apply_liger_kernel_to_llama
# 1a. Adding this line automatically monkey-patches the model with the optimized Liger kernels
apply_liger_kernel_to_llama()
# 1b. You could alternatively specify exactly which kernels are applied
apply_liger_kernel_to_llama(
rope=True,
swiglu=True,
cross_entropy=True,
fused_linear_cross_entropy=False,
rms_norm=False
)
# 2. Instantiate patched model
model = transformers.AutoModelForCausalLM("path/to/llama/model")
3. Compose Your Own Model
You can take individual kernels to compose your models.
!!! Example
from liger_kernel.transformers import LigerFusedLinearCrossEntropyLoss
import torch.nn as nn
import torch
model = nn.Linear(128, 256).cuda()
# fuses linear + cross entropy layers together and performs chunk-by-chunk computation to reduce memory
loss_fn = LigerFusedLinearCrossEntropyLoss()
input = torch.randn(4, 128, requires_grad=True, device="cuda")
target = torch.randint(256, (4, ), device="cuda")
loss = loss_fn(model.weight, input, target)
loss.backward()