mirror of
https://github.com/linkedin/Liger-Kernel.git
synced 2026-10-12 02:17:47 +00:00
13e1bbe59b
## Summary Add Liger kernel support for EXAONE4 models (LG AI Research's EXAONE 4.0 series). ### Changes - Add `src/liger_kernel/transformers/model/exaone4.py` with fused linear cross entropy forward - Add `apply_liger_kernel_to_exaone4()` function in `monkey_patch.py` - Register in `__init__.py` - Add `revert_liger_kernel_to_exaone4()` in `test/utils.py` - Add convergence tests in `test/convergence/bf16/` and `test/convergence/fp32/` ### Supported Kernels - RMSNorm (including QK-Norm in attention) - SwiGLU MLP - RoPE - Fused Linear Cross Entropy ### Note on in_place=False for RMSNorm EXAONE4 requires `in_place=False` for RMSNorm due to its attention implementation pattern: ```python # EXAONE4 pattern - separate assignment query_states = self.q_proj(hidden_states).view(...).transpose(1, 2) query_states = self.q_norm(query_states) # reassignment to same variable # vs Qwen3 pattern - chained query_states = self.q_norm(self.q_proj(hidden_states).view(...)).transpose(1, 2) ``` The view/transpose operations share storage with the original tensor. In-place modification corrupts the autograd graph, causing NaN gradients. ### References - Model: https://huggingface.co/LGAI-EXAONE/EXAONE-4.0-1.2B ## Testing Done - Hardware Type: H100 - [x] run `make test` to ensure correctness - [x] run `make checkstyle` to ensure code style - [x] run `make test-convergence` to ensure convergence --------- Co-authored-by: Sunghyun Cho <[email protected]> Co-authored-by: Steven Shimizu <[email protected]>