mirror of
https://github.com/linkedin/Liger-Kernel.git
synced 2026-10-12 02:17:47 +00:00
main
14
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Buffett LiuandZhipeng Wang
|
e6a81bb0c3 |
Add Liger Kernel support for Muse Glimmer (#1390)
## Summary Adds Liger Kernel support for Muse Glimmer (`model_type=muse_glimmer`), covering both capabilities requested in #1378: 1. FusedLinearCrossEntropy / skip-logits loss path, avoiding the `[B, S, V]` logits tensor (Muse Glimmer's `vocab_size` is 202,048) 2. `return_token_accuracy=True` → `outputs.token_accuracy`, so TRL's `SFTTrainer` can log `mean_token_accuracy` without a trainer-side fallback Registered in `MODEL_TYPE_TO_APPLY_LIGER_FN`, so `use_liger_kernel=True` picks it up. Patches RoPE, RMSNorm, SwiGLU, vision LayerNorm (instance path only), CrossEntropy and FusedLinearCrossEntropy. ## Details **Softcap.** HF computes `T * tanh(logits * m / T)` where `m = output_multiplier` and `T = final_logit_softcapping`, but Liger's fused softcap only implements `T * tanh(logits / T)`. `m` is folded into the hidden states instead, since `(m * h) @ W.T == m * (h @ W.T)` — scaling `[batch, seq, hidden]` rather than `[vocab, hidden]`. **Two RMSNorm variants.** `MuseGlimmerRMSNorm` scales by `weight` directly (gemma-style fp32 compute, offset 0); `MuseGlimmerTextCenteredRMSNorm` scales by `(1 + weight)` from zero-init (offset 1). The `with_scale=False` instances, `qk_norm`, `embed_norm`, `perception_emb_norm`, have no weight parameter and fall back to a torch implementation matching HF exactly; an assert guards against that fallback silently dropping a scale. **`in_place=False`** on the TextCentered norms is required, not conservative: each decoder layer hands `pre_feedforward_layernorm` the same tensor it retains as the residual. Note that `token_accuracy` is populated on the fused path only; with `skip_logits=False` (eval) it stays `None`, matching existing patched models such as Qwen3. ## Testing Done - `test/transformers/test_muse_glimmer.py` (new) — instance-path numerical parity against the unpatched model: loss plus embedding / SwiGLU / RMSNorm gradients in fp32 and bf16, the non-fused softcap branch, vision LayerNorm, and the `token_accuracy` / `predicted_tokens` contract. The convergence suite patches at module level, so this is what covers `_apply_liger_kernel_to_instance` — the path TRL actually takes. - `test/transformers/test_monkey_patch.py` — instance patching - Convergence in bf16 and fp32: `test_mini_models`, `test_mini_models_multimodal`, `test_mini_models_with_logits` ### Memory H100, bf16, forward + backward, no optimizer step. Randomly initialised (no public checkpoint), real `hidden_size=6656` / `vocab_size=202048`, `num_hidden_layers` reduced from 52 so a 30B-class model fits on one GPU. | seq_len | peak mem (HF) | peak mem (Liger) | saved | |--------:|--------------:|-----------------:|------:| | 1024 | 18.4 GB | 20.0 GB | -9% | | 2048 | 19.6 GB | 21.2 GB | -8% | | 4096 | 30.4 GB | 23.5 GB | 23% | | 8192 | 52.0 GB | 28.2 GB | 46% | | 16384 | OOM | 38.2 GB | — | The crossover sits near 3K: FLCE accumulates `grad_weight` in fp32, a fixed 5.4 GB for `[202048, 6656]` against bf16's 2.7 GB. Below ~3K that outweighs the logits it avoids; above it the saving compounds, and 16K only runs at all with Liger. ### Throughput FLCE trades speed for memory by design. End to end that penalty is diluted by the transformer body, and the reduced depth above overstates it: | layers | lm_head share | tok/s (HF) | tok/s (Liger) | ratio | |-------:|--------------:|-----------:|--------------:|------:| | 4 | 41% | 22,405 | 13,433 | 0.60x | | 8 | 26% | 13,685 | 10,396 | 0.76x | | 16 | 15% | 7,705 | 7,063 | 0.92x | `lm_head share` is measured from the model's parameter split, which tracks the matmul FLOP split. The shipping model's 52 layers put it at ~5%, below the last row here. - Hardware Type: <BLANK> - [x] run `make test` to ensure correctness - [x] run `make checkstyle` to ensure code style - [x] run `make test-convergence` to ensure convergence Co-authored-by: Zhipeng Wang <[email protected]> |
||
|
Dimitar DimitrovandClaude Opus 4.7
|
decb1b73d7 |
[Gemma 4] Add multimodal support (apply_liger_kernel_to_gemma4 for Gemma4ForConditionalGeneration) (#1203)
## Summary Follow-up to #1196 (the text path) — adds `apply_liger_kernel_to_gemma4` for `Gemma4ForConditionalGeneration` (multimodal class; includes E2B / E4B / E4B-it which are loaded by `AutoModelForCausalLM` as the multimodal class even when only text is being trained). Closes the multimodal half of #1186. ### Why Gemma 4's text vocab is 262,144. Without FLCE the (B, T, V) bf16 logits tensor is ~17 GB at T=8192 (and ~34 GB once the loss path upcasts to fp32 for cross-entropy), which OOMs even 96 GB cards on `Gemma4ForConditionalGeneration` SFT — the OOM that originally motivated #1186. Routing loss through `LigerForCausalLMLoss` materializes only the loss scalar. ### Shape A single unified entry point that dispatches on class, per @Mecoli1219's preference in #1186: - `apply_liger_kernel_to_gemma4(model=Gemma4ForConditionalGeneration_instance)` — multimodal path. Class-level RMSNorm + GeGLU swaps via `apply_liger_kernel_to_gemma4_text`, FLCE forward via the new `multimodal_forward`, recurses into `model.model.language_model` for instance-level patches. - `apply_liger_kernel_to_gemma4(model=Gemma4ForCausalLM_instance)` — routes to `apply_liger_kernel_to_gemma4_text` for backwards compatibility, so the same entry point works for either shape. - Registry: adds `"gemma4": apply_liger_kernel_to_gemma4` alongside the existing `"gemma4_text"`. ### Drive-by fixes Two `isinstance(model, tuple_with_None_filter)` sites (one in our new dispatcher, one in the existing `apply_liger_kernel_to_gemma4_text`) raised `TypeError: isinstance() arg 2 must be a type, a tuple of types, or a union` when called under `with patch("transformers.models.gemma4.modeling_gemma4")`: `getattr(MagicMock_module, "Gemma4TextForCausalLM", None)` returns a MagicMock (not `None`) because `MagicMock` auto-creates attributes, so the `cls is not None` filter let it slip into the `isinstance` tuple. The text-path version was dormant — its existing test passes a `Gemma4ForCausalLM` which short-circuits the `isinstance` match before reaching the bad entry — but the multimodal recursive call into the text path passes a `Gemma4TextModel`, so no early match. Both sites now use `isinstance(cls, type)` as the filter. ### Out of scope (deferred) - **Vision / audio tower kernel swaps.** Gemma 4 loads both via `AutoModel.from_config(config.{vision,audio}_config)`, so the module classes are polymorphic. Out of scope here; FLCE on the LM head is what unblocks training OOM. - **`Gemma4MultimodalEmbedder` / projector norms.** Analogous to gemma3's `mm_soft_emb_norm` patching. Skipped for the same minimal-surface reason. - **PLE (Per-Layer Embeddings).** State passes through the inner forward unchanged; verified end-to-end on `google/gemma-4-E4B-it`. - **MoE experts (`Gemma4TextExperts`).** Guarded out via the same `enable_moe_block` check used by the text path in #1196. - **Multimodal mini convergence test (`mini_gemma4`).** Would need new `test/resources/fake_configs/Google/Gemma4/...` scaffolding for an image / audio processor — PR #1196 followed the same pattern (only added `mini_gemma4_text` to the non-multimodal convergence files). Happy to add it as a follow-up if you'd prefer it bundled here — let me know. ## Testing Done ### Hardware - **Hardware Type: RTX 5090 (Blackwell, sm_120, 32 GB), Vast.ai instance** - CUDA 13.0, NVIDIA driver 590.48.01 - Python 3.10.12, torch 2.11.0+cu130, transformers 5.7.0.dev0 (built from `huggingface/transformers@main` — gemma4 requires ≥ 5.5.0) - bf16 throughout for end-to-end numerics ### End-to-end numerical equivalence on real `google/gemma-4-E4B-it` Verified before authoring this PR with our internal `verify_patch_equivalence.py` (same shape as #1196's verification): | Axis | Threshold | Measured | | --- | --- | --- | | Fused-CE vs reference loss diff (seq=512) | `< 5e-3` | `0.0016` | | End-to-end top-1 token agreement (seq=256, patched vs stock SDPA) | `> 99 %` | **>99 %** | | End-to-end loss diff (patched vs stock SDPA) | `< 5e-3` | `0.0016` | | Logit-distribution KL (informational) | — | `~3.5e-2` | | SDPA EFFICIENT_ATTENTION vs MATH cos-sim (seq=4096) | `> 0.9999` | **>0.9999** | ### Liger-Kernel test gates - [x] `make checkstyle` — `All checks passed!`, 267 files already formatted - [x] `make test` — see log - [x] `make test-convergence` — see logs (per file) #### `make test` `3131 passed, 903 skipped, 12 xfailed, 3 failed` in 35:42. Our two new unit tests pass cleanly: ``` PASSED test/transformers/test_monkey_patch.py::test_apply_liger_kernel_to_instance_for_gemma4_text (0.70s) PASSED test/transformers/test_monkey_patch.py::test_apply_liger_kernel_to_instance_for_gemma4_conditional_generation (0.22s) ``` All six `LigerGEGLUMLPForGemma4` edge-case tests added by #1196 also pass. The 3 unrelated failures are pre-existing on the parent branch (untouched by this PR) and reproduce on `main`: ``` FAILED test/transformers/test_grpo_loss.py::test_grpo_loss_with_bias_correction_kl[...] FAILED test/transformers/test_monkey_patch.py::test_apply_liger_kernel_to_instance_for_paligemma FAILED test/transformers/test_monkey_patch.py::test_apply_liger_kernel_to_instance_for_gemma3_conditional_generation ``` #### `make test-convergence` (per file) | File | Result | | --- | --- | | `fp32/test_mini_models.py` | 32 passed, 3 skipped | | `fp32/test_mini_models_multimodal.py` | 8 passed, 4 skipped, 1 xfailed, **1 failed** (`mini_qwen2_vl`) | | `fp32/test_mini_models_with_logits.py` | 29 passed, 3 skipped, 1 xpassed | | `bf16/test_mini_models.py` | 33 passed, **2 failed** (`mini_llama4`, `mini_gemma4_text`) | | `bf16/test_mini_models_multimodal.py` | 11 passed, 1 skipped, **2 failed** (`mini_qwen2_vl`, `mini_llama4`) | | `bf16/test_mini_models_with_logits.py` | 30 passed, 1 skipped, **2 failed** (`mini_llama4`, `mini_qwen3_moe`) | `mini_gemma4_text` passes in `bf16/test_mini_models_with_logits.py` and `fp32/test_mini_models.py` (PR #1196's text path). It fails only in `bf16/test_mini_models.py` with the same Blackwell bf16 logprob drift @eqy and reviewers flagged on #1196 (review comment `r4321013177`); this is in PR #1196's territory, not introduced by this multimodal patch. The other failures (`mini_llama4`, `mini_qwen2_vl`, `mini_qwen3_moe`) are also in models we don't touch and are pre-existing test instability on consumer Blackwell. cc @Mecoli1219 @lardinator @ruilin-gif --- 🤖 Drafted with [Claude Code](https://claude.com/claude-code) (Claude Opus 4.7), reviewed and posted by me. --------- Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]> |
||
|
|
6cd9fd64a5 |
Support for Qwen3-VL models (#911)
## Summary Support for Qwen3-VL models Solves https://github.com/linkedin/Liger-Kernel/issues/897 The branch currently already has LCE op implemented with all tests passing for qwen3VL. Currently tests are failing for qwen3VL moe I will be adding ops in order of difficulty. Please reach out if you want an op prioritized Update : Current status: Qwen3VL supported for rope, cross entropy, rmsnorm, flce -> All tests passing Qwen3VLMoe supported for rope, cross entropy, rmsnorm, flce -> Convergence failing for the two following tests ``` FAILED test/convergence/fp32/test_mini_models_multimodal.py::test_mini_model_multimodal[mini_qwen3_vl_moe-32-0.0001-dtype5-1e-08-1e-05-0.005-1e-05-0.005-1e-05] - AssertionError: [Loss]Number of mismatched elements: 28 FAILED test/convergence/bf16/test_mini_models_with_logits.py::test_mini_model[mini_qwen3_vl_moe-32-1e-05-dtype9-0.01-0.05-0.1-0.01-0.01-0.01] - AssertionError: [Top k logprobs]Number of mismatched elements: 1 ``` ## Details NA ## Testing Done NA <!-- Replace BLANK with your device type. For example, A100-80G-PCIe Complete the following tasks before sending your PR, and replace `[ ]` with `[x]` to indicate you have done them. --> - Hardware Type: <BLANK> - [ ] run `make test` to ensure correctness - [ ] run `make checkstyle` to ensure code style - [ ] run `make test-convergence` to ensure convergence --------- Co-authored-by: Mayank Agarwal <[email protected]> Co-authored-by: Steven Shimizu <[email protected]> |
||
|
MilkCloudsandTcc0403
|
f7c17b586a |
Add SmolVLM2 support (#919)
## Summary This PR adds support for SmolVLM2 to liger-kernel. SmolVLM2 is present on transformers after 4.50.0 version: https://github.com/huggingface/transformers/releases/tag/v4.50.0 ## Testing Done <!-- Replace BLANK with your device type. For example, A100-80G-PCIe Complete the following tasks before sending your PR, and replace `[ ]` with `[x]` to indicate you have done them. --> - Hardware Type: A100 - [x] run `make test` to ensure correctness ``` (4480 durations < 0.005s hidden. Use -vv to show these durations.) =============================================================================================== 2142 passed, 215 skipped, 45 warnings in 1941.33s (0:32:21) =============================================================================================== ``` - [x] run `make checkstyle` to ensure code style - [x] run `make test-convergence` to ensure convergence ``` (35 durations < 0.005s hidden. Use -vv to show these durations.) ================================================================================================================= short test summary info ================================================================================================================= FAILED test/convergence/bf16/test_mini_models_with_logits.py::test_mini_model[mini_glm4v_moe-32-1e-05-dtype17-0.01-0.01-0.1-0.01-0.01-0.01] - AssertionError: [Loss]Number of mismatched elements: 2 =================================================================================================== 1 failed, 22 passed, 1 warning in 262.08s (0:04:22) =================================================================================================== make: *** [Makefile:36: test-convergence] Error 1 ``` There's no test failure for smolvlm2! --------- Co-authored-by: Tcc0403 <[email protected]> |
||
|
|
9584b94119 |
Add InternVL3 support (#878)
## Summary This PR added InternVL3 models(e.g. https://huggingface.co/OpenGVLab/InternVL3-1B-hf) to liger-kernel. From InternVL2.5-MPO(e.g. https://huggingface.co/OpenGVLab/InternVL2_5-8B-MPO-hf) to InternVL3.5 they share same model type `internvl` but I focused on supporting InternVL3. Other versions may work but not sure. This model is present after transformers v4.52.1: https://github.com/huggingface/transformers/releases/tag/v4.52.1 ## Testing Done - Hardware Type: H100 - [x] run `make test` to ensure correctness: all passed ``` ====================================================================================================== 2072 passed, 215 skipped, 44 warnings in 918.28s (0:15:18) ======================================================================================================= ``` - [x] run `make checkstyle` to ensure code style - [x] run `make test-convergence` to ensure convergence: glm fail, not sure why ``` ======================================================================================================================== short test summary info ======================================================================================================================== FAILED test/convergence/bf16/test_mini_models.py::test_mini_model[mini_glm4v-32-1e-05-dtype14-0.01-0.01-0.1-0.01-0.01-0.01] - AssertionError: [Loss]Number of mismatched elements: 3 FAILED test/convergence/bf16/test_mini_models.py::test_mini_model[mini_glm4v_moe-32-1e-05-dtype15-0.01-0.2-0.1-0.01-0.01-0.01] - AssertionError: [Loss]Number of mismatched elements: 2 ========================================================================================================== 2 failed, 19 passed, 1 warning in 117.34s (0:01:57) ========================================================================================================== ``` ## Important note for testing Existing test failed for multimodal model test, so I fixed some implementation, not sure I've done right. Please have a look and give me review. https://github.com/linkedin/Liger-Kernel/pull/878/files#diff-76dbff2421c7cc7c9833d3bfaa5e1c55da5ad7ce96087d356aefb7d340acaf5bR954-R962 --------- Co-authored-by: Shao Tang <[email protected]> Co-authored-by: Vaibhav Jindal <[email protected]> |
||
|
|
8e231a3538 |
Liger support for Llama4 (#740)
## Summary <!--- This is a required section; please describe the main purpose of this proposed code change. ---> Adding liger support to llama4 (text and multimodal). Liger rope is not supported yet for llama4 as it has a different implementation. The tolerance are set higher than usual to pass the tests. <!--- ## Details This is an optional section; is there anything specific that reviewers should be aware of? ---> ## Testing Done <!--- This is a required section; please describe how this change was tested. ---> <!-- Replace BLANK with your device type. For example, A100-80G-PCIe Complete the following tasks before sending your PR, and replace `[ ]` with `[x]` to indicate you have done them. --> - Hardware Type: NVIDIA H100 80GB HBM3 - [ ] run `make test` to ensure correctness - [x] run `make checkstyle` to ensure code style - [ ] run `make test-convergence` to ensure convergence --------- Co-authored-by: Shao Tang <[email protected]> Co-authored-by: Manan Shah <[email protected]> Co-authored-by: Vaibhav Jindal <[email protected]> |
||
|
AbdelKarim ELJANDOUBI
|
799a14d053 |
Gemma3 (Text and Multimodal) (#621)
## Summary This PR adds Gemm3 (Text and Multimodal) support to the Liger-Kernel. You need to install transformers>=4.50.0 ## Testing Done - Hardware Type: RTX 3070 - [x] run `make test` to ensure correctness - [x] run `make checkstyle` to ensure code style - [x] run `make test-convergence` to ensure convergence |
||
|
|
f49da8e8c1 |
Add huggingface llava (#524)
## Summary #514 transformer > pip install git+https://github.com/huggingface/transformers.git@5fa35344755d8d9c29610b57d175efd03776ae9e ## Testing Done <!--- This is a required section; please describe how this change was tested. ---> <!-- Replace BLANK with your device type. For example, A100-80G-PCIe Complete the following tasks before sending your PR, and replace `[ ]` with `[x]` to indicate you have done them. --> <details> <summary>huggingface-env</summary> - transformers: 4.49.0.dev0 - huggingface_hub version: 0.26.2 - Platform: Linux-5.15.0-124-generic-x86_64-with-glibc2.35 - Python version: 3.10.12 - Running in iPython ?: No - Running in notebook ?: No - Running in Google Colab ?: No - Running in Google Colab Enterprise ?: No - Token path ?: /root/.cache/huggingface/token - Has saved token ?: True - Who am I ?: jp1924 - Configured git credential helpers: - FastAI: N/A - Tensorflow: N/A - Torch: 2.5.1+cu121 - Jinja2: 3.1.4 - Graphviz: N/A - keras: N/A - Pydot: N/A - Pillow: 11.0.0 - hf_transfer: N/A - gradio: N/A - tensorboard: N/A - numpy: 2.1.3 - pydantic: 2.9.2 - aiohttp: 3.11.6 - ENDPOINT: https://huggingface.co - HF_HUB_CACHE: /root/.cache/huggingface/hub - HF_ASSETS_CACHE: /root/.cache/huggingface/assets - HF_TOKEN_PATH: /root/.cache/huggingface/token - HF_STORED_TOKENS_PATH: /root/.cache/huggingface/stored_tokens - HF_HUB_OFFLINE: False - HF_HUB_DISABLE_TELEMETRY: False - HF_HUB_DISABLE_PROGRESS_BARS: None - HF_HUB_DISABLE_SYMLINKS_WARNING: False - HF_HUB_DISABLE_EXPERIMENTAL_WARNING: False - HF_HUB_DISABLE_IMPLICIT_TOKEN: False - HF_HUB_ENABLE_HF_TRANSFER: False - HF_HUB_ETAG_TIMEOUT: 10 - HF_HUB_DOWNLOAD_TIMEOUT: 10 </details> <details> <summary>torch&hw-env</summary> Collecting environment information... PyTorch version: 2.5.1+cu121 Is debug build: False CUDA used to build PyTorch: 12.1 ROCM used to build PyTorch: N/A OS: Ubuntu 22.04.3 LTS (x86_64) GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 Clang version: Could not collect CMake version: Could not collect Libc version: glibc-2.35 Python version: 3.10.12 (main, Jul 29 2024, 16:56:48) [GCC 11.4.0] (64-bit runtime) Python platform: Linux-5.15.0-124-generic-x86_64-with-glibc2.35 Is CUDA available: True CUDA runtime version: 12.1.105 CUDA_MODULE_LOADING set to: LAZY GPU models and configuration: GPU 0: NVIDIA A100 80GB PCIe GPU 1: NVIDIA A100 80GB PCIe GPU 2: NVIDIA A100 80GB PCIe GPU 3: NVIDIA A100 80GB PCIe Nvidia driver version: 560.35.03 cuDNN version: Probably one of the following: /usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.0 /usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.0 /usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.0 /usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.0 /usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.0 /usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.0 /usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.0 HIP runtime version: N/A MIOpen runtime version: N/A Is XNNPACK available: True CPU: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 46 bits physical, 57 bits virtual Byte Order: Little Endian CPU(s): 48 On-line CPU(s) list: 0-47 Vendor ID: GenuineIntel Model name: Intel(R) Xeon(R) Silver 4310 CPU @ 2.10GHz CPU family: 6 Model: 106 Thread(s) per core: 2 Core(s) per socket: 12 Socket(s): 2 Stepping: 6 CPU max MHz: 3300.0000 CPU min MHz: 800.0000 BogoMIPS: 4200.00 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 invpcid_single ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local split_lock_detect wbnoinvd dtherm ida arat pln pts avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq la57 rdpid fsrm md_clear pconfig flush_l1d arch_capabilities Virtualization: VT-x L1d cache: 1.1 MiB (24 instances) L1i cache: 768 KiB (24 instances) L2 cache: 30 MiB (24 instances) L3 cache: 36 MiB (2 instances) NUMA node(s): 2 NUMA node0 CPU(s): 0-11,24-35 NUMA node1 CPU(s): 12-23,36-47 Vulnerability Gather data sampling: Mitigation; Microcode Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Mitigation; Clear CPU buffers; SMT vulnerable Vulnerability Reg file data sampling: Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; RSB filling; PBRSB-eIBRS SW sequence; BHI SW loop, KVM SW loop Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Not affected Versions of relevant libraries: [pip3] numpy==2.1.3 [pip3] nvidia-cublas-cu12==12.1.3.1 [pip3] nvidia-cuda-cupti-cu12==12.1.105 [pip3] nvidia-cuda-nvrtc-cu12==12.1.105 [pip3] nvidia-cuda-runtime-cu12==12.1.105 [pip3] nvidia-cudnn-cu12==9.1.0.70 [pip3] nvidia-cufft-cu12==11.0.2.54 [pip3] nvidia-curand-cu12==10.3.2.106 [pip3] nvidia-cusolver-cu12==11.4.5.107 [pip3] nvidia-cusparse-cu12==12.1.0.106 [pip3] nvidia-nccl-cu12==2.21.5 [pip3] nvidia-nvjitlink-cu12==12.4.127 [pip3] nvidia-nvtx-cu12==12.1.105 [pip3] torch==2.5.1+cu121 [pip3] torchvision==0.20.1+cu121 [pip3] triton==3.1.0 [conda] Could not collect </details> - [x] run `make test` to ensure correctness - [x] run `make checkstyle` to ensure code style - [x] run `make test-convergence` to ensure convergence --------- Co-authored-by: Shao Tang <[email protected]> Co-authored-by: Tcc0403 <[email protected]> |
||
|
AbdelKarim ELJANDOUBIandShao Tang
|
cc0c945eb2 |
Paligemma support (#608)
## Summary This PR adds Paligemma support to the Liger-Kernel. ## Testing Done - Hardware Type: RTX 3070 - [x] run `make test` to ensure correctness - [x] run `make checkstyle` to ensure code style - [x] run `make test-convergence` to ensure convergence --------- Co-authored-by: Shao Tang <[email protected]> |
||
|
|
9c20bd4156 |
Monkeypatch for Qwen2.5-VL (#552)
## Summary <!--- This is a required section; please describe the main purpose of this proposed code change. ---> [Qwen2.5-VL](https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct) was released a few days ago. This PR aims to add support for it. There are some small changes in its architecture compared to Qwen2-VL. ## Details As of February 1, the latest git version of `transformers` ([62db3e6](https://github.com/huggingface/transformers/commit/62db3e6ed67a74cc1ed1436acd9973915c0a4475)) supports it, but it's not yet in the latest release of `transformers`. Needs help with the tests, as I got OOMs on my laptop GPU. I have tried LoRA with PR, which works great. ## Testing Done <!--- This is a required section; please describe how this change was tested. ---> Can't run the tests on my hardware, got OOMs. <!-- Replace BLANK with your device type. For example, A100-80G-PCIe Complete the following tasks before sending your PR, and replace `[ ]` with `[x]` to indicate you have done them. --> - Hardware Type: RTX 4070 laptop - [ ] run `make test` to ensure correctness - [ ] run `make checkstyle` to ensure code style - [ ] run `make test-convergence` to ensure convergence --------- Co-authored-by: Tyler Romero <[email protected]> Co-authored-by: Tyler Romero <[email protected]> Co-authored-by: Shao Tang <[email protected]> Co-authored-by: shivam15s <[email protected]> |
||
|
Shivam Sahni
|
77ff1a9b7d | Move the checkstyle to [Ruff](https://docs.astral.sh/ruff/) (#483) | ||
|
Tyler Romero
|
dc74fa401b |
Qwen2-VL Bug / Incompatibility Fixes (#388)
## Summary Two fixes: 1. https://github.com/linkedin/Liger-Kernel/pull/276/files created a backwards-incompatible change. So that when users use transformers < 4.47.0.dev0 they will see different behavior from liger (which mimics behavior of 4.47.0.dev0). By adding a conditional we can support the behavior of all versions of transformers that include qwen2_vl. 2. Recent versions of qwen2_vl from transformers make various qwen2_vl specific tokens mandatory to specify, so those are now specified to avoid AttributeErrors. ## Testing Done Tested locally against both transformers 4.46.2 and transformers 4.47.0.dev0 (from git+https://github.com/huggingface/transformers.git@52ea4aa589324bae43dfb1b6db70335da7b68654) - Hardware Type: RTX 4090 - [x] run `make test` to ensure correctness - [x] run `make checkstyle` to ensure code style - [x] run `make test-convergence` to ensure convergence |
||
|
|
9b10f48f17 |
Monkeypatch for Llama 3.2-Vision (#282)
## Summary Add monkeypatch to support [Llama 3.2-Vision](https://github.com/huggingface/transformers/pull/33703/files#diff-d804e851851cdebeb8048938f1f8beec1cfa78bf7b1f06af86faa450f9d18def) models. ## Details Llama 3.2-Vision is a multimodal model. It is also only available in transformers>=4.45.0. Torchvision is required to run the multimodal tests for Llama 3.2-Vision (the image processor requires it). ## Testing Done - Hardware Type: RTX 4090 - [x] run `make test` to ensure correctness - [x] run `make checkstyle` to ensure code style - [x] run `make test-convergence` to ensure convergence --------- Co-authored-by: Shao Tang <[email protected]> Co-authored-by: Steven Shimizu <[email protected]> Co-authored-by: shivam15s <[email protected]> |
||
|
Steven Shimizu
|
8ce3b5310e |
Refactored convergence tests to be portable (#41)
## Summary
- Make convergence tests more portable and easier to run by using
pre-tokenized data. This removes internal paths and allows users to not
have to download specific model tokenizers in a certain location.
- Since we're just testing convergence on a mini model with random
weights, the specific tokenizer doesn't really matter.
- Convergence tests also finish faster: **~95 sec -> ~60 sec**
Alternatives:
- Provide users the ability to configure paths to the different models
used in convergence test or HF token to download the tokenizer
(inconvenient to configure/download things run tests, not portable,
different tokenizer versions could break tests)
- Save the tokenizers in the repo (licensing issues)
- Save a small, completely OS tokenizer in the repo and use across all
tests (could do this, but also more performant to just pre-tokenize the
data)
## Testing Done
Ran convergence tests successfully
- [x] run `make test` to ensure correctness
- [x] run `make checkstyle` to ensure code style
- [x] run `make test-convergence` to ensure convergence
```
jobuser [ ~/Liger-Kernel ]$ make checkstyle
flake8 .; flake8_status=$?; \
isort .; isort_status=$?; \
black .; black_status=$?; \
if [ $flake8_status -ne 0 ] || [ $isort_status -ne 0 ] || [ $black_status -ne 0 ]; then \
exit 1; \
fi
Fixing /home/jobuser/Liger-Kernel/test/convergence/test_mini_models.py
Skipped 1 files
All done! ✨ 🍰 ✨
45 files left unchanged.
jobuser [ ~/Liger-Kernel ]$ make test
pytest --disable-warnings test/ --ignore=test/convergence
===================================================================================================================== test session starts ======================================================================================================================
platform linux -- Python 3.10.14, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/jobuser/Liger-Kernel
plugins: lipy-config-base-30.6.1, lipy-fabric-35.2.3, lipy-test-8.0.52, datadir-1.3.1, lipy-mp-34.4.191
collected 111 items
test/transformers/test_cross_entropy.py .......................................................... [ 52%]
test/transformers/test_fused_linear_cross_entropy.py ...... [ 57%]
test/transformers/test_geglu.py ........ [ 64%]
test/transformers/test_rms_norm.py ................ [ 79%]
test/transformers/test_rope.py ............ [ 90%]
test/transformers/test_swiglu.py ........ [ 97%]
test/transformers/test_transformers_monkey_patch.py . [ 98%]
test/triton/test_triton_monkey_patch.py .. [100%]
================================================================================================================ 111 passed in 60.81s (0:01:00) ================================================================================================================
jobuser [ ~/Liger-Kernel ]$ make test-convergence
HF_DATASETS_OFFLINE=1 pytest --disable-warnings test/convergence
===================================================================================================================== test session starts ======================================================================================================================
platform linux -- Python 3.10.14, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/jobuser/Liger-Kernel
plugins: lipy-config-base-30.6.1, lipy-fabric-35.2.3, lipy-test-8.0.52, datadir-1.3.1, lipy-mp-34.4.191
collected 8 items
test/convergence/test_mini_models.py ...... [ 75%]
test/convergence/test_mini_models_no_logits.py .. [100%]
====================================================================================================================== 8 passed in 58.41s ======================================================================================================================
```
|