mirror of
https://github.com/linkedin/Liger-Kernel.git
synced 2026-10-12 02:17:47 +00:00
3dd9e3a2b7
## Summary
- Add fused MoE expert (LigerExperts) monkey-patching to
**qwen3_vl_moe** and **glm4v_moe**, the two remaining MoE models that
were missing it
- Fix buggy glm4v_moe instance patching that was outside the `for` loop
with duplicate logic and setting `decoder_layer.mlp = None`
- Enable 3 previously-skipped qwen3_vl_moe convergence tests (bf16/fp32
with_logits, fp32 multimodal)
- All patching gated behind `IS_TRANSFORMERS_V5_OR_LATER` for backward
compatibility with transformers v4
### Models patched
| Model | Class-level patch | Instance-level patch |
|---|---|---|
| `qwen3_vl_moe` | `Qwen3VLMoeTextExperts = LigerExperts` |
Attribute-based detection of `experts` on MoE layers |
| `glm4v_moe` | `Glm4vMoeTextNaiveMoe = LigerExperts` | Attribute-based
detection of `experts` + `shared_experts` |
### Why not llama4?
`Llama4TextExperts.forward(hidden_states)` takes only 1 arg (routing
done externally in MoE block), incompatible with LigerExperts'
`forward(hidden_states,
top_k_index, top_k_weights)`.
### Compatibility
Verified across transformers v4.57.6, v5.0–v5.4, and v5.5.4. Class names
and forward signatures are stable across all v5 releases.
## Test plan
- [x] Instance patching tests (`test_monkey_patch.py`) for all 3
qwen3_vl_moe variants and glm4v_moe
- [x] Convergence tests (bf16 + fp32, FLCE + with_logits) for
qwen3_vl_moe and glm4v_moe
- [x] Enabled 3 previously-skipped qwen3_vl_moe convergence tests
- [x] Verified backward compatibility: on v4,
`IS_TRANSFORMERS_V5_OR_LATER` gates skip all LigerExperts code paths
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>
Co-authored-by: Vaibhav Jindal <[email protected]>