mirror of
https://github.com/linkedin/Liger-Kernel.git
synced 2026-10-12 02:17:47 +00:00
fix(cutedsl): self-skip SM100-only FLCE tests on SM90 (H100 CI) (#1431)
## Summary Fixes the **H100 (SM90)** half of the `NVIDIA GPU` daily CI job `cuTeDSL SM90 (H100)`. `make test-cutedsl` (`LIGER_KERNEL_IMPL=cutedsl pytest test/cutedsl/`) runs the whole `test/cutedsl/` folder on **both** H100 (SM90) and B200 (SM100) — per `nvi-ci.yml`, each arch is meant to *self-skip the other half*. But the native FLCE op `liger_kernel.ops.cutedsl.ops.fused_linear_cross_entropy` is **SM100/Blackwell-only** and raises ``` RuntimeError: Native CuTe DSL FLCE requires exact SM100 hardware, FP16/BF16 input and weight, and mean/sum reduction. ``` on non-SM100 GPUs. On the H100 job this made **214 FLCE tests fail instead of skip**, because the `_run_or_skip` helper only caught `NotImplementedError`. ## Root cause The file mixes two skip mechanisms. The 6 `@sm100_required` tests skip correctly, but the ~34 parity tests are `@cuda_required` and route through `_run_or_skip`, which only swallowed `NotImplementedError` — not the SM100 `RuntimeError`. A few tests also call the native kernel directly (outside `_run_or_skip`), and `test_flce_native_support_requires_exact_sm100` implicitly assumes Blackwell hardware. ## Changes (test-only) - **Broaden `_run_or_skip`** to also `pytest.skip` on the SM100-hardware `RuntimeError` (message contains `SM100`), re-raising every other `RuntimeError` so genuine SM100 failures still surface on B200. - **Route the 3 direct-call tests** through `_run_or_skip`: `test_flce_native_z_loss_preserves_input_dtype`, `test_flce_independent_requires_grad_matches_torch`, `test_flce_all_ignored_targets_are_valid`. - **Mark `test_flce_native_support_requires_exact_sm100` `@sm100_required`**: on SM90 `_native_sm100_supported` short-circuits at `infer_device_arch() != "blackwell"`, so `get_device_capability` is never called and its `assert seen == [tensor.device]` can't hold. Guard/validation tests that use `pytest.raises` (`rejects_fp32`, `rejects_non_sm100`, `reduction_none`, OOB targets, `invalid_inputs`) still **run and pass** on SM90 — coverage preserved. ## Verification (on an H100 node) - FLCE file alone: **214 failed → 0 failed** (`12 passed, 225 skipped`, 6.9s). - Full `test/cutedsl/` (with #1430 also applied): **462 passed, 229 skipped, 0 failed**. - `ruff check` + `ruff format --check`: clean. ## Relationship to #1430 The other 28 SM90 failures (26 CE-fp32 `assumed_align` + 2 routing) are fixed by #1430. This PR is complementary and independent (touches only `test/cutedsl/test_fused_linear_cross_entropy.py`); together they make the H100 job fully green. --------- Co-authored-by: svc-kingkong_LinkedIn <[email protected]> Co-authored-by: Copilot <[email protected]>
This commit is contained in:
co-authored by
svc-kingkong_LinkedIn
Copilot
parent
083c92342b
commit
6b7e3dd6fa
@@ -97,11 +97,30 @@ def _apply(
|
||||
|
||||
|
||||
def _run_or_skip(thunk):
|
||||
"""Run ``thunk``; if the cutedsl branch is still a Stage-1 stub, SKIP instead of fail."""
|
||||
"""Run ``thunk``; SKIP (don't fail) when the native cutedsl path can't run on this GPU.
|
||||
|
||||
Two conditions self-skip instead of failing:
|
||||
* a Stage-1 ``NotImplementedError`` stub, and
|
||||
* the SM100-only hardware gate — the native FLCE kernel raises a ``RuntimeError``
|
||||
mentioning SM100 on non-SM100 GPUs (e.g. the H100/SM90 daily-CI half, which must
|
||||
self-skip this SM100 suite per the nvi-ci.yml arch split).
|
||||
|
||||
Note: this file targets the SM100 *native* op ``LigerFusedLinearCrossEntropyFunction``,
|
||||
whose full feature set (fp16, bias, label smoothing, z-loss, softcap, ce-weight, token
|
||||
scaling, arbitrary shapes) the separate Hopper op
|
||||
``LigerFusedLinearCrossEntropySM90Function`` deliberately rejects — it is a narrow
|
||||
BF16/mean-sum/aligned-shape MVP. SM90 FLCE has its own coverage in
|
||||
``test_fused_linear_cross_entropy_sm90.py`` (which runs on the H100/SM90 CI half), so
|
||||
these skips do not leave SM90 FLCE untested.
|
||||
"""
|
||||
try:
|
||||
return thunk()
|
||||
except NotImplementedError as exc:
|
||||
pytest.skip(f"cutedsl FLCE branch not implemented yet: {exc}")
|
||||
except RuntimeError as exc:
|
||||
if "SM100" in str(exc):
|
||||
pytest.skip(f"native cutedsl FLCE requires SM100 hardware: {exc}")
|
||||
raise
|
||||
|
||||
|
||||
class _Masters:
|
||||
@@ -656,13 +675,15 @@ def test_flce_native_z_loss_preserves_input_dtype(dtype):
|
||||
x = masters.input.to(dtype).requires_grad_(True)
|
||||
w = masters.weight.to(dtype).requires_grad_(True)
|
||||
target = _make_target(BT, V, ignore_frac=0.25)
|
||||
_, z_loss, *_ = _apply(
|
||||
_cutedsl_flce(),
|
||||
x,
|
||||
w,
|
||||
target,
|
||||
lse_square_scale=1e-4,
|
||||
return_z_loss=True,
|
||||
_, z_loss, *_ = _run_or_skip(
|
||||
lambda: _apply(
|
||||
_cutedsl_flce(),
|
||||
x,
|
||||
w,
|
||||
target,
|
||||
lse_square_scale=1e-4,
|
||||
return_z_loss=True,
|
||||
)
|
||||
)
|
||||
assert z_loss.dtype == dtype
|
||||
|
||||
@@ -1059,7 +1080,7 @@ def test_flce_independent_requires_grad_matches_torch(trainable):
|
||||
leaf = {"input": x, "weight": w, "bias": b}[trainable]
|
||||
return loss.detach().float(), leaf.grad.detach().float()
|
||||
|
||||
out = one_native()
|
||||
out = _run_or_skip(one_native)
|
||||
ref = one_torch()
|
||||
_assert_close(out[0], ref[0], 5e-3, 5e-2, "independent requires_grad loss")
|
||||
_assert_close(out[1], ref[1], 5e-3, 5e-2, f"independent grad_{trainable}")
|
||||
@@ -1069,7 +1090,7 @@ def test_flce_independent_requires_grad_matches_torch(trainable):
|
||||
def test_flce_all_ignored_targets_are_valid():
|
||||
_input, weight, target = _basic_args(BT=16, V=256, dtype=torch.bfloat16)
|
||||
target.fill_(-100)
|
||||
out = _apply(_cutedsl_flce(), _input, weight, target, reduction="mean")[0]
|
||||
out = _run_or_skip(lambda: _apply(_cutedsl_flce(), _input, weight, target, reduction="mean")[0])
|
||||
assert out.item() == 0.0
|
||||
|
||||
|
||||
@@ -1101,7 +1122,7 @@ def test_flce_invalid_inputs_raise(mutation, error):
|
||||
_apply(_cutedsl_flce(), _input, weight, target)
|
||||
|
||||
|
||||
@cuda_required
|
||||
@sm100_required
|
||||
def test_flce_native_support_requires_exact_sm100(monkeypatch):
|
||||
import liger_kernel.ops.cutedsl.ops.fused_linear_cross_entropy as flce
|
||||
|
||||
|
||||
Reference in New Issue
Block a user