fix(cutedsl): self-skip SM100-only FLCE tests on SM90 (H100 CI) (#1431)

## Summary

Fixes the **H100 (SM90)** half of the `NVIDIA GPU` daily CI job `cuTeDSL
SM90 (H100)`.

`make test-cutedsl` (`LIGER_KERNEL_IMPL=cutedsl pytest test/cutedsl/`)
runs the whole `test/cutedsl/` folder on **both** H100 (SM90) and B200
(SM100) — per `nvi-ci.yml`, each arch is meant to *self-skip the other
half*. But the native FLCE op
`liger_kernel.ops.cutedsl.ops.fused_linear_cross_entropy` is
**SM100/Blackwell-only** and raises

```
RuntimeError: Native CuTe DSL FLCE requires exact SM100 hardware, FP16/BF16 input and weight, and mean/sum reduction.
```

on non-SM100 GPUs. On the H100 job this made **214 FLCE tests fail
instead of skip**, because the `_run_or_skip` helper only caught
`NotImplementedError`.

## Root cause

The file mixes two skip mechanisms. The 6 `@sm100_required` tests skip
correctly, but the ~34 parity tests are `@cuda_required` and route
through `_run_or_skip`, which only swallowed `NotImplementedError` — not
the SM100 `RuntimeError`. A few tests also call the native kernel
directly (outside `_run_or_skip`), and
`test_flce_native_support_requires_exact_sm100` implicitly assumes
Blackwell hardware.

## Changes (test-only)

- **Broaden `_run_or_skip`** to also `pytest.skip` on the SM100-hardware
`RuntimeError` (message contains `SM100`), re-raising every other
`RuntimeError` so genuine SM100 failures still surface on B200.
- **Route the 3 direct-call tests** through `_run_or_skip`:
`test_flce_native_z_loss_preserves_input_dtype`,
`test_flce_independent_requires_grad_matches_torch`,
`test_flce_all_ignored_targets_are_valid`.
- **Mark `test_flce_native_support_requires_exact_sm100`
`@sm100_required`**: on SM90 `_native_sm100_supported` short-circuits at
`infer_device_arch() != "blackwell"`, so `get_device_capability` is
never called and its `assert seen == [tensor.device]` can't hold.

Guard/validation tests that use `pytest.raises` (`rejects_fp32`,
`rejects_non_sm100`, `reduction_none`, OOB targets, `invalid_inputs`)
still **run and pass** on SM90 — coverage preserved.

## Verification (on an H100 node)

- FLCE file alone: **214 failed → 0 failed** (`12 passed, 225 skipped`,
6.9s).
- Full `test/cutedsl/` (with #1430 also applied): **462 passed, 229
skipped, 0 failed**.
- `ruff check` + `ruff format --check`: clean.

## Relationship to #1430

The other 28 SM90 failures (26 CE-fp32 `assumed_align` + 2 routing) are
fixed by #1430. This PR is complementary and independent (touches only
`test/cutedsl/test_fused_linear_cross_entropy.py`); together they make
the H100 job fully green.

---------

Co-authored-by: svc-kingkong_LinkedIn <[email protected]>
Co-authored-by: Copilot <[email protected]>
This commit is contained in:
Vaibhav Jindal
2026-09-01 20:26:39 +00:00
committed by GitHub
co-authored by svc-kingkong_LinkedIn Copilot
parent 083c92342b
commit 6b7e3dd6fa
+32 -11
View File
@@ -97,11 +97,30 @@ def _apply(
def _run_or_skip(thunk):
"""Run ``thunk``; if the cutedsl branch is still a Stage-1 stub, SKIP instead of fail."""
"""Run ``thunk``; SKIP (don't fail) when the native cutedsl path can't run on this GPU.
Two conditions self-skip instead of failing:
* a Stage-1 ``NotImplementedError`` stub, and
* the SM100-only hardware gate — the native FLCE kernel raises a ``RuntimeError``
mentioning SM100 on non-SM100 GPUs (e.g. the H100/SM90 daily-CI half, which must
self-skip this SM100 suite per the nvi-ci.yml arch split).
Note: this file targets the SM100 *native* op ``LigerFusedLinearCrossEntropyFunction``,
whose full feature set (fp16, bias, label smoothing, z-loss, softcap, ce-weight, token
scaling, arbitrary shapes) the separate Hopper op
``LigerFusedLinearCrossEntropySM90Function`` deliberately rejects — it is a narrow
BF16/mean-sum/aligned-shape MVP. SM90 FLCE has its own coverage in
``test_fused_linear_cross_entropy_sm90.py`` (which runs on the H100/SM90 CI half), so
these skips do not leave SM90 FLCE untested.
"""
try:
return thunk()
except NotImplementedError as exc:
pytest.skip(f"cutedsl FLCE branch not implemented yet: {exc}")
except RuntimeError as exc:
if "SM100" in str(exc):
pytest.skip(f"native cutedsl FLCE requires SM100 hardware: {exc}")
raise
class _Masters:
@@ -656,13 +675,15 @@ def test_flce_native_z_loss_preserves_input_dtype(dtype):
x = masters.input.to(dtype).requires_grad_(True)
w = masters.weight.to(dtype).requires_grad_(True)
target = _make_target(BT, V, ignore_frac=0.25)
_, z_loss, *_ = _apply(
_cutedsl_flce(),
x,
w,
target,
lse_square_scale=1e-4,
return_z_loss=True,
_, z_loss, *_ = _run_or_skip(
lambda: _apply(
_cutedsl_flce(),
x,
w,
target,
lse_square_scale=1e-4,
return_z_loss=True,
)
)
assert z_loss.dtype == dtype
@@ -1059,7 +1080,7 @@ def test_flce_independent_requires_grad_matches_torch(trainable):
leaf = {"input": x, "weight": w, "bias": b}[trainable]
return loss.detach().float(), leaf.grad.detach().float()
out = one_native()
out = _run_or_skip(one_native)
ref = one_torch()
_assert_close(out[0], ref[0], 5e-3, 5e-2, "independent requires_grad loss")
_assert_close(out[1], ref[1], 5e-3, 5e-2, f"independent grad_{trainable}")
@@ -1069,7 +1090,7 @@ def test_flce_independent_requires_grad_matches_torch(trainable):
def test_flce_all_ignored_targets_are_valid():
_input, weight, target = _basic_args(BT=16, V=256, dtype=torch.bfloat16)
target.fill_(-100)
out = _apply(_cutedsl_flce(), _input, weight, target, reduction="mean")[0]
out = _run_or_skip(lambda: _apply(_cutedsl_flce(), _input, weight, target, reduction="mean")[0])
assert out.item() == 0.0
@@ -1101,7 +1122,7 @@ def test_flce_invalid_inputs_raise(mutation, error):
_apply(_cutedsl_flce(), _input, weight, target)
@cuda_required
@sm100_required
def test_flce_native_support_requires_exact_sm100(monkeypatch):
import liger_kernel.ops.cutedsl.ops.fused_linear_cross_entropy as flce