Vaibhav JindalandCopilot 00e72b86ca ci: schedule NVIDIA tests by cadence/GPU with a hard 3-job concurrency cap (#1415) ...

## Summary

Reorganizes the **NVIDIA GPU** workflow into a cadence- and GPU-aware
schedule, and adds the Modal functions to back it. Follows up #1410
(which added the per-DSL test folders and `make test-cutile` /
`test-cutedsl` / `test-cute` targets).

### Schedule

**Daily** (cron `24 9 * * *`):
| Suite | GPU | Modal fn |
|---|---|---|
| general correctness | H100 | `liger_correctness_tests` |
| cuTile | H100 | `liger_cutile_tests` (new) |
| cuTeDSL — SM90 half | H100 | `liger_cutedsl_tests_h100` (new) |
| cuTeDSL — SM100 half | B200 | `liger_cutedsl_tests_b200` (new) |

cuTeDSL runs on **both** archs on purpose: its SM90 tests only run on
H100 and its SM100 tests only run on B200 — each arch self-skips the
other half, so both are needed for full coverage.

**Weekly** (cron `24 13 * * 0`, Sundays, H100): `convergence` (latest
tf) + oldest-supported-tf **correctness** and **convergence**.

**Not scheduled:** `cute` (fused-MoE) tests, for now.

### The ≤3-concurrent fix

A 4th simultaneous Modal GPU call was being rejected (one job died
instantly at ~0.7 min). This is now capped two ways:
- **`strategy.max-parallel: 3`** on each matrix job (daily's 4 entries
run 3-then-1; weekly's 3 run together).
- **Workflow-level `concurrency: { group: <workflow>,
cancel-in-progress: false }`** serializes whole runs, so the daily and
weekly runs never overlap.

Only **one** cadence job activates per trigger (the job-level `if`
selects daily vs weekly from `github.event.schedule` / the
`workflow_dispatch` `suite` input), so at most **3** GPU jobs ever run
together. The `if` deliberately uses only the `github` context — **not**
the `matrix` context, which isn't available in a job-level `if`.

### Manual runs
`workflow_dispatch` gains a `suite` input (`daily` / `weekly`, default
`daily`).


## Notes / caveats
- **Convergence is currently red** on `main`'s scheduled runs
(pre-existing; correctness passes). Weekly scheduling is fine but expect
failures until that's fixed separately.
- **DSL suites run in CI for the first time** — the cuTeDSL-B200 (SM100)
path and the cuTile routing test may surface latent failures on first
run.
- **B200 access**: uses `gpu="B200"` on Modal — confirm the token/plan
has B200 capacity.
- `cuTile` installs the `cutile-tileiras` extra so `import cuda.tile`
resolves (otherwise those tests importorskip).
- Validated locally with a YAML parser + a gating simulation (exactly
one job per trigger) + modal-fn-name cross-check; `actionlint` wasn't
installable in the dev sandbox, and GPU suites can't run there.

Co-authored-by: Copilot <[email protected]>

2026-08-28 04:53:37 +00:00

..

ISSUE_TEMPLATE

Makefile command for env-report (#114)

2024-08-27 09:42:22 -07:00

workflows

ci: schedule NVIDIA tests by cadence/GPU with a hard 3-job concurrency cap (#1415)

2026-08-28 04:53:37 +00:00

pull_request_template.md

Update pull_request_template.md

2024-08-25 11:03:15 -07:00