Use the non-deprecated torch.amp APIs when available while preserving the torch.cuda.amp fallback and legacy autocast argument semantics on older PyTorch releases.
Synchronize DDP/FSDP gradients on the final microbatch of each accumulation window.
Verified with a two-rank CPU/Gloo reproduction, py_compile, diff checks, and a clean synthetic merge against current main.
* fix(trainer): avoid KeyError when a checkpoint is saved at an unvalidated step
When save_checkpoint_interval is not a multiple of validate_interval, a
checkpoint can be saved at a step that was never validated. The best-ckpt
comparison then indexes val_acc/val_loss_step_or_epoch[ckpt_name], raising
KeyError right after the checkpoint was already written and aborting the run.
Guard both the DeepSpeed and the torch.save checkpoint paths with .get() so
an unvalidated checkpoint is skipped for the "best" bookkeeping (and logged)
instead of crashing; fall back to 0.0 when recording saved_ckpts.
Co-Authored-By: Claude <[email protected]>
* fix(trainer): exclude unvalidated-step checkpoints from best-model ranking
Address review feedback on PR #3455. A checkpoint saved at a step that was
never validated (save_checkpoint_interval not a multiple of
validate_interval) used to be inserted into `saved_ckpts` with a fabricated
score of 0.0. Under loss-based ranking that synthetic "perfect" loss could
evict a real validated best checkpoint via keep_nbest_models pruning.
Now such checkpoints are kept on disk but excluded from `saved_ckpts`
entirely, so they never compete in best-model ranking or pruning, and
`best_step_or_epoch` only ever references validated checkpoints.
Adds regression coverage for both loss and acc ranking on the DeepSpeed
and torch.save paths, asserting an unvalidated checkpoint cannot evict a
validated best checkpoint.
* test(trainer): make unvalidated-ckpt regression catch the DeepSpeed+acc case
The fake DeepSpeed engine now writes a placeholder file per tag (mirroring
DeepSpeed's per-tag checkpoint dirs), so keep_nbest_models pruning's
smart_remove() is observable. This closes a gap where the [DeepSpeed, acc]
combination passed on the buggy code: under acc ranking the fabricated 0.0
is the minimum, so pruning deletes the unvalidated checkpoint itself rather
than the validated best, and without an on-disk artifact that deletion was
invisible.
With the fix, all 8 cases pass; on the previous code all 4 unvalidated-step
cases now fail (loss: fabricated 0.0 evicts the validated best; acc: the
unvalidated checkpoint file is pruned).
* fix(trainer): apply the same unvalidated-step guard to the non-DeepSpeed Trainer
`funasr-train` (funasr/train_utils/trainer.py) shares the identical
checkpoint-ranking code as trainer_ds.py and had the same bug: direct
indexing of `val_{acc,loss}_step_or_epoch[ckpt_name]` raised KeyError when
a checkpoint was saved at a step that was never validated, and the
keep_nbest_models bookkeeping inserted a fabricated score for such
checkpoints.
Apply the same fix: checkpoints saved at an unvalidated step are kept on
disk but excluded from saved_ckpts (no synthetic score), best_step_or_epoch
only ever references validated checkpoints, and the metric dicts are read
via .get().
Tests now parametrize over all three checkpointing paths (trainer_ds
torch.save, trainer_ds DeepSpeed, and trainer torch.save) x loss/acc.
On upstream main all 6 unvalidated-step cases fail with
`KeyError: 'model.pt.ep1.2'`; with the fix all 12 cases pass.
---------
Co-authored-by: Claude <[email protected]>
- Fix TypeError when scheduler_conf is not provided (NoneType unpacking)
- Add explicit scheduler and scheduler_conf to VAD finetune.sh
- Fixes user report in #2961
- Use non_blocking=True for CPU→GPU data transfer (pin_memory is already enabled)
- Add persistent_workers=True and prefetch_factor=2 to DataLoader
- Reduce unnecessary all_reduce syncs to only fire at log_interval
Verified on AISHELL-1 Paraformer (H100x2, 10 epochs): val_loss matches
baseline exactly at all 10 checkpoints (10.420 final), step time reduced
from 0.2034s to 0.1972s.
The FSDP branch in save_checkpoint was silently doing nothing (pass),
causing all checkpoints to be lost during FSDP training without any
warning. Now raises NotImplementedError with a clear message.
- trainer.py/trainer_ds.py: replace eval(var) with float(var) for
speed_stats logging. The values are numeric strings that only need
type conversion, not arbitrary code execution.
- train.py/train_ds.py: replace eval(freeze_param) with
freeze_param.split(",") for parsing comma-separated parameter names.
* Fix wandb log
* fix validation loss is not logged
batch_idx got reset for each epoch.
use the global step counter instead
* LR should only be updated per step, not per step+ per epoch
* add early stopping
* Fix bf16 handling
scaler is only needed for fp16
* more logs
---------
Co-authored-by: Tony Mak <[email protected]>
zhifu gao