feat: add log_key_prefix to Trainer for Trainer-generated metric keys
Adds a `log_key_prefix` parameter to `Trainer` that prepends a string
to Trainer-generated metric keys such as `epoch`. Defaults to bare
`epoch` (no prefix), so existing users see no change.
Co-authored-by: Bhimraj Yadav <[email protected]>
* lightning.pytorch.profilers: use os.PathLike
Signed-off-by: Adam J. Stewart <[email protected]>
* fix(profilers): resolve `dirpath` via `os.fspath` for chrome trace export
`PyTorchProfiler` passed `str(self.dirpath)` to `tensorboard_trace_handler`,
which calls `os.makedirs` on whatever string it receives. That was harmless
while `dirpath` was `str | Path`, but now that it accepts any
`os.PathLike[str]`, `__str__` is no longer guaranteed to be the path: a
PathLike whose `__str__` returns a repr made the chrome trace land in a
literal directory named after that repr, created under the cwd, leaving
`dirpath` empty.
Use `os.fspath`, which is the documented way to get the path out of a
PathLike and passes `str` through unchanged. The neighbouring `os.path.join`
calls already do this internally.
Add tests covering a non-`Path` PathLike `dirpath` for `SimpleProfiler`,
`AdvancedProfiler`, and the `PyTorchProfiler` chrome export.
* docs: add CHANGELOG entry for the profiler `dirpath` type change
---------
Signed-off-by: Adam J. Stewart <[email protected]>
Co-authored-by: Bhimraj Yadav <[email protected]>
* CUDAAccelerator.setup_device: fix unrelated device init by matmul precision check
Without this fix, CUDAAccelerator.setup_device may initialize an unrelated device, via
- _check_cuda_matmul_precision
- _is_ampere_or_later
- torch.cuda.get_device_capability
- torch.cuda.get_device_properties
- torch.cuda._lazy_init
* Added tests asserting CUDAAccelerator setup sets device before triggering
initialization
* test: extract the spawned-subprocess CUDA check into a helper
The check was written as a test permanently marked `pytest.mark.skip` and
invoked by name from the test that spawns it. That overloaded the skip
marker, left `RunIf(min_cuda_gpus=1)` on a function pytest never evaluates,
and reported two permanently skipped tests on every run.
Make it a plain module-level helper instead and give the remaining test the
clearer name. Same coverage, no phantom skips.
* test: cover the set_device ordering on CPU runners
Both existing ordering checks are gated behind `RunIf(min_cuda_gpus=1)`, so
nothing fails on a CPU-only run if the two lines in `setup_device` are
swapped back.
Add a mock-based check that asserts the call order without touching CUDA. It
only proves ordering, so it complements the subprocess test rather than
replacing it: that one exercises the real `_lazy_init` and establishes that
the matmul precision check reaches it at all.
* docs: add CHANGELOG entries for the CUDA device init fix
The fix is user-facing and has a linked issue, so it falls outside the
template's exemption for internal changes. It touches both packages.
---------
Co-authored-by: Justus Perillieux <[email protected]>
Co-authored-by: Bhimraj Yadav <[email protected]>
Co-authored-by: thomas chaton <[email protected]>
The scaler guard read self.precision before it was assigned, so it saw the base
class default 32-true and raised for a 16-mixed scaler; the else branch also
dropped a user-provided scaler. Use the local precision argument in the guard and
keep the scaler.
* fix(checkpoint): block untrusted _class_path imports in load_from_checkpoint
The _instantiator allowlist added in #21832 for CVE-2026-58659 left a second
attacker-controlled import path open. The one allowlisted instantiator,
lightning.pytorch.cli.instantiate_module, passes the checkpoint's _class_path
to jsonargparse, whose import_object imports the named module before checking
that the class is a subclass of the expected type. A weights_only=True
checkpoint could therefore still execute module-level code of its choosing.
_load_state now rejects a _class_path that does not resolve to an already
imported subclass of the class being loaded. Resolution reads sys.modules
only, so loading a checkpoint never imports anything new.
Also reject a non-string _instantiator, which weights_only=True permits and
which previously raised TypeError: unhashable type from the allowlist lookup.
* refactor: align `_class_path` guard with repo conventions
- reword `_is_imported_subclass` docstring to lead with the predicate,
matching the "Check whether ..." style used for private predicates
- drop "the remaining" from the CHANGELOG entry, since nested hparams
import paths are still open, and link the PR instead of the issue
- remove a test comment that restated the docstring below it
* trigger:ci
---------
Co-authored-by: bhimrazy <[email protected]>
* fix: avoid deprecated jsonargparse APIs in LightningCLI
jsonargparse 4.49 deprecated `ArgumentParser.instantiate_classes` and 4.50
the `skip_none` argument of `dump`/`save`, so `LightningCLI` emitted
deprecation warnings on recent versions.
Both are now version-gated on jsonargparse>=4.50, since `skip_unset` does
not exist in 4.49 and `dump`/`save` accept `**kwargs`, which would silently
drop `None` entries from the saved config.
The parser-scoped `add_instantiator` is deliberately kept: its replacement
registers instantiators process-wide and would leak between LightningCLI
instances, so the warning is silenced instead.
Also raises the jsonargparse upper pin so CI exercises the new code path.
Fixes#21900
* refactor: tighten jsonargparse compat shims and comments
Condense the version-gate and `add_instantiator` comments to what is not
derivable from the code, drop the docstring from `_instantiate` to match the
other private helpers in the module, and collapse `_KEEP_NONE_KWARGS` into a
single expression.
Drop `test_cli_no_jsonargparse_deprecation_warnings` and its now-unused
imports. Keep the changelog entry version-agnostic, since the single 4.50 gate
leaves `instantiate_classes` in use on 4.49.
* ci: bump torch ceiling to <2.14.0 now that torch-tensorrt 2.13 is out
Follow-up to #21851, which capped torch at <2.13.0 because torch-tensorrt
had no stable build past it yet. torch-tensorrt 2.13.0 (released 2026-07-28)
now requires torch>=2.13.0,<2.14.0, so the cap can move up.
* test: skip TS-frontend torch-tensorrt cases when the build lacks it
CI on torch-tensorrt 2.13 failed:
ValueError: Requested using the TS frontend but the TS frontend is not
available in this build of Torch-TensorRT
The TorchScript frontend has been deprecated since torch-tensorrt 2.6 and is
no longer bundled in some prebuilt wheels (e.g. 2.13+), independent of the
torch version. The existing torch==2.9 skip only covered a different failure
mode (a crash), so add a runtime check against
torch_tensorrt.ENABLED_FEATURES.torchscript_frontend and skip ir="ts" cases
whenever the installed build doesn't have it.
* test: report accurate skip reason for missing TorchScript frontend
Corrects the previous commit's rationale. The TorchScript frontend is not
removed in torch-tensorrt 2.13 -- the PyPI wheel still ships libtorchtrt.so
and the release notes even include TS bug fixes. What changed is packaging:
the 2.13.0+cu130 wheel on download.pytorch.org is pure Python (608 KB, zero
.so files), while 2.12.1+cu130 bundled libtorchtrt.so. GPU CI installs from
that index via --find-links, so it gets a build without the frontend.
Feature detection via ENABLED_FEATURES.torchscript_frontend stays (it is the
portable check -- has_torchscript_frontend() only exists in newer releases),
but the two skip conditions are now separate marks so the report names the
one that actually applies instead of a misleading combined message.
* test: ignore torch 2.13 all_gather_into_tensor deprecation from DeepSpeed
The call comes from DeepSpeed's ZeRO-3 checkpointing; the API appears nowhere
in Lightning's source or tests, so there is nothing to migrate on our side.
* ci: resolve torch-tensorrt from PyPI to keep the TorchScript frontend
The 2.13.0+cu130 wheel on download.pytorch.org is pure Python (608 KB, zero
.so files); 2.12.1+cu130 bundled libtorchtrt.so. Since a local version sorts
above the plain release, the find-links entry always won and the TorchScript
frontend went missing. PyPI's 2.13.0 ships the libraries and is likewise a
CUDA 13 build.
* test: widen the DeepSpeed collective filter to the module
torch 2.13 renamed the `*_tensor` collectives to `*_single`, and DeepSpeed's
backend calls both `all_gather_into_tensor` and `reduce_scatter_tensor` from
deepspeed/comm/torch.py, so filtering one message at a time is whack-a-mole.
Filter on the module instead, matching the existing ddp_comm_hooks entry.
Bumping DeepSpeed would not help: 0.19.4 still binds
torch.distributed.reduce_scatter_tensor via hasattr and has no migration to
the new names.
* test: ignore the exact torch 2.13 collective deprecation messages
The earlier module-based filter (`deepspeed.comm.torch`) never matched:
`typing_extensions.deprecated()` uses stacklevel=2, which attributes the
warning to whoever called the deprecated function, and every collective is
wrapped by torch's own `c10d_logger`, so that's always the immediate caller
- not DeepSpeed. A module filter scoped to `torch.distributed.c10d_logger`
would work but also swallow any *other* collective deprecation raised through
that same wrapper, which is broader than intended.
Match the two known messages instead, same as the other entries in this list.
Verified locally against torch 2.13 (CPU-only) with a fake DeepSpeed-shaped
caller.
* Fix PermissionError being swallowed in _atomic_save fallback path
Non-EXDEV PermissionErrors were silently discarded instead of being
re-raised, hiding real permission failures during checkpoint saves.
* fix pre-commit
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* upd chlog
---------
Co-authored-by: deependujha <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Bhimraj Yadav <[email protected]>
* Remove dead PyTorch < 2.6 version guards
The torch floor was raised to 2.6 in #21851, so the
_TORCH_GREATER_EQUAL_2_2 .. _TORCH_GREATER_EQUAL_2_5 guards (plus the
unused _TORCH_EQUAL_2_4_0 / _TORCH_GREATER_EQUAL_2_4_1 constants) are
always true and their else-branches unreachable. Drop the constants,
inline the surviving branches, and remove the tests that only covered
the now-impossible paths.
* Add CHANGELOG entries for the removed torch version guards
* Fix tests referencing symbols dropped from the strategy modules
- import the DCP reader/writer helpers from lightning.fabric.utilities.cloud_io,
where they are defined, instead of the FSDP re-export
- stop patching model_parallel._lazy_load, which the module no longer uses;
the remote-vs-mmap regression test now watches torch.load instead
* Address review: drop kwargs indirection and stale comment
The `kwargs` dict in the six `_setup_distributed` implementations only
existed to build the argument list conditionally. Now that both entries
are unconditional, pass them directly as keyword arguments.
Also drop the `_lazy_load` mention from the model_parallel comment; that
path uses `torch.load(mmap=True)` only.
* ci: drop PyTorch 2.1-2.5 and raise minimum to 2.6 (1/n)
Prune the oldest PyTorch minors (2.1-2.5) from the CPU test matrices now
that we test up to 2.13, and raise the supported floor to 2.6:
- ci-tests-{pytorch,fabric}: replace the 2.1-2.5 'lightning' rows with
coverage at the new floor (2.6) and latest (2.13); bump the 'oldest'
job from 2.1 to 2.6.
- requirements: torch >=2.1.0 -> >=2.6.0 (base/pytorch/fabric); fix the
stale upper bound <2.10.0 -> <2.14.0; torchvision >=0.16.0 -> >=0.21.0
with upper bound <0.26.0 -> <0.29.0.
- remove per-version special-cases for versions no longer tested: the
torch 2.7 test-channel URL switch and the torch 2.4 Windows libuv
workaround.
PyTorch upstream only patches the newest minor, and Lightning's own
policy supports the latest 5 minors (2.9-2.13); 2.6 is a conservative
floor above every current ecosystem minimum.
* ci: raise deepspeed and onnx floors for torch 2.6
The 'oldest' job pins every dependency to its lower bound, which broke
against the new torch 2.6 floor:
- deepspeed: Lightning itself raises 'PyTorch >= 2.6 requires DeepSpeed
>= 0.16.0' (strategies/deepspeed.py), but the pin allowed 0.15.0.
- onnxruntime: 1.12.0 supports at most model IR version 8, while torch
2.6 exports IR version 10; bump to the first release covering it.
onnx is bumped alongside for IR-10 consistency.
* ci: retire CUDA 12.1 images in favor of 12.6 (cu126)
The cu121 wheel index stops at torch 2.5.1, so the GPU jobs could not
resolve any torch once the floor moved to >=2.6.0:
Because only torch<=2.5.1+cu121 is available and
lightning-fabric==2.6.2 depends on torch>=2.6.0,<2.14.0 ...
cu126 publishes wheels for the whole 2.6-2.13 range (cp310 and cp312),
so point the remaining nvidia/cuda:12.1.1 images at 12.6.3. Adding a
cu130 lane is left for a follow-up since it needs a runner driver bump.
* ci: cap torch <2.13 and move GPU 3.12 lanes to cu130
torch-tensorrt has no build for torch 2.13 on any CUDA lane, so the GPU
install was unresolvable. cu130 is the only lane shipping torch-tensorrt
for torch 2.12; the oldest 3.10 lane stays on cu126, which is the only
lane carrying torch 2.6.
Images are now set per matrix row rather than inherited from the default,
so checkgroup job names are updated to match.
* fix: honor torch.device context in _DeviceDtypeModuleMixin on torch <2.8
torch.get_default_device() only started reading the function mode stack
in 2.8, so on 2.3-2.7 it returned cpu inside a 'with torch.device(...)'
block and the mixin recorded the wrong device. Gate on 2.8 instead of 2.3
and cover it with a meta-device parametrization.
Also drops a duplicated _TORCH_GREATER_EQUAL_2_6 definition.
* ci: decouple torch-tensorrt from the GPU install
torch-tensorrt hard-pins to one torch minor and lags new releases, so as a
hard dependency it made every new torch version unresolvable. Install it
separately, matched to the resolved torch: when no build exists the step is
a no-op and the RunIf(tensorrt=True) tests skip instead of failing CI.
Pinning to the torch minor is deliberate - _TORCH_TRT_AVAILABLE falls back
to importing the module, so a mismatched build would pass the gate and then
fail on import.
* update image names and versions
* update image versions
* revert a few changes
* tests: ignore torch.distributed.nn.functional.all_gather FutureWarning
Deprecated in torch 2.12 and escalated to an error by our filterwarnings
config. The suggested replacement is a private torch API, so the ignore is
keyed to the message (stacklevel makes a module-based filter unreliable),
matching the existing torch deprecation entries.
* add ignore
* update
* revert the test gpu change
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* ci: raise bitsandbytes ceiling to pick up cuda130 binaries
bitsandbytes only started shipping a libbitsandbytes_cuda130.so in
0.48.0, so the <0.47.0 cap was resolving to a build with no cu130
binary on the L4 cu130 lane.
* ci: drop redundant Darwin marker for torch-tensorrt in test_gpu
This file is only ever installed by the GPU CI workflow, which never
runs on Darwin, so the platform_system check was dead weight.
* tests: tolerate torch.compile cleanup leakers on newer torch
Testing against newer torch surfaced new torch.compile subprocess-pool
side effects that the cleanup fixtures didn't account for:
- allowlist the TRITON_PTXAS_PATH env var (set by inductor)
- teach thread_police about the subproc_worker_timer watchdog thread
and native _DummyThreads that can't be joined/stopped
* update
* revert change in test_gpu.txt
* update comments
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Deependu <[email protected]>
tests: fix flaky model summary precision test on CPU
`16-mixed` autocasts to bfloat16 on CPU, and not every CI machine supports
it, so `fit` intermittently failed with "could not create a primitive" or
SIGILL. Only the attached trainer's precision is needed here, so attach it
directly instead of running a step.
* ci: add pytorch 2.12 and 2.13 to cpu tests
* test: avoid global privateuse1 backend pollution breaking CI on PyTorch 2.12/2.13
* test: simplify with monkeypatch.setitem
Prevent arbitrary code execution via _instantiator in load_from_checkpoint
Restrict the checkpoint _instantiator hyperparameter to an allowlist of
trusted instantiators. Previously any import path stored in a checkpoint
was imported and called, allowing arbitrary code execution even under
torch.load(weights_only=True). Fixes#21822.
* fix: use fsspec for DeepSpeed checkpoint path validation to support remote URIs
Fixes#21635. _validate_checkpoint_directory was wrapping paths in
pathlib.Path, which mangles remote URI schemes (e.g. s3:// to s3:/)
and uses local-only .is_dir()/.is_file() checks that always return
False for remote paths like S3, GCS, or HDFS.
Replace with get_filesystem() + fs.isdir()/fs.isfile() from
cloud_io, which is the established pattern used by ModelCheckpoint,
TorchCheckpointIO, and CheckpointConnector.
* test: add coverage for file-inside-checkpoint case with remote URIs
Adds a test for Case 2 (user passes a file path inside the checkpoint
subfolder, e.g. s3://bucket/ckpt/checkpoint/model_states.pt) to ensure
the grandparent path suggestion works correctly with remote URIs.
* docs: add CHANGELOG entry for DeepSpeed remote URI fix
* fix: use posixpath.dirname instead of private fs._parent for path traversal
Replace fs._parent() (private fsspec API) with posixpath.dirname()
(Python stdlib) for computing parent paths. posixpath handles
/‑separated paths correctly for both local paths and remote URIs
on all platforms, without relying on fsspec internals.
Also adds 3 unit tests covering remote URI validation and a shared
_make_s3_mock_fs() helper to reduce duplication.
* Use os.path.dirname for DeepSpeed checkpoint parent hints and cover local paths
Follow-up to the fsspec migration: os.path.dirname is the repo idiom for
parents of possibly-remote paths and keeps local Windows paths working,
where posixpath.dirname dropped the parent-directory suggestion. Also
consolidate the remote-URI tests and add local-path coverage that runs
without deepspeed installed.
* Escape tmp_path in regex match for Windows local-path test
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Stream atomic checkpoint saves to local disk instead of buffering
Avoids holding a second full in-memory copy of the checkpoint when
saving to local filesystems, which matters for large FSDP/ModelParallel
full state dicts. Object storage (S3/GCS/Azure) still buffers to bytes
since those backends can't stream torch.save directly.
* Add test for atomic save cleanup on interrupted torch.save
Verifies that a torch.save failure partway through serialization
leaves no partial file behind, guarding the streaming atomic save
change in fa0fbc744.
---------
Co-authored-by: Deependu <[email protected]>
* Add fsspec URL support for distributed checkpoints in ModelParallelStrategy (#21797)
Support remote storage (fsspec URLs) when saving and loading distributed checkpoints with ModelParallelStrategy in both Fabric and PyTorch Lightning.
TAG=agy
CONV=624ebd36-33ec-4389-8f79-843e487e6872
* Add missing RunIf(min_torch=2.4) guard to remote-path regression test
The test instantiates ModelParallelStrategy, which raises ImportError on
torch < 2.4. Without the guard it errored instead of skipping on the
mirrored pytorch_lightning CI run against older torch. Matches the pattern
used by every other test in the file.
* Add missing RunIf(min_torch=2.4) guard to fabric remote-path regression tests
Three tests monkeypatch "torch.distributed.checkpoint.state_dict.
get_model_state_dict", which forces an import of that submodule. It does not
exist on torch < 2.2, so the tests errored with ImportError instead of skipping
on the CI run against older torch (observed on torch 2.1). Guard them at
min_torch=2.4 to match every other test in the file (ModelParallelStrategy
requires torch >= 2.4), mirroring 5ac3fce04 on the pytorch side.
* Support remote storage (fsspec URLs) for FSDP checkpoints
* Add tests for torch's private fsspec DCP classes
* Respect the user-provided weights_only argument on load_checkpoint
* Fix _resolve_path for local file URIs
* Fix weights_only default for metadata in sharded FSDP checkpoints
Default to weights_only=False for metadata loads to match full-checkpoint
behavior and allow non-tensor metadata on torch>=2.6, while still honoring
explicit user-provided values.
* Add and improve remote checkpoint tests
* add comment
---------
Co-authored-by: Deependu <[email protected]>
* fix(strategy): pass CPU device to FSDP device_id on CPU
Passing `device_id=None` (which is `root_device.index` for CPU) triggers a guard in PyTorch >= 2.5 ("FSDP needs a non-CPU accelerator device"). Passing the actual `torch.device("cpu")` avoids this guard and allows FSDP to run on CPU.
- Update `FSDPStrategy` in both Fabric and PyTorch to pass `self.root_device` if it is a CPU device, otherwise `self.root_device.index`.
- Add unit tests to verify `device_id` selection for CPU and GPU devices.
CONV=516b4463-4a64-48ac-aabe-20fc589f838a
TAG=agy
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* chore: update CHANGELOG
* refactor(FSDP): address review feedback on device_id CPU fix
- common out `device_id` into a local so the debug log reflects the
device actually passed (was logging root_device.index = None on CPU)
- add call-site comment noting CPU is not a supported FSDP path; the
branch only honors the torch>=2.5 device_id contract
- replace the parametrized CPU/GPU device_id tests with a single
minimal CPU-only assertion per strategy: guards the device_id=None
regression without codifying the GPU path as a tested contract
- reword CHANGELOG to frame this as a narrow robustness fix
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <[email protected]>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* apply copilot suggestion
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Deependu <[email protected]>
Co-authored-by: Copilot Autofix powered by AI <[email protected]>
* Fix: add weights_only parameter to LearningRateFinder and propagate through LR finder call chain
* Test: ensure LearningRateFinder supports weights_only=False during checkpoint restore
* Test: ensure LearningRateFinder supports weights_only=False during checkpoint restore and apply pre-commit
* Style: apply pre-commit formatting to LR Finder patch
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* jsonargparse 4.39 required for CLI, check at run-time
Signed-off-by: Adam J. Stewart <[email protected]>
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Signed-off-by: Adam J. Stewart <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix toggle_optimizer breaking under torch.compile (#21513)
`LightningModule.toggle_optimizer` and `untoggle_optimizer` mutate
`requires_grad` on parameters to implement multi-optimizer gradient
masking. Dynamo/AOTAutograd does not support `setattr()` on
`Tensor.requires_grad` because it can change a tensor's leaf-ness
mid-graph, so when the `LightningModule` is wrapped with
`torch.compile` tracing either graph-breaks with
"Unsupported: setattr() on Tensor.requires_grad" or raises a
`KeyError` on the internal `param_requires_grad_state` mapping when
the traced parameter references diverge from those held by
`trainer.optimizers`.
Decorate both helpers with `@torch.compiler.disable` (the same
pattern already used for logging bookkeeping in
`logger_connector/result.py`) so they run as opaque Python when
called from a compiled `training_step`. Eager behavior is unchanged.
Adds a CPU regression test that compiles a two-optimizer
`LightningModule` calling `toggle_optimizer` / `untoggle_optimizer`
in `training_step` and exercises one training iteration, plus a
CHANGELOG entry.
* Narrow test_toggle_untoggle to check compiler.disable attribute (#21513)
The previous regression test compiled a `LightningModule` end-to-end
and called `self.optimizers()` inside the compiled `training_step`,
which unrelated to the toggle_optimizer fix trips a separate Dynamo
limitation: tracing `self.trainer.strategy._lightning_optimizers`
raises `InternalTorchDynamoError: GetAttrVariable(...) has no type`
across all CI platforms and torch versions.
The shipped fix — `@torch.compiler.disable` on `toggle_optimizer` /
`untoggle_optimizer` — does not require a full compiled trainer run
to verify; it only guarantees Dynamo skips those two methods.
Replace the integration test with a direct attribute check that both
methods carry the `_torchdynamo_disable` marker installed by
`torch.compiler.disable`, following the same `has_dynamo(fn)` pattern
already used by `tests/utilities/test_compile.py::test_compile_uncompile`.
Toggle/untoggle functional correctness remains covered by the existing
`test_toggle_untoggle_2_optimizers_no_shared_parameters` and
`test_toggle_untoggle_3_optimizers_shared_parameters` tests in this
file.
---------
Co-authored-by: Deependu <[email protected]>
* fix: use del instead of explicit shutdown in CombinedLoader.reset()
* Update src/lightning/pytorch/CHANGELOG.md
---------
Co-authored-by: Deependu <[email protected]>
* Add tied weight detection warning for FSDP auto-wrap policy
Detect shared parameters that would be placed in separate FSDP units
by the auto-wrap policy and emit a warning before wrapping. This turns
a cryptic RuntimeError (size mismatch) into an actionable message.
Applies to both Fabric and PyTorch Lightning FSDP strategies.
Closes#21403
* Add tests for FSDP tied weight detection warning
Cover four scenarios: tied weights across units (warns), tied weights
in same unit (no warn), no shared params (no warn), no policy (no warn).
---------
Co-authored-by: Deependu <[email protected]>
* Add `log_key_prefix` parameter to `LearningRateMonitor` callback
Allow users to prepend a configurable prefix to all metric names logged
by `LearningRateMonitor`. This is useful for grouping learning rate
metrics in loggers like TensorBoard (e.g., `optim/lr-Adam` instead of
`lr-Adam`).
Fixes#21590
* Update src/lightning/pytorch/CHANGELOG.md
---------
Co-authored-by: Deependu <[email protected]>
* fix: set nonzero exit code for SIGTERMException (#19916)
`SIGTERMException(SystemExit)` was raised without arguments, causing
`SystemExit.code` to default to `None`, which Python converts to exit
code 0. Scripts monitoring the training process exit code could not
distinguish a SIGTERM termination from a successful exit.
Override `__init__` to pass 143 (128 + SIGTERM signal number 15) to
`SystemExit`, following the standard Unix convention.
Fixes#19916
* Apply suggestion from @deependujha
* Apply suggestion from @deependujha
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Update src/lightning/pytorch/utilities/exceptions.py
---------
Co-authored-by: Deependu <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Use fs.pipe() for S3/GCS checkpoint uploads in _atomic_save
Replace fs.open() + f.write() with fs.pipe() for S3 and GCS
filesystems, enabling parallel multipart uploads. This gives
4-5x throughput improvement for checkpoints >= 500 MB.
Azure is excluded because adlfs stages blocks sequentially,
making fs.pipe() slower than f.write().
Fixes#21499
Signed-off-by: c-pozzi <[email protected]>
* Use isinstance check for Azure exclusion in _atomic_save
Use module_available + isinstance instead of string comparison
to detect AzureBlobFileSystem. Fix test mocking to patch
module_available so the isinstance check works without adlfs
installed.
Signed-off-by: c-pozzi <[email protected]>
* Fix ruff SIM117: combine nested with statements in tests
Signed-off-by: c-pozzi <[email protected]>
---------
Signed-off-by: c-pozzi <[email protected]>
* Fix LearningRateMonitor TypeError on MPS with float64 lr values
When an optimizer's learning rate is np.float64, the LearningRateMonitor
crashes on MPS because MPS doesn't support float64 tensors. Fix by
converting values to Python float before creating the tensor, which
ensures compatibility with all device backends.
Fixes#20250
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* update
* update
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: deependujha <[email protected]>
* Remove NeptuneLogger integration from PyTorch Lightning
- Deleted NeptuneLogger from the logger initialization and related files.
- Updated documentation and tests to reflect the removal of NeptuneLogger.
- Adjusted logger-related code to ensure compatibility without NeptuneLogger.
* keep `NeptuneLogger` class as a wrapper and raise an error suggesting litlogger instead
* update
* update
* Update src/lightning/pytorch/loggers/neptune.py
---------
Co-authored-by: Bhimraj Yadav <[email protected]>
Co-authored-by: Justus Schock <[email protected]>
* update litlogger integration to latest version
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* update
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* fix: Skip val_check_interval validation when limit_val_batches=0
When limit_val_batches=0 (validation disabled), the val_check_interval
sanity check is now skipped since validation will not run anyway.
Fixes#21553
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Move comments inside if-elif block per review feedback
* Remove extra blank line in test docstring
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Apply suggestion from @deependujha
* Apply suggestion from @deependujha
* Add changelog entry for val_check_interval fix
* Collapse test docstring to single line
* Add co-author credit for issue reporter
Co-authored-by: Taha Yassine <[email protected]>
* update
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Deependu <[email protected]>
Co-authored-by: Taha Yassine <[email protected]>
