* fix lora full state checkpointing in FSDP2
* Add CPU regression test for FSDP2 adapter-only state dict
* Update src/accelerate/utils/fsdp_utils.py
* Update src/accelerate/utils/fsdp_utils.py
* Update tests/fsdp/peft_checkpointing.py
---------
Co-authored-by: Marc Sun <[email protected]>
* update docker
* fix
* other fixes
* fix
* remove
* fix
* not needed anymore
* Rework nightly CI for speed and drop comet-ml test dependency
The nightly run took 1h30, dominated by run_deepspeed_tests_multi_gpu (88min)
which ran the whole tests/deepspeed suite twice: once via `make test_deepspeed`
and again via `make test_integrations` (which included tests/deepspeed).
- nightly.yml: collapse the four copy-pasted jobs into a single matrix job;
each suite now runs exactly once per GPU shape (deepspeed jobs only run
test_deepspeed, fsdp/tp/examples are covered by the core-image entries)
- Makefile: `test` now runs all suites even if one fails (full nightly signal)
and fails at the end; remove `test_integrations` (orphaned by the above) and
`extra_quality_checks` (references utils/check_*.py scripts that never
existed in this repo)
- setup.py: remove comet-ml from the test_trackers extra; its import-time
auto-patching breaks the example tests, which is why every CI job had to
`pip uninstall comet_ml` before running them. The CometMLTracker itself is
unaffected and its tests skip via require_comet_ml.
- test.yml / run_merge_tests.yml / self_hosted_integration_tests.yml: drop the
now-unneeded comet_ml uninstall steps
* TEMP: trigger image build + chained nightly tests on this branch for validation
Rebuilds the docker images on pushes touching docker/** or setup.py (including
this branch, temporarily) and runs the nightly test suite on the fresh images.
Before merging, scope the push trigger back to main only.
* Fix remaining multi-GPU nightly failures
- test_peak_memory_usage.py: last remaining bare "glue" dataset id; new
huggingface_hub rejects un-namespaced repo ids (HfUriError), use nyu-mll/glue
like every other test script
- test_sync.py: raise rtol to 1e-2 in the grad-accumulation sync check; the
observed reduction-order drift reaches ~5e-3 by iteration 4 on 2 GPUs, above
the previous 2e-3
* Document comet-ml removal in test_trackers extra
* TEMP: run nightly tests on pushes to this branch (validation only, remove before merge)
* Pin debug_launcher gloo mesh to loopback
test_cpu.py::MultiCPUTester::test_ops flakes on hosted runners with
"Gloo connectFullMesh failed ... Connection closed by peer": gloo resolves the
machine hostname to pick the interface for its TCP mesh, which on CI runners
points at a transient internal IP. debug_launcher only ever spawns processes on
the local machine, so force the mesh over loopback via GLOO_SOCKET_IFNAME=lo.
* Fix grad-sync test at the source: split tolerance by assertion intent
check_model_parameters used one tolerance for two opposite checks: synced
grads must match (needs loose compare, reduction roundoff reaches ~5e-3) and
unsynced grads must differ (needs strict compare, the real cross-rank gap can
be as small as ~1e-2). No single rtol fits between those, so the failure just
moved between the two assertions depending on the run. The loosened rtol now
only applies to the must-match check; the must-differ check uses strict
allclose defaults.
* Remove temporary CI-validation triggers
Reverts the branch-scoped push triggers and the docker-build -> nightly chain
that were only there to validate the CI rework on this branch:
- nightly.yml: back to workflow_dispatch + schedule only
- build_docker_images.yml: restored to its main state
* Trim verbose comments
* Remove comet-ml removal note from setup.py
* add support for SwanLabTracker and update related documentation
* add emoji in FRAMWORK
* apply the style corrections and quality control
* add support for SwanLabTracker in tests
* fix bug in test_tracking
* Initial test
* Try on push
* Only wf dispatch now
* keep trying
* Try again
* Try again
* source activate?
* Force bash
* Source activate accelerate to make it get the env propelry
* try using nightly docker
* Try this?
* Try this?
* Try this, proper output
* Try this, proper output
* Try via full conda activate(?)
* rm conda
* te fp8 tests
* add ao
* ao in setup too
* actually include fp8 deps
* FP8 docker image, use newer version
* Update docker image to take in input
* Test
* prior month
* igpu?
* Use only last 2 digits of year
* Build rest
* Apply style fixes
---------
Co-authored-by: [[ -z $EMAIL ]] && read -e -p "Enter your email (for git configuration): " EMAIL <[email protected]>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* Added artifacts and figure tracking at MLFlow tracker
* Added `log_artifact` to the MLFlowTracker
* Remove changes
* Added artifacts, artifacts and figure tracking at MLFlow tracker
* Improved the docstring
* added require_mlflow function at test_utils
* add test for MLflowTracker
* Bit of litting
* Refactor to a more robust test
* Revised the test asserts to something more robust.
* Removed incorrect import and some litting.
* removed commented code
* initiate tracker using Accelerator
* Added mlflow and matplotlib to setup.py. Guarded and decoredated the functions that required them.
* Guarded mlflow import
* added matplotlib required warning.
* ran style and quality
* init
* style
* is_hpu_available
* fix
* import habana_frameworks.torch.distributed.hccl
* style
* test
* initialize dist proc group
* revert
* set backend to hccl only if hccl initialization sets a local rank
* force backend hccl and multi_hpu type when sure of distributed launch
* style
* pass accelerator tests
* pas big modeling tests with bigger atol/rtol for accelerators
* fix hpu device count and skip tests requiring hpu:x
* hpu autocast
* hpu rng_state
* hpu launch
* hpu special device placement
* hpu launch
* rng state
* distributed data loop tests
* enforce non contiguity after device memory allocation
* pass fsdp tests
* enforce pt_hpu_lazy_mode=0 when fsdp testing
* pass cli tests
* pass and document grad sync tests
* pass kwargs handler and autocast tests
* memory utils
* found source of int64 errors
* skip some modeling utils tests
* enable int64
* skip optimizer tests
* pass checkpointing tests
* pass accelerator tests with safetensors main
* more hpu stuff
* style
* remove PT_HPU_LAZY_MODE and PT_ENABLE_INT64_SUPPORT as they should be in the testing environment
* start testing on gaudi2
* support fp16 on gaudi2
* add testing order
* custom hpu fsdp env dict
* fix torch trace malloc
* test ddp half precision comm hooks
* fix
* fix
* remove lower bound for hpu
* use 0.72 as lower bound
* lower lower bound
* order deepspeed tests
* fix
* deepspeed_use_hpu
* assert non lazy mode with offloaded optimizer
* make patching torch with habana frameworks the default
* less of require_non_hpu
* skip test_multi_device_merge_fsdp_weights for now as it halts
* skip another flaky test
* format
* use habana_visible_modules
* patch torch hpu device count
* avoid setting HABANA_VISIBLE_MODULES
* don't play with habana visible devices/modules
* only with hpu
* fixes and skips
* skip
* fix device ids and add some todos
* skip offloading with generate()
* fix
* reduced atol/rtol for hpu
* fix
* tag deepspeed tests that should run first
* enable a test path that was skipped
* revert a test that was customized for gaudi1
* some patching to enable HABANA_VISIBLE_MODULES
* fix zero3 test
* misc
* test DTensor TP
* remove gaudi1
* test
* style
* comment
* pass pad_across_processes
* require_fp16
* pass memory utils test
* test_ddp_comm_hook
* skip half precision comm hooks on hpu
* fix
* is_fp16_available
* fp16
* tp as part of integration tests
* fix
* write_basic_config
* safetensors
* local sgd and masked_fill_fwd_i64
* fix num_processes in test_load_states_by_steps
* fp8 support
* test
* fix
* add a workflow
* Update src/accelerate/accelerator.py
* review comments
* ci
* style
* comments
* test
* habana_frameworks.torch
* patch device count
* fix
* fix
* require_fp8
* fix
* fix
* gaudi 1
* remove unnecessary
* fixed maskd fill error in transformers
* style
* balanced_memory pass on hpu
* remove for now
* run first
* Apply suggestions from code review
* style after merge
* Update src/accelerate/accelerator.py
Co-authored-by: Zach Mueller <[email protected]>
* Update src/accelerate/utils/transformer_engine.py
Co-authored-by: Zach Mueller <[email protected]>
* empty cache review comments
* test_scirpt.py error messages
* AccelerateTestCase for accelerator state cleanup
* test
* add gaudi1 workflow
* fp8 avilability
* fix
* reduce batch size
* concurrency
* check cuda as well
* nits and comments
* mark fsdp tests that require_fp16
* style
* mark deepspeed fp16 tests
* update image
* fix
* updated
* better msgs
* skip pippy
* test
* test on 2 device
* support up to 1% relative error in test_accelerate
* skip hpu fp16
* allow for 1 byte differene
* revert torch_device change
* style
* skip memory release since it's flaky
* add accelerator state cleanup to fixture
* fix
* atol
* fix
* more rtol
* equal grad test
* revert
* pass pippy on gaudi2 and skip on gaudi1
* enable sd 1.5 test with require fp16
* added warning on memory release
* don't log warning in memory release as it requires PartialState to be initialized
* Apply suggestions from code review
---------
Co-authored-by: Zach Mueller <[email protected]>
* MNT Upgrade ruff to 0.6.4
Currently used version, 0.2.1, is quite old at this point.
Not a lot needed to be changed:
- Change ruff version in setup.py
- Remove deprecated ignore-init-module-imports option for ruff
- Type comparison should use is and not ==
- Use f-string instead of % formatting
- Some line wrapping and empty lines
* Oops
* Initial commit
* Now to test
* Store false
* Slight tweaks
* Fix naming
* Got it all working with tests
* Use not for safetensors arg
* rm change
* Add docs
* Adjust based on Marc's feedback
* Specify just weights
* Update tests to include CLI and swap namings
* Fin
* Rm unused
* Rm again
Marc Sun