mirror of
https://github.com/huggingface/accelerate.git
synced 2026-10-12 02:15:39 +00:00
Marc Sun
a5cb37c178
* update docker * fix * other fixes * fix * remove * fix * not needed anymore * Rework nightly CI for speed and drop comet-ml test dependency The nightly run took 1h30, dominated by run_deepspeed_tests_multi_gpu (88min) which ran the whole tests/deepspeed suite twice: once via `make test_deepspeed` and again via `make test_integrations` (which included tests/deepspeed). - nightly.yml: collapse the four copy-pasted jobs into a single matrix job; each suite now runs exactly once per GPU shape (deepspeed jobs only run test_deepspeed, fsdp/tp/examples are covered by the core-image entries) - Makefile: `test` now runs all suites even if one fails (full nightly signal) and fails at the end; remove `test_integrations` (orphaned by the above) and `extra_quality_checks` (references utils/check_*.py scripts that never existed in this repo) - setup.py: remove comet-ml from the test_trackers extra; its import-time auto-patching breaks the example tests, which is why every CI job had to `pip uninstall comet_ml` before running them. The CometMLTracker itself is unaffected and its tests skip via require_comet_ml. - test.yml / run_merge_tests.yml / self_hosted_integration_tests.yml: drop the now-unneeded comet_ml uninstall steps * TEMP: trigger image build + chained nightly tests on this branch for validation Rebuilds the docker images on pushes touching docker/** or setup.py (including this branch, temporarily) and runs the nightly test suite on the fresh images. Before merging, scope the push trigger back to main only. * Fix remaining multi-GPU nightly failures - test_peak_memory_usage.py: last remaining bare "glue" dataset id; new huggingface_hub rejects un-namespaced repo ids (HfUriError), use nyu-mll/glue like every other test script - test_sync.py: raise rtol to 1e-2 in the grad-accumulation sync check; the observed reduction-order drift reaches ~5e-3 by iteration 4 on 2 GPUs, above the previous 2e-3 * Document comet-ml removal in test_trackers extra * TEMP: run nightly tests on pushes to this branch (validation only, remove before merge) * Pin debug_launcher gloo mesh to loopback test_cpu.py::MultiCPUTester::test_ops flakes on hosted runners with "Gloo connectFullMesh failed ... Connection closed by peer": gloo resolves the machine hostname to pick the interface for its TCP mesh, which on CI runners points at a transient internal IP. debug_launcher only ever spawns processes on the local machine, so force the mesh over loopback via GLOO_SOCKET_IFNAME=lo. * Fix grad-sync test at the source: split tolerance by assertion intent check_model_parameters used one tolerance for two opposite checks: synced grads must match (needs loose compare, reduction roundoff reaches ~5e-3) and unsynced grads must differ (needs strict compare, the real cross-rank gap can be as small as ~1e-2). No single rtol fits between those, so the failure just moved between the two assertions depending on the run. The loosened rtol now only applies to the must-match check; the must-differ check uses strict allclose defaults. * Remove temporary CI-validation triggers Reverts the branch-scoped push triggers and the docker-build -> nightly chain that were only there to validate the CI rework on this branch: - nightly.yml: back to workflow_dispatch + schedule only - build_docker_images.yml: restored to its main state * Trim verbose comments * Remove comet-ml removal note from setup.py