Marc Sun
a5cb37c178
* update docker * fix * other fixes * fix * remove * fix * not needed anymore * Rework nightly CI for speed and drop comet-ml test dependency The nightly run took 1h30, dominated by run_deepspeed_tests_multi_gpu (88min) which ran the whole tests/deepspeed suite twice: once via `make test_deepspeed` and again via `make test_integrations` (which included tests/deepspeed). - nightly.yml: collapse the four copy-pasted jobs into a single matrix job; each suite now runs exactly once per GPU shape (deepspeed jobs only run test_deepspeed, fsdp/tp/examples are covered by the core-image entries) - Makefile: `test` now runs all suites even if one fails (full nightly signal) and fails at the end; remove `test_integrations` (orphaned by the above) and `extra_quality_checks` (references utils/check_*.py scripts that never existed in this repo) - setup.py: remove comet-ml from the test_trackers extra; its import-time auto-patching breaks the example tests, which is why every CI job had to `pip uninstall comet_ml` before running them. The CometMLTracker itself is unaffected and its tests skip via require_comet_ml. - test.yml / run_merge_tests.yml / self_hosted_integration_tests.yml: drop the now-unneeded comet_ml uninstall steps * TEMP: trigger image build + chained nightly tests on this branch for validation Rebuilds the docker images on pushes touching docker/** or setup.py (including this branch, temporarily) and runs the nightly test suite on the fresh images. Before merging, scope the push trigger back to main only. * Fix remaining multi-GPU nightly failures - test_peak_memory_usage.py: last remaining bare "glue" dataset id; new huggingface_hub rejects un-namespaced repo ids (HfUriError), use nyu-mll/glue like every other test script - test_sync.py: raise rtol to 1e-2 in the grad-accumulation sync check; the observed reduction-order drift reaches ~5e-3 by iteration 4 on 2 GPUs, above the previous 2e-3 * Document comet-ml removal in test_trackers extra * TEMP: run nightly tests on pushes to this branch (validation only, remove before merge) * Pin debug_launcher gloo mesh to loopback test_cpu.py::MultiCPUTester::test_ops flakes on hosted runners with "Gloo connectFullMesh failed ... Connection closed by peer": gloo resolves the machine hostname to pick the interface for its TCP mesh, which on CI runners points at a transient internal IP. debug_launcher only ever spawns processes on the local machine, so force the mesh over loopback via GLOO_SOCKET_IFNAME=lo. * Fix grad-sync test at the source: split tolerance by assertion intent check_model_parameters used one tolerance for two opposite checks: synced grads must match (needs loose compare, reduction roundoff reaches ~5e-3) and unsynced grads must differ (needs strict compare, the real cross-rank gap can be as small as ~1e-2). No single rtol fits between those, so the failure just moved between the two assertions depending on the run. The loosened rtol now only applies to the must-match check; the must-differ check uses strict allclose defaults. * Remove temporary CI-validation triggers Reverts the branch-scoped push triggers and the docker-build -> nightly chain that were only there to validate the CI rework on this branch: - nightly.yml: back to workflow_dispatch + schedule only - build_docker_images.yml: restored to its main state * Trim verbose comments * Remove comet-ml removal note from setup.py
Official Hugging Face Accelerate Docker Images
Accelerate publishes a variety of docker versions as part of our CI that users can also use. These are stable images that Accelerate can run off of which comes with a variety of different setup configurations, all of which are officially hosted on Docker Hub.
A breakdown of each are given below
Naming Conventions
Accelerate docker images follow a tagging convention of:
huggingface/accelerate:{accelerator}-{nightly,release}
accelerator in this instance is one of many applical pre-configured backend supports:
gpu: Comes compiled off of thenvidia/cudaimage and includes core parts likebitsandbytes. Runs off python 3.9.cpu: Comes compiled off ofpython:3.9-slimand is designed for non-CUDA based workloads.- More to come soon
gpu-deepspeed: Comes compiled off of thenvidia/cudaimage and includes core parts likebitsandbytesas well as the latestdeepspeedversion. Runs off python 3.10.gpu-fp8-transformerengine: Comes compiled off ofnvcr.io/nvidia/pytorchand is specifically for running thebenchmarks/fp8scripts on devices which support FP8 operations using theTransformerEnginelibrary (RTX 4090, H100, etc)
Nightlies vs Releases
Each release a new build is pushed with a version number included in the name. For a GPU-supported image of version 0.28.0 for instance, it would look like the following:
huggingface/accelerate:gpu-release-0.28.0
Nightlies contain two different image tags. There is a general nightly tag which is built each night, and a nightly-YYYY-MM-DD which corresponds to a build from a particular date.
For instance, here is an example nightly CPU image from 3/14/2024
huggingface/accelerate:cpu-nightly-2024-03-14
Running the images
Each image comes compiled with conda and an accelerate environment contains all of the installed dependencies.
To pull down the latest nightly run:
docker pull huggingface/accelerate:gpu-nightly
To then run it in interactive mode with GPU-memory available, run:
docker container run --gpus all -it huggingface/accelerate:gpu-nightly
DEPRECATED IMAGES
CPU and GPU docker images were hosted at huggingface/accelerate-gpu and huggingface/accelerate-cpu. These builds are now outdated and will not receive updates.
The builds at the corresponding huggingface/accelerate:{gpu,cpu} contain the same Dockerfile, so it's as simple as changing the docker image to the desired ones from above. We will not be deleting these images for posterity, but they will not be receiving updates going forward.