From 00928e3f7628b8288c17cfece5ee9600a8287f56 Mon Sep 17 00:00:00 2001 From: Michael Norris Date: Thu, 10 Sep 2026 08:01:09 -0700 Subject: [PATCH] Fix ROCm CI: build in a dedicated conda env instead of the runner's broken base (#5545) Summary: Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5545 The `linux-x86_64-GPU-w-ROCm-cmake` job fails installing AMD's ROCm torch wheels: they exist for one CPython ABI only, and the MI325 runner's interpreter is 3.14, not the 3.12 the action asks for. Nothing in `build_cmake` actually constrained the interpreter. `setup-miniconda` requests `python-version: '3.12'` and `Configure build environment` asks for `python=3.12` again, but the runner's conda base has `conda-meta` listing both `python 3.12.9` and `cpython 3.14.6`, so conda reports "already installed" while `bin/python` is 3.14. Forcing it does downgrade the interpreter, but strands conda's own compiled extensions (`pydantic_core`), and conda builds its argument parser through its plugins -- so every subsequent conda call in the job dies. Working around that with `CONDA_NO_PLUGINS` in turn drops the libmamba solver, which conda then rejects because `solver: libmamba` stays configured. Meanwhile the base env is inconsistent across three Python ABIs (conda lists ~200 conflicting packages), so there is no in-place repair to converge on. So stop repairing base. For ROCm only, `conda create` a fresh `$CONDA/envs/faiss` prefix on 3.12 and point `$CONDA` at it, leaving base untouched so conda keeps running on its own interpreter. `$CONDA` is the single lever the later steps already read (`-DPYTHON_EXECUTABLE`, `setup.py install`, the import smoke tests), so redirecting it is enough. Making that lever actually hold: - Pass `-p "$CONDA"` on every `conda install`, and `conda activate "$CONDA"`. A bare `conda install` targets the active env, which is base regardless of `$CONDA`. No-op for the non-ROCm jobs, where `$CONDA` is base. - Write base's `bin` to `$GITHUB_PATH` *before* the env's. The runner prepends entries as it reads them, so the last line written ends up first on PATH: this leaves the env's interpreter ahead of base's while keeping base reachable for `conda` itself. - Drive the ROCm torch step through `"$CONDA/bin/python" -m pip` rather than bare `python`/`pip3`, so it does not depend on PATH ordering at all. - `rm -rf` the prefix before `conda create`: it survives between runs on this self-hosted runner and `conda create` refuses an existing one. A fresh prefix each run is the point. - Derive the `libpthread_nonshared.a` symlink from `$CONDA` instead of a hardcoded `$HOME/miniconda3`, since `sysroot_linux-64` now installs into the build's prefix, and assert the target exists -- `ln -s` does not check, so a wrong path would surface much later as a link failure. - `conda list -p "$CONDA"` for the diagnostics, so the logs describe the env under test rather than base. Torch wheel installation: - Resolve through `--find-links` as a flat index instead of hardcoding a `cp312-cp312-linux_x86_64` filename; the hardcoded tag is what made an interpreter mismatch surface as "not a supported wheel on this platform". - Keep a retag fallback for pip builds that reject the unqualified `linux_x86_64` tag, but gate it on the resolver's own error text so a transient failure (network, disk) fails loudly instead of being retagged over. - `--ignore-installed`, because this runner is self-hosted and a wrong-ABI torch from an earlier run otherwise satisfies the requirement and skips tag matching. - Use AMD's wheels built against the system ROCm rather than the PyTorch Foundation `rocm7.2` wheels, which bundle an older ROCr lacking `hsa_amd_memory_get_preferred_copy_engine` and collide with the system `libamdhip64`. `ROCM_VERSION` is exported to `$GITHUB_ENV` so the apt release and the wheel release cannot drift apart. - Assert on interpreter version, torch's install location, and `torch.version.hip` rather than printing them. Each of these failures previously surfaced several steps later as something unrelated-looking. Note for infra: the base env on this runner is corrupt independently of this change (mixed 3.12/3.13/3.14 packages, and truncated `lib/python3.1/site-packages` paths from a `sys.version[:3]` bug upstream). Wiping it would let several of the workarounds above be dropped. Reviewed By: alibeklfc Differential Revision: D117896658 fbshipit-source-id: 26f98c53e5daf747dd81e0fd35232b4160f62377 --- .github/actions/build_cmake/action.yml | 191 ++++++++++++++++++------- 1 file changed, 139 insertions(+), 52 deletions(-) diff --git a/.github/actions/build_cmake/action.yml b/.github/actions/build_cmake/action.yml index 3a6f298a3..22e782f0d 100644 --- a/.github/actions/build_cmake/action.yml +++ b/.github/actions/build_cmake/action.yml @@ -50,22 +50,55 @@ runs: if: inputs.metal != 'ON' shell: bash run: | + : "${CONDA:?setup-miniconda did not export CONDA}" + # Ensure starting packages are from conda-forge. - conda list --show-channel-urls + conda list -p "$CONDA" --show-channel-urls + + # The MI325 runner's conda base cannot be repaired in place: its + # conda-meta lists python 3.12.9 and cpython 3.14.6 at once, so + # `conda install python=3.12` reports "already installed" while + # bin/python is 3.14, and forcing it strands conda's own compiled + # plugins. Build in a fresh prefix and leave base alone so conda keeps + # running on its own interpreter. ROCm-only; other runners' bases are + # fine. $CONDA is the single lever the later steps read, so pointing it + # at the new prefix redirects them all. + CONDA_HOME="$CONDA" + if [ "${{ inputs.rocm }}" = "ON" ]; then + # rm -rf, not plain create: the prefix survives on this self-hosted + # runner, and `conda create` refuses an existing one. A fresh prefix + # each run is the point -- reusing one reintroduces the stale-state + # problems this is meant to escape. + rm -rf "$CONDA_HOME/envs/faiss" + conda create -y -q -p "$CONDA_HOME/envs/faiss" python=3.12 + CONDA="$CONDA_HOME/envs/faiss" + echo "CONDA=$CONDA" >> "$GITHUB_ENV" + "$CONDA/bin/python" -c 'import sys; assert sys.version_info[:2] == (3, 12), sys.version' + fi + + # The runner prepends $GITHUB_PATH entries as it reads them, so the last + # line written ends up first on PATH: base goes first here so the env + # wins, while base stays reachable for `conda` itself. Steps that must + # not depend on this ordering call "$CONDA/bin/python" directly. + if [ "$CONDA" != "$CONDA_HOME" ]; then + echo "$CONDA_HOME/bin" >> $GITHUB_PATH + fi echo "$CONDA/bin" >> $GITHUB_PATH - conda install -y -q python=3.12 cmake=3.30.4 make=4.2 swig=4.0 "numpy>=2.0,<3.0" scipy=1.16 pytest=7.4 pytest-timeout gflags=2.2 setuptools + # -p "$CONDA" on every install below: a bare `conda install` targets + # whatever env is active, which is base regardless of the above. + conda install -y -q -p "$CONDA" python=3.12 cmake=3.30.4 make=4.2 swig=4.0 "numpy>=2.0,<3.0" scipy=1.16 pytest=7.4 pytest-timeout gflags=2.2 setuptools # install base packages for ARM64 if [ "${{ runner.arch }}" = "ARM64" ]; then - conda install -y -q -c conda-forge openblas=0.3.33 gxx_linux-aarch64=14.2 sysroot_linux-aarch64=2.17 + conda install -y -q -p "$CONDA" -c conda-forge openblas=0.3.33 gxx_linux-aarch64=14.2 sysroot_linux-aarch64=2.17 fi # install base packages for X86_64 if [ "${{ runner.arch }}" = "X64" ]; then # TODO: merge this with ARM64 - conda install -y -q -c conda-forge gxx_linux-64=14.2 sysroot_linux-64=2.17 - conda install -y -q mkl=2024.2.2 mkl-devel=2024.2.2 + conda install -y -q -p "$CONDA" -c conda-forge gxx_linux-64=14.2 sysroot_linux-64=2.17 + conda install -y -q -p "$CONDA" mkl=2024.2.2 mkl-devel=2024.2.2 fi # no CUDA needed for ROCm so skip this @@ -86,7 +119,7 @@ runs: # an older libnvJitLink.so.13 is left in the env and libcuvs fails to # load at import. PyTorch via pip supports cu132, avoiding the # conda-forge pytorch-gpu CUDA-12 limit. - conda install -y -q \ + conda install -y -q -p "$CONDA" \ cuda-nvcc=13.2 cuda-nvtx=13.2 cuda-cupti=13.2 cuda-cudart-dev=13.2 \ 'libnvjitlink>=13.3,<14' \ libcublas-dev libcusolver-dev libcusparse-dev \ @@ -95,19 +128,20 @@ runs: -c rapidsai -c rapidsai-nightly -c conda-forge -c nvidia conda clean --index-cache 2>/dev/null || true else - conda install -y -q cuda-libraries-dev=13.2 cuda-nvcc=13.2 cuda-nvtx=13.2 cuda-cupti=13.2 cuda-cudart-dev=13.2 gxx_linux-64=14.2 -c "nvidia/label/cuda-13.2" + conda install -y -q -p "$CONDA" cuda-libraries-dev=13.2 cuda-nvcc=13.2 cuda-nvtx=13.2 cuda-cupti=13.2 cuda-cudart-dev=13.2 gxx_linux-64=14.2 -c "nvidia/label/cuda-13.2" fi fi # install SVS runtime for SVS builds if [ "${{ inputs.svs }}" = "ON" ]; then - conda install -y -q libsvs-runtime=0.4.0 -c conda-forge + conda install -y -q -p "$CONDA" libsvs-runtime=0.4.0 -c conda-forge fi # install test packages if [ "${{ inputs.rocm }}" = "ON" ]; then - : # skip torch install via conda, we need to install via pip to get - # ROCm-enabled version until it's supported in conda by PyTorch + # Skip torch install via conda, we need to install via pip to get the + # ROCm-enabled version until it's supported in conda by PyTorch. + "$CONDA/bin/python" -VV elif [ "${{ inputs.gpu }}" = "ON" ]; then # PyTorch deprecated its official Anaconda channel (pytorch/pytorch#138506), # so we install via pip for CUDA builds. --break-system-packages is needed @@ -127,7 +161,7 @@ runs: fi else # TestLowLevelIVF.IVFRQ hangs on pytorch>=2.7, so left it as <2.5 for now. - conda install -y -q "pytorch<2.5" -c pytorch + conda install -y -q -p "$CONDA" "pytorch<2.5" -c pytorch fi - name: ROCm - Install dependencies if: inputs.rocm == 'ON' @@ -148,8 +182,9 @@ runs: # Get UBUNTU version name UBUNTU_VERSION_NAME=`cat /etc/os-release | grep UBUNTU_CODENAME | awk -F= '{print $2}'` - # Set ROCm version + # Exported so the torch wheel step resolves the same release. ROCM_VERSION="7.2" + echo "ROCM_VERSION=${ROCM_VERSION}" >> "$GITHUB_ENV" sudo mkdir -p /etc/apt/keyrings wget -qO /tmp/rocm.gpg.key https://repo.radeon.com/rocm/rocm.gpg.key @@ -169,7 +204,7 @@ runs: if: inputs.rocm == 'ON' shell: bash run: | - conda install -y \ + conda install -y -p "$CONDA" \ "libblas=3.9.0=35_*" \ "libcblas=3.9.0=35_*" \ "liblapack=3.9.0=35_*" @@ -189,7 +224,13 @@ runs: sudo ln -s /lib/x86_64-linux-gnu/libc.so.6 /lib64/libc.so.6 sudo ln -s /lib/x86_64-linux-gnu/libc_nonshared.a /usr/lib64/libc_nonshared.a sudo ln -s /usr/lib/x86_64-linux-gnu/libpthread.so.0 /lib64/libpthread.so.0 - sudo ln -s $HOME/miniconda3/x86_64-conda-linux-gnu/sysroot/usr/lib64/libpthread_nonshared.a /usr/lib64/libpthread_nonshared.a + # From $CONDA, not $HOME/miniconda3: sysroot_linux-64 is installed into + # whichever prefix the build uses. `ln -s` does not check its target, so + # a wrong path here would surface much later as a missing + # libpthread_nonshared.a at link time. + SYSROOT_LIB="$CONDA/x86_64-conda-linux-gnu/sysroot/usr/lib64" + test -f "$SYSROOT_LIB/libpthread_nonshared.a" + sudo ln -s "$SYSROOT_LIB/libpthread_nonshared.a" /usr/lib64/libpthread_nonshared.a - name: Print NVIDIA GPU info if: inputs.gpu == 'ON' && inputs.rocm != 'ON' shell: bash @@ -215,8 +256,8 @@ runs: shell: bash run: | eval "$(conda shell.bash hook)" - conda activate - conda list --show-channel-urls + conda activate "$CONDA" + conda list -p "$CONDA" --show-channel-urls cmake -B build \ -DBUILD_TESTING=ON \ -DBUILD_SHARED_LIBS=ON \ @@ -252,7 +293,7 @@ runs: if: inputs.metal != 'ON' shell: bash run: | - conda list --show-channel-urls + conda list -p "$CONDA" --show-channel-urls export GTEST_OUTPUT="xml:$(realpath .)/test-results/googletest/" make -C build test - name: C++ tests (Metal) @@ -272,64 +313,110 @@ runs: if: inputs.rocm == 'OFF' && inputs.metal != 'ON' shell: bash run: | - conda list --show-channel-urls + conda list -p "$CONDA" --show-channel-urls find ./build/perf_tests/ -executable -type f -name "bench*" -exec '{}' -v \; - name: Install Python extension if: inputs.metal != 'ON' shell: bash working-directory: build/faiss/python run: | - conda list --show-channel-urls + conda list -p "$CONDA" --show-channel-urls $CONDA/bin/python setup.py install - name: ROCm - install ROCm-enabled torch via pip if: inputs.rocm == 'ON' shell: bash run: | - conda list --show-channel-urls - conda install -y -q "pip<26" + set -euo pipefail + conda list -p "$CONDA" --show-channel-urls - # Install AMD's torch wheels built against the exact system ROCm (7.2.0, - # from apt/7.2). The PyTorch Foundation whl/rocm7.2 wheels float and bundle - # an older ROCr that lacks hsa_amd_memory_get_preferred_copy_engine (ROCr - # 1.18.0, added in ROCm 7.0.0), which collides with the system 7.2.0 - # libamdhip64. Matching both sides to 7.2.0 avoids the dual-runtime - # undefined-symbol/segfault failures. - BASE="https://repo.radeon.com/rocm/manylinux/rocm-rel-7.2" - HTML="$(wget -qO- "${BASE}/")" - pick() { - printf '%s\n' "$HTML" \ - | grep -oE "$1-[0-9][^\"]*rocm7\.2\.0[^\"]*cp312-cp312-linux_x86_64\.whl" \ - | sort -V | tail -1 - } - TORCH_WHL="$(pick torch)" - TVISION_WHL="$(pick torchvision)" - TAUDIO_WHL="$(pick torchaudio)" - echo "Selected: ${TORCH_WHL} ${TVISION_WHL} ${TAUDIO_WHL}" - if [ -z "${TORCH_WHL}" ] || [ -z "${TVISION_WHL}" ] || [ -z "${TAUDIO_WHL}" ]; then - echo "Could not locate +rocm7.2.0 cp312 wheels at ${BASE}" >&2 - exit 1 + conda install -y -q -p "$CONDA" "pip<26" wheel + + # Name the interpreter rather than trusting PATH, and drive pip through + # it, so nothing here depends on $GITHUB_PATH ordering. + PY="$CONDA/bin/python" + + # These wheels exist for one CPython ABI only, so bail with the version in + # hand rather than letting it read as "no matching distribution". + "$PY" -c 'import sys; assert sys.version_info[:2] == (3, 12), "need CPython 3.12 for the ROCm wheels, got %s" % sys.version' + + # Install AMD's torch wheels built against the exact system ROCm, from + # apt. The PyTorch Foundation wheels float and bundle an older ROCr that + # lacks hsa_amd_memory_get_preferred_copy_engine (ROCr 1.18.0, added in + # ROCm 7.0.0), which collides with the system libamdhip64. Matching both + # sides avoids the dual-runtime undefined-symbol/segfault failures. + ROCM_WHEEL_VERSION="${ROCM_VERSION}.0" + BASE="https://repo.radeon.com/rocm/manylinux/rocm-rel-${ROCM_VERSION}" + + "$PY" -m pip --version + "$PY" -c "import sys, sysconfig; print(sys.version, sysconfig.get_platform())" + + # Treat the directory listing as a flat index so pip matches wheel tags + # against the running interpreter. --no-deps avoids pulling + # triton/pytorch-triton-rocm (only on AMD's index, not PyPI); torch's + # import-time deps are installed explicitly below. --ignore-installed + # because this runner is self-hosted: a wrong-ABI torch left by an earlier + # run otherwise satisfies the requirement and skips tag matching entirely. + # `|| true` then inspect: a transient failure (network, disk) must not be + # mistaken for a tag rejection and retagged over. + RESOLVE_LOG="$(mktemp)" + if ! "$PY" -m pip install --no-deps --ignore-installed --no-index \ + --find-links "${BASE}/" torch torchvision torchaudio 2>&1 \ + | tee "$RESOLVE_LOG"; then + if ! grep -qE "No matching distribution found|Could not find a version" "$RESOLVE_LOG"; then + echo "pip failed for a reason other than wheel-tag rejection" >&2 + exit 1 + fi + # pip builds that reject the unqualified linux_x86_64 tag these wheels + # carry see an empty index, so retag: the wheels need a glibc well + # below the runner's, only the tag is wrong. + echo "Flat-index resolve found no candidate; retagging wheels" >&2 + PYTAG="$("$PY" -c 'import sys; print("cp%d%d" % sys.version_info[:2])')" + ROCM_RE="${ROCM_WHEEL_VERSION//./\\.}" + HTML="$(wget -qO- "${BASE}/")" + WHEELS=() + for pkg in torch torchvision torchaudio; do + # `|| true`: no match exits grep 1, which pipefail turns into a bare + # abort before the diagnostic below can run. + WHL="$(printf '%s\n' "$HTML" \ + | grep -oE "${pkg}-[0-9][^\"]*rocm${ROCM_RE}[^\"]*${PYTAG}-${PYTAG}-linux_x86_64\.whl" \ + | sort -V | tail -1 || true)" + if [ -z "${WHL}" ]; then + echo "Could not locate +rocm${ROCM_WHEEL_VERSION} ${PYTAG} ${pkg} wheel at ${BASE}" >&2 + exit 1 + fi + echo "Selected: ${WHL}" + # The listing percent-encodes the '+'; the URL needs it encoded, the + # file on disk must not be. + wget -q "${BASE}/${WHL}" -O "${WHL//%2B/+}" + WHEELS+=("$("$PY" -m wheel tags --platform-tag manylinux_2_28_x86_64 \ + --remove "${WHL//%2B/+}")") + done + "$PY" -m pip install --no-deps --ignore-installed "${WHEELS[@]}" fi + "$PY" -m pip install numpy pillow filelock typing-extensions sympy networkx jinja2 fsspec - # --no-deps avoids pulling triton/pytorch-triton-rocm (only on AMD's index, - # not PyPI); install torch's import-time deps explicitly instead. - pip3 install --no-deps \ - "${BASE}/${TORCH_WHL}" "${BASE}/${TVISION_WHL}" "${BASE}/${TAUDIO_WHL}" - pip3 install numpy pillow filelock typing-extensions sympy networkx jinja2 fsspec + # Check torch resolves to this interpreter's own site-packages before + # trusting it: a stale tree left on the self-hosted runner has shadowed + # the fresh install before, and a wrong-ABI hit there surfaces as an + # unrelated-looking "Failed to load PyTorch C extensions". + "$PY" -c "import sysconfig, torch; p = sysconfig.get_paths()['purelib']; assert torch.__file__.startswith(p), 'torch loaded from %s, expected under %s' % (torch.__file__, p)" - # Confirm torch links against ROCm 7.2 (matches the system /opt/rocm-7.2.0). - python -c "import torch; print('torch', torch.__version__, 'hip', torch.version.hip)" + # A HIP mismatch here is the dual-runtime problem above, much cheaper to + # read than the resulting segfault. Kept on one line: an unindented + # continuation would terminate this block scalar and break the manifest. + "$PY" -c "import os, sys, torch; hip = torch.version.hip or ''; want = os.environ['ROCM_VERSION']; print('torch', torch.__version__, 'hip', hip); sys.exit(None if hip.startswith(want) else 'expected a HIP %s build, got %s' % (want, hip))" - name: Python tests (CPU only) if: inputs.gpu == 'OFF' && inputs.metal != 'ON' shell: bash run: | - conda list --show-channel-urls + conda list -p "$CONDA" --show-channel-urls pytest --junitxml=test-results/pytest/results.xml tests/test_*.py pytest --junitxml=test-results/pytest/results-torch.xml tests/torch_*.py - name: Python tests (CPU + GPU) if: inputs.gpu == 'ON' shell: bash run: | - conda list --show-channel-urls + conda list -p "$CONDA" --show-channel-urls pytest --junitxml=test-results/pytest/results.xml tests/test_*.py pytest --junitxml=test-results/pytest/results-torch.xml tests/torch_*.py cp tests/common_faiss_tests.py faiss/gpu/test @@ -342,7 +429,7 @@ runs: if: inputs.opt_level == 'avx2' shell: bash run: | - conda list --show-channel-urls + conda list -p "$CONDA" --show-channel-urls FAISS_DISABLE_CPU_FEATURES=AVX2 LD_DEBUG=libs $CONDA/bin/python -c "import faiss" 2>&1 | grep faiss.so LD_DEBUG=libs $CONDA/bin/python -c "import faiss" 2>&1 | grep faiss_avx2.so - name: Upload test results @@ -356,4 +443,4 @@ runs: shell: bash run: | # Shows that all installed packages are from conda-forge. - conda list --show-channel-urls + conda list -p "$CONDA" --show-channel-urls