facebookresearch/faiss
1
0
Fork 0
mirror of https://github.com/facebookresearch/faiss.git synced 2026-10-11 22:50:00 +00:00
Code Issues Packages Projects Releases Wiki Activity

Fix ROCm CI: build in a dedicated conda env instead of the runner's broken base (#5545)

Browse Source 
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5545

The `linux-x86_64-GPU-w-ROCm-cmake` job fails installing AMD's ROCm torch wheels: they exist for one CPython ABI only, and the MI325 runner's interpreter is 3.14, not the 3.12 the action asks for.

Nothing in `build_cmake` actually constrained the interpreter. `setup-miniconda` requests `python-version: '3.12'` and `Configure build environment` asks for `python=3.12` again, but the runner's conda base has `conda-meta` listing both `python 3.12.9` and `cpython 3.14.6`, so conda reports "already installed" while `bin/python` is 3.14. Forcing it does downgrade the interpreter, but strands conda's own compiled extensions (`pydantic_core`), and conda builds its argument parser through its plugins -- so every subsequent conda call in the job dies. Working around that with `CONDA_NO_PLUGINS` in turn drops the libmamba solver, which conda then rejects because `solver: libmamba` stays configured. Meanwhile the base env is inconsistent across three Python ABIs (conda lists ~200 conflicting packages), so there is no in-place repair to converge on.

So stop repairing base. For ROCm only, `conda create` a fresh `$CONDA/envs/faiss` prefix on 3.12 and point `$CONDA` at it, leaving base untouched so conda keeps running on its own interpreter. `$CONDA` is the single lever the later steps already read (`-DPYTHON_EXECUTABLE`, `setup.py install`, the import smoke tests), so redirecting it is enough.

Making that lever actually hold:
- Pass `-p "$CONDA"` on every `conda install`, and `conda activate "$CONDA"`. A bare `conda install` targets the active env, which is base regardless of `$CONDA`. No-op for the non-ROCm jobs, where `$CONDA` is base.
- Write base's `bin` to `$GITHUB_PATH` *before* the env's. The runner prepends entries as it reads them, so the last line written ends up first on PATH: this leaves the env's interpreter ahead of base's while keeping base reachable for `conda` itself.
- Drive the ROCm torch step through `"$CONDA/bin/python" -m pip` rather than bare `python`/`pip3`, so it does not depend on PATH ordering at all.
- `rm -rf` the prefix before `conda create`: it survives between runs on this self-hosted runner and `conda create` refuses an existing one. A fresh prefix each run is the point.
- Derive the `libpthread_nonshared.a` symlink from `$CONDA` instead of a hardcoded `$HOME/miniconda3`, since `sysroot_linux-64` now installs into the build's prefix, and assert the target exists -- `ln -s` does not check, so a wrong path would surface much later as a link failure.
- `conda list -p "$CONDA"` for the diagnostics, so the logs describe the env under test rather than base.

Torch wheel installation:
- Resolve through `--find-links` as a flat index instead of hardcoding a `cp312-cp312-linux_x86_64` filename; the hardcoded tag is what made an interpreter mismatch surface as "not a supported wheel on this platform".
- Keep a retag fallback for pip builds that reject the unqualified `linux_x86_64` tag, but gate it on the resolver's own error text so a transient failure (network, disk) fails loudly instead of being retagged over.
- `--ignore-installed`, because this runner is self-hosted and a wrong-ABI torch from an earlier run otherwise satisfies the requirement and skips tag matching.
- Use AMD's wheels built against the system ROCm rather than the PyTorch Foundation `rocm7.2` wheels, which bundle an older ROCr lacking `hsa_amd_memory_get_preferred_copy_engine` and collide with the system `libamdhip64`. `ROCM_VERSION` is exported to `$GITHUB_ENV` so the apt release and the wheel release cannot drift apart.
- Assert on interpreter version, torch's install location, and `torch.version.hip` rather than printing them. Each of these failures previously surfaced several steps later as something unrelated-looking.

Note for infra: the base env on this runner is corrupt independently of this change (mixed 3.12/3.13/3.14 packages, and truncated `lib/python3.1/site-packages` paths from a `sys.version[:3]` bug upstream). Wiping it would let several of the workarounds above be dropped.

Reviewed By: alibeklfc

Differential Revision: D117896658

fbshipit-source-id: 26f98c53e5daf747dd81e0fd35232b4160f62377
This commit is contained in:
Michael Norris
2026-09-10 08:01:09 -07:00
committed by meta-codesync[bot]
parent 5702cafcb7
commit 00928e3f76
1 changed files with 139 additions and 52 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/actions/build_cmake/action.yml
+139 -52
View File
@@ -50,22 +50,55 @@ runs:
if: inputs.metal != 'ON'
shell: bash
run: |
: "${CONDA:?setup-miniconda did not export CONDA}"
# Ensure starting packages are from conda-forge.
conda list --show-channel-urls
conda list -p "$CONDA" --show-channel-urls
# The MI325 runner's conda base cannot be repaired in place: its
# conda-meta lists python 3.12.9 and cpython 3.14.6 at once, so
# `conda install python=3.12` reports "already installed" while
# bin/python is 3.14, and forcing it strands conda's own compiled
# plugins. Build in a fresh prefix and leave base alone so conda keeps
# running on its own interpreter. ROCm-only; other runners' bases are
# fine. $CONDA is the single lever the later steps read, so pointing it
# at the new prefix redirects them all.
CONDA_HOME="$CONDA"
if [ "${{ inputs.rocm }}" = "ON" ]; then
# rm -rf, not plain create: the prefix survives on this self-hosted
# runner, and `conda create` refuses an existing one. A fresh prefix
# each run is the point -- reusing one reintroduces the stale-state
# problems this is meant to escape.
rm -rf "$CONDA_HOME/envs/faiss"
conda create -y -q -p "$CONDA_HOME/envs/faiss" python=3.12
CONDA="$CONDA_HOME/envs/faiss"
echo "CONDA=$CONDA" >> "$GITHUB_ENV"
"$CONDA/bin/python" -c 'import sys; assert sys.version_info[:2] == (3, 12), sys.version'
fi
# The runner prepends $GITHUB_PATH entries as it reads them, so the last
# line written ends up first on PATH: base goes first here so the env
# wins, while base stays reachable for `conda` itself. Steps that must
# not depend on this ordering call "$CONDA/bin/python" directly.
if [ "$CONDA" != "$CONDA_HOME" ]; then
echo "$CONDA_HOME/bin" >> $GITHUB_PATH
fi
echo "$CONDA/bin" >> $GITHUB_PATH
conda install -y -q python=3.12 cmake=3.30.4 make=4.2 swig=4.0 "numpy>=2.0,<3.0" scipy=1.16 pytest=7.4 pytest-timeout gflags=2.2 setuptools
# -p "$CONDA" on every install below: a bare `conda install` targets
# whatever env is active, which is base regardless of the above.
conda install -y -q -p "$CONDA" python=3.12 cmake=3.30.4 make=4.2 swig=4.0 "numpy>=2.0,<3.0" scipy=1.16 pytest=7.4 pytest-timeout gflags=2.2 setuptools
# install base packages for ARM64
if [ "${{ runner.arch }}" = "ARM64" ]; then
conda install -y -q -c conda-forge openblas=0.3.33 gxx_linux-aarch64=14.2 sysroot_linux-aarch64=2.17
conda install -y -q -p "$CONDA" -c conda-forge openblas=0.3.33 gxx_linux-aarch64=14.2 sysroot_linux-aarch64=2.17
fi
# install base packages for X86_64
if [ "${{ runner.arch }}" = "X64" ]; then
# TODO: merge this with ARM64
conda install -y -q -c conda-forge gxx_linux-64=14.2 sysroot_linux-64=2.17
conda install -y -q mkl=2024.2.2 mkl-devel=2024.2.2
conda install -y -q -p "$CONDA" -c conda-forge gxx_linux-64=14.2 sysroot_linux-64=2.17
conda install -y -q -p "$CONDA" mkl=2024.2.2 mkl-devel=2024.2.2
fi
# no CUDA needed for ROCm so skip this
@@ -86,7 +119,7 @@ runs:
# an older libnvJitLink.so.13 is left in the env and libcuvs fails to
# load at import. PyTorch via pip supports cu132, avoiding the
# conda-forge pytorch-gpu CUDA-12 limit.
conda install -y -q \
conda install -y -q -p "$CONDA" \
cuda-nvcc=13.2 cuda-nvtx=13.2 cuda-cupti=13.2 cuda-cudart-dev=13.2 \
'libnvjitlink>=13.3,<14' \
libcublas-dev libcusolver-dev libcusparse-dev \
@@ -95,19 +128,20 @@ runs:
-c rapidsai -c rapidsai-nightly -c conda-forge -c nvidia
conda clean --index-cache 2>/dev/null || true
else
conda install -y -q cuda-libraries-dev=13.2 cuda-nvcc=13.2 cuda-nvtx=13.2 cuda-cupti=13.2 cuda-cudart-dev=13.2 gxx_linux-64=14.2 -c "nvidia/label/cuda-13.2"
conda install -y -q -p "$CONDA" cuda-libraries-dev=13.2 cuda-nvcc=13.2 cuda-nvtx=13.2 cuda-cupti=13.2 cuda-cudart-dev=13.2 gxx_linux-64=14.2 -c "nvidia/label/cuda-13.2"
fi
fi
# install SVS runtime for SVS builds
if [ "${{ inputs.svs }}" = "ON" ]; then
conda install -y -q libsvs-runtime=0.4.0 -c conda-forge
conda install -y -q -p "$CONDA" libsvs-runtime=0.4.0 -c conda-forge
fi
# install test packages
if [ "${{ inputs.rocm }}" = "ON" ]; then
: # skip torch install via conda, we need to install via pip to get
# ROCm-enabled version until it's supported in conda by PyTorch
# Skip torch install via conda, we need to install via pip to get the
# ROCm-enabled version until it's supported in conda by PyTorch.
"$CONDA/bin/python" -VV
elif [ "${{ inputs.gpu }}" = "ON" ]; then
# PyTorch deprecated its official Anaconda channel (pytorch/pytorch#138506),
# so we install via pip for CUDA builds. --break-system-packages is needed
@@ -127,7 +161,7 @@ runs:
fi
else
# TestLowLevelIVF.IVFRQ hangs on pytorch>=2.7, so left it as <2.5 for now.
conda install -y -q "pytorch<2.5" -c pytorch
conda install -y -q -p "$CONDA" "pytorch<2.5" -c pytorch
fi
- name: ROCm - Install dependencies
if: inputs.rocm == 'ON'
@@ -148,8 +182,9 @@ runs:
# Get UBUNTU version name
UBUNTU_VERSION_NAME=`cat /etc/os-release | grep UBUNTU_CODENAME | awk -F= '{print $2}'`
# Set ROCm version
# Exported so the torch wheel step resolves the same release.
ROCM_VERSION="7.2"
echo "ROCM_VERSION=${ROCM_VERSION}" >> "$GITHUB_ENV"
sudo mkdir -p /etc/apt/keyrings
wget -qO /tmp/rocm.gpg.key https://repo.radeon.com/rocm/rocm.gpg.key
@@ -169,7 +204,7 @@ runs:
if: inputs.rocm == 'ON'
shell: bash
run: |
conda install -y \
conda install -y -p "$CONDA" \
"libblas=3.9.0=35_*" \
"libcblas=3.9.0=35_*" \
"liblapack=3.9.0=35_*"
@@ -189,7 +224,13 @@ runs:
sudo ln -s /lib/x86_64-linux-gnu/libc.so.6 /lib64/libc.so.6
sudo ln -s /lib/x86_64-linux-gnu/libc_nonshared.a /usr/lib64/libc_nonshared.a
sudo ln -s /usr/lib/x86_64-linux-gnu/libpthread.so.0 /lib64/libpthread.so.0
sudo ln -s $HOME/miniconda3/x86_64-conda-linux-gnu/sysroot/usr/lib64/libpthread_nonshared.a /usr/lib64/libpthread_nonshared.a
# From $CONDA, not $HOME/miniconda3: sysroot_linux-64 is installed into
# whichever prefix the build uses. `ln -s` does not check its target, so
# a wrong path here would surface much later as a missing
# libpthread_nonshared.a at link time.
SYSROOT_LIB="$CONDA/x86_64-conda-linux-gnu/sysroot/usr/lib64"
test -f "$SYSROOT_LIB/libpthread_nonshared.a"
sudo ln -s "$SYSROOT_LIB/libpthread_nonshared.a" /usr/lib64/libpthread_nonshared.a
- name: Print NVIDIA GPU info
if: inputs.gpu == 'ON' && inputs.rocm != 'ON'
shell: bash
@@ -215,8 +256,8 @@ runs:
shell: bash
run: |
eval "$(conda shell.bash hook)"
conda activate
conda list --show-channel-urls
conda activate "$CONDA"
conda list -p "$CONDA" --show-channel-urls
cmake -B build \
-DBUILD_TESTING=ON \
-DBUILD_SHARED_LIBS=ON \
@@ -252,7 +293,7 @@ runs:
if: inputs.metal != 'ON'
shell: bash
run: |
conda list --show-channel-urls
conda list -p "$CONDA" --show-channel-urls
export GTEST_OUTPUT="xml:$(realpath .)/test-results/googletest/"
make -C build test
- name: C++ tests (Metal)
@@ -272,64 +313,110 @@ runs:
if: inputs.rocm == 'OFF' && inputs.metal != 'ON'
shell: bash
run: |
conda list --show-channel-urls
conda list -p "$CONDA" --show-channel-urls
find ./build/perf_tests/ -executable -type f -name "bench*" -exec '{}' -v \;
- name: Install Python extension
if: inputs.metal != 'ON'
shell: bash
working-directory: build/faiss/python
run: |
conda list --show-channel-urls
conda list -p "$CONDA" --show-channel-urls
$CONDA/bin/python setup.py install
- name: ROCm - install ROCm-enabled torch via pip
if: inputs.rocm == 'ON'
shell: bash
run: |
conda list --show-channel-urls
conda install -y -q "pip<26"
set -euo pipefail
conda list -p "$CONDA" --show-channel-urls
# Install AMD's torch wheels built against the exact system ROCm (7.2.0,
# from apt/7.2). The PyTorch Foundation whl/rocm7.2 wheels float and bundle
# an older ROCr that lacks hsa_amd_memory_get_preferred_copy_engine (ROCr
# 1.18.0, added in ROCm 7.0.0), which collides with the system 7.2.0
# libamdhip64. Matching both sides to 7.2.0 avoids the dual-runtime
# undefined-symbol/segfault failures.
BASE="https://repo.radeon.com/rocm/manylinux/rocm-rel-7.2"
HTML="$(wget -qO- "${BASE}/")"
pick() {
printf '%s\n' "$HTML" \
| grep -oE "$1-[0-9][^\"]*rocm7\.2\.0[^\"]*cp312-cp312-linux_x86_64\.whl" \
| sort -V | tail -1
}
TORCH_WHL="$(pick torch)"
TVISION_WHL="$(pick torchvision)"
TAUDIO_WHL="$(pick torchaudio)"
echo "Selected: ${TORCH_WHL} ${TVISION_WHL} ${TAUDIO_WHL}"
if [ -z "${TORCH_WHL}" ] || [ -z "${TVISION_WHL}" ] || [ -z "${TAUDIO_WHL}" ]; then
echo "Could not locate +rocm7.2.0 cp312 wheels at ${BASE}" >&2
exit 1
conda install -y -q -p "$CONDA" "pip<26" wheel
# Name the interpreter rather than trusting PATH, and drive pip through
# it, so nothing here depends on $GITHUB_PATH ordering.
PY="$CONDA/bin/python"
# These wheels exist for one CPython ABI only, so bail with the version in
# hand rather than letting it read as "no matching distribution".
"$PY" -c 'import sys; assert sys.version_info[:2] == (3, 12), "need CPython 3.12 for the ROCm wheels, got %s" % sys.version'
# Install AMD's torch wheels built against the exact system ROCm, from
# apt. The PyTorch Foundation wheels float and bundle an older ROCr that
# lacks hsa_amd_memory_get_preferred_copy_engine (ROCr 1.18.0, added in
# ROCm 7.0.0), which collides with the system libamdhip64. Matching both
# sides avoids the dual-runtime undefined-symbol/segfault failures.
ROCM_WHEEL_VERSION="${ROCM_VERSION}.0"
BASE="https://repo.radeon.com/rocm/manylinux/rocm-rel-${ROCM_VERSION}"
"$PY" -m pip --version
"$PY" -c "import sys, sysconfig; print(sys.version, sysconfig.get_platform())"
# Treat the directory listing as a flat index so pip matches wheel tags
# against the running interpreter. --no-deps avoids pulling
# triton/pytorch-triton-rocm (only on AMD's index, not PyPI); torch's
# import-time deps are installed explicitly below. --ignore-installed
# because this runner is self-hosted: a wrong-ABI torch left by an earlier
# run otherwise satisfies the requirement and skips tag matching entirely.
# `|| true` then inspect: a transient failure (network, disk) must not be
# mistaken for a tag rejection and retagged over.
RESOLVE_LOG="$(mktemp)"
if ! "$PY" -m pip install --no-deps --ignore-installed --no-index \
--find-links "${BASE}/" torch torchvision torchaudio 2>&1 \
| tee "$RESOLVE_LOG"; then
if ! grep -qE "No matching distribution found|Could not find a version" "$RESOLVE_LOG"; then
echo "pip failed for a reason other than wheel-tag rejection" >&2
exit 1
fi
# pip builds that reject the unqualified linux_x86_64 tag these wheels
# carry see an empty index, so retag: the wheels need a glibc well
# below the runner's, only the tag is wrong.
echo "Flat-index resolve found no candidate; retagging wheels" >&2
PYTAG="$("$PY" -c 'import sys; print("cp%d%d" % sys.version_info[:2])')"
ROCM_RE="${ROCM_WHEEL_VERSION//./\\.}"
HTML="$(wget -qO- "${BASE}/")"
WHEELS=()
for pkg in torch torchvision torchaudio; do
# `|| true`: no match exits grep 1, which pipefail turns into a bare
# abort before the diagnostic below can run.
WHL="$(printf '%s\n' "$HTML" \
| grep -oE "${pkg}-[0-9][^\"]*rocm${ROCM_RE}[^\"]*${PYTAG}-${PYTAG}-linux_x86_64\.whl" \
| sort -V | tail -1 || true)"
if [ -z "${WHL}" ]; then
echo "Could not locate +rocm${ROCM_WHEEL_VERSION} ${PYTAG} ${pkg} wheel at ${BASE}" >&2
exit 1
fi
echo "Selected: ${WHL}"
# The listing percent-encodes the '+'; the URL needs it encoded, the
# file on disk must not be.
wget -q "${BASE}/${WHL}" -O "${WHL//%2B/+}"
WHEELS+=("$("$PY" -m wheel tags --platform-tag manylinux_2_28_x86_64 \
--remove "${WHL//%2B/+}")")
done
"$PY" -m pip install --no-deps --ignore-installed "${WHEELS[@]}"
fi
"$PY" -m pip install numpy pillow filelock typing-extensions sympy networkx jinja2 fsspec
# --no-deps avoids pulling triton/pytorch-triton-rocm (only on AMD's index,
# not PyPI); install torch's import-time deps explicitly instead.
pip3 install --no-deps \
"${BASE}/${TORCH_WHL}" "${BASE}/${TVISION_WHL}" "${BASE}/${TAUDIO_WHL}"
pip3 install numpy pillow filelock typing-extensions sympy networkx jinja2 fsspec
# Check torch resolves to this interpreter's own site-packages before
# trusting it: a stale tree left on the self-hosted runner has shadowed
# the fresh install before, and a wrong-ABI hit there surfaces as an
# unrelated-looking "Failed to load PyTorch C extensions".
"$PY" -c "import sysconfig, torch; p = sysconfig.get_paths()['purelib']; assert torch.__file__.startswith(p), 'torch loaded from %s, expected under %s' % (torch.__file__, p)"
# Confirm torch links against ROCm 7.2 (matches the system /opt/rocm-7.2.0).
python -c "import torch; print('torch', torch.__version__, 'hip', torch.version.hip)"
# A HIP mismatch here is the dual-runtime problem above, much cheaper to
# read than the resulting segfault. Kept on one line: an unindented
# continuation would terminate this block scalar and break the manifest.
"$PY" -c "import os, sys, torch; hip = torch.version.hip or ''; want = os.environ['ROCM_VERSION']; print('torch', torch.__version__, 'hip', hip); sys.exit(None if hip.startswith(want) else 'expected a HIP %s build, got %s' % (want, hip))"
- name: Python tests (CPU only)
if: inputs.gpu == 'OFF' && inputs.metal != 'ON'
shell: bash
run: |
conda list --show-channel-urls
conda list -p "$CONDA" --show-channel-urls
pytest --junitxml=test-results/pytest/results.xml tests/test_*.py
pytest --junitxml=test-results/pytest/results-torch.xml tests/torch_*.py
- name: Python tests (CPU + GPU)
if: inputs.gpu == 'ON'
shell: bash
run: |
conda list --show-channel-urls
conda list -p "$CONDA" --show-channel-urls
pytest --junitxml=test-results/pytest/results.xml tests/test_*.py
pytest --junitxml=test-results/pytest/results-torch.xml tests/torch_*.py
cp tests/common_faiss_tests.py faiss/gpu/test
@@ -342,7 +429,7 @@ runs:
if: inputs.opt_level == 'avx2'
shell: bash
run: |
conda list --show-channel-urls
conda list -p "$CONDA" --show-channel-urls
FAISS_DISABLE_CPU_FEATURES=AVX2 LD_DEBUG=libs $CONDA/bin/python -c "import faiss" 2>&1 | grep faiss.so
LD_DEBUG=libs $CONDA/bin/python -c "import faiss" 2>&1 | grep faiss_avx2.so
- name: Upload test results
@@ -356,4 +443,4 @@ runs:
shell: bash
run: |
# Shows that all installed packages are from conda-forge.
conda list --show-channel-urls
conda list -p "$CONDA" --show-channel-urls