facebookresearch/faiss
1
0
Fork 0
mirror of https://github.com/facebookresearch/faiss.git synced 2026-10-11 22:50:00 +00:00
Code Issues Packages Projects Releases Wiki Activity

Fix cuVS GPU CI timeout: pin CUDA toolkit to 13.2 (#5401)

Browse Source 
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5401

`faiss/gpu/test/test_gpu_index_serialize.py` was timing out (10 min hard cap) on the GitHub `Linux x86_64 GPU w/ cuVS (cmake)` job, hanging inside `index_cpu_to_gpu` on the first cuVS GPU operation.

Root cause: D106837134 bumped the conda cuVS toolkit from CUDA 12.9 to 13.3. The 13.3 `cudart` deadlocks cuVS GPU kernels on the T4 runner. The change's stated intent was only to pull `libnvJitLink` up to 13.3 — the conda `libcuvs=26.06` build links it and needs the `__nvJitLinkCreate_13_3` symbol — but `cuda-nvcc`, `cuda-cudart-dev`, `cuda-cupti`, and `cuda-nvtx` were bumped to 13.3 alongside it. CUDA 13.2 is already the version used by both the non-cuVS GPU job and the pip-wheel cuVS build (whose GPU smoke test passes on the same T4 runner class), and it matches the runner driver. Only `libnvJitLink` actually needs to be 13.3.

Fix: pin the CUDA toolkit (`cuda-nvcc`/`cuda-cudart-dev`/`cuda-cupti`/`cuda-nvtx` and `cuda-version`) back to 13.2 for the cuVS cmake build, keeping the explicit `libnvjitlink>=13.3,<14` pin so `libcuvs` still loads.

passed in https://github.com/facebookresearch/faiss/actions/runs/29109874975/job/86419354334?pr=5401

Reviewed By: alibeklfc

Differential Revision: D111466548

fbshipit-source-id: 6eef20488605981f2326aaa2fb628b3315b9ed75
This commit is contained in:
Michael Norris
2026-07-10 17:33:47 -07:00
committed by meta-codesync[bot]
parent ea107641d4
commit 63378f0ace
2 changed files with 22 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/actions/build_cmake/action.yml
+15 -10
View File
@@ -74,19 +74,24 @@ runs:
# CUDA install for GPU builds
elif [ "${{ inputs.gpu }}" = "ON" ]; then
if [ "${{ inputs.cuvs }}" = "ON" ]; then
# Use CUDA 13.3 for cuVS cmake builds: the conda libcuvs 26.06 build
# links libnvJitLink 13.3 (needs the __nvJitLinkCreate_13_3 symbol).
# The nvjitlink runtime lib is a separate package that cuda-nvcc /
# cuda-cudart-dev / cuda-version do NOT pull to 13.3, so pin
# libnvjitlink explicitly — otherwise an older libnvJitLink.so.13 is
# left in the env and libcuvs fails to load at import. PyTorch via pip
# supports cu132, avoiding the conda-forge pytorch-gpu CUDA-12 limit.
# Pin the CUDA toolkit (nvcc + cudart/cupti/nvtx) to 13.2, not 13.3:
# the 13.3 cudart deadlocks cuVS GPU kernels on the T4 runner (the
# first cuVS op in index_cpu_to_gpu hangs, timing out
# test_gpu_index_serialize). CUDA 13.2 matches both the non-cuVS GPU
# job and the pip-wheel cuVS build (whose GPU smoke test passes), and
# the T4 driver. Only libnvJitLink must be 13.3: the conda libcuvs
# 26.06 build links it (needs the __nvJitLinkCreate_13_3 symbol), and
# it is a standalone package that cuda-nvcc / cuda-cudart-dev /
# cuda-version do NOT pull to 13.3, so pin it explicitly — otherwise
# an older libnvJitLink.so.13 is left in the env and libcuvs fails to
# load at import. PyTorch via pip supports cu132, avoiding the
# conda-forge pytorch-gpu CUDA-12 limit.
conda install -y -q \
cuda-nvcc=13.3 cuda-nvtx=13.3 cuda-cupti=13.3 cuda-cudart-dev=13.3 \
cuda-nvcc=13.2 cuda-nvtx=13.2 cuda-cupti=13.2 cuda-cudart-dev=13.2 \
'libnvjitlink>=13.3,<14' \
libcublas-dev libcusolver-dev libcusparse-dev \
gxx_linux-64=14.2 \
libcuvs=26.06 'cuda-version>=13.3,<14' sysroot_linux-64=2.34 \
libcuvs=26.06 'cuda-version>=13.2,<14' sysroot_linux-64=2.34 \
-c rapidsai -c rapidsai-nightly -c conda-forge -c nvidia
conda clean --index-cache 2>/dev/null || true
else
@@ -116,7 +121,7 @@ runs:
# conda 13.3 lib that cuVS 26.06 needs, so faiss then fails to load with
# "undefined symbol __nvJitLinkCreate_13_3". No torch cu133 build exists,
# so upgrade the pip nvjitlink to 13.3 (symbol-complete, CUDA-13 forward
# compatible). cuVS-only: the non-cuVS GPU path is CUDA 12.
# compatible). cuVS-only: the non-cuVS GPU path does not use libcuvs.
if [ "${{ inputs.cuvs }}" = "ON" ]; then
pip install --break-system-packages --upgrade "nvidia-nvjitlink>=13.3,<14"
fi