Summary:
## What
Split the existing VPOPCNTDQ RaBitQ and Hamming kernels from the full Sapphire Rapids SIMD level.
DD builds now expose an `AVX512_VPOPCNT` capability for CPUs such as Ice Lake and Zen 4. The fallback chain is:
`AVX512_SPR -> AVX512_VPOPCNT -> AVX512 -> AVX2 -> NONE`
The ordinary static AVX512 target remains unchanged and contains no VPOPCNT instructions.
## Why
The VPOPCNT kernels only require baseline AVX-512 plus `AVX512_VPOPCNTDQ`. Tying them to the SPR level unnecessarily excluded CPUs that support VPOPCNTDQ but not AVX512-FP16, notably Zen 4.
## Changes made while importing
Two changes were needed on top of the pull request.
**Feature detection read the wrong CPUID leaf.** `AVX512_VPOPCNTDQ` and `AVX512_VNNI` were read from leaf 1 ECX. Both live in leaf 7 subleaf 0 ECX, at bits 14 and 11. Leaf 1 ECX holds unrelated bits at those positions, so the new level was never selected on any CPU. Measured on an AMD Genoa host before the fix: the level reported as unavailable and detection returned `AVX512`. After the fix the same host reports `AVX512_VPOPCNT`.
**The build definitions outside CMake were not updated.** The pull request renames three files. The other build description still named the old paths and had no entry for the new level, so the library did not build. This adds the level constant, its compiler flags, its `COMPILE_SIMD_AVX512_VPOPCNT` define, and the renamed paths.
**The new level was not registered in the compiled-level mask.** `compiled_simd_levels()` reports which levels a binary holds, and `FAISS_SIMD_LEVEL` walks down from the requested level until it finds one that is present. The new level had no entry, so the override always stepped past it to `AVX512` and printed that the level was not compiled in. This was reported by `test_dispatch_with_env_var` on the open-source dynamic-dispatch build.
**The test helper inferred available levels from `/proc/cpuinfo`.** A CPU feature does not imply that a level is compiled into the build, so the helper listed a level the override then refused. It now asks `SIMDConfig::is_simd_level_available()`, which reports compiled and supported. That is a subset of compiled, so every level it returns is one the override accepts.
The mask and helper names also follow the naming used in the parent diff, so `AVAILABLE_SIMD_LEVELS_BASE_WITH_VPOPCNT` and `with_simd_level_with_vpopcnt`.
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5531
Test Plan:
```
buck2 test fbcode//mode/opt -c faiss.dynamic_dispatch=true \
fbcode//faiss/tests:test_simd_levels fbcode//faiss/tests:test_rabitq_simd
```
Result: all pass.
The build failures the imported version reported are cleared. All three configurations build:
```
buck2 build fbcode//mode/dev fbcode//faiss:faiss fbcode//faiss:faiss_no_multithreading fbcode//faiss:faiss_omp_mock
buck2 build fbcode//mode/opt -c faiss.dynamic_dispatch=true <same three targets>
buck2 build fbcode//mode/dev -c fbcode.arch=aarch64 <same three targets>
```
Detection on an AMD Genoa host, which has AVX512_VPOPCNTDQ and AVX512_BF16 but not AVX512_FP16:
| | before the CPUID fix | after |
| --- | --- | --- |
| detected level | `AVX512` | `AVX512_VPOPCNT` |
| `AVX512_VPOPCNT` available | 0 | 1 |
| `AVX512_SPR` available | 0 | 0 |
Reviewed By: alibeklfc
Differential Revision: D118969754
Pulled By: mnorris11
fbshipit-source-id: d1ca52b614829a3d4b322c9fecf0c004f2174dab
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5349
# this is a no-op change
Ran `black -l 80` over all tracked Python files under `fbcode/faiss/`, using line length 80 to match faiss's documented coding style (`CONTRIBUTING.md`: "80 character line length (both for C++ and Python)"). 172 files were reformatted; 28 were already compliant.
This is a pure-formatting change. `black` runs its AST-equivalence safety check on every file, so no identifiers, string/numeric values, operators, or control flow were altered — only whitespace, indentation, line wrapping, blank lines, trailing commas, and string-quote normalization (single to double quotes).
`fbcode/faiss/` is intentionally excluded from Meta's standard Python autoformatter (`arc f` / `arc lint -a` report no formatter for these paths) because it mirrors to public GitHub, so the Python had drifted out of any consistent style. The C++ is already clang-format-clean (0 of 807 files change), so this effort is Python-only. This diff brings the Python to canonical `black` formatting.
Excluded `demos/index_pq_flat_separate_codes_from_codebook.py`: it contains a Jupyter `!rm` shell-magic line and is not valid Python, so `black` cannot parse it.
This is the base of a 3-diff stack: (1) this black reformat; (2) `.flake8` alignment (max-line-length 80 + black-compatible ignores E203/W503); (3) genuine lint fixes (F401/F403/F841/B011).
Reviewed By: mnorris11
Differential Revision: D109736099
fbshipit-source-id: 9a054635766ed11aa3760a788395babd3d198abf
Summary:
I've verified the successful compilation via docker + qemu combination.
Also, this PR introduces all the needed overrides, which rely on default implementation, but later can be populated with the vectorized code.
Also, fixes a typo in SQ4U.
algoriddle I would appreciate if you could take a look whenever you have time. Thanks!
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5156
Reviewed By: mdouze
Differential Revision: D102911491
Pulled By: mnorris11
fbshipit-source-id: f3555987cdbf8d32a4fbc9d07ec2307f0f8a112a
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5158
Several DD-dispatched code paths produce integer outputs (codes,
top-k indices) where bit-exact equivalence between SIMD levels is the
right correctness bar, but the existing tests assert only level-
independent properties (decode error, recall) that wouldn't catch
silent dispatch drift between AVX2/AVX512/NEON/NONE implementations.
Adds NoneSIMDLevel context manager in common_faiss_tests.py:
codes = quantizer.compute_codes(xb)
with NoneSIMDLevel():
codes_none = quantizer.compute_codes(xb)
np.testing.assert_array_equal(codes, codes_none)
The context manager pins SIMDConfig to NONE for the duration of the
block, restores the previous level on exit (incl. exception), and
raises SkipTest when NONE isn't compiled in (static-AVX2 lane).
Adds test_codes_match_none / test_PQ4_search_matches_none methods to
five existing for_all_simd_levels test classes:
* TestScalarQuantizerEncodeDecode (test_scalar_quantizer_correctness)
* TestProductQuantizer (test_product_quantizer)
* TestRaBitQuantizerEncodeDecode (test_rabitq)
* TestSearch (test_fast_scan; checks (D, I) output of PQ16x4fs search)
* TestResidualQuantizer (test_residual_quantizer; replaces inlined
set_level/restore from D102821184 with the new helper)
Caveat documented in the docstring: SkipTest raised inside subTest is
swallowed; tests using both must do an explicit availability check
before the loop. Applied where needed.
Reviewed By: mdouze
Differential Revision: D102989453
fbshipit-source-id: 475ee504271e0505615841ef82263776245419e0
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/4798
Add ARM SVE (Scalable Vector Extension) support for FAISS distance
functions.
- Add ARM_SVE to SIMDLevel enum and dispatch macros
- Add distances_arm_sve.cpp with SVE implementations
- Wire up SVE in both build systems:
- Buck: SIMD_FILES/SIMD_FLAGS in xplat.bzl
- CMake: FAISS_SIMD_SVE_SRC, COMPILE_SIMD_ARM_SVE definition, and per-file -march=armv8.2-a+sve flags in DD mode
- SVE is disabled by default (ARM_SVE: False in dispatch_config)
Reviewed By: mdouze
Differential Revision: D92819013
fbshipit-source-id: cb37fc09efb34364fc88dde3389b76232fb29d59
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3452
Delete all remaining print within the Tests to improve the readability and effectiveness of the codebase.
Reviewed By: junjieqi
Differential Revision: D57466393
fbshipit-source-id: 6ebd66ae2e769894d810d4ba7a5f69fc865b797d
Summary: Useful info on github test runs is burried in spurious logging. Avoid this.
Reviewed By: mlomeli1
Differential Revision: D47209139
fbshipit-source-id: b5111c91e2b94f0c3678d599197f8e7094993df1
lyang24