Summary:
This diff adds native ARM NEON kernels for the RaBitQ bitwise popcount
operations and for the multi-bit inner product. The kernels replace the scalar
fallbacks on ARM. They handle every supported bit width and every unaligned
dimension.
- `tests/test_rabitq_simd_neon.cpp` holds the correctness test, and
`tests/BUCK` holds its `cpp_unittest` target. The target carries
`target_compatible_with` on `ovr_config//cpu:arm64`, which mirrors the
x86_64 gate on `test_rabitq_simd`.
- `tests/test_rabitq_simd_util.h` holds the shared `kDims` list and the shared
`random_bytes` helper. `test_rabitq_simd.cpp` and `test_rabitq_simd_dd.cpp`
now use them too.
- The three kernels that index a fixed array of eight accumulators assert
`qb <= kMaxQueryBits`. `RaBitQuantizer::set_query` caps `qb` at 8, but the
kernels are public templates and carry no other bound. The scalar kernel and
the AVX-512 kernel need no such assertion, because they shift inside the loop
and hold one accumulator.
The multi-bit test compares with a relative tolerance of `2e-5`. The NEON kernel
accumulates in eight lanes, so it does not reproduce the scalar sum bit for bit.
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5532
Test Plan:
`buck2 test fbcode//faiss/tests:test_ivf_index fbcode//faiss/tests:test_custom_result_handler fbcode//faiss/tests:test_hnsw fbcode//faiss/tests:test_rabitq_simd_neon fbcode//faiss/tests:test_params_override fbcode//faiss/tests:test_hnsw_rabitq fbcode//faiss/tests:test_rabitq fbcode//faiss/tests:test_search_params fbcode//faiss/tests:test_io_corrupted fbcode//faiss/tests:test_rabitq_utils`, run at the top of the stack on aarch64: 173 pass, 0 fail, 3 skip.
`buck2 build fbcode//faiss:faiss` succeeds in the static aarch64 build and with `-c faiss.dynamic_dispatch=true`.
The host holds Neoverse V2 cores with ASIMDDP, so `test_rabitq_simd_neon` runs
its kernels natively rather than skipping.
Reviewed By: alibeklfc
Differential Revision: D121846717
Pulled By: mnorris11
fbshipit-source-id: de079613a52ced326045830fb799dfc9b169fe2e
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5620
**TL;DR:** Eight SIMD kernel tests never ran, and two byte-domain kernels trust their caller. This change makes three fixes:
1. **Six tests never compiled.** `test_rabitq_simd.cpp` hides them behind `COMPILE_SIMD_*` macros that the faiss target does not export.
2. **Two tests skipped themselves in a static build.** `TestSQByteDirectAcrossSIMDLevels` needs `SIMDLevel::NONE` as a reference, and a static build does not report that level.
3. **The byte kernels never test their dimension requirement.** `DistanceComputerByte` and `DistanceComputerByteSigned` read 16 bytes per loop step.
## 1. Six tests never compiled
`test_rabitq_simd.cpp` wraps two blocks in `#ifdef COMPILE_SIMD_AVX2` and `#ifdef COMPILE_SIMD_AVX512`. The faiss target sets those macros as `preprocessor_flags`, and it does not export them. A test translation unit never sees them. So the preprocessor deleted both blocks. The file holds nine tests and ran three.
The AVX2 block loses its guard, because every x86 faiss build holds the AVX2 kernels. Its three tests also lose an `#ifdef FAISS_ENABLE_DD` around a runtime skip, which the same problem disabled. `is_simd_level_available()` reports the right answer in a static build too.
The AVX512 block moves to a dynamic dispatch target. A static x86 build compiles the AVX2 kernels only, so the test fails to link there rather than skips. `test_rabitq_simd_vpopcnt.cpp` becomes `test_rabitq_simd_dd.cpp`, because it now holds every RaBitQ test that a static build cannot link. A new `test_rabitq_simd_util.h` holds the scalar reference that both files call.
## 2. Two tests skipped themselves in a static build
`TestSQByteDirectAcrossSIMDLevels` measured the index a second time at `SIMDLevel::NONE`, then compared the two runs. A static build reports only the level it compiles, and omits `NONE`, so both tests skipped.
The tests now compute the distances from the definition. The distances are integers, so the reference is exact. The tests check absolute values now, instead of the agreement of two faiss paths.
## 3. The byte kernels trust the caller
`DistanceComputerByte` and `DistanceComputerByteSigned` read 16 bytes per loop step, so they require `d % 16 == 0`. `sq-dispatch.h` enforces that, and no other caller builds them.
A caller that ignores the requirement reads up to 15 bytes past each code and returns a wrong distance. Each constructor now tests the requirement and throws. The AVX2 and the ARM_NEON specializations both get the test.
Reviewed By: alibeklfc
Differential Revision: D119545379
fbshipit-source-id: ddfdcc30d3662f469b2ec3dbb9d5b9f4dafa21aa
Summary:
## What
Split the existing VPOPCNTDQ RaBitQ and Hamming kernels from the full Sapphire Rapids SIMD level.
DD builds now expose an `AVX512_VPOPCNT` capability for CPUs such as Ice Lake and Zen 4. The fallback chain is:
`AVX512_SPR -> AVX512_VPOPCNT -> AVX512 -> AVX2 -> NONE`
The ordinary static AVX512 target remains unchanged and contains no VPOPCNT instructions.
## Why
The VPOPCNT kernels only require baseline AVX-512 plus `AVX512_VPOPCNTDQ`. Tying them to the SPR level unnecessarily excluded CPUs that support VPOPCNTDQ but not AVX512-FP16, notably Zen 4.
## Changes made while importing
Two changes were needed on top of the pull request.
**Feature detection read the wrong CPUID leaf.** `AVX512_VPOPCNTDQ` and `AVX512_VNNI` were read from leaf 1 ECX. Both live in leaf 7 subleaf 0 ECX, at bits 14 and 11. Leaf 1 ECX holds unrelated bits at those positions, so the new level was never selected on any CPU. Measured on an AMD Genoa host before the fix: the level reported as unavailable and detection returned `AVX512`. After the fix the same host reports `AVX512_VPOPCNT`.
**The build definitions outside CMake were not updated.** The pull request renames three files. The other build description still named the old paths and had no entry for the new level, so the library did not build. This adds the level constant, its compiler flags, its `COMPILE_SIMD_AVX512_VPOPCNT` define, and the renamed paths.
**The new level was not registered in the compiled-level mask.** `compiled_simd_levels()` reports which levels a binary holds, and `FAISS_SIMD_LEVEL` walks down from the requested level until it finds one that is present. The new level had no entry, so the override always stepped past it to `AVX512` and printed that the level was not compiled in. This was reported by `test_dispatch_with_env_var` on the open-source dynamic-dispatch build.
**The test helper inferred available levels from `/proc/cpuinfo`.** A CPU feature does not imply that a level is compiled into the build, so the helper listed a level the override then refused. It now asks `SIMDConfig::is_simd_level_available()`, which reports compiled and supported. That is a subset of compiled, so every level it returns is one the override accepts.
The mask and helper names also follow the naming used in the parent diff, so `AVAILABLE_SIMD_LEVELS_BASE_WITH_VPOPCNT` and `with_simd_level_with_vpopcnt`.
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5531
Test Plan:
```
buck2 test fbcode//mode/opt -c faiss.dynamic_dispatch=true \
fbcode//faiss/tests:test_simd_levels fbcode//faiss/tests:test_rabitq_simd
```
Result: all pass.
The build failures the imported version reported are cleared. All three configurations build:
```
buck2 build fbcode//mode/dev fbcode//faiss:faiss fbcode//faiss:faiss_no_multithreading fbcode//faiss:faiss_omp_mock
buck2 build fbcode//mode/opt -c faiss.dynamic_dispatch=true <same three targets>
buck2 build fbcode//mode/dev -c fbcode.arch=aarch64 <same three targets>
```
Detection on an AMD Genoa host, which has AVX512_VPOPCNTDQ and AVX512_BF16 but not AVX512_FP16:
| | before the CPUID fix | after |
| --- | --- | --- |
| detected level | `AVX512` | `AVX512_VPOPCNT` |
| `AVX512_VPOPCNT` available | 0 | 1 |
| `AVX512_SPR` available | 0 | 0 |
Reviewed By: alibeklfc
Differential Revision: D118969754
Pulled By: mnorris11
fbshipit-source-id: d1ca52b614829a3d4b322c9fecf0c004f2174dab
Summary:
Fuse RaBitQ’s AND-dot and doc-side popcount into a single SIMD pass, avoiding a second scan of the same binary code.
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5412
Reviewed By: mnorris11
Differential Revision: D112829641
Pulled By: alibeklfc
fbshipit-source-id: d8015661fe27e656d34a210ddeec7789aed1802e
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5392
{F1991891064}
RaBitQ distance computation needs the quantized query bits laid out in "bit-plane" format. The transformation operation for this has to happen for every single residual therefore it is a fixed cost per-iterator. I noticed that this was quite significant for very high cluster counts.
the code for it is:
```
for (size_t idim = 0; idim < d; idim++) {
for (size_t iv = 0; iv < qb; iv++) {
const bool bit = ((rotated_qq[idim] & (1 << iv)) != 0);
rearranged_rotated_qq[iv * offset + idim / 8] |=
bit ? (1 << (idim % 8)) : 0;
}
}
```
When qb=4, each dimension is 4-bits wide representing values 0-15. `rotated_qq[i]` is all 4 bits for dimension i. We need to rotate into bit-plan so that `bit_plane_j` is all the j-bits. That is, `bit_plane_3` is `d` bits, each one the 3rd bit of its qb=4 dimension. Right now this is done with a for-each-dimension for-each-bit double loop as seen above.
Instead of going one-dimension-at-a-time we can do SIMD 32 dimensions at once with a 256-bit AVX2 register. Turn those 0-15 into uint8 (this way it works for any qb <= 8, the higher bits are just set to 0) and load 32 into a register, then for each bit plane j, mask every uint8 with 1 << j and check if the bit is set. Then use `movemask` to compress just the 1/0 result from 256 bits down to 32 bits. That's the block the plane needs.
Number of operations is reduced by 32.
Reviewed By: alibeklfc
Differential Revision: D101289092
fbshipit-source-id: baa81f56edfdab70d0129d1319a089b7acfef54f
lyang24