mirror of
https://github.com/facebookresearch/faiss.git
synced 2026-10-12 02:13:36 +00:00
main
4
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Jaeyoon Jeong
|
e7c44eb000 |
Add AVX-512 fp16 conversion kernels (#5694)
Summary: Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5694 Add exact AVX-512 widening and fp32 accumulation specializations for packed fp16 data. Reuse the existing AVX-512 SIMD translation unit, expand runtime dispatch to the AVX-512 level, and cover conversion tails, large blocks, weighted and unweighted centroid updates, and architecture fallbacks. Reviewed By: mnorris11 Differential Revision: D122781509 fbshipit-source-id: 67cbc4546149f069706e80dd25cae25d3588b0cc |
||
|
lyang24
|
e6bd1d252c |
Add AVX512 VPOPCNT dynamic dispatch level (#5531)
Summary: ## What Split the existing VPOPCNTDQ RaBitQ and Hamming kernels from the full Sapphire Rapids SIMD level. DD builds now expose an `AVX512_VPOPCNT` capability for CPUs such as Ice Lake and Zen 4. The fallback chain is: `AVX512_SPR -> AVX512_VPOPCNT -> AVX512 -> AVX2 -> NONE` The ordinary static AVX512 target remains unchanged and contains no VPOPCNT instructions. ## Why The VPOPCNT kernels only require baseline AVX-512 plus `AVX512_VPOPCNTDQ`. Tying them to the SPR level unnecessarily excluded CPUs that support VPOPCNTDQ but not AVX512-FP16, notably Zen 4. ## Changes made while importing Two changes were needed on top of the pull request. **Feature detection read the wrong CPUID leaf.** `AVX512_VPOPCNTDQ` and `AVX512_VNNI` were read from leaf 1 ECX. Both live in leaf 7 subleaf 0 ECX, at bits 14 and 11. Leaf 1 ECX holds unrelated bits at those positions, so the new level was never selected on any CPU. Measured on an AMD Genoa host before the fix: the level reported as unavailable and detection returned `AVX512`. After the fix the same host reports `AVX512_VPOPCNT`. **The build definitions outside CMake were not updated.** The pull request renames three files. The other build description still named the old paths and had no entry for the new level, so the library did not build. This adds the level constant, its compiler flags, its `COMPILE_SIMD_AVX512_VPOPCNT` define, and the renamed paths. **The new level was not registered in the compiled-level mask.** `compiled_simd_levels()` reports which levels a binary holds, and `FAISS_SIMD_LEVEL` walks down from the requested level until it finds one that is present. The new level had no entry, so the override always stepped past it to `AVX512` and printed that the level was not compiled in. This was reported by `test_dispatch_with_env_var` on the open-source dynamic-dispatch build. **The test helper inferred available levels from `/proc/cpuinfo`.** A CPU feature does not imply that a level is compiled into the build, so the helper listed a level the override then refused. It now asks `SIMDConfig::is_simd_level_available()`, which reports compiled and supported. That is a subset of compiled, so every level it returns is one the override accepts. The mask and helper names also follow the naming used in the parent diff, so `AVAILABLE_SIMD_LEVELS_BASE_WITH_VPOPCNT` and `with_simd_level_with_vpopcnt`. Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5531 Test Plan: ``` buck2 test fbcode//mode/opt -c faiss.dynamic_dispatch=true \ fbcode//faiss/tests:test_simd_levels fbcode//faiss/tests:test_rabitq_simd ``` Result: all pass. The build failures the imported version reported are cleared. All three configurations build: ``` buck2 build fbcode//mode/dev fbcode//faiss:faiss fbcode//faiss:faiss_no_multithreading fbcode//faiss:faiss_omp_mock buck2 build fbcode//mode/opt -c faiss.dynamic_dispatch=true <same three targets> buck2 build fbcode//mode/dev -c fbcode.arch=aarch64 <same three targets> ``` Detection on an AMD Genoa host, which has AVX512_VPOPCNTDQ and AVX512_BF16 but not AVX512_FP16: | | before the CPUID fix | after | | --- | --- | --- | | detected level | `AVX512` | `AVX512_VPOPCNT` | | `AVX512_VPOPCNT` available | 0 | 1 | | `AVX512_SPR` available | 0 | 0 | Reviewed By: alibeklfc Differential Revision: D118969754 Pulled By: mnorris11 fbshipit-source-id: d1ca52b614829a3d4b322c9fecf0c004f2174dab |
||
|
marcelo-cjl
|
256980cb6e |
Add SVE low-dimensional nearest fast path and spherical SuperKMeans support (#5530)
Summary:
This PR adds two related changes for the SCANN coarse quantizer / PQ encoding path on ARM SVE: a low-dimensional nearest-neighbor fast path and spherical SuperKMeans support.
### 1. Low-dimensional L2sqr nearest fast path (d in {2,4,8})
`fvec_L2sqr_ny_nearest<ARM_SVE>` previously wrote all `ny` distances to the scratch buffer and then performed a separate linear scan. For PQ encoding (`compute_1_code`) each call has `d = dsub` and `ny = ksub`, making this the hot path. The new path maps one centroid to each SVE lane, gathers its `D` components, and keeps lane-local minimum distances and ids across batches. A predicated tail handles non-multiple `ny` values, equal distances preserve the first index, and only one final horizontal reduction is needed. The scratch buffer is not written, matching the x86 AVX2/AVX512 `D2/D4/D8` implementations.
`ProductQuantizer::compute_code` already uses the A1 dispatch on current `main`, so the ARM_SVE specialization remains reachable from PQ encoding after this rebase.
### 2. SuperKMeans spherical (inner-product) support
SuperKMeans previously only supported L2 clustering; the header explicitly excluded IP/cosine. With `cp.spherical=true`:
- `TrainState::R` becomes `std::unique_ptr<VectorTransform>`; power-of-two `d` uses the fast `HadamardRotation`, L2 keeps `RandomRotationMatrix`.
- Forgy initialization produces unit centroids before the first assignment. Each iteration computes and splits centroids first, then applies spherical centroid post-processing once, matching the `Clustering` flow.
- `HadamardRotation::reverse_transform` fills the missing inverse.
- `ClusteringParameters::use_super_kmeans` (default false) lets `Level1Quantizer::train_q1` route coarse quantizer training through SuperKMeans when explicitly enabled; the Python type stubs expose the new option.
- `block_l2<ARM_SVE>` completes the SuperKMeans SIMD kernels for the ADSampling pruning loop.
## Tests
- Low-dim nearest across SIMD levels: `d` in `{2,4,8}` and `ny` around common SIMD widths through 257, including non-multiple tails and first-index tie behavior. The dispatch test source is included in DD builds.
- Spherical objective close to vanilla spherical Clustering; unit-norm centroids.
- `use_super_kmeans` field inherited from `ClusteringParameters`.
## Benchmark
qwen 4096-dim, IP, nlist=1024, sub_dim=4, 1 thread, 50k rows:
| stage | baseline | optimized | change |
| --- | ---: | ---: | --- |
| Train | 67.73s | 24.36s | -64% |
| Add | 10.86s | 7.17s | -34% |
| Build | 78.59s | 31.52s | -60% |
Train speedup comes from SuperKMeans spherical coarse quantizer training; Add speedup comes from the low-dim SVE nearest fast path in PQ encoding. recall@10 unchanged vs Clustering baseline (diff <= 0.0002).
## Notes
This change targets ARM SVE. The x86 (AVX2/AVX512) implementations already have equivalent low-dim nearest specializations. The generic ARM SVE distance path comes from current `main`; this PR adds the low-dimensional nearest specializations on top of it.
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5530
Reviewed By: mnorris11
Differential Revision: D117563661
Pulled By: alibeklfc
fbshipit-source-id: 47ab1064d781a45d009923b4d5e62525718e83bf
|
||
|
Gergely Szilvasy
|
c7f07bbc0a |
Add C++ per-SIMD-level tests for distance utility functions (#5154)
Summary: Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5154 The templated distance functions in faiss/utils/distances.h -- fvec_L1, fvec_Linf, fvec_L2sqr_batch_4, fvec_inner_product_batch_4, fvec_L2sqr_ny_transposed, fvec_L2sqr_ny_nearest -- are exercised transitively by IndexFlat search at the binary's compiled/dispatched SIMD level. Direct per-level tests catch tail-handling bugs that the typical aligned-d transitive coverage masks. Adds tests/test_distances_dispatch.cpp with one TEST per function, sweeping every SIMDConfig-available level via SIMDConfig::set_level() and asserting equivalence against the SIMDLevel::NONE reference. Dimensions sweep {1, 3, 7, 8, 9, 15, 16, 17, 31, 32, 33, 64} so non- multiple-of-8 dims exercise the tail paths. Tolerances are calibrated to function semantics: * fvec_L2sqr_ny_nearest returns size_t; compared bit-exactly via EXPECT_EQ on the index. The internal distances_tmp_buffer is not part of the public contract (some SIMD impls don't write into it), so it isn't compared. * fvec_Linf is a max, not a sum, so reordering doesn't change the result; compared bit-exactly with EXPECT_NEAR(0.0). * The L1 / batch_4 / L2sqr_ny_transposed variants accumulate over d terms; SIMD reductions can differ by a few ULPs from scalar. Tolerance scales linearly with d (1e-6 per term, floor 1e-6). For fvec_L2sqr_ny_transposed the test uses the contract layout y[i + k * d_offset] (per distances_simd.cpp:62) and picks d_offset = ny + 5 so the stride is non-trivial AND every (i, k) access in [0, ny) x [0, d) is in-bounds. NaN/Inf inputs are out of scope: rand_vec produces uniform [-1, 1]. ___ overriding_review_checks_triggers_an_audit_and_retroactive_review Oncall Short Name: fair_umami_cluster Differential Revision: D102821270 fbshipit-source-id: d25a9da80f5c623487bdf4868a107d2b59dfe833 |