Files
faiss/tutorial/cpp/10-SVS-Vamana-LVQ.cpp
Aaron Lin a695814f4c Enable Intel ScalableVectorSearch support (#4548)
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/issues/4450

This pull request introduces support for [Intel ScalableVectorSearch](https://github.com/intel/ScalableVectorSearch), integrating Intel's proprietary LVQ and LeanVec technologies in binary form. The following index types are now supported: `IndexSVSVamana`, `IndexSVSVamanaLVQ`, `IndexSVSVamanaLeanVec`, and `IndexSVSFlat`. `IndexSVSVamana` and `IndexSVSFlat` utilizes SVS open-source float32/float16/int8 implementation.

Key features and enhancements include:
- Implemented `search`, `add`, and `remove_ids`
- Implemented filter search and `range_search`
- Enabled SVS open-source fp16 and scalar quantization implementation
- Enabled factory methods
- Enabled save/load
- Enabled Python bindings
- Enable fallback mechanism that fallbacks to 8-bit scalar quantization if LVQ/LeanVec is used on non-intel hardware CPUs
- Added examples in both Python and C++ under the `tutorial/` directory
- Added `FAISS_ENABLE_SVS` flag to allows users to optionally enable SVS

TODOs:
- Support more indices such as `IndexSVSIVF` and `IndexSVSFlat` with LVQ and LeanVec
- Documentation

Pull Request resolved: https://github.com/facebookresearch/faiss/pull/4548

Reviewed By: subhadeepkaran

Differential Revision: D84713772

Pulled By: mnorris11

fbshipit-source-id: 290365eb3337f685dd46696935b621c57dad3174
2025-12-05 16:03:58 -08:00

90 lines
2.2 KiB
C++

/*
* Portions Copyright (c) Meta Platforms, Inc. and affiliates.
*
* This source code is licensed under the MIT license found in the
* LICENSE file in the root directory of this source tree.
*/
/*
* Portions Copyright 2025 Intel Corporation
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
#include <cassert>
#include <cstdio>
#include <cstdlib>
#include <random>
#include <faiss/svs/IndexSVSVamanaLVQ.h>
using idx_t = faiss::idx_t;
int main() {
int d = 64; // dimension
int nb = 100000; // database size
int nq = 10000; // nb of queries
std::mt19937 rng;
std::uniform_real_distribution<> distrib;
float* xb = new float[d * nb];
float* xq = new float[d * nq];
for (int i = 0; i < nb; i++) {
for (int j = 0; j < d; j++)
xb[d * i + j] = distrib(rng);
xb[d * i] += i / 1000.;
}
for (int i = 0; i < nq; i++) {
for (int j = 0; j < d; j++)
xq[d * i + j] = distrib(rng);
xq[d * i] += i / 1000.;
}
int k = 4;
faiss::IndexSVSVamanaLVQ index(d, 64);
index.add(nb, xb);
{ // search xq
idx_t* I = new idx_t[k * nq];
float* D = new float[k * nq];
index.search(nq, xq, k, D, I);
printf("I=\n");
for (int i = nq - 5; i < nq; i++) {
for (int j = 0; j < k; j++)
printf("%5zd ", I[i * k + j]);
printf("\n");
}
printf("D=\n");
for (int i = nq - 5; i < nq; i++) {
for (int j = 0; j < k; j++)
printf("%5f ", D[i * k + j]);
printf("\n");
}
delete[] I;
delete[] D;
}
delete[] xb;
delete[] xq;
return 0;
}