Summary:
Allow avoiding storing original vectors, which may not necessarily be used, in which case they would be wasting memory
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5525
Reviewed By: alibeklfc
Differential Revision: D118722620
Pulled By: mnorris11
fbshipit-source-id: 9757fa3d272187308c750fb6c6d0e3a6e8ab2bb3
Summary:
This pull request adds support for memory-mapped I/O (mmap) loading of static SVS (Scalable Vector Search) indices in Faiss, enabling faster and more memory-efficient index loading. The main changes include implementing mmap-based deserialization for static Vamana and Flat indices, updating the index loading logic to use mmap when requested, and adding comprehensive tests for the new functionality.
**Memory-mapped index loading support:**
* Added `map_to` methods to `IndexSVSVamana` and `IndexSVSFlat` for loading static indices directly from memory-mapped files, including logic to manage memory mapping ownership and adjust file positions after mapping.
**Index and file structure updates:**
* Extended `IndexSVSVamana` and `IndexSVSFlat` classes to hold a reference to the mmap owner, ensuring the mapped memory remains valid for the lifetime of the index. Also added forward declarations and includes for mmap-related types.
* Modified `reset` methods to release memory mappings when the index is reset, preventing resource leaks.
**Testing and interface improvements:**
* Enhanced test utilities and added new tests to verify correct mmap loading for various static Vamana index types, including FP32, FP16, SQI8, LVQ4x4, and LeanVec4x4 in `tests/test_svs.cpp`
These changes collectively provide robust and efficient support for memory-mapped loading of static SVS indices, improving both performance and usability for large-scale vector search applications.
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5271
Reviewed By: alibeklfc
Differential Revision: D108313487
Pulled By: mnorris11
fbshipit-source-id: 4ae6dc3e69363b0c792cd56ad26b01bf6a853d28
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5349
# this is a no-op change
Ran `black -l 80` over all tracked Python files under `fbcode/faiss/`, using line length 80 to match faiss's documented coding style (`CONTRIBUTING.md`: "80 character line length (both for C++ and Python)"). 172 files were reformatted; 28 were already compliant.
This is a pure-formatting change. `black` runs its AST-equivalence safety check on every file, so no identifiers, string/numeric values, operators, or control flow were altered — only whitespace, indentation, line wrapping, blank lines, trailing commas, and string-quote normalization (single to double quotes).
`fbcode/faiss/` is intentionally excluded from Meta's standard Python autoformatter (`arc f` / `arc lint -a` report no formatter for these paths) because it mirrors to public GitHub, so the Python had drifted out of any consistent style. The C++ is already clang-format-clean (0 of 807 files change), so this effort is Python-only. This diff brings the Python to canonical `black` formatting.
Excluded `demos/index_pq_flat_separate_codes_from_codebook.py`: it contains a Jupyter `!rm` shell-magic line and is not valid Python, so `black` cannot parse it.
This is the base of a 3-diff stack: (1) this black reformat; (2) `.flake8` alignment (max-line-length 80 + black-compatible ignores E203/W503); (3) genuine lint fixes (F401/F403/F841/B011).
Reviewed By: mnorris11
Differential Revision: D109736099
fbshipit-source-id: 9a054635766ed11aa3760a788395babd3d198abf
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5337
We will use SQ8 more frequently in Vamana in particular, so this will reduce friction
Reviewed By: junjieqi
Differential Revision: D109480499
fbshipit-source-id: 8d8a3982f459eb75b1d63e7032b8abacf7f3a41a
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5201
Currently SVSVamana failed with error:
Mitigation: Error in std::unique_ptr<Index> faiss::(anonymous namespace)::index_factory_sub(int, std::string, MetricType, bool) at fbcode/faiss/index_factory.cpp:1176: could not parse index string IVF524288_SVSVamana64_SQI8,SQ4
In this diff we will fix the issue
f1078497787
Reviewed By: mnorris11
Differential Revision: D104326684
fbshipit-source-id: 974723c880d235cfee4ec36b3f4c91d251f408b4
Summary:
This PR introduces SVS IVF (Inverted File) index support in FAISS, complementing the existing SVS Vamana graph‑based index. The implementation integrates Intel's Scalable Vector Search (SVS) runtime to provide a high‑performance, partition‑based ANN index with strong scalability on modern hardware.
## Key Advantages of SVS IVF
- **AMX‑accelerated builds** for significantly faster index training on supported Intel platforms
- **Hierarchical clustering** (default ON) enabling much faster k‑means on large datasets
- **LVQ and LeanVec compression** applied at search time to boost throughput while preserving recall
- **Intra-query parallelism** to reduce latency for small‑batch or single‑query workloads
- **Static index mode** for read‑only workloads with optimized memory layout and search performance
## New Factory Strings
| Factory String | IndexClass | Storage |
|---|---|---|
| `SVSIVF{nlist}` | IndexSVSIVF | FP32 (default) |
| `SVSIVF{nlist},FP16` | IndexSVSIVF | Float16 |
| `SVSIVF{nlist},SQI8` | IndexSVSIVF | Scalar quantized int8 |
| `SVSIVF{nlist},LVQ4x0` | IndexSVSIVFLVQ | LVQ primary=4, no residual |
| `SVSIVF{nlist},LVQ4x4` | IndexSVSIVFLVQ | LVQ primary=4, residual=4 |
| `SVSIVF{nlist},LVQ4x8` | IndexSVSIVFLVQ | LVQ primary=4, residual=8 |
| `SVSIVF{nlist},LVQ8x0` | IndexSVSIVFLVQ | LVQ primary=8, no residual |
| `SVSIVF{nlist},LeanVec4x4_{dims}` | IndexSVSIVFLeanVec | LeanVec 4x4 |
| `SVSIVF{nlist},LeanVec4x8_{dims}` | IndexSVSIVFLeanVec | LeanVec 4x8 |
| `SVSIVF{nlist},LeanVec8x8_{dims}` | IndexSVSIVFLeanVec | LeanVec 8x8 |
## Static vs Dynamic IVF Index
SVS IVF supports two modes controlled by the `is_static` constructor parameter:
### Dynamic IVF (default, `is_static = false`)
- Supports `train()`, `add()`, `remove_ids()`, and `search()`
- Vectors can be added or removed after index creation
- Suitable for workloads where the dataset changes over time
### Static IVF (`is_static = true`)
- **All data must be provided during `train()`** — the index is built and populated in a single step
- `add()` and `remove_ids()` will throw an exception after creation
- Optimized for **read‑only workloads** where the dataset is fixed at build time
- Uses `svs_runtime::IVFIndex` (static) instead of `svs_runtime::DynamicIVFIndex`
- Serialization and deserialization are fully supported
**Usage Examples(C++):**
```cpp
// Dynamic IVF with intra-query parallelism
faiss::IndexSVSIVF index(d, nlist);
// intra_query_threads must be set BEFORE train() — changes after
// train() are not propagated to the SVS backend (current SVS runtime
// API limitation, will be fixed in a future SVS version).
index.intra_query_threads = 4;
index.n_probes = 20;
index.train(n, train_data);
index.add(n, base_data);
index.search(nq, queries, k, distances, labels);
// num_threads CAN be changed between searches (synced at search time).
index.num_threads = 8;
index.search(nq, queries, k, distances, labels);
// Static IVF (read-only, all data at build time)
faiss::IndexSVSIVF index(d, nlist, faiss::METRIC_L2,
faiss::SVSStorageKind::SVS_FP32, /*is_static=*/true);
index.intra_query_threads = 4; // set before train()
index.train(n, data); // builds and populates the index
index.search(nq, queries, k, distances, labels);
// index.add(...) // throws — not supported on static index
// index.remove_ids(...) // throws — not supported on static index
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/4801
Reviewed By: limqiying
Differential Revision: D102536698
Pulled By: mnorris11
fbshipit-source-id: cbc2b882c1f7728ed54a0bd25ceacb225d5a6c74
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5162
T225351715 Fixing failed unit test as part of oncall duty. test_svs_py are failing due to timeout.
In this diff, we are fixing 2 issues with the tests:
1. setUp runs before every test method — with 11 tests per class × 8 classes = ~57 times, each regenerating the same numpy arrays. setUpClass runs once per class, so data is generated ~8 times instead of ~57. The data is deterministic (fixed np.random.seed(1234)) and no test mutates xb/xq — tests only read them. So regenerating identical arrays per-method is pure waste.
2. Reduce test size
Reviewed By: mnorris11
Differential Revision: D103043664
fbshipit-source-id: ac79b23910c2d936168dc7021b06f57c47a32782
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/4774
Fix FLAKE8 and CLANGTIDY lint errors in the SVS integration test file and index_factory to resolve CI failures from D88785138.
Reviewed By: mnorris11
Differential Revision: D91659701
fbshipit-source-id: 7bc3d47fdb76a620056295fcde9ac1b09769e2bb
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/4737
Add SVS as dependency of Faiss in BUCK files so we can use SVS internally. Also makes minor changes to index_read and index_factory to fix compiler warnings generated internally.
Reviewed By: mdouze
Differential Revision: D88785138
fbshipit-source-id: 975ade365cd6775582526aa2c852644ef4676610
Mihai Capotă