Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5718
`test_reader` searches one IVFPQ index loaded twice, with and without the precomputed table, and required the neighbor ids to be identical. The two paths add the distance terms in a different order, so near-tied neighbors can swap. With unseeded data it failed in 7 of 400 random draws, and once in seven runs of the Cogwheel faiss suite on T1_BGM (2 of 10000 ids).
Compare with `check_ref_knn_with_draws`, which allows swaps only among tied distances, and seed the data so a failure reproduces. `test_io` moves to `py_tests_with_contrib` for the helper.
Reviewed By: mnorris11
Differential Revision: D124383290
fbshipit-source-id: ceb81262ffaba81ac23e5a592e1658741192060d
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5707
`IndexFlatCodes::reset()` called `codes.clear()`. When the index was read with `IO_FLAG_MMAP_IFC` (or through a zero-copy reader), `codes` is a non-owning view, and `MaybeOwnedVector::clear()` asserts `is_owned`, so `reset()` aborted the process. This hits `IndexFlat`, `IndexScalarQuantizer` and every index that resets a flat-codes storage, e.g. `IndexHNSWFlat` and `IndexNSGFlat`.
`reset()` now keeps `clear()` for owned codes (capacity is kept, as before) and replaces a view with an empty owned vector. The index can then be refilled with `add()`.
___
Differential Revision: D123073895
fbshipit-source-id: 892e07eaa81d0176e34769714292c036c772a4d7
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5349
# this is a no-op change
Ran `black -l 80` over all tracked Python files under `fbcode/faiss/`, using line length 80 to match faiss's documented coding style (`CONTRIBUTING.md`: "80 character line length (both for C++ and Python)"). 172 files were reformatted; 28 were already compliant.
This is a pure-formatting change. `black` runs its AST-equivalence safety check on every file, so no identifiers, string/numeric values, operators, or control flow were altered — only whitespace, indentation, line wrapping, blank lines, trailing commas, and string-quote normalization (single to double quotes).
`fbcode/faiss/` is intentionally excluded from Meta's standard Python autoformatter (`arc f` / `arc lint -a` report no formatter for these paths) because it mirrors to public GitHub, so the Python had drifted out of any consistent style. The C++ is already clang-format-clean (0 of 807 files change), so this effort is Python-only. This diff brings the Python to canonical `black` formatting.
Excluded `demos/index_pq_flat_separate_codes_from_codebook.py`: it contains a Jupyter `!rm` shell-magic line and is not valid Python, so `black` cannot parse it.
This is the base of a 3-diff stack: (1) this black reformat; (2) `.flake8` alignment (max-line-length 80 + black-compatible ignores E203/W503); (3) genuine lint fixes (F401/F403/F841/B011).
Reviewed By: mnorris11
Differential Revision: D109736099
fbshipit-source-id: 9a054635766ed11aa3760a788395babd3d198abf
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5305
The serialization for storage is already separate, it might as well be unbounded.
Reviewed By: mnorris11
Differential Revision: D108328669
fbshipit-source-id: 27951676b56a2222786798580b845551cf1722f6
Summary:
On a 148 GB `IndexIVFRaBitQ`, `faiss.read_index(path, IO_FLAG_MMAP_IFC)` currently spikes to **165 GB of committed private heap** for ~50 seconds during load, before dropping back to the intended ~9 GB steady state. The spike is transient but matches the entire invlist payload, which defeats the purpose of `IO_FLAG_MMAP_IFC` (keep invlist codes and ids file-cache-backed, not pagefile-committed).
Root cause is a two-loop pattern in `read_InvertedLists_up` (`faiss/impl/index_read.cpp`) for both the `ilar` (`ArrayInvertedLists`) and `ilpn` (`ArrayInvertedListsPanorama`) branches:
```cpp
// Loop A: pre-allocate every list's owning std::vector (= full invlist size on heap)
for (i = 0 .. nlist) {
ails->codes[i].resize(sizes[i] * code_size);
ails->ids[i].resize(sizes[i]);
}
// Loop B: replace each owning vector with a view into mmap
for (i = 0 .. nlist) {
read_vector_with_known_size(ails->codes[i], ...);
read_vector_with_known_size(ails->ids[i], ...);
}
```
`read_vector_with_known_size` detects a `MappedFileIOReader` and replaces the target `MaybeOwnedVector` via `create_view`, correctly releasing the owning storage. But Loop A has already committed 100% of the invlist size to the heap before Loop B can release any of it.
## Fix
Merge the two loops into a single pass per list, so each list's owning heap allocation is released via the view-substitution before the next list's is made. Peak heap during load is now bounded by a single list's worth (~one nprobe cluster, typically under 1 MB), not the sum of all lists'.
Applied to both the `ilar` and `ilpn` branches. End state is byte-identical: with `MappedFileIOReader` every `MaybeOwnedVector` ends up as a view; with a regular `FileIOReader` every `MaybeOwnedVector` ends up as owning storage.
## Measured impact
Same IndexIVFRaBitQ file (IVF262144_HNSW32,RaBitQ8, 99.7 M vectors, nlist=262144, code_size=1556, 148 GB on disk), loaded via `faiss.read_index(path, IO_FLAG_MMAP_IFC)`:
| Metric | Before | After |
|---------------------|--------:|------:|
| Peak private memory | 164 GB | 10 GB |
| Load time | 99 s | 16 s |
| Steady-state RSS | 9 GB | 9 GB |
Load time drops because ~90 s of the original was spent in `std::vector::resize` zero-filling pages that were about to be discarded.
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5122
Test Plan:
- [x] Manual: `faiss.read_index(path, IO_FLAG_MMAP_IFC)` on the 148 GB production index; `search(q, 20)` at `nprobe=64` returns the same top-k ids before and after the patch.
- [x] Manual: 1 Hz private-memory polling during a 10-query, `k=500`, `nprobe=256` stress — RSS grows identically to the pre-patch behaviour (file-cache page-fault growth, not private heap), confirming the view substitution still works.
- [ ] Existing FAISS CI — no new tests added. The bug is a transient peak that existing tests cannot observe, and the fix is an ordering change with no new code path.
## Safety
- No API, ABI, serialization-format, or behavioural change.
- `FAISS_CHECK_DESERIALIZATION_LOOP_LIMIT` hardening still runs in the same place, before the merged loop.
- Works identically on `FileIOReader` (heap-owning end state) and `MappedFileIOReader` (view end state).
Made with [Cursor](https://cursor.com)
Reviewed By: mdouze
Differential Revision: D103055438
Pulled By: mnorris11
fbshipit-source-id: 5cdc371aa78547f00a4e8e854cac84945d81b6b5
Summary:
Extend the POSIX mmap reader to Apple platforms and use MAP_FAILED for mmap error checks while keeping madvise best-effort.
Update the C++ and Python mmap tests to exercise Darwin, and stop linking faiss_test against the Python example extension so Python-enabled test builds can run on macOS.
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5058
Reviewed By: junjieqi
Differential Revision: D100351635
Pulled By: alibeklfc
fbshipit-source-id: f384251a634c7d3154103dbc763293b52b093ee8
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/4856
Using a Hadamard transformation is very similar to using a random rotation matrix, but we can calculate it in `d log d` time instead of `d ^ 2`. I initially started exploring this using a fast fourier transform which does outperform a random rotation, but a Hadamard transformation has the same computation complexity as FFT and in practice it is much easier to implement. Also because it is an orthonormal transformation, the recall is identical to a random rotation matrix.
I added a benchmark to compare both.
Reviewed By: mdouze
Differential Revision: D94658424
fbshipit-source-id: bb2f9dc0ab5a41ce58874a1e72a3e85c22e58ea2
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/4851
Roundtrip tests only test serialize -> deserialize. They should also test the deserialize -> serialize sequence.
Reviewed By: mdouze
Differential Revision: D93678044
fbshipit-source-id: 1503b5b954d49df509d9f785d35911ade989290e
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/4809
Convert all allocations in faiss/impl/index_read.cpp to std::unique_ptr<>
so that exceptions thrown during deserialization do not leak memory.
The conversion is mostly mechanical in nature except that it includes
std::unique_ptr<> versions of the public API to faciliate future adoption
of this approach to exception safety in calling contexts.
NOTE: This covers the most obvious exception related memory leaks, but
additional auditing and testing may find others. These will be addressed in
a separate commit.
Reviewed By: junjieqi
Differential Revision: D93139218
fbshipit-source-id: 48ac606abb6c8e6eb5bb06d04988cb06cdb34120
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/4700
A pass over the serialization / deserialization tests, so that they don't use temp files when not needed.
Reviewed By: pankajsingh88, junjieqi
Differential Revision: D87615610
fbshipit-source-id: 3e6366b938d41e083781745179ff19ecf5247ba2
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/4541
Per PEP 8 style guide (https://pep8.org/#blank-lines), top-level class
definitions should be surrounded by two blank lines. This commit addresses
\E302 lint violations across the faiss codebase by ensuring all class
definitions have exactly 2 blank lines before them.
Changes:
- Fixed \E302 errors in 28 Python files across tests, benchmarks, contrib,
and GPU modules
- Applied consistent PEP 8 formatting for class definitions
- No functional changes - formatting only
Files affected include test files, benchmark scripts, contrib modules,
and GPU test utilities. All changes maintain existing functionality
while improving code style compliance.
Reviewed By: junjieqi
Differential Revision: D80682723
fbshipit-source-id: 6ad5458c3bf1e5c06f35dc8fcdbea2ac4ba83c5f
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/4250
This is an attempt to re-land the diff stack D69972250 D70982449
It was reverted because the bottom of the stack did not pass the tests.
The original code comes from Alexandr Guzhva's https://github.com/facebookresearch/faiss/pull/4199
To the adsmarket steward: the diff was already accepted by your team (see D70982449), but reverted for an independent reason. So should be easy to accept now.
Reviewed By: mengdilin
Differential Revision: D71614511
fbshipit-source-id: 94139b4a4d457afe0d37ac95342537414aa81e7a
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3455
Code quality control by reducing the number of prints
Reviewed By: junjieqi
Differential Revision: D57502194
fbshipit-source-id: a6cd65ed4cc49590ce73d2978d41b640b5259c17
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/2582
A few more or less cosmetic improvements
* Index::idx_t was in the Index object, which does not make much sense, this diff moves it to faiss::idx_t
* replace multiprocessing.dummy with multiprocessing.pool
* add Alexandr as a core contributor of Faiss in the README ;-)
```
for i in $( find . -name \*.cu -o -name \*.cuh -o -name \*.h -o -name \*.cpp ) ; do
sed -i s/Index::idx_t/idx_t/ $i
done
```
For the fbcode deps:
```
for i in $( fbgs Index::idx_t --exclude fbcode/faiss -l ) ; do
sed -i s/Index::idx_t/idx_t/ $i
done
```
Reviewed By: algoriddle
Differential Revision: D41437507
fbshipit-source-id: 8300f2a3ae97cace6172f3f14a9be3a83999fb89
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/2489
Currently, Faiss does not save the precomputed_table for IVFPQ when saving an index, and reconstructs one on the fly upon loading. The introduced flag allows to skip the reconstruction during faiss.read_index()
Reviewed By: mdouze
Differential Revision: D39747978
fbshipit-source-id: 47d3cfc59e791e2e0b986301764ccc5b220292f4
Summary:
support merge for all IndexFlatCodes children
make merge_from and check_compatible_for_merge methods of Index and IndexIVF and IndexFlatCodes(the only supported types) inherit them from Index.
This is part 1 of 2 as merge_into still not updated
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/2488
Test Plan:
cd build
make -j
make test
cd faiss/python && python setup.py build
cd ../../..
PYTHONPATH="$(ls -d ./build/faiss/python/build/lib*/)" pytest tests/test_*.py
# fbcode
buck test //faiss/tests/:test_index_merge
buck test //faiss/tests/:test_io
Reviewed By: mdouze
Differential Revision: D39726378
Pulled By: AbdelrahmanElmeniawy
fbshipit-source-id: 6739477fddcad3c7a990f3aae9be07c1b2b74fef
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/2132
This diff adds the class IndexFlatCodes that becomes the parent of all "flat" encodings.
IndexPQ
IndexFlat
IndexAdditiveQuantizer
IndexScalarQuantizer
IndexLSH
Index2Layer
The other changes are:
- for IndexFlat, there is no vector<float> with the data anymore. It is replaced with a `get_xb()` function. This broke quite a few external codes, that this diff also attempts to fix.
- I/O functions needed to be adapted. This is done without changing the I/O format for any index.
- added a small contrib function to get the data from the IndexFlat
- the functionality has been made uniform, for example remove_ids and add are now in the parent class.
Eventually, we may support generic storage for flat indexes, similar to `InvertedLists`, eg to memmap the data, but this will again require a big change.
Reviewed By: wickedfoo
Differential Revision: D32646769
fbshipit-source-id: 04a1659173fd51b130ae45d345176b72183cae40
Summary: add getstate / setstate to serialize indexes. Seems to work properly with object ownership etc.
Reviewed By: wickedfoo
Differential Revision: D26521228
fbshipit-source-id: ebbe08cfe2c15af2aa5b7ea1fc1bf87546066c23
Summary:
Added a few functions in contrib to:
- run range searches by batches on the query or the database side
- emulate range search on GPU: search on GPU with k=1024, if the farthest neighbor is still within range, re-perform search on CPU
- as reference implementations for precision-recall on range search datasets
- optimized code to plot precision-recall plots (ie. sweep over thresholds)
The new functions are mainly in a new `evaluation.py`
Reviewed By: wickedfoo
Differential Revision: D25627619
fbshipit-source-id: 58f90654c32c925557d7bbf8083efbb710712e03
Weixiang Hong