Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5444
We fix every remaining flake8 E501 (line > 80 chars) violation in
fbcode/faiss/ (excluding gpu/), closing out a lint-cleanup initiative that a
June 29, 2026 diff stack (9a05463576 -> d304e29b9a -> 7d140d86ca) explicitly
promised but left 41% incomplete. The middle diff in that stack aligned
.flake8's max-line-length to black's 80-char width and stated the remaining
277 violations after the config change were "genuine lint (E501 long
strings/URLs, F401/F841/B011, etc.), addressed in later commits in this
stack." The follow-up commit fixed only the F401/F403/F841/B011 categories
(45 issues) and never touched E501, leaving 192 violations across 53 files
live in trunk. A direct re-scan today found 201 violations across 54 files
(two files gained lines since the original count).
We fixed 183 violations across 52 files. Two of the originally-flagged
files, benchs/bench_rabitq.py and
demos/index_pq_flat_separate_codes_from_codebook.py, already carry a
file-level `# flake8: noqa` directive (the second is a Bento/Grimaldi
notebook export that also fails standalone py_compile due to embedded shell
magic) -- flake8 ignores both files entirely, so no changes were needed
there and they are excluded from this diff.
Fix approach, by category:
- The large majority of violations are prose: docstrings and comments
rewrapped at word boundaries, preserving each file's existing indentation
and docstring conventions. No wording was changed.
- Code lines (function calls, f-strings, assert messages, long argument
lists) were reflowed in black-compatible style: parenthesization, comma
breaks, or implicit adjacent string-literal concatenation. Every touched
file is confirmed unchanged by `black --line-length 80 --diff` after the
edit.
- A small number of genuinely unwrappable long URLs (no reasonable break
point) were suppressed with `# noqa: E501` rather than force-wrapped,
matching the precedent set by the original black-format diff stack.
- demos/offline_ivf/offline_ivf.py: a repeated, non-trivial ratio expression
used in both an assert condition and its (lazily-evaluated) failure
message was hoisted into a single local variable, eliminating the
duplicate computation. The expression is pure and deterministic (built
from already-computed arrays), so this is behaviorally identical to the
original -- not a functional change.
- `arc lint -a` additionally auto-fixed a real bug it detected while
processing these files: `raise NotImplemented` -> `raise
NotImplementedError` in contrib/evaluation.py (the sentinel object
NotImplemented is not an exception type; raising it produces a TypeError
masking the intended NotImplementedError). It also added a missing
trailing newline in benchs/fb/bench_index_binary_from_float.py and, in
the same file, rewrote the `#!/usr/bin/env python2` shebang to
`#!/usr/bin/env fbpython` -- that rewrite was reverted by hand, since the
file still does `import cPickle` (a Python 2-only module), so the
auto-fix would have made the shebang describe an interpreter the script
cannot actually run under. Fixing the underlying py2-to-py3 migration is
out of scope for this line-length-only sweep.
- Rebasing onto master (this diff's base commit had drifted) surfaced two
concurrent upstream fixes to `check_ref_knn_with_draws(...)` calls in
tests/test_ivf_flat_panorama.py and tests/test_refine_panorama.py (an
unrelated in-flight correctness fix passing `D_panorama`/`atol` instead
of a stale `D_regular` arg with no `atol`). The rebase took upstream's
corrected arguments and re-applied only this diff's line-wrap on top.
We verified zero functional change with an AST-level check: every string
and f-string literal in every touched file was extracted before and after
the edit and compared with whitespace normalized, confirming no wording,
value, or expression was altered beyond the intended `# noqa: E501`
suffixes and the one documented ratio-hoisting simplification above.
Reviewed By: mnorris11
Differential Revision: D112525458
fbshipit-source-id: e4b85d73dafcaac779f3522a383a66e490fa942a
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5351
Fixes 45 genuine (non-style) flake8 issues across 20 faiss Python files, on top of the black-80 base in this stack. These are real code issues, not formatting.
- `B011` (19): replaced `assert False` / `assert False, "msg"` with `raise AssertionError(...)`. `assert` statements are stripped under `python -O`, so an `assert False` guard silently vanishes in optimized builds; `raise AssertionError` always fires.
- `F401` (10) / `F403` (7): the intentional star-import re-exports in `python/loader.py` and `python/loader_fb.py` (which expose the SWIG symbols as `faiss.X`) are annotated with `# noqa: F401,F403`; genuinely unused imports in test/bench scripts are removed.
- `F841` (9): unused local variables removed, or replaced with `_` in tuple unpacking, preserving behavior.
No runtime behavior changes; formatting stays at black width 80.
Reviewed By: mnorris11
Differential Revision: D109739445
fbshipit-source-id: 7d140d86cab9b2df691003bd043fc880a31410cd
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5349
# this is a no-op change
Ran `black -l 80` over all tracked Python files under `fbcode/faiss/`, using line length 80 to match faiss's documented coding style (`CONTRIBUTING.md`: "80 character line length (both for C++ and Python)"). 172 files were reformatted; 28 were already compliant.
This is a pure-formatting change. `black` runs its AST-equivalence safety check on every file, so no identifiers, string/numeric values, operators, or control flow were altered — only whitespace, indentation, line wrapping, blank lines, trailing commas, and string-quote normalization (single to double quotes).
`fbcode/faiss/` is intentionally excluded from Meta's standard Python autoformatter (`arc f` / `arc lint -a` report no formatter for these paths) because it mirrors to public GitHub, so the Python had drifted out of any consistent style. The C++ is already clang-format-clean (0 of 807 files change), so this effort is Python-only. This diff brings the Python to canonical `black` formatting.
Excluded `demos/index_pq_flat_separate_codes_from_codebook.py`: it contains a Jupyter `!rm` shell-magic line and is not valid Python, so `black` cannot parse it.
This is the base of a 3-diff stack: (1) this black reformat; (2) `.flake8` alignment (max-line-length 80 + black-compatible ignores E203/W503); (3) genuine lint fixes (F401/F403/F841/B011).
Reviewed By: mnorris11
Differential Revision: D109736099
fbshipit-source-id: 9a054635766ed11aa3760a788395babd3d198abf
Summary:
- open SIFT demo vector files with binary mode
## Why
The SIFT demo reads .fvecs/.ivecs data as fixed-width binary records. Opening the file with rb avoids platform text-mode translation, especially on Windows, and keeps the byte-size checks aligned with the actual file contents.
## Testing
- git diff --check
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5213
Reviewed By: junjieqi
Differential Revision: D106137008
Pulled By: alibeklfc
fbshipit-source-id: 4610e646fab4409c0e551162ac9f144382a02337
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/4765
There are use cases where there is a group id associated to each vector id.
We want no more than max_per_group results per group but of course we also want to keep the best results for each group.
This diff demonstrates how to obtain this functionality, using a custom result handler.
Differential Revision: D91020690
fbshipit-source-id: 92669cf8396ad8b11faa87c4420b35a3db2de7f9
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/4560
**What:** Fixed 73 Python \E302 linting errors by adding missing blank lines before function definitions.
**Why:** Python PEP 8 style guide requires 2 blank lines before top-level function and class definitions. The codebase had functions with only 1 blank line preceding them, causing linting failures.
**Changes:** Added one additional blank line before function definitions across 12 files in the faiss codebase to meet the "expected 2 blank lines, found 1" requirement.
**Impact:** Brings Python code into compliance with standard formatting rules without affecting functionality.
Reviewed By: trang-nm-nguyen
Differential Revision: D81080938
fbshipit-source-id: dd5e02c52229ca77db38b465bdaadd9bfa83801a
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/4250
This is an attempt to re-land the diff stack D69972250 D70982449
It was reverted because the bottom of the stack did not pass the tests.
The original code comes from Alexandr Guzhva's https://github.com/facebookresearch/faiss/pull/4199
To the adsmarket steward: the diff was already accepted by your team (see D70982449), but reverted for an independent reason. So should be easy to accept now.
Reviewed By: mengdilin
Differential Revision: D71614511
fbshipit-source-id: 94139b4a4d457afe0d37ac95342537414aa81e7a
Summary:
For the release, we want to standardize on Faiss instead of FAISS.
Changed everything except the CHANGELOG.md which I assume should not be changed after it lands.
This doesn't aim to fix any existing lints / errors. Those can be handled at another time.
Reviewed By: junjieqi
Differential Revision: D68842649
fbshipit-source-id: c0b60d5baa0e1f710db3638ffcc6f223fb3408ad
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/4029
Remove headers flagged by facebook-unused-include-check over fbcode.faiss.
+ format and autodeps
This is a codemod. It was automatically generated and will be landed once it is approved and tests are passing in sandcastle.
You have been added as a reviewer by Sentinel or Butterfly.
Autodiff project: uif
Autodiff partition: fbcode.faiss
Autodiff bookmark: ad.uif.fbcode.faiss
Reviewed By: dtolnay
Differential Revision: D65957849
fbshipit-source-id: f6199250db595defd56f5e7b2828f838702e9a16
Summary:
Demonstrate IndexLSH does not need training or codebook serialization
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/4009
Reviewed By: junjieqi
Differential Revision: D65274645
Pulled By: asadoughi
fbshipit-source-id: c9af463757edbd07cc07b1cf607b88373fa334c4
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3989
Moved add_sa_codes, sa_code_size to Index, IndexBinary base classes from IndexIVF to support adding coded vectors with ids using IDMap2,PQ
For an alternative approach, see previous attempt with merge_ids and merge_codes: D64941798
Reviewed By: mnorris11
Differential Revision: D64972587
fbshipit-source-id: 71622fc35a378d9892569a56442a872f0c9c9e83
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3987
Created a new notebook demonstrating how to separate serializing and deserializing the PQ codebook (via faiss.write_index for IndexPQ) independently of the vector codes. For example, in the case where you have a few vector embeddings per user and want to shard the flat index by user you can re-use the same PQ method for all users but store each user's codes independently.
Reviewed By: junjieqi
Differential Revision: D64844978
fbshipit-source-id: ad6434101fbb3ef84999527a577ecb9b503e556c
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3876
Demo script for distributed kmeans. It provides a `DatasetAssign` object and shows how to run it with torch.distributed.
Reviewed By: asadoughi, pankajsingh88
Differential Revision: D63013820
fbshipit-source-id: 22c959f3afdc04fd4aa8b9aeed309ea6290b1328
Summary:
`torch.load` without `weights_only` parameter is unsafe. Explicitly set `weights_only` to False only if you trust the data you load and full pickle functionality is needed, otherwise set `weights_only=True`.
If `weights_only=True` doesn't work for some cases, then explicit `weights_only=False` should be used.
Found with https://github.com/pytorch-labs/torchfix/
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3796
Reviewed By: asadoughi
Differential Revision: D61824340
Pulled By: kit1980
fbshipit-source-id: bc013d06d4f368f730ffee6898e75fd0b0ff1d40
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3608
This is a straightforward implementation of QINCo in CPU Faiss, with encoding and decoding capabilities (not training).
For this, we translate a simplified version of some torch classes:
- tensors, restricted to 2D and int32 + float32
- Linear and Embedding layer
Then the QINCoStep and QINCo can just be defined as C++ objects that are copy-constructable.
There is some plumbing required in the wrapping layers to support the integration. Pytroch tensors are converted to numpy for getting / setting them in C++.
Reviewed By: asadoughi
Differential Revision: D59132952
fbshipit-source-id: eea4856507a5b7c5f219efcf8d19fe56944df088
Summary:
LLVM-15 has a warning `-Wunused-but-set-variable` which we treat as an error because it's so often diagnostic of a code issue. Unused variables can compromise readability or, worse, performance.
This diff either (a) removes an unused variable and, possibly, it's associated code, or (b) qualifies the variable with `[[maybe_unused]]`, mostly in cases where the variable _is_ used, but, eg, in an `assert` statement that isn't present in production code.
- If you approve of this diff, please use the "Accept & Ship" button :-)
Reviewed By: dmm-fb
Differential Revision: D56065763
fbshipit-source-id: b0541b8a759c4b6ca0e8753fc24b8c227047bd3d
Summary:
This PR adds support for dimensionality reduction in OIVFBBS. I tested the code with an index `OPQ64_128,IVF4096,PQ64` using the ssnpp embeddings - this index string is added to the config_ssnpp.yaml to showcase this functionality.
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3290
Reviewed By: junjieqi
Differential Revision: D54878345
Pulled By: mlomeli1
fbshipit-source-id: 98ecdeb2224ce0325e37720cc113d82f9c6c75d6
Summary:
This PR introduces the offline IVF (OIVF) framework which contains some tooling to run search using IVFPQ indexes (plus OPQ pretransforms) for large batches of queries using [big_batch_search](https://github.com/mlomeli1/faiss/blob/main/contrib/big_batch_search.py) and GPU faiss. See the [README](https://github.com/facebookresearch/faiss/blob/36226f5fe8412162863d7d5d4cab97809f4d211d/demos/offline_ivf/README.md) for details about using this framework.
This PR includes the following unit tests, which can be run with the unittest library as so:
````
~/faiss/demos/offline_ivf$ python3 -m unittest tests/test_iterate_input.py -k test_iterate_back
````
In test_offline_ivf:
````
test_consistency_check
test_train_index
test_index_shard_equal_file_sizes
test_index_shard_unequal_file_sizes
test_search
test_evaluate_without_margin
test_evaluate_without_margin_OPQ
````
In test_iterate_input:
````
test_iterate_input_file_larger_than_batch
test_get_vs_iterate
test_iterate_back
````
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3202
Reviewed By: algoriddle
Differential Revision: D52734222
Pulled By: mlomeli1
fbshipit-source-id: 61fd0084277c1b14bdae1189db8ae43340611e16
Summary: Revert so that the test data can be replaced with synthetic
Reviewed By: mdouze
Differential Revision: D52388604
fbshipit-source-id: c0037635a4e66c54d42400294d13d9a80610b845
Summary:
This PR introduces the offline IVF (OIVF) framework which contains some tooling to run search using IVFPQ indexes (plus OPQ pretransforms) for large batches of queries using [big_batch_search](https://github.com/mlomeli1/faiss/blob/main/contrib/big_batch_search.py) and GPU faiss. See the [README](https://github.com/mlomeli1/faiss/blob/oivf/demos/offline_ivf/README.md) for details about using this framework.
This PR includes the following unit tests, which can be run with the unittest library as so:
````
~/faiss/demos/offline_ivf$ python3 -m unittest tests/test_iterate_input.py -k test_iterate_back
````
In test_offline_ivf:
````
test_consistency_check
test_train_index
test_index_shard_equal_file_sizes
test_index_shard_unequal_file_sizes
test_search
test_evaluate_without_margin
test_evaluate_without_margin_OPQ
test_evaluate_with_margin
test_split_batch_size_bigger_than_file_sizes
test_split_batch_size_smaller_than_file_sizes
test_split_files_with_corrupted_input_file
````
In test_iterate_input:
````
test_iterate_input_file_larger_than_batch
test_get_vs_iterate
test_iterate_back
````
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3175
Reviewed By: algoriddle
Differential Revision: D52218447
Pulled By: mlomeli1
fbshipit-source-id: 78b12457c79b02eb2c9ae993560f2e295798e7e5
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/2582
A few more or less cosmetic improvements
* Index::idx_t was in the Index object, which does not make much sense, this diff moves it to faiss::idx_t
* replace multiprocessing.dummy with multiprocessing.pool
* add Alexandr as a core contributor of Faiss in the README ;-)
```
for i in $( find . -name \*.cu -o -name \*.cuh -o -name \*.h -o -name \*.cpp ) ; do
sed -i s/Index::idx_t/idx_t/ $i
done
```
For the fbcode deps:
```
for i in $( fbgs Index::idx_t --exclude fbcode/faiss -l ) ; do
sed -i s/Index::idx_t/idx_t/ $i
done
```
Reviewed By: algoriddle
Differential Revision: D41437507
fbshipit-source-id: 8300f2a3ae97cace6172f3f14a9be3a83999fb89
Summary:
As discussed in https://github.com/facebookresearch/faiss/issues/685, I'm going to add an NSG index to faiss. This PR which adds an NNDescent index is the first step as I commented [here ](https://github.com/facebookresearch/faiss/issues/685#issuecomment-760608431).
**Changes:**
1. Add an `IndexNNDescent` and an `IndexNNDescentFlat` which allow users to construct a KNN graph on a million scale dataset using CPU and search NN on it. The implementation part is put under `faiss/impl`.
2. Add compilation entries to `CMakeLists.txt` for C++ and `swigfaiss.swig` for Python. `IndexNNDescentFlat` could be directly called by users in C++ and Python.
3. `VisitedTable` struct in `HNSW.h` is moved into `AuxIndexStructures.h`.
3. Add a demo `demo_nndescent.cpp` to demonstrate the effectiveness.
**TODO**
1. Support index factor.
2. Implement `IndexNNDescentPQ` and `IndexNNDescentSQ`
3. More comments in the code.
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/1654
Test Plan:
buck test //faiss/tests/:test_index_accuracy -- TestNNDescent
buck test //faiss/tests/:test_build_blocks -- TestNNDescentKNNG
Reviewed By: wickedfoo
Differential Revision: D26309716
Pulled By: mdouze
fbshipit-source-id: 2abade9708d29023f8bccbf77143e8eea14f66c4
Summary: Fixes 2 bugs spotted by ASAN in the demo.
Reviewed By: wenjieX
Differential Revision: D25897053
fbshipit-source-id: fd2bed13faded42426cefc5ebe9d027adec78015
Alibek Zhakubayev