Summary:
Three defects in `contrib/factory_tools.py` where a reversed factory string rebuilds a different index or fails to parse. `test_get_code_size_hnsw_roundtrip` already pins this invariant for flat HNSW; these cases escape it.
1. **HNSW storage dropped.** The `IndexHNSW` branch never read `index.storage`, so `HNSW32,SQ8` and `HNSW32,PQ16x8` both reversed to `"HNSW32"` and rebuilt as `IndexHNSWFlat`. `get_code_size` then applied the flat `d*4` formula: at d=128 it reported 768 bytes/vector for an index whose storage is 128 plus the 256-byte graph estimate, so 384.
2. **IMI regex fields transposed.** `index_factory.cpp:295` parses `IMI2x([0-9]+)`, but the regex read `IMI\d+x2`. Every real IMI key (`IMI2x10,PQ16`, and the ones in `demos/demo_auto_tune.py`) raised `RuntimeError("cannot parse")`.
3. **IndexIVFPQR matched by the IndexIVFPQ check**, dropping the refinement stage: `IVF64,PQ8+16` reversed to `IVF64,PQ8x8`, 8 bytes instead of 24.
Consumers are `benchs/bench_all_ivf/parse_bench_all_ivf.py` (imports `get_code_size` as `unitsize`) and `benchs/bench_fw/index.py`.
Flat storage stays implicit as the bare `HNSW<M>` string, and a null `storage` pointer still returns `HNSW<M>` instead of raising.
**Behavior change worth flagging:** `IndexHNSW2Level` storage is `Index2Layer`, which `reverse_index_factory` does not handle, so it now raises `NotImplementedError` instead of silently returning a wrong `"HNSW32"`. It is not reachable from `index_factory`.
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5564
Test Plan:
Four tests added to `TestFactoryTools`. All four fail on unpatched `factory_tools.py` and pass with it:
```
# pristine contrib/factory_tools.py, new tests present
4 failed, 7 passed
FAILED test_get_code_size_hnsw_non_flat_storage
FAILED test_get_code_size_imi
FAILED test_hnsw_storage_reverse_index_factory
FAILED test_ivfpqr_reverse_index_factory
# with the fix
11 passed
```
Full `tests/test_contrib.py tests/test_factory.py`, pristine vs branch:
```
pristine: 2 failed, 88 passed
branch: 2 failed, 92 passed
```
Failure sets are identical. The two are `TestComputeGT::test_compute_GT_gpu` and `test_compute_GT_ip_gpu`, pre-existing and unrelated (`AttributeError: module 'faiss' has no attribute 'GpuMultipleClonerOptions'` on a CPU-only build).
An A/B sweep over 40 factory strings changes exactly the 7 expected lines and nothing else:
```
- HNSW32,SQ8 IndexHNSWSQ HNSW32 768
+ HNSW32,SQ8 IndexHNSWSQ HNSW32,SQ8 384
- HNSW32,SQfp16 IndexHNSWSQ HNSW32 768
+ HNSW32,SQfp16 IndexHNSWSQ HNSW32,SQfp16 512
- HNSW32,PQ16x8 IndexHNSWPQ HNSW32 768
+ HNSW32,PQ16x8 IndexHNSWPQ HNSW32,PQ16x8 272
- IVF64,PQ8+16 IndexIVFPQR IVF64,PQ8x8 8
+ IVF64,PQ8+16 IndexIVFPQR IVF64,PQ8+16 24
- IMI2x5,Flat IndexIVFFlat IMI2x5,Flat <RuntimeError>
+ IMI2x5,Flat IndexIVFFlat IMI2x5,Flat 512
- IMI2x5,PQ8 IndexIVFPQ IMI2x5,PQ8x8 <RuntimeError>
+ IMI2x5,PQ8 IndexIVFPQ IMI2x5,PQ8x8 8
- IMI2x8,PQ8x8 IndexIVFPQ IMI2x8,PQ8x8 <RuntimeError>
+ IMI2x8,PQ8x8 IndexIVFPQ IMI2x8,PQ8x8 8
```
flake8 at the 80-char limit from CONTRIBUTING reports the same 12 pre-existing findings before and after, none in the changed lines.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Reviewed By: mnorris11
Differential Revision: D118661065
Pulled By: trang-nm-nguyen
fbshipit-source-id: b50a9522361e3fd14f276b36196d3c728567b980
Summary:
## Bug
`range_search_max_results()` (and its helper `apply_maxres()`) in `contrib/exhaustive_search.py` decide whether to keep the highest- or lowest-scoring results when pruning the accumulated results down to `max_results`, based on:
```python
keep_max=index.metric_type == faiss.METRIC_INNER_PRODUCT
```
This hardcodes the assumption that `METRIC_INNER_PRODUCT` is the only "similarity" metric (higher = better). But `faiss.is_similarity_metric()` (see `faiss/MetricType.h`) also treats `METRIC_Jaccard` as a similarity metric, and this same, correct check is already used a few lines above in `range_search_gpu()` in the very same file (`keep_max = faiss.is_similarity_metric(index_gpu.metric_type)`).
As a result, when `range_search_max_results` is used with a `METRIC_Jaccard` index and the result set grows past `max_results`, the pruning logic incorrectly discards the *highest*-similarity matches and keeps the *lowest*-similarity ones, and moves the returned radius in the wrong direction.
## Root cause
Two call sites in `contrib/exhaustive_search.py` compare `index.metric_type` directly against `faiss.METRIC_INNER_PRODUCT` instead of using the library's own `faiss.is_similarity_metric()` helper, which is the single source of truth for which metrics are "larger is better".
## Fix
Replace both occurrences of `index.metric_type == faiss.METRIC_INNER_PRODUCT` with `faiss.is_similarity_metric(index.metric_type)`, matching the existing pattern used in `range_search_gpu` in the same module.
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5533
Test Plan:
Added `test_query_iterator_similarity_metric` to `tests/test_contrib.py`. It exercises `range_search_max_results` against a lightweight fake index using `METRIC_Jaccard` (real `IndexFlat.range_search` doesn't support `METRIC_Jaccard` directly, so a fake index is used to isolate the pruning logic under test) and asserts that pruning keeps the highest-scoring results and raises the radius accordingly.
- Confirmed the new test fails on `main` (old code keeps the *lowest*-similarity results and lowers the radius) and passes after the fix, by swapping the module in-place against an installed `faiss-cpu` and running the assertions directly (full repo `pytest` collection isn't possible in this environment without rebuilding the C++ extension for `main`, since the installed wheel is older and `tests/common_faiss_tests.py` on `main` references APIs from a newer build).
- The fix is a pure two-line change that only affects the boolean passed to `apply_maxres`, verified by direct comparison of before/after behavior with the reproduction script.
Reviewed By: mnorris11
Differential Revision: D117001619
Pulled By: alibeklfc
fbshipit-source-id: 27e6757951d63a4d4f4f998640a1726ef8f6c220
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5444
We fix every remaining flake8 E501 (line > 80 chars) violation in
fbcode/faiss/ (excluding gpu/), closing out a lint-cleanup initiative that a
June 29, 2026 diff stack (9a05463576 -> d304e29b9a -> 7d140d86ca) explicitly
promised but left 41% incomplete. The middle diff in that stack aligned
.flake8's max-line-length to black's 80-char width and stated the remaining
277 violations after the config change were "genuine lint (E501 long
strings/URLs, F401/F841/B011, etc.), addressed in later commits in this
stack." The follow-up commit fixed only the F401/F403/F841/B011 categories
(45 issues) and never touched E501, leaving 192 violations across 53 files
live in trunk. A direct re-scan today found 201 violations across 54 files
(two files gained lines since the original count).
We fixed 183 violations across 52 files. Two of the originally-flagged
files, benchs/bench_rabitq.py and
demos/index_pq_flat_separate_codes_from_codebook.py, already carry a
file-level `# flake8: noqa` directive (the second is a Bento/Grimaldi
notebook export that also fails standalone py_compile due to embedded shell
magic) -- flake8 ignores both files entirely, so no changes were needed
there and they are excluded from this diff.
Fix approach, by category:
- The large majority of violations are prose: docstrings and comments
rewrapped at word boundaries, preserving each file's existing indentation
and docstring conventions. No wording was changed.
- Code lines (function calls, f-strings, assert messages, long argument
lists) were reflowed in black-compatible style: parenthesization, comma
breaks, or implicit adjacent string-literal concatenation. Every touched
file is confirmed unchanged by `black --line-length 80 --diff` after the
edit.
- A small number of genuinely unwrappable long URLs (no reasonable break
point) were suppressed with `# noqa: E501` rather than force-wrapped,
matching the precedent set by the original black-format diff stack.
- demos/offline_ivf/offline_ivf.py: a repeated, non-trivial ratio expression
used in both an assert condition and its (lazily-evaluated) failure
message was hoisted into a single local variable, eliminating the
duplicate computation. The expression is pure and deterministic (built
from already-computed arrays), so this is behaviorally identical to the
original -- not a functional change.
- `arc lint -a` additionally auto-fixed a real bug it detected while
processing these files: `raise NotImplemented` -> `raise
NotImplementedError` in contrib/evaluation.py (the sentinel object
NotImplemented is not an exception type; raising it produces a TypeError
masking the intended NotImplementedError). It also added a missing
trailing newline in benchs/fb/bench_index_binary_from_float.py and, in
the same file, rewrote the `#!/usr/bin/env python2` shebang to
`#!/usr/bin/env fbpython` -- that rewrite was reverted by hand, since the
file still does `import cPickle` (a Python 2-only module), so the
auto-fix would have made the shebang describe an interpreter the script
cannot actually run under. Fixing the underlying py2-to-py3 migration is
out of scope for this line-length-only sweep.
- Rebasing onto master (this diff's base commit had drifted) surfaced two
concurrent upstream fixes to `check_ref_knn_with_draws(...)` calls in
tests/test_ivf_flat_panorama.py and tests/test_refine_panorama.py (an
unrelated in-flight correctness fix passing `D_panorama`/`atol` instead
of a stale `D_regular` arg with no `atol`). The rebase took upstream's
corrected arguments and re-applied only this diff's line-wrap on top.
We verified zero functional change with an AST-level check: every string
and f-string literal in every touched file was extracted before and after
the edit and compared with whitespace normalized, confirming no wording,
value, or expression was altered beyond the intended `# noqa: E501`
suffixes and the one documented ratio-hoisting simplification above.
Reviewed By: mnorris11
Differential Revision: D112525458
fbshipit-source-id: e4b85d73dafcaac779f3522a383a66e490fa942a
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5427
# this is a no-op change
Part 2 of a 3-diff split of the faiss `black -l 80` reformat into small, reviewable diffs (each 3-4 files, <100 changed lines). Follow-up to D109736099 (landed), which formatted the faiss Python tree; these files drifted or are new.
Ran `black -l 80` (line length 80, matching faiss's `CONTRIBUTING.md`: "80 character line length (both for C++ and Python)") over the test files: `tests/test_build_blocks.py`, `tests/test_contrib.py`, `tests/test_eden.py`, `tests/test_ivf_flat_panorama.py`.
Pure-formatting only. `black`'s AST-equivalence safety check guarantees no identifier, string/numeric value, operator, or control-flow change — only whitespace, wrapping, trailing commas, and quote normalization. Independently verified: `ast.dump` equality (docstrings normalized) and comment-content preservation. `fbcode/faiss/` is excluded from Meta's standard Python autoformatter (it mirrors to public GitHub), so `black -l 80` is the correct tool.
___
Reviewed By: trang-nm-nguyen
Differential Revision: D112191719
fbshipit-source-id: 20fc89768077a16fb6abd6c2fe58981c3febf0ea
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5419
### THIS DIFF
Fix two bugs in `reverse_index_factory` for multi-bit RaBitQ indexes, in
both the C++ and Python implementations.
**C++ (`factory_tools.cpp`):**
- `IndexIVFRaBitQ` case returned `prefix + ",RaBitQ"` unconditionally,
regardless of `rabitq.nb_bits`. For a 2-bit index built from
`"IVF_nlist,RaBitQ2"`, the function returned `"IVF_nlist,RaBitQ"` (1-bit
factory string). This means `TelemetryWrapper` logs the wrong index type
for every production multi-bit RaBitQ IVF index.
- `IndexRaBitQ` (standalone) had the same flaw — always returned
`"RaBitQ"` regardless of `nb_bits`.
**Python (`contrib/factory_tools.py`):**
- Neither `IndexIVFRaBitQ` nor `IndexRaBitQ` had a case in
`reverse_index_factory`, so any Python caller using this API on a RaBitQ
index raised `NotImplementedError`.
**Fix:** Read `rabitq.nb_bits` from the typed cast result and append the
bit count when `nb_bits > 1` (e.g., `"RaBitQ2"`). For `nb_bits == 1`
the output is still `"RaBitQ"`, preserving backward compatibility with
existing logs and round-trip tests.
### CONTEXT
Multi-bit RaBitQ (`nb_bits=2..9`) was added alongside standard 1-bit
RaBitQ. `index_factory` has parsed `"RaBitQ2"`, `"RaBitQ4"`, etc., since
the feature landed, but `reverse_index_factory` was never updated to emit
these strings. Every production `IndexIVFRaBitQ` with `nb_bits > 1`
(including the RaBitQ2 deployment) logs `",RaBitQ"` (1-bit) in telemetry.
The Python API has silently been broken for all RaBitQ index types.
The fix pattern matches how other parameterized types (`PQMxN`, `HNSWNM`,
etc.) embed their parameters in the factory string.
Reviewed By: mnorris11
Differential Revision: D110042828
fbshipit-source-id: d4e976bd9fc99691a20fb33d893af2d36deeb1e1
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5416
We fix two bugs in `get_code_size` (`contrib/factory_tools.py`) that raise
`RuntimeError` for any valid HNSW factory string with M≠32.
**Bug 1 — IVF+HNSW regex hardcodes M=32 (line 20):**
`re.match("IVF\\d+(_HNSW32)?,(.*)")` — `_HNSW8`, `_HNSW16`, `_HNSW64` do
not match. Execution falls through all branches to
`raise RuntimeError("cannot parse IVF64_HNSW16,Flat")`.
Fix: change `_HNSW32` to `_HNSW\\d+`.
**Bug 2 — Standalone HNSW string comparison hardcodes M=32 (lines 48–49):**
`if indexkey == "HNSW32" or indexkey == "HNSW32,Flat"` — any other M falls
through to `RuntimeError`. Fix: replace with
`re.match("HNSW(\\d+)(,Flat)?$")` and generalized formula
`d * 4 + M * 2 * 4` (equivalent to the original `d * 4 + 64 * 4` for M=32;
the `# roughly` comment is preserved since HNSW graph overhead depends on
ntotal and level distribution).
**Root cause:** `reverse_index_factory` already correctly handles any M via
`get_hnsw_M()` — line 96 produces `"IVF{n}_HNSW{M},..."`, line 125 produces
`"HNSW{M}"`. The standard roundtrip
`get_code_size(d, reverse_index_factory(index))` crashes for every
IVF+HNSW or standalone HNSW index with M≠32. HNSW M=16 and M=64 are common
production configurations.
Reviewed By: limqiying
Differential Revision: D109142320
fbshipit-source-id: 7ddcfd83211b7e95e1bab20a3dee017c4d08e450
Summary:
`OperatingPoints.add_operating_point` (in `contrib/evaluation.py`) maintains a Pareto-optimal set of `(key, perf, t)` points where higher `perf` and lower `t` are better. The admission test `is_pareto_optimal` rejects a new point when an existing one dominates it using **non-strict** comparisons in both dimensions:
```python
if perf >= perf_new and t <= t_new: # existing dominates new
return False
```
But the prune loop that removes previously-stored points the new point now dominates used a **strict** time comparison:
```python
if perf >= perf2 and t < t2: # <-- strict on time
self.suboptimal_points.append(self.operating_points.pop(i))
```
Because of this asymmetry, when a new point has **strictly better performance at the same time** as a stored point, the new point is admitted (it is not dominated), but the older point it now dominates is **not** removed. The frontier then contains a point dominated by another point in the same set, breaking the Pareto invariant. Dominated operating points leak into the "optimal" set used by the benchmark/auto-tune machinery (`benchs/bench_fw/utils.py`, `benchs/bench_hybrid_cpu_gpu.py`). This also diverges from the C++ `AutoTune::add` this is documented to port (`faiss/AutoTune.cpp`), which replaces an equal-perf point at lower-or-equal time.
## Fix
Use `t <= t2` in the prune loop so it matches the admission test:
```diff
- if perf >= perf2 and t < t2:
+ if perf >= perf2 and t <= t2:
```
This is safe: identical points never reach the loop (they fail `is_pareto_optimal` and go straight to `suboptimal_points`), so only genuinely dominated points are removed.
## Validation
Extracted the real `is_pareto_optimal` / `add_operating_point` logic verbatim and ran:
- **Targeted:** add `A=(0.5, t=1.0)` then `B=(0.9, t=1.0)`. Before: frontier `['A', 'B']` (A dominated by B). After: `['B']`.
- **Fuzz (2000 random sequences, coarse `t` to force equal-time ties):** dominated points left in the frontier — **before: 249, after: 0**.
Added `TestOperatingPoints.test_equal_time_better_perf_shadows` to `tests/test_contrib.py` (pure-Python, no compiled core needed); it fails on `main` and passes with the fix.
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5334
Reviewed By: trang-nm-nguyen
Differential Revision: D110116217
Pulled By: limqiying
fbshipit-source-id: 554b2bb92c22eb1c4b00896ec36fa1db46756f06
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5351
Fixes 45 genuine (non-style) flake8 issues across 20 faiss Python files, on top of the black-80 base in this stack. These are real code issues, not formatting.
- `B011` (19): replaced `assert False` / `assert False, "msg"` with `raise AssertionError(...)`. `assert` statements are stripped under `python -O`, so an `assert False` guard silently vanishes in optimized builds; `raise AssertionError` always fires.
- `F401` (10) / `F403` (7): the intentional star-import re-exports in `python/loader.py` and `python/loader_fb.py` (which expose the SWIG symbols as `faiss.X`) are annotated with `# noqa: F401,F403`; genuinely unused imports in test/bench scripts are removed.
- `F841` (9): unused local variables removed, or replaced with `_` in tuple unpacking, preserving behavior.
No runtime behavior changes; formatting stays at black width 80.
Reviewed By: mnorris11
Differential Revision: D109739445
fbshipit-source-id: 7d140d86cab9b2df691003bd043fc880a31410cd
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5349
# this is a no-op change
Ran `black -l 80` over all tracked Python files under `fbcode/faiss/`, using line length 80 to match faiss's documented coding style (`CONTRIBUTING.md`: "80 character line length (both for C++ and Python)"). 172 files were reformatted; 28 were already compliant.
This is a pure-formatting change. `black` runs its AST-equivalence safety check on every file, so no identifiers, string/numeric values, operators, or control flow were altered — only whitespace, indentation, line wrapping, blank lines, trailing commas, and string-quote normalization (single to double quotes).
`fbcode/faiss/` is intentionally excluded from Meta's standard Python autoformatter (`arc f` / `arc lint -a` report no formatter for these paths) because it mirrors to public GitHub, so the Python had drifted out of any consistent style. The C++ is already clang-format-clean (0 of 807 files change), so this effort is Python-only. This diff brings the Python to canonical `black` formatting.
Excluded `demos/index_pq_flat_separate_codes_from_codebook.py`: it contains a Jupyter `!rm` shell-magic line and is not valid Python, so `black` cannot parse it.
This is the base of a 3-diff stack: (1) this black reformat; (2) `.flake8` alignment (max-line-length 80 + black-compatible ignores E203/W503); (3) genuine lint fixes (F401/F403/F841/B011).
Reviewed By: mnorris11
Differential Revision: D109736099
fbshipit-source-id: 9a054635766ed11aa3760a788395babd3d198abf
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5322
`reverse_index_factory` in `contrib/factory_tools.py` had two bugs:
1. `IndexIVFScalarQuantizer` branch hardcoded `prefix + ",SQ8"` for every quantizer type. Any IVF+SQfp16, IVF+SQ4, IVF+SQbf16, IVF+SQ8_direct, IVF+SQ8_direct_signed, or IVF+TurboQuant index returned the wrong factory string. When used to reconstruct the index via `index_factory`, the string produced a wrong type. All non-SQ8 IVF scalar-quantizer indexes were mislogged by the telemetry wrapper.
2. `IndexScalarQuantizer` branch's `sqtypes` dict held 5 entries (QT_8bit, QT_4bit, QT_6bit, QT_fp16, QT_bf16) and used `sqtypes[index.sq.qtype]` directly. Any standalone SQ index with QT_8bit_direct, QT_8bit_direct_signed, or any of the 9 TurboQuant types (QT_1bit_tqmse through QT_5bit_tq, added in b985e9c43271, 2026-06-04) raises KeyError with no error message.
We fix both by hoisting a single `sq_names` dict to the top of `reverse_index_factory`, covering all 19 `ScalarQuantizer::QuantizerType` values. Both the IVF and standalone branches look up from this shared dict.
Addressing review feedback (mnorris): the dict now also includes the three remaining enum values that were initially missed — `QT_0bit` ("SQ0", parsed by `index_factory` and intended for the IVF path), and `QT_8bit_uniform`/`QT_4bit_uniform` ("SQ8u"/"SQ4u", synthetic names since `index_factory` does not parse the uniform variants). The dict is ordered to mirror the C++ enum so it can be checked for completeness against it.
`test_ivf_scalar_quantizer_types` verifies all 19 types in the IVF context; `test_scalar_quantizer_types` verifies the 18 standalone types (`QT_0bit` is IVF-only).
Reviewed By: mnorris11
Differential Revision: D108272403
fbshipit-source-id: 3f747966495cfe7eee1cc4c328cba51b6ee9bcba
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5266
`reverse_index_factory` in `contrib/factory_tools.py` raised `NotImplementedError` for `IndexIDMap` and `IndexIDMap2`, even though both simply wrap an inner index. This adds handling for both: an `IndexIDMap` is described as `IDMap,<inner>` and an `IndexIDMap2` as `IDMap2,<inner>`, recursing into the wrapped index. `IndexIDMap2` is checked first because it subclasses `IndexIDMap`. This matches the C++ `reverse_index_factory` in `faiss/factory_tools.cpp`, which already handles `IDMap`, and round-trips with `index_factory` for IDMap-wrapped indexes such as `IDMap,Flat` and `IDMap,IVF100,Flat`.
Reviewed By: mnorris11
Differential Revision: D107159123
fbshipit-source-id: 40dca17a78743bf53373ec3077fd640e0fff76d4
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5221
## TL;DR
Two related bugs in the Python wrappers for `search_preassigned` and `range_search_preassigned`: passing `Dq=None` crashes with `ValueError`, and `np.empty` in `ivf_tools.range_search_preassigned` produces nondeterministic results for IVFPQ with `by_residual=True`. We fix both by substituting `np.zeros` when the caller omits coarse distances.
## Bug 1: Dq=None crashes (class_wrappers.py, 4 sites)
The Python wrappers for `search_preassigned` and `range_search_preassigned` on both `Index` and `IndexBinaryIVF` document `Dq` as optional, but passing `None` crashes:
```
ValueError: argument is not a numpy array
```
The cause: each wrapper validates `Dq` inside `if Dq is not None:` but has no `else` branch. When `Dq` is `None`, `swig_ptr(Dq)` is called with `None`, which SWIG rejects.
## Bug 2: np.empty nondeterminism (ivf_tools.py)
`ivf_tools.range_search_preassigned` allocates a dummy `coarse_dis` with `np.empty` when the caller does not provide one. `np.empty` returns uninitialized memory. For `IndexIVFPQ` with L2 metric and `by_residual=True` (the default), the C++ search kernel reads `coarse_dis` for distance correction. Uninitialized values produce nondeterministic search results with no error or warning.
The sibling function `ivf_tools.search_preassigned` already uses `np.zeros` for the same purpose. The `range_search_preassigned` variant diverged.
## Fix
**class_wrappers.py (4 sites):** Add `else: Dq = np.zeros(Iq.shape, dtype=DTYPE)` after each `if Dq is not None:` block. Float wrappers use `dtype='float32'`, binary wrappers use `dtype='int32'`. `Iq` is already coerced and shape-validated at each site, so the shape is correct.
**ivf_tools.py (1 site):** Change `np.empty` to `np.zeros` at line 75. Restores parity with `search_preassigned` at line 53.
## Why zeros
`Dq` represents the distances from queries to their assigned IVF centroids. Zeros means "no distance correction." For most index types (`IVF,Flat`, `IVF,SQ8`, `IndexBinaryIVF`), the C++ code ignores `Dq` entirely in scoring. For `IVFPQ` with `by_residual=True`, zeros omit the coarse component from the returned distances but preserve the neighbor ranking (because all vectors in the same inverted list share the same coarse distance, so it cancels in relative comparisons). Users who need accurate absolute distances should compute and pass real `Dq` via `index.quantizer.search()`.
Zeros is deterministic, semantically neutral, and consistent with the existing convention in `ivf_tools.search_preassigned`.
Reviewed By: junjieqi
Differential Revision: D105299681
fbshipit-source-id: 9f59a7bd839b253ee31d42c1ed013234a7adfd87
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5050
- Implement balanced assignment in clustering.py based on notebook N10159950
- add a test that shows we improve the imabalace at some cost in MSE
Reviewed By: algoriddle
Differential Revision: D99819394
fbshipit-source-id: 568b6deb7d2b95b8228dbb276c5578df23b01a96
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/4523
This upgrades OSS Faiss to numpy 2.3.3. I tried to fix issues based on https://numpy.org/devdocs/numpy_2_0_migration_guide.html.
Overall, numpy2 causes some test issues on various platforms/optimization levels.
Test changes that are OK:
- test_contrib.py: test_float: changing to use item(): this is just the new way of getting the value, like recommended in the migration guide.
Test changes that indicate some problem:
- test_contrib.py: test_ivf_train_2level: This fails on many platforms (avx2, conda installs). I could not find what the issue was. I had to increase the bounds slightly.
- test_index_accuracy.py: test_OIVFPQ: This fails just on AVX512. I had to increase the iterations from 10 to 12.
- test_local_search_quantizer.py and test_standalone_codec.py: Some of these break on Windows. Others needed lighter tolerances.
Debugging attempts: Especially for test_ivf_train_2level, I tried to unsuccessfully track down what might have changed. I suspected the type promotion change from numpy1 to 2.
```
- np.float32(3) + 3. now returns a float32 when it previously returned a float64.
- np.array([3], dtype=np.float32) + np.float64(3) will now return a float64 array. (The higher precision of the scalar is not ignored.)
```
I changed all datasets to be explicitly float64s, and checked for any constants in the code path. No luck.
Devmate was less than useless, it was actively a waste of time (as usual...)
Reviewed By: subhadeepkaran
Differential Revision: D79900113
fbshipit-source-id: 541cfa6f1fa1d60c84d855ce71f0e1de7ea7c7f1
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/4541
Per PEP 8 style guide (https://pep8.org/#blank-lines), top-level class
definitions should be surrounded by two blank lines. This commit addresses
\E302 lint violations across the faiss codebase by ensuring all class
definitions have exactly 2 blank lines before them.
Changes:
- Fixed \E302 errors in 28 Python files across tests, benchmarks, contrib,
and GPU modules
- Applied consistent PEP 8 formatting for class definitions
- No functional changes - formatting only
Files affected include test files, benchmark scripts, contrib modules,
and GPU test utilities. All changes maintain existing functionality
while improving code style compliance.
Reviewed By: junjieqi
Differential Revision: D80682723
fbshipit-source-id: 6ad5458c3bf1e5c06f35dc8fcdbea2ac4ba83c5f
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/4126
Good resource on overriding channels to make sure we aren't using `defaults`:https://stackoverflow.com/questions/67695893/how-do-i-completely-purge-and-disable-the-default-channel-in-anaconda-and-switch
Explanation of changes:
-
- changed to miniforge from miniconda: this ensures we only pull in from conda-defaults when creating the environment
- architecture: ARM64 and aarch64 are the same thing. But there is no miniforge package for ARM64, so we need to make it check for aarch64 instead. However, mac breaks this rule, and does have macOS-arm64! So there is a conditional for mac to use arm64. https://github.com/conda-forge/miniforge/releases/
- action.yml mkl 2022.2.1 change: conda-forge and defaults have completely different dependencies. Defaults required intel-openmp, but now on conda-forge, mkl 2023.1 or higher requires llvm-openmp >=14.0.6, but this is incompatible with the pytorch build <2.5 which requires llvm-openmp<14.0. We would need to upgrade Python to 3.12 first, upgrade Pytorch build, then upgrade this mkl. (The meta.yaml changes are the ones that narrow it to 2022.2.1 during `conda build faiss`.) So, this has just been changed to 2022.2.1.
- mkl now requires _openmp_mutex of type "llvm" instead of "gnu": prior non-cuVS builds all used gnu, because intel-openmp from anaconda defaults channel does not require llvm-openmp. Now we need to remove the gnu one which is automatically pulled in during miniconda setup, and only keep the llvm version of _openmp_mutex.
- liblief: The above changes tried to pull in liblief 0.15. This results in an error like `AttributeError: module 'lief._lief.ELF' has no attribute 'ELF_CLASS'`. When I checked passing PR builds on defaults, they use lief 0.12, so I pinned that one for Python 3.9 3.10 3.11. For Python 3.12, we need lief 0.14 or higher.
- gcc_linux-64 =11.2 for faiss-gpu on cudatoolkit-11.2: GPU builds kept trying to reference 11.2 when 14.2 was installed. I couldn't figure out why, or how to point it to the 14.2 installed on the host. Current nightly builds still reference 11.2, so I gave up and pinned 11.2 to keep it the same. Moving to 14.2 will take some more investigation.
- meta.yaml mkl 2023.0 vs 2023.1 with python versions: 3.9, 3.10, and 3.11 pass with 2023.0, but python 3.12 needs mkl 2023.1 or higher. Otherwise we get:
```
INTEL MKL ERROR: $PREFIX/lib/python3.12/site-packages/faiss/../../.././libmkl_def.so.2: undefined symbol: mkl_sparse_optimize_bsr_trsm_i8.
Intel MKL FATAL ERROR: Cannot load libmkl_def.so.2.
```
so the solution was to put a bunch of conditions in in faiss/meta.yaml.
We should be able to use Jinja macros to reduce duplication but it requires some investigation. It was failing: https://github.com/facebookresearch/faiss/actions/runs/12915187334/job/36016477707?pr=4126 (paste of logs here: P1716887936). This can be a future BE task.
Macro example (the `-` signs remove whitespace lines before and after)
```
{% macro inclmkldevel() %}
{%- if PY_VER == '3.9' or PY_VER == '3.10' or PY_VER == '3.11' -%}
- mkl-devel =2023.0 # [x86_64]
- liblief =0.12.3 # [not win]
- python_abi <3.12
{%- elif PY_VER == '3.12' %}
- mkl-devel >=2023.2.0 # [x86_64]
- liblief =0.15.1 # [not win]
- python_abi =3.12
{% endif -%}
{% endmacro %}
```
The python_abi was required to be pinned inside these conditions because otherwise several builds got this error:
```
File "/Users/runner/miniconda3/lib/python3.12/site-packages/conda_build/utils.py", line 1919, in insert_variant_versions
matches = [regex.match(pkg) for pkg in reqs]
^^^^^^^^^^^^^^^^
TypeError: expected string or bytes-like object, got 'list'
```
Unit test notes:
-
- test_gpu_basics.py: GPU residual quantizer: Debugged extensively with Matthijs. The problem is in the C++ -> Python conversion. The C++ side prints the right values, but when getting it back to Python, it is filled with junk data. It is only reproducible on CUDA 11.4.4 after switching channels. It is likely a compiler problem. We discussed, and resolved to create a C++ side unit test (so this diff creates TestGpuResidualQuantizer) to verify the functionality and disable the Python unit test, but leave it in the codebase with a comment. Matthijs made extensive notes in https://docs.google.com/document/d/1MjMdOpPgx-MArdrYJZCaQlRqlrhSj5Y1Z9lTyiab8jc/edit?usp=sharing .
- test_contrib.py: this now hangs forever and times out the runner for Windows on Python 3.12. I have it skipping now.
- test_mem_leak.cpp seems flaky. It sometimes fails, then passes with rerun.
Unfixed issues:
-
- I noticed sometimes downloads will fail with the text like below. It passes on re-run.
```
libgomp-14.2.0-h77fa898_1.conda extraction failed
Warning: error libmamba Error when extracting package: Could not chdir info/recipe/parent/patches/0005-Hardcode-HAVE_ALIGNED_ALLOC-1-in-libstdc-v3-configur.patch
error libmamba Error when extracting package: Could not chdir info/recipe/parent/patches/0005-Hardcode-HAVE_ALIGNED_ALLOC-1-in-libstdc-v3-configur.patch
Warning: Found incorrect download: libgomp. Aborting
Found incorrect download: libgomp. Aborting
Warning:
```
Green build and tests for both build pull request and nightlies: https://github.com/facebookresearch/faiss/actions/runs/12956402963/job/36148818361
Reviewed By: asadoughi
Differential Revision: D68043874
fbshipit-source-id: b105a1e3e6272763ad9daab7fc6f05a79f01c9e2
Summary:
This pull request introduces a new default argument, `ngpu=-1`, to the `knn_ground_truth` function in the `faiss.contrib`.
## Purpose of Change
### Bug Fix
In the current implementation, running tests under the tests directory (CPU tests) in an environment with faiss-gpu installed would inadvertently use the GPU and cause unintended behavior.
This pull request prevents the GPU from being used during CPU-only tests by explicitly controlling GPU allocation via the ngpu parameter.
### API Consistency
Other functions that call `faiss.get_num_gpus` in `faiss.contrib`, such as `range_search_max_results` and `range_ground_truth`, already include the `ngpu` argument.
Adding this parameter to `knn_ground_truth` will ensure consistency across the API, reduce potential confusion, and improve ease of use.
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/4123
Reviewed By: asadoughi
Differential Revision: D68199506
Pulled By: junjieqi
fbshipit-source-id: cb50e206d8a1a982c21b0ccb42825ea45873f3ef
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3872
The contrib.torch subdirectory is intended to receive modules in python that are useful for similarity search and that apply to CPU or GPU pytorch tensors.
The current version includes CPU clustering on torch tensors. To be added:
* implementation of PQ
Reviewed By: asadoughi
Differential Revision: D62759207
fbshipit-source-id: 87dbaa5083e3f2f4f60526815e22ded4e83e8559
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3873
The previous version required scipy to do the accumulation, which is replaced here with a nifty piece of numpy accumulation.
This removes the need for scipy for non-sparse data.
Reviewed By: junjieqi
Differential Revision: D62884307
fbshipit-source-id: 5443634e487387a2b518fd2a7f9a3d9a40abd4b4
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3452
Delete all remaining print within the Tests to improve the readability and effectiveness of the codebase.
Reviewed By: junjieqi
Differential Revision: D57466393
fbshipit-source-id: 6ebd66ae2e769894d810d4ba7a5f69fc865b797d
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3327
**Context**
1. [Issue 2621](https://github.com/facebookresearch/faiss/issues/2621) discuss inconsistency between OnDiskInvertedList and InvertedList. OnDiskInvertedList is supposed to handle disk based multiple Index Shards. Thus, we should name it differently when merging invls from index shard.
2. [Issue 2876](https://github.com/facebookresearch/faiss/issues/2876) provides usecase of shifting ids when merging invls from different shards.
**In this diff**,
1. To address #1 above, I renamed the merge_from function to merge_from_multiple without touching merge_from base class.
why so? To continue to allow merge invl from one index to ondiskinvl from other index.
2. To address #2 above, I have added support of shift_ids in merge_from_multiple to shift ids from different shards. This can be used when each shard has same set of ids but different data. This is not recommended if id is already unique across shards.
Reviewed By: mdouze
Differential Revision: D55482518
fbshipit-source-id: 95470c7449160488d2b45b024d134cbc037a2083
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/2984
It is not entirely trivial to access the NSG graph structure from Python (although it is a fixed size N-by-K matrix of vector ids).
This diff adds an inspect_tools function to do that.
Reviewed By: algoriddle
Differential Revision: D48026775
fbshipit-source-id: 94cd7be7f656bcd333d62586531f287ea8e052e5
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/2949
A more scalable alternative to `np.unique` for deduping large datasets with a quantized code.
Reviewed By: mlomeli1
Differential Revision: D47443953
fbshipit-source-id: 4a1554d4d4200b5fa657e9d8b7395bba9856a8e3
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/2916
Overall better support for binary indexes:
- cloning (to CPU and GPU), only for BinaryFlat for now
- fix bug in reconstruct_n
- range_search_max_results
Reviewed By: algoriddle
Differential Revision: D46755778
fbshipit-source-id: 777ad90aff5c54a77f9685ed6512247a922c6ef5
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/2901
This diff allows each GPU to work independently, a hot centroid (eg. out-of-distribution queries that hit a centroid heavily) will only block the one GPU that is processing it, others will continue to pick up work independently.
Reviewed By: mdouze
Differential Revision: D46521298
fbshipit-source-id: 171cb06cce8b2d16b7bd744799b105b3cd525be3
Summary: In the IndexIVFIndepenentQuantizer, the coarse quantizer is applied on the input vectors, but the encoding is performed on a vector-transformed version of the database elements.
Reviewed By: alexanderguzhva
Differential Revision: D45950970
fbshipit-source-id: 30f6cf46d44174b1d99a12384b7d5e2d475c1f88
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/2860
Optimized range search function where the GPU computes by default and falls back on gpu for queries where there are too many results.
Parallelize the CPU to GPU cloning, it seems to work.
Support range_search_preassigned in Python
Fix long-standing issue with SWIG exposed functions that did not release the GIL (in particular the MapLong2Long).
Adds a MapInt64ToInt64 that is more efficient than MapLong2Long.
Reviewed By: algoriddle
Differential Revision: D45672301
fbshipit-source-id: 2e77397c40083818584dbafa5427149359a2abfd
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/2846
Adds a function to ivf_contrib to sort the inverted lists by size without changing the results. Also moves big_batch_search to its own module.
Reviewed By: algoriddle
Differential Revision: D45565880
fbshipit-source-id: 091a1c1c074f860d6953bf20d04523292fb55e1a
Summary: Big batch search can be running for hours so it's useful to have a checkpointing mechanism in case it's run on a best-effort cluster queue.
Reviewed By: algoriddle
Differential Revision: D44059758
fbshipit-source-id: 5cb5e80800c6d2bf76d9f6cb40736009cd5d4b8e
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/2567
Intuitively, it should be easier to handle big-batch searches because all distance computations for a set of queries can be done locally within each inverted list.
This benchmark implements this in pure python (but should be close to optimal in terms of speed), on CPU for IndexIVFFlat, IndexIVFPQ and IndexIVFScalarQuantizer. GPU is also supported.
The results are not systematically better, see https://docs.google.com/document/d/1d3YuV8uN7hut6aOATCOMx8Ut-QEl_oRnJdPgDBRF1QA/edit?usp=sharing
Reviewed By: algoriddle
Differential Revision: D41098338
fbshipit-source-id: 479e471b0d541f242d420f581775d57b708a61b8
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/2294
there is a weird CI failure on one of the platforms occurring in the PR
https://github.com/facebookresearch/faiss/pull/2291
This diff makes the test a bit more robust, correcting inter_perf to computer the intersection measure. Hopefully this will make the bug go away.
Reviewed By: beauby
Differential Revision: D35558855
fbshipit-source-id: f5a926d9d8ebee975e538c65ac37b15d485798aa
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/2217
This diff introduces a new Faiss contrib module that contains:
- generic k-means implemented in python (was in distributed_ondisk)
- the two-level clustering code, including a simple function that runs it on a Faiss IVF index.
- sparse clustering code (new)
The main idea is that that code is often re-used so better have it in contrib.
Reviewed By: beauby
Differential Revision: D34170932
fbshipit-source-id: cc297cc56d241b5ef421500ed410d8e2be0f1b77
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/2132
This diff adds the class IndexFlatCodes that becomes the parent of all "flat" encodings.
IndexPQ
IndexFlat
IndexAdditiveQuantizer
IndexScalarQuantizer
IndexLSH
Index2Layer
The other changes are:
- for IndexFlat, there is no vector<float> with the data anymore. It is replaced with a `get_xb()` function. This broke quite a few external codes, that this diff also attempts to fix.
- I/O functions needed to be adapted. This is done without changing the I/O format for any index.
- added a small contrib function to get the data from the IndexFlat
- the functionality has been made uniform, for example remove_ids and add are now in the parent class.
Eventually, we may support generic storage for flat indexes, similar to `InvertedLists`, eg to memmap the data, but this will again require a big change.
Reviewed By: wickedfoo
Differential Revision: D32646769
fbshipit-source-id: 04a1659173fd51b130ae45d345176b72183cae40
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/1972
This fixes a few issues that I ran into + adds tests:
- range_search_max_results with IP search
- a few missing downcasts for VectorTRansforms
- ResultHeap supports max IP search
Reviewed By: wickedfoo
Differential Revision: D29525093
fbshipit-source-id: d4ff0aff1d83af9717ff1aaa2fe3cda7b53019a3
Summary:
Adds the preassigned add and search python wrappers to contrib.
Adds the preassigned search for the binary case (was missing before).
Also adds a real test for that functionality.
Reviewed By: beauby
Differential Revision: D26560021
fbshipit-source-id: 330b715a9ed0073cfdadbfbcb1c23b10bed963a5
Summary: The order of xb an xq was different between `faiss.knn` and `faiss.knn_gpu`. Also the metric argument was called distance_type. This diff fixes both. Hopefully not too much external code depends on it.
Reviewed By: wickedfoo
Differential Revision: D26222853
fbshipit-source-id: b43e143d64d9ecbbdf541734895c13847cf2696c
Summary:
Added a few functions in contrib to:
- run range searches by batches on the query or the database side
- emulate range search on GPU: search on GPU with k=1024, if the farthest neighbor is still within range, re-perform search on CPU
- as reference implementations for precision-recall on range search datasets
- optimized code to plot precision-recall plots (ie. sweep over thresholds)
The new functions are mainly in a new `evaluation.py`
Reviewed By: wickedfoo
Differential Revision: D25627619
fbshipit-source-id: 58f90654c32c925557d7bbf8083efbb710712e03
kzahiri1