mirror of
https://github.com/facebookresearch/faiss.git
synced 2026-10-12 02:13:36 +00:00
main
10
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Tia Lim
|
0f0f728166 |
Format faiss Python files with black (line length 80) (#5349)
Summary: Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5349 # this is a no-op change Ran `black -l 80` over all tracked Python files under `fbcode/faiss/`, using line length 80 to match faiss's documented coding style (`CONTRIBUTING.md`: "80 character line length (both for C++ and Python)"). 172 files were reformatted; 28 were already compliant. This is a pure-formatting change. `black` runs its AST-equivalence safety check on every file, so no identifiers, string/numeric values, operators, or control flow were altered — only whitespace, indentation, line wrapping, blank lines, trailing commas, and string-quote normalization (single to double quotes). `fbcode/faiss/` is intentionally excluded from Meta's standard Python autoformatter (`arc f` / `arc lint -a` report no formatter for these paths) because it mirrors to public GitHub, so the Python had drifted out of any consistent style. The C++ is already clang-format-clean (0 of 807 files change), so this effort is Python-only. This diff brings the Python to canonical `black` formatting. Excluded `demos/index_pq_flat_separate_codes_from_codebook.py`: it contains a Jupyter `!rm` shell-magic line and is not valid Python, so `black` cannot parse it. This is the base of a 3-diff stack: (1) this black reformat; (2) `.flake8` alignment (max-line-length 80 + black-compatible ignores E203/W503); (3) genuine lint fixes (F401/F403/F841/B011). Reviewed By: mnorris11 Differential Revision: D109736099 fbshipit-source-id: 9a054635766ed11aa3760a788395babd3d198abf |
||
|
Alibek Zhakubayev
|
fd7be0cfb0 |
Add QT_0bit and QT_2bit_tq..QT_5bit_tq to ScalarQuantizer pyi stub (#5323)
Summary: Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5323 QT_2bit_tq, QT_3bit_tq, QT_4bit_tq, and QT_5bit_tq were added in D102408184 (TurboQuant full algorithm, June 4) but were not added to the authoritative `__init__.pyi` stub wired in D107274305 (June 3). `QT_0bit` (the centroid-only scalar quantizer used by IVF) is likewise missing from the stub. Pyre users referencing these types (e.g. `faiss.ScalarQuantizer.QT_2bit_tq` or `faiss.ScalarQuantizer.QT_0bit`) get [missing-attribute] errors even though the values are accessible at runtime (`QT_0bit` = 9, `QT_2bit_tq`..`QT_5bit_tq` = 15-18). We add all five enum values to the `ScalarQuantizer` stub, ordered to mirror the C++ enum in `impl/ScalarQuantizer.h`. Two tests in `test_swig_wrapper` assert the ordinal values: `test_scalar_quantizer_tq_enum_values` (15-18) and `test_scalar_quantizer_0bit_enum_value` (9). This is the same pattern as D108001955 (IVFSQTurboQSearchParameters stub), which fixed the search-params side of the same June 4 commit. The `QT_0bit` entry also unblocks D108272403, which references `faiss.ScalarQuantizer.QT_0bit` in `reverse_index_factory`. Reviewed By: limqiying Differential Revision: D108738051 fbshipit-source-id: 1557a0baa30ce046cef54cf0f47960dd8b90ebe3 |
||
|
Alibek Zhakubayev
|
261d8f25aa |
Add IVFSQTurboQSearchParameters to __init__.pyi stub (#5304)
Summary: Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5304 Add IVFSQTurboQSearchParameters to the authoritative __init__.pyi type stub. TurboQuant was added to ScalarQuantizer in D102408184, which introduced IVFSQTurboQSearchParameters in IndexScalarQuantizer.h. The struct controls two new per-search parameters: - qb: query quantization bits for MSE pre-screening (0 = float path, 1-8 = integer popcount path) - int_qjl: use integer popcount for the QJL sign-bit stage (requires qb > 0) IndexScalarQuantizer.h is `%include`d in swigfaiss.swig so SWIG wraps the struct at runtime. However, __init__.pyi (the authoritative Pyre type source since D107274305) was not updated when the struct was added. Any Pyre-checked code that uses IVFSQTurboQSearchParameters receives [missing-attribute] errors. Pattern matches the approach used in D107508282, which added SearchParametersHNSW and SearchParametersPQ for the same gap. Reviewed By: limqiying, mnorris11 Differential Revision: D108001955 fbshipit-source-id: 6cfbb0dddea1e09562b8185888c571aea229b747 |
||
|
Alibek Zhakubayev
|
34eb98936f |
fix(python): int64 coercions for MapLong2Long + InvertedLists DOWNCAST chain (#5257)
Summary: Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5257 Two Python binding correctness fixes. Both are silent data-corruption bugs that affect callers using standard numpy idioms. ## Fix 1: MapLong2Long missing int64 dtype coercions `handle_MapLong2Long` in `class_wrappers.py` passed `keys` and `vals` directly to `swig_ptr` without int64 coercion. `swig_ptr` returns the raw memory pointer of the numpy array, ignoring its dtype. The C++ `MapLong2Long::add` and `search_multiple` methods take `const int64_t*`, so when callers pass `int32` arrays (e.g., `np.arange(n)`, which defaults to int32 on most platforms), C++ reads every 8 bytes as one `int64_t`. For a 5-element int32 array, this reads 2 int64 values instead of 5, and each int64 is a bit-merge of two adjacent int32 values. The map is silently populated with completely wrong keys and values, with no exception or warning. This is the identical bug class as the `add_sa_codes` int64 coercion fix (D103384269, 2026-05-01), and the identical pattern omission. Every other ids-accepting wrapper in `class_wrappers.py` (`replacement_add_with_ids`, `replacement_remove_ids`, `replacement_permute_entries`) applies `np.ascontiguousarray(..., dtype='int64')` before `swig_ptr`. `handle_MapLong2Long` was added separately and missed this step. **Fix:** Add `np.ascontiguousarray(keys, dtype='int64')` and `np.ascontiguousarray(vals, dtype='int64')` in `replacement_map_add`, and `np.ascontiguousarray(keys, dtype='int64')` in `replacement_map_search_multiple`. ## Fix 2: InvertedLists* DOWNCAST chain missing ArrayInvertedListsPanorama and SliceInvertedLists The `%typemap(out) faiss::InvertedLists *` in `swigfaiss.swig` runs whenever a C++ function returns `faiss::InvertedLists*` (including `downcast_InvertedLists`). Before this fix, `ArrayInvertedListsPanorama` was absent from the DOWNCAST chain, so `dynamic_cast<faiss::ArrayInvertedLists*>` succeeded first (since `ArrayInvertedListsPanorama` IS-A `ArrayInvertedLists`) and the object was returned as `faiss.ArrayInvertedLists`, hiding all Panorama-specific fields (`cum_sums`, `n_levels`, `level_width`, `get_cum_sums()`). `SliceInvertedLists` fell through all entries to the base `DOWNCAST(InvertedLists)`, returning the base type. `ArrayInvertedListsPanorama` and `SliceInvertedLists` are both defined in `InvertedLists.h` (already `%include`d at line 551) and are fully exposed as Python types. The DOWNCAST entries were simply missing. **Fix:** Add `DOWNCAST(ArrayInvertedListsPanorama)` immediately before `DOWNCAST(ArrayInvertedLists)` (subclass must precede parent); add `DOWNCAST(SliceInvertedLists)` before the base `DOWNCAST(InvertedLists)` catch-all. `downcast_InvertedLists` now returns the correct subtype for both. Note: `%typemap(out)` applies to function return types, not raw member field access. Tests use `faiss.downcast_InvertedLists(index.invlists)`, not `index.invlists` directly — the same pattern as `faiss.downcast_index()` for the `Index*` chain. ## Tests 4 new tests in `TestMapLong2Long`: verify int32 keys/vals are stored and looked up correctly, verify int64 is unchanged, verify search_multiple with int32 keys, verify missing key returns -1. 2 new tests in `TestInvertedListsDowncast`: verify `downcast_InvertedLists` returns `ArrayInvertedListsPanorama` for a Panorama index (with Panorama-specific field access), and `SliceInvertedLists` after `replace_invlists` (with `i0`/`i1` field access). Reviewed By: junjieqi Differential Revision: D106074206 fbshipit-source-id: d0f59ac1e27c37add9c21927ce1dc9af9771d17b |
||
|
Alibek Zhakubayev
|
cb69d7c566 |
Fix Dq=None crash and np.empty nondeterminism in search_preassigned wrappers (#5221)
Summary: Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5221 ## TL;DR Two related bugs in the Python wrappers for `search_preassigned` and `range_search_preassigned`: passing `Dq=None` crashes with `ValueError`, and `np.empty` in `ivf_tools.range_search_preassigned` produces nondeterministic results for IVFPQ with `by_residual=True`. We fix both by substituting `np.zeros` when the caller omits coarse distances. ## Bug 1: Dq=None crashes (class_wrappers.py, 4 sites) The Python wrappers for `search_preassigned` and `range_search_preassigned` on both `Index` and `IndexBinaryIVF` document `Dq` as optional, but passing `None` crashes: ``` ValueError: argument is not a numpy array ``` The cause: each wrapper validates `Dq` inside `if Dq is not None:` but has no `else` branch. When `Dq` is `None`, `swig_ptr(Dq)` is called with `None`, which SWIG rejects. ## Bug 2: np.empty nondeterminism (ivf_tools.py) `ivf_tools.range_search_preassigned` allocates a dummy `coarse_dis` with `np.empty` when the caller does not provide one. `np.empty` returns uninitialized memory. For `IndexIVFPQ` with L2 metric and `by_residual=True` (the default), the C++ search kernel reads `coarse_dis` for distance correction. Uninitialized values produce nondeterministic search results with no error or warning. The sibling function `ivf_tools.search_preassigned` already uses `np.zeros` for the same purpose. The `range_search_preassigned` variant diverged. ## Fix **class_wrappers.py (4 sites):** Add `else: Dq = np.zeros(Iq.shape, dtype=DTYPE)` after each `if Dq is not None:` block. Float wrappers use `dtype='float32'`, binary wrappers use `dtype='int32'`. `Iq` is already coerced and shape-validated at each site, so the shape is correct. **ivf_tools.py (1 site):** Change `np.empty` to `np.zeros` at line 75. Restores parity with `search_preassigned` at line 53. ## Why zeros `Dq` represents the distances from queries to their assigned IVF centroids. Zeros means "no distance correction." For most index types (`IVF,Flat`, `IVF,SQ8`, `IndexBinaryIVF`), the C++ code ignores `Dq` entirely in scoring. For `IVFPQ` with `by_residual=True`, zeros omit the coarse component from the returned distances but preserve the neighbor ranking (because all vectors in the same inverted list share the same coarse distance, so it cancels in relative comparisons). Users who need accurate absolute distances should compute and pass real `Dq` via `index.quantizer.search()`. Zeros is deterministic, semantically neutral, and consistent with the existing convention in `ivf_tools.search_preassigned`. Reviewed By: junjieqi Differential Revision: D105299681 fbshipit-source-id: 9f59a7bd839b253ee31d42c1ed013234a7adfd87 |
||
|
Alibek Zhakubayev
|
9942229393 |
Fix add_sa_codes silently accepting non-int64 ids, corrupting stored labels (#5171)
Summary: Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5171 **Problem 1 — add_sa_codes silently stores wrong ids when passed non-int64 input** replacement_add_sa_codes in class_wrappers.py passed the ids array directly to swig_ptr without dtype coercion. The C++ signature is add_sa_codes(idx_t n, const uint8_t* codes, const idx_t* xids) where idx_t = int64_t (8 bytes per element). When the caller passed int32 ids — the default output of np.arange(n) on most platforms — swig_ptr returned a pointer to 4-byte data. The C++ layer then read every 8 consecutive bytes as one int64, producing garbage labels for every stored vector. No exception was raised, no warning was emitted. This is inconsistent with every other ids-accepting wrapper in the same file. replacement_add_with_ids, replacement_remove_ids, and replacement_permute_entries all call np.ascontiguousarray(ids, dtype='int64') before swig_ptr. add_sa_codes was the only exception. Callers using np.arange(n) without an explicit dtype — common in tutorials and quick-start code — silently ingested a corrupt index. Fix: add ids = np.ascontiguousarray(ids, dtype='int64') before swig_ptr(ids), matching the pattern used by all other ids-accepting wrappers. The call is a no-op for callers already passing int64, so there is no behavior change for correct inputs. **Problem 2 — sa_encode calls sa_code_size() twice (one redundant SWIG dispatch)** replacement_sa_encode called self.sa_code_size() once in the if codes is None branch and once in the else branch. Since sa_code_size() is a virtual C++ method accessed via SWIG, this is one avoidable cross-boundary call per sa_encode invocation. The value is a pure function of index state and cannot change within a call. replacement_sa_decode in the same block already uses the correct pattern, reusing the code size from codes.shape. The inconsistency between sa_encode and sa_decode confirms this was an oversight. Fix: hoist self.sa_code_size() into code_size = self.sa_code_size() once before the branch, matching the pattern in sa_decode. Reviewed By: limqiying Differential Revision: D103384269 fbshipit-source-id: 3f28a18b7e951c24d23db676be475bb6b298df2f |
||
|
Andrejs Krasilnikovs
|
db5aef2253 |
Strengthen roundtrip tests by doing serialization after deserialization (#4851)
Summary: Pull Request resolved: https://github.com/facebookresearch/faiss/pull/4851 Roundtrip tests only test serialize -> deserialize. They should also test the deserialize -> serialize sequence. Reviewed By: mdouze Differential Revision: D93678044 fbshipit-source-id: 1503b5b954d49df509d9f785d35911ade989290e |
||
|
Alibek Zhakubayev
|
1deba7b90f |
Add attribute validation to prevent silent failures in SWIG wrappers (#4583)
Summary: Pull Request resolved: https://github.com/facebookresearch/faiss/pull/4583 This diff implements attribute validation in Faiss Python wrappers to prevent silent failures when setting non-existent attributes on SWIG-wrapped objects. Previously, attempting to set invalid attributes would fail silently, leading to bugs that were difficult to detect. ## Key Changes **Core Implementation:** - Added `replacement_setattr` method in `class_wrappers.py` that validates attribute existence before setting - Raises `AttributeError` with descriptive messages for invalid attributes - Preserves normal behavior for valid attributes and delegation patterns **Bug Fixes Discovered and Resolved:** The validation implementation successfully caught and helped fix **6 real bugs** in the codebase: * **test_fast_scan.py** : - Removed invalid `index.nprobe = 16` on `IndexResidualQuantizer` (RQ indices don't support nprobe) - Similar fix in subtest_from_idxaq method * **test_meta_index.py**: - Fixed `IndexShards.threaded` attribute - `threaded` is a construction-time parameter, not runtime settable - Solution: Create separate threaded/non-threaded instances instead of toggling at runtime * **test_refine.py**: - Fixed `IndexRefine.nprobe` access - nprobe belongs to the base index, not the refine wrapper - Solution: Use `index.base_index.nprobe = 4` instead of direct access * **test_graph_based.py**: - Fixed `IndexHNSWFlat.search_bounded_queue` access - belongs to hnsw sub-object - Solution: Use `index.hnsw.search_bounded_queue = False` instead of direct access * **test_index_accuracy.py**: - Fixed typo: `index.npobe = 16` → `index.nprobe = 16` - This typo caused search quality degradation (intersection count improved from ~460 to ~945 after fix) * **xri_algo/lyra/features2d/match_faiss.py**: - Added conditional check before setting nprobe: `if hasattr(self.index, 'nprobe'): self.index.nprobe = nprobe` - Prevents AttributeError on flat indices like IndexFlatL2 that don't support nprobe ## Impact - **Improved Code Quality**: Converts silent failures into explicit errors with clear messages - **Better Developer Experience**: Developers now get immediate feedback for invalid attribute usage - **Prevents Future Bugs**: Makes it impossible to silently set non-existent attributes - **Production Ready**: All tests pass, demonstrating the validation works without breaking existing functionality The implementation has proven its value by catching multiple real bugs that were previously going unnoticed, including a performance-affecting typo that significantly degraded search accuracy. Reviewed By: subhadeepkaran Differential Revision: D82774442 fbshipit-source-id: 4cf7ff9dff2bc934abe6bbb04a6d1601f311890f |
||
|
Michael Norris
|
eff0898a13 |
Enable linting: lint config changes plus arc lint command (#3966)
Summary: Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3966 This actually enables the linting. Manual changes: - tools/arcanist/lint/fbsource-licenselint-config.toml - tools/arcanist/lint/fbsource-lint-engine.toml Automated changes: `arc lint --apply-patches --take LICENSELINT --paths-cmd 'hg files faiss'` Reviewed By: asadoughi Differential Revision: D64484165 fbshipit-source-id: 4f2f6e953c94ef6ebfea8a5ae035ccfbea65ed04 |
||
|
Matthijs Douze
|
52cf9af18a |
group SWIG tests into one file (#3807)
Summary: Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3807 This diff re-organizes the tests a bit: * groups tests related to SWIG into a single file * enable doxygen test conditionally * removes a dep on platform.Version that does not exist on some envs * move a few tests out of build_blocks to avoid it being a catch-all file for uncategorized tests Reviewed By: asadoughi Differential Revision: D61959750 fbshipit-source-id: fb6ada5b5a980ac07088254183d80b80ddfb1f45 |