Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5349
# this is a no-op change
Ran `black -l 80` over all tracked Python files under `fbcode/faiss/`, using line length 80 to match faiss's documented coding style (`CONTRIBUTING.md`: "80 character line length (both for C++ and Python)"). 172 files were reformatted; 28 were already compliant.
This is a pure-formatting change. `black` runs its AST-equivalence safety check on every file, so no identifiers, string/numeric values, operators, or control flow were altered — only whitespace, indentation, line wrapping, blank lines, trailing commas, and string-quote normalization (single to double quotes).
`fbcode/faiss/` is intentionally excluded from Meta's standard Python autoformatter (`arc f` / `arc lint -a` report no formatter for these paths) because it mirrors to public GitHub, so the Python had drifted out of any consistent style. The C++ is already clang-format-clean (0 of 807 files change), so this effort is Python-only. This diff brings the Python to canonical `black` formatting.
Excluded `demos/index_pq_flat_separate_codes_from_codebook.py`: it contains a Jupyter `!rm` shell-magic line and is not valid Python, so `black` cannot parse it.
This is the base of a 3-diff stack: (1) this black reformat; (2) `.flake8` alignment (max-line-length 80 + black-compatible ignores E203/W503); (3) genuine lint fixes (F401/F403/F841/B011).
Reviewed By: mnorris11
Differential Revision: D109736099
fbshipit-source-id: 9a054635766ed11aa3760a788395babd3d198abf
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5134
Expand DD test coverage by applying for_all_simd_levels to existing test
classes that exercise DD-dispatched code paths but were previously pinned
to a single SIMD level.
This is the "mega decorator" diff from the DD test coverage gaps plan --
pure decorator additions, no new test logic. Follow-up diffs add new
test files and numerical cross-level assertions for gaps the decorator
alone cannot close.
Classes decorated (grouped by area):
* Binary Hamming non-IVF: TestRange and TestKnn in
test_binary_hashindex.py; TestBinarySearchParams in
test_binary_search_params.py; TestIndexBinaryFromFloat in
test_index_binary_from_float.py; TestSpectralHash in
test_index_accuracy.py.
* IVFPQ / search: EvalIVFPQAccuracy in test_index.py;
TestSelector and TestSearchParams in test_search_params.py.
* Flat / refine / Panorama: TestIndexFlat, TestIndexFlatL2,
TestIndexFlatL2Panorama, TestScalarQuantizer in test_index.py;
TestDistanceComputer, TestIndexRefineSearchParams,
TestIndexRefineRangeSearch in test_refine.py;
TestIndexRefinePanorama, TestIndexFlatPanorama,
TestIndexHNSWFlatPanorama, TestIndexIVFFlatPanorama in their
respective files.
* Quantizer encode: TestResidualQuantizer,
TestIndexResidualQuantizerSearch in test_residual_quantizer.py;
TestComponents, TestLocalSearchQuantizer in
test_local_search_quantizer.py.
* Fast scan: TestFastScanFiltering, TestBlockSkipConsistency,
TestFastScanRangeSearchFilter in test_fastscan_filter.py.
* Broader index tests (search-exercising classes only):
TestParameterSpace in test_autotune.py; TestSpectralHash in
test_factory.py; TestMerge1, TestMerge2, TestRemoveFastScan in
test_merge_index.py.
BUCK changes move decorated tests from the legacy-listing lists to the
simd_levels lists (which use supports_static_listing = False, required
for dynamic class name generation), and add the new entries
test_binary_search_params, test_fastscan_filter, test_refine_panorama,
test_hnsw_panorama.
___
overriding_review_checks_triggers_an_audit_and_retroactive_review
Oncall Short Name: fair_umami_cluster
Differential Revision: D101822335
fbshipit-source-id: 9a432c6d6bee201c3731713226405a8c8ecebbe6
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/4523
This upgrades OSS Faiss to numpy 2.3.3. I tried to fix issues based on https://numpy.org/devdocs/numpy_2_0_migration_guide.html.
Overall, numpy2 causes some test issues on various platforms/optimization levels.
Test changes that are OK:
- test_contrib.py: test_float: changing to use item(): this is just the new way of getting the value, like recommended in the migration guide.
Test changes that indicate some problem:
- test_contrib.py: test_ivf_train_2level: This fails on many platforms (avx2, conda installs). I could not find what the issue was. I had to increase the bounds slightly.
- test_index_accuracy.py: test_OIVFPQ: This fails just on AVX512. I had to increase the iterations from 10 to 12.
- test_local_search_quantizer.py and test_standalone_codec.py: Some of these break on Windows. Others needed lighter tolerances.
Debugging attempts: Especially for test_ivf_train_2level, I tried to unsuccessfully track down what might have changed. I suspected the type promotion change from numpy1 to 2.
```
- np.float32(3) + 3. now returns a float32 when it previously returned a float64.
- np.array([3], dtype=np.float32) + np.float64(3) will now return a float64 array. (The higher precision of the scalar is not ignored.)
```
I changed all datasets to be explicitly float64s, and checked for any constants in the code path. No luck.
Devmate was less than useless, it was actively a waste of time (as usual...)
Reviewed By: subhadeepkaran
Differential Revision: D79900113
fbshipit-source-id: 541cfa6f1fa1d60c84d855ce71f0e1de7ea7c7f1
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3452
Delete all remaining print within the Tests to improve the readability and effectiveness of the codebase.
Reviewed By: junjieqi
Differential Revision: D57466393
fbshipit-source-id: 6ebd66ae2e769894d810d4ba7a5f69fc865b797d
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/2650
LocalSearchQuantizer::icm_encode_step was optimized:
* The loop orders were changes
* Memory allocations were optimized
* Different order for accessing 'binaries' tensor, because it is symmetric
* SIMD for the lookup of the best code.
Results are unchanged.
Also, fixes incorrect test in test_local_search_quantizer.py
Reviewed By: mdouze
Differential Revision: D42352124
fbshipit-source-id: bf7e349f2123e6ee99e0776cf15ad5fc1cf2439a
Summary:
Work in progress.
This PR is going to implement the following search methods for ProductAdditiveQuantizer, including index factory and I/O:
- [x] IndexProductAdditiveQuantizer
- [x] IndexIVFProductAdditiveQuantizer
- [x] IndexProductAdditiveQuantizerFastScan
- [x] IndexIVFProductAdditiveQuantizerFastScan
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/2336
Test Plan:
buck test //faiss/tests/:test_fast_scan
buck test //faiss/tests/:test_fast_scan_ivf
buck test //faiss/tests/:test_local_search_quantizer
buck test //faiss/tests/:test_residual_quantizer
Reviewed By: alexanderguzhva
Differential Revision: D37172745
Pulled By: mdouze
fbshipit-source-id: 6ff18bfc462525478c90cd42e21805ab8605bd0f
Summary:
This diff added ProductAdditiveQuantizer.
A Simple Algo description:
1. Divide the vector space into several orthogonal sub-spaces, just like PQ does.
2. Quantize each sub-space by an independent additive quantizer.
Usage:
Construct a ProductAdditiveQuantizer object:
- `d`: dimensionality of the input vectors
- `nsplits`: number of sub-spaces divided into
- `Msub`: `M` of each additive quantizer
- `nbits`: `nbits` of each additive quantizer
```python
d = 128
nsplits = 2
Msub = 4
nbits = 8
plsq = faiss.ProductLocalSearchQuantizer(d, nsplits, Msub, nbits)
prq = faiss.ProductResidualQuantizer(d, nsplits, Msub, nbits)
```
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/2286
Test Plan:
```
buck test //faiss/tests/:test_local_search_quantizer -- TestProductLocalSearchQuantizer
buck test //faiss/tests/:test_residual_quantizer -- TestProductResidualQuantizer
```
Reviewed By: alexanderguzhva
Differential Revision: D35907702
Pulled By: mdouze
fbshipit-source-id: 7428a196e6bd323569caa585c57281dd70e547b1
Tia Lim