Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5549
`IndexBinaryFromFloat::search()` converted the wrapped float index's distances back to
Hamming distances with a hardcoded `round(df[i] / 4.0)`, which is only correct for
`METRIC_L2`. For any other metric the returned distances were silently wrong.
`binary_to_real()` maps bits to `{-1, +1}`, so the relation between the float distance and
the Hamming distance depends on the metric:
- `METRIC_L2` (squared): `float_dist = 4 * hamming`
- `METRIC_INNER_PRODUCT`: `float_dist = d - 2 * hamming`
- `METRIC_L1`: `float_dist = 2 * hamming`
Branch on `index->metric_type` and apply the matching inverse. `METRIC_L2` remains the
default/fallback, so existing behavior is unchanged. Each mapping is monotonic in the
Hamming distance, so the result order the wrapped index produces is already correct and
needs no reordering.
Reported in https://github.com/facebookresearch/faiss/issues/5546.
___
Differential Revision: D118160087
fbshipit-source-id: f2ce2b97ebfd47461346508f9878b1af7e321b12
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5349
# this is a no-op change
Ran `black -l 80` over all tracked Python files under `fbcode/faiss/`, using line length 80 to match faiss's documented coding style (`CONTRIBUTING.md`: "80 character line length (both for C++ and Python)"). 172 files were reformatted; 28 were already compliant.
This is a pure-formatting change. `black` runs its AST-equivalence safety check on every file, so no identifiers, string/numeric values, operators, or control flow were altered — only whitespace, indentation, line wrapping, blank lines, trailing commas, and string-quote normalization (single to double quotes).
`fbcode/faiss/` is intentionally excluded from Meta's standard Python autoformatter (`arc f` / `arc lint -a` report no formatter for these paths) because it mirrors to public GitHub, so the Python had drifted out of any consistent style. The C++ is already clang-format-clean (0 of 807 files change), so this effort is Python-only. This diff brings the Python to canonical `black` formatting.
Excluded `demos/index_pq_flat_separate_codes_from_codebook.py`: it contains a Jupyter `!rm` shell-magic line and is not valid Python, so `black` cannot parse it.
This is the base of a 3-diff stack: (1) this black reformat; (2) `.flake8` alignment (max-line-length 80 + black-compatible ignores E203/W503); (3) genuine lint fixes (F401/F403/F841/B011).
Reviewed By: mnorris11
Differential Revision: D109736099
fbshipit-source-id: 9a054635766ed11aa3760a788395babd3d198abf
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5134
Expand DD test coverage by applying for_all_simd_levels to existing test
classes that exercise DD-dispatched code paths but were previously pinned
to a single SIMD level.
This is the "mega decorator" diff from the DD test coverage gaps plan --
pure decorator additions, no new test logic. Follow-up diffs add new
test files and numerical cross-level assertions for gaps the decorator
alone cannot close.
Classes decorated (grouped by area):
* Binary Hamming non-IVF: TestRange and TestKnn in
test_binary_hashindex.py; TestBinarySearchParams in
test_binary_search_params.py; TestIndexBinaryFromFloat in
test_index_binary_from_float.py; TestSpectralHash in
test_index_accuracy.py.
* IVFPQ / search: EvalIVFPQAccuracy in test_index.py;
TestSelector and TestSearchParams in test_search_params.py.
* Flat / refine / Panorama: TestIndexFlat, TestIndexFlatL2,
TestIndexFlatL2Panorama, TestScalarQuantizer in test_index.py;
TestDistanceComputer, TestIndexRefineSearchParams,
TestIndexRefineRangeSearch in test_refine.py;
TestIndexRefinePanorama, TestIndexFlatPanorama,
TestIndexHNSWFlatPanorama, TestIndexIVFFlatPanorama in their
respective files.
* Quantizer encode: TestResidualQuantizer,
TestIndexResidualQuantizerSearch in test_residual_quantizer.py;
TestComponents, TestLocalSearchQuantizer in
test_local_search_quantizer.py.
* Fast scan: TestFastScanFiltering, TestBlockSkipConsistency,
TestFastScanRangeSearchFilter in test_fastscan_filter.py.
* Broader index tests (search-exercising classes only):
TestParameterSpace in test_autotune.py; TestSpectralHash in
test_factory.py; TestMerge1, TestMerge2, TestRemoveFastScan in
test_merge_index.py.
BUCK changes move decorated tests from the legacy-listing lists to the
simd_levels lists (which use supports_static_listing = False, required
for dynamic class name generation), and add the new entries
test_binary_search_params, test_fastscan_filter, test_refine_panorama,
test_hnsw_panorama.
___
overriding_review_checks_triggers_an_audit_and_retroactive_review
Oncall Short Name: fair_umami_cluster
Differential Revision: D101822335
fbshipit-source-id: 9a432c6d6bee201c3731713226405a8c8ecebbe6
Summary: A test was timing out but the culprit was not the functionality being tested but instead a very slow list comprehension. Also relaxed the test very slightly as it failed from time to time.
Reviewed By: wickedfoo
Differential Revision: D26727507
fbshipit-source-id: 5b3352674fbef1f0cb6155452e4a93adc631d6a7
Changelog:
- changed license: BSD+Patents -> MIT
- propagates exceptions raised in sub-indexes of IndexShards and IndexReplicas
- support for searching several inverted lists in parallel (parallel_mode != 0)
- better support for PQ codes where nbit != 8 or 16
- IVFSpectralHash implementation: spectral hash codes inside an IVF
- 6-bit per component scalar quantizer (4 and 8 bit were already supported)
- combinations of inverted lists: HStackInvertedLists and VStackInvertedLists
- configurable number of threads for OnDiskInvertedLists prefetching (including 0=no prefetch)
- more test and demo code compatible with Python 3 (print with parentheses)
- refactored benchmark code: data loading is now in a single file
Features:
- automatic tracking of C++ references in Python
- non-intel platforms supported -- some functions optimized for ARM
- override nprobe for concurrent searches
- support for floating-point quantizers in binary indexes
Bug fixes:
- no more segfaults in python (I know it's the same as the first feature but it's important!)
- fix GpuIndexIVFFlat issues for float32 with 64 / 128 dims
- fix sharding of flat indexes on GPU with index_cpu_to_gpu_multiple
Trang Nguyen