Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5361
`test_batch_boundaries` flaked (~36%) on the `AVX512_SPR` dispatch variant. The reference `IndexFlat` distance kernel is SIMD-dispatched, so its float reduction order changes with the runtime SIMD level (wide FMA under `AVX512_SPR`), while the `IndexFlatPanorama` kernel is compiled once at baseline and uses the algebraically-equivalent `a*a + b*b - 2ab` decomposition instead of `(a-b)^2`. Both are exact but round differently, so near-tied neighbors reorder at batch boundaries. The earlier scalar de-flake (D109488316 / D109488320, T276491016) only widened the `atol`/`otol` tolerance for the `_NONE` gap, leaving the larger `AVX512_SPR` gap.
Fix: replace the brittle exact-id overlap-rate gate in `assert_search_results_equal` with the tie-aware comparator `faiss.contrib.evaluation.check_ref_knn_with_draws`, which requires distances to match within `rtol` and ids to agree only as sets within each distance tie-group. This is variant-agnostic (covers `NONE` / `AVX2` / `AVX512` / `AVX512_SPR`) and is the standard faiss kNN comparison helper used across the suite.
Reviewed By: mnorris11
Differential Revision: D110092203
fbshipit-source-id: ec6fc8c6114d7dc9bb7b6911478a6d5b0b993b5c
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5349
# this is a no-op change
Ran `black -l 80` over all tracked Python files under `fbcode/faiss/`, using line length 80 to match faiss's documented coding style (`CONTRIBUTING.md`: "80 character line length (both for C++ and Python)"). 172 files were reformatted; 28 were already compliant.
This is a pure-formatting change. `black` runs its AST-equivalence safety check on every file, so no identifiers, string/numeric values, operators, or control flow were altered — only whitespace, indentation, line wrapping, blank lines, trailing commas, and string-quote normalization (single to double quotes).
`fbcode/faiss/` is intentionally excluded from Meta's standard Python autoformatter (`arc f` / `arc lint -a` report no formatter for these paths) because it mirrors to public GitHub, so the Python had drifted out of any consistent style. The C++ is already clang-format-clean (0 of 807 files change), so this effort is Python-only. This diff brings the Python to canonical `black` formatting.
Excluded `demos/index_pq_flat_separate_codes_from_codebook.py`: it contains a Jupyter `!rm` shell-magic line and is not valid Python, so `black` cannot parse it.
This is the base of a 3-diff stack: (1) this black reformat; (2) `.flake8` alignment (max-line-length 80 + black-compatible ignores E203/W503); (3) genuine lint fixes (F401/F403/F841/B011).
Reviewed By: mnorris11
Differential Revision: D109736099
fbshipit-source-id: 9a054635766ed11aa3760a788395babd3d198abf
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5339
These three faiss CPU test modules were filed as flaky by the test-health tracker. Root causes and fixes (test-only):
- `test_flat_panorama.py::test_batch_boundaries`: the query matrix `xq` was drawn from an unseeded global NumPy RNG (the existing `np.random.seed(987)` only governs the database `xb`), so near-tied neighbors occasionally reordered between `IndexFlat` and `IndexFlatPanorama` and breached the tight comparison. Seed the query draw, and loosen the distance `atol` in `assert_search_results_equal` from `1e-7` to `1e-4` (the two distance formulas `(a-b)^2` vs `a*a+b*b-2ab` differ at the float32 ULP level).
- `test_graph_based.py`: `test_hnsw_flat_IP`, `test_nsg_pq`, and `test_nsg_sq` asserted recall/agreement floors only a few counts above the observed values, so OpenMP-parallel graph-build nondeterminism intermittently breached them. Loosened each floor with documented ~5-10% headroom; these still collapse by tens-to-hundreds of points on a real regression.
- `test_hamming_utils.py` (`BUCK` only): `for_all_simd_levels` mutates the process-global `faiss.SIMDConfig` level in `setUp`/`tearDown`; concurrent SIMD-level variant cases within one binary could stomp it. Gave `test_hamming_utils` a dedicated `python_unittest` target carrying `tpx_labels.serialize_test_cases` so its cases run one at a time. No source change to the test itself.
___
Differential Revision: D109488320
fbshipit-source-id: b9ec2ab6547fc4424072588558409de62f3dac01
Summary:
In the current code you can segfault Panorama indexes by loading a dataset with 960 dims and setting `nlevels` to 128.
In the case where `nlevels` does not evenly divide `d`, Panorama currently calculates `level_width = ceil(d / nlevels)`. We cannot change this behavior due to backwards compatibility. However, for our case of 960 dims and 128 levels, this would mean each level is 8 dims. Thus, the last 8 levels have 0 dims each. Since we calculate `actual_level_width` using `size_t` params, it overflows and produces 8 when it should produce 0, leading to out-of-bounds memory accesses.
The fix is to simply truncate `nlevels` in the `Panorama` object when this would happen. A debug log is emitted to the user when this happens. In the aforementioned example, the user would see `WARNING truncating nlevels from 128 to 120`.
This PR also fixes a similar latent bug in `IndexFlatPanorama::reconstruct` where we would perform out-of-bounds memory accesses when `nlevels` doesn't evenly divide `d`.
Existing test cases are strengthened to ensure bugs are fixed and to avoid any future regressions.
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5200
Reviewed By: junjieqi
Differential Revision: D104764705
Pulled By: mnorris11
fbshipit-source-id: e55577b8ce5508516ee5dd4758f31acb4a28179a
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5134
Expand DD test coverage by applying for_all_simd_levels to existing test
classes that exercise DD-dispatched code paths but were previously pinned
to a single SIMD level.
This is the "mega decorator" diff from the DD test coverage gaps plan --
pure decorator additions, no new test logic. Follow-up diffs add new
test files and numerical cross-level assertions for gaps the decorator
alone cannot close.
Classes decorated (grouped by area):
* Binary Hamming non-IVF: TestRange and TestKnn in
test_binary_hashindex.py; TestBinarySearchParams in
test_binary_search_params.py; TestIndexBinaryFromFloat in
test_index_binary_from_float.py; TestSpectralHash in
test_index_accuracy.py.
* IVFPQ / search: EvalIVFPQAccuracy in test_index.py;
TestSelector and TestSearchParams in test_search_params.py.
* Flat / refine / Panorama: TestIndexFlat, TestIndexFlatL2,
TestIndexFlatL2Panorama, TestScalarQuantizer in test_index.py;
TestDistanceComputer, TestIndexRefineSearchParams,
TestIndexRefineRangeSearch in test_refine.py;
TestIndexRefinePanorama, TestIndexFlatPanorama,
TestIndexHNSWFlatPanorama, TestIndexIVFFlatPanorama in their
respective files.
* Quantizer encode: TestResidualQuantizer,
TestIndexResidualQuantizerSearch in test_residual_quantizer.py;
TestComponents, TestLocalSearchQuantizer in
test_local_search_quantizer.py.
* Fast scan: TestFastScanFiltering, TestBlockSkipConsistency,
TestFastScanRangeSearchFilter in test_fastscan_filter.py.
* Broader index tests (search-exercising classes only):
TestParameterSpace in test_autotune.py; TestSpectralHash in
test_factory.py; TestMerge1, TestMerge2, TestRemoveFastScan in
test_merge_index.py.
BUCK changes move decorated tests from the legacy-listing lists to the
simd_levels lists (which use supports_static_listing = False, required
for dynamic class name generation), and add the new entries
test_binary_search_params, test_fastscan_filter, test_refine_panorama,
test_hnsw_panorama.
___
overriding_review_checks_triggers_an_audit_and_retroactive_review
Oncall Short Name: fair_umami_cluster
Differential Revision: D101822335
fbshipit-source-id: 9a432c6d6bee201c3731713226405a8c8ecebbe6
Tia Lim