facebookresearch/faiss
1
0
Fork 0
mirror of https://github.com/facebookresearch/faiss.git synced 2026-10-11 22:50:00 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,354 Commits 267 Branches 33 Tags
main
Commit Graph
10 Commits
Author SHA1 Message Date
Michael Norris b5a14632a6 Run the eight SIMD kernel tests that no build executed (#5620) 
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5620

**TL;DR:** Eight SIMD kernel tests never ran, and two byte-domain kernels trust their caller. This change makes three fixes:

1. **Six tests never compiled.** `test_rabitq_simd.cpp` hides them behind `COMPILE_SIMD_*` macros that the faiss target does not export.
2. **Two tests skipped themselves in a static build.** `TestSQByteDirectAcrossSIMDLevels` needs `SIMDLevel::NONE` as a reference, and a static build does not report that level.
3. **The byte kernels never test their dimension requirement.** `DistanceComputerByte` and `DistanceComputerByteSigned` read 16 bytes per loop step.

## 1. Six tests never compiled

`test_rabitq_simd.cpp` wraps two blocks in `#ifdef COMPILE_SIMD_AVX2` and `#ifdef COMPILE_SIMD_AVX512`. The faiss target sets those macros as `preprocessor_flags`, and it does not export them. A test translation unit never sees them. So the preprocessor deleted both blocks. The file holds nine tests and ran three.

The AVX2 block loses its guard, because every x86 faiss build holds the AVX2 kernels. Its three tests also lose an `#ifdef FAISS_ENABLE_DD` around a runtime skip, which the same problem disabled. `is_simd_level_available()` reports the right answer in a static build too.

The AVX512 block moves to a dynamic dispatch target. A static x86 build compiles the AVX2 kernels only, so the test fails to link there rather than skips. `test_rabitq_simd_vpopcnt.cpp` becomes `test_rabitq_simd_dd.cpp`, because it now holds every RaBitQ test that a static build cannot link. A new `test_rabitq_simd_util.h` holds the scalar reference that both files call.

## 2. Two tests skipped themselves in a static build

`TestSQByteDirectAcrossSIMDLevels` measured the index a second time at `SIMDLevel::NONE`, then compared the two runs. A static build reports only the level it compiles, and omits `NONE`, so both tests skipped.

The tests now compute the distances from the definition. The distances are integers, so the reference is exact. The tests check absolute values now, instead of the agreement of two faiss paths.

## 3. The byte kernels trust the caller

`DistanceComputerByte` and `DistanceComputerByteSigned` read 16 bytes per loop step, so they require `d % 16 == 0`. `sq-dispatch.h` enforces that, and no other caller builds them.

A caller that ignores the requirement reads up to 15 bytes past each code and returns a wrong distance. Each constructor now tests the requirement and throws. The AVX2 and the ARM_NEON specializations both get the test.

Reviewed By: alibeklfc

Differential Revision: D119545379

fbshipit-source-id: ddfdcc30d3662f469b2ec3dbb9d5b9f4dafa21aa
 2026-09-10 20:05:56 -07:00
Michael Norris fc13c81fb8 Stop aarch64 falling back to SIMDLevel::NONE in faiss dispatch (#5572) 
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5572

**TL;DR:** Stops aarch64 running scalar code where a NEON or SVE kernel exists, adds the two missing NEON byte-domain kernels, and fixes `FAISS_SIMD_LEVEL` falling all the way to `NONE` on an uncompiled level.

On aarch64 several faiss dispatch paths ran scalar code, or ran an `ARM_NEON` kernel where an `ARM_SVE` kernel already existed. This change enforces the order SVE -> NEON -> NONE on an ARM host. It also adds the one kernel pair that was missing.

T287037898 reported the scalar-quantizer half. On aarch64 `IndexScalarQuantizer` with `QT_8bit_direct` or `QT_8bit_direct_signed` saved memory but lost throughput. `Refine(SQ8)` has the same problem, because `IndexRefine::search` calls `refine_index->get_distance_computer()`.

## What each call site gets

| Call site | Before | After |
| --- | --- | --- |
| `QT_8bit_direct` on aarch64 | float-domain `DCTemplate` | NEON `DistanceComputerByte` |
| `QT_8bit_direct_signed` on aarch64 | float-domain `DCTemplate` | NEON `DistanceComputerByteSigned` (new) |
| `IndexFlat` distance computers, SVE host | `ARM_NEON` | `ARM_SVE` |
| `AdditiveQuantizer::compute_centroid_norms`, SVE host | `ARM_NEON` | `ARM_SVE` |
| `SuperKMeans` `block_l2`, SVE host | `ARM_NEON` | `ARM_SVE` |
| `pq_code_distance` wrappers, SVE host | `ARM_NEON` | `ARM_SVE` |
| `with_VectorDistance`, SVE host | `ARM_NEON` | `ARM_SVE` |
| `FAISS_SIMD_LEVEL=ARM_SVE`, build without SVE | `NONE` | nearest compiled level |

## Three causes, fixed separately

**1. Two missing kernels.** `sq-neon.cpp` held a scalar `DistanceComputerByte<Sim, ARM_NEON>` that nothing could reach, and no `DistanceComputerByteSigned<Sim, ARM_NEON>` at all. Both are now real NEON kernels. L2 and the unsigned inner product use `vabdq_u8`, `vmull_u8` and `vpadalq_u16`. The bias-encoded inner product uses `veorq_u8`, `vmull_s8` and `vpadalq_s16`. Both agree bit for bit with the AVX2 specializations, which the tests require.

Two invariants deserve a note:

- The L2 path keeps an unsigned accumulator. A `vmull_u8` square reaches 255^2 = 65025, which an int16 lane would read as negative.
- For x in 0 to 255, `x ^ 0x80` read as `int8` is exactly `x - 128`. That is how the kernel removes the +128 bias before `vmull_s8`.

**2. Dispatch chains that list only x86 levels.** The `if constexpr` chains in `sq-dispatch.h` enumerated x86 levels only, so an ARM host fell through to the float path. This adds `ARM_NEON` to four chains: two in `select_distance_computer_body`, and two in `sq_select_InvertedListScanner`. The chain in `is_dimension_compatible` already included ARM.

This does not add `ARM_SVE`. The scalar-quantizer entry points dispatch with a mask that holds no `ARM_SVE` bit, so an SVE host already falls through to the `ARM_NEON` case and now gets these kernels. Adding `ARM_SVE` would instantiate the empty primary template in `distance_computers.h`. This corrects that template's stale comment.

**3. Level masks that hide existing SVE kernels.** The default mask holds no `ARM_SVE` bit, so the dispatch fell through `case ARM_SVE` to `ARM_NEON`. This uses `with_simd_level_a1` at every site where a real SVE kernel exists and links: `IndexFlat.cpp` at four sites, `AdditiveQuantizer::compute_centroid_norms`, `block_l2` in `SuperKMeans.cpp`, all three wrappers in `pq_code_distance-generic.cpp`, and `with_VectorDistance` in `distances_dispatch.h`.

The `IndexFlat` sites matter most. `faiss::fvec_L2sqr()` already used the SVE mask, so on an SVE host the free function ran SVE while `IndexFlatL2`'s distance computer ran NEON.

Also: `FAISS_SIMD_LEVEL=ARM_SVE` on a build without SVE compiled in used to skip every level and run at `NONE`, because `with_selected_simd_levels` has no case label for an uncompiled level. It now walks down to the nearest compiled level. The override is still honoured when the CPU lacks the level, since forcing a level is the point of it. Only uncompiled levels are corrected.

The unused `AVAILABLE_SIMD_LEVELS_A2` is deleted. It is the NEON-to-NONE trap in constant form, with no users.

This change keeps the existing mask names. A follow-up renames them to say what they hold.

## Out of scope

In rough order of remaining value:

- `IndexPQ.cpp` and `IndexIVFPQ.cpp` still pin PQ to `ARM_NEON` on an SVE host. A mask change is not enough. `pq_code_distance-sve.cpp` includes only `pq_scan_impl.h`, and `with_HammingComputer<ARM_SVE>` has no complete type.
- `distances_aarch64.cpp` forwards five `ARM_NEON` specializations to `<SIMDLevel::NONE>`, so a non-SVE aarch64 host runs scalar code in the IVFFlat scan.
- `rabitq_neon.cpp` forwards all six `ARM_NEON` specializations to `<SIMDLevel::NONE>`. There is no SVE variant.
- There is no `block_l2<ARM_NEON>` and no `exhaustive_L2sqr_blas_cmax<ARM_NEON>`.
- Part two of T287037898, which is SDOT and SMMLA. `FEAT_DotProd` and `FEAT_I8MM` are not expressible as a `SIMDLevel`. The right shape is a runtime `getauxval(AT_HWCAP)` check inside the ARM translation unit, which follows the `SIMDConfig::avx512_split` precedent. Note the reporter's caveat: SMMLA shows no improvement until the scan loop is tiled, so the tiling must land in the same change.

Reviewed By: alibeklfc

Differential Revision: D118682598

fbshipit-source-id: 1ab588d9fa130e8246bf996025fc11d8954323be
 2026-09-10 09:32:36 -07:00
Michael Norris e2f9ccaccc Fix SQtqmse codebook scale: fold 1/sqrt(d) into Lloyd-Max table (#5517) 
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5517

`SQtqmse` reconstructions have inflated norms and badly degraded recall since
D102408184. Reported as a 1.14.2 -> 1.14.3 regression: at `d=64`, median decoded
norm of a unit-norm input went `0.9965` -> `1.1503` (+15%), while `SQtqmse8`
looked "essentially unchanged".

Fixes https://github.com/facebookresearch/faiss/issues/5317

Root cause
-
D102408184 replaced the `tqmse` codebook construction with hardcoded Lloyd-Max
tables:

```
-    scalar_quantizer::train_TurboQuantMSE(d, 4, trained);
+    populate_lloyd_max_trained(4, trained);
```

The old codebook was built for one coordinate of a unit-norm vector in R^d, so
its centroids scaled as `1/sqrt(d)`. The new tables are optimal for `N(0, 1)`
and do not depend on `d` at all.

That is correct for `_eden` and `_tq`, which rescale each vector to unit
variance before lookup. Plain `tqmse` does not: it encodes raw unit-norm
vectors, whose components are ~`1/sqrt(d)` (~`0.125` at `d=64`), against a
codebook ~50x too wide. Nearly everything falls in the innermost cells, so the
codebook collapses to a few levels and every component decodes too large:

| d | bits | distinct codes used (before D102408184 -> after) |
| --- | --- | --- |
| 64 | 4 | 16/16 -> **4/16** |
| 768 | 4 | 16/16 -> **2/16** |
| 768 | 8 | 256/256 -> **32/256** |

Gets worse as `d` grows.

Fix
-
Give `populate_lloyd_max_trained` a `scale` argument that multiplies the table.
The five `tqmse` cases pass `1/sqrt(d)` (the standard deviation of a unit-norm
vector's components) so the codebook once again (like 1.14.2) matches the data it encodes.

**No change for `_eden` / `_tq`.** They pass no `scale`, so it defaults to `1`
and every table entry is bit-identical to today.

Scaling `trained` rather than the encode path is what keeps this a one-line fix
per callsite: every SIMD specialization and distance computer reads
`this->centroids`, a pointer into `trained`.

Notes
-
- NOT A REVERT TO 1.14.2, but that is fine. 1.14.2 trained on the exact
  unit-sphere marginal; this scales a Gaussian approximation of it, so centroids
  differ by up to ~20% at 8 bits. The approximation is not worse: max component
  error is lower than 1.14.2 and recall is restored (see test plan). The only
  consequence is that `tqmse` codes are not comparable across 1.14.2 / 1.14.3 /
  this version -- harmless, because the codebook ships with the index (below).
- SERIALIZATION STILL COMPATIBLE: Existing 1.14.3-written `tqmse` indexes stay readable and self-consistent:
  `read_ScalarQuantizer` loads `trained` verbatim, so decode matches how they
  were encoded, and `trained` length is unchanged so size validation still
  passes. They are degraded, not corrupt, and need re-indexing to benefit. No
  version guard added.

Reviewed By: alibeklfc

Differential Revision: D115440646

fbshipit-source-id: a559f6a6a50a65dc3ab32ff3f17cda8796b2d4ae
 2026-08-12 13:46:19 -07:00
Tia Lim 0f0f728166 Format faiss Python files with black (line length 80) (#5349) 
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5349

# this is a no-op change

Ran `black -l 80` over all tracked Python files under `fbcode/faiss/`, using line length 80 to match faiss's documented coding style (`CONTRIBUTING.md`: "80 character line length (both for C++ and Python)"). 172 files were reformatted; 28 were already compliant.

This is a pure-formatting change. `black` runs its AST-equivalence safety check on every file, so no identifiers, string/numeric values, operators, or control flow were altered — only whitespace, indentation, line wrapping, blank lines, trailing commas, and string-quote normalization (single to double quotes).

`fbcode/faiss/` is intentionally excluded from Meta's standard Python autoformatter (`arc f` / `arc lint -a` report no formatter for these paths) because it mirrors to public GitHub, so the Python had drifted out of any consistent style. The C++ is already clang-format-clean (0 of 807 files change), so this effort is Python-only. This diff brings the Python to canonical `black` formatting.

Excluded `demos/index_pq_flat_separate_codes_from_codebook.py`: it contains a Jupyter `!rm` shell-magic line and is not valid Python, so `black` cannot parse it.

This is the base of a 3-diff stack: (1) this black reformat; (2) `.flake8` alignment (max-line-length 80 + black-compatible ignores E203/W503); (3) genuine lint fixes (F401/F403/F841/B011).

Reviewed By: mnorris11

Differential Revision: D109736099

fbshipit-source-id: 9a054635766ed11aa3760a788395babd3d198abf
 2026-06-29 17:29:05 -07:00
Michael Norris 74d36198fa TurboQuant in ScalarQuantizer: full Algorithm 2 (QJL stage) with SIMD and optimizations (#5170) 
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5170

Adds the complete TurboQuant implementation (paper Algorithm 2, "TurboQuant_prod") as a standalone TurboQuantizer class alongside the MSE-only ScalarQuantizer integration from D102195204.

Benchmark results
-
TLDR: TQ has a similar operating point to RaBitQ in terms of memory savings, but it never hit RaBitQ levels of recall. We do not really recommend TurboQuant for most ranking scenarios.

We implemented and compared TurboQuant to our existing implementation of RaBitQ across 20+ datasets. Some were internal, 18 were open source (from https://vector-index-bench.github.io/datasets.html, a nice collection)
- RaBitQ had 4% to 53% higher recall than TurboQuant at equivalent bit widths (i.e. TurboQuant2 vs RaBitQ2), for every single parameter swept. 2 bit is one of the most interesting operating points. TurboQuant recall was notably coupled with dataset characteristics: recall correlates heavily with dataset effective dimensions.
- TurboQuant had ~1.7x faster scoring than non-FastScan RaBitQ at equivalent bit widths for practical qb values. But RaBitQFS is faster and has the same recall as non-FS RaBitQ for reasonable datasets.
- Fast Walsh Hadamard Transform rather than Gaussian in the QJL step improved TurboQuant recall 7-15%. Random rotation was also better than Gaussian, and even better than FWHT on 5/18 datasets. So we just dropped Gaussian here and kept RR and FWHT.
- TurboQuant may have some recall benefit on datasets with very high effective dimensions, e.g. after applying heavy PCA, but in these cases recall is pretty degenerate anyway (50% ish).
- TurboQuant adaptive bit lengths (2.5, 3.5) raised recall only marginally over 2 and 3 respectively, 2-4%. However there is no reason this can't also be applied to RaBitQ for additional operating points. We plan to investigate this sometime soon.

Description
-
**Core algorithm:**
- TurboQuantizer: two-stage quantizer (Lloyd-Max MSE + QJL sign bits)
- IndexIVFTurboQ: IVF index with by_residual=false
- Bit-plane encoding layout for MSE codes
- Per-vector factors (norm, gamma) for distance computation
- FWHT and Random Rotation QJL projection options
- Custom distance computer with direct MSE+QJL scoring

**SIMD and pre-screening:**
- turboq::masked_sum SIMD kernel (AVX2/AVX512) for 1-bit MSE and QJL dot products
- Public TurboQDistanceComputer with pre-computed query state
- MSE-first pre-screening with worst-case L1 QJL error bound
- Adaptive bit width (TurboQ2.5, TurboQ3.5 factory strings)

**Integer popcount paths:**
- Integer popcount MSE path (qb parameter) reusing rabitq::bitwise_and_dot_product — 1.2-2.2x QPS with <1% recall loss
- Integer QJL path (int_qjl) — additional 1.3x QPS on top of qb

**Random Rotation QJL:**
- QJLProjectionType enum (FWHT, RANDOM_ROTATION)
- QR-orthogonalized Gaussian matrix for exact orthogonality
- TurboQ{n}r factory string

**Removed after benchmarking (not useful):**
- Gaussian QJL (TurboQ{n}g): strictly worse recall AND QPS than FWHT on all 26 datasets
- Statistical bound (stat_bound): catastrophic recall loss on real data (mean -0.20 R@10) due to CLT assumption failing on real embedding distributions
- Two-pass batch screening (two_pass_c): no benefit with practical IVF settings (candidate count > list size makes it a no-op)

Factory strings: TurboQ, TurboQ{1-5}, TurboQ{n}r, TurboQ2.5, TurboQ3.5
All work standalone or with IVF prefix (e.g., IVF100,TurboQ2).

Reviewed By: mdouze

Differential Revision: D102408184

fbshipit-source-id: b985e9c4327169a95c5bb9042a5ca3a093bcc405
 2026-06-04 07:59:24 -07:00
Gergely Szilvasy 417c53e0ce Add NoneSIMDLevel context manager for cross-level reference checks (#5158) 
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5158

Several DD-dispatched code paths produce integer outputs (codes,
top-k indices) where bit-exact equivalence between SIMD levels is the
right correctness bar, but the existing tests assert only level-
independent properties (decode error, recall) that wouldn't catch
silent dispatch drift between AVX2/AVX512/NEON/NONE implementations.

Adds NoneSIMDLevel context manager in common_faiss_tests.py:

    codes = quantizer.compute_codes(xb)
    with NoneSIMDLevel():
        codes_none = quantizer.compute_codes(xb)
    np.testing.assert_array_equal(codes, codes_none)

The context manager pins SIMDConfig to NONE for the duration of the
block, restores the previous level on exit (incl. exception), and
raises SkipTest when NONE isn't compiled in (static-AVX2 lane).

Adds test_codes_match_none / test_PQ4_search_matches_none methods to
five existing for_all_simd_levels test classes:
* TestScalarQuantizerEncodeDecode (test_scalar_quantizer_correctness)
* TestProductQuantizer (test_product_quantizer)
* TestRaBitQuantizerEncodeDecode (test_rabitq)
* TestSearch (test_fast_scan; checks (D, I) output of PQ16x4fs search)
* TestResidualQuantizer (test_residual_quantizer; replaces inlined
  set_level/restore from D102821184 with the new helper)

Caveat documented in the docstring: SkipTest raised inside subTest is
swallowed; tests using both must do an explicit availability check
before the loop. Applied where needed.

Reviewed By: mdouze

Differential Revision: D102989453

fbshipit-source-id: 475ee504271e0505615841ef82263776245419e0
 2026-05-02 14:30:13 -07:00
Fabrizio (Misto) Milo a7116ed4ec Add TurboQuant (CPU) (#5049) 
Summary:
This PR adds initial TurboQuant (see https://github.com/facebookresearch/faiss/issues/4990) support to Faiss and integrates it into the main codepaths needed for local evaluation.

Changes in this PR:
- add `IndexTurboQuantMSE` and the underlying `TurboQuantizer` implementation
- add cloning and serialization support for TurboQuant indexes
- expose TurboQuant through the Python SWIG bindings
- extend quantizer benchmarking to cover TurboQuant and additional datasets
- add unit tests for TurboQuant reconstruction behavior

## Preliminary Benchmarks

These are preliminary local results from a macOS CPU-only run.

Command:

```bash
python bench_quantizer.py glove 100x4 turboquant pq rq
eval on glove 100x4 maxtrain=100000
No training set: training on database
===== PQ
        training time: 0.594 s
        encode time: 0.317 reconstruction error: 0.010 recall@1: 0.7036 recons_err_compat 0.100 code_size: 50 B/vector
===== RQ
        training time: 208.554 s
max_beam_size=1
        encode time: 2.977 reconstruction error: 0.027 recall@1: 0.6034 recons_err_compat 0.162 code_size: 50 B/vector
max_beam_size=2
        encode time: 5.774 reconstruction error: 0.023 recall@1: 0.6280 recons_err_compat 0.151 code_size: 50 B/vector
max_beam_size=4
        encode time: 12.271 reconstruction error: 0.020 recall@1: 0.6512 recons_err_compat 0.140 code_size: 50 B/vector
max_beam_size=8
        encode time: 25.617 reconstruction error: 0.018 recall@1: 0.6751 recons_err_compat 0.131 code_size: 50 B/vector
max_beam_size=16
        encode time: 50.257 reconstruction error: 0.016 recall@1: 0.6890 recons_err_compat 0.123 code_size: 50 B/vector
max_beam_size=32
        encode time: 105.272 reconstruction error: 0.014 recall@1: 0.7067 recons_err_compat 0.116 code_size: 50 B/vector
===== TurboQuant
        training time: 0.002 s
        encode time: 0.080 reconstruction error: 0.009 recall@1: 0.7189 recons_err_compat 0.095 code_size: 50 B/vector
```

### Initial takeaway

On this macOS CPU-only benchmark, TurboQuant shows:
- better `recall@1` than PQ at the same code size
- lower reconstruction error than PQ
- higher `recall@1` than the best RQ setting tested here
- much lower training and encoding cost than RQ

### TODO
- [x] add JQL support (only MSE at the moment)
- reproduce the paper benchmarks with same datasets (need a beefy machine but script is ready)
- ~add CUDA support~ (will be in another PR if this is accepted)

Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5049

Test Plan:
buck build fbcode//faiss:faiss
buck test fbcode//faiss/tests:test_factory
buck test fbcode//faiss/tests:test_scalar_quantizer
buck test fbcode//faiss/tests:test_scalar_quantizer_correctness
buck build fbcode//faiss/tests/fb:faiss_fuzzer_helpers

Reviewed By: mdouze

Differential Revision: D102195204

Pulled By: mnorris11

fbshipit-source-id: a302541a1fc5c229e697f28e639cba633babd072
 2026-04-24 16:32:53 -07:00
Gergely Szilvasy f226be432c Roll out @for_all_simd_levels to key test files (#5061) 
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5061

Roll out the `for_all_simd_levels` decorator to 62 test classes across
11 test files. This ensures that every available SIMD level (NONE, AVX2,
AVX512, etc.) is exercised in CI, rather than only testing at the
auto-detected (highest) level.

Previously only `TestExtraDistances` in `test_extra_distances.py` used
the decorator. Now all test files covering DD-dispatched code paths are
parameterized: distances, PQ, SQ, fast scan, RaBitQ, partitioning,
HNSW, and binary indices.

Changes:
- Add `for_all_simd_levels` to 62 test classes across 11 test files
- BUCK: move decorated test targets to `supports_static_listing = False`
  (required because the decorator replaces class names with None,
  breaking TPX's static test enumeration)
- test_fast_scan.py, test_fast_scan_ivf.py: apply decorator manually
  after dynamic `setattr` method generation for TestAQFastScan and
  TestIVFAQFastScan (the `setattr` loops reference the class by name)
- test_rabitq_fastscan.py: extract `_create_fastscan_index` as
  module-level helper to fix cross-class method reference that broke
  when TestRaBitQFastScan was replaced by the decorator
- IndexFastScan.cpp: fix `search_implem_14` to auto-cap `qbs` based
  on `bbs`. The accumulate loop dispatch table only instantiates
  certain (nq, BB) pairs (e.g. BB=2 only has nq=1,2). Previously,
  using bbs=64 with the default qbs (batch of 4) would crash with
  "nq=3 bbs=64 not instantiated". Now the query batch size is
  automatically capped to the max nq supported for the given BB.
  Exposed by per-level testing of test_factory_with_batch_size.

Reviewed By: mdouze

Differential Revision: D99978401

fbshipit-source-id: 54858f3810bab91ac5ec8bf0ce0d55a77710727b
 2026-04-08 10:56:41 -07:00
Gergely Szilvasy cd4898bc13 Address review comments on SQ correctness tests (#4771) 
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/4771

Follow-up to D91104886 addressing Matthijs's review comments:

1. Use SyntheticDataset from faiss.contrib.datasets instead of manual
   np.random.random() calls throughout the tests

2. Factorize test_d1 and test_d1024 into a single test_extreme_dims
   test that iterates over both dimensions with subTest

3. Use self.subTest() in test_non_simd_dims for finer error granularity
   so each dimension reports as a separate subtest

Also moved the test to py_tests_with_contrib in BUCK since it now
depends on faiss.contrib.

Reviewed By: mdouze

Differential Revision: D91560577

fbshipit-source-id: 2f2890328a98254f16f0d836e824e897b1c194ae
 2026-01-27 03:31:28 -08:00
Gergely Szilvasy 49c64f9704 Add comprehensive ScalarQuantizer correctness tests (#4766) 
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/4766

Add test_scalar_quantizer_correctness.py with 13 tests covering:
- Encode/decode roundtrip for 4-bit, fp16, and uniform quantizer variants
- Search accuracy tests for SQ4 and SQfp16
- Distance computation correctness verification
- Edge cases: zero vectors, constant vectors, d=1, d=1024
- Non-SIMD-aligned dimensions (7, 9, 15, 17, 31, 33, 63, 65)
- Inner product metric testing

These tests complement the existing SQ tests in test_build_blocks.py
which only cover 8-bit and 6-bit quantizers. The non-SIMD-aligned
dimension tests are particularly important for validating SIMD
implementations handle remainder elements correctly.

Reviewed By: mdouze

Differential Revision: D91104886

fbshipit-source-id: 19dc7400bc28ce5a4af92a41a94864312a66d1b8
 2026-01-21 05:10:16 -08:00