Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5419
### THIS DIFF
Fix two bugs in `reverse_index_factory` for multi-bit RaBitQ indexes, in
both the C++ and Python implementations.
**C++ (`factory_tools.cpp`):**
- `IndexIVFRaBitQ` case returned `prefix + ",RaBitQ"` unconditionally,
regardless of `rabitq.nb_bits`. For a 2-bit index built from
`"IVF_nlist,RaBitQ2"`, the function returned `"IVF_nlist,RaBitQ"` (1-bit
factory string). This means `TelemetryWrapper` logs the wrong index type
for every production multi-bit RaBitQ IVF index.
- `IndexRaBitQ` (standalone) had the same flaw — always returned
`"RaBitQ"` regardless of `nb_bits`.
**Python (`contrib/factory_tools.py`):**
- Neither `IndexIVFRaBitQ` nor `IndexRaBitQ` had a case in
`reverse_index_factory`, so any Python caller using this API on a RaBitQ
index raised `NotImplementedError`.
**Fix:** Read `rabitq.nb_bits` from the typed cast result and append the
bit count when `nb_bits > 1` (e.g., `"RaBitQ2"`). For `nb_bits == 1`
the output is still `"RaBitQ"`, preserving backward compatibility with
existing logs and round-trip tests.
### CONTEXT
Multi-bit RaBitQ (`nb_bits=2..9`) was added alongside standard 1-bit
RaBitQ. `index_factory` has parsed `"RaBitQ2"`, `"RaBitQ4"`, etc., since
the feature landed, but `reverse_index_factory` was never updated to emit
these strings. Every production `IndexIVFRaBitQ` with `nb_bits > 1`
(including the RaBitQ2 deployment) logs `",RaBitQ"` (1-bit) in telemetry.
The Python API has silently been broken for all RaBitQ index types.
The fix pattern matches how other parameterized types (`PQMxN`, `HNSWNM`,
etc.) embed their parameters in the factory string.
Reviewed By: mnorris11
Differential Revision: D110042828
fbshipit-source-id: d4e976bd9fc99691a20fb33d893af2d36deeb1e1
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5417
`reverse_index_factory` in `factory_tools.cpp` has four correctness bugs that
affect production telemetry. `TelemetryWrapper::TelemetryWrapper()` calls
`reverse_index_factory(this)` with no try/catch; any exception propagates
out of the constructor.
**Bug 1 — `sq_types.at()` throws for QT_0bit, QT_8bit_uniform, QT_4bit_uniform**
`IndexIVFScalarQuantizer` and `IndexScalarQuantizer` constructed with these
qtypes pass a key absent from `sq_types` to `std::map::at()`, which throws
`std::out_of_range`. Any `TelemetryWrapper<IndexIVFScalarQuantizer>` with one
of these qtypes aborts the process.
Fix: add all three missing entries.
- `QT_0bit → "SQ0"`: accepted by `index_factory` (listed in its name map).
- `QT_8bit_uniform → "SQ8u"`, `QT_4bit_uniform → "SQ4u"`: synthetic names,
not round-trippable through `index_factory`, but match `contrib/
factory_tools.py` and uniquely identify the type in telemetry.
**Bug 2 — IndexIDMap2 returns "IDMap,..." instead of "IDMap2,..."**
`IndexIDMap2` inherits `IndexIDMap`. The existing
`dynamic_cast<const faiss::IndexIDMap*>` check catches both. Every
`TelemetryWrapper<IndexIDMap2>` is logged as `"IDMap,..."` — wrong type.
Fix: add `dynamic_cast<const faiss::IndexIDMap2*>` check before the
`IndexIDMap` branch, matching the ordering in `contrib/factory_tools.py`.
Reviewed By: mnorris11
Differential Revision: D109830472
fbshipit-source-id: 58659b8d54810df89800ad46ef57ffb51c5eaca6
Summary:
This PR adds initial TurboQuant (see https://github.com/facebookresearch/faiss/issues/4990) support to Faiss and integrates it into the main codepaths needed for local evaluation.
Changes in this PR:
- add `IndexTurboQuantMSE` and the underlying `TurboQuantizer` implementation
- add cloning and serialization support for TurboQuant indexes
- expose TurboQuant through the Python SWIG bindings
- extend quantizer benchmarking to cover TurboQuant and additional datasets
- add unit tests for TurboQuant reconstruction behavior
## Preliminary Benchmarks
These are preliminary local results from a macOS CPU-only run.
Command:
```bash
python bench_quantizer.py glove 100x4 turboquant pq rq
eval on glove 100x4 maxtrain=100000
No training set: training on database
===== PQ
training time: 0.594 s
encode time: 0.317 reconstruction error: 0.010 recall@1: 0.7036 recons_err_compat 0.100 code_size: 50 B/vector
===== RQ
training time: 208.554 s
max_beam_size=1
encode time: 2.977 reconstruction error: 0.027 recall@1: 0.6034 recons_err_compat 0.162 code_size: 50 B/vector
max_beam_size=2
encode time: 5.774 reconstruction error: 0.023 recall@1: 0.6280 recons_err_compat 0.151 code_size: 50 B/vector
max_beam_size=4
encode time: 12.271 reconstruction error: 0.020 recall@1: 0.6512 recons_err_compat 0.140 code_size: 50 B/vector
max_beam_size=8
encode time: 25.617 reconstruction error: 0.018 recall@1: 0.6751 recons_err_compat 0.131 code_size: 50 B/vector
max_beam_size=16
encode time: 50.257 reconstruction error: 0.016 recall@1: 0.6890 recons_err_compat 0.123 code_size: 50 B/vector
max_beam_size=32
encode time: 105.272 reconstruction error: 0.014 recall@1: 0.7067 recons_err_compat 0.116 code_size: 50 B/vector
===== TurboQuant
training time: 0.002 s
encode time: 0.080 reconstruction error: 0.009 recall@1: 0.7189 recons_err_compat 0.095 code_size: 50 B/vector
```
### Initial takeaway
On this macOS CPU-only benchmark, TurboQuant shows:
- better `recall@1` than PQ at the same code size
- lower reconstruction error than PQ
- higher `recall@1` than the best RQ setting tested here
- much lower training and encoding cost than RQ
### TODO
- [x] add JQL support (only MSE at the moment)
- reproduce the paper benchmarks with same datasets (need a beefy machine but script is ready)
- ~add CUDA support~ (will be in another PR if this is accepted)
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5049
Test Plan:
buck build fbcode//faiss:faiss
buck test fbcode//faiss/tests:test_factory
buck test fbcode//faiss/tests:test_scalar_quantizer
buck test fbcode//faiss/tests:test_scalar_quantizer_correctness
buck build fbcode//faiss/tests/fb:faiss_fuzzer_helpers
Reviewed By: mdouze
Differential Revision: D102195204
Pulled By: mnorris11
fbshipit-source-id: a302541a1fc5c229e697f28e639cba633babd072
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/4782
This diff makes several cleanups to prepare for Dynamic Dispatch:
Move factory_tools from cppcontrib to main faiss library:
- Moves factory_tools.cpp/h from faiss/cppcontrib/ to faiss/
- Removes the separate //faiss/cppcontrib:factory_tools target
- Updates all includes to use <faiss/factory_tools.h>
- Updates xplat.bzl to include factory_tools in source/header lists
Fix unaligned SIMD store in simd_result_handlers.h:
- Changes store() to storeu() for potentially unaligned memory
- Fixes crashes in test_pqfs_unaligned when compiled with SIMD flags
Sync CMake tests with BUCK:
- Add missing tests: test_NSG_compressed_graph, test_pqfs_unaligned, test_scalar_quantizer
- Remove test_mem_leak.cpp (already removed from BUCK in D73790172)
- Move test_factory_tools from cppcontrib tests to main tests
Add autodeps-skip to faiss library targets:
- Prevents autodeps from modifying carefully configured SIMD build settings
Reviewed By: junjieqi
Differential Revision: D91814033
fbshipit-source-id: 4ac00f4fca53725a9441cd81573736d0b13a88ec
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/4003
It was emitting "SQSQ8" for `IndexScalarQuantizer`. The enum names already have `SQ` in the prefix.
Reviewed By: kuarora
Differential Revision: D65083146
fbshipit-source-id: 564f6d3d8987a2188fb7aa82b5ba772034225550
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/3879
1. Adds JK `faiss/telemetry:use_faiss_telemetry_core` to the top level logging util in `wrapper_logging_utils.h`. This is currently set to false. I plan to deprecate the other knobs under https://www.internalfb.com/intern/justknobs/?name=faiss%2Ftelemetry and just use one, as Unicorn can't really have their own JK easily (they subclass a lot of FAISS classes too).
2. Copied StringIOReader from Unicorn to telemetry wrapper in `io.h`. This will be deleted from Unicorn in the follow up diff.
3. Updated Laser tests to reflect correct index_read factory string changes.
4. Adds reverse_index_factory. More tests for it in subsequent diff.
Reviewed By: junjieqi
Differential Revision: D62670316
fbshipit-source-id: de1b2ed385593bb43798d29d16d90407920a3251
Alibek Zhakubayev