kzahiri1
b4c66ba603
Summary:
Three defects in `contrib/factory_tools.py` where a reversed factory string rebuilds a different index or fails to parse. `test_get_code_size_hnsw_roundtrip` already pins this invariant for flat HNSW; these cases escape it.
1. **HNSW storage dropped.** The `IndexHNSW` branch never read `index.storage`, so `HNSW32,SQ8` and `HNSW32,PQ16x8` both reversed to `"HNSW32"` and rebuilt as `IndexHNSWFlat`. `get_code_size` then applied the flat `d*4` formula: at d=128 it reported 768 bytes/vector for an index whose storage is 128 plus the 256-byte graph estimate, so 384.
2. **IMI regex fields transposed.** `index_factory.cpp:295` parses `IMI2x([0-9]+)`, but the regex read `IMI\d+x2`. Every real IMI key (`IMI2x10,PQ16`, and the ones in `demos/demo_auto_tune.py`) raised `RuntimeError("cannot parse")`.
3. **IndexIVFPQR matched by the IndexIVFPQ check**, dropping the refinement stage: `IVF64,PQ8+16` reversed to `IVF64,PQ8x8`, 8 bytes instead of 24.
Consumers are `benchs/bench_all_ivf/parse_bench_all_ivf.py` (imports `get_code_size` as `unitsize`) and `benchs/bench_fw/index.py`.
Flat storage stays implicit as the bare `HNSW<M>` string, and a null `storage` pointer still returns `HNSW<M>` instead of raising.
**Behavior change worth flagging:** `IndexHNSW2Level` storage is `Index2Layer`, which `reverse_index_factory` does not handle, so it now raises `NotImplementedError` instead of silently returning a wrong `"HNSW32"`. It is not reachable from `index_factory`.
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5564
Test Plan:
Four tests added to `TestFactoryTools`. All four fail on unpatched `factory_tools.py` and pass with it:
```
# pristine contrib/factory_tools.py, new tests present
4 failed, 7 passed
FAILED test_get_code_size_hnsw_non_flat_storage
FAILED test_get_code_size_imi
FAILED test_hnsw_storage_reverse_index_factory
FAILED test_ivfpqr_reverse_index_factory
# with the fix
11 passed
```
Full `tests/test_contrib.py tests/test_factory.py`, pristine vs branch:
```
pristine: 2 failed, 88 passed
branch: 2 failed, 92 passed
```
Failure sets are identical. The two are `TestComputeGT::test_compute_GT_gpu` and `test_compute_GT_ip_gpu`, pre-existing and unrelated (`AttributeError: module 'faiss' has no attribute 'GpuMultipleClonerOptions'` on a CPU-only build).
An A/B sweep over 40 factory strings changes exactly the 7 expected lines and nothing else:
```
- HNSW32,SQ8 IndexHNSWSQ HNSW32 768
+ HNSW32,SQ8 IndexHNSWSQ HNSW32,SQ8 384
- HNSW32,SQfp16 IndexHNSWSQ HNSW32 768
+ HNSW32,SQfp16 IndexHNSWSQ HNSW32,SQfp16 512
- HNSW32,PQ16x8 IndexHNSWPQ HNSW32 768
+ HNSW32,PQ16x8 IndexHNSWPQ HNSW32,PQ16x8 272
- IVF64,PQ8+16 IndexIVFPQR IVF64,PQ8x8 8
+ IVF64,PQ8+16 IndexIVFPQR IVF64,PQ8+16 24
- IMI2x5,Flat IndexIVFFlat IMI2x5,Flat <RuntimeError>
+ IMI2x5,Flat IndexIVFFlat IMI2x5,Flat 512
- IMI2x5,PQ8 IndexIVFPQ IMI2x5,PQ8x8 <RuntimeError>
+ IMI2x5,PQ8 IndexIVFPQ IMI2x5,PQ8x8 8
- IMI2x8,PQ8x8 IndexIVFPQ IMI2x8,PQ8x8 <RuntimeError>
+ IMI2x8,PQ8x8 IndexIVFPQ IMI2x8,PQ8x8 8
```
flake8 at the 80-char limit from CONTRIBUTING reports the same 12 pre-existing findings before and after, none in the changed lines.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Reviewed By: mnorris11
Differential Revision: D118661065
Pulled By: trang-nm-nguyen
fbshipit-source-id: b50a9522361e3fd14f276b36196d3c728567b980
The contrib modules
The contrib directory contains helper modules for Faiss for various tasks.
Code structure
The contrib directory gets compiled in the module faiss.contrib. Note that although some of the modules may depend on additional modules (eg. GPU Faiss, pytorch, hdf5), they are not necessarily compiled in to avoid adding dependencies. It is the user's responsibility to provide them.
List of contrib modules
rpc.py
A very simple Remote Procedure Call library, where function parameters and results are pickled, for use with client_server.py
client_server.py
The server handles requests to a Faiss index. The client calls the remote index. This is mainly to shard datasets over several machines, see Distributed index
ondisk.py
Encloses the main logic to merge indexes into an on-disk index. See On-disk storage
exhaustive_search.py
Computes the ground-truth search results for a dataset that possibly does not fit in RAM. Uses GPU if available.
Tested in tests/test_contrib.TestComputeGT
torch_utils.py
Interoperability functions for pytorch and Faiss: Importing this will allow pytorch Tensors (CPU or GPU) to be used as arguments to Faiss indexes and other functions. Torch GPU tensors can only be used with Faiss GPU indexes. If this is imported with a package that supports Faiss GPU, the necessary stream synchronization with the current pytorch stream will be automatically performed.
Numpy ndarrays can continue to be used in the Faiss python interface after importing this file. All arguments must be uniformly either numpy ndarrays or Torch tensors; no mixing is allowed.
Tested in tests/test_contrib_torch.py (CPU) and gpu/test/test_contrib_torch_gpu.py (GPU).
inspect_tools.py
Functions to inspect C++ objects wrapped by SWIG. Most often this just means reading fields and converting them to the proper python array.
ivf_tools.py
A few functions to override the coarse quantizer in IVF, providing additional flexibility for assignment.
datasets.py
(may require h5py)
Definition of how to access data for some standard datasets.
factory_tools.py
Functions related to factory strings.
evaluation.py
A few non-trivial evaluation functions for search results
clustering.py
Contains:
-
a Python implementation of kmeans, that can be used for special datatypes (eg. sparse matrices).
-
a 2-level clustering routine and a function that can apply it to train an IndexIVF
big_batch_search.py
Search IVF indexes with one centroid after another. Useful for large databases that do not fit in RAM and a large number of queries.