2 Commits

Include renames

Author	SHA1	Message	Date
Michael Norris	ec94295adc	Bound the fast scan LUT scale by the accumulated sum (#5658) ... Summary: Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5658 A fast scan kernel sums `M` uint8 table entries into a uint16 that can wrap around to 0. The scale that quantizes the table must therefore bound the whole sum that gets accumulated, not only one entry. Seven call sites each held their own copy of that rule. Only four applied it. This adds `quantize_lut::fastscan_lut_scale` and routes all seven through it. The bound also reserves a margin for `round_tab`, which rounds half up, so `n` terms can gain `n/2` over the unrounded sum. `IndexIVFRaBitQFastScan` also folds its per-probe bias into the bound, because that bias is rounded into the same accumulator. `IndexPQFastScan` only reaches the bound when a sub-vector holds one dimension. A nearest centroid keeps the selected entries near the bottom of each column. `IndexSQFastScan` sets `M = d`, so it fails above about `d = 512`. At `d = 1024` it goes from R@1 0.00 to 1.00. Search speed does not change. The scale is computed once per query, and the inner loop is untouched. An index at `M <= 256` quantizes to the same bytes as before, because the sum term does not bind there. Reviewed By: junjieqi Differential Revision: D120638521 fbshipit-source-id: 27d5dd15f5a60a9c80cec1f63f53ac9c6c51b8ff	2026-09-25 13:47:19 -07:00
Mulugeta Mammo	b0d9e80288	Add fast scan variants of SQ indexes - IndexSQFastScan and IndexIVFSQFastScan (#5341) ... Summary: This PR introduces `IndexSQFastScan` and `IndexIVFSQFastScan`, fast scan variants of scalar quantized indexes. **IndexSQFastScan** (33f2cb639bd7a888b2ab7578083b73753c2fd3d3) Introduces `IndexSQFastScan`, a fast scan variant of `IndexScalarQuantizer` that leverages the PQ4 FastScan SIMD infrastructure (`vpshufb`) for accelerated search over scalar-quantized codes. Three code paths depending on quantizer type: - Native 4-bit (`QT_4bit`, `QT_4bit_uniform`): codes map directly onto the 16-level PQ4 fast scan. ~11x speedup, no precision loss. - Rerank (`QT_6bit`, `QT_8bit`, `QT_8bit_uniform`, `QT_8bit_direct`, `QT_8bit_direct_signed`): re-quantize to 4-bit for a coarse fast scan first pass, then rerank top-k*rerank_factor candidates with exact original-precision distances. ~7-9x speedup, near-zero precision loss at rerank_factor=4. - Fallback (`QT_fp16`, `QT_bf16`, `TurboQuant`, etc.): delegates to the ScalarQuantizer's own SIMD-optimized scanner. Same speed as IndexScalarQuantizer — no fast scan benefit, but provides a unified interface for all quantizer types. **benchmark**: `benchs/bench_index_sq_fastscan.py` ```text +---------------------+----------------------+---------+---------+--------+--------+ | Category | QType | Speedup | ms/q SQ | R@1 | R@10 | +---------------------+----------------------+---------+---------+--------+--------+ | Native 4-bit (SIMD) | QT_4bit | 11.9x | 0.240 | 0.7714 | 0.9985 | | | QT_4bit_uniform | 11.7x | 0.235 | 0.7180 | 0.9952 | +---------------------+----------------------+----------+---------+--------+--------+ | Rerank (rf=4) | QT_6bit | 9.3x | 0.202 | 0.9371 | 1.0000 | | | QT_8bit | 8.4x | 0.173 | 0.9783 | 1.0000 | | | QT_8bit_uniform | 6.7x | 0.138 | 0.9728 | 1.0000 | | | QT_8bit_direct | 4.7x | 0.101 | 0.9914 | 1.0000 | | | QT_8bit_direct_signed| 4.9x | 0.101 | 0.5999 | 0.6591 | +---------------------+----------------------+----------+---------+--------+--------+ | Fallback (~1.0x) | QT_fp16 | 0.9x | 0.373 | 0.9914 | 1.0000 | | | QT_bf16 | 1.0x | 0.365 | 0.9914 | 1.0000 | | | QT_8bit_tqmse | 1.0x | 0.261 | 0.6033 | 0.9561 | | | QT_5bit_tq | 1.0x | 3.013 | 0.2680 | 0.6248 | | | QT_4bit_tq | 1.0x | 2.416 | 0.1511 | 0.3994 | | | QT_3bit_tq | 1.0x | 0.411 | 0.0487 | 0.1691 | | | QT_2bit_tq | 1.1x | 0.231 | 0.0376 | 0.1532 | | | QT_4bit_tqmse | 1.0x | 0.290 | 0.0012 | 0.0064 | | | QT_3bit_tqmse | 1.0x | 0.345 | 0.0000 | 0.0001 | | | QT_2bit_tqmse | 1.0x | 0.266 | 0.0000 | 0.0001 | | | QT_1bit_tqmse | 1.0x | 0.280 | 0.0000 | 0.0001 | +---------------------+----------------------+----------+---------+--------+--------+ Key: Speedup = IndexScalarQuantizer ms / IndexSQFastScan ms R@1, R@10 = IndexSQFastScan recall (matches SQ for all fallback types) Rerank shows rf=4 (rf=2 and rf=8 yield same recall, ~0.1x speedup variation) Dataset: SIFT1M, k=32 ``` **IndexIVFSQFastScan** (ebfa072816df2fbd80a6b463a913866f862587f6) Similar to the above, this commit introduces `IndexIVFSQFastScan`, an IVF variant of scalar quantized fast scan that extends the PQ4 FastScan SIMD infrastructure (vpshufb) to inverted-list indexes with scalar quantization. Uses BlockInvertedLists for packed 4-bit codes scanned via SIMD. Will continue to update the PR until it's feature complete. Three code paths depending on quantizer type: - Native 4-bit (QT_4bit, QT_4bit_uniform): codes pack directly into the SIMD block layout with no precision loss. ~12x speedup. - Rerank (QT_6bit, QT_8bit, QT_8bit_uniform, QT_8bit_direct, QT_8bit_direct_signed): re-quantized to 4-bit for the SIMD coarse pass, then reranked using exact original-precision codes stored in a parallel ArrayInvertedLists. ~8-10x speedup, near-zero precision loss at rerank_factor=2. - Fallback (QT_fp16, QT_bf16, TurboQuant, etc.): delegates to IndexIVF's InvertedListScanner path. Same behavior as IndexIVFScalarQuantizer — no fast scan benefit, but provides a unified interface for all quantizer types. **benchmark (SIFT1M, d=128, k=32, SapphireRapids)**: `benchs/bench_index_ivf_sq_fastscan.py` ```text +---------------------+----------------------+--------+-------------------------------------------+ | Category | QType | nprobe | Speedup | ms/q SQ | R@1 FS | R@10 FS | +---------------------+----------------------+--------+---------+---------+--------+-------------+ | Native 4-bit (SIMD) | QT_4bit | 1 | 7.1x | 0.007 | 0.4582 | 0.5456 | | | | 4 | 10.4x | 0.007 | 0.6903 | 0.8667 | | | | 16 | 12.9x | 0.022 | 0.7668 | 0.9885 | | | | 64 | 12.1x | 0.070 | 0.7725 | 0.9983 | | | | 256 | 11.3x | 0.230 | 0.7725 | 0.9983 | | +----------------------+--------+---------+---------+--------+-------------+ | | QT_4bit_uniform | 1 | 15.8x | 0.006 | 0.4397 | 0.5449 | | | | 4 | 10.5x | 0.007 | 0.6419 | 0.8650 | | | | 16 | 10.4x | 0.018 | 0.7113 | 0.9855 | | | | 64 | 10.9x | 0.063 | 0.7162 | 0.9951 | | | | 256 | 11.0x | 0.222 | 0.7162 | 0.9951 | +---------------------+----------------------+--------+---------+---------+--------+-------------+ | Rerank (rf=4) | QT_6bit | 1 | 4.9x | 0.007 | 0.4582 | 0.5456 | | | | 4 | 4.8x | 0.007 | 0.6903 | 0.8667 | | | | 16 | 7.9x | 0.018 | 0.7668 | 0.9885 | | | | 64 | 10.4x | 0.067 | 0.7725 | 0.9983 | | | | 256 | 10.2x | 0.221 | 0.7725 | 0.9983 | | +----------------------+--------+---------+---------+--------+-------------+ | | QT_8bit | 1 | 17.1x | 0.007 | 0.4582 | 0.5456 | | | | 4 | 12.5x | 0.008 | 0.6903 | 0.8667 | | | | 16 | 8.9x | 0.015 | 0.7668 | 0.9885 | | | | 64 | 8.4x | 0.049 | 0.7725 | 0.9983 | | | | 256 | 9.3x | 0.188 | 0.7725 | 0.9983 | | +----------------------+--------+---------+---------+--------+-------------+ | | QT_8bit_uniform | 1 | 18.2x | 0.007 | 0.4397 | 0.5449 | | | | 4 | 11.1x | 0.008 | 0.6419 | 0.8650 | | | | 16 | 8.3x | 0.014 | 0.7113 | 0.9855 | | | | 64 | 7.8x | 0.045 | 0.7162 | 0.9951 | | | | 256 | 7.8x | 0.164 | 0.7162 | 0.9951 | | +----------------------+--------+---------+---------+--------+-------------+ | | QT_8bit_direct | 1 | 8.6x | 0.007 | 0.4083 | 0.5426 | | | | 4 | 7.0x | 0.008 | 0.5987 | 0.8587 | | | | 16 | 6.8x | 0.013 | 0.6550 | 0.9756 | | | | 64 | 6.1x | 0.035 | 0.6581 | 0.9850 | | | | 256 | 7.4x | 0.150 | 0.6581 | 0.9850 | | +----------------------+--------+---------+---------+--------+-------------+ | | QT_8bit_direct_signed| 1 | 16.5x | 0.006 | 0.4257 | 0.5447 | | | | 4 | 10.1x | 0.007 | 0.6286 | 0.8633 | | | | 16 | 7.1x | 0.012 | 0.6919 | 0.9831 | | | | 64 | 5.7x | 0.033 | 0.6963 | 0.9923 | | | | 256 | 7.1x | 0.145 | 0.6963 | 0.9923 | +---------------------+----------------------+--------+---------+---------+--------+-------------+ | Fallback (~1.0x) | QT_fp16 | 1 | 1.0x | 0.007 | 0.5434 | 0.5462 | | | | 4 | 1.1x | 0.013 | 0.8607 | 0.8678 | | | | 16 | 0.9x | 0.035 | 0.9814 | 0.9902 | | | | 64 | 0.9x | 0.115 | 0.9912 | 1.0000 | | | | 256 | 0.9x | 0.481 | 0.9912 | 1.0000 | | +----------------------+--------+---------+---------+--------+-------------+ | | QT_bf16 | 1 | 1.0x | 0.008 | 0.5427 | 0.5462 | | | | 4 | 1.1x | 0.013 | 0.8581 | 0.8678 | | | | 16 | 1.0x | 0.035 | 0.9778 | 0.9902 | | | | 64 | 1.0x | 0.115 | 0.9876 | 1.0000 | | | | 256 | 1.0x | 0.487 | 0.9876 | 1.0000 | | +----------------------+--------+---------+---------+--------+-------------+ | | QT_5bit_tq | 1 | 1.0x | 0.016 | 0.1866 | 0.3827 | | | | 64 | 1.0x | 0.794 | 0.2680 | 0.6248 | | | | 256 | 1.0x | 2.885 | 0.2680 | 0.6248 | | +----------------------+--------+---------+---------+--------+-------------+ | | QT_4bit_tq | 1 | 1.0x | 0.013 | 0.1182 | 0.2841 | | | | 64 | 0.9x | 0.618 | 0.1511 | 0.3994 | | | | 256 | 0.9x | 2.298 | 0.1511 | 0.3994 | | +----------------------+--------+---------+---------+--------+-------------+ | | QT_3bit_tq | 1 | 1.1x | 0.008 | 0.0526 | 0.1599 | | | | 64 | 0.8x | 0.092 | 0.0488 | 0.1696 | | | | 256 | 0.8x | 0.357 | 0.0487 | 0.1691 | | +----------------------+--------+---------+---------+--------+-------------+ | | QT_2bit_tq | 1 | 1.2x | 0.008 | 0.0450 | 0.1526 | | | | 64 | 1.1x | 0.072 | 0.0377 | 0.1534 | | | | 256 | 1.1x | 0.224 | 0.0376 | 0.1532 | | +----------------------+--------+---------+---------+--------+-------------+ | | QT_4bit_tqmse | 1 | 1.0x | 0.007 | 0.0619 | 0.1851 | | | | 64 | 1.0x | 0.072 | 0.0002 | 0.0010 | | | | 256 | 0.9x | 0.275 | 0.0000 | 0.0000 | | +----------------------+--------+---------+---------+--------+-------------+ | | QT_3bit_tqmse | 1 | 1.0x | 0.007 | 0.0621 | 0.1852 | | | | 64 | 1.0x | 0.090 | 0.0006 | 0.0032 | | | | 256 | 1.0x | 0.350 | 0.0000 | 0.0000 | | +----------------------+--------+---------+---------+--------+-------------+ | | QT_2bit_tqmse | 1 | 1.1x | 0.007 | 0.0621 | 0.1852 | | | | 64 | 1.0x | 0.068 | 0.0015 | 0.0080 | | | | 256 | 1.0x | 0.244 | 0.0000 | 0.0000 | +---------------------+----------------------+--------+---------+---------+--------+-------------+ Key: Speedup = IndexIVFScalarQuantizer ms / IndexIVFSQFastScan ms R@1, R@10 = IVFSQFastScan recall (matches IVF-SQ for all fallback types) Rerank shows rf=4 only (rf=2,8 yield similar results) Fallback shows nprobe=1,64,256 only (abbreviated for low-recall types) Dataset: SIFT1M, nlist=256, k=32 ``` Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5341 Reviewed By: junjieqi Differential Revision: D117259284 Pulled By: mnorris11 fbshipit-source-id: 41455097ff3ae1285aeb4cf88e07abc5b5b13171	2026-09-18 20:24:54 -07:00