Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5616
Follow up on D118495504 by closing two arithmetic gaps in the Panorama
bounds validation. Avoid signed overflow and reject negative counts in
`IndexFlatPanorama::reconstruct_n`, and compute the serialized
`IndexFlatPanorama` batch count without overflowing
`ntotal + batch_size - 1`.
Add regression coverage for negative reconstruction counts and a maximal
serialized batch size that previously admitted empty storage for a
non-empty index.
Reviewed By: mnorris11
Differential Revision: D119272683
fbshipit-source-id: 7ea841bd4bd90d0358b78797441fa3cc815db46a
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5560
`OnDiskInvertedListsIOHook::read` allocates its `OnDiskInvertedLists` with a
bare `new` and then keeps reading. Any throw after that point leaks the whole
object -- including `od->lists`, which `READVECTOR` will happily size up to
`get_deserialization_vector_byte_limit()`. Throwing after that point is the
common case for malformed input: the `slots` and `filename` reads can fail, and
`do_mmap()` on an attacker-controlled filename fails essentially always.
This is what the `faiss_read_index_binary_fuzzer` health check has been tripping
over. The one CI stack that survived tpx truncation names
`OnDiskInvertedLists.cpp:722` (`vector<OnDiskOneList>::resize`) as the largest
live allocation, reached via `IBFf` -> `read_index` -> `IxRP` -> IVF ->
`read_InvertedLists`. Because the harness runs with `detect_leaks: 0`, the
leaked blocks accumulate across a 120 s session; peak RSS across the failing
runs climbed 978 -> 1142 -> 1249 -> 1815 -> 4356 MB against `-rss_limit_mb=4096`.
That is why the check reads as flaky (23.6%) rather than as a hard red, and why
the saved crash artifacts do not reproduce when replayed one at a time.
Switch both `read` and `read_ArrayInvertedLists` to hold the object in a
`std::unique_ptr` and `release()` on success. The public signature still returns
`InvertedLists*`, so this is a local scope guard only -- the same shape
`read_InvertedLists_up` in `index_read.cpp` already uses.
Also reject `lists.size() != nlist` at read time. Every `OnDiskInvertedLists`
accessor (`list_size`, `get_codes`, `get_ids`, ...) indexes `lists[list_no]`
with `list_no` bounded only against `nlist`, so a file whose `lists` vector is
shorter than its `nlist` is a straight out-of-bounds read. Every construction
path in the class keeps `lists.size() == nlist`, so this rejects only malformed
input.
Reviewed By: mnorris11
Differential Revision: D118487565
fbshipit-source-id: 6f01c937b24a2a98620e4ca2bde141674af1b19d
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5618
Remove the unused projection buffers and eager `init_projection` work from `ScalarQuantizer::TurboQuantRefine`. Quantizers construct the required projection from `trained` when used, so rebuilding it during training and deserialization only added an unbounded quadratic allocation and cubic QR operation.
Keep `seed` and `qjl_type` as the serialized configuration, enforce that the helper remains a small config-only struct, and cover both projection modes in deserialization tests.
___
Differential Revision: D119243693
fbshipit-source-id: 53f982b00dd849cbe88983ffd27809c2fa738a4c
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5562
Lionhead harness `faiss_rmdt_post_read_reconstruct_fuzzer` found a
heap out-of-bounds read at `Panorama.cpp:171`, reported as
`READ of size 5122903332` in `__asan_memcpy`.
Decoding the 232-byte crash input, it is `IxPT -> IxPT -> IxPT -> IxFp`
where every `IndexPreTransform` carries an **empty** transform chain and
the innermost `IndexFlatIPPanorama` has `d = 0x4C565349 = 1280725833`
(so `code_size = d * 4 = 5122903332`, exactly the ASAN read size),
`ntotal = 0`, and an empty `codes` vector. Three independent defects
combine:
1. `IndexFlatPanorama::reconstruct` never bounds-checks `key`. It
overrides `IndexFlat::reconstruct`, which does have
`FAISS_THROW_IF_NOT(key < ntotal)`, and drops the guard. Overriding
`reconstruct_n` likewise removes `IndexFlatCodes::reconstruct_n`'s
`i0 + ni <= ntotal` check. With `ntotal == 0` and empty `codes`,
`reconstruct(0, ...)` memcpy's `code_size` bytes out of a zero-length
buffer. `Panorama::reconstruct` takes an unsized `const uint8_t*`, so it
cannot self-validate -- the caller has to.
2. The `IxFP`/`IxFp` read branch does not range-check `d`. It is the one
index type that does not go through `read_index_header`, so it misses
that function's `FAISS_CHECK_RANGE(d, 0, (1 << 20) + 1)` and
`ntotal >= 0` checks. The `codes.size()` equality check below it does
not catch this, because `ntotal == 0` makes `num_slots == 0` and an
empty `codes` is then "valid".
3. `IndexPreTransform` deserialization skips all consistency checks when
the chain is empty -- the `chain[0].d_in == ixpt->d` /
`chain[nt-1].d_out == index->d` validation sits entirely inside
`if (nt > 0)`. With `nt == 0`, `IndexPreTransform::reconstruct` passes
the caller's buffer (sized from the outer `d`) straight to the
sub-index, so a mismatched sub-index `d` is also a write-side overflow,
for any sub-index type.
Each is fixed:
- `IndexFlatPanorama::reconstruct` / `reconstruct_n` now range-check
against `ntotal`.
- The `IxFP`/`IxFp` branch applies the same `d` and `ntotal` checks as
`read_index_header`.
- `read_index` requires `index->d == ixpt->d` when the `IxPT` chain is
empty.
Two adjacent gaps in the same audit are closed as well:
- `IHfP` (`IndexHNSWFlatPanorama`) never validated `cum_sums.size()`,
unlike the equivalent `IxFP` check. Both `get_cum_sum()` and
`HNSW.cpp`'s Panorama search path index it at stride
`pano.n_levels + 1`, so a short vector is an OOB read at search time.
It also had no explicit `n_levels > 0` guard.
- `ArrayInvertedListsPanorama::get_single_code` bounds-checked with
`assert`, which is compiled out in opt builds. It reaches the same
`Panorama::reconstruct` sink via
`IndexIVFFlatPanorama::reconstruct_from_offset`.
Reviewed By: mnorris11
Differential Revision: D118495504
fbshipit-source-id: 46bc7a1a6c9335a5589e20221b9478ae5505b9da
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5561
`svs::lib::DirectoryArchiver::read_file` builds its output path as `root / filename`, where `filename` is read straight off the deserialization stream. Neither guard on that path contains it:
- `std::filesystem::operator/` **discards `root` entirely** when the right operand is absolute, so `filename = "/etc/cron.d/x"` yields `path = "/etc/cron.d/x"`.
- A `..` component climbs out of `root` normally.
- `check_file(path, out|binary)` only verifies that `path.parent_path()` is an existing directory. It performs no containment check — and that check is satisfied by construction anyway, because `read_file` calls `fs::create_directories(parent_dir)` two lines earlier.
The result is that `faiss::read_index_up()` on untrusted bytes can create or overwrite an arbitrary file with attacker-controlled content, bounded only by the process's filesystem permissions. The harness manifest already rates this surface `REMOTE` / `PRE_AUTH` / `ZERO_CLICK`.
This is not theoretical. The new tests fail on unpatched code with `std::filesystem::exists(probe)` returning **true** — the file really is written outside the temp root. Note the deserialization then fails with an unrelated `"File could not be opened for reading"`, so the write is a silent side effect that never surfaces as an error to the caller.
The fix normalizes both sides and rejects any entry whose resolved path is not under `root`, before `create_directories` runs so that a rejected entry leaves nothing behind. Legitimate entries are unaffected: plain names, nested `sub/dir/name`, `./name`, and interior `..` that stays inside `root` all still resolve normally — only escapes are rejected.
`third-party/svs/v0.4.0` is pristine vendored code (`c5e5dbb35bbf`, never patched internally), so this also needs reporting upstream to Intel via their vulnerability handling process (`src/SECURITY.md`) rather than as a public PR, so it survives the next version bump.
The related unbounded-allocation defect in `Archiver::read_name`, on the same call path, is fixed separately in the next diff in this stack (T287110736).
Reviewed By: mnorris11
Differential Revision: D118494326
fbshipit-source-id: 7f7851111375eda831e4a7dac0acb7cdd1f80dc8
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5485
Fixes T280514309: an attacker-tunable heap-buffer-overflow (write) that
fires during `faiss::read_index()` when deserializing a malformed
`ResidualCoarseQuantizer` index (`ImRQ` fourcc).
Root cause: the FAISS index format encodes the centroid count twice - once
as the index header field `ntotal`, and once implicitly as
`2^sum(rq.nbits)` (`tot_bits`). On the `ImRQ` read path these two fields
are restored independently and never reconciled. `set_beam_factor()` then
sizes the `centroid_norms` buffer from `ntotal`
(IndexAdditiveQuantizer.cpp:516) while `compute_centroid_norms()` writes
`2^tot_bits` floats (AdditiveQuantizer.cpp:386). A crafted index with
`ntotal < 2^tot_bits`, `beam_factor <= 0`, and `metric_type == METRIC_L2`
overflows the heap allocation. The trained-constructor invariant
`FAISS_THROW_IF_NOT(rq.tot_bits <= 63)` and `ntotal == 1 << tot_bits` are
bypassed by deserialization, which assigns fields directly onto a
default-constructed object.
Fix: reconcile the two fields in the `ImRQ` branch of `read_index_up()`
before `set_beam_factor()` runs, matching the cross-field consistency
checks that sibling index types already perform. We restore the
`tot_bits <= 63` bound (also avoiding UB in `(size_t)1 << tot_bits`) and
require `ntotal == 2^tot_bits`, which holds by construction for every
legitimately-trained ResidualCoarseQuantizer.
Reviewed By: mnorris11
Differential Revision: D113832696
fbshipit-source-id: 4bc2f9f5451ba6e6b95cdd33abbe8bafba295b6f
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5464
The `faiss_read_index_binary_fuzzer_ci_diff_time` health-check has been failing (T276499517). Reproduced at HEAD: libFuzzer reports `out-of-memory (malloc(2147483648))` with the stack `AlignedTableTightAlloc<float,32>::resize` <- `initialize_IVFPQ_precomputed_table` <- `read_ivfpq` <- `read_index_binary` (the `IBFf` path).
Root cause: `read_ivfpq()` calls `IndexIVFPQ::precompute_table()`, which builds a derived table of `quantizer->ntotal * pq.M * pq.ksub` floats. That table is NOT stored in the serialized index — it is recomputed on load from attacker-controlled header fields — and faiss only capped it at `precomputed_table_max_bytes` (2 GB). Every other derived allocation in `index_read.cpp` is already bounded by the runtime-configurable `get_deserialization_vector_byte_limit()` (which the fuzzer sets to 128 MB), but this precompute path was not, so a tiny crafted header can drive a multi-GB allocation and OOM.
Fix: bound `ntotal * pq.M * pq.ksub` by `get_deserialization_vector_byte_limit()` (overflow-safe via `mul_no_overflow`) before calling `precompute_table()`, mirroring the existing `read_ProductQuantizer` guard style. For normal library use the limit defaults to 1 TB, so realistic tables (<= 2 GB) still load exactly as before — no behavioral change.
Also adds a deterministic regression test `ReadIndexDeserialize.IVFPQPrecomputedTableExceedsByteLimit` that serializes a real trained IVFPQ and confirms reading it back under a low byte limit throws instead of allocating.
Reviewed By: mnorris11, alibeklfc
Differential Revision: D111974816
fbshipit-source-id: 89b23905cd1b9406d957471c574afc0c2b1c91a9
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5279
READ1(bool) reads a single byte directly from the serialized stream
into a bool's storage. The set of valid byte values for a bool is
implementation-defined - on every C++ ABI any byte outside the
implementation's canonical encoding is undefined behavior, and UBSan's
invalid-bool-load check aborts on it.
FAISS itself only ever writes the canonical encoding via WRITE1(bool),
so well-formed indices roundtrip cleanly. Corrupt or attacker-controlled
input that places a non-canonical byte at a bool-field offset, however,
can introduce UB-inducing data instead of being rejected as a
deserialization error.
Add a READ1_BOOL helper to faiss/impl/io_macros.h that reads one byte,
assigns it to the destination bool via the language-defined conversion
(b != 0) - which is well-defined for any input byte - and then compares
the resulting bool's storage byte back against the byte we read. The
roundtrip succeeds iff the input byte was already the platform's
canonical bool encoding. This stays ABI-portable: no assumption that 0
and 1 are the only valid encodings is baked into the source. A
static_assert(sizeof(bool) == 1) makes the existing implicit FAISS
assumption (WRITE1/READ1 of bool transfers exactly one byte) explicit
at the one call site that depends on it for the byte compare.
Apply READ1_BOOL to every READ1 call in faiss/impl/index_read.cpp whose
destination is a bool - covering the local SVS variables (initialized,
trained, has_vt) and bool fields across IVF, PQ, HNSW, NSG, NNDescent,
SVS, additive-quantizer, lattice, scalar-quantizer, refine, residual,
and binary index types.
Reviewed By: mnorris11
Differential Revision: D107454749
fbshipit-source-id: bc550083a83e9bd5fe09a58a9b698e9380ef9cc1
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5239
Two deserialize sites in `read_index_up` / `read_index_binary_up` accept a `nullptr` sub-index without validation:
1. `IndexIDMap` (`IxMp` / `IxM2`): the recently-added inner-ntotal cross-validator compares `idxmap->index->ntotal` against `idxmap->ntotal` without first verifying `idxmap->index` is non-null. A corrupt or maliciously constructed payload with a `"null"` inner fourcc dereferences the null pointer inside the validator that was meant to protect against exactly this class of failure. The binary `IndexBinaryIDMap` (`IBMp` / `IBM2`) path has the same cross-check shape, but is not reachable directly because `read_index_binary` rejects an unknown `"null"` fourcc at dispatch; the same payload class still reaches the float `IxMp` / `IxM2` validator transitively when an embedded float index sits inside a binary wrapper (e.g. `IndexBinaryFromFloat`).
2. `IndexBinaryFromFloat` (`IBFf`): the deserialize site reads `idxff->index = read_index(...)` and unconditionally accepts the result. When the inner fourcc is `"null"` the field is left as `nullptr`, and every subsequent operation (`search`, `add`, `sa_code_size`, ...) dereferences it. Unlike the `IxMp` / `IxM2` case the failure is deferred to a downstream call site rather than crashing at the deserialization boundary, but the deserialized object is unusable.
Reject a null inner index immediately after the inner `read_index` returns at both sites, with a clear `FaissException` message naming the field. The new guard sits before any field access, so subsequent validators (`id_map.size() == ntotal`, `index->ntotal == ntotal`) can safely deref. The guard preserves the existing semantics for well-formed payloads: a legitimately-missing inner index never reaches `IxMp` / `IxM2` / `IBFf` deserialization in any documented `faiss` workflow. The public single-argument constructors of `IndexIDMap` / `IndexIDMap2` / `IndexBinaryFromFloat` already enforce that `index` is non-null on construction; this change makes the post-deserialization invariant match.
Reviewed By: mnorris11
Differential Revision: D106309947
fbshipit-source-id: e8455fe5ece9ec4df462f7b26d6fc8deba14a2fe
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5238
`ZnSphereCodecRec` is constructed at `IndexLattice` deserialization time and populates a decode cache by calling `decode()` once per cache entry. The existing in-codec guard caps total cache float entries at `2^27 / dimsub`, which bounds the cache memory footprint but not the CPU cost of populating it. For small `dimsub` the cap permits up to roughly `2^27` `decode()` invocations, each doing `O(dim * log(r2))` work, which can run well past any reasonable load-time budget. A corrupt or maliciously constructed serialized index file can choose parameters that pass every existing structural check (`d`, `nsq`, `dsq`, `r2 > 0`, `code_size`) yet still produce this load-time CPU blowup.
Add a configurable per-payload `r2` ceiling mirroring the existing `deserialization_loop_limit` and `deserialization_vector_byte_limit` APIs. `get_deserialization_lattice_r2_limit()` and `set_deserialization_lattice_r2_limit()` expose the knob; the default of `0` preserves existing behavior so library callers see no change. `read_index_up` rejects `IndexLattice` payloads whose `r2` exceeds the limit before constructing the `IndexLattice` (and hence before invoking the underlying `ZnSphereCodecRec`), raising a `FaissException` with a message naming the limit.
The new limit is scoped specifically to `r2` rather than reused from `deserialization_loop_limit` because the cost-per-unit of these fields is fundamentally different. `nlist` drives `O(nlist)` allocations and legitimately reaches the thousands in real workloads; `r2` drives polynomial decode-cache build work and stays in the tens for any documented `IndexLattice` configuration (`ZnSphereCodecAlt` hardcodes `r2 = 14` for its fallback codec). Reusing the existing limit would either be a no-op for the `r2` cost class if sized for `nlist`, or incorrectly reject legitimate IVF workloads if sized for `r2`.
Reviewed By: mnorris11
Differential Revision: D106309812
fbshipit-source-id: 360b406d2b178e13cd6ad9671207c431563bd8af
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5204
read_index_up reads the SVS storage_kind field from two distinct fourcc paths — the IndexSVSVamana family (ILVQ/ISVL/ISVD/ISV2) and the IndexSVSIVF family (ISIQ/ISIL/ISID). The Vamana branch read the value into a plain int, range-checked it against [0, SVS_count), and then cast to SVSStorageKind, so a malformed payload was rejected at the deserialization boundary with a clear error. The IVF branch read the value directly into the SVSStorageKind enum field with no range check. A corrupt or maliciously constructed payload could store an out-of-range enum value and continue reading several more fields; the bad value was only noticed later from to_svs_storage_kind() inside IndexSVSIVF::deserialize_impl, after the SVS runtime load had been entered, producing a less actionable error from a deeper site.
Replace the open-coded Vamana check and the missing IVF check with a single static read_svs_storage_kind(IOReader*) helper in index_read.cpp. The helper performs the four-byte read and the [0, SVS_count) validation in one place and returns the validated enum, so each call site collapses to a one-line assignment. Future SVS read sites cannot diverge in their validation strategy without going around this helper. The helper preserves the existing "invalid SVS storage_kind=N (must be in [0, M))" error message, so no externally-visible error string changes.
Reviewed By: mnorris11
Differential Revision: D104481541
fbshipit-source-id: 7e67d84b93ed142fcac39336c6bd6fdeda48cdb5
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5203
IndexIDMap::search_ex translates each label returned by the inner index through this->id_map. The outer IDMap header and the inner index header carry independent ntotal fields on disk, and the existing read_index check only enforces id_map.size() == idxmap->ntotal — the inner index's ntotal is not cross-validated. When the two disagree, the translation loop can index past the end of id_map. With id_map empty its data() pointer can be null, so the offending read dereferences a null pointer and aborts under ASAN; without ASAN the access is a heap-buffer overflow. The same gap exists for the binary counterpart at the IBMp/IBM2 read site.
Two layers of protection:
1. read_index now validates idxmap->index->ntotal == idxmap->ntotal in addition to the existing id_map.size() == ntotal check, for both IxMp/IxM2 and IBMp/IBM2. The new check rejects corrupt or maliciously constructed serialized payloads that pass each per-index validator individually but disagree at the IDMap layer.
2. IndexIDMap::search_ex and IndexIDMap::range_search bound-check each translated label against id_map.size() and remap out-of-range labels to the -1 "no result" sentinel rather than indexing past the end of id_map. This is needed because the same invariant can be transiently violated by ordinary in-memory code paths outside of deserialization — for example mid-construction (vectors added directly to the inner index before the IDMap layer has caught up), or after a recoverable allocation failure in add_with_ids_ex, add_sa_codes, or merge_from leaves the IDMap behind the inner index. The added comparison sits adjacent to the existing label sign check on the search hot path and adds a single branch.
Reviewed By: mnorris11
Differential Revision: D104468890
fbshipit-source-id: d7323adb62b9c5c18173fd43f029a8adbe4ba986
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5187
ProductQuantizer::set_derived_values only checks d % M == 0, which is satisfied by any M when d == 0. A corrupt or maliciously constructed serialized index can therefore carry an enormous M alongside an empty centroids vector and pass read_ProductQuantizer's existing centroids validation. Downstream allocations sized M * ksub — most notably the r_norms vector in initialize_IVFPQ_precomputed_table when an IVFPQ is precomputed at deserialization time — then exceed std::vector::max_size() and raise std::length_error. In code paths invoked from noexcept callers this aborts the process via std::terminate.
Two layers of protection:
1. read_ProductQuantizer now bounds M * ksub by the configurable deserialization vector byte limit, mirroring the existing d * ksub check that protects the centroids vector. This caps every PQ-derived M * ksub allocation (residual norms, search-time distance tables) at the same byte budget the caller has already chosen for vector deserialization, and rejects pathological M values up front with a FaissException rather than letting them propagate to allocation sites.
2. initialize_IVFPQ_precomputed_table now uses mul_no_overflow when computing table_size = pq.M * pq.ksub * nlist * sizeof(float). The existing precomputed_table_max_bytes guard depended on this product not wrapping size_t; with raw multiplication a sufficiently large M, ksub, or nlist silently wrapped to a small value, bypassed the guard, and proceeded to the std::vector::max_size() failure described above. Switching to mul_no_overflow makes the guard work correctly for any input that reaches it, and also defends the in-memory IVFPQ construction paths (IndexIVFPQ::train, IndexHNSW post-init, IndexIVFPQFastScan) without changing their behavior on well-formed inputs.
Reviewed By: mnorris11
Differential Revision: D104129118
fbshipit-source-id: c273aecd22572d734f535df98127dbb7a437ac4b
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5151
Several index types read code_size directly from the serialized stream independently of the quantizer parameters that determine its correct value. When the stored code_size is corrupt but ntotal is 0, the existing consistency check (codes.size() == ntotal * code_size) passes trivially. A subsequent search then allocates (code_size * sizeof(float)) bytes in GenericFlatCodesDistanceComputer, which can trigger an OOM exception.
Two layers of protection:
1. Cross-validate the deserialized code_size against the quantizer-derived value for all index types that read code_size from the stream: IndexResidualQuantizer, IndexLocalSearchQuantizer, IndexProductResidualQuantizer, IndexProductLocalSearchQuantizer, IndexIVFAdditiveQuantizer, IndexIVFScalarQuantizer, IndexLSH, and Index2Layer. The quantizer code_size is computed from validated parameters via set_derived_values() and is always authoritative.
2. For IndexLattice, where code_size is derived from constructor parameters (scale_nbit, lattice_nbit, nsq) rather than read from the stream, validate that code_size does not exceed the uncompressed vector size (d * sizeof(float)). IndexLattice is a lossy compressor, so its code_size must always be smaller than the uncompressed representation. A corrupt scale_nbit can overflow the total_nbit computation, producing a code_size that wraps to a huge value; this bound catches that before any allocation is attempted.
Reviewed By: mnorris11
Differential Revision: D102360605
fbshipit-source-id: 1f0e7262a0e0e4566d7813b7da1bf0102e6fd9bf
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5147
Index2Layer deserialization reads its sub-quantizer via read_index() which allocates a new Index on the heap, but never sets q1.own_fields = true. Since Level1Quantizer::own_fields defaults to false, the deserialized Index2Layer does not own its quantizer and never frees it — leaking the allocation both on the normal path (when the Index2Layer is eventually destroyed) and on the error path (when a subsequent deserialization step throws and the partially-constructed Index2Layer is cleaned up).
Every other index type in read_index_up that reads a sub-index via read_index sets own_fields = true afterwards (IndexIVF, IndexPreTransform, IndexIDMap, IndexRefine, IndexHNSW, IndexNSG, IndexNNDescent, etc.). Index2Layer was the only one missing it.
Reviewed By: mnorris11
Differential Revision: D102357926
fbshipit-source-id: 66a64ebe406c27aeebe45924028670227feeb9d0
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5117
Add deserialization byte limit checks before vector::resize calls in read_InvertedLists_up() for both ArrayInvertedListsPanorama ("ilpn") and ArrayInvertedLists ("ilar") paths. Previously, per-list sizes read from serialized data were used directly in .resize() calls without checking against get_deserialization_vector_byte_limit(). The READVECTOR macro enforces this limit, but explicit .resize() calls bypassed it.
Also add mul_no_overflow protection for the ilpn codes allocation (num_elems * code_size) which previously had no overflow check.
Reviewed By: mnorris11
Differential Revision: D101260923
fbshipit-source-id: 24287740642cc9647115676c71508faf8bf8f48e
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5115
Add deserialization-time validation for VectorTransform dimension invariants that are enforced by constructors but not by deserialization:
1. NormalizationTransform (VNrm): Require d_in == d_out. The constructor enforces this (both set to d), but deserialization reads them independently. A crafted file with d_in > d_out causes memcpy in apply_noalloc to overflow the output buffer (allocated as n * d_out floats but copied as n * d_in).
2. CenteringTransform (VCnt): Same d_in == d_out invariant.
3. IndexPreTransform (IxPT) chain consistency: Validate that chain[0].d_in == index.d, chain[i].d_in == chain[i-1].d_out for consecutive transforms, and chain.back().d_out == sub_index.d. Without this, mismatched dimensions between transforms cause out-of-bounds reads when one transform produces fewer elements than the next expects.
Reviewed By: mnorris11
Differential Revision: D101244181
fbshipit-source-id: fbe88bf63d42381297319d4125e750f6d47bc333
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5114
Add FAISS_THROW_IF_NOT(is_trained) to IndexIVF::search(), IndexIVF::search_preassigned(), IndexIVF::range_search(), and IndexIVF::range_search_preassigned(), mirroring the existing check in IndexScalarQuantizer::search(). This prevents querying untrained IVF indexes deserialized from corrupt data where the ScalarQuantizer trained vector is empty.
The existing deserialization validation in read_ScalarQuantizer correctly allows untrained indexes (is_trained=false with empty trained) to be deserialized, since these are legitimately produced by index_factory before training. However, IndexIVF search methods lacked the is_trained guard that IndexScalarQuantizer::search() has, allowing a deserialized untrained IndexIVFScalarQuantizer to be queried, which causes null-deref in QuantizerTemplate when it indexes into the empty trained vector.
Reviewed By: mnorris11
Differential Revision: D101243973
fbshipit-source-id: eca68dc82e5cca37d4c461b735c5d59a66349248
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5113
Add validation that IndexHNSW2Level (fourcc "IHN2") has storage of an appropriate type, both at deserialization time and at search time.
IndexHNSW2Level::search() uses dynamic_cast to dispatch between Index2Layer and IndexIVFPQ storage types. When storage is null or a different type (e.g. IndexFlat from corrupt serialized data, or a programmatically misconfigured index), the dynamic_cast returns nullptr which is then unconditionally dereferenced, causing a segfault.
Deserialization-time fix: After reading the HNSW storage sub-index for IHN2, validate that storage is non-null and is either Index2Layer or IndexIVFPQ.
Search-time defense-in-depth: Add a FAISS_THROW_IF_NOT check on the dynamic_cast result in IndexHNSW2Level::search() before dereferencing. This protects against programmatically constructed indexes that bypass deserialization validation.
Reviewed By: mnorris11
Differential Revision: D101243603
fbshipit-source-id: f3d75c1b19e68bf8539c55877c94749ef2899445
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5112
Add validation in read_ivf_header() to reject a null quantizer sub-index read from serialized data. The IVF deserialization reads the quantizer via read_index(), which returns nullptr when the stream contains the "null" fourcc. A null quantizer is fundamentally invalid for any IVF index type. Without this check, downstream code (e.g. initialize_IVFPQ_precomputed_table, IndexIVF::search) dereferences the null pointer.
This single validation protects all IVF index types that share read_ivf_header: IndexIVFFlat, IndexIVFPQ, IndexIVFScalarQuantizer, IndexIVFAdditiveQuantizer, and others.
Reviewed By: mnorris11
Differential Revision: D101236489
fbshipit-source-id: d9eb6759024ee2a4a59b838367ebf9299759ff23
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5104
Add deserialization-time validation for the `k_factor` field in
IndexRefine (IxRF), IndexRefinePanorama (IxRP), and IndexIVFPQR
(IwQR) to reject corrupted or malicious index files that would
cause out-of-memory crashes during search.
`k_factor` is read from the serialized file via `READ1()` without
validation. At search time, `k_base = k * k_factor` drives
allocations of `n * k_base` elements for labels and distances
arrays. A crafted file with an enormous `k_factor` value causes
these allocations to exhaust available memory.
Fix: Add validation after both `READ1(k_factor)` call sites,
rejecting values outside the finite range [1, 1000]. The cap
follows the same pattern as `beam_factor` validation for
`ResidualCoarseQuantizer`. The value 1000 provides ample headroom
beyond any known usage (defaults are 1 for IndexRefine and 4 for
IndexIVFPQR; AutoTune explores powers-of-two up to 64) while
blocking OOM from crafted files.
Reviewed By: mnorris11
Differential Revision: D100855031
fbshipit-source-id: ead0d3a89dd8368699f543ecc79acfe09168442b
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5056
Add deserialization-time validation for FastScan `M2` across all six
FastScan index types to reject corrupted or malicious index files that
would cause heap buffer overflows during search.
During normal construction, `M2 = roundup(M, 2)` is an invariant
maintained by `init_fastscan()`. During deserialization, `M2` is read
directly from the file and was not validated. A corrupted file with
`M2 < M` causes `compute_quantized_LUT` to write `M * ksub` bytes
into a buffer sized for `M2 * ksub` bytes, producing an out-of-bounds
write. The `memset` that zeroes padding from M to M2 additionally
underflows as an unsigned subtraction when `M2 < M`.
1. **Added `validate_fastscan_fields()` helper**:
Consolidates all FastScan field validation into a single function:
M > 0, ksub > 0, bbs > 0 and 32-aligned, M2 == roundup(M, 2),
and overflow checks for ksub * M and ksub * M2.
2. **Non-IVF FastScan paths (already had partial validation)**:
Replaced inline checks in IndexPQFastScan, IndexAdditiveQuantizer-
FastScan, and IndexRaBitQFastScan with calls to the new helper,
adding the missing M2 consistency check.
3. **IVF FastScan paths (had no validation at all)**:
Added validation to IndexIVFPQFastScan, IndexIVFAdditiveQuantizer-
FastScan, and IndexIVFRaBitQFastScan, which previously had zero
checks on any FastScan fields.
Reviewed By: mnorris11
Differential Revision: D99738294
fbshipit-source-id: 8e334993b0e8b4375f9ec173c20754e301b7c9f6
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5055
Add deserialization-time validation for the `storage_kind` field in SVS
index types (IndexSVSVamana, IndexSVSVamanaLVQ, IndexSVSVamanaLeanVec)
to reject corrupted or malicious index files before they can cause
crashes.
1. **Read `storage_kind` as int and range-check before cast**:
`storage_kind` was previously read directly into the `SVSStorageKind`
enum via `READ1`, which is undefined behavior for out-of-range values.
Now read into a temporary `int`, validate the value is in
`[0, SVS_count)`, and only then cast to `SVSStorageKind`. This rejects
invalid values at deserialization time with a `FaissException` instead
of reaching `to_svs_storage_kind()` where the `default` branch calls
`FAISS_ASSERT(false)` and aborts the process.
2. **Add `SVS_count` sentinel to `SVSStorageKind` enum**:
Follows the convention used by `QT_count`, `ST_count`, and
`DMT_count` in other FAISS enums. The deserialization validation
uses this sentinel so it automatically stays correct when new
storage kinds are added.
Reviewed By: mnorris11
Differential Revision: D99722676
fbshipit-source-id: 557cf91a963d1d93171fea2a67ba99f19b9b3420
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5040
Add the missing `key < nlist` upper-bound check in
`IndexIVF::search1()`, which was the only IVF search entry point
lacking this validation. The other two paths —
`search_preassigned()` and `range_search_preassigned()` — already
had this check.
Also add deserialization acceptance tests verifying that IVF indexes
with various quantizer states deserialize correctly:
1. **Surplus centroids** (`ntotal > nlist`):
Produced by `shard_ivf_index_centroids()`, which distributes
all of the original quantizer's centroids across shards without
adjusting `nlist`. The search-time `key < nlist` bounds check
prevents OOB access if the quantizer returns out-of-range keys.
2. **Trained quantizer** (`ntotal == nlist`):
The normal trained IVF state.
3. **Sharded quantizer** (`0 < ntotal < nlist`):
Also produced by `shard_ivf_index_centroids()`, when the
original quantizer has `ntotal == nlist` and centroids are
split across shards.
4. **Untrained quantizer** (`ntotal == 0`):
Legitimate for custom inverted list management.
Reviewed By: mnorris11
Differential Revision: D99494237
fbshipit-source-id: 6a76b55f104b9c233dfdd2625bb0336ed8061463
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5033
Add test coverage for `validate_HNSW()` checks in `index_read.cpp` that
were previously untested. These validations were introduced in D92907607
(Feb 2026) but had no dedicated unit tests.
Five new tests in `test_read_index_deserialize.cpp`:
1. **`HNSWNeighborNegativeId`**:
Neighbor ID of -5 (not the -1 sentinel) is rejected during deserialization.
2. **`HNSWNeighborIdTooLarge`**:
Neighbor ID >= ntotal is rejected during deserialization.
3. **`HNSWEntryPointOutOfRange`**:
Entry point >= ntotal is rejected during deserialization.
4. **`HNSWValidNeighborsSearchWorks`**:
A hand-crafted HNSW with valid neighbor data deserializes and searches
correctly, confirming the happy path through `validate_HNSW()`.
5. **`BinaryHNSWNeighborNegativeId`**:
Same as (1) for the binary HNSW index path (`IBHf` fourcc).
A `push_hnsw_with_neighbors()` helper is added to construct serialized
HNSW graphs with explicit neighbor data for use by these and future tests.
Reviewed By: mnorris11
Differential Revision: D99334840
fbshipit-source-id: 6bb38eb78ea20f7215eb0046de7398d07f14ac35
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5029
1. **Null pointer dereference in `deserialize_impl`**:
`IndexSVSVamana::deserialize_impl()` and `IndexSVSFlat::deserialize_impl()`
called `impl->load(...)` as a member function on a null `impl` pointer,
causing a SIGSEGV. The `load` function is actually a static factory method
(like `build` and `destroy`), so it should be called as
`ClassName::load(&impl, ...)` instead. This matches the pattern used in
`IndexSVSVamanaLeanVec::deserialize_training_data()` which correctly calls
`svs_runtime::LeanVecTrainingData::load(&tdata, in)`.
2. **Added post-load validation**:
Added `FAISS_THROW_IF_NOT_MSG(impl, ...)` after `load()` returns to ensure
the impl pointer was actually populated.
3. **Added tests**:
Added deserialization tests that provide invalid SVS stream data and verify
the load fails with a FaissException rather than crashing.
Reviewed By: mnorris11
Differential Revision: D99117347
fbshipit-source-id: 3165048d06dc5ab66fcf83ecba3a144605c5732f
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5028
The ZnSphereCodecRec constructor builds a decode cache by calling
decode() for every lattice point at the cache level. The number of
lattice points grows as O(r2^(dim/2)), so moderate r2 values with
higher dimensions cause excessive CPU usage.
The existing nv_cum memory cap (2^27 uint64_t entries) does not catch
this because nv_cum grows as O(r2^2) -- the decode cache is the
binding constraint for these inputs.
**Fix: Add decode cache size check in ZnSphereCodecRec constructor**:
Track cumulative decode cache entries within the existing decode
cache loop. Cap at 2^27 total floats (~512 MB), aligned with
the nv_cum memory cap which also uses 2^27 entries. This bounds
both memory and computation for all callers, not just
deserialization.
Reviewed By: mnorris11
Differential Revision: D99166802
fbshipit-source-id: 442ba2f5ea6b46309e3598b97953cf4c38fa443a
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5024
Fixes a null-deref crash in `IndexIDMap2::construct_rev_map()` when
deserializing a crafted index where `id_map.size() != ntotal`.
1. **Validation in index_read.cpp**:
Added a `FAISS_THROW_IF_NOT_FMT` check after reading the `id_map`
vector in the `IxMp`/`IxM2` deserialization path, verifying that
`id_map.size() == ntotal` before `construct_rev_map()` is called.
The binary index path (`IBMp`/`IBM2`) already had this check.
2. **Test in test_read_index_deserialize.cpp**:
Added `IndexIDMap2IdMapSizeMismatch` test that constructs a serialized
`IndexIDMap2` with `ntotal=2` but `id_map` containing 5 entries, and
verifies deserialization throws with the expected error message.
Reviewed By: mnorris11
Differential Revision: D99138339
fbshipit-source-id: cfcc5a4daf83bb6fbda324d15f946340eee259c0
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5023
Add deserialization-time validation to catch AdditiveQuantizer dimension
mismatches that would cause crashes during search operations.
1. **AQ dimension vs index dimension cross-check**:
Added `validate_aq_dimension_match()` helper that verifies `aq.d == idx.d`
for all AdditiveQuantizer-based index types. Without this check, a
corrupted index where AQ.d != Index.d causes `compute_LUT()` to read
codebooks out of bounds (query vectors are sized for Index.d while
codebooks are laid out for AQ.d).
2. **Coverage across all AQ index types**:
- IndexResidualQuantizer (IxRq/IxRQ)
- IndexLocalSearchQuantizer (IxLS)
- IndexProductResidualQuantizer (IxPR)
- IndexProductLocalSearchQuantizer (IxPL)
- ResidualCoarseQuantizer (ImRQ)
- IndexAdditiveQuantizerFastScan (IRfs/ILfs/IPRf/IPLf)
- IndexIVFAdditiveQuantizerFastScan (IVLf/IVRf/NPLf/NPRf)
- IndexIVFAdditiveQuantizer (IwLS/IwRQ/IwPL/IwPR)
Reviewed By: mnorris11
Differential Revision: D99137777
fbshipit-source-id: bb4201f32b1268d66907f9a3f34eff869c80dc9c
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5020
The prior deserialization validation for training data in
`read_ScalarQuantizer` skipped size checks when `trained` was empty,
allowing inconsistent index state to be materialized.
This diff hardens `read_ScalarQuantizer` in three ways:
1. **Trained size validation**:
Replace the `if (!ivsc->trained.empty())` guard with two-branch
validation. For qtypes that require training data
(UNIFORM/NON_UNIFORM), an empty `trained` vector is only accepted
when the index header has `is_trained == false` (a legitimate
untrained index). When `is_trained == true`, the trained vector
must match the expected size. Qtypes that need no training (fp16,
bf16, 8bit_direct*) still require `trained.size() == 0`.
2. **Dimension cross-validation**:
Validate that `ScalarQuantizer::d` matches `Index::d` from the index
header. A mismatch indicates a corrupted file and could cause the
trained size check to accept a wrong size.
3. **is_trained consistency**:
When training data is present, validate that the index header
`is_trained` flag is true. A false flag with valid training data
indicates file corruption.
To support these cross-validations, `read_ScalarQuantizer` now takes
a `const Index&` parameter for access to the index header fields.
Untrained indices (created via `index_factory` but not yet trained)
are legitimately serialized with `is_trained=false` and an empty
`trained` vector. The existing `FAISS_THROW_IF_NOT(is_trained)` guards
in `IndexScalarQuantizer::search()` and related methods already prevent
querying such indices, so no additional query-time validation is needed.
Reviewed By: mnorris11
Differential Revision: D98924927
fbshipit-source-id: cbb9906e031c871cb8d2baa7bb9cb9d2c6834e03
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5019
The `qbs` field (query block size) is an `int` read from serialized
FastScan index files. A negative value causes `pq4_qbs_to_nq()` to loop
infinitely at search time because arithmetic right shift on a signed `int`
preserves the sign bit, preventing the loop variable from reaching zero.
Validate `qbs >= 0` immediately after reading it during deserialization in
all three affected index formats:
- `IndexPQFastScan` (fourcc "IPfs")
- `IndexAdditiveQuantizerFastScan` (fourccs "IRfs", "ILfs", "IPLf", "IPRf")
- `IndexIVFAdditiveQuantizerFastScan` (fourccs "IVRf", "IVLf", "NPLf", "NPRf")
This catches malformed or adversarial index files at load time with a clean
`FaissException`, rather than hanging during search.
`IndexIVFPQFastScan` and `IndexIVFRaBitQFastScan` are not affected because
they only serialize `qbs2`, which is `size_t` (unsigned).
Reviewed By: mnorris11
Differential Revision: D98815835
fbshipit-source-id: 40c5f39c84bcc553b7031c601832ef2c7d0a4d64
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5011
Add a deserialization-time check in `read_index_binary` that rejects
`IndexBinaryHash` data where `b` exceeds `code_size * 8`. Without
this validation, a malformed index can cause `BitstringReader::read()`
to access past the allocated code buffer during search, resulting in a
heap-buffer-overflow.
Reviewed By: mnorris11
Differential Revision: D98811392
fbshipit-source-id: 6847bce77be6edb66d87bd28a770c2af615868d3
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5009
Add deserialization-time validation for all three FastScan index types
(IndexPQFastScan, IndexAdditiveQuantizerFastScan, IndexRaBitQFastScan)
in `index_read.cpp`. A crafted or corrupted index file could previously
supply invalid quantizer parameters that would cause crashes or undefined
behavior when the index was later used for search.
The following checks are added at deserialization time for each FastScan
variant:
1. **M > 0 and ksub > 0**:
These quantizer dimensions are used as divisors and loop bounds in
`compute_float_LUT` and search kernels. Zero values would cause
division-by-zero or empty allocations followed by out-of-bounds access.
2. **bbs > 0 and bbs % 32 == 0**:
The block batch size (`bbs`) is used as a divisor (e.g. `ntotal2 /
bbs`) in search and is assumed to be aligned to 32 by the SIMD packing
logic. A zero value causes division-by-zero; a misaligned value causes
incorrect memory access patterns.
3. **ksub * M and ksub * M2 overflow**:
These products are used to size lookup tables and scoring buffers. A
corrupted file could set M2 to a huge value, causing `size_t` overflow
and subsequent under-sized allocations with out-of-bounds writes. The
existing `mul_no_overflow` helper is used to detect this.
Reviewed By: mnorris11
Differential Revision: D98787875
fbshipit-source-id: 76656f1c1bea4afe2d268dbd46af5a6b3a7b5932
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5008
Add deserialization-time validation in `index_read.cpp` for
AdditiveQuantizer-based indices, catching corrupt or malicious index
files early rather than crashing during search.
1. **Codebooks divisibility check in `read_AdditiveQuantizer`**:
`codebooks.size()` must be a positive multiple of
`total_codebook_size`. This catches obvious corruption without
needing the effective dimension.
2. **`validate_codebooks_size` helper**:
Checks `codebooks.size() >= effective_d * total_codebook_size`.
Called from `read_ResidualQuantizer` and `read_LocalSearchQuantizer`
with `d`, and from `read_ProductAdditiveQuantizer` with
`d / nsplits`.
3. **`d > 0` validation**:
Reject zero dimension in `read_AdditiveQuantizer`.
4. **ProductAdditiveQuantizer checks**:
- `d % nsplits == 0` in `read_ProductAdditiveQuantizer`
- Sub-quantizer `d == d_sub` in `read_ProductResidualQuantizer`
and `read_ProductLocalSearchQuantizer`
Existing tests updated to write correctly-sized codebooks so they
continue to reach their intended validation points.
Reviewed By: mnorris11
Differential Revision: D98767295
fbshipit-source-id: 95d8debbbb0f3822bae087a546a573e5c2c3c9dd
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5007
Add input validation to VectorTransform deserialization to defend against
invalid or malicious serialized data:
1. **Negative dimension checks**:
Reject `d_in < 0` or `d_out < 0` immediately after reading, before
any allocations depend on these values.
2. **Byte limit check on `d_in * d_out`**:
Use `mul_no_overflow` to safely compute the product and verify it
does not exceed the deserialization vector byte limit. This prevents
allocation of excessively large matrices from corrupt data. The
existing per-type-specific `d_out > 0` check in HadamardRotation is
removed as redundant (the power-of-2 check already implies `d_out > 0`).
3. **IndexIVFIndependentQuantizer dimension consistency**:
When an `IndexIVFIndependentQuantizer` has a VectorTransform, validate
that `vt->d_in == indep->d` and `vt->d_out == index_ivf->d`. A mismatch
indicates a corrupt or tampered index file and would cause incorrect
behavior at query time.
Reviewed By: mnorris11
Differential Revision: D98683628
fbshipit-source-id: 0101ee1aa956c346b614d26569c20d6b442210b4
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/4997
Collectively validate ntotal, beam_factor, and max_beam_size during
deserialization to ensure allocation limits are honored, even when
those allocations occur in search context (ResidualCoarseQuantizer::search()).
Specifically, search() allocates O(beam_size * M * n) int32_t elements where
beam_size is derived from min(k * beam_factor, ntotal). With an adversarial
ntotal of 2^60 or a beam_factor of 1e10, these allocations exceed max_size()
and throw an uncaught std::length_error, terminating the process.
This diff adds three layers of protection:
1. read_ResidualQuantizer(): Validates max_beam_size > 0 and that
max_beam_size * M * sizeof(int32_t) fits within the configurable
deserialization vector byte limit.
2. ResidualCoarseQuantizer ("ImRQ") deserialization path:
Validates beam_factor <= 1000 to prevent int overflow in beam_size
computation, and validates ntotal * M * sizeof(int32_t) against the
byte limit since search() can allocate O(ntotal * M) when beam_size
is capped to ntotal.
Why 1000? The default beam_size is 4 (expand candidate serarch by 4x).
A value of even 1000 makes no practical sense as in most cases it will
be equivalent to an exhaustive search (beam_size = -1), but is sufficient
to prevent overflow.
3. ResidualCoarseQuantizer::search():
Replaces bare arithmetic with mul_no_overflow() for the codes and
beam_distances allocations, so any overflow throws a catchable
FaissException instead of silently wrapping and causing undefined
behavior.
Reviewed By: mnorris11
Differential Revision: D98169828
fbshipit-source-id: a8cd067ebb0fcb3ca25bed80a6d2ccc0cdbc0a32
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/4984
Add two input validation checks when deserializing IndexBinaryMultiHash
("IBHm") from untrusted index files:
1. **b > 0**:
The `b` field specifies the number of bits extracted per hash
map. During search, `BitstringReader::read(b)` is called for each hash
map to extract hash values from query vectors. If `b` is zero or
negative, `read(0)` produces undefined hash values and the hash lookup
silently returns garbage results. This mirrors the existing `b > 0`
validation already present for IndexBinaryHash ("IBHh").
2. **nhash * b <= code_size * 8**:
The search path in `search_1_query_multihash()` creates a single
`BitstringReader` over the query vector and sequentially reads `b` bits
`nhash` times, consuming `nhash * b` total bits. The vector buffer is
`code_size` bytes ('= d/8'), providing `code_size * 8` bits. If `nhash
* b` exceeds this, the reader advances past the buffer boundary,
triggering an out-of-bounds read. The constructor already enforces
`nhash * b <= d`, but deserialization skipped this check, allowing
a crafted file to bypass the constraint.
Both checks use FAISS_THROW_IF_NOT_FMT to throw descriptive exceptions
with the invalid values, consistent with the existing nhash > 0 validation
in the same code path.
Reviewed By: mnorris11
Differential Revision: D97990226
fbshipit-source-id: 62d2f8f79e02f6d473676e90981d201ef57a2e00
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/4980
IndexBinaryIVF::search dereferences invlists via
invlists->prefetch_lists() without any prior null check. The invlists
pointer can legitimately be nullptr through multiple paths:
1. Default construction: IndexBinaryIVF::invlists is initialized to
nullptr (IndexBinaryIVF.h:35) and the default constructor is
defaulted, leaving it null.
2. Deserialization with IO_FLAG_SKIP_IVF_DATA: When this flag is set,
read_InvertedLists_up returns nullptr, and the binary IVF reader
at index_read.cpp:2130 stores it directly (ils.release() on a null
unique_ptr yields nullptr). This is intentional — the flag allows
loading index metadata without the (potentially large) inverted
list data.
3. Deserialization with fourcc "il00": The serialized data explicitly
indicates inverted lists were not stored with the IVF object
(index_read.cpp:453), returning nullptr.
Without this check, any of these cases would segfault on the
invlists->prefetch_lists() call. With this check, the caller receives
a clear FaissException explaining the misconfiguration.
This mirrors the existing null check in the float IndexIVF::search
(IndexIVF.cpp:312) which already guards against the same condition.
Reviewed By: mnorris11
Differential Revision: D97848088
fbshipit-source-id: 406abb870dfa093ffc3ce49ba65904bebe4763a2
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/4979
These checks guard against invalid runtime state that would otherwise
cause undefined behavior or confusing errors during search operations.
They are not redundant with deserialization-time checks — they protect
against programmatic misuse at query time.
Specific validations and their consequences if absent:
1. get_distance_computer: Verify storage is IndexBinaryFlat via
dynamic_cast. Without this, a nullptr or wrong-type storage would
pass a garbage pointer to dispatch_HammingComputer, leading to
memory corruption. Deserialization already validates storage type,
but this also guards against programmatic construction with the
wrong storage type. Converted from FAISS_ASSERT (which aborts or
is compiled out) to FAISS_THROW_IF_NOT_MSG for safe error
propagation.
2. ntotal > 0 in base_level_only search: The entrypoint selection loop
constructs std::uniform_int_distribution<idx_t>(0, ntotal - 1).
When ntotal == 0, this becomes distribution(0, -1) — undefined
behavior per the C++ standard ([rand.dist.uni.int]). An empty
index is valid state that can arise after deserialization or
construction, but searching it in base_level_only mode must be
rejected.
3. num_base_level_search_entrypoints > 0: This is a public, user-
settable parameter (default 32). If set to 0, the entrypoint
selection loop at line 388 never executes, nearest[i] stays at -1,
and the downstream FAISS_THROW at line 397 fires with the
misleading message "Could not find a valid entrypoint" — obscuring
the actual misconfiguration. The early check provides an actionable
error message naming the offending parameter.
Reviewed By: mnorris11
Differential Revision: D97845054
fbshipit-source-id: d57d364889ce6a6bcfda741d6541c4cdbac44e85
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/4978
Add validation checks for binary index types at deserialization time.
Without these checks, malformed index files can cause out-of-bounds
memory accesses, null pointer dereferences, and type confusion during
subsequent search or add operations.
- **`read_index_binary_header`** —
strengthen `d >= 0` to `d > 0 && d % 8 == 0` and add `code_size == d /
8`. Binary indices represent vectors as bit arrays with one bit per
dimension; all operations compute byte offsets via `d / 8`. A dimension
that is not a multiple of 8 causes integer truncation in `code_size =
d / 8`, making every vector stride too short and causing out-of-bounds
reads/writes in `add()`, `search()`, and distance computations. A `d =
0` causes division by zero. A mismatched `code_size` has the same effect
— vector data is strided by `code_size` bytes (`x + i * code_size`),
so a wrong value means every vector access is at the wrong offset.
- **`read_binary_hash_invlists`** —
validate `vecs.size() == ids.size() * code_size`. Each inverted list
entry in `IndexBinaryHash` pairs a vector ID with `code_size` bytes of
raw vector data. If `vecs` is shorter than `ids.size() * code_size`,
distance computations that iterate through the vector bytes using the
ID count as stride will read past the end of the `vecs` buffer.
- **`read_binary_multi_hash_map`** —
validate `id < ntotal` for each deserialized ID. IDs in the multi-hash
map index into the underlying `IndexBinaryFlat` storage array. An
out-of-range ID causes an out-of-bounds read when the vector is looked
up during search, either returning garbage distances or crashing.
- **IndexBinaryHNSW (IBHf) and IndexBinaryHNSWCagra (IBHc)** —
validate three consistency invariants:
1. `hnsw.levels.size() == ntotal`: The HNSW graph stores one level entry
per vector. A mismatch causes out-of-bounds reads in the levels array
during graph traversal, which runs inside OpenMP parallel regions.
2. Storage is `IndexBinaryFlat`: `get_distance_computer()` does
`dynamic_cast<IndexBinaryFlat*>(storage)` and dereferences the result.
A non-flat storage type yields a null pointer, crashing the process.
3. `storage->ntotal == ntotal`: HNSW graph node IDs index into the
storage. If storage has fewer vectors than the graph expects,
search traversal accesses out-of-bounds storage entries.
Reviewed By: mnorris11
Differential Revision: D97819033
fbshipit-source-id: 88787c05b9edae522217a65c86eacc3488696736
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/4983
Validate that the `qb` (query quantization bits) field is valid for all
6 RaBitQ deserialization points (IndexRaBitQ, IndexRaBitQFastScan,
IndexIVFRaBitQ, IndexIVFRaBitQFastScan, and their multi-bit variants).
The `qb` parameter controls the scalar quantization precision of query
vectors in the RaBitQ algorithm.
- **'FastScan' formats: `qb = 0` is invalid for the quantized path.**
The quantized distance computer (`RaBitQDistanceComputerQ`) computes
`max_code = (1 << qb) - 1` and then divides by `delta = (v_max - v_min)
/ max_code`. With `qb = 0`, `max_code` is 0, producing a division by
zero. The `Z_MAX_BY_QB` lookup table is also indexed by `qb - 1`, so
`qb = 0` causes an out-of-bounds read from the 8-element array.
- **All other formats: `qb = 0` is valid and means "use unquantized float32 distance computation."**
When `qb == 0`, the code returns a `RaBitQDistanceComputerNotQ` that
operates on raw fp32 values without any scalar quantization of the query. Ixrq is the original/default 1-bit RaBitQ format and the only format that supports `qb==0`.
- **`qb > 8` overflows `uint8_t` quantized values.**
The query vector is scalar-quantized into `rotated_qq`, a
`std::vector<uint8_t>`, with values in `[0, (1 << qb) - 1]`. For `qb > 8`,
`(1 << qb) - 1` exceeds 255 and the cast to `uint8_t` silently
truncates, producing incorrect quantized values. The bit-rearrangement
loop in `set_query` also extracts individual bits via `rotated_qq[idim]
& (1 << iv)` for `iv` in `[0, qb)` — with `qb > 8` this shifts beyond
the 8-bit width of `uint8_t`, yielding undefined behavior.
- **Existing runtime checks are `FAISS_ASSERT`, not `FAISS_THROW`.**
Both `RaBitQDistanceComputerQ::set_query()` and
`compute_query_factors()` have `FAISS_ASSERT(qb > 0)` / `FAISS_ASSERT(qb
<= 8)` checks, but these compile to no-ops in release builds and call
`abort()` in debug builds. In either case, when reached inside an OMP
parallel region (as happens during search), `abort()` terminates the
entire process without cleanup. Validating at deserialization time
converts this into a catchable exception before the invalid value can
reach any parallel code path.
Reviewed By: mnorris11
Differential Revision: D97815297
fbshipit-source-id: 53fce4b8befe14bfd301451a125684c03e939b2b
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/4981
Validate internal consistency of VectorTransform data at deserialization
time to prevent crashes from malformed index files, leading to out-of-bounds
reads/writes.
- **HadamardRotation** — validate `d_out > 0` before calling `init()`.
`init()` calls `generate_signs(seed, d_out, …)` which `resize()`s three
`std::vector<float>` to `d_out` elements and then iterates over them.
A zero `d_out` would violate the `p > 0` assert inside `generate_signs`.
- **RemapDimensionsTransform** — validate `map.size() >= d_out`.
`apply_noalloc` loops `for (j = 0; j < d_out; j++) xt[j] = … x[map[j]]`,
indexing into `map` with `j`. If `map` is shorter than `d_out`, the loop
reads past the end of the vector, producing an out-of-bounds heap read
that can crash or leak memory contents.
- **CenteringTransform** — validate `mean.size() >= d_in`.
`apply_noalloc` loops `for (j = 0; j < d_in; j++) *xt++ = *x++ - mean[j]`,
so a `mean` vector shorter than `d_in` causes an out-of-bounds read on
every input vector processed.
- **ITQTransform** — validate `mean.size() >= d_in`.
`apply_noalloc` subtracts `mean[j]` for `j` in `[0, d_in)` before
passing the normalized data to the PCA+ITQ chain. The same out-of-bounds
read hazard as CenteringTransform applies here.
Reviewed By: mnorris11
Differential Revision: D97809504
fbshipit-source-id: da82d1c1b62d06b09b387d11a7934091b6754daf
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/4951
Add null-pointer checks for `invlists` in `IndexIVF::add_with_ids()`,
`IndexIVF::search()`, and `IndexIVF::search_preassigned()`.
When an IVF index is deserialized with `IO_FLAG_SKIP_IVF_DATA`, the
`invlists` pointer is null. Calling add or search on such an index
would dereference the null pointer, causing a crash. These checks
provide a clear error message instead.
Reviewed By: mnorris11
Differential Revision: D96967039
fbshipit-source-id: 288ee0964b5d9fffbf617d0d826b8182a46f7de9
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/4950
When `IndexHNSWCagra::search()` is called with `base_level_only=true`
on an empty index (`ntotal == 0`), it proceeds to create entry point
vectors and access random graph nodes, causing undefined behavior.
Add an early return that fills the output arrays with sentinel values
(max float distances and -1 labels) when the index is empty, consistent
with how other FAISS index types handle empty-index queries.
Reviewed By: mnorris11
Differential Revision: D96966891
fbshipit-source-id: 0b9cded2e0e14ad6e4b0869cce4da1311e595d96
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/4949
Add validation checks for graph-based index types during deserialization:
- NNDescent: validate that `final_graph` size matches `ntotal * K` and
that all neighbor IDs are in the valid range `[-1, ntotal)`.
- HNSW: cross-check that `hnsw.levels.size()` matches the index header
`ntotal`, and that the storage sub-index `ntotal` and `d` match the
parent index.
- NSG: cross-check that `nsg.ntotal` matches the index header `ntotal`
(when built), and that the storage sub-index `ntotal` and `d` match.
- NNDescentFlat: cross-check that `nndescent.ntotal` matches the index
header `ntotal` (when built), and that the storage sub-index `ntotal`
and `d` match.
These checks catch corrupt or maliciously crafted index files that could
cause out-of-bounds memory access during graph traversal in search.
Reviewed By: mnorris11
Differential Revision: D96966646
fbshipit-source-id: a3839e84884c45221aa8608cb1f4271eea7ca05e
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/4948
Add validation that the deserialized codes vector size matches
`ntotal * code_size` for the following index types: IndexPQ,
IndexResidualQuantizer, IndexLocalSearchQuantizer,
IndexProductResidualQuantizer, and IndexProductLocalSearchQuantizer.
A mismatch between the codes vector size and the expected size from the
index header indicates a corrupt or maliciously crafted index file. Without
this check, a truncated or inflated codes vector would cause out-of-bounds
memory access during search or reconstruction operations.
Reviewed By: mnorris11
Differential Revision: D96966390
fbshipit-source-id: 37602d0343f0d79a07487c7f54ed45fd09ae45dd
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/4947
Add validation checks during index deserialization for quantizer data:
- ProductQuantizer: verify that the centroids vector size matches
`d * ksub`. A mismatch indicates a corrupt or maliciously crafted
index file and would cause out-of-bounds reads during search.
- ScalarQuantizer: verify that the trained vector size matches the
expected size for the quantizer type and dimension. The expected size
depends on the quantizer type (e.g., 2*d for per-dimension types,
2 for uniform types, 0 for direct/fp16/bf16 types).
Reviewed By: mnorris11
Differential Revision: D96965538
fbshipit-source-id: b64a2e316b5fcdf1cebefe6ffa6081bab2ecc640
Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/4928
The original intention of the `READVECTOR` and `READXBVECTOR` macros in
`io_macros.h` was to limit reads to 256GB (see comment above the previous
version of READVECTOR). However, the enforced limit was actualy based on
element count not bytes, allowing extremely large reads — requiring TBs of
memory — to pass the check and be attempted.
In this diff, represent the limit in bytes, and set the default value to
1TB. This value can be changed at runtime via the
set_deserialization_vector_byte_limit() API.
Reviewed By: mnorris11
Differential Revision: D96559593
fbshipit-source-id: 38937c912aa5f7e17c45aed6f37b1bfd1b5b3e1a
Trang Nguyen