mirror of
https://github.com/facebookresearch/faiss.git
synced 2026-10-11 22:50:00 +00:00
Rafik Saliev
42dc2862c2
Summary: This pull request adds support for memory-mapped I/O (mmap) loading of static SVS (Scalable Vector Search) indices in Faiss, enabling faster and more memory-efficient index loading. The main changes include implementing mmap-based deserialization for static Vamana and Flat indices, updating the index loading logic to use mmap when requested, and adding comprehensive tests for the new functionality. **Memory-mapped index loading support:** * Added `map_to` methods to `IndexSVSVamana` and `IndexSVSFlat` for loading static indices directly from memory-mapped files, including logic to manage memory mapping ownership and adjust file positions after mapping. **Index and file structure updates:** * Extended `IndexSVSVamana` and `IndexSVSFlat` classes to hold a reference to the mmap owner, ensuring the mapped memory remains valid for the lifetime of the index. Also added forward declarations and includes for mmap-related types. * Modified `reset` methods to release memory mappings when the index is reset, preventing resource leaks. **Testing and interface improvements:** * Enhanced test utilities and added new tests to verify correct mmap loading for various static Vamana index types, including FP32, FP16, SQI8, LVQ4x4, and LeanVec4x4 in `tests/test_svs.cpp` These changes collectively provide robust and efficient support for memory-mapped loading of static SVS indices, improving both performance and usability for large-scale vector search applications. Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5271 Reviewed By: alibeklfc Differential Revision: D108313487 Pulled By: mnorris11 fbshipit-source-id: 4ae6dc3e69363b0c792cd56ad26b01bf6a853d28