mirror of
https://github.com/facebookresearch/faiss.git
synced 2026-10-11 22:50:00 +00:00
Alibek Zhakubayev
a238933d17
Summary: Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5444 We fix every remaining flake8 E501 (line > 80 chars) violation in fbcode/faiss/ (excluding gpu/), closing out a lint-cleanup initiative that a June 29, 2026 diff stack (9a05463576 -> d304e29b9a -> 7d140d86ca) explicitly promised but left 41% incomplete. The middle diff in that stack aligned .flake8's max-line-length to black's 80-char width and stated the remaining 277 violations after the config change were "genuine lint (E501 long strings/URLs, F401/F841/B011, etc.), addressed in later commits in this stack." The follow-up commit fixed only the F401/F403/F841/B011 categories (45 issues) and never touched E501, leaving 192 violations across 53 files live in trunk. A direct re-scan today found 201 violations across 54 files (two files gained lines since the original count). We fixed 183 violations across 52 files. Two of the originally-flagged files, benchs/bench_rabitq.py and demos/index_pq_flat_separate_codes_from_codebook.py, already carry a file-level `# flake8: noqa` directive (the second is a Bento/Grimaldi notebook export that also fails standalone py_compile due to embedded shell magic) -- flake8 ignores both files entirely, so no changes were needed there and they are excluded from this diff. Fix approach, by category: - The large majority of violations are prose: docstrings and comments rewrapped at word boundaries, preserving each file's existing indentation and docstring conventions. No wording was changed. - Code lines (function calls, f-strings, assert messages, long argument lists) were reflowed in black-compatible style: parenthesization, comma breaks, or implicit adjacent string-literal concatenation. Every touched file is confirmed unchanged by `black --line-length 80 --diff` after the edit. - A small number of genuinely unwrappable long URLs (no reasonable break point) were suppressed with `# noqa: E501` rather than force-wrapped, matching the precedent set by the original black-format diff stack. - demos/offline_ivf/offline_ivf.py: a repeated, non-trivial ratio expression used in both an assert condition and its (lazily-evaluated) failure message was hoisted into a single local variable, eliminating the duplicate computation. The expression is pure and deterministic (built from already-computed arrays), so this is behaviorally identical to the original -- not a functional change. - `arc lint -a` additionally auto-fixed a real bug it detected while processing these files: `raise NotImplemented` -> `raise NotImplementedError` in contrib/evaluation.py (the sentinel object NotImplemented is not an exception type; raising it produces a TypeError masking the intended NotImplementedError). It also added a missing trailing newline in benchs/fb/bench_index_binary_from_float.py and, in the same file, rewrote the `#!/usr/bin/env python2` shebang to `#!/usr/bin/env fbpython` -- that rewrite was reverted by hand, since the file still does `import cPickle` (a Python 2-only module), so the auto-fix would have made the shebang describe an interpreter the script cannot actually run under. Fixing the underlying py2-to-py3 migration is out of scope for this line-length-only sweep. - Rebasing onto master (this diff's base commit had drifted) surfaced two concurrent upstream fixes to `check_ref_knn_with_draws(...)` calls in tests/test_ivf_flat_panorama.py and tests/test_refine_panorama.py (an unrelated in-flight correctness fix passing `D_panorama`/`atol` instead of a stale `D_regular` arg with no `atol`). The rebase took upstream's corrected arguments and re-applied only this diff's line-wrap on top. We verified zero functional change with an AST-level check: every string and f-string literal in every touched file was extracted before and after the edit and compared with whitespace normalized, confirming no wording, value, or expression was altered beyond the intended `# noqa: E501` suffixes and the one documented ratio-hoisting simplification above. Reviewed By: mnorris11 Differential Revision: D112525458 fbshipit-source-id: e4b85d73dafcaac779f3522a383a66e490fa942a
Offline IVF
This folder contains the code for the offline ivf algorithm powered by faiss big batch search.
Create a conda env:
conda create --name oivf python=3.10
conda activate oivf
conda install -c pytorch/label/nightly -c nvidia faiss-gpu=1.7.4
conda install tqdm
conda install pyyaml
conda install -c conda-forge submitit
Run book
- Optionally shard your dataset (see create_sharded_dataset.py) and create the corresponding yaml file
config_ssnpp.yaml. You can usegenerate_config.pyby specifying the root directory of your dataset and the files with the data shards
python generate_config
- Run the train index command
python run.py --command train_index --config config_ssnpp.yaml --xb ssnpp_1B
- Run the index-shard command so it produces sharded indexes, required for the search step
python run.py --command index_shard --config config_ssnpp.yaml --xb ssnpp_1B
- Send jobs to the cluster to run search
python run.py --command search --config config_ssnpp.yaml --xb ssnpp_1B --cluster_run --partition <PARTITION-NAME>
Remarks about the search command: it is assumed that the database vectors are the query vectors when performing the search step.
a. If the query vectors are different than the database vectors, it should be passed in the xq argument
b. A new dataset needs to be prepared (step 1) before passing it to the query vectors argument –xq
python run.py --command search --config config_ssnpp.yaml --xb ssnpp_1B --xq <QUERIES_DATASET_NAME>
- We can always run the consistency-check for sanity checks!
python run.py --command consistency_check --config config_ssnpp.yaml --xb ssnpp_1B