mirror of
https://github.com/facebookresearch/faiss.git
synced 2026-10-11 22:50:00 +00:00
Alibek Zhakubayev
a238933d17
Summary: Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5444 We fix every remaining flake8 E501 (line > 80 chars) violation in fbcode/faiss/ (excluding gpu/), closing out a lint-cleanup initiative that a June 29, 2026 diff stack (9a05463576 -> d304e29b9a -> 7d140d86ca) explicitly promised but left 41% incomplete. The middle diff in that stack aligned .flake8's max-line-length to black's 80-char width and stated the remaining 277 violations after the config change were "genuine lint (E501 long strings/URLs, F401/F841/B011, etc.), addressed in later commits in this stack." The follow-up commit fixed only the F401/F403/F841/B011 categories (45 issues) and never touched E501, leaving 192 violations across 53 files live in trunk. A direct re-scan today found 201 violations across 54 files (two files gained lines since the original count). We fixed 183 violations across 52 files. Two of the originally-flagged files, benchs/bench_rabitq.py and demos/index_pq_flat_separate_codes_from_codebook.py, already carry a file-level `# flake8: noqa` directive (the second is a Bento/Grimaldi notebook export that also fails standalone py_compile due to embedded shell magic) -- flake8 ignores both files entirely, so no changes were needed there and they are excluded from this diff. Fix approach, by category: - The large majority of violations are prose: docstrings and comments rewrapped at word boundaries, preserving each file's existing indentation and docstring conventions. No wording was changed. - Code lines (function calls, f-strings, assert messages, long argument lists) were reflowed in black-compatible style: parenthesization, comma breaks, or implicit adjacent string-literal concatenation. Every touched file is confirmed unchanged by `black --line-length 80 --diff` after the edit. - A small number of genuinely unwrappable long URLs (no reasonable break point) were suppressed with `# noqa: E501` rather than force-wrapped, matching the precedent set by the original black-format diff stack. - demos/offline_ivf/offline_ivf.py: a repeated, non-trivial ratio expression used in both an assert condition and its (lazily-evaluated) failure message was hoisted into a single local variable, eliminating the duplicate computation. The expression is pure and deterministic (built from already-computed arrays), so this is behaviorally identical to the original -- not a functional change. - `arc lint -a` additionally auto-fixed a real bug it detected while processing these files: `raise NotImplemented` -> `raise NotImplementedError` in contrib/evaluation.py (the sentinel object NotImplemented is not an exception type; raising it produces a TypeError masking the intended NotImplementedError). It also added a missing trailing newline in benchs/fb/bench_index_binary_from_float.py and, in the same file, rewrote the `#!/usr/bin/env python2` shebang to `#!/usr/bin/env fbpython` -- that rewrite was reverted by hand, since the file still does `import cPickle` (a Python 2-only module), so the auto-fix would have made the shebang describe an interpreter the script cannot actually run under. Fixing the underlying py2-to-py3 migration is out of scope for this line-length-only sweep. - Rebasing onto master (this diff's base commit had drifted) surfaced two concurrent upstream fixes to `check_ref_knn_with_draws(...)` calls in tests/test_ivf_flat_panorama.py and tests/test_refine_panorama.py (an unrelated in-flight correctness fix passing `D_panorama`/`atol` instead of a stale `D_regular` arg with no `atol`). The rebase took upstream's corrected arguments and re-applied only this diff's line-wrap on top. We verified zero functional change with an AST-level check: every string and f-string literal in every touched file was extracted before and after the edit and compared with whitespace normalized, confirming no wording, value, or expression was altered beyond the intended `# noqa: E501` suffixes and the one documented ratio-hoisting simplification above. Reviewed By: mnorris11 Differential Revision: D112525458 fbshipit-source-id: e4b85d73dafcaac779f3522a383a66e490fa942a
68 lines
2.0 KiB
Python
68 lines
2.0 KiB
Python
# Copyright (c) Meta Platforms, Inc. and affiliates.
|
|
#
|
|
# This source code is licensed under the MIT license found in the
|
|
# LICENSE file in the root directory of this source tree.
|
|
|
|
import numpy as np
|
|
import argparse
|
|
import os
|
|
|
|
|
|
def xbin_mmap(fname, dtype, maxn=-1):
|
|
"""
|
|
Code from
|
|
https://github.com/harsha-simhadri/big-ann-benchmarks/blob/main/benchmark/dataset_io.py#L94 # noqa: E501
|
|
mmap the competition file format for a given type of items
|
|
"""
|
|
n, d = map(int, np.fromfile(fname, dtype="uint32", count=2))
|
|
assert os.stat(fname).st_size == 8 + n * d * np.dtype(dtype).itemsize
|
|
if maxn > 0:
|
|
n = min(n, maxn)
|
|
return np.memmap(fname, dtype=dtype, mode="r", offset=8, shape=(n, d))
|
|
|
|
|
|
def main(args: argparse.Namespace):
|
|
ssnpp_data = xbin_mmap(fname=args.filepath, dtype="uint8")
|
|
num_batches = ssnpp_data.shape[0] // args.data_batch
|
|
assert (
|
|
ssnpp_data.shape[0] % args.data_batch == 0
|
|
), "num of embeddings per file should divide total num of embeddings"
|
|
for i in range(num_batches):
|
|
xb_batch = ssnpp_data[
|
|
i * args.data_batch : (i + 1) * args.data_batch, :
|
|
]
|
|
filename = args.output_dir + f"/ssnpp_{(i):010}.npy"
|
|
np.save(filename, xb_batch)
|
|
print(f"File {filename} is saved!")
|
|
|
|
|
|
if __name__ == "__main__":
|
|
parser = argparse.ArgumentParser()
|
|
parser.add_argument(
|
|
"--data_batch",
|
|
dest="data_batch",
|
|
type=int,
|
|
default=50000000,
|
|
help="Number of embeddings per file, should be a divisor of 1B",
|
|
)
|
|
parser.add_argument(
|
|
"--filepath",
|
|
dest="filepath",
|
|
type=str,
|
|
default=(
|
|
"/datasets01/big-ann-challenge-data/FB_ssnpp/"
|
|
"FB_ssnpp_database.u8bin"
|
|
),
|
|
help="path of 1B ssnpp database vectors' original file",
|
|
)
|
|
parser.add_argument(
|
|
"--filepath",
|
|
dest="output_dir",
|
|
type=str,
|
|
default="/checkpoint/marialomeli/ssnpp_data",
|
|
help="path to put sharded files",
|
|
)
|
|
|
|
args = parser.parse_args()
|
|
main(args)
|