HashPrune reservoir and sketch kernel (#5708)

Summary:
Pull Request resolved: https://github.com/facebookresearch/faiss/pull/5708

We add the HashPrune reservoir and its sketch kernel, the first building block of a PiPNN graph index (Rubel et al., KDD 2026, arXiv:2602.21247).

**Motivation**
PiPNN builds a navigable graph without beam searches during construction: it partitions the data into overlapping leaves, picks candidate edges inside each leaf, and keeps a bounded set of candidates per point with HashPrune. This diff adds only that per-point candidate set and the hash it buckets by. Its result must not depend on the order of offers, so that the build in a later diff can process leaves in parallel and stay deterministic. The rest of the stack adds, in order, the numeric kernels, the partition into leaves, the graph build, `IndexPiPNN` with search, and serialization with `index_factory` support.

**Implementation**
- `distance_key`: an order-preserving 16-bit key built from the scalar bf16 encoding. NaN maps to +inf and -0 to +0 on the bit pattern, which also holds under `-ffast-math`; the sign-magnitude to unsigned mapping makes signed inner-product distances compare correctly.
- `ReservoirRef::offer`: the HashPrune insert over 8-byte slots kept sorted by hash. Bucket replacement, eviction and the farthest-slot scan all use the strict `(key, id)` order, so the final reservoir is a function of the set of offers, whatever their order. A full reservoir rejects an offer that is not below its cached maximum before the binary search, which is expected to be the common case at steady state.
- `Hyperplanes` and `Hyperplanes::sketch`: Gaussian hyperplanes stored transposed and zero-padded, and a fixed-order `FAISS_NOINLINE` sketch kernel, so sketches computed on the fly are bitwise identical within one build. `residual_hash` derives the bucket bits.
- `FAISS_NOINLINE` in `impl/platform_macros.h`.
- No caller yet. The graph build uses this code in a later diff of the stack.

Reviewed By: mnorris11

Differential Revision: D123079760

fbshipit-source-id: f4278cf1d960b1756cdd54b6fbb14dec6eca3b3c
This commit is contained in:
Alibek Zhakubayev
2026-10-06 14:45:14 -07:00
committed by meta-codesync[bot]
parent 83ae8b0908
commit 7a057952fd
6 changed files with 554 additions and 0 deletions
+2
View File
@@ -166,6 +166,7 @@ set(FAISS_SRC
impl/hnsw/MinimaxHeap.cpp
impl/result_handler/ResultHandler.cpp
impl/NSG.cpp
impl/pipnn/HashPrune.cpp
impl/PolysemousTraining.cpp
impl/ProductQuantizer.cpp
impl/pq_code_distance/pq_code_distance-generic.cpp
@@ -313,6 +314,7 @@ set(FAISS_HEADERS
impl/maybe_owned_vector.h
impl/NNDescent.h
impl/NSG.h
impl/pipnn/HashPrune.h
impl/Panorama.h
impl/PanoramaStats.h
impl/PdxLayout.h
+159
View File
@@ -0,0 +1,159 @@
/*
* Copyright (c) Meta Platforms, Inc. and affiliates.
*
* This source code is licensed under the MIT license found in the
* LICENSE file in the root directory of this source tree.
*/
#include <faiss/impl/pipnn/HashPrune.h>
#include <cstring>
#include <faiss/impl/FaissAssert.h>
#include <faiss/impl/platform_macros.h>
#include <faiss/utils/bf16.h>
#include <faiss/utils/random.h>
namespace faiss {
namespace pipnn {
uint16_t distance_key(float v) {
// Normalized on the bit pattern, so -ffast-math cannot remove the checks.
uint32_t u;
std::memcpy(&u, &v, sizeof(u));
if ((u & 0x7fffffffu) == 0) {
u = 0; // -0 -> +0
} else if ((u & 0x7f800000u) == 0x7f800000u && (u & 0x007fffffu) != 0) {
u = 0x7f800000u; // NaN -> +inf
}
std::memcpy(&v, &u, sizeof(v));
// Scalar `encode_bf16` only: the SIMD path rounds differently, so keys
// would depend on the CPU.
const uint16_t b = encode_bf16(v);
return (b & 0x8000) ? static_cast<uint16_t>(~b)
: static_cast<uint16_t>(b | 0x8000);
}
bool ReservoirRef::offer(int32_t id, uint16_t hash, uint16_t key) {
const int n = *count;
// A full reservoir rejects anything not below its maximum (rules 1 and 3).
if (n == capacity && *farthest != kNoFarthest) {
const Slot& f = slots[*farthest];
if (!slot_less(key, id, f.key, f.id)) {
return false;
}
}
int lo = 0;
int hi = n;
while (lo < hi) {
const int mid = (lo + hi) / 2;
if (slots[mid].hash < hash) {
lo = mid + 1;
} else {
hi = mid;
}
}
// Rule 1: bucket hit. The cached farthest stays valid only when a
// non-farthest slot is lowered.
if (lo < n && slots[lo].hash == hash) {
Slot& s = slots[lo];
if (!slot_less(key, id, s.key, s.id)) {
return false;
}
s.id = id;
s.key = key;
if (*farthest == lo) {
*farthest = kNoFarthest;
}
return true;
}
// Rule 2: miss with room.
if (n < capacity) {
std::memmove(
slots + lo + 1,
slots + lo,
sizeof(Slot) * static_cast<size_t>(n - lo));
slots[lo] = Slot{id, hash, key};
*count = static_cast<uint8_t>(n + 1);
*farthest = kNoFarthest;
return true;
}
// Rule 3: miss, full. Compare against the slot_less-maximum.
int z = *farthest;
if (z == kNoFarthest) {
z = 0;
for (int i = 1; i < n; i++) {
if (slot_less(
slots[z].key, slots[z].id, slots[i].key, slots[i].id)) {
z = i;
}
}
*farthest = static_cast<uint8_t>(z);
}
const Slot& zs = slots[z];
if (!slot_less(key, id, zs.key, zs.id)) {
return false;
}
if (z < lo) {
std::memmove(
slots + z,
slots + z + 1,
sizeof(Slot) * static_cast<size_t>(lo - 1 - z));
slots[lo - 1] = Slot{id, hash, key};
} else {
std::memmove(
slots + lo + 1,
slots + lo,
sizeof(Slot) * static_cast<size_t>(z - lo));
slots[lo] = Slot{id, hash, key};
}
*farthest = kNoFarthest;
return true;
}
Hyperplanes::Hyperplanes(int d, int m, int64_t seed) : d(d), m(m) {
FAISS_THROW_IF_NOT_FMT(d >= 1, "Hyperplanes: d must be >= 1, got %d", d);
FAISS_THROW_IF_NOT_FMT(
m >= 1 && m <= 16, "Hyperplanes: m must be in [1, 16], got %d", m);
std::vector<float> h(static_cast<size_t>(m) * d);
float_randn(h.data(), h.size(), seed);
ht.assign(static_cast<size_t>(d) * 16, 0.0f);
for (int i = 0; i < m; i++) {
for (int j = 0; j < d; j++) {
ht[static_cast<size_t>(j) * 16 + i] =
h[static_cast<size_t>(i) * d + j];
}
}
}
FAISS_NOINLINE void Hyperplanes::sketch(const float* x, float* out16) const {
// Local accumulators, j ascending: the result depends only on the values
// of x, never on its buffer, alignment or caller.
float acc[16] = {};
const float* h = ht.data();
for (int j = 0; j < d; j++) {
const float xj = x[j];
const float* hj = h + static_cast<size_t>(j) * 16;
for (int i = 0; i < 16; i++) {
acc[i] += xj * hj[i];
}
}
std::memcpy(out16, acc, sizeof(acc));
}
uint16_t residual_hash(const float* sketch_p, const float* sketch_c, int m) {
uint16_t h = 0;
for (int i = 0; i < m; i++) {
if (sketch_c[i] - sketch_p[i] >= 0) {
h = static_cast<uint16_t>(h | (1u << i));
}
}
return h;
}
} // namespace pipnn
} // namespace faiss
+69
View File
@@ -0,0 +1,69 @@
/*
* Copyright (c) Meta Platforms, Inc. and affiliates.
*
* This source code is licensed under the MIT license found in the
* LICENSE file in the root directory of this source tree.
*/
#pragma once
#include <cstdint>
#include <vector>
namespace faiss {
namespace pipnn {
/// One reservoir slot: candidate id, residual-LSH bucket, order-preserving key.
struct Slot {
int32_t id;
uint16_t hash;
uint16_t key;
};
static_assert(sizeof(Slot) == 8, "PiPNN reservoir slots must be 8 bytes");
/// "No cached farthest slot"; never a slot index since capacity <= 255.
constexpr uint8_t kNoFarthest = 0xFF;
/// Monotone 16-bit key of a distance, which may be negative, from its scalar
/// bf16 encoding. NaN maps to +inf and -0 to +0.
uint16_t distance_key(float v);
/// Strict (key, id) order. Deciding key ties by id makes the reservoir
/// independent of the offer order.
inline bool slot_less(uint16_t ka, int32_t ia, uint16_t kb, int32_t ib) {
return ka < kb || (ka == kb && ia < ib);
}
/// One point's reservoir in caller-owned storage; not thread-safe. Slots are
/// sorted by distinct hash, and *farthest caches the slot_less-maximum.
struct ReservoirRef {
Slot* slots;
uint8_t* count; // 0 when fresh
uint8_t* farthest; // kNoFarthest when fresh or unknown
int capacity; // 1..255
/// Keeps the smaller offer on a bucket hit, else inserts or evicts the
/// maximum. An id must always come with the same hash. True iff changed.
bool offer(int32_t id, uint16_t hash, uint16_t key);
};
/// m <= 16 Gaussian hyperplanes from float_randn(seed), stored transposed
/// and zero-padded: ht[j * 16 + i].
struct Hyperplanes {
int d = 0;
int m = 0;
std::vector<float> ht; // d * 16
Hyperplanes() = default;
Hyperplanes(int d, int m, int64_t seed);
/// out16[i] = sum over ascending j of x[j] * ht[j * 16 + i]. Not inlined,
/// so all call sites produce bitwise-identical sketches.
void sketch(const float* x, float* out16) const;
};
/// Bit i (i < m <= 16) is set iff sketch_c[i] - sketch_p[i] >= 0.
uint16_t residual_hash(const float* sketch_p, const float* sketch_c, int m);
} // namespace pipnn
} // namespace faiss
+12
View File
@@ -147,6 +147,18 @@ inline int __builtin_clzll(uint64_t x) {
#define FAISS_DEPRECATED(msg)
#endif // GCC or Clang
// Never inlined, so floating-point results do not depend on the compile
// flags of the call site (e.g. under LTO).
#if defined(_MSC_VER)
#define FAISS_NOINLINE __declspec(noinline)
#elif defined(__clang__)
#define FAISS_NOINLINE __attribute__((noinline))
#elif defined(__GNUC__)
#define FAISS_NOINLINE __attribute__((noinline, noclone))
#else
#define FAISS_NOINLINE
#endif
namespace faiss {
namespace detail {
+1
View File
@@ -47,6 +47,7 @@ set(FAISS_TEST_SRC
test_rabitq_fastscan.cpp
test_rabitq_batch_estimates.cpp
test_super_kmeans_foundations.cpp
test_pipnn.cpp
)
# Tests that use POSIX APIs or have platform-specific issues on Windows
+311
View File
@@ -0,0 +1,311 @@
/*
* Copyright (c) Meta Platforms, Inc. and affiliates.
*
* This source code is licensed under the MIT license found in the
* LICENSE file in the root directory of this source tree.
*/
#include <gtest/gtest.h>
#include <algorithm>
#include <cmath>
#include <cstdint>
#include <cstring>
#include <limits>
#include <map>
#include <set>
#include <tuple>
#include <utility>
#include <vector>
#include <faiss/impl/FaissException.h>
#include <faiss/impl/pipnn/HashPrune.h>
#include <faiss/utils/random.h>
using faiss::pipnn::distance_key;
using faiss::pipnn::Hyperplanes;
using faiss::pipnn::kNoFarthest;
using faiss::pipnn::ReservoirRef;
using faiss::pipnn::residual_hash;
using faiss::pipnn::Slot;
using faiss::pipnn::slot_less;
TEST(PiPNNDistanceKey, SpecialValues) {
const float inf = std::numeric_limits<float>::infinity();
EXPECT_EQ(distance_key(0.0f), uint16_t(0x8000));
EXPECT_EQ(distance_key(-0.0f), distance_key(0.0f));
EXPECT_EQ(distance_key(1.0f), uint16_t(0xBF80));
EXPECT_EQ(distance_key(-1.0f), uint16_t(0x407F));
EXPECT_EQ(distance_key(inf), uint16_t(0xFF80));
EXPECT_EQ(
distance_key(std::numeric_limits<float>::quiet_NaN()),
distance_key(inf));
}
TEST(PiPNNDistanceKey, Monotone) {
// Strictly increasing over all non-NaN bf16 values.
std::vector<float> grid;
for (uint32_t b = 0; b < 65536; b++) {
const uint32_t bits = b << 16;
float f;
std::memcpy(&f, &bits, sizeof(f));
if (!std::isnan(f)) {
grid.push_back(f);
}
}
std::sort(grid.begin(), grid.end());
for (size_t i = 1; i < grid.size(); i++) {
if (grid[i - 1] != grid[i]) { // skips the -0, +0 pair
ASSERT_LT(distance_key(grid[i - 1]), distance_key(grid[i]))
<< grid[i - 1] << " vs " << grid[i];
}
}
// Non-decreasing over arbitrary floats, which bf16 rounding maps between
// grid points.
std::vector<float> v;
faiss::SplitMix64RandomGenerator rng(42);
for (int i = 0; i < 100000; i++) {
const uint32_t bits = static_cast<uint32_t>(rng.next());
float f;
std::memcpy(&f, &bits, sizeof(f));
if (!std::isnan(f)) {
v.push_back(f);
}
}
std::sort(v.begin(), v.end());
for (size_t i = 1; i < v.size(); i++) {
ASSERT_LE(distance_key(v[i - 1]), distance_key(v[i]))
<< v[i - 1] << " vs " << v[i];
}
}
namespace {
struct TestReservoir {
std::vector<Slot> slots;
uint8_t count = 0;
uint8_t farthest = kNoFarthest;
ReservoirRef ref;
explicit TestReservoir(int capacity)
: slots(capacity), ref{slots.data(), &count, &farthest, capacity} {}
TestReservoir(const TestReservoir&) = delete;
TestReservoir& operator=(const TestReservoir&) = delete;
bool offer(int32_t id, uint16_t hash, uint16_t key) {
return ref.offer(id, hash, key);
}
};
using SlotTuple = std::tuple<int32_t, uint16_t, uint16_t>; // (id, hash, key)
std::vector<SlotTuple> contents(const TestReservoir& r) {
std::vector<SlotTuple> out;
out.reserve(r.count);
for (int i = 0; i < r.count; i++) {
out.emplace_back(r.slots[i].id, r.slots[i].hash, r.slots[i].key);
}
return out;
}
/// Checks the ReservoirRef invariant documented in HashPrune.h.
void check_invariant(const TestReservoir& r) {
ASSERT_LE(int(r.count), r.ref.capacity);
std::set<int32_t> ids;
for (int i = 0; i < r.count; i++) {
if (i > 0) {
ASSERT_LT(r.slots[i - 1].hash, r.slots[i].hash);
}
ASSERT_TRUE(ids.insert(r.slots[i].id).second);
}
if (r.farthest != kNoFarthest) {
ASSERT_LT(int(r.farthest), int(r.count));
const Slot& f = r.slots[r.farthest];
for (int i = 0; i < r.count; i++) {
if (i != r.farthest) {
ASSERT_TRUE(
slot_less(r.slots[i].key, r.slots[i].id, f.key, f.id));
}
}
}
}
} // namespace
TEST(PiPNNReservoir, Rules) {
TestReservoir r(3);
EXPECT_TRUE(r.offer(1, 30, 0x9000));
EXPECT_TRUE(r.offer(2, 10, 0x9300));
EXPECT_FALSE(r.offer(3, 10, 0x9400)); // bucket hit, larger key
EXPECT_FALSE(r.offer(4, 10, 0x9300)); // bucket hit, equal key, larger id
EXPECT_TRUE(r.offer(0, 10, 0x9300)); // bucket hit, equal key, smaller id
EXPECT_TRUE(r.offer(5, 20, 0x9100));
std::vector<SlotTuple> expected = {
{0, 10, 0x9300}, {5, 20, 0x9100}, {1, 30, 0x9000}};
EXPECT_EQ(contents(r), expected);
EXPECT_FALSE(r.offer(6, 40, 0x9300)); // full: ties the max, larger id
EXPECT_TRUE(r.offer(7, 40, 0x9200)); // full: evicts the max, id 0
expected = {{5, 20, 0x9100}, {1, 30, 0x9000}, {7, 40, 0x9200}};
EXPECT_EQ(contents(r), expected);
}
TEST(PiPNNReservoir, FarthestCache) {
TestReservoir r(3);
ASSERT_TRUE(r.offer(1, 10, 0x9000));
ASSERT_TRUE(r.offer(2, 20, 0x9300));
ASSERT_TRUE(r.offer(3, 30, 0x9100));
EXPECT_FALSE(r.offer(4, 40, 0x9400)); // the scan caches the max, id 2
EXPECT_EQ(int(r.farthest), 1);
EXPECT_TRUE(r.offer(3, 30, 0x8F00)); // lowering another slot keeps it
EXPECT_EQ(int(r.farthest), 1);
EXPECT_TRUE(r.offer(2, 20, 0x8E00)); // lowering the max clears it
EXPECT_EQ(int(r.farthest), int(kNoFarthest));
}
namespace {
struct Offer {
int32_t id;
uint16_t hash;
uint16_t key;
};
/// Reference model of the reservoir (Lemma A.2 of the PiPNN paper, with key
/// ties decided by id): the min(capacity, B) slot_less-smallest bucket
/// minima, listed in hash order.
std::vector<SlotTuple> model_reservoir(
const std::vector<Offer>& offers,
int capacity) {
std::map<uint16_t, std::pair<uint16_t, int32_t>> minima; // hash->(key,id)
for (const Offer& o : offers) {
auto it = minima.find(o.hash);
if (it == minima.end()) {
minima.emplace(o.hash, std::make_pair(o.key, o.id));
} else if (slot_less(
o.key, o.id, it->second.first, it->second.second)) {
it->second = std::make_pair(o.key, o.id);
}
}
std::vector<std::tuple<uint16_t, int32_t, uint16_t>> ranked; // key,id,hash
ranked.reserve(minima.size());
for (const auto& [hash, key_id] : minima) {
ranked.emplace_back(key_id.first, key_id.second, hash);
}
std::sort(ranked.begin(), ranked.end());
if (ranked.size() > static_cast<size_t>(capacity)) {
ranked.resize(capacity);
}
std::vector<SlotTuple> out;
out.reserve(ranked.size());
for (const auto& [key, id, hash] : ranked) {
out.emplace_back(id, hash, key);
}
std::sort(
out.begin(), out.end(), [](const SlotTuple& a, const SlotTuple& b) {
return std::get<1>(a) < std::get<1>(b);
});
return out;
}
/// Ids are bucket * 4 + r (r < 4), so an id always lands in the same bucket,
/// as the residual hash guarantees in the build. Ids and keys repeat, so the
/// offers include re-offers and key ties.
std::vector<Offer> random_offers(
size_t n,
int n_buckets,
int n_keys,
faiss::SplitMix64RandomGenerator& rng) {
std::vector<Offer> offers(n);
for (Offer& o : offers) {
const int bucket = static_cast<int>(rng.next() % n_buckets);
o.hash = static_cast<uint16_t>(bucket * 37);
o.id = static_cast<int32_t>(bucket * 4 + rng.next() % 4);
o.key = static_cast<uint16_t>(0x8000 + rng.next() % n_keys);
}
return offers;
}
} // namespace
TEST(PiPNNReservoir, MatchesModelInAnyOrder) {
faiss::SplitMix64RandomGenerator rng(77);
for (int capacity : {1, 3, 8, 255}) {
const int n_buckets = 2 * capacity + 8;
for (int n_keys : {3, 4096}) {
for (int trial = 0; trial < 3; trial++) {
std::vector<Offer> offers =
random_offers(3 * n_buckets, n_buckets, n_keys, rng);
const std::vector<SlotTuple> expected =
model_reservoir(offers, capacity);
for (int shuffle = 0; shuffle < 10; shuffle++) {
SCOPED_TRACE(
testing::Message()
<< "capacity " << capacity << " keys " << n_keys
<< " trial " << trial << " shuffle " << shuffle);
for (size_t i = offers.size(); i > 1; i--) {
std::swap(offers[i - 1], offers[rng.next() % i]);
}
TestReservoir r(capacity);
for (const Offer& o : offers) {
const std::vector<SlotTuple> before = contents(r);
const bool changed = r.offer(o.id, o.hash, o.key);
ASSERT_EQ(changed, contents(r) != before);
ASSERT_NO_FATAL_FAILURE(check_invariant(r));
}
ASSERT_EQ(contents(r), expected);
}
}
}
}
}
TEST(PiPNNSketch, InvalidArgs) {
EXPECT_THROW(Hyperplanes(0, 4, 1), faiss::FaissException);
EXPECT_THROW(Hyperplanes(8, 0, 1), faiss::FaissException);
EXPECT_THROW(Hyperplanes(8, 17, 1), faiss::FaissException);
}
TEST(PiPNNSketch, ProjectionIndependentOfBuffer) {
const int d = 37;
const int m = 12;
Hyperplanes hp(d, m, 7);
std::vector<float> h(size_t(m) * d);
faiss::float_randn(h.data(), h.size(), 7);
std::vector<float> x(d);
faiss::float_randn(x.data(), d, 99);
// The same vector at an odd offset of another buffer, with a misaligned
// output, must give a bitwise-identical sketch.
std::vector<float> shifted(3 + d);
std::copy(x.begin(), x.end(), shifted.begin() + 3);
float a[16];
std::vector<float> b(17);
hp.sketch(x.data(), a);
hp.sketch(shifted.data() + 3, b.data() + 1);
EXPECT_EQ(0, std::memcmp(a, b.data() + 1, sizeof(a)));
for (int i = 0; i < m; i++) {
double s = 0;
for (int j = 0; j < d; j++) {
s += double(x[j]) * double(h[size_t(i) * d + j]);
}
EXPECT_NEAR(a[i], s, 1e-4 * (1 + std::abs(s))) << i;
}
for (int i = m; i < 16; i++) {
EXPECT_EQ(a[i], 0.0f) << i; // zero-padded hyperplanes
}
}
TEST(PiPNNSketch, ResidualHash) {
float p[16];
float c[16];
for (int i = 0; i < 16; i++) {
p[i] = float(i);
c[i] = i % 3 == 0 ? float(i) - 1 : float(i) + 1;
}
// Bit i is set iff i % 3 != 0.
EXPECT_EQ(residual_hash(p, c, 1), uint16_t(0));
EXPECT_EQ(residual_hash(p, c, 6), uint16_t(0x36));
EXPECT_EQ(residual_hash(p, c, 16), uint16_t(0x6DB6));
// A zero residual sets all m bits and none above.
EXPECT_EQ(residual_hash(p, p, 12), uint16_t(0x0FFF));
}