25 Commits
Author SHA1 Message Date
Ni0ri 20c9b8e0be Add Zvec Sparse ANN (#1307) 2026-10-01 09:53:38 -04:00
Hua.T 8ef454d0bf Preserve keyword document ID types across save/load (#1274) (#1275) 2026-09-30 10:57:41 -04:00
davidmezzetti 5cba72c88a Update pylintrc file and remove warning overrides for new limits 2026-09-29 07:11:03 -04:00
David Mezzetti 4e2eb242b6 Support explicit LIKE escapes in wildcard term queries (#1259) 2026-09-24 16:34:16 -04:00
LimbC-C f47d389735 Preserve wildcard compatibility and support explicit LIKE escapes 2026-09-24 23:54:33 +08:00
LimbC-C e7894cdb00 Escape literal SQL wildcard characters in term expansion (#1258) 2026-09-23 13:06:27 +08:00
LimbC-C 47d5f56768 Keep deleted keyword results excluded after common-term scoring (#1247) 2026-09-19 09:46:09 +08:00
Morgan Carr 20e2dc41e1 Save empty keyword and content indexes without errors (#1224) 2026-09-04 08:21:11 -04:00
Morgan Carr 7e98746d78 Delete every index entry for an id in Terms (#1219) 2026-09-04 08:11:47 -04:00
Sainikhil Juluri e90c67ef06 fix(scoring): keep count() accurate when an id is deleted more than once (#1202) 2026-08-24 21:55:11 -04:00
Amir Fathi bd199158b6 Fix Scoring.delete() crashing on ids that were never indexed 2026-07-30 12:35:35 +00:00
davidmezzetti fcfd340ff1 Cleanup tests 2026-07-21 12:18:35 -04:00
Andrew Chen 3bb34eae2e Fix Terms.search AttributeError on an empty terms index (#1143) 2026-07-21 11:40:43 -04:00
davidmezzetti 638a2e12b1 Validate resolver target classes, closes #1153 2026-07-21 09:13:43 -04:00
Jaepil Jeong ba0d1eaead Add BB25 normalization for sparse encoders (#1046)
* Add log-odds conjunction fusion for BB25 hybrid search

BB25 normalization outputs calibrated probabilities, but the existing
hybrid fusion uses convex combination which discards the Bayesian
probability semantics. This causes BB25 to regress on 4/5 BEIR datasets.

Add log-odds conjunction fusion (from "From Bayesian Inference to Neural
Computation") that correctly combines probability signals in logit space
with per-query dynamic calibration for dense cosine scores.

- scoring/normalize.py: Extract Bayesian method check into isbayes()
- scoring/base.py: Add default isbayes() returning False
- scoring/tfidf.py: Add isbayes() delegating to normalizer
- search/base.py: Add logodds(), convex(), rrf() fusion methods;
  dispatch based on isbayes()

BEIR nDCG@10 results (BB25+LogOdds vs Default):
  arguana +2.23, fiqa +2.03, scidocs +0.62, scifact +1.33, nfcorpus -1.96

* Extract Hybrid class for score fusion strategies

Move logodds, convex, and rrf fusion methods from Search into
a dedicated Hybrid class, following the same pattern as Normalize.

* Fix coding convention issues in Hybrid class for CI

- Fix black formatting: remove unnecessary parentheses, remove spaces around **
- Fix pylint too-many-branches: extract calibrate() method from logodds()
- Fix pylint unused-variable: rename score to _ in rrf()

* Add BB25 normalization for sparse encoders and fix IVFSparse topn bug

- Support `normalize: bb25` config for sparse encoder scoring, enabling
  Bayesian sigmoid calibration as an alternative to default linear
  normalization. Reuses existing Normalize.bayes() infrastructure.

- Fix dimension check in IVFSparse.topn(): use scores.shape[1] (number
  of data items) instead of scores.shape[0] (number of queries) for the
  argpartition kth bound check. The previous code caused ValueError when
  the number of centroids was less than nprobe.

* Add tests for BB25 sparse normalization and IVFSparse topn fix
2026-02-20 14:40:18 -05:00
davidmezzetti 89299926bd Add additional methods 2026-02-18 15:44:59 -05:00
davidmezzetti 00acd243b0 Code standardization and added tests #1041 2026-02-18 14:18:11 -05:00
davidmezzetti 371d5d8a19 Add test for bb25 2026-02-16 19:01:44 -05:00
Sigrid Jin (ง'̀-'́)ง oO 90e66075f6 Add Bayesian normalization "bb25" method for keyword scoring (#1037)
* Add Bayesian normalization for keyword scoring

* Add BB25 alias docs and references

* Align BB25 normalization with cognica candidate-set behavior
2026-02-16 15:53:26 -05:00
davidmezzetti 38170a4ab1 Support ngram tokenization, closes #1017 2026-02-16 09:41:41 -05:00
davidmezzetti bfd1f13f2b Workaround macOS test failure 2026-02-10 12:20:32 -05:00
davidmezzetti 7132f63c46 Add additional tokenization methods for keyword search, closes #1006 2026-01-11 09:04:36 -05:00
davidmezzetti d75be702dd Add support for wildcards in scoring term search, closes #990 2025-11-29 06:52:41 -05:00
davidmezzetti b53c966fa7 Change sparse vector normalization logic, closes #946 2025-08-05 18:26:49 -04:00
davidmezzetti e765be3f6c Refactor of ann and vectors packages for sparse vectors. Closes #930. Closes #934. Closes #935. Closes #936. Closes #937. 2025-07-14 07:48:23 -04:00