458 Commits
Author SHA1 Message Date
davidmezzetti 3073589ad6 Add deterministic setting to database functions closes #1048. Add ability to index database expressions #1049 2026-02-22 16:38:54 -05:00
davidmezzetti a53b431492 Add support for custom SQL functions with DuckDB, closes #1047 2026-02-21 06:43:56 -05:00
Jaepil Jeong ba0d1eaead Add BB25 normalization for sparse encoders (#1046)
* Add log-odds conjunction fusion for BB25 hybrid search

BB25 normalization outputs calibrated probabilities, but the existing
hybrid fusion uses convex combination which discards the Bayesian
probability semantics. This causes BB25 to regress on 4/5 BEIR datasets.

Add log-odds conjunction fusion (from "From Bayesian Inference to Neural
Computation") that correctly combines probability signals in logit space
with per-query dynamic calibration for dense cosine scores.

- scoring/normalize.py: Extract Bayesian method check into isbayes()
- scoring/base.py: Add default isbayes() returning False
- scoring/tfidf.py: Add isbayes() delegating to normalizer
- search/base.py: Add logodds(), convex(), rrf() fusion methods;
  dispatch based on isbayes()

BEIR nDCG@10 results (BB25+LogOdds vs Default):
  arguana +2.23, fiqa +2.03, scidocs +0.62, scifact +1.33, nfcorpus -1.96

* Extract Hybrid class for score fusion strategies

Move logodds, convex, and rrf fusion methods from Search into
a dedicated Hybrid class, following the same pattern as Normalize.

* Fix coding convention issues in Hybrid class for CI

- Fix black formatting: remove unnecessary parentheses, remove spaces around **
- Fix pylint too-many-branches: extract calibrate() method from logodds()
- Fix pylint unused-variable: rename score to _ in rrf()

* Add BB25 normalization for sparse encoders and fix IVFSparse topn bug

- Support `normalize: bb25` config for sparse encoder scoring, enabling
  Bayesian sigmoid calibration as an alternative to default linear
  normalization. Reuses existing Normalize.bayes() infrastructure.

- Fix dimension check in IVFSparse.topn(): use scores.shape[1] (number
  of data items) instead of scores.shape[0] (number of queries) for the
  argpartition kth bound check. The previous code caused ValueError when
  the number of centroids was less than nprobe.

* Add tests for BB25 sparse normalization and IVFSparse topn fix
2026-02-20 14:40:18 -05:00
davidmezzetti 89299926bd Add additional methods 2026-02-18 15:44:59 -05:00
davidmezzetti 00acd243b0 Code standardization and added tests #1041 2026-02-18 14:18:11 -05:00
davidmezzetti 371d5d8a19 Add test for bb25 2026-02-16 19:01:44 -05:00
Sigrid Jin (ง'̀-'́)ง oO 90e66075f6 Add Bayesian normalization "bb25" method for keyword scoring (#1037)
* Add Bayesian normalization for keyword scoring

* Add BB25 alias docs and references

* Align BB25 normalization with cognica candidate-set behavior
2026-02-16 15:53:26 -05:00
davidmezzetti 38170a4ab1 Support ngram tokenization, closes #1017 2026-02-16 09:41:41 -05:00
Rin 5a66065b34 Escape single quotes in dynamic column names to prevent SQL injection (#1026) 2026-02-15 10:50:45 -05:00
davidmezzetti 90bee00f1d Add ability to store agent memory by session, closes #1025 2026-02-11 13:18:18 -05:00
davidmezzetti f013ca7d1d Workaround macOS test failure 2026-02-10 13:12:37 -05:00
davidmezzetti bfd1f13f2b Workaround macOS test failure 2026-02-10 12:20:32 -05:00
davidmezzetti 4758907f3b Add input merging modes for language modeling training, closes #1021. Closes #1020 2026-02-10 09:42:23 -05:00
davidmezzetti 3a46f51f25 Add support for agents.md and skill.md. Closes #1014. Closes #1015 2026-01-30 15:35:19 -05:00
davidmezzetti 7672f42743 httpx import at module level causes ModuleNotFoundError for Embeddings-only usage, closes #1012 2026-01-23 11:39:11 -05:00
davidmezzetti c48ecb3062 Add OpenCode LLM, closes #1010 2026-01-19 21:14:37 -05:00
davidmezzetti 7132f63c46 Add additional tokenization methods for keyword search, closes #1006 2026-01-11 09:04:36 -05:00
davidmezzetti 2a63caf83e Reformat code 2026-01-05 12:05:33 -05:00
davidmezzetti 86e768bccb Load instruction prompts for Sentence Transformers models, closes #1004 2026-01-05 12:03:48 -05:00
davidmezzetti 861545a1ed Update test 2025-12-11 21:30:52 -05:00
davidmezzetti 75c4f54f63 Add new LLM default parameters, closes #998. Closes #999. 2025-12-11 20:47:45 -05:00
davidmezzetti d75be702dd Add support for wildcards in scoring term search, closes #990 2025-11-29 06:52:41 -05:00
davidmezzetti 99e3457b1f Fix compatibility with latest version of Faiss, closes #986 2025-11-18 13:28:46 -05:00
davidmezzetti 594145840f Support stripthink in streaming LLM responses, closes #981 2025-11-10 16:32:37 -05:00
davidmezzetti 8794d714cc Add tuples option to Segmentation pipeline, closes #979 2025-11-07 14:26:03 -05:00
davidmezzetti cf45a400ce Add GGML ANN, closes #975 2025-10-30 17:53:44 -04:00
davidmezzetti e662a631e2 Skip quantization tests for macOS 2025-10-17 08:11:05 -04:00
davidmezzetti e6e1d57ac4 Add quantization support to Torch ANN, closes #971 2025-10-17 07:24:05 -04:00
davidmezzetti 8dcab1addc Add support for Safetensors storage to NumPy/Torch ANN backend, closes #969 2025-10-13 15:23:03 -04:00
davidmezzetti 9a1a1e400c Update LLM stripthink parameter to support gpt-oss patterns, closes #966 2025-09-17 16:16:34 -04:00
davidmezzetti a543b028ef Syntax update 2025-09-17 07:59:20 -04:00
davidmezzetti a5306ae5d7 Code cleanup 2025-09-11 11:48:59 -04:00
davidmezzetti 303a257615 Additional checks for tar file extraction, closes #965 2025-09-11 11:41:09 -04:00
davidmezzetti 1f5a352261 Add exponential http backoff to work around constant HF Hub 429 errors 2025-08-28 09:38:59 -04:00
davidmezzetti b8980bc5be Add Reranker pipeline, closes #960 2025-08-27 15:00:12 -04:00
davidmezzetti 0e54ad3c3f Add ability to merge late interaction vectors into fixed dimensional MUVERA vectors, closes #952 2025-08-27 14:58:05 -04:00
davidmezzetti 2866dd962e Update similarity pipeline to support late interaction models, closes #953 2025-08-18 14:15:28 -04:00
davidmezzetti ad55344f26 Change default behavior for pickle serialization to disabled, closes #949 2025-08-13 12:20:41 -04:00
davidmezzetti 2956bd8789 Modify LiteLLM path detection logic, closes #947 2025-08-06 09:17:29 -04:00
davidmezzetti b53c966fa7 Change sparse vector normalization logic, closes #946 2025-08-05 18:26:49 -04:00
davidmezzetti f04a43578b IVFSparse improvements, closes #943 2025-07-19 08:16:15 -04:00
davidmezzetti 6cf12b97a0 Limit pgsparse input vectors to 1000 non-zero elements, closes #939 2025-07-16 16:30:58 -04:00
davidmezzetti dc57d6bf0d Add test for embeddings shortcuts 2025-07-14 08:31:45 -04:00
davidmezzetti e765be3f6c Refactor of ann and vectors packages for sparse vectors. Closes #930. Closes #934. Closes #935. Closes #936. Closes #937. 2025-07-14 07:48:23 -04:00
davidmezzetti af264bd0bb Update test 2025-07-08 22:13:45 -04:00
davidmezzetti db60bd76e6 Add IVFFlat indexing for sparse vectors, closes #929 2025-07-08 21:44:35 -04:00
davidmezzetti 2549c6598d Update test 2025-07-04 09:33:03 -04:00
davidmezzetti 6ca8a61856 Update test to workaround HF Hub HTTP 429 issues, closes #926 2025-07-04 09:27:20 -04:00
davidmezzetti d9c50c84b7 Add sparse vector scoring, closes #924. Update default Embeddings parameters, closes #925 2025-07-04 09:26:36 -04:00
davidmezzetti 484b366608 Handle GrandCypher API change, closes #909 2025-05-26 20:15:48 -04:00