* Fix translation chunk concatenation when showmodels is enabled
When translating long text that gets split into multiple chunks by the
tokenizer, the translate() method combines results from the same source
text segment. With showmodels=True, each result is a tuple (path, text).
Using += to combine tuples produces (path, text1, path, text2) instead
of the expected (path, text1+text2). This causes a ValueError when the
caller unpacks with "model, op = outputs[y]" since there are too many
values.
Fix by explicitly concatenating only the text portion of the tuple when
showmodels is enabled.
Signed-off-by: JiangNan <[email protected]>
* Add unit test for showmodels with chunked translation
Adds testLongTranslationWithShowmodels to verify that when text is long
enough to be chunked by the tokenizer, the showmodels result is still a
proper 3-tuple (translation, language, model) and not a malformed tuple
from incorrect concatenation.
Signed-off-by: JiangNan <[email protected]>
* Shorten test method name to fix build
Rename testLongTranslationWithShowmodels -> testShowmodelsChunked
as the previous name was too long for the test runner.
Signed-off-by: JiangNan <[email protected]>
---------
Signed-off-by: JiangNan <[email protected]>
* Add log-odds conjunction fusion for BB25 hybrid search
BB25 normalization outputs calibrated probabilities, but the existing
hybrid fusion uses convex combination which discards the Bayesian
probability semantics. This causes BB25 to regress on 4/5 BEIR datasets.
Add log-odds conjunction fusion (from "From Bayesian Inference to Neural
Computation") that correctly combines probability signals in logit space
with per-query dynamic calibration for dense cosine scores.
- scoring/normalize.py: Extract Bayesian method check into isbayes()
- scoring/base.py: Add default isbayes() returning False
- scoring/tfidf.py: Add isbayes() delegating to normalizer
- search/base.py: Add logodds(), convex(), rrf() fusion methods;
dispatch based on isbayes()
BEIR nDCG@10 results (BB25+LogOdds vs Default):
arguana +2.23, fiqa +2.03, scidocs +0.62, scifact +1.33, nfcorpus -1.96
* Extract Hybrid class for score fusion strategies
Move logodds, convex, and rrf fusion methods from Search into
a dedicated Hybrid class, following the same pattern as Normalize.
* Fix coding convention issues in Hybrid class for CI
- Fix black formatting: remove unnecessary parentheses, remove spaces around **
- Fix pylint too-many-branches: extract calibrate() method from logodds()
- Fix pylint unused-variable: rename score to _ in rrf()
* Add BB25 normalization for sparse encoders and fix IVFSparse topn bug
- Support `normalize: bb25` config for sparse encoder scoring, enabling
Bayesian sigmoid calibration as an alternative to default linear
normalization. Reuses existing Normalize.bayes() infrastructure.
- Fix dimension check in IVFSparse.topn(): use scores.shape[1] (number
of data items) instead of scores.shape[0] (number of queries) for the
argpartition kth bound check. The previous code caused ValueError when
the number of centroids was less than nprobe.
* Add tests for BB25 sparse normalization and IVFSparse topn fix
* Add log-odds conjunction fusion for BB25 hybrid search
BB25 normalization outputs calibrated probabilities, but the existing
hybrid fusion uses convex combination which discards the Bayesian
probability semantics. This causes BB25 to regress on 4/5 BEIR datasets.
Add log-odds conjunction fusion (from "From Bayesian Inference to Neural
Computation") that correctly combines probability signals in logit space
with per-query dynamic calibration for dense cosine scores.
- scoring/normalize.py: Extract Bayesian method check into isbayes()
- scoring/base.py: Add default isbayes() returning False
- scoring/tfidf.py: Add isbayes() delegating to normalizer
- search/base.py: Add logodds(), convex(), rrf() fusion methods;
dispatch based on isbayes()
BEIR nDCG@10 results (BB25+LogOdds vs Default):
arguana +2.23, fiqa +2.03, scidocs +0.62, scifact +1.33, nfcorpus -1.96
* Extract Hybrid class for score fusion strategies
Move logodds, convex, and rrf fusion methods from Search into
a dedicated Hybrid class, following the same pattern as Normalize.
* Fix coding convention issues in Hybrid class for CI
- Fix black formatting: remove unnecessary parentheses, remove spaces around **
- Fix pylint too-many-branches: extract calibrate() method from logodds()
- Fix pylint unused-variable: rename score to _ in rrf()
davidmezzetti