mirror of
https://github.com/neuml/txtai.git
synced 2026-10-12 02:19:52 +00:00
JiangNan
79908eee5a
* Fix translation chunk concatenation when showmodels is enabled When translating long text that gets split into multiple chunks by the tokenizer, the translate() method combines results from the same source text segment. With showmodels=True, each result is a tuple (path, text). Using += to combine tuples produces (path, text1, path, text2) instead of the expected (path, text1+text2). This causes a ValueError when the caller unpacks with "model, op = outputs[y]" since there are too many values. Fix by explicitly concatenating only the text portion of the tuple when showmodels is enabled. Signed-off-by: JiangNan <[email protected]> * Add unit test for showmodels with chunked translation Adds testLongTranslationWithShowmodels to verify that when text is long enough to be chunked by the tokenizer, the showmodels result is still a proper 3-tuple (translation, language, model) and not a malformed tuple from incorrect concatenation. Signed-off-by: JiangNan <[email protected]> * Shorten test method name to fix build Rename testLongTranslationWithShowmodels -> testShowmodelsChunked as the previous name was too long for the test runner. Signed-off-by: JiangNan <[email protected]> --------- Signed-off-by: JiangNan <[email protected]>