diff --git a/docs/embeddings/configuration/vectors.md b/docs/embeddings/configuration/vectors.md index dce8c4e6..d4d4f88a 100644 --- a/docs/embeddings/configuration/vectors.md +++ b/docs/embeddings/configuration/vectors.md @@ -155,6 +155,18 @@ muvera: Settings to control the size of MUVERA fixed dimensional outputs. Default is 20 * 2^5 * 16 = 10,240 dimensions. +### lemur +```yaml +lemur: + path: local artifact directory or Hugging Face Hub path +``` + +Loads a trained [LEMUR](../../../pipeline/train/lemur) fixed dimensional encoder. LEMUR artifacts are corpus-specific and must be +created with `LemurTrainer` before configuring an embeddings index. Query vectors are summed learned features and document vectors +are ordinary least squares weights over the artifact's stored token sample. Each artifact contains `config.json` and +`model.safetensors`; the Safetensors file contains inference state only. See the trainer page for score filtering and Faiss exact/IVF +search behavior. + ### trust_remote_code ```yaml trust_remote_code: boolean diff --git a/docs/pipeline/train/lemur.md b/docs/pipeline/train/lemur.md new file mode 100644 index 00000000..20b7de0a --- /dev/null +++ b/docs/pipeline/train/lemur.md @@ -0,0 +1,82 @@ +# LemurTrainer + +![pipeline](../../images/pipeline.png#only-light) +![pipeline](../../images/pipeline-dark.png#only-dark) + +Trains a LEMUR fixed dimensional encoder for a late interaction model and corpus. Training is a separate pipeline step; the saved +artifact is then loaded through the embeddings `vectors.lemur` configuration. + +## Example + +```python +from txtai.pipeline import LemurTrainer + +corpus = [ + "First document", + "Second document", + "Third document", +] + +trainer = LemurTrainer() +trainer( + "colbert-ir/colbertv2.0", + corpus, + "lemur-model", + epochs=100, + validation_split=0.1, +) +``` + +`epochs` is required so the training mode is explicit. Use `epochs=100` for the quality-oriented MLP setting. This can take hours on +a CPU. Use `epochs=0` to select deterministic random ELM features as a lower-cost fallback. + +The learn distribution defaults to `learn_category="query"`. Target documents are always encoded with the data encoder, so the +default encodes selected corpus texts once as data and once as queries. Set `learn_category="data"` to reuse the data encodings, or +pass a separate iterable of texts with `learn`. + +Set `corpus_subset_size` to a positive integer to sample that many raw corpus texts under `seed` before either encoding pass. The +default is `None`, which uses every corpus text. `train_subset_size` and `learn_subset_size` apply later, after token vectors have +already been created. `corpus_subset_size` applies to `data`; a separate `learn` iterable remains caller-sized. + +For trained MLP features, set `validation_split` to a fraction greater than zero and less than one to retain the epoch with the lowest +held-out loss. The default is `0.0`, which preserves training-loss selection. The selected one-based epoch, loss and metric are +available as `selected_epoch`, `selected_loss` and `selection_metric` on the fitted or reloaded encoder. + +The artifact contains `config.json` and `model.safetensors`. It stores the inference feature model, output-normalization statistics +and token sample needed to encode documents added after training. The training-only output readout is not saved. + +Load the artifact in an embeddings configuration. + +```yaml +embeddings: + path: colbert-ir/colbertv2.0 + vectors: + lemur: + path: lemur-model +``` + +## Search behavior + +LEMUR approximates standardized MaxSim targets, so useful ranking scores can be negative. txtai's dense vector path L2-normalizes the +fixed vectors and removes results with scores less than or equal to zero. This can change ordering and return fewer than the requested +number of candidates compared with raw maximum inner-product search. + +The Faiss backend uses exact `IDMap,Flat` search through 5,000 rows and switches to an IVF index above that threshold. In the measured +scifact run, default IVF reduced LEMUR NDCG@10 by 43% relative to exact search, compared with 25% for MUVERA. For LEMUR corpora above +5,000 rows, either pin `faiss.components` to an exact index or tune the IVF settings for the corpus. The exact configuration is: + +```yaml +embeddings: + path: colbert-ir/colbertv2.0 + vectors: + lemur: + path: lemur-model + faiss: + components: IDMap,Flat +``` + +## Methods + +Python documentation for the pipeline. + +### ::: txtai.pipeline.LemurTrainer.__call__ diff --git a/mkdocs.yml b/mkdocs.yml index 6af5c533..848322f6 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -109,6 +109,7 @@ nav: - Translation: pipeline/text/translation.md - Train: - HF ONNX: pipeline/train/hfonnx.md + - LEMUR Trainer: pipeline/train/lemur.md - ML ONNX: pipeline/train/mlonnx.md - Trainer: pipeline/train/trainer.md - Workflow: