From fa342a4f6a69fb1e2cea0e85e39915f055ee8503 Mon Sep 17 00:00:00 2001 From: davidmezzetti <561939+davidmezzetti@users.noreply.github.com> Date: Thu, 17 Mar 2022 10:05:03 -0400 Subject: [PATCH] Add clarification on building word vector backed models with scoring #247 --- docs/embeddings/index.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/docs/embeddings/index.md b/docs/embeddings/index.md index 02d712f4..da14fdf7 100644 --- a/docs/embeddings/index.md +++ b/docs/embeddings/index.md @@ -71,6 +71,16 @@ The input iterable can be a list and/or a generator. [Generators](https://wiki.p txtai buffers data to temporary storage along the way during indexing as embeddings vectors can be quite large (for example 768 dimensions of float32 is 768 * 4 = 3072 bytes per vector). +### Scoring +When using [word vector backed models](./configuration#words) with scoring set, a separate call is required before calling `index` as follows: + +```python +embeddings.score([(uid, text, None) for uid, text in enumerate(data)]) +embeddings.index([(uid, text, None) for uid, text in enumerate(data)]) +``` + +Two calls are required to support generator-backed iteration of data. The scoring index requires a separate full-pass of the data. + ## Search Once data is indexed, it is ready for search.