neuml/txtai
1
0
Fork 0
mirror of https://github.com/neuml/txtai.git synced 2026-10-12 02:19:52 +00:00
Code Issues Packages Projects Releases Wiki Activity

Add clarification on building word vector backed models with scoring #247

Browse Source
This commit is contained in:
davidmezzetti
2022-03-17 10:05:03 -04:00
parent fd954eaf68
commit fa342a4f6a
1 changed files with 10 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/embeddings/index.md
+10
View File
@@ -71,6 +71,16 @@ The input iterable can be a list and/or a generator. [Generators](https://wiki.p
txtai buffers data to temporary storage along the way during indexing as embeddings vectors can be quite large (for example 768 dimensions of float32 is 768 * 4 = 3072 bytes per vector).
### Scoring
When using [word vector backed models](./configuration#words) with scoring set, a separate call is required before calling `index` as follows:
```python
embeddings.score([(uid, text, None) for uid, text in enumerate(data)])
embeddings.index([(uid, text, None) for uid, text in enumerate(data)])
```
Two calls are required to support generator-backed iteration of data. The scoring index requires a separate full-pass of the data.
## Search
Once data is indexed, it is ready for search.