neuml/txtai
1
0
Fork 0
mirror of https://github.com/neuml/txtai.git synced 2026-10-12 02:19:52 +00:00
Code Issues Packages Projects Releases Wiki Activity

Add indexing guide to docs, closes #312

Browse Source
This commit is contained in:
davidmezzetti
2022-07-22 11:44:17 -04:00
parent 4db72769c0
commit eae295bbfa
5 changed files with 98 additions and 24 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/embeddings/configuration.md
+16 -8
View File
@@ -65,6 +65,20 @@ transform: function
When method is `external`, this function transforms input content into embeddings. The input to this function is a list of data. This method must return either a numpy array or list of numpy arrays.
### batch
```yaml
batch: int
```
Sets the transform batch size. This parameter controls how input streams are chunked and vectorized.
### encodebatch
```yaml
encodebatch: int
```
Sets the encode batch size. This parameter controls the underlying vector model batch size. This often corresponds to a GPU batch size, which controls GPU memory usage.
### backend
```yaml
backend: faiss|hnsw|annoy
@@ -83,6 +97,8 @@ faiss:
x = 4 * sqrt(embeddings count)
nprobe: search probe setting (int) - defaults to x/16 (as defined above)
for larger indexes
quantize: store vectors with 8-bit precision vs 32-bit (boolean)
defaults to false
mmap: load as on-disk index (boolean) - trade query response time for a
smaller RAM footprint, defaults to false
```
@@ -145,14 +161,6 @@ query:
Query translation model. Translates natural language queries to txtai compatible SQL statements.
### quantize
```yaml
quantize: boolean
```
Enables quanitization of generated sentence embeddings. If the index backend supports it, sentence embeddings will be stored with 8-bit precision vs 32-bit.
Only Faiss currently supports quantization.
### tokenize
```yaml
tokenize: boolean
docs/embeddings/index.md
+11 -15
View File
@@ -49,11 +49,13 @@ An embeddings instance can be created as follows:
embeddings = Embeddings({"path": "sentence-transformers/nli-mpnet-base-v2"})
```
The example above builds a transformers based embeddings instance. In this case, when loading and searching for data, a transformers model is used to vectorize data. The embeddings instance is [configuration driven](configuration) based on what is passed in the constructor. A number of different options are supported to cover a wide variety of use cases.
The example above builds a transformers-based embeddings instance. In this case, when loading and searching for data, a transformers model is used to vectorize data.
The embeddings instance is [configuration-driven](configuration) based on what is passed in the constructor. Embeddings indexes store vectors and can optionally [store content](configuration#content). Content storage enables additional filtering and data retrieval options.
## Index
After creating a new Embeddings instance, the next step is adding data to it.
After creating a new embeddings instance, the next step is adding data to it.
```python
embeddings.index([(uid, text, None) for uid, text in enumerate(data)])
@@ -67,19 +69,11 @@ The index method takes an iterable collection of tuples with three values.
| data | input data to index, can be text, a dictionary or object |
| tags | optional tags string, used to mark/label data as it's indexed |
The input iterable can be a list and/or a generator. [Generators](https://wiki.python.org/moin/Generators) help with indexing very large datasets as only portions of the data is in memory at any given time.
When the data element is a dictionary and it has a field named `text`, that will be used for indexing.
txtai buffers data to temporary storage along the way during indexing as embeddings vectors can be quite large (for example 768 dimensions of float32 is 768 * 4 = 3072 bytes per vector).
The input iterable can be a list or generator. [Generators](https://wiki.python.org/moin/Generators) help with indexing very large datasets as only portions of the data is in memory at any given time.
### Scoring
When using [word vector backed models](./configuration#words) with scoring set, a separate call is required before calling `index` as follows:
```python
embeddings.score([(uid, text, None) for uid, text in enumerate(data)])
embeddings.index([(uid, text, None) for uid, text in enumerate(data)])
```
Two calls are required to support generator-backed iteration of data. The scoring index requires a separate full-pass of the data.
More information on indexing can be found in the [index guide](indexing).
## Search
@@ -91,8 +85,10 @@ embeddings.search(query, limit)
The search method takes two parameters, the query and query limit. The results format is different based on whether [content](configuration/#content) is stored or not.
- List of `(id, score)` when content is disabled
- List of `{**query columns}` when content is enabled
- List of `(id, score)` when content is _not_ stored
- List of `{**query columns}` when content is stored
Both natural language and SQL queries are supported. More information can be found in the [query guide](query).
## More examples
docs/embeddings/indexing.md
+69
View File
@@ -0,0 +1,69 @@
# Index guide
This section gives an in-depth overview on how to index data with txtai. We'll cover vectorization, indexing, updating and deleting data.
## Vectorization
The most compute intensive step in building an index is vectorization. The [path](../configuration#path) parameter sets the path to the vector model. There is logic to automatically detect the vector model [method](../configuration#method) but it can also be set directly.
The [batch](../configuration#batch) and [encodebatch](../configuration#encodebatch) parameters control the vectorization process. Larger values for `batch` will pass larger batches to the vectorization method. Larger values for `encodebatch` will pass larger batches for each vector encode call. In the case of GPU vector models, larger values will consume more GPU memory.
Data is buffered to temporary storage during indexing as embeddings vectors can be quite large (for example 768 dimensions of float32 is 768 * 4 = 3072 bytes per vector). Once vectorization is complete, a mmapped array is created with all vectors for [Approximate Nearest Neighbor (ANN)](../configuration#backend) indexing.
## Setting a backend
As mentioned above, computed vectors are stored in an ANN. There are various index [backends](../configuration#backend) that can be configured. Faiss is the default backend.
## Content storage
Embeddings indexes can optionally [store content](../configuration#content). When this is enabled, the input content is saved in a database alongside the computed vectors. This enables filtering on additional fields and content retrieval.
## Index vs Upsert
Data is loaded into an index with either an [index](../methods#txtai.embeddings.base.Embeddings.index) or [upsert](../methods#txtai.embeddings.base.Embeddings.upsert) call.
```python
embeddings.index([(uid, text, None) for uid, text in enumerate(data)])
embeddings.upsert([(uid, text, None) for uid, text in enumerate(data)])
```
The `index` call will build a brand new index replacing an existing one. `upsert` will insert or update records. `upsert` ops do _not_ require a full index rebuild.
## Save
Indexes don't automatically persist. Indexes can be persisted using the [save](../methods/#txtai.embeddings.base.Embeddings.save) method.
```python
embeddings.save("/path/to/save")
```
Embeddings indexes can be restored using the [load](../methods/#txtai.embeddings.base.Embeddings.load) method.
In addition to saving indexes locally, they can also be persisted to [cloud storage](../configuration#cloud). This is especially useful when running from a serverless context or otherwise running on temporary compute.
## Delete
Content can be removed from the index with the [delete](../methods#txtai.embeddings.base.Embeddings.delete) method. This method takes a list of ids to delete.
```python
embeddings.delete(ids)
```
## Reindex
When [content storage](../configuration#content) is enabled, [reindex](../methods#txtai.embeddings.base.Embeddings.reindex) can be called to rebuild the index with new settings. For example, the backend can be switched from faiss to hnsw or the vector model can be updated. This prevents having to go back to the original raw data.
```python
embeddings.reindex({"path": "sentence-transformers/all-MiniLM-L6-v2", "backend": "hnsw"})
```
## Scoring
When using [word vector backed models](../configuration#words) with scoring set, a separate call is required before calling `index` as follows:
```python
embeddings.score([(uid, text, None) for uid, text in enumerate(data)])
embeddings.index([(uid, text, None) for uid, text in enumerate(data)])
```
Two calls are required to support generator-backed iteration of data. The scoring index requires a separate full-pass of the data.
docs/embeddings/query.md
+1 -1
View File
@@ -11,7 +11,7 @@ embeddings.search("feel good story")
embeddings.search("wildlife")
```
The queries above search the index for similarity matches on `feel good story` and `wildlife`. If content storage is enabled, a list of `{**query columns}` is returned. Otherwise, a list of `(id, score)` tuples are returned.
The queries above [search](../methods#txtai.embeddings.base.Embeddings.search) the index for similarity matches on `feel good story` and `wildlife`. If content storage is enabled, a list of `{**query columns}` is returned. Otherwise, a list of `(id, score)` tuples are returned.
## SQL
mkdocs.yml
+1
View File
@@ -52,6 +52,7 @@ nav:
- Embeddings:
- embeddings/index.md
- Configuration: embeddings/configuration.md
- Index Guide: embeddings/indexing.md
- Methods: embeddings/methods.md
- Query Guide: embeddings/query.md
- Pipeline: