Documentation update

Modified logger configuration in __init__.py (libraries shouldn't modify root logger)

.coveragerc

@@ -1,7 +1,8 @@
 [run]
 source = src/python
-concurrency = multiprocessing
+concurrency = multiprocessing,thread
 disable_warnings = no-data-collected
+omit = **/__main__.py
 
 [combine]
 disable_warnings = no-data-collected

.github/workflows/build.yml

@@ -10,21 +10,37 @@ jobs:
       matrix:
         os: [ubuntu-latest, macos-latest, windows-latest]
 
+    timeout-minutes: 60
     steps:
-      - uses: actions/checkout@v2
-      - uses: actions/setup-python@v2
+      - name: Checkout code
+        uses: actions/checkout@v3
+
+      - name: Install Python - Linux
+        uses: actions/setup-python@v4
         with:
-          python-version: 3.6
-      - uses: actions/setup-java@v1
+          python-version: 3.7
+        if: matrix.os == 'ubuntu-latest'
+
+      - name: Install Python - macOS/Windows
+        uses: actions/setup-python@v4
         with:
-          java-version: 1.8
+          python-version: 3.9
+        if: matrix.os != 'ubuntu-latest'
+
+      - name: Install Java
+        uses: actions/setup-java@v2
+        with:
+          distribution: "zulu"
+          java-version: "8"
 
       - name: Install dependencies - Linux
         run: sudo apt-get install libsndfile1
         if: matrix.os == 'ubuntu-latest'
 
-      - name: Install dependencies - MacOS
-        run: brew install libomp
+      - name: Install dependencies - macOS
+        run: |
+          brew install libomp
+          pip install torch==1.12.1 torchvision==0.13.1
         if: matrix.os == 'macos-latest'
 
       - name: Install dependencies - Windows
@@ -33,13 +49,13 @@ jobs:
 
       - name: Build
         run: |
-          pip install -U pip wheel coverage coveralls
-          pip install .[all]
+          pip install -U pip wheel
+          pip install .[all,dev]
           python -c "import nltk; nltk.download('punkt')"
           python --version
           make data coverage
 
-      - uses: pre-commit/action@v2.0.0
+      - uses: pre-commit/action@v3.0.0
         if: matrix.os == 'ubuntu-latest'
 
       - name: Test Coverage

.github/workflows/docs.yml

@@ -10,7 +10,9 @@ jobs:
       - uses: actions/checkout@v2
       - uses: actions/setup-python@v2
         with:
-          python-version: 3.6
-      - run: pip install mkdocs-material mkdocstrings git+https://github.com/neuml/txtai#egg=txtai[all]
+          python-version: 3.7
+      - run: |
+          pip install -U pip wheel
+          pip install .[all,dev]
       - run: mkdocs gh-deploy --force
  

.gitignore

@@ -1,5 +1,6 @@
 build/
 dist/
+docker/**/*.yml
 htmlcov/
 *egg-info/
 __pycache__/

.pre-commit-config.yaml

@@ -1,13 +1,13 @@
 repos:
   - repo: https://github.com/pycqa/pylint
-    rev: pylint-2.6.0
+    rev: v2.12.1
     hooks:
     - id: pylint
       args:
       - -d import-error
       - -d duplicate-code
   - repo: https://github.com/ambv/black
-    rev: stable
+    rev: 22.3.0
     hooks:
     - id: black
       language_version: python3

CITATION.cff

@@ -0,0 +1,11 @@
+cff-version: 1.2.0
+date-released: 2020-08-11
+message: "If you use this software, please cite it as below."
+title: "txtai"
+abstract: "Build AI-powered semantic search applications"
+url: "https://github.com/neuml/txtai"
+authors:
+- family-names: "Mezzetti"
+  given-names: "David"
+  affiliation: NeuML
+license: Apache-2.0

README.md

@@ -10,20 +10,20 @@
     <a href="https://github.com/neuml/txtai/releases">
         <img src="https://img.shields.io/github/release/neuml/txtai.svg?style=flat&color=success" alt="Version"/>
     </a>
-    <a href="https://github.com/neuml/txtai/releases">
-        <img src="https://img.shields.io/github/release-date/neuml/txtai.svg?style=flat&color=blue" alt="GitHub Release Date"/>
+    <a href="https://github.com/neuml/txtai">
+        <img src="https://img.shields.io/github/last-commit/neuml/txtai.svg?style=flat&color=blue" alt="GitHub last commit"/>
     </a>
     <a href="https://github.com/neuml/txtai/issues">
         <img src="https://img.shields.io/github/issues/neuml/txtai.svg?style=flat&color=success" alt="GitHub issues"/>
     </a>
-    <a href="https://github.com/neuml/txtai">
-        <img src="https://img.shields.io/github/last-commit/neuml/txtai.svg?style=flat&color=blue" alt="GitHub last commit"/>
+    <a href="https://join.slack.com/t/txtai/shared_invite/zt-1cagya4yf-DQeuZbd~aMwH5pckBU4vPg">
+        <img src="https://img.shields.io/badge/slack-join-blue?style=flat&logo=slack&logocolor=white" alt="Join Slack"/>
     </a>
     <a href="https://github.com/neuml/txtai/actions?query=workflow%3Abuild">
         <img src="https://github.com/neuml/txtai/workflows/build/badge.svg" alt="Build Status"/>
     </a>
     <a href="https://coveralls.io/github/neuml/txtai?branch=master">
-        <img src="https://img.shields.io/coveralls/github/neuml/txtai" alt="Coverage Status">
+        <img src="https://img.shields.io/coverallsCoverage/github/neuml/txtai" alt="Coverage Status">
     </a>
 </p>
 
@@ -35,6 +35,9 @@ txtai executes machine-learning workflows to transform data and build AI-powered
 
 Traditional search systems use keywords to find data. Semantic search applications have an understanding of natural language and identify results that have the same meaning, not necessarily the same keywords.
 
+![search](https://raw.githubusercontent.com/neuml/txtai/master/docs/images/search.png#gh-light-mode-only)
+![search](https://raw.githubusercontent.com/neuml/txtai/master/docs/images/search-dark.png#gh-dark-mode-only)
+
 Backed by state-of-the-art machine learning models, data is transformed into vector representations for search (also known as embeddings). Innovation is happening at a rapid pace, models can understand concepts in documents, audio, images and more.
 
 Summary of txtai features:
@@ -43,10 +46,17 @@ Summary of txtai features:
 - 📄 Create embeddings for text snippets, documents, audio, images and video. Supports transformers and word vectors.
 - 💡 Machine-learning pipelines to run extractive question-answering, zero-shot labeling, transcription, translation, summarization and text extraction
 - ↪️️ Workflows that join pipelines together to aggregate business logic. txtai processes can be microservices or full-fledged indexing workflows.
-- 🔗 API bindings for [JavaScript](https://github.com/neuml/txtai.js), [Java](https://github.com/neuml/txtai.java), [Rust](https://github.com/neuml/txtai.rs) and [Go](https://github.com/neuml/txtai.go)
+- ⚙️ Build with Python or YAML. API bindings available for [JavaScript](https://github.com/neuml/txtai.js), [Java](https://github.com/neuml/txtai.java), [Rust](https://github.com/neuml/txtai.rs) and [Go](https://github.com/neuml/txtai.go).
 - ☁️ Cloud-native architecture that scales out with container orchestration systems (e.g. Kubernetes)
 
-Applications range from similarity search to complex NLP-driven data extractions to generate structured databases. The following applications are powered by txtai.
+Applications range from similarity search to complex NLP-driven data extractions to generate structured databases. Semantic workflows transform and find data driven by user intent.
+
+![flows](https://raw.githubusercontent.com/neuml/txtai/master/docs/images/flows.png#gh-light-mode-only)
+![flows](https://raw.githubusercontent.com/neuml/txtai/master/docs/images/flows-dark.png#gh-dark-mode-only)
+
+The following applications are powered by txtai.
+
+![apps](https://raw.githubusercontent.com/neuml/txtai/master/apps.jpg)
 
 | Application  | Description  |
 |:----------|:-------------|
@@ -55,53 +65,88 @@ Applications range from similarity search to complex NLP-driven data extractions
 | [neuspo](https://neuspo.com) | Fact-driven, real-time sports event and news site |
 | [codequestion](https://github.com/neuml/codequestion) | Ask coding questions directly from the terminal |
 
-txtai is built with Python 3.6+, [Hugging Face Transformers](https://github.com/huggingface/transformers), [Sentence Transformers](https://github.com/UKPLab/sentence-transformers) and [FastAPI](https://github.com/tiangolo/fastapi)
+txtai is built with Python 3.7+, [Hugging Face Transformers](https://github.com/huggingface/transformers), [Sentence Transformers](https://github.com/UKPLab/sentence-transformers) and [FastAPI](https://github.com/tiangolo/fastapi)
 
 ## Why txtai?
 
+![why](https://raw.githubusercontent.com/neuml/txtai/master/docs/images/why.png#gh-light-mode-only)
+![why](https://raw.githubusercontent.com/neuml/txtai/master/docs/images/why-dark.png#gh-dark-mode-only)
+
 In addition to traditional search systems, a growing number of semantic search solutions are available, so why txtai?
 
-- `pip install txtai` is all that is needed to get started
-- Works well with both small and big data, prototypes can be built in a couple lines of code, scale up as needed
-- Rich data processing framework (pipelines and workflows) to pre and post process data
-- Work in your programming language of choice via the API
-- Modular with low install footprint, most dependencies optional and only required when you need them
-- Learn by example, notebooks cover all available functionality
+- Up and running in minutes with [pip](https://neuml.github.io/txtai/install/) or [Docker](https://neuml.github.io/txtai/cloud/)
+```python
+# Get started in a couple lines
+from txtai.embeddings import Embeddings
+
+embeddings = Embeddings({"path": "sentence-transformers/all-MiniLM-L6-v2"})
+embeddings.index([(0, "Correct", None), (1, "Not what we hoped", None)])
+embeddings.search("positive", 1)
+#[(0, 0.2986203730106354)]
+```
+- Build applications in your programming language of choice via the API
+```yaml
+# app.yml
+embeddings:
+    path: sentence-transformers/all-MiniLM-L6-v2
+```
+```bash
+CONFIG=app.yml uvicorn "txtai.api:app"
+curl -X GET "http://localhost:8000/search?query=positive"
+```
+- Connect machine learning models together to build intelligent data processing workflows
+- Works with both small and big data - scale when needed
+- Low footprint - install additional dependencies when you need them
+- [Learn by example](#examples) - notebooks cover all available functionality
 
 ## Installation
 
+![install](https://raw.githubusercontent.com/neuml/txtai/master/docs/images/install.png#gh-light-mode-only)
+![install](https://raw.githubusercontent.com/neuml/txtai/master/docs/images/install-dark.png#gh-dark-mode-only)
+
 The easiest way to install is via pip and PyPI
 
-    pip install txtai
+```
+pip install txtai
+```
 
-Python 3.6+ is supported. Using a Python [virtual environment](https://docs.python.org/3/library/venv.html) is recommended.
+Python 3.7+ is supported. Using a Python [virtual environment](https://docs.python.org/3/library/venv.html) is recommended.
 
 See the detailed [install instructions](https://neuml.github.io/txtai/install) for more information covering
-[installing from source](https://neuml.github.io/txtai/install/#install-from-source),
-[environment specific prerequisites](https://neuml.github.io/txtai/install/#environment-specific-prerequisites) and
-[optional dependencies](https://neuml.github.io/txtai/install/#optional-dependencies).
+[optional dependencies](https://neuml.github.io/txtai/install/#optional-dependencies), [environment specific prerequisites](https://neuml.github.io/txtai/install/#environment-specific-prerequisites), [installing from source](https://neuml.github.io/txtai/install/#install-from-source), [conda support](https://neuml.github.io/txtai/install/#conda) and how to [run with containers](https://neuml.github.io/txtai/cloud).
 
 ## Examples
 
+![examples](https://raw.githubusercontent.com/neuml/txtai/master/docs/images/examples.png#gh-light-mode-only)
+![examples](https://raw.githubusercontent.com/neuml/txtai/master/docs/images/examples-dark.png#gh-dark-mode-only)
+
 The examples directory has a series of notebooks and applications giving an overview of txtai. See the sections below.
 
 ### Semantic Search
 
-Build semantic/similarity/vector search applications.
+Build semantic/similarity/vector/neural search applications.
 
 | Notebook  | Description  |       |
 |:----------|:-------------|------:|
-| [Introducing txtai](https://github.com/neuml/txtai/blob/master/examples/01_Introducing_txtai.ipynb) | Overview of the functionality provided by txtai | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/01_Introducing_txtai.ipynb) |
+| [Introducing txtai](https://github.com/neuml/txtai/blob/master/examples/01_Introducing_txtai.ipynb) [▶️](https://www.youtube.com/watch?v=SIezMnVdmMs) | Overview of the functionality provided by txtai | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/01_Introducing_txtai.ipynb) |
 | [Build an Embeddings index with Hugging Face Datasets](https://github.com/neuml/txtai/blob/master/examples/02_Build_an_Embeddings_index_with_Hugging_Face_Datasets.ipynb) | Index and search Hugging Face Datasets | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/02_Build_an_Embeddings_index_with_Hugging_Face_Datasets.ipynb) |
 | [Build an Embeddings index from a data source](https://github.com/neuml/txtai/blob/master/examples/03_Build_an_Embeddings_index_from_a_data_source.ipynb)  | Index and search a data source with word embeddings | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/03_Build_an_Embeddings_index_from_a_data_source.ipynb) |
 | [Add semantic search to Elasticsearch](https://github.com/neuml/txtai/blob/master/examples/04_Add_semantic_search_to_Elasticsearch.ipynb)  | Add semantic search to existing search systems | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/04_Add_semantic_search_to_Elasticsearch.ipynb) |
-| [API Gallery](https://github.com/neuml/txtai/blob/master/examples/08_API_Gallery.ipynb) | Using txtai in JavaScript, Java, Rust and Go | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/08_API_Gallery.ipynb) |
 | [Similarity search with images](https://github.com/neuml/txtai/blob/master/examples/13_Similarity_search_with_images.ipynb) | Embed images and text into the same space for search | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/13_Similarity_search_with_images.ipynb) |
 | [Distributed embeddings cluster](https://github.com/neuml/txtai/blob/master/examples/15_Distributed_embeddings_cluster.ipynb) | Distribute an embeddings index across multiple data nodes | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/15_Distributed_embeddings_cluster.ipynb) |
+| [What's new in txtai 4.0](https://github.com/neuml/txtai/blob/master/examples/24_Whats_new_in_txtai_4_0.ipynb) | Content storage, SQL, object storage, reindex and compressed indexes | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/24_Whats_new_in_txtai_4_0.ipynb) |
+| [Anatomy of a txtai index](https://github.com/neuml/txtai/blob/master/examples/29_Anatomy_of_a_txtai_index.ipynb) | Deep dive into the file formats behind a txtai embeddings index | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/29_Anatomy_of_a_txtai_index.ipynb) |
+| [Custom Embeddings SQL functions](https://github.com/neuml/txtai/blob/master/examples/30_Embeddings_SQL_custom_functions.ipynb) | Add user-defined functions to Embeddings SQL | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/30_Embeddings_SQL_custom_functions.ipynb) |
+| [Model explainability](https://github.com/neuml/txtai/blob/master/examples/32_Model_explainability.ipynb) | Explainability for semantic search | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/32_Model_explainability.ipynb) |
+| [Query translation](https://github.com/neuml/txtai/blob/master/examples/33_Query_translation.ipynb) | Domain-specific natural language queries with query translation | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/33_Query_translation.ipynb) |
+| [Build a QA database](https://github.com/neuml/txtai/blob/master/examples/34_Build_a_QA_database.ipynb) | Question matching with semantic search | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/34_Build_a_QA_database.ipynb) |
+| [Embeddings components](https://github.com/neuml/txtai/blob/master/examples/37_Embeddings_index_components.ipynb) | Composable search with vector, SQL and scoring components | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/37_Embeddings_index_components.ipynb) |
+| [Semantic Graphs](https://github.com/neuml/txtai/blob/master/examples/38_Introducing_the_Semantic_Graph.ipynb) | Explore topics, data connectivity and run network analysis| [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/38_Introducing_the_Semantic_Graph.ipynb) |
+| [Topic Modeling with BM25](https://github.com/neuml/txtai/blob/master/examples/39_Classic_Topic_Modeling_with_BM25.ipynb) | Topic modeling backed by a BM25 index | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/39_Classic_Topic_Modeling_with_BM25.ipynb) |
 
-### Pipelines and Workflows
+### Pipelines
 
-NLP-backed data transformation pipelines and workflows.
+Transform data with NLP-backed pipelines.
 
 | Notebook  | Description  |       |
 |:----------|:-------------|------:|
@@ -111,10 +156,27 @@ NLP-backed data transformation pipelines and workflows.
 | [Apply labels with zero shot classification](https://github.com/neuml/txtai/blob/master/examples/07_Apply_labels_with_zero_shot_classification.ipynb) | Use zero shot learning for labeling, classification and topic modeling | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/07_Apply_labels_with_zero_shot_classification.ipynb) |
 | [Building abstractive text summaries](https://github.com/neuml/txtai/blob/master/examples/09_Building_abstractive_text_summaries.ipynb) | Run abstractive text summarization | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/09_Building_abstractive_text_summaries.ipynb) |
 | [Extract text from documents](https://github.com/neuml/txtai/blob/master/examples/10_Extract_text_from_documents.ipynb) | Extract text from PDF, Office, HTML and more | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/10_Extract_text_from_documents.ipynb) |
+| [Text to speech generation](https://github.com/neuml/txtai/blob/master/examples/40_Text_to_Speech_Generation.ipynb) | Generate speech from text | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/40_Text_to_Speech_Generation.ipynb) |
 | [Transcribe audio to text](https://github.com/neuml/txtai/blob/master/examples/11_Transcribe_audio_to_text.ipynb) | Convert audio files to text | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/11_Transcribe_audio_to_text.ipynb) |
 | [Translate text between languages](https://github.com/neuml/txtai/blob/master/examples/12_Translate_text_between_languages.ipynb) | Streamline machine translation and language detection | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/12_Translate_text_between_languages.ipynb) |
-| [Run pipeline workflows](https://github.com/neuml/txtai/blob/master/examples/14_Run_pipeline_workflows.ipynb) | Simple yet powerful constructs to efficiently process data | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/14_Run_pipeline_workflows.ipynb) |
-| [Transform tabular data with composable workflows](https://github.com/neuml/txtai/blob/master/examples/22_Transform_tabular_data_with_composable_workflows.ipynb) | Transform, index and search tabular data with workflows | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/22_Transform_tabular_data_with_composable_workflows.ipynb) |
+| [Generate image captions and detect objects](https://github.com/neuml/txtai/blob/master/examples/25_Generate_image_captions_and_detect_objects.ipynb) | Captions and object detection for images | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/25_Generate_image_captions_and_detect_objects.ipynb) |
+| [Near duplicate image detection](https://github.com/neuml/txtai/blob/master/examples/31_Near_duplicate_image_detection.ipynb) | Identify duplicate and near-duplicate images | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/31_Near_duplicate_image_detection.ipynb) |
+| [API Gallery](https://github.com/neuml/txtai/blob/master/examples/08_API_Gallery.ipynb) | Using txtai in JavaScript, Java, Rust and Go | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/08_API_Gallery.ipynb) |
+
+### Workflows
+
+Efficiently process data at scale.
+
+| Notebook  | Description  |       |
+|:----------|:-------------|------:|
+| [Run pipeline workflows](https://github.com/neuml/txtai/blob/master/examples/14_Run_pipeline_workflows.ipynb) [▶️](https://www.youtube.com/watch?v=UBMPDCn1gEU) | Simple yet powerful constructs to efficiently process data | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/14_Run_pipeline_workflows.ipynb) |
+| [Transform tabular data with composable workflows](https://github.com/neuml/txtai/blob/master/examples/22_Transform_tabular_data_with_composable_workflows.ipynb) | Transform, index and search tabular data | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/22_Transform_tabular_data_with_composable_workflows.ipynb) |
+| [Tensor workflows](https://github.com/neuml/txtai/blob/master/examples/23_Tensor_workflows.ipynb) | Performant processing of large tensor arrays | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/23_Tensor_workflows.ipynb) |
+| [Entity extraction workflows](https://github.com/neuml/txtai/blob/master/examples/26_Entity_extraction_workflows.ipynb) | Identify entity/label combinations | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/26_Entity_extraction_workflows.ipynb) |
+| [Workflow Scheduling](https://github.com/neuml/txtai/blob/master/examples/27_Workflow_scheduling.ipynb) | Schedule workflows with cron expressions | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/27_Workflow_scheduling.ipynb) |
+| [Push notifications with workflows](https://github.com/neuml/txtai/blob/master/examples/28_Push_notifications_with_workflows.ipynb) | Generate and push notifications with workflows | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/28_Push_notifications_with_workflows.ipynb) |
+| [Pictures are a worth a thousand words](https://github.com/neuml/txtai/blob/master/examples/35_Pictures_are_worth_a_thousand_words.ipynb) | Generate webpage summary images with DALL-E mini | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/35_Pictures_are_worth_a_thousand_words.ipynb) |
+| [Run txtai with native code](https://github.com/neuml/txtai/blob/master/examples/36_Run_txtai_in_native_code.ipynb) | Execute workflows in native code with the Python C API | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/36_Run_txtai_in_native_code.ipynb) |
 
 ### Model Training
 
@@ -147,10 +209,16 @@ Series of example applications with txtai. Links to hosted versions on [Hugging
 
 ### Further Reading
 
-- [Introducing txtai, AI-powered semantic search built on Transformers](https://towardsdatascience.com/introducing-txtai-an-ai-powered-search-engine-built-on-transformers-37674be252ec)
-- [Run machine-learning workflows to transform data and build AI-powered semantic search applications with txtai](https://towardsdatascience.com/run-machine-learning-workflows-to-transform-data-and-build-ai-powered-text-indices-with-txtai-43d769b566a7)
-- [Semantic search on the cheap](https://towardsdatascience.com/semantic-search-on-the-cheap-55940c0fcdab)
-- [Tutorial series on dev.to](https://dev.to/neuml/tutorial-series-on-txtai-ibg)
+![further](https://raw.githubusercontent.com/neuml/txtai/master/docs/images/further.png)
+
+- [Introducing txtai, AI-powered semantic search built on Transformers](https://medium.com/neuml/introducing-txtai-an-ai-powered-search-engine-built-on-transformers-37674be252ec)
+- [Tutorial series on Hashnode](https://neuml.hashnode.dev/series/txtai-tutorial) | [dev.to](https://dev.to/neuml/tutorial-series-on-txtai-ibg)
+- [What's new in txtai 5.0](https://medium.com/neuml/whats-new-in-txtai-5-0-e5c75a13b101) | [4.0](https://medium.com/neuml/whats-new-in-txtai-4-0-bbc3a65c3d1c)
+- [Getting started with semantic search](https://medium.com/neuml/getting-started-with-semantic-search-a9fd9d8a48cf)
+- [Run machine-learning workflows to transform data and build AI-powered semantic search applications with txtai](https://medium.com/neuml/run-machine-learning-workflows-to-transform-data-and-build-ai-powered-text-indices-with-txtai-43d769b566a7)
+- [Semantic search on the cheap](https://medium.com/neuml/semantic-search-on-the-cheap-55940c0fcdab)
+- [Serverless vector search with txtai](https://medium.com/neuml/serverless-vector-search-with-txtai-96f6163ab972)
+- [Insights from the txtai console](https://medium.com/neuml/insights-from-the-txtai-console-d307c28e149e)
 
 ### Contributing
 

README_ko.md

@@ -1,155 +0,0 @@
-<p align="center">
-    <img src="https://raw.githubusercontent.com/neuml/txtai/master/logo.png"/>
-</p>
-
-<h3 align="center">
-    <p>Build AI-powered semantic search applications</p>
-</h3>
-
-<p align="center">
-    <a href="https://github.com/neuml/txtai/releases">
-        <img src="https://img.shields.io/github/release/neuml/txtai.svg?style=flat&color=success" alt="Version"/>
-    </a>
-    <a href="https://github.com/neuml/txtai/releases">
-        <img src="https://img.shields.io/github/release-date/neuml/txtai.svg?style=flat&color=blue" alt="GitHub Release Date"/>
-    </a>
-    <a href="https://github.com/neuml/txtai/issues">
-        <img src="https://img.shields.io/github/issues/neuml/txtai.svg?style=flat&color=success" alt="GitHub issues"/>
-    </a>
-    <a href="https://github.com/neuml/txtai">
-        <img src="https://img.shields.io/github/last-commit/neuml/txtai.svg?style=flat&color=blue" alt="GitHub last commit"/>
-    </a>
-    <a href="https://github.com/neuml/txtai/actions?query=workflow%3Abuild">
-        <img src="https://github.com/neuml/txtai/workflows/build/badge.svg" alt="Build Status"/>
-    </a>
-    <a href="https://coveralls.io/github/neuml/txtai?branch=master">
-        <img src="https://img.shields.io/coveralls/github/neuml/txtai" alt="Coverage Status">
-    </a>
-</p>
-
--------------------------------------------------------------------------------------------------------------------------------------------------------
-
-txtai는 데이터를 변환하고 AI를 기반으로 한 의미론적 검색 애플리케이션을 만들기 위해 머신러닝 워크플로우를 실행합니다.
-
-![demo](https://raw.githubusercontent.com/neuml/txtai/master/demo.gif)
-
-기존에 사용하던 검색 시스템은 데이터를 찾기 위해서 키워드를 사용합니다. 의미론적 검색 애플리케이션은 자연어를 이해하고, 단지 동일한 키워드를 가진게 아닌 동일한 의미를 갖는 결과를 식별합니다.
-
-최첨단 머신 러닝 모델의 지원에 의해, 데이터는 검색을 위한 벡터 표현(임베딩(embeddings)이라고 알려진)으로 변환됩니다. 혁신은 빠른 속도로 일어나고 있으며, 이 모델은 문서, 오디오, 이미지 등의 개념을 이해할 수 있습니다.
-
-txtai 특징 요약:
-
-- 🔎 다수의 index backends를 이용해 대규모 유사성 검색 ([Faiss](https://github.com/facebookresearch/faiss), [Annoy](https://github.com/spotify/annoy), [Hnswlib](https://github.com/nmslib/hnswlib))
-- 📄 텍스트 조각, 문서, 오디오, 이미지 및 비디오에 대한 embeddings를 만듭니다. 변환기와 단어 벡터를 지원합니다.
-- 💡 머신 러닝 파이프라인은 회신, zero-shot 라벨링, 표기, 번역, 요약 및 텍스트 추출과 같은 질문을 추출하기 위해 사용됩니다.
-- ↪️️ 워크플로우는 파이프라인을 연결하여 비즈니스 로직을 합칩니다. txtai 프로세스는 마이크로 서비스 또는 완전한 인덱싱 워크플로우일 수 있습니다.
-- 🔗 [JavaScript](https://github.com/neuml/txtai.js), [Java](https://github.com/neuml/txtai.java), [Rust](https://github.com/neuml/txtai.rs)와 [Go](https://github.com/neuml/txtai.go)용 API 바인딩
-- ☁️ 컨테이너 배포 관리 시스템 (예: Kubernetes)으로 확장되는 Cloud-native 구조
-
-응용 프로그램은 복잡한 NLP기반 데이터 추출을 위한 유사성 검색에서 구조화된 데이터베이스를 생성하기까지의 범위입니다. 다음의 애플리케이션은 txtai를 기반으로 구동됩니다.
-
-| 응용 프로그램  | 설명  |
-|:----------|:-------------|
-| [paperai](https://github.com/neuml/paperai) | 의료/과학 논문을 위한 AI기반 문헌 발견 및 검토 엔진 |
-| [tldrstory](https://github.com/neuml/tldrstory) | 헤드라인과 본문에 대한 AI기반 이해 |
-| [neuspo](https://neuspo.com) | 사실 기반의 실시간 스포츠 이벤트 및 뉴스 사이트 |
-| [codequestion](https://github.com/neuml/codequestion) | 터미널에서 직접 코딩에 관한 질문을 합니다. |
-
-txtai는 Python 3.6+, [Hugging Face Transformers](https://github.com/huggingface/transformers), [Sentence Transformers](https://github.com/UKPLab/sentence-transformers) 그리고 [FastAPI](https://github.com/tiangolo/fastapi) 로 만들어졌습니다.
-
-## 왜 txtai인가요?
-
-기존 검색 시스템 외에도 점점 더 많은 의미론적 검색 솔루션을 사용할 수 있는데, 왜 txtai를 사용해야 하죠?
-
-- `pip install txtai` 만 있으면 시작할 수 있습니다.
-- 소규모 데이터와 빅 데이터에서 모두 잘 작동하며, 프로토타입을 몇 줄의 코드로 제작할 수 있고, 필요에 따라 확장할 수 있습니다.
-- 사전 및 사후 처리 데이터를 위한 풍부한 데이터 프로세싱 프레임워크(파이프라인과 워크플로우)
--	API를 통해 선택한 프로그래밍 언어로 작업할 수 있습니다.
--	설치 공간이 적고, 대부분의 종속성이 선택적이며, 필요할 때만 모듈을 요구할 수 있습니다. 
--	예시로 알아봅시다. 설명서는 모든 기능을 다룹니다.
-
-## 설치
-
-가장 쉬운 설치 방법은 via pip 과 PyPI입니다.
-
-    pip install txtai
-
-Python 3.6+ 이 지원됩니다.. 파이썬 [가상 환경](https://docs.python.org/3/library/venv.html) 을 사용하는 걸 추천드립니다.
-
-[소스 설치](https://neuml.github.io/txtai/install/#install-from-source),
-[환경별 필수 구성 요소](https://neuml.github.io/txtai/install/#environment-specific-prerequisites) 및
-[선택적 종속성](https://neuml.github.io/txtai/install/#optional-dependencies) 에 대한 자세한 정보는 [설치 지침](https://neuml.github.io/txtai/install) 을 참조하십시오.
-
-## 예시
-
-examples 디렉토리에는 txtai의 개요를 제공하는 일련의 설명서(문서) 및 애플리케이션이 있습니다. 아래 섹션을 참조하십시오.
-
-### 의미론적 검색
-
-의미론적/유사성/벡터 검색 애플리케이션을 제작합니다. 
-
-| 문서  | 설명  |       |
-|:----------|:-------------|------:|
-| [txtai 소개](https://github.com/neuml/txtai/blob/master/examples/01_Introducing_txtai.ipynb) | txtai에서 제공하는 기능 개요 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/01_Introducing_txtai.ipynb) |
-| [Hugging Face 데이터 셋으로 임베딩 인덱스 구축](https://github.com/neuml/txtai/blob/master/examples/02_Build_an_Embeddings_index_with_Hugging_Face_Datasets.ipynb) | Hugging Face 데이터 셋 색인 및 검색 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/02_Build_an_Embeddings_index_with_Hugging_Face_Datasets.ipynb) |
-| [데이터 소스로 임베딩 인덱스 구축](https://github.com/neuml/txtai/blob/master/examples/03_Build_an_Embeddings_index_from_a_data_source.ipynb)  | 단어 임베딩을 이용한 데이터 소스 색인 및 검색 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/03_Build_an_Embeddings_index_from_a_data_source.ipynb) |
-| [Elasticsearch에 의미론적 검색 추가](https://github.com/neuml/txtai/blob/master/examples/04_Add_semantic_search_to_Elasticsearch.ipynb)  | 존재하는 검색 시스템에 의미론적 검색 추가 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/04_Add_semantic_search_to_Elasticsearch.ipynb) |
-| [API 갤러리](https://github.com/neuml/txtai/blob/master/examples/08_API_Gallery.ipynb) | JavaScript, Java, Rust와 Go에서 txtai 사용 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/08_API_Gallery.ipynb) |
-| [이미지를 이용한 유사성 검색](https://github.com/neuml/txtai/blob/master/examples/13_Similarity_search_with_images.ipynb) | 검색을 위해 포함된 이미지와 텍스트를 동일한 공간에 삽입 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/13_Similarity_search_with_images.ipynb) |
-| [배포된 임베딩 클러스터](https://github.com/neuml/txtai/blob/master/examples/15_Distributed_embeddings_cluster.ipynb) | 여러 데이터 노드에 임베딩 인덱스 배포 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/15_Distributed_embeddings_cluster.ipynb) |
-
-### 파이프라인 및 워크플로우
-
-NLP 지원 데이터를 파이프라인과 워크플로우로 변환합니다.
-
-| 문서  | 설명  |       |
-|:----------|:-------------|------:|
-| [txtai에 관한 질의응답](https://github.com/neuml/txtai/blob/master/examples/05_Extractive_QA_with_txtai.ipynb) | txtai에 관한 질의응답 소개 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/05_Extractive_QA_with_txtai.ipynb) |
-| [Elasticsearch에 관한 질의응답](https://github.com/neuml/txtai/blob/master/examples/06_Extractive_QA_with_Elasticsearch.ipynb) | Elasticsearch에 관한 질의응답을 실행 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/06_Extractive_QA_with_Elasticsearch.ipynb) |
-| [구조화된 데이터 구축에 관한 질의응답](https://github.com/neuml/txtai/blob/master/examples/20_Extractive_QA_to_build_structured_data.ipynb) | 질의응답을 통해 구조화된 데이터 셋을 구축 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/20_Extractive_QA_to_build_structured_data.ipynb) |
-| [zero-shot 분류로 레이블을 적용](https://github.com/neuml/txtai/blob/master/examples/07_Apply_labels_with_zero_shot_classification.ipynb) | 라벨링, 분류 및 주제 모델링을 위해 zero-shot 학습을 사용 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/07_Apply_labels_with_zero_shot_classification.ipynb) |
-| [추상적인 텍스트 요약 작성](https://github.com/neuml/txtai/blob/master/examples/09_Building_abstractive_text_summaries.ipynb) | 추상적인 텍스트 요약을 실행 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/09_Building_abstractive_text_summaries.ipynb) |
-| [문서로부터 텍스트를 추출](https://github.com/neuml/txtai/blob/master/examples/10_Extract_text_from_documents.ipynb) | PDF, Office, HTML 등으로부터 텍스트를 추출 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/10_Extract_text_from_documents.ipynb) |
-| [오디오를 텍스트로 변환](https://github.com/neuml/txtai/blob/master/examples/11_Transcribe_audio_to_text.ipynb) | 오디오 파일을 텍스트로 변환 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/11_Transcribe_audio_to_text.ipynb) |
-| [언어 간에 텍스트를 번역](https://github.com/neuml/txtai/blob/master/examples/12_Translate_text_between_languages.ipynb) | 기계 번역 및 언어 감지를 간소화 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/12_Translate_text_between_languages.ipynb) |
-| [파이프라인과 워크플로우를 실행](https://github.com/neuml/txtai/blob/master/examples/14_Run_pipeline_workflows.ipynb) | 효율적으로 데이터를 처리하기 위한 간단하면서도 강력한 구조 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/14_Run_pipeline_workflows.ipynb) |
-| [구성 가능한 워크플로우로 테이블 형식 데이터 변환](https://github.com/neuml/txtai/blob/master/examples/22_Transform_tabular_data_with_composable_workflows.ipynb) | 워크플로우로 테이블 형식 데이터를 변환, 색인 생성및 검색 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/22_Transform_tabular_data_with_composable_workflows.ipynb) |
-
-### 모델 훈련
-
-NLP 모델을 훈련한다.
-
-| 문서  | 설명  |       |
-|:----------|:-------------|------:|
-| [텍스트 레이블러를 훈련](https://github.com/neuml/txtai/blob/master/examples/16_Train_a_text_labeler.ipynb) | 텍스트 시퀀스 분류 모델을 구축 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/16_Train_a_text_labeler.ipynb) |
-| [레이블을 제외하고 훈련](https://github.com/neuml/txtai/blob/master/examples/17_Train_without_labels.ipynb) | 새로운 모델을 훈련하기위해 zero-shot 분류기를 사용 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/17_Train_without_labels.ipynb) |
-| [QA 모델을 훈련](https://github.com/neuml/txtai/blob/master/examples/19_Train_a_QA_model.ipynb) | 질의응답 모델 구축 및 미세 조정 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/19_Train_a_QA_model.ipynb) |
-| [ONNX로 모델 내보내기 및 실행](https://github.com/neuml/txtai/blob/master/examples/18_Export_and_run_models_with_ONNX.ipynb) | ONNX로 모델을 내보내고, JavaScript, Java와 Rust에서 기본적으로 실행 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/18_Export_and_run_models_with_ONNX.ipynb) |
-| [다른 머신 러닝 모델들을 내보내기 및 실행](https://github.com/neuml/txtai/blob/master/examples/21_Export_and_run_other_machine_learning_models.ipynb) | scikit-learn, PyTorch 등에서 모델을 내보내기 및 실행 | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/21_Export_and_run_other_machine_learning_models.ipynb) |
-
-### 응용 프로그램
-
-txtai를 이용한 일련의 애플리케이션 예시입니다. [Hugging Face Spaces](https://hf.co/spaces)에서 호스팅된 버전에 대한 링크도 제공됩니다.
-
-| 응용 프로그램  | 설명  |       |
-|:-------------|:-------------|------:|
-| [Basic similarity search](https://github.com/neuml/txtai/blob/master/examples/similarity.py) | 기본 유사성 검색 예시입니다. 원래 txtai 데모에서 사용되었습니다. |[🤗](https://hf.co/spaces/NeuML/similarity)|
-| [Book search](https://github.com/neuml/txtai/blob/master/examples/books.py) | 도서 유사성 검색 애플리케이션입니다. 자연어 문장을 사용하여 책 설명 및 요청을 색인화합니다. |*오직 로컬에서만 실행*|
-| [Image search](https://github.com/neuml/txtai/blob/master/examples/images.py) | 이미지 유사성 검색 애플리케이션입니다. 이미지 디렉토리를 인덱싱하고 검색을 실행하여 입력한 요청과 유사한 이미지를 식별합니다. |[🤗](https://hf.co/spaces/NeuML/imagesearch)|
-| [Wiki search](https://github.com/neuml/txtai/blob/master/examples/wiki.py) | 위키백과를 검색하는 애플리케이션입니다. 위키백과 API를 요청하고 상위 결과를 요약합니다. |[🤗](https://hf.co/spaces/NeuML/wikisummary)|
-| [Workflow builder](https://github.com/neuml/txtai/blob/master/examples/workflows.py) | txtai 워크플로우를 구축하고 실행합니다. 요약, 텍스트 추출, 사본(transcription), 번역 및 유사성 검색 파이프라인을 함께 연결하여 통합 워크플로우를 실행합니다. |[🤗](https://hf.co/spaces/NeuML/txtai)|
-
-### 문서
-
-파이프라인, 워크플로우, 인덱싱 및 API에 대한 구성 설정을 포함한 [txtai 전체 설명서](https://neuml.github.io/txtai)입니다.
-
-### 추가적인 읽기
-
-- [Introducing txtai, AI-powered semantic search built on Transformers](https://towardsdatascience.com/introducing-txtai-an-ai-powered-search-engine-built-on-transformers-37674be252ec)
-- [Run machine-learning workflows to transform data and build AI-powered semantic search applications with txtai](https://towardsdatascience.com/run-machine-learning-workflows-to-transform-data-and-build-ai-powered-text-indices-with-txtai-43d769b566a7)
-- [Semantic search on the cheap](https://towardsdatascience.com/semantic-search-on-the-cheap-55940c0fcdab)
-- [Tutorial series on dev.to](https://dev.to/neuml/tutorial-series-on-txtai-ibg)
-
-### 기여하기
-
-txtai에 기여하고 싶은 사람은 [해당 가이드](https://github.com/neuml/.github/blob/master/CONTRIBUTING.md)를 참조하십시오.

docker/api.Dockerfile

@@ -1,42 +0,0 @@
-FROM nvidia/cuda:10.2-base-ubuntu18.04
-LABEL maintainer="NeuML"
-LABEL repository="txtai"
-
-# Locale environment variables
-ENV LC_ALL=C.UTF-8
-ENV LANG=C.UTF-8
-
-# Expose default uvicorn port
-EXPOSE 8000
-
-# Install required packages
-RUN apt-get update && \
-    apt-get -y --no-install-recommends install libgomp1 libsndfile1 gcc g++ python3.6 python3-dev python3-pip && \
-    rm -rf /var/lib/apt/lists
-
-# Install txtai project and dependencies
-RUN ln -s /usr/bin/python3.6 /usr/bin/python && \
-    python -m pip install --no-cache-dir -U pip wheel setuptools && \
-    python -m pip install --no-cache-dir txtai[api,pipeline] && \
-    python -c "import nltk; nltk.download('punkt')"
-
-# Cleanup build packages
-RUN apt-get -y purge gcc g++ python3-dev && apt-get -y autoremove
-
-# Generate YAML file
-RUN echo "path: /txtai" > index.yml && \
-    echo "writable: True" >> index.yml && \
-    echo "embeddings:" >> index.yml && \ 
-    echo "  method: transformers" >> index.yml && \
-    echo "  path: sentence-transformers/nli-mpnet-base-v2" >> index.yml && \
-    echo "extractor:" >> index.yml && \
-    echo "  path: distilbert-base-cased-distilled-squad" >> index.yml && \
-    echo "labels:" >> index.yml
-
-# Cache models in Docker image
-RUN python -c "from transformers import AutoModel; AutoModel.from_pretrained('sentence-transformers/nli-mpnet-base-v2')" && \
-    python -c "from transformers import AutoModel; AutoModel.from_pretrained('distilbert-base-cased-distilled-squad')" && \
-    python -c "from transformers import AutoModel; AutoModel.from_pretrained('facebook/bart-large-mnli')"
-
-# Start script on all interfaces (localhost + bridged interface)
-ENTRYPOINT CONFIG=index.yml uvicorn --host 0.0.0.0 "txtai.api:app"

docker/api/Dockerfile

@@ -0,0 +1,13 @@
+# Set base image
+ARG BASE_IMAGE=neuml/txtai-cpu
+FROM $BASE_IMAGE
+
+# Copy configuration
+COPY config.yml .
+
+# Run local API instance to cache models in container
+RUN python -c "from txtai.api import API; API('config.yml', False)"
+
+# Start server and listen on all interfaces
+ENV CONFIG "config.yml"
+ENTRYPOINT ["uvicorn", "--host", "0.0.0.0", "txtai.api:app"]

docker/aws/Dockerfile

@@ -0,0 +1,23 @@
+# Set base image
+ARG BASE_IMAGE=neuml/txtai-cpu
+FROM $BASE_IMAGE
+
+# Application script to copy into image
+ARG APP=api.py
+
+# Install Lambda Runtime Interface Client and Mangum ASGI bindings
+RUN pip install awslambdaric mangum
+
+# Copy configuration
+COPY config.yml .
+
+# Run local API instance to cache models in container
+RUN python -c "from txtai.api import API; API('config.yml', False)"
+
+# Copy application
+COPY $APP ./app.py
+
+# Start runtime client using default application handler
+ENV CONFIG "config.yml"
+ENTRYPOINT ["python", "-m", "awslambdaric"]
+CMD ["app.handler"]

docker/aws/api.py

@@ -0,0 +1,17 @@
+"""
+Lambda handler for a txtai API instance
+"""
+
+from mangum import Mangum
+
+from txtai.api import app, start
+
+# pylint: disable=C0103
+# Create FastAPI application instance wrapped by Mangum
+handler = None
+if not handler:
+    # Start application
+    start()
+
+    # Create handler
+    handler = Mangum(app, lifespan="off")

docker/aws/workflow.py

@@ -0,0 +1,32 @@
+"""
+Lambda handler for txtai workflows
+"""
+
+import json
+
+from txtai.api import API
+
+APP = None
+
+# pylint: disable=W0603,W0613
+def handler(event, context):
+    """
+    Runs a workflow using input event parameters.
+
+    Args:
+        event: input event
+        context: input context
+
+    Returns:
+        Workflow results
+    """
+
+    # Create (or get) global app instance
+    global APP
+    APP = APP if APP else API("config.yml")
+
+    # Get parameters from event body
+    event = json.loads(event["body"])
+
+    # Run workflow and return results
+    return {"statusCode": 200, "headers": {"Content-Type": "application/json"}, "body": list(APP.workflow(event["name"], event["elements"]))}

docker/base/Dockerfile

@@ -0,0 +1,38 @@
+# Set base image
+ARG BASE_IMAGE=ubuntu:20.04
+FROM $BASE_IMAGE
+
+# Install GPU-enabled version of PyTorch if set
+ARG GPU
+
+# Target CPU architecture
+ARG TARGETARCH
+
+# Set Python version (i.e. 3, 3.7, 3.8)
+ARG PYTHON_VERSION=3
+
+# List of txtai components to install
+ARG COMPONENTS=[all]
+
+# Locale environment variables
+ENV LC_ALL=C.UTF-8
+ENV LANG=C.UTF-8
+
+RUN \
+    # Install required packages
+    apt-get update && \
+    apt-get -y --no-install-recommends install libgomp1 libsndfile1 gcc g++ python${PYTHON_VERSION} python${PYTHON_VERSION}-dev python3-pip && \
+    rm -rf /var/lib/apt/lists && \
+    \
+    # Install txtai project and dependencies
+    ln -s /usr/bin/python${PYTHON_VERSION} /usr/bin/python && \
+    python -m pip install --no-cache-dir -U pip wheel setuptools && \
+    if [ -z ${GPU} ] && { [ -z ${TARGETARCH} ] || [ ${TARGETARCH} = "amd64" ] ;}; then pip install --no-cache-dir pip install torch==1.12.1+cpu torchvision==0.13.1+cpu -f https://download.pytorch.org/whl/torch_stable.html; fi && \
+    python -m pip install --no-cache-dir txtai${COMPONENTS} && \
+    python -c "import sys, importlib.util as util; 1 if util.find_spec('nltk') else sys.exit(); import nltk; nltk.download('punkt')" && \
+    \
+    # Cleanup build packages
+    apt-get -y purge gcc g++ python${PYTHON_VERSION}-dev && apt-get -y autoremove
+
+# Set default working directory
+WORKDIR /app

docker/schedule/Dockerfile

@@ -0,0 +1,12 @@
+# Set base image
+ARG BASE_IMAGE=neuml/txtai-cpu
+FROM $BASE_IMAGE
+
+# Copy configuration
+COPY config.yml .
+
+# Run local API instance to cache models in container
+RUN python -c "from txtai.api import API; API('config.yml', False)"
+
+# Start application and wait for completion. Scheduled workflows can run indefinitely. 
+ENTRYPOINT ["python", "-c", "from txtai.api import API; API('config.yml').wait()"]

docker/workflow/Dockerfile

@@ -0,0 +1,13 @@
+# Set base image
+ARG BASE_IMAGE=neuml/txtai-cpu
+FROM $BASE_IMAGE
+
+# Copy configuration
+COPY config.yml .
+
+# Run local API instance to cache models in container
+RUN python -c "from txtai.api import API; API('config.yml', False)"
+
+# Run workflow. Requires two command line arguments: name of workflow and input elements
+ENTRYPOINT ["python", "-c", "import sys; from txtai.api import API\nfor _ in API('config.yml').workflow(sys.argv[1], sys.argv[2:]): pass"]
+CMD ["workflow"]

docs/api.md

@@ -1,121 +0,0 @@
-# API
-
-txtai has a full-featured API that can optionally be enabled for any txtai process. All functionality found in txtai can be accessed via the API. The following is an example configuration and startup script for the API.
-
-Note that this configuration file enables all functionality. It is suggested that separate processes are used for each instance of a txtai component. Components can be joined together with workflows.
-
-```yaml
-# Index file path
-path: /tmp/index
-
-# Allow indexing of documents
-writable: True
-
-# Enbeddings index
-embeddings:
-  path: sentence-transformers/nli-mpnet-base-v2
-
-# Extractive QA
-extractor:
-  path: distilbert-base-cased-distilled-squad
-
-# Zero-shot labeling
-labels:
-
-# Similarity
-similarity:
-
-# Text segmentation
-segmentation:
-    sentences: true
-
-# Text summarization
-summary:
-
-# Text extraction
-textractor:
-    paragraphs: true
-    minlength: 100
-    join: true
-
-# Transcribe audio to text
-transcription:
-
-# Translate text between languages
-translation:
-
-# Workflow definitions
-workflow:
-    sumfrench:
-        tasks:
-            - action: textractor
-              task: storage
-              ids: false
-            - action: summary
-            - action: translation
-              args: ["fr"]
-    sumspanish:
-        tasks:
-            - action: textractor
-              task: url
-            - action: summary
-            - action: translation
-              args: ["es"]
-```
-
-Assuming this YAML content is stored in a file named index.yml, the following command starts the API process.
-
-```
-CONFIG=index.yml uvicorn "txtai.api:app"
-```
-
-uvicorn is a full-featured production ready server with support for SSL and more. See the [uvicorn deployment guide](https://www.uvicorn.org/deployment/) for details.
-
-## Docker
-
-A Dockerfile with commands to install txtai, all dependencies and default configuration is available in this repository.
-
-The Dockerfile can be copied from the docker directory on GitHub locally. The following commands show how to run the API process.
-
-```bash
-docker build -t txtai.api -f docker/api.Dockerfile .
-docker run --name txtai.api -p 8000:8000 --rm -it txtai.api
-
-# Alternatively, if nvidia-docker is installed, the build will support GPU runtimes
-docker run --name txtai.api --runtime=nvidia -p 8000:8000 --rm -it txtai.api
-```
-
-This will bring up an API instance without having to install Python, txtai or any dependencies on your machine!
-
-## Distributed embeddings clusters
-
-The API supports combining multiple API instances into a single logical embeddings index. An example configuration is shown below.
-
-```yaml
-cluster:
-    shards:
-        - http://127.0.0.1:8002
-        - http://127.0.0.1:8003
-```
-
-This configuration aggregates the API instances above as index shards. Data is evenly split among each of the shards at index time. Queries are run in parallel against each shard and the results are joined together. This method allows horizontal scaling and supports very large index clusters.
-
-This method is only recommended for data sets in the 1 billion+ records. The ANN libraries can easily support smaller data sizes and this method is not worth the additional complexity. At this time, new shards can not be added after building the initial index.
-
-## Differences between Python and API
-
-The txtai API provides all the major functionality found in this project. But there are differences due to the nature of JSON and differences across the supported programming languages. For example, any Python callable method is available at a named endpoint (i.e. instead of summary() the method call would be summary.summary()).
-Return types vary as tuples are returned as objects via the API.
-
-## Supported language bindings
-
-The following programming languages have txtai bindings:
-
-- [JavaScript](https://github.com/neuml/txtai.js)
-- [Java](https://github.com/neuml/txtai.java)
-- [Rust](https://github.com/neuml/txtai.rs)
-- [Go](https://github.com/neuml/txtai.go)
-
-See each of the projects above for details on how to install and use. Please add an issue to request additional language bindings!
-
-::: txtai.api.application

docs/api/cluster.md

@@ -0,0 +1,20 @@
+# Distributed embeddings clusters
+
+The API supports combining multiple API instances into a single logical embeddings index. An example configuration is shown below.
+
+```yaml
+cluster:
+    shards:
+        - http://127.0.0.1:8002
+        - http://127.0.0.1:8003
+```
+
+This configuration aggregates the API instances above as index shards. Data is evenly split among each of the shards at index time. Queries are run in parallel against each shard and the results are joined together. This method allows horizontal scaling and supports very large index clusters.
+
+This method is only recommended for data sets in the 1 billion+ records. The ANN libraries can easily support smaller data sizes and this method is not worth the additional complexity. At this time, new shards can not be added after building the initial index.
+
+See the link below for a detailed example covering distributed embeddings clusters.
+
+| Notebook  | Description  |       |
+|:----------|:-------------|------:|
+| [Distributed embeddings cluster](https://github.com/neuml/txtai/blob/master/examples/15_Distributed_embeddings_cluster.ipynb) | Distribute an embeddings index across multiple data nodes | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/15_Distributed_embeddings_cluster.ipynb) |

docs/api/configuration.md

@@ -0,0 +1,140 @@
+# Configuration
+
+Configuration is set through YAML. In most cases, YAML keys map to fields names in Python. The [example in the previous section](../) gave a full-featured example covering a wide array of configuration options.
+
+Each section below describes the available configuration settings.
+
+## Embeddings
+
+The configuration parser expects a top level `embeddings` key to be present in the YAML. All [embeddings configuration](../../embeddings/configuration) is supported.
+
+The following example defines an embeddings index.
+
+```yaml
+path: index path
+writable: true
+
+embeddings:
+  path: vector model
+  content: true
+```
+
+Three top level settings are available to control where indexes are saved and if an index is a read-only index.
+
+### path
+```yaml
+path: string
+```
+
+Path to save and load the embeddings index. Each API instance can only access a single index at a time.
+
+### writable
+```yaml
+writable: boolean
+```
+
+Determines if the input embeddings index is writable (true) or read-only (false). This allows serving a read-only index.
+
+### cloud
+[Cloud storage settings](../../embeddings/configuration/cloud) can be set under a `cloud` top level configuration group.
+
+## Pipeline
+
+Pipelines are loaded as top level configuration parameters. Pipeline names are automatically detected in the YAML configuration and created upon startup. All [pipelines](../../pipeline) are supported.
+
+The following example defines a series of pipelines. Note that entries below are the lower-case names of the pipeline class.
+
+```yaml
+caption:
+
+extractor:
+  path: model path
+
+labels:
+
+summary:
+
+tabular:
+
+translation:
+```
+
+Under each pipeline name, configuration settings for the pipeline can be set.
+
+## Workflow
+
+Workflows are defined under a top level `workflow` key. Each key under the `workflow` key is the name of the workflow. Under that is a `tasks` key with each task definition.
+
+The following example defines a workflow.
+
+```yaml
+workflow:
+  sumtranslate:
+    tasks:
+        - action: summary
+        - action: translation
+```
+
+### schedule
+
+Schedules a workflow using a [cron expression](../../workflow/schedule).
+
+```yaml
+workflow:
+  index:
+    schedule:
+      cron: 0/10 * * * * *
+      elements: ["api params"] 
+    tasks:
+      - task: service
+        url: api url
+      - action: index
+```
+
+### tasks
+```yaml
+tasks: list
+```
+
+Expects a list of workflow tasks. Each element defines a single workflow task. All [task configuration](../../workflow/task) is supported.
+
+A shorthand syntax for creating tasks is supported. This syntax will automatically map task strings to an `action:value` pair.
+
+Example below.
+
+```yaml
+workflow:
+  index:
+    tasks:
+      - action1
+      - action2
+```
+
+Each task element supports the following additional arguments.
+
+#### action
+```yaml
+action: string|list
+```
+
+Both single and multi-action tasks are supported.
+
+The action parameter works slightly different when passed via configuration. The parameter(s) needs to be converted into callable method(s). If action is a pipeline that has been defined in the current configuration, it will use that pipeline as the action.
+
+There are three special action names `index`, `upsert` and `search`. If `index` or `upsert` are used as the action, the task will collect workflow data elements and load them into defined the embeddings index. If `search` is used, the task will execute embeddings queries for each input data element.
+
+Otherwise, the action must be a path to a callable object or function. The configuration parser will resolve the function name and use that as the task action.
+
+#### task
+```yaml
+task: string
+```
+
+Optionally sets the type of task to create. For example, this could be a `file` task or a `retrieve` task. If this is not specified, a generic task is created. [The list of workflow tasks can be found here](../../workflow).
+
+#### args
+```yaml
+args: list
+```
+
+Optional list of static arguments to pass to the workflow task. These are combined with workflow data to pass to each `__call__`.

docs/api/index.md

@@ -0,0 +1,126 @@
+# API
+
+![api](../images/api.png#only-light)
+![api](../images/api-dark.png#only-dark)
+
+txtai has a full-featured API, backed by [FastAPI](https://github.com/tiangolo/fastapi), that can optionally be enabled for any txtai process. All functionality found in txtai can be accessed via the API.
+
+The following is an example configuration and startup script for the API.
+
+Note: This configuration file enables all functionality. For memory-bound systems, splitting pipelines into multiple instances is a best practice.
+
+```yaml
+# Index file path
+path: /tmp/index
+
+# Allow indexing of documents
+writable: True
+
+# Enbeddings index
+embeddings:
+  path: sentence-transformers/nli-mpnet-base-v2
+
+# Extractive QA
+extractor:
+  path: distilbert-base-cased-distilled-squad
+
+# Zero-shot labeling
+labels:
+
+# Similarity
+similarity:
+
+# Text segmentation
+segmentation:
+    sentences: true
+
+# Text summarization
+summary:
+
+# Text extraction
+textractor:
+    paragraphs: true
+    minlength: 100
+    join: true
+
+# Transcribe audio to text
+transcription:
+
+# Translate text between languages
+translation:
+
+# Workflow definitions
+workflow:
+    sumfrench:
+        tasks:
+            - action: textractor
+              task: url
+            - action: summary
+            - action: translation
+              args: ["fr"]
+    sumspanish:
+        tasks:
+            - action: textractor
+              task: url
+            - action: summary
+            - action: translation
+              args: ["es"]
+```
+
+Assuming this YAML content is stored in a file named config.yml, the following command starts the API process.
+
+```bash
+CONFIG=config.yml uvicorn "txtai.api:app"
+```
+
+uvicorn is a full-featured production ready server with support for SSL and more. See the [uvicorn deployment guide](https://www.uvicorn.org/deployment/) for details.
+
+## Connect to API
+
+The default port for the API is 8000. See the uvicorn link above to change this.
+
+txtai has a number of language bindings which abstract the API (see links below). Alternatively, code can be written to connect directly to the API. Documentation for a live running instance can be found at the `/docs` url (i.e. http://localhost:8000/docs). The following example runs a workflow using cURL.
+
+```bash
+curl \
+  -X POST "http://localhost:8000/workflow" \
+  -H "Content-Type: application/json" \
+  -d '{"name":"sumfrench", "elements": ["https://github.com/neuml/txtai"]}'
+```
+
+## Local instance
+
+A local instance can be instantiated. In this case, a txtai application runs internally, without any network connections, providing the same consolidated functionality. This enables running txtai in Python with configuration.
+
+The configuration above can be run in Python with:
+
+```python
+from txtai.app import Application
+
+# Load and run workflow
+app = Application(config.yml)
+app.workflow("sumfrench", ["https://github.com/neuml/txtai"])
+```
+
+See this [link for a full list of methods](./methods).
+
+## Run with containers
+
+The API can be containerized and run. This will bring up an API instance without having to install Python, txtai or any dependencies on your machine!
+
+[See this section for more information](../cloud/#api).
+
+## Supported language bindings
+
+The following programming languages have bindings with the txtai API:
+
+- [JavaScript](https://github.com/neuml/txtai.js)
+- [Java](https://github.com/neuml/txtai.java)
+- [Rust](https://github.com/neuml/txtai.rs)
+- [Go](https://github.com/neuml/txtai.go)
+
+See the link below for a detailed example covering how to use the API.
+
+| Notebook  | Description  |       |
+|:----------|:-------------|------:|
+| [API Gallery](https://github.com/neuml/txtai/blob/master/examples/08_API_Gallery.ipynb) | Using txtai in JavaScript, Java, Rust and Go | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/08_API_Gallery.ipynb) |

docs/api/methods.md

@@ -0,0 +1,13 @@
+# Methods
+
+::: txtai.api.API
+    selection:
+        inherited_members: true
+        filters:
+            - "!__del__"
+            - "!flows"
+            - "!function"
+            - "!limit"
+            - "!pipes"
+            - "!read"
+            - "!resolve"

docs/articles.md

@@ -1,6 +0,0 @@
-# Further reading
-
-- [Introducing txtai, AI-powered semantic search built on Transformers](https://towardsdatascience.com/introducing-txtai-an-ai-powered-search-engine-built-on-transformers-37674be252ec)
-- [Run machine-learning workflows to transform data and build AI-powered semantic search applications with txtai](https://towardsdatascience.com/run-machine-learning-workflows-to-transform-data-and-build-ai-powered-text-indices-with-txtai-43d769b566a7)
-- [Semantic search on the cheap](https://towardsdatascience.com/semantic-search-on-the-cheap-55940c0fcdab)
-- [Tutorial series on dev.to](https://dev.to/neuml/tutorial-series-on-txtai-ibg)

docs/cloud.md

@@ -0,0 +1,177 @@
+# Cloud
+
+![cloud](images/cloud.png#only-light)
+![cloud](images/cloud-dark.png#only-dark)
+
+Scalable cloud-native applications can be built with txtai. The following container runtimes are supported.
+
+- Container Orchestration Systems (i.e. Kubernetes)
+- Docker Engine
+- Serverless Compute
+
+Images for txtai are available on Docker Hub for [CPU](https://hub.docker.com/r/neuml/txtai-cpu) and [GPU](https://hub.docker.com/r/neuml/txtai-gpu) installs. The CPU install is recommended when GPUs aren't available given the image is half the size.
+
+The base txtai images have no models installed and models will be downloaded each time the container starts. Caching the models is recommended as that will significantly reduce container start times. This can be done a couple different ways.
+
+- Create a container with the [models cached](#container-image-model-caching)
+- Set the transformers cache environment variable and mount that volume when starting the image
+    ```bash
+    docker run -v <local dir>:/models -e TRANSFORMERS_CACHE=/models --rm -it <docker image>
+    ```
+
+## Build txtai images
+
+The txtai images found on Docker Hub are configured to support most situations. This image can be locally built with different options as desired.
+
+Examples build commands below.
+
+```bash
+# Get Dockerfile
+wget https://raw.githubusercontent.com/neuml/txtai/master/docker/base/Dockerfile
+
+# Build Ubuntu 18.04 image running Python 3.7
+docker build -t txtai --build-arg BASE_IMAGE=ubuntu:18.04 --build-arg PYTHON_VERSION=3.7 .
+
+# Build image with GPU support
+docker build -t txtai --build-arg GPU=1 .
+
+# Build minimal image with the base txtai components
+docker build -t txtai --build-arg COMPONENTS= .
+```
+
+## Container image model caching
+
+As mentioned previously, model caching is recommended to reduce container start times. The following commands demonstrate this. In all cases, it is assumed a config.yml file is present in the local directory with the desired configuration set.
+
+### API
+This section builds an image that caches models and starts an API service. The config.yml file should be configured with the desired components to expose via the API.
+
+The following is a sample config.yml file that creates an Embeddings API service.
+
+```yaml
+# config.yml
+writable: true
+
+embeddings:
+  path: sentence-transformers/nli-mpnet-base-v2
+  content: true
+```
+
+The next section builds the image and starts an instance.
+
+```bash
+# Get Dockerfile
+wget https://raw.githubusercontent.com/neuml/txtai/master/docker/api/Dockerfile
+
+# CPU build
+docker build -t txtai-api .
+
+# GPU build
+docker build -t txtai-api --build-arg BASE_IMAGE=neuml/txtai-gpu .
+
+# Run
+docker run -p 8000:8000 --rm -it txtai-api
+```
+
+### Service
+This section builds a scheduled workflow service. [More on scheduled workflows can be found here.](../workflow/schedule)
+
+```bash
+# Get Dockerfile
+wget https://raw.githubusercontent.com/neuml/txtai/master/docker/service/Dockerfile
+
+# CPU build
+docker build -t txtai-service .
+
+# GPU build
+docker build -t txtai-service --build-arg BASE_IMAGE=neuml/txtai-gpu .
+
+# Run
+docker run --rm -it txtai-service
+```
+
+### Workflow
+This section builds a single run workflow. [Example workflows can be found here.](../examples/#workflows)
+
+```bash
+# Get Dockerfile
+wget https://raw.githubusercontent.com/neuml/txtai/master/docker/workflow/Dockerfile
+
+# CPU build
+docker build -t txtai-workflow . 
+
+# GPU build
+docker build -t txtai-workflow --build-arg BASE_IMAGE=neuml/txtai-gpu .
+
+# Run
+docker run --rm -it txtai-workflow <workflow name> <workflow parameters>
+```
+
+## Serverless Compute
+
+One of the most powerful features of txtai is building YAML-configured applications with the "build once, run anywhere" approach. API instances and workflows can run locally, on a server, on a cluster or serverless.
+
+Serverless instances of txtai are supported on frameworks such as [AWS Lambda](https://aws.amazon.com/lambda/), [Google Cloud Functions](https://cloud.google.com/functions), [Azure Cloud Functions](https://azure.microsoft.com/en-us/services/functions/) and [Kubernetes](https://kubernetes.io/) with [Knative](https://knative.dev/docs/).
+
+### AWS Lambda
+
+The following steps show a basic example of how to build a serverless API instance with [AWS SAM](https://github.com/aws/serverless-application-model).
+
+- Create config.yml and template.yml
+
+```yaml
+# config.yml
+writable: true
+
+embeddings:
+  path: sentence-transformers/nli-mpnet-base-v2
+  content: true
+```
+
+```yaml
+# template.yml
+Resources:
+  txtai:
+    Type: AWS::Serverless::Function
+    Properties:
+      PackageType: Image
+      MemorySize: 3000
+      Timeout: 20
+      Events:
+        Api:
+          Type: Api
+          Properties:
+            Path: "/{proxy+}"
+            Method: ANY
+    Metadata:
+      Dockerfile: Dockerfile
+      DockerContext: ./
+      DockerTag: api
+```
+
+- Install [AWS SAM](https://pypi.org/project/aws-sam-cli/)
+
+- Run following
+
+```bash
+# Get Dockerfile and application
+wget https://raw.githubusercontent.com/neuml/txtai/master/docker/aws/api.py
+wget https://raw.githubusercontent.com/neuml/txtai/master/docker/aws/Dockerfile
+
+# Build the docker image
+sam build
+
+# Start API gateway and Lambda instance locally
+sam local start-api -p 8000 --warm-containers LAZY
+
+# Verify instance running (should return 0)
+curl http://localhost:8080/count
+```
+
+If successful, a local API instance is now running in a "serverless" fashion. This configuration can be deployed to AWS using SAM. [See this link for more information.](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-cli-command-reference-sam-deploy.html)
+
+### Kubernetes with Knative
+
+txtai scales with container orchestration systems. This can be self-hosted or with a cloud provider such as [Amazon Elastic Kubernetes Service](https://aws.amazon.com/eks/), [Google Kubernetes Engine](https://cloud.google.com/kubernetes-engine) and [Azure Kubernetes Service](https://azure.microsoft.com/en-us/services/kubernetes-service/). There are also other smaller providers with a managed Kubernetes offering.
+
+A full example covering how to build a serverless txtai application on Kubernetes with Knative [can be found here](https://medium.com/neuml/serverless-vector-search-with-txtai-96f6163ab972).

docs/embeddings.md

@@ -1,148 +0,0 @@
-# Embeddings
-An Embeddings instance is the engine that provides similarity search. Embeddings can be used to run ad-hoc similarity comparisions or build/search large indices.
-
-Embeddings parameters are set through the constructor. Examples below.
-
-```python
-# Transformers embeddings model
-Embeddings({"method": "transformers",
-            "path": "sentence-transformers/nli-mpnet-base-v2"})
-
-# Word embeddings model
-Embeddings({"method": "words",
-            "path": vectors,
-            "storevectors": True,
-            "scoring": "bm25",
-            "pca": 3,
-            "quantize": True})
-```
-
-## Configuration
-
-### method
-```yaml
-method: transformers|sentence-transformers|words
-```
-
-Sentence embeddings method to use. Options listed below.
-
-#### transformers
-
-Builds sentence embeddings using a transformers model. While this can be any transformers model, it works best with
-[models trained](https://huggingface.co/models?search=sentence-transformers) to build sentence embeddings.
-
-#### sentence-transformers
-
-Same as transformers but loads models with the sentence-transformers library.
-
-#### words
-
-Builds sentence embeddings using a word embeddings model.
-
-sentence-transformers and words require the [similarity](https://neuml.github.io/txtai/install/#similarity) extras package to be installed.
-
-The method is inferred using the _path_ if not provided.
-
-### path
-```yaml
-path: string
-```
-
-Required field that sets the path for a vectors model. When using a transformers/sentence-transformers model, this can be any model on the
-[Hugging Face Model Hub](https://huggingface.co/models) or a local file path. Otherwise, it must be a local file path to a word embeddings model.
-
-### backend
-```yaml
-backend: faiss|annoy|hnsw
-```
-
-Approximate Nearest Neighbor (ANN) index backend for storing generated sentence embeddings. Defaults to Faiss. Additional backends require the
-[similarity](https://neuml.github.io/txtai/install/#similarity) extras package to be installed.
-
-Backend-specific settings are set with a corresponding configuration object having the same name as the backend (i.e. annoy, faiss, or hnsw). None of these are required and are set to defaults if omitted.
-
-### faiss
-```yaml
-faiss:
-    components: Comma separated list of components - defaults to None
-    nprobe: search probe setting (int) - defaults to 6
-```
-
-See Faiss documentation on the [index factory](https://github.com/facebookresearch/faiss/wiki/The-index-factory) and [search](https://github.com/facebookresearch/faiss/wiki/Faster-search) for more information on these parameters.
-
-### annoy
-```yaml
-annoy:
-    ntrees: number of trees (int) - defaults to 10
-    searchk: search_k search setting (int) - defaults to -1
-```
-
-See [Annoy documentation](https://github.com/spotify/annoy#full-python-api) for more information on these parameters.
-
-### hnsw
-```yaml
-hnsw:
-    efconstruction:  ef_construction param for init_index (int) - defaults to 200
-    m: M param for init_index (int) - defaults to 16
-    randomseed: random-seed param for init_index (init) - defaults to 100
-    efsearch: ef search param (int) - defaults to None and not set
-```
-
-See [Hnswlib documentation](https://github.com/nmslib/hnswlib/blob/master/ALGO_PARAMS.md) for more information on these parameters.
-
-### quantize
-```yaml
-quantize: boolean
-```
-
-Enables quanitization of generated sentence embeddings. If the index backend supports it, sentence embeddings will be stored with 8-bit precision vs 32-bit.
-Only Faiss currently supports quantization.
-
-## Additional configuration for Transformers models
-
-### tokenize
-```yaml
-tokenize: boolean
-```
-
-Enables string tokenization (defaults to false). This method applies tokenization rules that only work with English language text and may increase the quality of
-English language sentence embeddings in some situations.
-
-## Additional configuration for Word embedding models
-
-Word embeddings provide a good tradeoff of performance to functionality for a similarity search system. With that being said, Transformers models are making great progress in scaling performance down to smaller models and are the preferred vector backend in txtai for most cases.
-
-Word embeddings models require the [similarity](https://neuml.github.io/txtai/install/#similarity) extras package to be installed.
-
-### storevectors
-```yaml
-storevectors: boolean
-```
-
-Enables copying of a vectors model set in path into the embeddings models output directory on save. This option enables a fully encapsulated index with no external file dependencies.
-
-### scoring
-```yaml
-scoring: bm25|tfidf|sif
-```
-
-A scoring model builds weighted averages of word vectors for a given sentence. Supports BM25, TF-IDF and SIF (smooth inverse frequency) methods. If a scoring method is not provided, mean sentence embeddings are built.
-
-### pca
-```yaml
-pca: int
-```
-
-Removes _n_ principal components from generated sentence embeddings. When enabled, a TruncatedSVD model is built to help with dimensionality reduction. After pooling of vectors creates a single sentence embedding, this method is applied.
-
-::: txtai.embeddings.Embeddings.__init__
-::: txtai.embeddings.Embeddings.batchsearch
-::: txtai.embeddings.Embeddings.batchsimilarity
-::: txtai.embeddings.Embeddings.batchtransform
-::: txtai.embeddings.Embeddings.index
-::: txtai.embeddings.Embeddings.load
-::: txtai.embeddings.Embeddings.save
-::: txtai.embeddings.Embeddings.score
-::: txtai.embeddings.Embeddings.search
-::: txtai.embeddings.Embeddings.similarity
-::: txtai.embeddings.Embeddings.transform

docs/embeddings/configuration/cloud.md

@@ -0,0 +1,59 @@
+# Cloud
+
+This section describes parameters used to sync compressed indexes with cloud storage. These parameters are only enabled if an embeddings index is stored as compressed. They are set via the [embeddings.load](../../methods/#txtai.embeddings.base.Embeddings.load) and [embeddings.save](../../methods/#txtai.embeddings.base.Embeddings.save) methods.
+
+## provider
+```yaml
+provider: string
+```
+
+The cloud storage provider, see [full list of providers here](https://libcloud.readthedocs.io/en/stable/storage/supported_providers.html).
+
+## container
+```yaml
+container: string
+```
+
+Container/bucket/directory name.
+
+## key
+```yaml
+key: string
+```
+
+Provider-specific access key. Can also be set via ACCESS_KEY environment variable. Ensure the configuration file is secured if added to the file.
+
+## secret
+```yaml
+secret: string
+```
+
+Provider-specific access secret. Can also be set via ACCESS_SECRET environment variable. Ensure the configuration file is secured if added to the file.
+
+## host
+```yaml
+host: string
+```
+
+Optional server host name. Set when using a local cloud storage server.
+
+## port
+```yaml
+port: int
+```
+
+Optional server port. Set when using a local cloud storage server.
+
+## token
+```yaml
+token: string
+```
+
+Optional temporary session token
+
+## region
+```yaml
+region: string
+```
+
+Optional parameter to specify the storage region, provider-specific.

docs/embeddings/configuration/index.md

@@ -0,0 +1,205 @@
+# Configuration
+
+This following describes available embeddings configuration. These parameters are set via the [Embeddings constructor](../methods#txtai.embeddings.base.Embeddings.__init__).
+
+## path
+```yaml
+path: string
+```
+
+Sets the path for a vectors model. When using a transformers/sentence-transformers model, this can be any model on the
+[Hugging Face Model Hub](https://huggingface.co/models) or a local file path. Otherwise, it must be a local file path to a word embeddings model.
+
+## method
+```yaml
+method: transformers|sentence-transformers|words|external
+```
+
+Sentence embeddings method to use. If the method is not provided, it is inferred using the `path`.
+
+`sentence-transformers` and `words` require the [similarity](../../install/#similarity) extras package to be installed.
+
+### transformers
+
+Builds sentence embeddings using a transformers model. While this can be any transformers model, it works best with
+[models trained](https://huggingface.co/models?pipeline_tag=sentence-similarity) to build sentence embeddings.
+
+### sentence-transformers
+
+Same as transformers but loads models with the [sentence-transformers](https://github.com/UKPLab/sentence-transformers) library.
+
+### words
+
+Builds sentence embeddings using a word embeddings model. Transformers models are the preferred vector backend in most cases. Word embeddings models may be deprecated in the future.
+
+#### storevectors
+```yaml
+storevectors: boolean
+```
+
+Enables copying of a vectors model set in path into the embeddings models output directory on save. This option enables a fully encapsulated index with no external file dependencies.
+
+#### scoring
+```yaml
+scoring: bm25|tfidf|sif
+```
+
+A scoring model builds weighted averages of word vectors for a given sentence. Supports BM25, TF-IDF and SIF (smooth inverse frequency) methods. If a scoring method is not provided, mean sentence embeddings are built.
+
+#### pca
+```yaml
+pca: int
+```
+
+Removes _n_ principal components from generated sentence embeddings. When enabled, a TruncatedSVD model is built to help with dimensionality reduction. After pooling of vectors creates a single sentence embedding, this method is applied.
+
+### external
+
+Sentence embeddings are loaded via an external model or API. Requires setting the [transform](#transform) parameter to a function that translates data into vectors.
+
+#### transform
+```yaml
+transform: function
+```
+
+When method is `external`, this function transforms input content into embeddings. The input to this function is a list of data. This method must return either a numpy array or list of numpy arrays.
+
+## batch
+```yaml
+batch: int
+```
+
+Sets the transform batch size. This parameter controls how input streams are chunked and vectorized.
+
+## encodebatch
+```yaml
+encodebatch: int
+```
+
+Sets the encode batch size. This parameter controls the underlying vector model batch size. This often corresponds to a GPU batch size, which controls GPU memory usage.
+
+## tokenize
+```yaml
+tokenize: boolean
+```
+
+Enables string tokenization (defaults to false). This method applies tokenization rules that only work with English language text and may increase the quality of
+English language sentence embeddings in some situations.
+
+## backend
+```yaml
+backend: faiss|hnsw|annoy
+```
+
+Approximate Nearest Neighbor (ANN) index backend for storing generated sentence embeddings. `Defaults to faiss`. Additional backends require the
+[similarity](../../install/#similarity) extras package to be installed.
+
+Backend-specific settings are set with a corresponding configuration object having the same name as the backend (i.e. annoy, faiss, or hnsw). None of these are required and are set to defaults if omitted.
+
+### faiss
+```yaml
+faiss:
+    components: comma separated list of components - defaults to "Flat" for small
+                indices and "IVFx,Flat" for larger indexes where
+                x = 4 * sqrt(embeddings count)
+    nprobe: search probe setting (int) - defaults to x/16 (as defined above)
+            for larger indexes
+    quantize: store vectors with 8-bit precision vs 32-bit (boolean)
+              defaults to false
+    mmap: load as on-disk index (boolean) - trade query response time for a
+          smaller RAM footprint, defaults to false
+```
+
+See the following Faiss documentation links for more information.
+
+- [Guidelines for choosing an index](https://github.com/facebookresearch/faiss/wiki/Guidelines-to-choose-an-index)
+- [Index configuration summary](https://github.com/facebookresearch/faiss/wiki/Faiss-indexes)
+- [Index Factory](https://github.com/facebookresearch/faiss/wiki/The-index-factory)
+- [Search Tuning](https://github.com/facebookresearch/faiss/wiki/Faster-search)
+
+### hnsw
+```yaml
+hnsw:
+    efconstruction:  ef_construction param for init_index (int) - defaults to 200
+    m: M param for init_index (int) - defaults to 16
+    randomseed: random-seed param for init_index (int) - defaults to 100
+    efsearch: ef search param (int) - defaults to None and not set
+```
+
+See [Hnswlib documentation](https://github.com/nmslib/hnswlib/blob/master/ALGO_PARAMS.md) for more information on these parameters.
+
+### annoy
+```yaml
+annoy:
+    ntrees: number of trees (int) - defaults to 10
+    searchk: search_k search setting (int) - defaults to -1
+```
+
+See [Annoy documentation](https://github.com/spotify/annoy#full-python-api) for more information on these parameters. Note that annoy indexes can not be modified after creation, upserts/deletes and other modifications are not supported.
+
+## content
+```yaml
+content: string|boolean
+```
+
+Enables content storage. When true, the default content storage engine will be used. Otherwise, the string must specify the supported content storage engine to use.
+
+## functions
+```yaml
+functions: list
+```
+
+List of functions with user-defined SQL functions, only used when [content](#content) is enabled. Each list element must be one of the following:
+
+- function
+- callable object
+- dict with fields for name, argcount and function
+
+[An example can be found here](../query#custom-sql-functions).
+
+## query
+```yaml
+query:
+    path: sets the path for the query model - this can be any model on the
+          Hugging Face Model Hub or a local file path.
+    prefix: text prefix to prepend to all inputs
+    maxlength: maximum generated sequence length
+```
+
+Query translation model. Translates natural language queries to txtai compatible SQL statements.
+
+## graph
+```yaml
+graph:
+    backend: graph network backend (string), defaults to "networkx"
+    batchsize: batch query size, used to query embeddings index (int)
+               defaults to 256
+    limit: maximum number of results to return per embeddings query (int)
+           defaults to 15
+    minscore: minimum score required to consider embeddings query matches (float)
+              defaults to 0.1
+    approximate: when true, queries only run for nodes without edges (boolean)
+                 defaults to true
+    topics: see below
+```
+
+Enables graph storage. When set, a graph network is built using the embeddings index. Graph nodes are synced with each embeddings index operation (index/upsert/delete). Graph edges are created using the embeddings index upon completion of each index/upsert/delete embeddings index call.
+
+Defaults are tuned so that in most cases these values don't need to be changed. 
+
+### topics
+```yaml
+topics:
+    algorithm: community detection algorithm (string), options are
+               louvain (default), greedy, lpa
+    level: controls number of topics (string), options are best (default) or first
+    resolution: controls number of topics (int), larger values create more
+                topics (int), defaults to 100
+    labels: scoring index method used to build topic labels (string)
+            options are bm25 (default), tfidf, sif
+    terms: number of frequent terms to use for topic labels (int), defaults to 4
+    categories: optional list of categories used to group topics, allows
+                granular topics with broad categories grouping topics
+```
+
+Enables topic modeling. Defaults are tuned so that in most cases these values don't need to be changed (except for categories). These parameters are available for advanced use cases where one wants full control over the community detection process.

docs/embeddings/index.md

@@ -0,0 +1,95 @@
+# Embeddings
+
+![embeddings](../images/embeddings.png#only-light)
+![embeddings](../images/embeddings-dark.png#only-dark)
+
+Embeddings is the engine that delivers semantic search. Data is transformed into embeddings vectors where similar concepts will produce similar vectors. Indexes both large and small are built with these vectors. The indexes are used to find results that have the same meaning, not necessarily the same keywords.
+
+The following code snippet shows how to build and search an embeddings index.
+
+```python
+from txtai.embeddings import Embeddings
+
+# Create embeddings model, backed by sentence-transformers & transformers
+embeddings = Embeddings({"path": "sentence-transformers/nli-mpnet-base-v2"})
+
+data = [
+  "US tops 5 million confirmed virus cases",
+  "Canada's last fully intact ice shelf has suddenly collapsed, " +
+  "forming a Manhattan-sized iceberg",
+  "Beijing mobilises invasion craft along coast as Taiwan tensions escalate",
+  "The National Park Service warns against sacrificing slower friends " +
+  "in a bear attack",
+  "Maine man wins $1M from $25 lottery ticket",
+  "Make huge profits without work, earn up to $100,000 a day"
+]
+
+# Create an index for the list of text
+embeddings.index([(uid, text, None) for uid, text in enumerate(data)])
+
+print("%-20s %s" % ("Query", "Best Match"))
+print("-" * 50)
+
+# Run an embeddings search for each query
+for query in ("feel good story", "climate change", "public health story", "war",
+              "wildlife", "asia", "lucky", "dishonest junk"):
+    # Extract uid of first result
+    # search result format: (uid, score)
+    uid = embeddings.search(query, 1)[0][0]
+
+    # Print text
+    print("%-20s %s" % (query, data[uid]))
+```
+
+## Build
+
+An embeddings instance can be created as follows:
+
+```python
+embeddings = Embeddings({"path": "sentence-transformers/nli-mpnet-base-v2"})
+```
+
+The example above builds a transformers-based embeddings instance. In this case, when loading and searching for data, a transformers model is used to vectorize data.
+
+The embeddings instance is [configuration-driven](configuration) based on what is passed in the constructor. Embeddings indexes store vectors and can optionally [store content](configuration#content). Content storage enables additional filtering and data retrieval options.
+
+## Index
+
+After creating a new embeddings instance, the next step is adding data to it.
+
+```python
+embeddings.index([(uid, text, None) for uid, text in enumerate(data)])
+```
+
+The index method takes an iterable collection of tuples with three values. 
+
+| Element     | Description                                                   |
+| ----------- | ------------------------------------------------------------- |
+| id          | unique record id                                              |
+| data        | input data to index, can be text, a dictionary or object      |
+| tags        | optional tags string, used to mark/label data as it's indexed |
+
+When the data element is a dictionary and it has a field named `text`, that will be used for indexing.
+
+The input iterable can be a list or generator. [Generators](https://wiki.python.org/moin/Generators) help with indexing very large datasets as only portions of the data is in memory at any given time.
+
+More information on indexing can be found in the [index guide](indexing).
+
+## Search
+
+Once data is indexed, it is ready for search.
+
+```python
+embeddings.search(query, limit)
+```
+
+The search method takes two parameters, the query and query limit. The results format is different based on whether [content](configuration/#content) is stored or not.
+
+- List of `(id, score)` when content is _not_ stored
+- List of `{**query columns}` when content is stored
+
+Both natural language and SQL queries are supported. More information can be found in the [query guide](query).
+
+## More examples
+
+See [this link](../examples/#semantic-search) for a full list of embeddings examples.

docs/embeddings/indexing.md

@@ -0,0 +1,118 @@
+# Index guide
+
+![indexing](../images/indexing.png#only-light)
+![indexing](../images/indexing-dark.png#only-dark)
+
+This section gives an in-depth overview on how to index data with txtai. We'll cover vectorization, indexing, updating and deleting data.
+
+## Vectorization
+
+The most compute intensive step in building an index is vectorization. The [path](../configuration#path) parameter sets the path to the vector model. There is logic to automatically detect the vector model [method](../configuration#method) but it can also be set directly.
+
+The [batch](../configuration#batch) and [encodebatch](../configuration#encodebatch) parameters control the vectorization process. Larger values for `batch` will pass larger batches to the vectorization method. Larger values for `encodebatch` will pass larger batches for each vector encode call. In the case of GPU vector models, larger values will consume more GPU memory.
+
+Data is buffered to temporary storage during indexing as embeddings vectors can be quite large (for example 768 dimensions of float32 is 768 * 4 = 3072 bytes per vector). Once vectorization is complete, a mmapped array is created with all vectors for [Approximate Nearest Neighbor (ANN)](../configuration#backend) indexing.
+
+## Setting a backend
+
+As mentioned above, computed vectors are stored in an ANN. There are various index [backends](../configuration#backend) that can be configured. Faiss is the default backend.
+
+## Content storage
+
+Embeddings indexes can optionally [store content](../configuration#content). When this is enabled, the input content is saved in a database alongside the computed vectors. This enables filtering on additional fields and content retrieval.
+
+## Index vs Upsert
+
+Data is loaded into an index with either an [index](../methods#txtai.embeddings.base.Embeddings.index) or [upsert](../methods#txtai.embeddings.base.Embeddings.upsert) call.
+
+```python
+embeddings.index([(uid, text, None) for uid, text in enumerate(data)])
+embeddings.upsert([(uid, text, None) for uid, text in enumerate(data)])
+```
+
+The `index` call will build a brand new index replacing an existing one. `upsert` will insert or update records. `upsert` ops do _not_ require a full index rebuild.
+
+## Save
+
+Indexes can be stored in a directory using the [save](../methods/#txtai.embeddings.base.Embeddings.save) method.
+
+```python
+embeddings.save("/path/to/save")
+```
+
+Compressed indexes are also supported.
+
+```python
+embeddings.save("/path/to/save/index.tar.gz")
+```
+
+In addition to saving indexes locally, they can also be persisted to [cloud storage](../configuration/cloud).
+
+```python
+embeddings.save("/path/to/save/index.tar.gz", cloud={...})
+```
+
+This is especially useful when running in a serverless context or otherwise running on temporary compute. Cloud storage is only supported with compressed indexes.
+
+Embeddings indexes can be restored using the [load](../methods/#txtai.embeddings.base.Embeddings.load) method.
+
+```python
+embeddings.load("/path/to/load")
+```
+
+## Delete
+
+Content can be removed from the index with the [delete](../methods#txtai.embeddings.base.Embeddings.delete) method. This method takes a list of ids to delete.
+
+```python
+embeddings.delete(ids)
+```
+
+## Reindex
+
+When [content storage](../configuration#content) is enabled, [reindex](../methods#txtai.embeddings.base.Embeddings.reindex) can be called to rebuild the index with new settings. For example, the backend can be switched from faiss to hnsw or the vector model can be updated. This prevents having to go back to the original raw data. 
+
+```python
+embeddings.reindex({"path": "sentence-transformers/all-MiniLM-L6-v2", "backend": "hnsw"})
+```
+
+## Graph
+
+Dimensionality reduction with UMAP combined with HDBSCAN is a popular topic modeling method found in a number of libraries. txtai takes a different approach with a semantic graph.
+
+Enabling a [graph network](../configuration#graph) adds a semantic graph at index time as data is being vectorized. Vector embeddings are used to create relationships in the graph. Finally, community detection algorithms build topic clusters. Semantic graphs can also be used to analyze data connectivity.
+
+This approach has the advantage of only having to vectorize data once. It also has the advantage of better topic precision given there isn't a dimensionality reduction operation (UMAP). Semantic graph examples are shown below.
+
+Get a mapping of discovered topics to associated ids.
+
+```python
+embeddings.graph.topics
+```
+
+Show the most central nodes in the index.
+
+```python
+embeddings.graph.centrality()
+```
+
+Show how node 1 and node 2 are connected in the graph.
+
+```python
+embeddings.graph.showpath(id1, id2)
+```
+
+Graphs are persisted alongside an embeddings index. Each save and load will also save and load the graph.
+
+## Scoring
+
+When using [word vector backed models](../configuration#words) with scoring set, a separate call is required before calling `index` as follows:
+
+```python
+embeddings.score([(uid, text, None) for uid, text in enumerate(data)])
+embeddings.index([(uid, text, None) for uid, text in enumerate(data)])
+```
+
+Two calls are required to support generator-backed iteration of data. The scoring index requires a separate full-pass of the data.
+
+Scoring instances can also create a standalone keyword-based index (BM25, TF-IDF). See [this link](../../examples/#semantic-search) to learn more.

docs/embeddings/methods.md

@@ -0,0 +1,12 @@
+# Methods
+
+::: txtai.embeddings.Embeddings
+    selection:
+        filters:
+            - "!createdatabase"
+            - "!creategraph"
+            - "!checkarchive"
+            - "!configure"
+            - "!loadquery"
+            - "!loadvectors"
+            - "!normalize"

docs/embeddings/query.md

@@ -0,0 +1,167 @@
+# Query guide
+
+![query](../images/query.png#only-light)
+![query](../images/query-dark.png#only-dark)
+
+This section covers how to query data with txtai. The simplest way to search for data is building a natural language string with the desired content to find. txtai also supports querying with SQL. We'll cover both methods here.
+
+## Natural language queries
+
+In the simplest case, the query is text and the results are index text that is most similar to the query text.
+
+```python
+embeddings.search("feel good story")
+embeddings.search("wildlife")
+```
+
+The queries above [search](../methods#txtai.embeddings.base.Embeddings.search) the index for similarity matches on `feel good story` and `wildlife`. If content storage is enabled, a list of `{**query columns}` is returned. Otherwise, a list of `(id, score)` tuples are returned.
+
+## SQL
+
+txtai supports more complex queries with SQL. This is only supported if [content storage](../configuration/#content) is enabled. txtai has a translation layer that analyzes input SQL statements and combines similarity results with content stored in a relational database.
+
+SQL queries are run through `embeddings.search` like natural language queries but the examples below only show the SQL query for conciseness.
+
+```python
+embeddings.search("SQL query")
+```
+
+### Similar clause
+
+The similar clause is a txtai function that enables similarity searches with SQL.
+
+```sql
+SELECT id, text, score FROM txtai WHERE similar('feel good story')
+SELECT id, text, score FROM txtai WHERE similar('feel good story')
+```
+
+The similar clause takes two arguments:
+
+```sql
+similar("query", "number of candidates")
+```
+
+| Argument              | Description                            |
+| --------------------- | ---------------------------------------|
+| query                 | natural language query to run          |
+| number of candidates  | number of candidate results to return  |
+
+The txtai query layer has to join results from two separate components, a relational store and a similarity index. With a similar clause, a similarity search is run and those ids are fed to the underlying database query.
+
+The number of candidates should be larger than the desired number of results when applying additional filter clauses. This ensures that `limit` results are still returned after applying additional filters. If the number of candidates is not specified, it is defaulted as follows:
+
+- For a single query filter clause, the default is the query limit
+- With multiple filtering clauses, the default is 10x the query limit
+
+### Dynamic columns
+
+Content can be indexed in multiple ways when content storage is enabled. [Remember that input documents](../#index) take the form of `(id, data, tags)` tuples. If data is a string, then content is primarily filtered with similar clauses. If data is a dictionary, then all fields in the dictionary are indexed and searchable.
+
+For example:
+
+```python
+embeddings.index([(0, {"text": "text to index", "flag": True,
+                       "entry": "2022-01-01"}, None)])
+```
+
+With the above input data, queries can now have more complex filters.
+
+```sql
+SELECT text, flag, entry FROM txtai WHERE similar('query') AND flag = 1
+AND entry >= '2022-01-01'
+```
+
+txtai's query layer automatically detects columns and translates queries into a format that can be understood by the underlying database.
+
+Nested dictionaries/JSON is supported and can be escaped with bracket statements.
+
+```python
+embeddings.index([(0, {"text": "text to index",
+                       "parent": {"child element": "abc"}}, None)])
+```
+
+```sql
+SELECT text FROM txtai WHERE [parent.child element] ='abc'
+```
+
+Note the bracket statement escaping the nested column with spaces in the name.
+
+### Aggregation queries
+
+The goal of txtai's query language is to closely support all functions in the underlying database engine. The main challenge is ensuring dynamic columns are properly escaped into the engines native query function. 
+
+Aggregation query examples.
+
+```sql
+SELECT count(*) FROM txtai WHERE similar('feel good story') AND score >= 0.15
+SELECT max(length(text)) FROM txtai WHERE similar('feel good story')
+AND score >= 0.15
+SELECT count(*), flag FROM txtai GROUP BY flag ORDER BY count(*) DESC
+```
+
+## Binary objects
+
+txtai has support for storing and retrieving binary objects. Binary objects can be retrieved as shown in the example below.
+
+```python
+# Get an image
+request = open("demo.gif")
+
+# Insert record
+embeddings.index([("txtai", {"text": "txtai executes machine-learning workflows.",
+                             "object": request.read()}, None)])
+
+# Query txtai and get associated object
+query = "select object from txtai where similar('machine learning') limit 1"
+result = embeddings.search(query)[0]["object"]
+```
+
+## Custom SQL functions
+
+Custom, user-defined SQL functions extend selection, filtering and ordering clauses with additional logic. For example, the following snippet defines a function that translates text using a translation pipeline.
+
+```python
+# Translation pipeline
+translate = Translation()
+
+# Create embeddings index
+embeddings = Embeddings({"path": "sentence-transformers/nli-mpnet-base-v2",
+                         "content": True,
+                         "functions": [translate]})
+
+# Run a search using a custom SQL function
+embeddings.search("""
+select
+  text,
+  translation(text, 'de', null) 'text (DE)',
+  translation(text, 'es', null) 'text (ES)',
+  translation(text, 'fr', null) 'text (FR)'
+from txtai where similar('feel good story')
+limit 1
+""")
+```
+
+## Query translation
+
+Natural language queries with filters can be converted to txtai-compatible SQL statements with query translation. For example:
+
+```python
+embeddings.search("feel good story since yesterday")
+```
+
+can be converted to a SQL statement with a similar clause and date filter.
+
+```sql
+select id, text, score from txtai where similar('feel good story') and
+entry >= date('now', '-1 day')
+```
+
+This requires setting a [query translation model](../configuration#query). The default query translation model is [t5-small-txtsql](https://huggingface.co/NeuML/t5-small-txtsql) but this can easily be finetuned to handle different use cases.
+
+## Combined index architecture
+
+When content storage is enabled, txtai becomes a dual storage engine. Content is stored in an underlying database (currently supports SQLite) along with an Approximate Nearest Neighbor (ANN) index. These components combine to deliver similarity search alongside traditional structured search.
+
+The ANN index stores ids and vectors for each input element. When a natural language query is run, the query is translated into a vector and a similarity query finds the best matching ids. When a database is added into the mix, an additional step is applied. This step takes those ids and effectively inserts them as part of the underlying database query.
+
+Dynamic columns are supported via the underlying engine. For SQLite, data is stored as JSON and dynamic columns are converted into `json_extract` clauses. This same concept can be expanded to other storage engines like PostgreSQL and could even work with NoSQL stores. 

docs/examples.md

@@ -1,24 +1,35 @@
 # Examples
 
+![examples](images/examples.png#only-light)
+![examples](images/examples-dark.png#only-dark)
+
 The examples directory has a series of notebooks and applications giving an overview of txtai. See the sections below.
 
 ## Semantic Search
 
-Build semantic/similarity/vector search applications.
+Build semantic/similarity/vector/neural search applications.
 
 | Notebook  | Description  |       |
 |:----------|:-------------|------:|
-| [Introducing txtai](https://github.com/neuml/txtai/blob/master/examples/01_Introducing_txtai.ipynb) | Overview of the functionality provided by txtai | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/01_Introducing_txtai.ipynb) |
+| [Introducing txtai](https://github.com/neuml/txtai/blob/master/examples/01_Introducing_txtai.ipynb) [▶️](https://www.youtube.com/watch?v=SIezMnVdmMs) | Overview of the functionality provided by txtai | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/01_Introducing_txtai.ipynb) |
 | [Build an Embeddings index with Hugging Face Datasets](https://github.com/neuml/txtai/blob/master/examples/02_Build_an_Embeddings_index_with_Hugging_Face_Datasets.ipynb) | Index and search Hugging Face Datasets | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/02_Build_an_Embeddings_index_with_Hugging_Face_Datasets.ipynb) |
 | [Build an Embeddings index from a data source](https://github.com/neuml/txtai/blob/master/examples/03_Build_an_Embeddings_index_from_a_data_source.ipynb)  | Index and search a data source with word embeddings | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/03_Build_an_Embeddings_index_from_a_data_source.ipynb) |
 | [Add semantic search to Elasticsearch](https://github.com/neuml/txtai/blob/master/examples/04_Add_semantic_search_to_Elasticsearch.ipynb)  | Add semantic search to existing search systems | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/04_Add_semantic_search_to_Elasticsearch.ipynb) |
-| [API Gallery](https://github.com/neuml/txtai/blob/master/examples/08_API_Gallery.ipynb) | Using txtai in JavaScript, Java, Rust and Go | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/08_API_Gallery.ipynb) |
 | [Similarity search with images](https://github.com/neuml/txtai/blob/master/examples/13_Similarity_search_with_images.ipynb) | Embed images and text into the same space for search | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/13_Similarity_search_with_images.ipynb) |
 | [Distributed embeddings cluster](https://github.com/neuml/txtai/blob/master/examples/15_Distributed_embeddings_cluster.ipynb) | Distribute an embeddings index across multiple data nodes | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/15_Distributed_embeddings_cluster.ipynb) |
+| [What's new in txtai 4.0](https://github.com/neuml/txtai/blob/master/examples/24_Whats_new_in_txtai_4_0.ipynb) | Content storage, SQL, object storage, reindex and compressed indexes | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/24_Whats_new_in_txtai_4_0.ipynb) |
+| [Anatomy of a txtai index](https://github.com/neuml/txtai/blob/master/examples/29_Anatomy_of_a_txtai_index.ipynb) | Deep dive into the file formats behind a txtai embeddings index | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/29_Anatomy_of_a_txtai_index.ipynb) |
+| [Custom Embeddings SQL functions](https://github.com/neuml/txtai/blob/master/examples/30_Embeddings_SQL_custom_functions.ipynb) | Add user-defined functions to Embeddings SQL | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/30_Embeddings_SQL_custom_functions.ipynb) |
+| [Model explainability](https://github.com/neuml/txtai/blob/master/examples/32_Model_explainability.ipynb) | Explainability for semantic search | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/32_Model_explainability.ipynb) |
+| [Query translation](https://github.com/neuml/txtai/blob/master/examples/33_Query_translation.ipynb) | Domain-specific natural language queries with query translation | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/33_Query_translation.ipynb) |
+| [Build a QA database](https://github.com/neuml/txtai/blob/master/examples/34_Build_a_QA_database.ipynb) | Question matching with semantic search | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/34_Build_a_QA_database.ipynb) |
+| [Embeddings components](https://github.com/neuml/txtai/blob/master/examples/37_Embeddings_index_components.ipynb) | Composable search with vector, SQL and scoring components | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/37_Embeddings_index_components.ipynb) |
+| [Semantic Graphs](https://github.com/neuml/txtai/blob/master/examples/38_Introducing_the_Semantic_Graph.ipynb) | Explore topics, data connectivity and run network analysis| [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/38_Introducing_the_Semantic_Graph.ipynb) |
+| [Topic Modeling with BM25](https://github.com/neuml/txtai/blob/master/examples/39_Classic_Topic_Modeling_with_BM25.ipynb) | Topic modeling backed by a BM25 index | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/39_Classic_Topic_Modeling_with_BM25.ipynb) |
 
-## Pipelines and Workflows
+## Pipelines
 
-NLP-backed data transformation pipelines and workflows.
+Transform data with NLP-backed pipelines.
 
 | Notebook  | Description  |       |
 |:----------|:-------------|------:|
@@ -28,10 +39,27 @@ NLP-backed data transformation pipelines and workflows.
 | [Apply labels with zero shot classification](https://github.com/neuml/txtai/blob/master/examples/07_Apply_labels_with_zero_shot_classification.ipynb) | Use zero shot learning for labeling, classification and topic modeling | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/07_Apply_labels_with_zero_shot_classification.ipynb) |
 | [Building abstractive text summaries](https://github.com/neuml/txtai/blob/master/examples/09_Building_abstractive_text_summaries.ipynb) | Run abstractive text summarization | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/09_Building_abstractive_text_summaries.ipynb) |
 | [Extract text from documents](https://github.com/neuml/txtai/blob/master/examples/10_Extract_text_from_documents.ipynb) | Extract text from PDF, Office, HTML and more | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/10_Extract_text_from_documents.ipynb) |
+| [Text to speech generation](https://github.com/neuml/txtai/blob/master/examples/40_Text_to_Speech_Generation.ipynb) | Generate speech from text | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/40_Text_to_Speech_Generation.ipynb) |
 | [Transcribe audio to text](https://github.com/neuml/txtai/blob/master/examples/11_Transcribe_audio_to_text.ipynb) | Convert audio files to text | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/11_Transcribe_audio_to_text.ipynb) |
 | [Translate text between languages](https://github.com/neuml/txtai/blob/master/examples/12_Translate_text_between_languages.ipynb) | Streamline machine translation and language detection | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/12_Translate_text_between_languages.ipynb) |
-| [Run pipeline workflows](https://github.com/neuml/txtai/blob/master/examples/14_Run_pipeline_workflows.ipynb) | Simple yet powerful constructs to efficiently process data | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/14_Run_pipeline_workflows.ipynb) |
-| [Transform tabular data with composable workflows](https://github.com/neuml/txtai/blob/master/examples/22_Transform_tabular_data_with_composable_workflows.ipynb) | Transform, index and search tabular data with workflows | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/22_Transform_tabular_data_with_composable_workflows.ipynb) |
+| [Generate image captions and detect objects](https://github.com/neuml/txtai/blob/master/examples/25_Generate_image_captions_and_detect_objects.ipynb) | Captions and object detection for images | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/25_Generate_image_captions_and_detect_objects.ipynb) |
+| [Near duplicate image detection](https://github.com/neuml/txtai/blob/master/examples/31_Near_duplicate_image_detection.ipynb) | Identify duplicate and near-duplicate images | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/31_Near_duplicate_image_detection.ipynb) |
+| [API Gallery](https://github.com/neuml/txtai/blob/master/examples/08_API_Gallery.ipynb) | Using txtai in JavaScript, Java, Rust and Go | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/08_API_Gallery.ipynb) |
+
+## Workflows
+
+Efficiently process data at scale.
+
+| Notebook  | Description  |       |
+|:----------|:-------------|------:|
+ [Run pipeline workflows](https://github.com/neuml/txtai/blob/master/examples/14_Run_pipeline_workflows.ipynb) [▶️](https://www.youtube.com/watch?v=UBMPDCn1gEU) | Simple yet powerful constructs to efficiently process data | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/14_Run_pipeline_workflows.ipynb) |
+| [Transform tabular data with composable workflows](https://github.com/neuml/txtai/blob/master/examples/22_Transform_tabular_data_with_composable_workflows.ipynb) | Transform, index and search tabular data | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/22_Transform_tabular_data_with_composable_workflows.ipynb) |
+| [Tensor workflows](https://github.com/neuml/txtai/blob/master/examples/23_Tensor_workflows.ipynb) | Performant processing of large tensor arrays | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/23_Tensor_workflows.ipynb) |
+| [Entity extraction workflows](https://github.com/neuml/txtai/blob/master/examples/26_Entity_extraction_workflows.ipynb) | Identify entity/label combinations | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/26_Entity_extraction_workflows.ipynb) |
+| [Workflow Scheduling](https://github.com/neuml/txtai/blob/master/examples/27_Workflow_scheduling.ipynb) | Schedule workflows with cron expressions | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/27_Workflow_scheduling.ipynb) |
+| [Push notifications with workflows](https://github.com/neuml/txtai/blob/master/examples/28_Push_notifications_with_workflows.ipynb) | Generate and push notifications with workflows | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/28_Push_notifications_with_workflows.ipynb) |
+| [Pictures are a worth a thousand words](https://github.com/neuml/txtai/blob/master/examples/35_Pictures_are_worth_a_thousand_words.ipynb) | Generate webpage summary images with DALL-E mini | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/35_Pictures_are_worth_a_thousand_words.ipynb) |
+| [Run txtai with native code](https://github.com/neuml/txtai/blob/master/examples/36_Run_txtai_in_native_code.ipynb) | Execute workflows in native code with the Python C API | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/36_Run_txtai_in_native_code.ipynb) |
 
 ## Model Training
 

docs/further.md

@@ -0,0 +1,12 @@
+# Further reading
+
+![further](images/further.png)
+
+- [Introducing txtai, AI-powered semantic search built on Transformers](https://medium.com/neuml/introducing-txtai-an-ai-powered-search-engine-built-on-transformers-37674be252ec)
+- [Tutorial series on Hashnode](https://neuml.hashnode.dev/series/txtai-tutorial) | [dev.to](https://dev.to/neuml/tutorial-series-on-txtai-ibg)
+- [What's new in txtai 5.0](https://medium.com/neuml/whats-new-in-txtai-5-0-e5c75a13b101) | [4.0](https://medium.com/neuml/whats-new-in-txtai-4-0-bbc3a65c3d1c)
+- [Getting started with semantic search](https://medium.com/neuml/getting-started-with-semantic-search-a9fd9d8a48cf)
+- [Run machine-learning workflows to transform data and build AI-powered semantic search applications with txtai](https://medium.com/neuml/run-machine-learning-workflows-to-transform-data-and-build-ai-powered-text-indices-with-txtai-43d769b566a7)
+- [Semantic search on the cheap](https://medium.com/neuml/semantic-search-on-the-cheap-55940c0fcdab)
+- [Serverless vector search with txtai](https://medium.com/neuml/serverless-vector-search-with-txtai-96f6163ab972)
+- [Insights from the txtai console](https://medium.com/neuml/insights-from-the-txtai-console-d307c28e149e)

docs/images/cloud.excalidraw

@@ -0,0 +1,551 @@
+{
+  "type": "excalidraw",
+  "version": 2,
+  "source": "https://excalidraw.com",
+  "elements": [
+    {
+      "type": "rectangle",
+      "version": 3020,
+      "versionNonce": 574956379,
+      "isDeleted": false,
+      "id": "Ufh5VUA3qmvJowuFyEWz4",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 570.7515859513394,
+      "y": 59.768341009859626,
+      "strokeColor": "#d0d9dd",
+      "backgroundColor": "transparent",
+      "width": 719.330217768941,
+      "height": 100.49478848195537,
+      "seed": 1218477931,
+      "groupIds": [
+        "xlXj5_q2htAY6GkFTIlfY"
+      ],
+      "strokeSharpness": "sharp",
+      "boundElements": [
+        {
+          "type": "arrow",
+          "id": "QPHaiLoteZgu8lrk22yKh"
+        }
+      ],
+      "updated": 1644866249723,
+      "link": null
+    },
+    {
+      "type": "rectangle",
+      "version": 1748,
+      "versionNonce": 1788187605,
+      "isDeleted": false,
+      "id": "NDWbJgM53uVck-LSQzD-0",
+      "fillStyle": "cross-hatch",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 740.6664767795138,
+      "y": 67.41041395399299,
+      "strokeColor": "#ffeb3b",
+      "backgroundColor": "#ffeb3b",
+      "width": 85,
+      "height": 84.17917209201391,
+      "seed": 1934943141,
+      "groupIds": [
+        "xlXj5_q2htAY6GkFTIlfY"
+      ],
+      "strokeSharpness": "sharp",
+      "boundElements": [
+        {
+          "type": "arrow",
+          "id": "xz4WiNbnK-neZmBN7EIi8"
+        },
+        {
+          "type": "arrow",
+          "id": "OH6z46_TWktLzJb31zpRF"
+        },
+        {
+          "type": "arrow",
+          "id": "b07pmUMsHbkm7ES3OaM12"
+        },
+        {
+          "id": "HXM4G7z36lQFPmJTUAy8G",
+          "type": "arrow"
+        },
+        {
+          "id": "BmtKHcjATX2AOVt484fin",
+          "type": "arrow"
+        },
+        {
+          "id": "nR4v3IHv93pXRqc8UUXjz",
+          "type": "arrow"
+        },
+        {
+          "id": "095ylTvsupuqCzyoolf44",
+          "type": "arrow"
+        },
+        {
+          "type": "text",
+          "id": "0VztU7lb7T9I5zoL2LjLt"
+        }
+      ],
+      "updated": 1644866249723,
+      "link": null
+    },
+    {
+      "id": "0VztU7lb7T9I5zoL2LjLt",
+      "type": "text",
+      "x": 745.6664767795138,
+      "y": 98.99999999999994,
+      "width": 75,
+      "height": 21,
+      "angle": 0,
+      "strokeColor": "#000000",
+      "backgroundColor": "#7950f2",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "groupIds": [
+        "xlXj5_q2htAY6GkFTIlfY"
+      ],
+      "strokeSharpness": "sharp",
+      "seed": 604922325,
+      "version": 333,
+      "versionNonce": 1646133243,
+      "isDeleted": false,
+      "boundElements": null,
+      "updated": 1644866249723,
+      "link": null,
+      "text": "Container",
+      "fontSize": 16,
+      "fontFamily": 1,
+      "textAlign": "center",
+      "verticalAlign": "middle",
+      "baseline": 15,
+      "containerId": "NDWbJgM53uVck-LSQzD-0",
+      "originalText": "Container"
+    },
+    {
+      "type": "rectangle",
+      "version": 1824,
+      "versionNonce": 1455313205,
+      "isDeleted": false,
+      "id": "rr3YJsAyrrPqZbZ4qXh5x",
+      "fillStyle": "cross-hatch",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 1175.874140249897,
+      "y": 70.42614920483032,
+      "strokeColor": "#00e676",
+      "backgroundColor": "#00e676",
+      "width": 85,
+      "height": 84.17917209201391,
+      "seed": 1158748789,
+      "groupIds": [
+        "xlXj5_q2htAY6GkFTIlfY"
+      ],
+      "strokeSharpness": "sharp",
+      "boundElements": [
+        {
+          "type": "arrow",
+          "id": "xz4WiNbnK-neZmBN7EIi8"
+        },
+        {
+          "type": "arrow",
+          "id": "OH6z46_TWktLzJb31zpRF"
+        },
+        {
+          "type": "arrow",
+          "id": "b07pmUMsHbkm7ES3OaM12"
+        },
+        {
+          "id": "HXM4G7z36lQFPmJTUAy8G",
+          "type": "arrow"
+        },
+        {
+          "id": "BmtKHcjATX2AOVt484fin",
+          "type": "arrow"
+        },
+        {
+          "id": "nR4v3IHv93pXRqc8UUXjz",
+          "type": "arrow"
+        },
+        {
+          "id": "095ylTvsupuqCzyoolf44",
+          "type": "arrow"
+        },
+        {
+          "id": "9NBdG6TyrBqjuDKvzAZXN",
+          "type": "text"
+        },
+        {
+          "id": "9NBdG6TyrBqjuDKvzAZXN",
+          "type": "text"
+        },
+        {
+          "type": "text",
+          "id": "9NBdG6TyrBqjuDKvzAZXN"
+        }
+      ],
+      "updated": 1644866249723,
+      "link": null
+    },
+    {
+      "type": "text",
+      "version": 404,
+      "versionNonce": 938625179,
+      "isDeleted": false,
+      "id": "9NBdG6TyrBqjuDKvzAZXN",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 1180.874140249897,
+      "y": 102.01573525083728,
+      "strokeColor": "#000000",
+      "backgroundColor": "#7950f2",
+      "width": 75,
+      "height": 21,
+      "seed": 603698011,
+      "groupIds": [
+        "xlXj5_q2htAY6GkFTIlfY"
+      ],
+      "strokeSharpness": "sharp",
+      "boundElements": [],
+      "updated": 1644866249723,
+      "link": null,
+      "fontSize": 16,
+      "fontFamily": 1,
+      "text": "Container",
+      "baseline": 15,
+      "textAlign": "center",
+      "verticalAlign": "middle",
+      "containerId": "rr3YJsAyrrPqZbZ4qXh5x",
+      "originalText": "Container"
+    },
+    {
+      "type": "rectangle",
+      "version": 1814,
+      "versionNonce": 766010005,
+      "isDeleted": false,
+      "id": "ji9UBE590YJ5e8DKUKbPG",
+      "fillStyle": "cross-hatch",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 598.874140249897,
+      "y": 68.42614920483032,
+      "strokeColor": "#ff7043",
+      "backgroundColor": "#ff7043",
+      "width": 85,
+      "height": 84.17917209201391,
+      "seed": 895199477,
+      "groupIds": [
+        "xlXj5_q2htAY6GkFTIlfY"
+      ],
+      "strokeSharpness": "sharp",
+      "boundElements": [
+        {
+          "type": "arrow",
+          "id": "xz4WiNbnK-neZmBN7EIi8"
+        },
+        {
+          "type": "arrow",
+          "id": "OH6z46_TWktLzJb31zpRF"
+        },
+        {
+          "type": "arrow",
+          "id": "b07pmUMsHbkm7ES3OaM12"
+        },
+        {
+          "id": "HXM4G7z36lQFPmJTUAy8G",
+          "type": "arrow"
+        },
+        {
+          "id": "BmtKHcjATX2AOVt484fin",
+          "type": "arrow"
+        },
+        {
+          "id": "nR4v3IHv93pXRqc8UUXjz",
+          "type": "arrow"
+        },
+        {
+          "id": "095ylTvsupuqCzyoolf44",
+          "type": "arrow"
+        },
+        {
+          "id": "2fdy7dRwF7ZVaYPrb1Q7p",
+          "type": "text"
+        },
+        {
+          "id": "2fdy7dRwF7ZVaYPrb1Q7p",
+          "type": "text"
+        },
+        {
+          "type": "text",
+          "id": "2fdy7dRwF7ZVaYPrb1Q7p"
+        }
+      ],
+      "updated": 1644866249723,
+      "link": null
+    },
+    {
+      "type": "text",
+      "version": 394,
+      "versionNonce": 695500091,
+      "isDeleted": false,
+      "id": "2fdy7dRwF7ZVaYPrb1Q7p",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 603.874140249897,
+      "y": 100.01573525083728,
+      "strokeColor": "#000000",
+      "backgroundColor": "#7950f2",
+      "width": 75,
+      "height": 21,
+      "seed": 507989211,
+      "groupIds": [
+        "xlXj5_q2htAY6GkFTIlfY"
+      ],
+      "strokeSharpness": "sharp",
+      "boundElements": [],
+      "updated": 1644866249723,
+      "link": null,
+      "fontSize": 16,
+      "fontFamily": 1,
+      "text": "Container",
+      "baseline": 15,
+      "textAlign": "center",
+      "verticalAlign": "middle",
+      "containerId": "ji9UBE590YJ5e8DKUKbPG",
+      "originalText": "Container"
+    },
+    {
+      "type": "rectangle",
+      "version": 1754,
+      "versionNonce": 873023477,
+      "isDeleted": false,
+      "id": "Kod3erHdgkFtJcsgqHeQQ",
+      "fillStyle": "cross-hatch",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 889.874140249897,
+      "y": 68.42614920483032,
+      "strokeColor": "#03a9f4",
+      "backgroundColor": "#03a9f4",
+      "width": 85,
+      "height": 84.17917209201391,
+      "seed": 1941388379,
+      "groupIds": [
+        "xlXj5_q2htAY6GkFTIlfY"
+      ],
+      "strokeSharpness": "sharp",
+      "boundElements": [
+        {
+          "type": "arrow",
+          "id": "xz4WiNbnK-neZmBN7EIi8"
+        },
+        {
+          "type": "arrow",
+          "id": "OH6z46_TWktLzJb31zpRF"
+        },
+        {
+          "type": "arrow",
+          "id": "b07pmUMsHbkm7ES3OaM12"
+        },
+        {
+          "id": "HXM4G7z36lQFPmJTUAy8G",
+          "type": "arrow"
+        },
+        {
+          "id": "BmtKHcjATX2AOVt484fin",
+          "type": "arrow"
+        },
+        {
+          "id": "nR4v3IHv93pXRqc8UUXjz",
+          "type": "arrow"
+        },
+        {
+          "id": "095ylTvsupuqCzyoolf44",
+          "type": "arrow"
+        },
+        {
+          "id": "vA5QfvPySVZE9WnP6d7dp",
+          "type": "text"
+        },
+        {
+          "type": "text",
+          "id": "vA5QfvPySVZE9WnP6d7dp"
+        }
+      ],
+      "updated": 1644866249723,
+      "link": null
+    },
+    {
+      "type": "text",
+      "version": 337,
+      "versionNonce": 1644633563,
+      "isDeleted": false,
+      "id": "vA5QfvPySVZE9WnP6d7dp",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 894.874140249897,
+      "y": 100.01573525083728,
+      "strokeColor": "#000000",
+      "backgroundColor": "#7950f2",
+      "width": 75,
+      "height": 21,
+      "seed": 867011285,
+      "groupIds": [
+        "xlXj5_q2htAY6GkFTIlfY"
+      ],
+      "strokeSharpness": "sharp",
+      "boundElements": [],
+      "updated": 1644866249723,
+      "link": null,
+      "fontSize": 16,
+      "fontFamily": 1,
+      "text": "Container",
+      "baseline": 15,
+      "textAlign": "center",
+      "verticalAlign": "middle",
+      "containerId": "Kod3erHdgkFtJcsgqHeQQ",
+      "originalText": "Container"
+    },
+    {
+      "type": "rectangle",
+      "version": 1863,
+      "versionNonce": 2078703957,
+      "isDeleted": false,
+      "id": "8p_AIsxC6UbaWESkrwl4g",
+      "fillStyle": "cross-hatch",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 1035.874140249897,
+      "y": 69.42614920483032,
+      "strokeColor": "#5f3dc4",
+      "backgroundColor": "#7950f2",
+      "width": 85,
+      "height": 84.17917209201391,
+      "seed": 1712249051,
+      "groupIds": [
+        "xlXj5_q2htAY6GkFTIlfY"
+      ],
+      "strokeSharpness": "sharp",
+      "boundElements": [
+        {
+          "type": "arrow",
+          "id": "xz4WiNbnK-neZmBN7EIi8"
+        },
+        {
+          "type": "arrow",
+          "id": "OH6z46_TWktLzJb31zpRF"
+        },
+        {
+          "type": "arrow",
+          "id": "b07pmUMsHbkm7ES3OaM12"
+        },
+        {
+          "id": "HXM4G7z36lQFPmJTUAy8G",
+          "type": "arrow"
+        },
+        {
+          "id": "BmtKHcjATX2AOVt484fin",
+          "type": "arrow"
+        },
+        {
+          "id": "nR4v3IHv93pXRqc8UUXjz",
+          "type": "arrow"
+        },
+        {
+          "id": "095ylTvsupuqCzyoolf44",
+          "type": "arrow"
+        },
+        {
+          "id": "AceuLTPujneYjRp_njhIl",
+          "type": "text"
+        },
+        {
+          "id": "AceuLTPujneYjRp_njhIl",
+          "type": "text"
+        },
+        {
+          "id": "AceuLTPujneYjRp_njhIl",
+          "type": "text"
+        },
+        {
+          "type": "text",
+          "id": "AceuLTPujneYjRp_njhIl"
+        }
+      ],
+      "updated": 1644866249723,
+      "link": null
+    },
+    {
+      "type": "text",
+      "version": 443,
+      "versionNonce": 2092739195,
+      "isDeleted": false,
+      "id": "AceuLTPujneYjRp_njhIl",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 1040.874140249897,
+      "y": 101.01573525083728,
+      "strokeColor": "#000000",
+      "backgroundColor": "#7950f2",
+      "width": 75,
+      "height": 21,
+      "seed": 1770780245,
+      "groupIds": [
+        "xlXj5_q2htAY6GkFTIlfY"
+      ],
+      "strokeSharpness": "sharp",
+      "boundElements": [],
+      "updated": 1644866249723,
+      "link": null,
+      "fontSize": 16,
+      "fontFamily": 1,
+      "text": "Container",
+      "baseline": 15,
+      "textAlign": "center",
+      "verticalAlign": "middle",
+      "containerId": "8p_AIsxC6UbaWESkrwl4g",
+      "originalText": "Container"
+    }
+  ],
+  "appState": {
+    "gridSize": null,
+    "viewBackgroundColor": "#ffffff"
+  },
+  "files": {}
+}

docs/images/flows.excalidraw

@@ -0,0 +1,821 @@
+{
+  "type": "excalidraw",
+  "version": 2,
+  "source": "https://excalidraw.com",
+  "elements": [
+    {
+      "type": "text",
+      "version": 536,
+      "versionNonce": 1665188440,
+      "isDeleted": false,
+      "id": "Buic2Lx427wuSIW8P_Rw5",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 781,
+      "y": 142,
+      "strokeColor": "#000000",
+      "backgroundColor": "#228be6",
+      "width": 371,
+      "height": 46,
+      "seed": 373648901,
+      "groupIds": [],
+      "strokeSharpness": "sharp",
+      "boundElements": [],
+      "updated": 1658678188283,
+      "link": null,
+      "locked": false,
+      "fontSize": 36,
+      "fontFamily": 1,
+      "text": "Application Workflows",
+      "baseline": 32,
+      "textAlign": "left",
+      "verticalAlign": "top",
+      "containerId": null,
+      "originalText": "Application Workflows"
+    },
+    {
+      "type": "rectangle",
+      "version": 1629,
+      "versionNonce": 653362472,
+      "isDeleted": false,
+      "id": "qYd3q0Vjks7VOHUC9RR51",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 425,
+      "y": 339.5,
+      "strokeColor": "#00e676",
+      "backgroundColor": "#00e676",
+      "width": 290.0000000000001,
+      "height": 46,
+      "seed": 1441952427,
+      "groupIds": [],
+      "strokeSharpness": "sharp",
+      "boundElements": [
+        {
+          "type": "text",
+          "id": "WPeWn6N4rCHf0jY16N9Ge"
+        }
+      ],
+      "updated": 1658678190330,
+      "link": null,
+      "locked": false
+    },
+    {
+      "type": "arrow",
+      "version": 311,
+      "versionNonce": 924248664,
+      "isDeleted": false,
+      "id": "tgnQzXC9s8RY4oImBOXuB",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 562.5285999651802,
+      "y": 272.4578674923631,
+      "strokeColor": "#000",
+      "backgroundColor": "#228be6",
+      "width": 0.4714000348197942,
+      "height": 72.5421325076369,
+      "seed": 650463755,
+      "groupIds": [],
+      "strokeSharpness": "round",
+      "boundElements": [],
+      "updated": 1658678202111,
+      "link": null,
+      "locked": false,
+      "startBinding": {
+        "elementId": "B435ajoI5vAQBDvzkd8aY",
+        "focus": 0.05963733900881362,
+        "gap": 5.457867492363107
+      },
+      "endBinding": null,
+      "lastCommittedPoint": null,
+      "startArrowhead": null,
+      "endArrowhead": "arrow",
+      "points": [
+        [
+          0,
+          0
+        ],
+        [
+          0.4714000348197942,
+          72.5421325076369
+        ]
+      ]
+    },
+    {
+      "type": "text",
+      "version": 1323,
+      "versionNonce": 515031384,
+      "isDeleted": false,
+      "id": "WPeWn6N4rCHf0jY16N9Ge",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 430,
+      "y": 344.5,
+      "strokeColor": "#000000",
+      "backgroundColor": "#fa5252",
+      "width": 280,
+      "height": 36,
+      "seed": 870516459,
+      "groupIds": [],
+      "strokeSharpness": "sharp",
+      "boundElements": [],
+      "updated": 1658678190330,
+      "link": null,
+      "locked": false,
+      "fontSize": 28,
+      "fontFamily": 1,
+      "text": "Translate",
+      "baseline": 25,
+      "textAlign": "center",
+      "verticalAlign": "middle",
+      "containerId": "qYd3q0Vjks7VOHUC9RR51",
+      "originalText": "Translate"
+    },
+    {
+      "type": "rectangle",
+      "version": 168,
+      "versionNonce": 2070244952,
+      "isDeleted": false,
+      "id": "IxMxusRKX2PpnJT1uY0cC",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 819,
+      "y": 217.5,
+      "strokeColor": "#ffeb3b",
+      "backgroundColor": "#ffeb3b",
+      "width": 290.0000000000001,
+      "height": 49,
+      "seed": 1364021803,
+      "groupIds": [],
+      "strokeSharpness": "sharp",
+      "boundElements": [
+        {
+          "type": "text",
+          "id": "SSXjX_URKvcVC8h-Qgz4j"
+        },
+        {
+          "id": "_YNRYTAnVzdhhKxFtkVyg",
+          "type": "arrow"
+        }
+      ],
+      "updated": 1658678121633,
+      "link": null,
+      "locked": false
+    },
+    {
+      "type": "text",
+      "version": 73,
+      "versionNonce": 1790001960,
+      "isDeleted": false,
+      "id": "SSXjX_URKvcVC8h-Qgz4j",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 824,
+      "y": 224,
+      "strokeColor": "#000000",
+      "backgroundColor": "#82c91e",
+      "width": 280,
+      "height": 36,
+      "seed": 948915563,
+      "groupIds": [],
+      "strokeSharpness": "sharp",
+      "boundElements": [],
+      "updated": 1658678121633,
+      "link": null,
+      "locked": false,
+      "fontSize": 28,
+      "fontFamily": 1,
+      "text": "Extract Text",
+      "baseline": 25,
+      "textAlign": "center",
+      "verticalAlign": "middle",
+      "containerId": "IxMxusRKX2PpnJT1uY0cC",
+      "originalText": "Extract Text"
+    },
+    {
+      "type": "rectangle",
+      "version": 205,
+      "versionNonce": 1180033880,
+      "isDeleted": false,
+      "id": "lATIKISgJPOGnUHleuFRH",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 821,
+      "y": 338,
+      "strokeColor": "#03a9f4",
+      "backgroundColor": "#03a9f4",
+      "width": 290.0000000000001,
+      "height": 46,
+      "seed": 1256311595,
+      "groupIds": [],
+      "strokeSharpness": "sharp",
+      "boundElements": [
+        {
+          "id": "21WiUuyDtpQ9FnJ74REpJ",
+          "type": "text"
+        },
+        {
+          "type": "text",
+          "id": "21WiUuyDtpQ9FnJ74REpJ"
+        },
+        {
+          "id": "Q51e0Hav-kYfjX3b8tt-r",
+          "type": "arrow"
+        }
+      ],
+      "updated": 1658678121633,
+      "link": null,
+      "locked": false
+    },
+    {
+      "type": "text",
+      "version": 84,
+      "versionNonce": 1559141928,
+      "isDeleted": false,
+      "id": "21WiUuyDtpQ9FnJ74REpJ",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 826,
+      "y": 343,
+      "strokeColor": "#000000",
+      "backgroundColor": "#fa5252",
+      "width": 280,
+      "height": 36,
+      "seed": 626266373,
+      "groupIds": [],
+      "strokeSharpness": "sharp",
+      "boundElements": [],
+      "updated": 1658678121633,
+      "link": null,
+      "locked": false,
+      "fontSize": 28,
+      "fontFamily": 1,
+      "text": "Summarize",
+      "baseline": 25,
+      "textAlign": "center",
+      "verticalAlign": "middle",
+      "containerId": "lATIKISgJPOGnUHleuFRH",
+      "originalText": "Summarize"
+    },
+    {
+      "type": "arrow",
+      "version": 150,
+      "versionNonce": 108519256,
+      "isDeleted": false,
+      "id": "_YNRYTAnVzdhhKxFtkVyg",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 962.2340210300499,
+      "y": 270,
+      "strokeColor": "#000",
+      "backgroundColor": "#228be6",
+      "width": 0.2659789699500834,
+      "height": 75,
+      "seed": 101249451,
+      "groupIds": [],
+      "strokeSharpness": "round",
+      "boundElements": [],
+      "updated": 1658678205029,
+      "link": null,
+      "locked": false,
+      "startBinding": {
+        "elementId": "IxMxusRKX2PpnJT1uY0cC",
+        "focus": 0.012856281024868153,
+        "gap": 3.5
+      },
+      "endBinding": null,
+      "lastCommittedPoint": null,
+      "startArrowhead": null,
+      "endArrowhead": "arrow",
+      "points": [
+        [
+          0,
+          0
+        ],
+        [
+          0.2659789699500834,
+          75
+        ]
+      ]
+    },
+    {
+      "type": "rectangle",
+      "version": 213,
+      "versionNonce": 1959303464,
+      "isDeleted": false,
+      "id": "6B3J0wJfi561c9gMsgtXG",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 824,
+      "y": 455,
+      "strokeColor": "#7950f2",
+      "backgroundColor": "#7950f2",
+      "width": 290.0000000000001,
+      "height": 46,
+      "seed": 1164190923,
+      "groupIds": [],
+      "strokeSharpness": "sharp",
+      "boundElements": [
+        {
+          "id": "qJDqmDEZF9Ewh2UrjXq5Z",
+          "type": "text"
+        },
+        {
+          "type": "text",
+          "id": "qJDqmDEZF9Ewh2UrjXq5Z"
+        },
+        {
+          "id": "Q51e0Hav-kYfjX3b8tt-r",
+          "type": "arrow"
+        }
+      ],
+      "updated": 1658678121634,
+      "link": null,
+      "locked": false
+    },
+    {
+      "type": "text",
+      "version": 135,
+      "versionNonce": 210169176,
+      "isDeleted": false,
+      "id": "qJDqmDEZF9Ewh2UrjXq5Z",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 829,
+      "y": 460,
+      "strokeColor": "#000000",
+      "backgroundColor": "#fa5252",
+      "width": 280,
+      "height": 36,
+      "seed": 1307943269,
+      "groupIds": [],
+      "strokeSharpness": "sharp",
+      "boundElements": [],
+      "updated": 1658678121634,
+      "link": null,
+      "locked": false,
+      "fontSize": 28,
+      "fontFamily": 1,
+      "text": "Build Vector Index",
+      "baseline": 25,
+      "textAlign": "center",
+      "verticalAlign": "middle",
+      "containerId": "6B3J0wJfi561c9gMsgtXG",
+      "originalText": "Build Vector Index"
+    },
+    {
+      "type": "arrow",
+      "version": 203,
+      "versionNonce": 133313624,
+      "isDeleted": false,
+      "id": "Q51e0Hav-kYfjX3b8tt-r",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 963.600775377322,
+      "y": 387.5,
+      "strokeColor": "#000",
+      "backgroundColor": "#228be6",
+      "width": 0.29392358842710564,
+      "height": 66.5,
+      "seed": 59173285,
+      "groupIds": [],
+      "strokeSharpness": "round",
+      "boundElements": [],
+      "updated": 1658678208037,
+      "link": null,
+      "locked": false,
+      "startBinding": {
+        "elementId": "lATIKISgJPOGnUHleuFRH",
+        "focus": 0.015253485349599789,
+        "gap": 3.5
+      },
+      "endBinding": {
+        "elementId": "6B3J0wJfi561c9gMsgtXG",
+        "focus": -0.039966641220930875,
+        "gap": 1
+      },
+      "lastCommittedPoint": null,
+      "startArrowhead": null,
+      "endArrowhead": "arrow",
+      "points": [
+        [
+          0,
+          0
+        ],
+        [
+          -0.29392358842710564,
+          66.5
+        ]
+      ]
+    },
+    {
+      "type": "rectangle",
+      "version": 1219,
+      "versionNonce": 1987359528,
+      "isDeleted": false,
+      "id": "5VuUdI_BsJ5pyE1nTqJUI",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 1195,
+      "y": 221,
+      "strokeColor": "#7950f2",
+      "backgroundColor": "#7950f2",
+      "width": 290.0000000000001,
+      "height": 46,
+      "seed": 1044404613,
+      "groupIds": [],
+      "strokeSharpness": "sharp",
+      "boundElements": [
+        {
+          "id": "bJJ9SGsJsvT071qBBH0w5",
+          "type": "text"
+        },
+        {
+          "id": "bJJ9SGsJsvT071qBBH0w5",
+          "type": "text"
+        },
+        {
+          "id": "Q51e0Hav-kYfjX3b8tt-r",
+          "type": "arrow"
+        },
+        {
+          "type": "text",
+          "id": "bJJ9SGsJsvT071qBBH0w5"
+        }
+      ],
+      "updated": 1658678209478,
+      "link": null,
+      "locked": false
+    },
+    {
+      "type": "text",
+      "version": 1403,
+      "versionNonce": 1434611496,
+      "isDeleted": false,
+      "id": "bJJ9SGsJsvT071qBBH0w5",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 1200,
+      "y": 226,
+      "strokeColor": "#000000",
+      "backgroundColor": "#fa5252",
+      "width": 280,
+      "height": 36,
+      "seed": 128953675,
+      "groupIds": [],
+      "strokeSharpness": "sharp",
+      "boundElements": [],
+      "updated": 1658678121634,
+      "link": null,
+      "locked": false,
+      "fontSize": 28,
+      "fontFamily": 1,
+      "text": "Run similarity query",
+      "baseline": 25,
+      "textAlign": "center",
+      "verticalAlign": "middle",
+      "containerId": "5VuUdI_BsJ5pyE1nTqJUI",
+      "originalText": "Run similarity query"
+    },
+    {
+      "type": "rectangle",
+      "version": 332,
+      "versionNonce": 226326360,
+      "isDeleted": false,
+      "id": "pSbgtf1qAB7tWl-pTld7e",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 1197,
+      "y": 343,
+      "strokeColor": "#f44336",
+      "backgroundColor": "#f44336",
+      "width": 290.0000000000001,
+      "height": 46,
+      "seed": 210689733,
+      "groupIds": [],
+      "strokeSharpness": "sharp",
+      "boundElements": [
+        {
+          "id": "pQzKSM3audka1kiQm_Sku",
+          "type": "text"
+        },
+        {
+          "id": "pQzKSM3audka1kiQm_Sku",
+          "type": "text"
+        },
+        {
+          "id": "Q51e0Hav-kYfjX3b8tt-r",
+          "type": "arrow"
+        },
+        {
+          "id": "pQzKSM3audka1kiQm_Sku",
+          "type": "text"
+        },
+        {
+          "type": "text",
+          "id": "pQzKSM3audka1kiQm_Sku"
+        },
+        {
+          "id": "k67YzXNtt1GLh4i8Es5zJ",
+          "type": "arrow"
+        }
+      ],
+      "updated": 1658678121634,
+      "link": null,
+      "locked": false
+    },
+    {
+      "type": "text",
+      "version": 290,
+      "versionNonce": 750406184,
+      "isDeleted": false,
+      "id": "pQzKSM3audka1kiQm_Sku",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 1202,
+      "y": 348,
+      "strokeColor": "#000000",
+      "backgroundColor": "#fa5252",
+      "width": 280,
+      "height": 36,
+      "seed": 1869028363,
+      "groupIds": [],
+      "strokeSharpness": "sharp",
+      "boundElements": [],
+      "updated": 1658678121635,
+      "link": null,
+      "locked": false,
+      "fontSize": 28,
+      "fontFamily": 1,
+      "text": "Send notifications",
+      "baseline": 25,
+      "textAlign": "center",
+      "verticalAlign": "middle",
+      "containerId": "pSbgtf1qAB7tWl-pTld7e",
+      "originalText": "Send notifications"
+    },
+    {
+      "type": "arrow",
+      "version": 308,
+      "versionNonce": 1589736280,
+      "isDeleted": false,
+      "id": "k67YzXNtt1GLh4i8Es5zJ",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 1339.1171359117898,
+      "y": 265.5,
+      "strokeColor": "#000",
+      "backgroundColor": "#228be6",
+      "width": 1.0119492860442278,
+      "height": 76.5,
+      "seed": 1656816747,
+      "groupIds": [],
+      "strokeSharpness": "round",
+      "boundElements": [],
+      "updated": 1658678212274,
+      "link": null,
+      "locked": false,
+      "startBinding": null,
+      "endBinding": {
+        "elementId": "pSbgtf1qAB7tWl-pTld7e",
+        "focus": -0.010690950588675632,
+        "gap": 1
+      },
+      "lastCommittedPoint": null,
+      "startArrowhead": null,
+      "endArrowhead": "arrow",
+      "points": [
+        [
+          0,
+          0
+        ],
+        [
+          1.0119492860442278,
+          76.5
+        ]
+      ]
+    },
+    {
+      "type": "text",
+      "version": 308,
+      "versionNonce": 641647912,
+      "isDeleted": false,
+      "id": "kKPPLMCj8QQIJLbW-B5hm",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 1180,
+      "y": 451,
+      "strokeColor": "#000",
+      "backgroundColor": "#fab005",
+      "width": 296,
+      "height": 52,
+      "seed": 1079990731,
+      "groupIds": [],
+      "strokeSharpness": "sharp",
+      "boundElements": [],
+      "updated": 1658678121635,
+      "link": null,
+      "locked": false,
+      "fontSize": 20,
+      "fontFamily": 1,
+      "text": "- API bindings for JavaScript,\n  Rust, Go and Java",
+      "baseline": 44,
+      "textAlign": "left",
+      "verticalAlign": "top",
+      "containerId": null,
+      "originalText": "- API bindings for JavaScript,\n  Rust, Go and Java"
+    },
+    {
+      "type": "text",
+      "version": 560,
+      "versionNonce": 1977188696,
+      "isDeleted": false,
+      "id": "1AQ3rj-V4weRtPA8a5-z-",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 437.5,
+      "y": 449.5,
+      "strokeColor": "#000",
+      "backgroundColor": "#fab005",
+      "width": 278,
+      "height": 52,
+      "seed": 836512075,
+      "groupIds": [],
+      "strokeSharpness": "sharp",
+      "boundElements": [],
+      "updated": 1658678121635,
+      "link": null,
+      "locked": false,
+      "fontSize": 20,
+      "fontFamily": 1,
+      "text": "- Build with Python or YAML\n- Run local or via API",
+      "baseline": 44,
+      "textAlign": "left",
+      "verticalAlign": "top",
+      "containerId": null,
+      "originalText": "- Build with Python or YAML\n- Run local or via API"
+    },
+    {
+      "type": "rectangle",
+      "version": 292,
+      "versionNonce": 259776552,
+      "isDeleted": false,
+      "id": "B435ajoI5vAQBDvzkd8aY",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 426,
+      "y": 221,
+      "strokeColor": "#03a9f4",
+      "backgroundColor": "#03a9f4",
+      "width": 290.0000000000001,
+      "height": 46,
+      "seed": 942981672,
+      "groupIds": [],
+      "strokeSharpness": "sharp",
+      "boundElements": [
+        {
+          "id": "AoGnxEHn4x-zq2-0C0VrT",
+          "type": "text"
+        },
+        {
+          "id": "AoGnxEHn4x-zq2-0C0VrT",
+          "type": "text"
+        },
+        {
+          "id": "Q51e0Hav-kYfjX3b8tt-r",
+          "type": "arrow"
+        },
+        {
+          "type": "text",
+          "id": "AoGnxEHn4x-zq2-0C0VrT"
+        },
+        {
+          "id": "tgnQzXC9s8RY4oImBOXuB",
+          "type": "arrow"
+        }
+      ],
+      "updated": 1658678121635,
+      "link": null,
+      "locked": false
+    },
+    {
+      "type": "text",
+      "version": 171,
+      "versionNonce": 2093287976,
+      "isDeleted": false,
+      "id": "AoGnxEHn4x-zq2-0C0VrT",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 431,
+      "y": 226,
+      "strokeColor": "#000000",
+      "backgroundColor": "#fa5252",
+      "width": 280,
+      "height": 36,
+      "seed": 604886104,
+      "groupIds": [],
+      "strokeSharpness": "sharp",
+      "boundElements": [],
+      "updated": 1658678188284,
+      "link": null,
+      "locked": false,
+      "fontSize": 28,
+      "fontFamily": 1,
+      "text": "Summarize",
+      "baseline": 25,
+      "textAlign": "center",
+      "verticalAlign": "middle",
+      "containerId": "B435ajoI5vAQBDvzkd8aY",
+      "originalText": "Summarize"
+    }
+  ],
+  "appState": {
+    "gridSize": null,
+    "viewBackgroundColor": "#3030"
+  },
+  "files": {}
+}

docs/images/further.excalidraw

@@ -0,0 +1,162 @@
+{
+  "type": "excalidraw",
+  "version": 2,
+  "source": "https://excalidraw.com",
+  "elements": [
+    {
+      "type": "rectangle",
+      "version": 188,
+      "versionNonce": 534828920,
+      "isDeleted": false,
+      "id": "cnELM0dCp_Q-tLXE_zwgZ",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 965,
+      "y": 277,
+      "strokeColor": "#ffeb3b",
+      "backgroundColor": "#ffeb3b",
+      "width": 182.99999999999997,
+      "height": 153.00000000000003,
+      "seed": 1391794808,
+      "groupIds": [],
+      "strokeSharpness": "sharp",
+      "boundElements": [],
+      "updated": 1641745297574
+    },
+    {
+      "type": "line",
+      "version": 298,
+      "versionNonce": 1982214008,
+      "isDeleted": false,
+      "id": "rXoyG_F7vbnUJMusXWFbX",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 720,
+      "y": 352,
+      "strokeColor": "#03a9f4",
+      "backgroundColor": "#03a9f4",
+      "width": 236,
+      "height": 1,
+      "seed": 1005064312,
+      "groupIds": [],
+      "strokeSharpness": "round",
+      "boundElements": [],
+      "updated": 1641745297579,
+      "startBinding": null,
+      "endBinding": null,
+      "lastCommittedPoint": null,
+      "startArrowhead": null,
+      "endArrowhead": null,
+      "points": [
+        [
+          0,
+          0
+        ],
+        [
+          236,
+          1
+        ]
+      ]
+    },
+    {
+      "type": "ellipse",
+      "version": 420,
+      "versionNonce": 170959880,
+      "isDeleted": false,
+      "id": "lRGCiDG9NSzsH050BbQc7",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 535,
+      "y": 259,
+      "strokeColor": "#03a9f4",
+      "backgroundColor": "#03a9f4",
+      "width": 183,
+      "height": 177,
+      "seed": 2018122872,
+      "groupIds": [],
+      "strokeSharpness": "sharp",
+      "boundElements": [],
+      "updated": 1641745297579
+    },
+    {
+      "type": "line",
+      "version": 196,
+      "versionNonce": 1401598328,
+      "isDeleted": false,
+      "id": "8xcVtQq1gXDL_EteX8-1D",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 1154,
+      "y": 349,
+      "strokeColor": "#00e676",
+      "backgroundColor": "#00e676",
+      "width": 228,
+      "height": 1,
+      "seed": 1003877752,
+      "groupIds": [],
+      "strokeSharpness": "round",
+      "boundElements": [],
+      "updated": 1641745297579,
+      "startBinding": null,
+      "endBinding": null,
+      "lastCommittedPoint": null,
+      "startArrowhead": null,
+      "endArrowhead": null,
+      "points": [
+        [
+          0,
+          0
+        ],
+        [
+          228,
+          -1
+        ]
+      ]
+    },
+    {
+      "type": "ellipse",
+      "version": 332,
+      "versionNonce": 80560648,
+      "isDeleted": false,
+      "id": "KMrkhYiwGuRrQNGu8CERi",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 1385.5,
+      "y": 262.5,
+      "strokeColor": "#00e676",
+      "backgroundColor": "#00e676",
+      "width": 183,
+      "height": 177,
+      "seed": 1536620040,
+      "groupIds": [],
+      "strokeSharpness": "sharp",
+      "boundElements": [],
+      "updated": 1641745297579
+    }
+  ],
+  "appState": {
+    "gridSize": null,
+    "viewBackgroundColor": "#ffffff"
+  },
+  "files": {}
+}

docs/images/indexing.excalidraw

@@ -0,0 +1,333 @@
+{
+  "type": "excalidraw",
+  "version": 2,
+  "source": "https://excalidraw.com",
+  "elements": [
+    {
+      "type": "rectangle",
+      "version": 953,
+      "versionNonce": 1696043931,
+      "isDeleted": false,
+      "id": "UO6MS3wSDu7yg2421__LI",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 933,
+      "y": 267,
+      "strokeColor": "#ffeb3b",
+      "backgroundColor": "#ffeb3b",
+      "width": 214,
+      "height": 49,
+      "seed": 1629565989,
+      "groupIds": [],
+      "strokeSharpness": "sharp",
+      "boundElements": [
+        {
+          "type": "text",
+          "id": "8sp7H8ijWBlh6aMgZ0XTP"
+        },
+        {
+          "id": "Qzp41i_jzQIBlAB_qFKFH",
+          "type": "arrow"
+        },
+        {
+          "id": "SJ0F0Y81z9hir5qQWAJjk",
+          "type": "arrow"
+        }
+      ],
+      "updated": 1669034076965,
+      "link": null,
+      "locked": false
+    },
+    {
+      "type": "rectangle",
+      "version": 1612,
+      "versionNonce": 1217480597,
+      "isDeleted": false,
+      "id": "qYd3q0Vjks7VOHUC9RR51",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 549,
+      "y": 267.5,
+      "strokeColor": "#03a9f4",
+      "backgroundColor": "#03a9f4",
+      "width": 219,
+      "height": 52,
+      "seed": 1441952427,
+      "groupIds": [],
+      "strokeSharpness": "sharp",
+      "boundElements": [
+        {
+          "type": "text",
+          "id": "WPeWn6N4rCHf0jY16N9Ge"
+        },
+        {
+          "id": "Qzp41i_jzQIBlAB_qFKFH",
+          "type": "arrow"
+        }
+      ],
+      "updated": 1669034076965,
+      "link": null,
+      "locked": false
+    },
+    {
+      "type": "text",
+      "version": 1298,
+      "versionNonce": 968386619,
+      "isDeleted": false,
+      "id": "WPeWn6N4rCHf0jY16N9Ge",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 554,
+      "y": 275.5,
+      "strokeColor": "#000",
+      "backgroundColor": "#fa5252",
+      "width": 209,
+      "height": 36,
+      "seed": 870516459,
+      "groupIds": [],
+      "strokeSharpness": "sharp",
+      "boundElements": [],
+      "updated": 1669034076965,
+      "link": null,
+      "locked": false,
+      "fontSize": 28,
+      "fontFamily": 1,
+      "text": "Vectorize",
+      "baseline": 25,
+      "textAlign": "center",
+      "verticalAlign": "middle",
+      "containerId": "qYd3q0Vjks7VOHUC9RR51",
+      "originalText": "Vectorize"
+    },
+    {
+      "type": "rectangle",
+      "version": 1199,
+      "versionNonce": 713919733,
+      "isDeleted": false,
+      "id": "5VuUdI_BsJ5pyE1nTqJUI",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 1332,
+      "y": 268,
+      "strokeColor": "#00e676",
+      "backgroundColor": "#00e676",
+      "width": 218,
+      "height": 49,
+      "seed": 1044404613,
+      "groupIds": [],
+      "strokeSharpness": "sharp",
+      "boundElements": [
+        {
+          "id": "bJJ9SGsJsvT071qBBH0w5",
+          "type": "text"
+        },
+        {
+          "id": "bJJ9SGsJsvT071qBBH0w5",
+          "type": "text"
+        },
+        {
+          "id": "Q51e0Hav-kYfjX3b8tt-r",
+          "type": "arrow"
+        },
+        {
+          "type": "text",
+          "id": "bJJ9SGsJsvT071qBBH0w5"
+        },
+        {
+          "id": "SJ0F0Y81z9hir5qQWAJjk",
+          "type": "arrow"
+        }
+      ],
+      "updated": 1669034076965,
+      "link": null,
+      "locked": false
+    },
+    {
+      "type": "text",
+      "version": 1382,
+      "versionNonce": 949332187,
+      "isDeleted": false,
+      "id": "bJJ9SGsJsvT071qBBH0w5",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 1337,
+      "y": 274.5,
+      "strokeColor": "#000",
+      "backgroundColor": "#fa5252",
+      "width": 208,
+      "height": 36,
+      "seed": 128953675,
+      "groupIds": [],
+      "strokeSharpness": "sharp",
+      "boundElements": [],
+      "updated": 1669034076965,
+      "link": null,
+      "locked": false,
+      "fontSize": 28,
+      "fontFamily": 1,
+      "text": "Search",
+      "baseline": 25,
+      "textAlign": "center",
+      "verticalAlign": "middle",
+      "containerId": "5VuUdI_BsJ5pyE1nTqJUI",
+      "originalText": "Search"
+    },
+    {
+      "type": "text",
+      "version": 943,
+      "versionNonce": 60434005,
+      "isDeleted": false,
+      "id": "8sp7H8ijWBlh6aMgZ0XTP",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 938,
+      "y": 273.5,
+      "strokeColor": "#000",
+      "backgroundColor": "transparent",
+      "width": 204,
+      "height": 36,
+      "seed": 1854823263,
+      "groupIds": [],
+      "strokeSharpness": "sharp",
+      "boundElements": [],
+      "updated": 1669034076965,
+      "link": null,
+      "locked": false,
+      "fontSize": 28,
+      "fontFamily": 1,
+      "text": "Index",
+      "baseline": 25,
+      "textAlign": "center",
+      "verticalAlign": "middle",
+      "containerId": "UO6MS3wSDu7yg2421__LI",
+      "originalText": "Index"
+    },
+    {
+      "type": "arrow",
+      "version": 1814,
+      "versionNonce": 797594901,
+      "isDeleted": false,
+      "id": "Qzp41i_jzQIBlAB_qFKFH",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 770.5,
+      "y": 289.8470411964629,
+      "strokeColor": "#000",
+      "backgroundColor": "#f44336",
+      "width": 158.1310513485223,
+      "height": 0.5692601572380909,
+      "seed": 660786897,
+      "groupIds": [],
+      "strokeSharpness": "round",
+      "boundElements": [],
+      "updated": 1669034076965,
+      "link": null,
+      "locked": false,
+      "startBinding": {
+        "elementId": "qYd3q0Vjks7VOHUC9RR51",
+        "focus": -0.15367587596362536,
+        "gap": 2.5
+      },
+      "endBinding": {
+        "elementId": "UO6MS3wSDu7yg2421__LI",
+        "focus": 0.027437144815141,
+        "gap": 4.3689486514776945
+      },
+      "lastCommittedPoint": null,
+      "startArrowhead": null,
+      "endArrowhead": "arrow",
+      "points": [
+        [
+          0,
+          0
+        ],
+        [
+          158.1310513485223,
+          0.5692601572380909
+        ]
+      ]
+    },
+    {
+      "type": "arrow",
+      "version": 1963,
+      "versionNonce": 616624827,
+      "isDeleted": false,
+      "id": "SJ0F0Y81z9hir5qQWAJjk",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 1149.5,
+      "y": 292.6790761701911,
+      "strokeColor": "#000",
+      "backgroundColor": "#f44336",
+      "width": 181.5,
+      "height": 1.5898915058209013,
+      "seed": 899541905,
+      "groupIds": [],
+      "strokeSharpness": "round",
+      "boundElements": [],
+      "updated": 1669034076965,
+      "link": null,
+      "locked": false,
+      "startBinding": {
+        "elementId": "UO6MS3wSDu7yg2421__LI",
+        "focus": 0.08406032225724415,
+        "gap": 2.5
+      },
+      "endBinding": {
+        "elementId": "5VuUdI_BsJ5pyE1nTqJUI",
+        "focus": 0.09327847520504394,
+        "gap": 1
+      },
+      "lastCommittedPoint": null,
+      "startArrowhead": null,
+      "endArrowhead": "arrow",
+      "points": [
+        [
+          0,
+          0
+        ],
+        [
+          181.5,
+          -1.5898915058209013
+        ]
+      ]
+    }
+  ],
+  "appState": {
+    "gridSize": null,
+    "viewBackgroundColor": "#ffffff"
+  },
+  "files": {}
+}

docs/images/install.excalidraw

@@ -0,0 +1,96 @@
+{
+  "type": "excalidraw",
+  "version": 2,
+  "source": "https://excalidraw.com",
+  "elements": [
+    {
+      "type": "rectangle",
+      "version": 1431,
+      "versionNonce": 726709768,
+      "isDeleted": false,
+      "id": "LgVOdCQnXRH2GYHIGLwOx",
+      "fillStyle": "cross-hatch",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 1.5707963267948957,
+      "x": 764,
+      "y": 123.99999999999991,
+      "strokeColor": "#00e676",
+      "backgroundColor": "#00e676",
+      "width": 20.000000000000092,
+      "height": 199.99999999999991,
+      "seed": 782310408,
+      "groupIds": [
+        "s5mCNZzJxsETmWhZoD3qR"
+      ],
+      "strokeSharpness": "sharp",
+      "boundElements": [],
+      "updated": 1641754344717
+    },
+    {
+      "type": "rectangle",
+      "version": 1531,
+      "versionNonce": 2009304696,
+      "isDeleted": false,
+      "id": "GG_mFtlk6nEiYCQ8vhzjZ",
+      "fillStyle": "cross-hatch",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 1.5707963267948957,
+      "x": 953,
+      "y": 136.9999999999999,
+      "strokeColor": "#ced4da",
+      "backgroundColor": "#ced4da",
+      "width": 17.99999999999998,
+      "height": 173.99999999999994,
+      "seed": 810177544,
+      "groupIds": [
+        "Rezl6DUXkEY8GT9kuDOQX"
+      ],
+      "strokeSharpness": "sharp",
+      "boundElements": [],
+      "updated": 1641754344717
+    },
+    {
+      "id": "NLfZWFLgvJj3ccuF5yxD8",
+      "type": "text",
+      "x": 793,
+      "y": 216.86805555555557,
+      "width": 80,
+      "height": 16,
+      "angle": 0,
+      "strokeColor": "#343a40",
+      "backgroundColor": "transparent",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "groupIds": [],
+      "strokeSharpness": "sharp",
+      "seed": 297813880,
+      "version": 288,
+      "versionNonce": 145457528,
+      "isDeleted": false,
+      "boundElements": null,
+      "updated": 1641754464371,
+      "text": "Installing.......",
+      "fontSize": 12.242424242424246,
+      "fontFamily": 1,
+      "textAlign": "left",
+      "verticalAlign": "top",
+      "baseline": 11,
+      "containerId": null,
+      "originalText": "Installing......."
+    }
+  ],
+  "appState": {
+    "gridSize": null,
+    "viewBackgroundColor": "#ffffff"
+  },
+  "files": {}
+}

docs/images/query.excalidraw

@@ -0,0 +1,243 @@
+{
+  "type": "excalidraw",
+  "version": 2,
+  "source": "https://excalidraw.com",
+  "elements": [
+    {
+      "type": "rectangle",
+      "version": 3038,
+      "versionNonce": 1386987419,
+      "isDeleted": false,
+      "id": "pjs0X-Y0kuuXB-vpf_dvP",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 934.5,
+      "y": -240.29250000000002,
+      "strokeColor": "#ffeb3b",
+      "backgroundColor": "#ffeb3b",
+      "width": 312,
+      "height": 146,
+      "seed": 1534167490,
+      "groupIds": [],
+      "strokeSharpness": "sharp",
+      "boundElements": [
+        {
+          "id": "of5l7HpIH0kPvZ95rOMwM",
+          "type": "text"
+        },
+        {
+          "id": "Qzp41i_jzQIBlAB_qFKFH",
+          "type": "arrow"
+        },
+        {
+          "id": "of5l7HpIH0kPvZ95rOMwM",
+          "type": "text"
+        },
+        {
+          "type": "text",
+          "id": "of5l7HpIH0kPvZ95rOMwM"
+        }
+      ],
+      "updated": 1669033845173,
+      "link": null,
+      "locked": false
+    },
+    {
+      "type": "text",
+      "version": 2554,
+      "versionNonce": 1319492501,
+      "isDeleted": false,
+      "id": "of5l7HpIH0kPvZ95rOMwM",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 939.5,
+      "y": -235.29250000000002,
+      "strokeColor": "#000",
+      "backgroundColor": "#fa5252",
+      "width": 260,
+      "height": 130,
+      "seed": 1972792158,
+      "groupIds": [],
+      "strokeSharpness": "sharp",
+      "boundElements": [],
+      "updated": 1669033845173,
+      "link": null,
+      "locked": false,
+      "fontSize": 20,
+      "fontFamily": 1,
+      "text": "SELECT\n  count(*), flag\nFROM txtai\nGROUP BY flag\nORDER BY count(*) DESC",
+      "baseline": 122,
+      "textAlign": "left",
+      "verticalAlign": "top",
+      "containerId": "pjs0X-Y0kuuXB-vpf_dvP",
+      "originalText": "SELECT\n  count(*), flag\nFROM txtai\nGROUP BY flag\nORDER BY count(*) DESC"
+    },
+    {
+      "type": "rectangle",
+      "version": 3354,
+      "versionNonce": 1004999739,
+      "isDeleted": false,
+      "id": "xMKXZw8aL27lv21eVGy_C",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 581,
+      "y": -238.29250000000002,
+      "strokeColor": "#03a9f4",
+      "backgroundColor": "#03a9f4",
+      "width": 302,
+      "height": 140,
+      "seed": 1704428866,
+      "groupIds": [],
+      "strokeSharpness": "sharp",
+      "boundElements": [
+        {
+          "id": "tlu8eCXVp5ApnEG74CF38",
+          "type": "text"
+        },
+        {
+          "id": "Qzp41i_jzQIBlAB_qFKFH",
+          "type": "arrow"
+        },
+        {
+          "type": "text",
+          "id": "tlu8eCXVp5ApnEG74CF38"
+        }
+      ],
+      "updated": 1669033845173,
+      "link": null,
+      "locked": false
+    },
+    {
+      "type": "text",
+      "version": 2750,
+      "versionNonce": 686792949,
+      "isDeleted": false,
+      "id": "tlu8eCXVp5ApnEG74CF38",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 586,
+      "y": -233.29250000000002,
+      "strokeColor": "#000",
+      "backgroundColor": "#fa5252",
+      "width": 290,
+      "height": 130,
+      "seed": 92268510,
+      "groupIds": [],
+      "strokeSharpness": "sharp",
+      "boundElements": [],
+      "updated": 1669033845173,
+      "link": null,
+      "locked": false,
+      "fontSize": 20,
+      "fontFamily": 1,
+      "text": "SELECT\n  text, flag, entry\nFROM txtai\nWHERE\n  similar('text') AND flag = 1",
+      "baseline": 122,
+      "textAlign": "left",
+      "verticalAlign": "top",
+      "containerId": "xMKXZw8aL27lv21eVGy_C",
+      "originalText": "SELECT\n  text, flag, entry\nFROM txtai\nWHERE\n  similar('text') AND flag = 1"
+    },
+    {
+      "type": "rectangle",
+      "version": 3169,
+      "versionNonce": 1477367003,
+      "isDeleted": false,
+      "id": "Tsn_ukSxtC7CIwEMUMHxI",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 1301.5,
+      "y": -236.79250000000002,
+      "strokeColor": "#00e676",
+      "backgroundColor": "#00e676",
+      "width": 304,
+      "height": 140,
+      "seed": 919682718,
+      "groupIds": [],
+      "strokeSharpness": "sharp",
+      "boundElements": [
+        {
+          "id": "t3OC1akz9Oe2w6N8OCavg",
+          "type": "text"
+        },
+        {
+          "id": "Qzp41i_jzQIBlAB_qFKFH",
+          "type": "arrow"
+        },
+        {
+          "id": "t3OC1akz9Oe2w6N8OCavg",
+          "type": "text"
+        },
+        {
+          "id": "t3OC1akz9Oe2w6N8OCavg",
+          "type": "text"
+        },
+        {
+          "type": "text",
+          "id": "t3OC1akz9Oe2w6N8OCavg"
+        }
+      ],
+      "updated": 1669033845173,
+      "link": null,
+      "locked": false
+    },
+    {
+      "type": "text",
+      "version": 2769,
+      "versionNonce": 345619029,
+      "isDeleted": false,
+      "id": "t3OC1akz9Oe2w6N8OCavg",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 1306.5,
+      "y": -231.79250000000002,
+      "strokeColor": "#000",
+      "backgroundColor": "#fa5252",
+      "width": 259,
+      "height": 130,
+      "seed": 1222509122,
+      "groupIds": [],
+      "strokeSharpness": "sharp",
+      "boundElements": [],
+      "updated": 1669033845173,
+      "link": null,
+      "locked": false,
+      "fontSize": 20,
+      "fontFamily": 1,
+      "text": "SELECT\n  translation(text, 'fr')\nFROM txtai\nWHERE\n  similar('feel good story')",
+      "baseline": 122,
+      "textAlign": "left",
+      "verticalAlign": "top",
+      "containerId": "Tsn_ukSxtC7CIwEMUMHxI",
+      "originalText": "SELECT\n  translation(text, 'fr')\nFROM txtai\nWHERE\n  similar('feel good story')"
+    }
+  ],
+  "appState": {
+    "gridSize": null,
+    "viewBackgroundColor": "#ffffff"
+  },
+  "files": {}
+}

docs/images/search.excalidraw

@@ -0,0 +1,555 @@
+{
+  "type": "excalidraw",
+  "version": 2,
+  "source": "https://excalidraw.com",
+  "elements": [
+    {
+      "type": "text",
+      "version": 354,
+      "versionNonce": 1260634512,
+      "isDeleted": false,
+      "id": "Buic2Lx427wuSIW8P_Rw5",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 829,
+      "y": 184,
+      "strokeColor": "#000000",
+      "backgroundColor": "#228be6",
+      "width": 452,
+      "height": 46,
+      "seed": 373648901,
+      "groupIds": [],
+      "strokeSharpness": "sharp",
+      "boundElements": [],
+      "updated": 1658676789638,
+      "link": null,
+      "locked": false,
+      "fontSize": 36,
+      "fontFamily": 1,
+      "text": "What is semantic search?",
+      "baseline": 32,
+      "textAlign": "left",
+      "verticalAlign": "top",
+      "containerId": null,
+      "originalText": "What is semantic search?"
+    },
+    {
+      "type": "rectangle",
+      "version": 786,
+      "versionNonce": 1756557200,
+      "isDeleted": false,
+      "id": "U2NgEIEiFpAlwmv5Xnyzr",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 50,
+      "angle": 0,
+      "x": 555.5,
+      "y": 425.30499999999995,
+      "strokeColor": "#7950f2",
+      "backgroundColor": "#7950f2",
+      "width": 995.3286713286714,
+      "height": 339.0000000000001,
+      "seed": 1946478225,
+      "groupIds": [
+        "C_65R9XVeMQED2nMfIW2D"
+      ],
+      "strokeSharpness": "sharp",
+      "boundElements": [],
+      "updated": 1658676804492,
+      "link": "",
+      "locked": false
+    },
+    {
+      "type": "text",
+      "version": 569,
+      "versionNonce": 1675727216,
+      "isDeleted": false,
+      "id": "fbSO8bnZmAdvIXZxBAtHY",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 570.7237762237763,
+      "y": 439.50080419580416,
+      "strokeColor": "#000",
+      "backgroundColor": "#03a9f4",
+      "width": 967,
+      "height": 312,
+      "seed": 1586673137,
+      "groupIds": [
+        "C_65R9XVeMQED2nMfIW2D"
+      ],
+      "strokeSharpness": "sharp",
+      "boundElements": [],
+      "updated": 1658676804492,
+      "link": null,
+      "locked": false,
+      "fontSize": 20.27972027972028,
+      "fontFamily": 1,
+      "text": "Query                  Best Match\n----------------------------------------------------\nfeel good story        Maine man wins $1M from $25 lottery ticket\nclimate change         Canada's last fully intact ice shelf has suddenly collapsed, forming a \n                       Manhattan-sized iceberg\npublic health story     US tops 5 million confirmed virus cases\nwar                    Beijing mobilises invasion craft along coast as Taiwan tensions escalate\nwildlife                 The National Park Service warns against sacrificing slower friends in a\n                        bear attack\nasia                   Beijing mobilises invasion craft along coast as Taiwan tensions escalate\nlucky                   Maine man wins $1M from $25 lottery ticket\ndishonest junk          Make huge profits without work, earn up to $100,000 a day",
+      "baseline": 304,
+      "textAlign": "left",
+      "verticalAlign": "top",
+      "containerId": null,
+      "originalText": "Query                  Best Match\n----------------------------------------------------\nfeel good story        Maine man wins $1M from $25 lottery ticket\nclimate change         Canada's last fully intact ice shelf has suddenly collapsed, forming a \n                       Manhattan-sized iceberg\npublic health story     US tops 5 million confirmed virus cases\nwar                    Beijing mobilises invasion craft along coast as Taiwan tensions escalate\nwildlife                 The National Park Service warns against sacrificing slower friends in a\n                        bear attack\nasia                   Beijing mobilises invasion craft along coast as Taiwan tensions escalate\nlucky                   Maine man wins $1M from $25 lottery ticket\ndishonest junk          Make huge profits without work, earn up to $100,000 a day"
+    },
+    {
+      "type": "rectangle",
+      "version": 952,
+      "versionNonce": 581259632,
+      "isDeleted": false,
+      "id": "UO6MS3wSDu7yg2421__LI",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 933,
+      "y": 267,
+      "strokeColor": "#ffeb3b",
+      "backgroundColor": "#ffeb3b",
+      "width": 214,
+      "height": 49,
+      "seed": 1629565989,
+      "groupIds": [
+        "3sURMvhuRfR0M-Q3VRPbg"
+      ],
+      "strokeSharpness": "sharp",
+      "boundElements": [
+        {
+          "type": "text",
+          "id": "8sp7H8ijWBlh6aMgZ0XTP"
+        },
+        {
+          "id": "Qzp41i_jzQIBlAB_qFKFH",
+          "type": "arrow"
+        },
+        {
+          "id": "SJ0F0Y81z9hir5qQWAJjk",
+          "type": "arrow"
+        }
+      ],
+      "updated": 1658676831306,
+      "link": null,
+      "locked": false
+    },
+    {
+      "type": "rectangle",
+      "version": 1611,
+      "versionNonce": 1109835152,
+      "isDeleted": false,
+      "id": "qYd3q0Vjks7VOHUC9RR51",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 549,
+      "y": 267.5,
+      "strokeColor": "#03a9f4",
+      "backgroundColor": "#03a9f4",
+      "width": 219,
+      "height": 52,
+      "seed": 1441952427,
+      "groupIds": [
+        "3sURMvhuRfR0M-Q3VRPbg"
+      ],
+      "strokeSharpness": "sharp",
+      "boundElements": [
+        {
+          "type": "text",
+          "id": "WPeWn6N4rCHf0jY16N9Ge"
+        },
+        {
+          "id": "Qzp41i_jzQIBlAB_qFKFH",
+          "type": "arrow"
+        }
+      ],
+      "updated": 1658676831306,
+      "link": null,
+      "locked": false
+    },
+    {
+      "type": "text",
+      "version": 1297,
+      "versionNonce": 1725539184,
+      "isDeleted": false,
+      "id": "WPeWn6N4rCHf0jY16N9Ge",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 554,
+      "y": 275.5,
+      "strokeColor": "#000",
+      "backgroundColor": "#fa5252",
+      "width": 209,
+      "height": 36,
+      "seed": 870516459,
+      "groupIds": [
+        "3sURMvhuRfR0M-Q3VRPbg"
+      ],
+      "strokeSharpness": "sharp",
+      "boundElements": [],
+      "updated": 1658676831306,
+      "link": null,
+      "locked": false,
+      "fontSize": 28,
+      "fontFamily": 1,
+      "text": "Vectorize",
+      "baseline": 25,
+      "textAlign": "center",
+      "verticalAlign": "middle",
+      "containerId": "qYd3q0Vjks7VOHUC9RR51",
+      "originalText": "Vectorize"
+    },
+    {
+      "type": "rectangle",
+      "version": 1198,
+      "versionNonce": 1201179536,
+      "isDeleted": false,
+      "id": "5VuUdI_BsJ5pyE1nTqJUI",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 1332,
+      "y": 268,
+      "strokeColor": "#00e676",
+      "backgroundColor": "#00e676",
+      "width": 218,
+      "height": 49,
+      "seed": 1044404613,
+      "groupIds": [
+        "3sURMvhuRfR0M-Q3VRPbg"
+      ],
+      "strokeSharpness": "sharp",
+      "boundElements": [
+        {
+          "id": "bJJ9SGsJsvT071qBBH0w5",
+          "type": "text"
+        },
+        {
+          "id": "bJJ9SGsJsvT071qBBH0w5",
+          "type": "text"
+        },
+        {
+          "id": "Q51e0Hav-kYfjX3b8tt-r",
+          "type": "arrow"
+        },
+        {
+          "type": "text",
+          "id": "bJJ9SGsJsvT071qBBH0w5"
+        },
+        {
+          "id": "SJ0F0Y81z9hir5qQWAJjk",
+          "type": "arrow"
+        }
+      ],
+      "updated": 1658676831306,
+      "link": null,
+      "locked": false
+    },
+    {
+      "type": "text",
+      "version": 1381,
+      "versionNonce": 78886256,
+      "isDeleted": false,
+      "id": "bJJ9SGsJsvT071qBBH0w5",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 1337,
+      "y": 274.5,
+      "strokeColor": "#000",
+      "backgroundColor": "#fa5252",
+      "width": 208,
+      "height": 36,
+      "seed": 128953675,
+      "groupIds": [
+        "3sURMvhuRfR0M-Q3VRPbg"
+      ],
+      "strokeSharpness": "sharp",
+      "boundElements": [],
+      "updated": 1658676831306,
+      "link": null,
+      "locked": false,
+      "fontSize": 28,
+      "fontFamily": 1,
+      "text": "Search",
+      "baseline": 25,
+      "textAlign": "center",
+      "verticalAlign": "middle",
+      "containerId": "5VuUdI_BsJ5pyE1nTqJUI",
+      "originalText": "Search"
+    },
+    {
+      "type": "text",
+      "version": 942,
+      "versionNonce": 429955472,
+      "isDeleted": false,
+      "id": "8sp7H8ijWBlh6aMgZ0XTP",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 938,
+      "y": 273.5,
+      "strokeColor": "#000",
+      "backgroundColor": "transparent",
+      "width": 204,
+      "height": 36,
+      "seed": 1854823263,
+      "groupIds": [
+        "3sURMvhuRfR0M-Q3VRPbg"
+      ],
+      "strokeSharpness": "sharp",
+      "boundElements": [],
+      "updated": 1658676831306,
+      "link": null,
+      "locked": false,
+      "fontSize": 28,
+      "fontFamily": 1,
+      "text": "Index",
+      "baseline": 25,
+      "textAlign": "center",
+      "verticalAlign": "middle",
+      "containerId": "UO6MS3wSDu7yg2421__LI",
+      "originalText": "Index"
+    },
+    {
+      "type": "text",
+      "version": 382,
+      "versionNonce": 1098614640,
+      "isDeleted": false,
+      "id": "jWJpSXHkTCzRTCA4tbAgv",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 548.5,
+      "y": 347.30499999999995,
+      "strokeColor": "#000",
+      "backgroundColor": "#03a9f4",
+      "width": 296,
+      "height": 42,
+      "seed": 1241563487,
+      "groupIds": [
+        "3sURMvhuRfR0M-Q3VRPbg"
+      ],
+      "strokeSharpness": "sharp",
+      "boundElements": [],
+      "updated": 1658676831307,
+      "link": null,
+      "locked": false,
+      "fontSize": 16,
+      "fontFamily": 1,
+      "text": "- Transform input into numbers\n- Similar concepts have similar values",
+      "baseline": 36,
+      "textAlign": "left",
+      "verticalAlign": "top",
+      "containerId": null,
+      "originalText": "- Transform input into numbers\n- Similar concepts have similar values"
+    },
+    {
+      "type": "text",
+      "version": 381,
+      "versionNonce": 1266083728,
+      "isDeleted": false,
+      "id": "qEnmXs0P_MQE8r4c4OWGh",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 931.5,
+      "y": 346.2074999999999,
+      "strokeColor": "#000",
+      "backgroundColor": "#f44336",
+      "width": 230,
+      "height": 42,
+      "seed": 1038536465,
+      "groupIds": [
+        "3sURMvhuRfR0M-Q3VRPbg"
+      ],
+      "strokeSharpness": "sharp",
+      "boundElements": [],
+      "updated": 1658676831307,
+      "link": null,
+      "locked": false,
+      "fontSize": 16,
+      "fontFamily": 1,
+      "text": "- Save vectors\n- Store content with vectors",
+      "baseline": 36,
+      "textAlign": "left",
+      "verticalAlign": "top",
+      "containerId": null,
+      "originalText": "- Save vectors\n- Store content with vectors"
+    },
+    {
+      "type": "text",
+      "version": 491,
+      "versionNonce": 2037979504,
+      "isDeleted": false,
+      "id": "1q8bzjK8lnKUZj8_A9v7D",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 1243,
+      "y": 349.2074999999999,
+      "strokeColor": "#000",
+      "backgroundColor": "#f44336",
+      "width": 333,
+      "height": 42,
+      "seed": 304472945,
+      "groupIds": [
+        "3sURMvhuRfR0M-Q3VRPbg"
+      ],
+      "strokeSharpness": "sharp",
+      "boundElements": [],
+      "updated": 1658676831307,
+      "link": null,
+      "locked": false,
+      "fontSize": 16,
+      "fontFamily": 1,
+      "text": "- Find similar vectors with cosine similarity\n- Add rule-based filters using content",
+      "baseline": 36,
+      "textAlign": "left",
+      "verticalAlign": "top",
+      "containerId": null,
+      "originalText": "- Find similar vectors with cosine similarity\n- Add rule-based filters using content"
+    },
+    {
+      "type": "arrow",
+      "version": 1813,
+      "versionNonce": 1230206352,
+      "isDeleted": false,
+      "id": "Qzp41i_jzQIBlAB_qFKFH",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 770.5,
+      "y": 289.8470411964629,
+      "strokeColor": "#000",
+      "backgroundColor": "#f44336",
+      "width": 158.1310513485223,
+      "height": 0.5692601572380909,
+      "seed": 660786897,
+      "groupIds": [
+        "3sURMvhuRfR0M-Q3VRPbg"
+      ],
+      "strokeSharpness": "round",
+      "boundElements": [],
+      "updated": 1658676831307,
+      "link": null,
+      "locked": false,
+      "startBinding": {
+        "elementId": "qYd3q0Vjks7VOHUC9RR51",
+        "focus": -0.15367587596362536,
+        "gap": 2.5
+      },
+      "endBinding": {
+        "elementId": "UO6MS3wSDu7yg2421__LI",
+        "focus": 0.027437144815141,
+        "gap": 4.3689486514776945
+      },
+      "lastCommittedPoint": null,
+      "startArrowhead": null,
+      "endArrowhead": "arrow",
+      "points": [
+        [
+          0,
+          0
+        ],
+        [
+          158.1310513485223,
+          0.5692601572380909
+        ]
+      ]
+    },
+    {
+      "type": "arrow",
+      "version": 1962,
+      "versionNonce": 1072982896,
+      "isDeleted": false,
+      "id": "SJ0F0Y81z9hir5qQWAJjk",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 1149.5,
+      "y": 292.6790761701911,
+      "strokeColor": "#000",
+      "backgroundColor": "#f44336",
+      "width": 181.5,
+      "height": 1.5898915058209013,
+      "seed": 899541905,
+      "groupIds": [
+        "3sURMvhuRfR0M-Q3VRPbg"
+      ],
+      "strokeSharpness": "round",
+      "boundElements": [],
+      "updated": 1658676831307,
+      "link": null,
+      "locked": false,
+      "startBinding": {
+        "elementId": "UO6MS3wSDu7yg2421__LI",
+        "focus": 0.08406032225724415,
+        "gap": 2.5
+      },
+      "endBinding": {
+        "elementId": "5VuUdI_BsJ5pyE1nTqJUI",
+        "focus": 0.09327847520504394,
+        "gap": 1
+      },
+      "lastCommittedPoint": null,
+      "startArrowhead": null,
+      "endArrowhead": "arrow",
+      "points": [
+        [
+          0,
+          0
+        ],
+        [
+          181.5,
+          -1.5898915058209013
+        ]
+      ]
+    }
+  ],
+  "appState": {
+    "gridSize": null,
+    "viewBackgroundColor": "#ffffff"
+  },
+  "files": {}
+}

docs/images/why.excalidraw

@@ -0,0 +1,263 @@
+{
+  "type": "excalidraw",
+  "version": 2,
+  "source": "https://excalidraw.com",
+  "elements": [
+    {
+      "type": "rectangle",
+      "version": 633,
+      "versionNonce": 705603336,
+      "isDeleted": false,
+      "id": "mG5J3YEl3JUm8tWD2gLax",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 840.1017043524607,
+      "y": 323.8650942251963,
+      "strokeColor": "#ff7043",
+      "backgroundColor": "#ff7043",
+      "width": 412.0000000000001,
+      "height": 54.999999999999964,
+      "seed": 322487816,
+      "groupIds": [],
+      "strokeSharpness": "sharp",
+      "boundElements": [],
+      "updated": 1641744014625
+    },
+    {
+      "type": "text",
+      "version": 202,
+      "versionNonce": 1087974776,
+      "isDeleted": false,
+      "id": "N9ytCRthOA395FeHYtiWS",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 1010.3997812755383,
+      "y": 335.8362480713501,
+      "strokeColor": "#343a40",
+      "backgroundColor": "#03a9f4",
+      "width": 53,
+      "height": 36,
+      "seed": 1396884744,
+      "groupIds": [],
+      "strokeSharpness": "sharp",
+      "boundElements": [],
+      "updated": 1641744014625,
+      "fontSize": 28,
+      "fontFamily": 1,
+      "text": "API",
+      "baseline": 25,
+      "textAlign": "left",
+      "verticalAlign": "top",
+      "containerId": null,
+      "originalText": "API"
+    },
+    {
+      "type": "rectangle",
+      "version": 741,
+      "versionNonce": 917800456,
+      "isDeleted": false,
+      "id": "uFbzRXgEesWgll3llORrW",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 839.3613197370764,
+      "y": 387.49009422519623,
+      "strokeColor": "#00e676",
+      "backgroundColor": "#00e676",
+      "width": 184,
+      "height": 60.846153846153825,
+      "seed": 531948552,
+      "groupIds": [],
+      "strokeSharpness": "sharp",
+      "boundElements": [
+        {
+          "type": "text",
+          "id": "Ctud-Ap642NinH0-64fKT"
+        }
+      ],
+      "updated": 1641744014625
+    },
+    {
+      "type": "text",
+      "version": 509,
+      "versionNonce": 171632136,
+      "isDeleted": false,
+      "id": "Ctud-Ap642NinH0-64fKT",
+      "fillStyle": "cross-hatch",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 844.3613197370764,
+      "y": 399.91317114827314,
+      "strokeColor": "#000000",
+      "backgroundColor": "#03a9f4",
+      "width": 174,
+      "height": 36,
+      "seed": 720254072,
+      "groupIds": [],
+      "strokeSharpness": "sharp",
+      "boundElements": [],
+      "updated": 1641743551018,
+      "fontSize": 28,
+      "fontFamily": 1,
+      "text": "Pipeline",
+      "baseline": 25,
+      "textAlign": "center",
+      "verticalAlign": "middle",
+      "containerId": "uFbzRXgEesWgll3llORrW",
+      "originalText": "Pipeline"
+    },
+    {
+      "type": "rectangle",
+      "version": 726,
+      "versionNonce": 558153336,
+      "isDeleted": false,
+      "id": "_zQO_kyS5Pvq1zDhEn9hv",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 1032.3805505063074,
+      "y": 389.3362480713501,
+      "strokeColor": "#ffeb3b",
+      "backgroundColor": "#ffeb3b",
+      "width": 219,
+      "height": 56,
+      "seed": 1988829960,
+      "groupIds": [],
+      "strokeSharpness": "sharp",
+      "boundElements": [
+        {
+          "id": "tL_gw7MpliV1bDFJp8Cql",
+          "type": "text"
+        },
+        {
+          "type": "text",
+          "id": "tL_gw7MpliV1bDFJp8Cql"
+        }
+      ],
+      "updated": 1641744014625
+    },
+    {
+      "type": "text",
+      "version": 514,
+      "versionNonce": 1694557448,
+      "isDeleted": false,
+      "id": "tL_gw7MpliV1bDFJp8Cql",
+      "fillStyle": "cross-hatch",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 1037.2555505063074,
+      "y": 399.3362480713501,
+      "strokeColor": "#000000",
+      "backgroundColor": "#03a9f4",
+      "width": 209,
+      "height": 36,
+      "seed": 274132856,
+      "groupIds": [],
+      "strokeSharpness": "sharp",
+      "boundElements": [],
+      "updated": 1641743551018,
+      "fontSize": 28,
+      "fontFamily": 1,
+      "text": "Workflow",
+      "baseline": 25,
+      "textAlign": "center",
+      "verticalAlign": "middle",
+      "containerId": "_zQO_kyS5Pvq1zDhEn9hv",
+      "originalText": "Workflow"
+    },
+    {
+      "type": "rectangle",
+      "version": 455,
+      "versionNonce": 289834248,
+      "isDeleted": false,
+      "id": "QKBksUpSSix31GSKFO1uV",
+      "fillStyle": "hachure",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 838.5920889678456,
+      "y": 456.87470960981165,
+      "strokeColor": "#03a9f4",
+      "backgroundColor": "#03a9f4",
+      "width": 417,
+      "height": 60.846153846153825,
+      "seed": 910262792,
+      "groupIds": [],
+      "strokeSharpness": "sharp",
+      "boundElements": [
+        {
+          "id": "4o-d8ZzuqEpB7ROBAyReL",
+          "type": "text"
+        },
+        {
+          "id": "4o-d8ZzuqEpB7ROBAyReL",
+          "type": "text"
+        },
+        {
+          "type": "text",
+          "id": "4o-d8ZzuqEpB7ROBAyReL"
+        }
+      ],
+      "updated": 1641744014625
+    },
+    {
+      "type": "text",
+      "version": 258,
+      "versionNonce": 347014152,
+      "isDeleted": false,
+      "id": "4o-d8ZzuqEpB7ROBAyReL",
+      "fillStyle": "cross-hatch",
+      "strokeWidth": 1,
+      "strokeStyle": "solid",
+      "roughness": 1,
+      "opacity": 100,
+      "angle": 0,
+      "x": 843.4670889678456,
+      "y": 469.29778653288855,
+      "strokeColor": "#343a40",
+      "backgroundColor": "#03a9f4",
+      "width": 407,
+      "height": 36,
+      "seed": 1820749832,
+      "groupIds": [],
+      "strokeSharpness": "sharp",
+      "boundElements": [],
+      "updated": 1641743551019,
+      "fontSize": 28,
+      "fontFamily": 1,
+      "text": "Embeddings",
+      "baseline": 25,
+      "textAlign": "center",
+      "verticalAlign": "middle",
+      "containerId": "QKBksUpSSix31GSKFO1uV",
+      "originalText": "Embeddings"
+    }
+  ],
+  "appState": {
+    "gridSize": null,
+    "viewBackgroundColor": "#ffffff"
+  },
+  "files": {}
+}

docs/index.md

@@ -12,20 +12,20 @@
     <a href="https://github.com/neuml/txtai/releases">
         <img src="https://img.shields.io/github/release/neuml/txtai.svg?style=flat&color=success" alt="Version"/>
     </a>
-    <a href="https://github.com/neuml/txtai/releases">
-        <img src="https://img.shields.io/github/release-date/neuml/txtai.svg?style=flat&color=blue" alt="GitHub Release Date"/>
+    <a href="https://github.com/neuml/txtai">
+        <img src="https://img.shields.io/github/last-commit/neuml/txtai.svg?style=flat&color=blue" alt="GitHub last commit"/>
     </a>
     <a href="https://github.com/neuml/txtai/issues">
         <img src="https://img.shields.io/github/issues/neuml/txtai.svg?style=flat&color=success" alt="GitHub issues"/>
     </a>
-    <a href="https://github.com/neuml/txtai">
-        <img src="https://img.shields.io/github/last-commit/neuml/txtai.svg?style=flat&color=blue" alt="GitHub last commit"/>
+    <a href="https://join.slack.com/t/txtai/shared_invite/zt-1cagya4yf-DQeuZbd~aMwH5pckBU4vPg">
+        <img src="https://img.shields.io/badge/slack-join-blue?style=flat&logo=slack&logocolor=white" alt="Join Slack"/>
     </a>
     <a href="https://github.com/neuml/txtai/actions?query=workflow%3Abuild">
         <img src="https://github.com/neuml/txtai/workflows/build/badge.svg" alt="Build Status"/>
     </a>
     <a href="https://coveralls.io/github/neuml/txtai?branch=master">
-        <img src="https://img.shields.io/coveralls/github/neuml/txtai" alt="Coverage Status">
+        <img src="https://img.shields.io/coverallsCoverage/github/neuml/txtai" alt="Coverage Status">
     </a>
 </p>
 
@@ -37,6 +37,9 @@ txtai executes machine-learning workflows to transform data and build AI-powered
 
 Traditional search systems use keywords to find data. Semantic search applications have an understanding of natural language and identify results that have the same meaning, not necessarily the same keywords.
 
+![search](images/search.png#only-light)
+![search](images/search-dark.png#only-dark)
+
 Backed by state-of-the-art machine learning models, data is transformed into vector representations for search (also known as embeddings). Innovation is happening at a rapid pace, models can understand concepts in documents, audio, images and more.
 
 Summary of txtai features:
@@ -45,10 +48,17 @@ Summary of txtai features:
 - 📄 Create embeddings for text snippets, documents, audio, images and video. Supports transformers and word vectors.
 - 💡 Machine-learning pipelines to run extractive question-answering, zero-shot labeling, transcription, translation, summarization and text extraction
 - ↪️️ Workflows that join pipelines together to aggregate business logic. txtai processes can be microservices or full-fledged indexing workflows.
-- 🔗 API bindings for [JavaScript](https://github.com/neuml/txtai.js), [Java](https://github.com/neuml/txtai.java), [Rust](https://github.com/neuml/txtai.rs) and [Go](https://github.com/neuml/txtai.go)
+- ⚙️ Build with Python or YAML. API bindings available for [JavaScript](https://github.com/neuml/txtai.js), [Java](https://github.com/neuml/txtai.java), [Rust](https://github.com/neuml/txtai.rs) and [Go](https://github.com/neuml/txtai.go).
 - ☁️ Cloud-native architecture that scales out with container orchestration systems (e.g. Kubernetes)
 
-Applications range from similarity search to complex NLP-driven data extractions to generate structured databases. The following applications are powered by txtai.
+Applications range from similarity search to complex NLP-driven data extractions to generate structured databases. Semantic workflows transform and find data driven by user intent.
+
+![flows](images/flows.png#only-light)
+![flows](images/flows-dark.png#only-dark)
+
+The following applications are powered by txtai.
+
+![apps](https://raw.githubusercontent.com/neuml/txtai/master/apps.jpg)
 
 | Application  | Description  |
 |:----------|:-------------|
@@ -57,4 +67,4 @@ Applications range from similarity search to complex NLP-driven data extractions
 | [neuspo](https://neuspo.com) | Fact-driven, real-time sports event and news site |
 | [codequestion](https://github.com/neuml/codequestion) | Ask coding questions directly from the terminal |
 
-txtai is built with Python 3.6+, [Hugging Face Transformers](https://github.com/huggingface/transformers), [Sentence Transformers](https://github.com/UKPLab/sentence-transformers) and [FastAPI](https://github.com/tiangolo/fastapi)
+txtai is built with Python 3.7+, [Hugging Face Transformers](https://github.com/huggingface/transformers), [Sentence Transformers](https://github.com/UKPLab/sentence-transformers) and [FastAPI](https://github.com/tiangolo/fastapi)

docs/install.md

@@ -1,41 +1,26 @@
 # Installation
 
+![install](images/install.png#only-light)
+![install](images/install-dark.png#only-dark)
+
 The easiest way to install is via pip and PyPI
 
-    pip install txtai
+```
+pip install txtai
+```
 
-Python 3.6+ is supported. Using a Python [virtual environment](https://docs.python.org/3/library/venv.html) is recommended.
-
-## Install from source
-
-txtai can also be installed directly from GitHub to access the latest, unreleased features.
-
-    pip install git+https://github.com/neuml/txtai
-
-## Environment specific prerequisites
-
-Additional environment specific prerequisites are below.
-
-### Linux
-
-Optional audio transcription requires a [system library to be installed](https://github.com/bastibe/python-soundfile#installation)
-
-### macOS
-
-Run `brew install libomp` see [this link](https://github.com/kyamagu/faiss-wheels#prerequisite)
-
-### Windows
-
-Optional dependencies require [C++ Build Tools](https://visualstudio.microsoft.com/visual-cpp-build-tools/)
+Python 3.7+ is supported. Using a Python [virtual environment](https://docs.python.org/3/library/venv.html) is recommended.
 
 ## Optional dependencies
 
 txtai has the following optional dependencies that can be installed as extras. The patterns below are supported
 in setup.py install_requires sections.
 
+_Note: Extras are provided for convenience. Alternatively, individual packages can be installed to limit dependencies._
+
 ### All
 
-Install all dependencies (mirrors txtai < 3.2)
+Install all dependencies.
 
 ```
 pip install txtai[all]
@@ -49,6 +34,46 @@ Serve txtai via a web API.
 pip install txtai[api]
 ```
 
+### Cloud
+
+Interface with cloud compute.
+
+```
+pip install txtai[cloud]
+```
+
+### Console
+
+Command line index query console.
+
+```
+pip install txtai[console]
+```
+
+### Database
+
+Additional content storage options.
+
+```
+pip install txtai[database]
+```
+
+### Graph
+
+Topic modeling, data connectivity and network analysis.
+
+```
+pip install txtai[graph]
+```
+
+### Model
+
+Additional non-standard models.
+
+```
+pip install txtai[model]
+```
+
 ### Pipeline
 
 All pipelines - default install comes with most common pipelines.
@@ -57,6 +82,8 @@ All pipelines - default install comes with most common pipelines.
 pip install txtai[pipeline]
 ```
 
+More granular extras are available for pipeline categories: `pipeline-audio`, `pipeline-data`, `pipeline-image`, `pipeline-text`, and `pipeline-train`.
+
 ### Similarity
 
 Word vectors, support for sentence-transformers models not on the HF Hub and additional ANN libraries.
@@ -73,8 +100,48 @@ All workflow tasks - default install comes with most common workflow tasks.
 pip install txtai[workflow]
 ```
 
+### Combining dependencies
+
 Multiple dependencies can be specified at the same time.
 
 ```
 pip install txtai[pipeline,workflow]
 ```
+
+## Environment specific prerequisites
+
+Additional environment specific prerequisites are below.
+
+### Linux
+
+Optional audio transcription requires a [system library to be installed](https://github.com/bastibe/python-soundfile#installation)
+
+### macOS
+
+Run `brew install libomp` see [this link](https://github.com/kyamagu/faiss-wheels#prerequisite)
+
+_Note: Older versions of Python had issues with PyTorch 1.12+. If issues are encountered, either upgrade Python or downgrade PyTorch._
+
+### Windows
+
+Optional dependencies require [C++ Build Tools](https://visualstudio.microsoft.com/visual-cpp-build-tools/)
+
+_Note: Older versions of Python did not have SQLite JSON support. If issues are encountered, upgrade to a later version of Python._
+
+The [txtai build workflow](https://github.com/neuml/txtai/blob/master/.github/workflows/build.yml) occasionally has work arounds for other known but temporary dependency issues.
+
+## Install from source
+
+txtai can also be installed directly from GitHub to access the latest, unreleased features.
+
+    pip install git+https://github.com/neuml/txtai
+
+## Conda
+
+A [community-supported txtai package](https://anaconda.org/conda-forge/txtai) is available via conda-forge.
+
+    conda install -c conda-forge txtai
+
+## Run with containers
+
+Docker images are available for txtai. [See this section](../cloud) for more information on container-based installs.

docs/overrides/main.html

@@ -0,0 +1,24 @@
+{% extends "base.html" %}
+
+{% block extrahead %}
+    {% set title = config.site_name %}
+    {% if page and page.meta and page.meta.title %}
+        {% set title = title ~ " - " ~ page.meta.title %}
+    {% elif page and page.title and not page.is_homepage %}
+        {% set title = title ~ " - " ~ page.title %}
+    {% endif %}
+
+    <meta property="og:type" content="website" />
+    <meta property="og:title" content="{{ title }}" />
+    <meta property="og:description" content="{{ config.site_description }}" />
+    <meta property="og:url" content="{{ page.canonical_url }}" />
+    <meta property="og:image" content="https://repository-images.githubusercontent.com/286301447/a5a3cc1d-0e3f-4fee-b1a4-a30a0d11e794" />
+    <meta property="og:image:type" content="image/png" />
+    <meta property="og:image:width" content="1920" />
+    <meta property="og:image:height" content="960" />
+
+    <meta name="twitter:card" content="summary_large_image" />
+    <meta name="twitter:title" content="{{ title }}" />
+    <meta name="twitter:description" content="{{ config.site_description }}" />
+    <meta name="twitter:image" content="https://repository-images.githubusercontent.com/286301447/a5a3cc1d-0e3f-4fee-b1a4-a30a0d11e794" />
+{% endblock %}

docs/pipeline/audio/texttospeech.md

@@ -0,0 +1,73 @@
+# Text To Speech
+
+![pipeline](../../images/pipeline.png#only-light)
+![pipeline](../../images/pipeline-dark.png#only-dark)
+
+The Text To Speech pipeline generates speech from text.
+
+## Example
+
+The following shows a simple example using this pipeline.
+
+```python
+from txtai.pipeline import TextToSpeech
+
+# Create and run pipeline
+tts = TextToSpeech()
+tts("Say something here")
+```
+
+See the link below for a more detailed example.
+
+| Notebook  | Description  |       |
+|:----------|:-------------|------:|
+| [Text to speech generation](https://github.com/neuml/txtai/blob/master/examples/40_Text_to_Speech_Generation.ipynb) | Generate speech from text | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/40_Text_to_Speech_Generation.ipynb) |
+
+This pipeline is backed by ONNX models from the Hugging Face Hub. The following models are currently available.
+
+- [ljspeech-jets-onnx](https://huggingface.co/NeuML/ljspeech-jets-onnx)
+- [ljspeech-vits-onnx](https://huggingface.co/NeuML/ljspeech-vits-onnx)
+
+## Configuration-driven example
+
+Pipelines are run with Python or configuration. Pipelines can be instantiated in [configuration](../../../api/configuration/#pipeline) using the lower case name of the pipeline. Configuration-driven pipelines are run with [workflows](../../../workflow/#configuration-driven-example) or the [API](../../../api#local-instance).
+
+### config.yml
+```yaml
+# Create pipeline using lower case class name
+texttospeech:
+
+# Run pipeline with workflow
+workflow:
+  tts:
+    tasks:
+      - action: texttospeech
+```
+
+### Run with Workflows
+
+```python
+from txtai.app import Application
+
+# Create and run pipeline with workflow
+app = Application("config.yml")
+list(app.workflow("tts", ["Say something here"]))
+```
+
+### Run with API
+
+```bash
+CONFIG=config.yml uvicorn "txtai.api:app" &
+
+curl \
+  -X POST "http://localhost:8000/workflow" \
+  -H "Content-Type: application/json" \
+  -d '{"name":"tts", "elements":["Say something here"]}'
+```
+
+## Methods
+
+Python documentation for the pipeline.
+
+### ::: txtai.pipeline.TextToSpeech.__init__
+### ::: txtai.pipeline.TextToSpeech.__call__

docs/pipeline/audio/transcription.md

@@ -0,0 +1,68 @@
+# Transcription
+
+![pipeline](../../images/pipeline.png#only-light)
+![pipeline](../../images/pipeline-dark.png#only-dark)
+
+The Transcription pipeline converts speech in audio files to text.
+
+## Example
+
+The following shows a simple example using this pipeline.
+
+```python
+from txtai.pipeline import Transcription
+
+# Create and run pipeline
+transcribe = Transcription()
+transcribe("path to wav file")
+```
+
+See the link below for a more detailed example.
+
+| Notebook  | Description  |       |
+|:----------|:-------------|------:|
+| [Transcribe audio to text](https://github.com/neuml/txtai/blob/master/examples/11_Transcribe_audio_to_text.ipynb) | Convert audio files to text | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/11_Transcribe_audio_to_text.ipynb) |
+
+## Configuration-driven example
+
+Pipelines are run with Python or configuration. Pipelines can be instantiated in [configuration](../../../api/configuration/#pipeline) using the lower case name of the pipeline. Configuration-driven pipelines are run with [workflows](../../../workflow/#configuration-driven-example) or the [API](../../../api#local-instance).
+
+### config.yml
+```yaml
+# Create pipeline using lower case class name
+transcription:
+
+# Run pipeline with workflow
+workflow:
+  transcribe:
+    tasks:
+      - action: transcription
+```
+
+### Run with Workflows
+
+```python
+from txtai.app import Application
+
+# Create and run pipeline with workflow
+app = Application("config.yml")
+list(app.workflow("transcribe", ["path to wav file"]))
+```
+
+### Run with API
+
+```bash
+CONFIG=config.yml uvicorn "txtai.api:app" &
+
+curl \
+  -X POST "http://localhost:8000/workflow" \
+  -H "Content-Type: application/json" \
+  -d '{"name":"transcribe", "elements":["path to wav file"]}'
+```
+
+## Methods
+
+Python documentation for the pipeline.
+
+### ::: txtai.pipeline.Transcription.__init__
+### ::: txtai.pipeline.Transcription.__call__

docs/pipeline/data/segmentation.md

@@ -0,0 +1,63 @@
+# Segmentation
+
+![pipeline](../../images/pipeline.png#only-light)
+![pipeline](../../images/pipeline-dark.png#only-dark)
+
+The Segmentation pipeline segments text into semantic units.
+
+## Example
+
+The following shows a simple example using this pipeline.
+
+```python
+from txtai.pipeline import Segmentation
+
+# Create and run pipeline
+segment = Segmentation(sentences=True)
+segment("This is a test. And another test.")
+```
+
+## Configuration-driven example
+
+Pipelines are run with Python or configuration. Pipelines can be instantiated in [configuration](../../../api/configuration/#pipeline) using the lower case name of the pipeline. Configuration-driven pipelines are run with [workflows](../../../workflow/#configuration-driven-example) or the [API](../../../api#local-instance).
+
+### config.yml
+```yaml
+# Create pipeline using lower case class name
+segmentation:
+  sentences: true
+
+# Run pipeline with workflow
+workflow:
+  segment:
+    tasks:
+      - action: segmentation
+```
+
+### Run with Workflows
+
+```python
+from txtai.app import Application
+
+# Create and run pipeline with workflow
+app = Application("config.yml")
+list(app.workflow("segment", ["This is a test. And another test."]))
+```
+
+### Run with API
+
+```bash
+CONFIG=config.yml uvicorn "txtai.api:app" &
+
+curl \
+  -X POST "http://localhost:8000/workflow" \
+  -H "Content-Type: application/json" \
+  -d '{"name":"segment", "elements":["This is a test. And another test."]}'
+```
+
+## Methods
+
+Python documentation for the pipeline.
+
+### ::: txtai.pipeline.Segmentation.__init__
+### ::: txtai.pipeline.Segmentation.__call__

docs/pipeline/data/tabular.md

@@ -0,0 +1,71 @@
+# Tabular
+
+![pipeline](../../images/pipeline.png#only-light)
+![pipeline](../../images/pipeline-dark.png#only-dark)
+
+The Tabular pipeline splits tabular data into rows and columns. The tabular pipeline is most useful in creating (id, text, tag) tuples to load into Embedding indexes. 
+
+## Example
+
+The following shows a simple example using this pipeline.
+
+```python
+from txtai.pipeline import Tabular
+
+# Create and run pipeline
+tabular = Tabular("id", ["text"])
+tabular("path to csv file")
+```
+
+See the link below for a more detailed example.
+
+| Notebook  | Description  |       |
+|:----------|:-------------|------:|
+| [Transform tabular data with composable workflows](https://github.com/neuml/txtai/blob/master/examples/22_Transform_tabular_data_with_composable_workflows.ipynb) | Transform, index and search tabular data | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/22_Transform_tabular_data_with_composable_workflows.ipynb) |
+
+## Configuration-driven example
+
+Pipelines are run with Python or configuration. Pipelines can be instantiated in [configuration](../../../api/configuration/#pipeline) using the lower case name of the pipeline. Configuration-driven pipelines are run with [workflows](../../../workflow/#configuration-driven-example) or the [API](../../../api#local-instance).
+
+### config.yml
+```yaml
+# Create pipeline using lower case class name
+tabular:
+    idcolumn: id
+    textcolumns:
+      - text
+
+# Run pipeline with workflow
+workflow:
+  tabular:
+    tasks:
+      - action: tabular
+```
+
+### Run with Workflows
+
+```python
+from txtai.app import Application
+
+# Create and run pipeline with workflow
+app = Application("config.yml")
+list(app.workflow("tabular", ["path to csv file"]))
+```
+
+### Run with API
+
+```bash
+CONFIG=config.yml uvicorn "txtai.api:app" &
+
+curl \
+  -X POST "http://localhost:8000/workflow" \
+  -H "Content-Type: application/json" \
+  -d '{"name":"tabular", "elements":["path to csv file"]}'
+```
+
+## Methods
+
+Python documentation for the pipeline.
+
+### ::: txtai.pipeline.Tabular.__init__
+### ::: txtai.pipeline.Tabular.__call__

docs/pipeline/data/textractor.md

@@ -0,0 +1,68 @@
+# Textractor
+
+![pipeline](../../images/pipeline.png#only-light)
+![pipeline](../../images/pipeline-dark.png#only-dark)
+
+The Textractor pipeline extracts and splits text from documents. This pipeline uses either an [Apache Tika](https://github.com/chrismattmann/tika-python) backend (if Java is available) or [BeautifulSoup4](https://pypi.org/project/beautifulsoup4/).
+
+## Example
+
+The following shows a simple example using this pipeline.
+
+```python
+from txtai.pipeline import Textractor
+
+# Create and run pipeline
+textract = Textractor()
+textract("https://github.com/neuml/txtai")
+```
+
+See the link below for a more detailed example.
+
+| Notebook  | Description  |       |
+|:----------|:-------------|------:|
+| [Extract text from documents](https://github.com/neuml/txtai/blob/master/examples/10_Extract_text_from_documents.ipynb) | Extract text from PDF, Office, HTML and more | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/10_Extract_text_from_documents.ipynb) |
+
+## Configuration-driven example
+
+Pipelines are run with Python or configuration. Pipelines can be instantiated in [configuration](../../../api/configuration/#pipeline) using the lower case name of the pipeline. Configuration-driven pipelines are run with [workflows](../../../workflow/#configuration-driven-example) or the [API](../../../api#local-instance).
+
+### config.yml
+```yaml
+# Create pipeline using lower case class name
+textractor:
+
+# Run pipeline with workflow
+workflow:
+  textract:
+    tasks:
+      - action: textractor
+```
+
+### Run with Workflows
+
+```python
+from txtai.app import Application
+
+# Create and run pipeline with workflow
+app = Application("config.yml")
+list(app.workflow("textract", ["https://github.com/neuml/txtai"]))
+```
+
+### Run with API
+
+```bash
+CONFIG=config.yml uvicorn "txtai.api:app" &
+
+curl \
+  -X POST "http://localhost:8000/workflow" \
+  -H "Content-Type: application/json" \
+  -d '{"name":"textract", "elements":["https://github.com/neuml/txtai"]}'
+```
+
+## Methods
+
+Python documentation for the pipeline.
+
+### ::: txtai.pipeline.Textractor.__init__
+### ::: txtai.pipeline.Textractor.__call__

docs/pipeline/image/caption.md

@@ -0,0 +1,68 @@
+# Caption
+
+![pipeline](../../images/pipeline.png#only-light)
+![pipeline](../../images/pipeline-dark.png#only-dark)
+
+The caption pipeline reads a list of images and returns a list of captions for those images.
+
+## Example
+
+The following shows a simple example using this pipeline.
+
+```python
+from txtai.pipeline import Caption
+
+# Create and run pipeline
+caption = Caption()
+caption("path to image file")
+```
+
+See the link below for a more detailed example.
+
+| Notebook  | Description  |       |
+|:----------|:-------------|------:|
+| [Generate image captions and detect objects](https://github.com/neuml/txtai/blob/master/examples/25_Generate_image_captions_and_detect_objects.ipynb) | Captions and object detection for images | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/25_Generate_image_captions_and_detect_objects.ipynb) |
+
+## Configuration-driven example
+
+Pipelines are run with Python or configuration. Pipelines can be instantiated in [configuration](../../../api/configuration/#pipeline) using the lower case name of the pipeline. Configuration-driven pipelines are run with [workflows](../../../workflow/#configuration-driven-example) or the [API](../../../api#local-instance).
+
+### config.yml
+```yaml
+# Create pipeline using lower case class name
+caption:
+
+# Run pipeline with workflow
+workflow:
+  caption:
+    tasks:
+      - action: caption
+```
+
+### Run with Workflows
+
+```python
+from txtai.app import Application
+
+# Create and run pipeline with workflow
+app = Application("config.yml")
+list(app.workflow("caption", ["path to image file"]))
+```
+
+### Run with API
+
+```bash
+CONFIG=config.yml uvicorn "txtai.api:app" &
+
+curl \
+  -X POST "http://localhost:8000/workflow" \
+  -H "Content-Type: application/json" \
+  -d '{"name":"caption", "elements":["path to image file"]}'
+```
+
+## Methods
+
+Python documentation for the pipeline.
+
+### ::: txtai.pipeline.Caption.__init__
+### ::: txtai.pipeline.Caption.__call__

docs/pipeline/image/imagehash.md

@@ -0,0 +1,68 @@
+# ImageHash
+
+![pipeline](../../images/pipeline.png#only-light)
+![pipeline](../../images/pipeline-dark.png#only-dark)
+
+The image hash pipeline generates perceptual image hashes. These hashes can be used to detect near-duplicate images. This method is not backed by machine learning models and not intended to find conceptually similar images.
+
+## Example
+
+The following shows a simple example using this pipeline.
+
+```python
+from txtai.pipeline import ImageHash
+
+# Create and run pipeline
+ihash = ImageHash()
+ihash("path to image file")
+```
+
+See the link below for a more detailed example.
+
+| Notebook  | Description  |       |
+|:----------|:-------------|------:|
+| [Near duplicate image detection](https://github.com/neuml/txtai/blob/master/examples/31_Near_duplicate_image_detection.ipynb) | Identify duplicate and near-duplicate images | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/31_Near_duplicate_image_detection.ipynb) |
+
+## Configuration-driven example
+
+Pipelines are run with Python or configuration. Pipelines can be instantiated in [configuration](../../../api/configuration/#pipeline) using the lower case name of the pipeline. Configuration-driven pipelines are run with [workflows](../../../workflow/#configuration-driven-example) or the [API](../../../api#local-instance).
+
+### config.yml
+```yaml
+# Create pipeline using lower case class name
+imagehash:
+
+# Run pipeline with workflow
+workflow:
+  imagehash:
+    tasks:
+      - action: imagehash
+```
+
+### Run with Workflows
+
+```python
+from txtai.app import Application
+
+# Create and run pipeline with workflow
+app = Application("config.yml")
+list(app.workflow("imagehash", ["path to image file"]))
+```
+
+### Run with API
+
+```bash
+CONFIG=config.yml uvicorn "txtai.api:app" &
+
+curl \
+  -X POST "http://localhost:8000/workflow" \
+  -H "Content-Type: application/json" \
+  -d '{"name":"imagehash", "elements":["path to image file"]}'
+```
+
+## Methods
+
+Python documentation for the pipeline.
+
+### ::: txtai.pipeline.ImageHash.__init__
+### ::: txtai.pipeline.ImageHash.__call__

docs/pipeline/image/objects.md

@@ -0,0 +1,68 @@
+# Objects
+
+![pipeline](../../images/pipeline.png#only-light)
+![pipeline](../../images/pipeline-dark.png#only-dark)
+
+The Objects pipeline reads a list of images and returns a list of detected objects.
+
+## Example
+
+The following shows a simple example using this pipeline.
+
+```python
+from txtai.pipeline import Objects
+
+# Create and run pipeline
+objects = Objects()
+objects("path to image file")
+```
+
+See the link below for a more detailed example.
+
+| Notebook  | Description  |       |
+|:----------|:-------------|------:|
+| [Generate image captions and detect objects](https://github.com/neuml/txtai/blob/master/examples/25_Generate_image_captions_and_detect_objects.ipynb) | Captions and object detection for images | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/25_Generate_image_captions_and_detect_objects.ipynb) |
+
+## Configuration-driven example
+
+Pipelines are run with Python or configuration. Pipelines can be instantiated in [configuration](../../../api/configuration/#pipeline) using the lower case name of the pipeline. Configuration-driven pipelines are run with [workflows](../../../workflow/#configuration-driven-example) or the [API](../../../api#local-instance).
+
+### config.yml
+```yaml
+# Create pipeline using lower case class name
+objects:
+
+# Run pipeline with workflow
+workflow:
+  objects:
+    tasks:
+      - action: objects
+```
+
+### Run with Workflows
+
+```python
+from txtai.app import Application
+
+# Create and run pipeline with workflow
+app = Application("config.yml")
+list(app.workflow("objects", ["path to image file"]))
+```
+
+### Run with API
+
+```bash
+CONFIG=config.yml uvicorn "txtai.api:app" &
+
+curl \
+  -X POST "http://localhost:8000/workflow" \
+  -H "Content-Type: application/json" \
+  -d '{"name":"objects", "elements":["path to image file"]}'
+```
+
+## Methods
+
+Python documentation for the pipeline.
+
+### ::: txtai.pipeline.Objects.__init__
+### ::: txtai.pipeline.Objects.__call__

docs/pipeline/index.md

@@ -0,0 +1,36 @@
+# Pipeline
+
+![pipeline](../images/pipeline.png#only-light)
+![pipeline](../images/pipeline-dark.png#only-dark)
+
+txtai provides a generic pipeline processing framework with the only interface requirement being a `__call__` method. Pipelines are flexible and process various types of data. Pipelines can wrap machine learning models as well as other processes.
+
+Pipelines are run with Python or configuration. Pipelines can be instantiated in [configuration](../api/configuration/#pipeline) using the lower case name of the pipeline. Configuration-driven pipelines are run with [workflows](../workflow/#configuration-driven-example) or the [API](../api#local-instance).
+
+## List of pipelines
+
+The following is a list of the default pipelines available in txtai.
+
+- Audio
+    - [TextToSpeech](audio/texttospeech)
+    - [Transcription](audio/transcription)
+- Data Processing
+    - [Segmentation](data/segmentation)
+    - [Tabular](data/tabular)
+    - [Text extraction](data/textractor)
+- Image
+    - [Caption](image/caption)
+    - [Image Hash](image/imagehash)
+    - [Objects](image/objects)
+- Text
+    - [Entity](text/entity)
+    - [Extractive QA](text/extractor)
+    - [Labeling](text/labels)
+    - [Sequences](text/sequences)
+    - [Similarity](text/similarity)
+    - [Summary](text/summary)
+    - [Translation](text/translation)
+- Training
+    - [HF ONNX](train/hfonnx)
+    - [ML ONNX](train/mlonnx)
+    - [Trainer](train/trainer)

docs/pipeline/text/entity.md

@@ -0,0 +1,69 @@
+# Entity
+
+![pipeline](../../images/pipeline.png#only-light)
+![pipeline](../../images/pipeline-dark.png#only-dark)
+
+The Entity pipeline applies a token classifier to text and extracts entity/label combinations.
+
+## Example
+
+The following shows a simple example using this pipeline.
+
+```python
+from txtai.pipeline import Entity
+
+# Create and run pipeline
+entity = Entity()
+entity("Canada's last fully intact ice shelf has suddenly collapsed, " \
+       "forming a Manhattan-sized iceberg")
+```
+
+See the link below for a more detailed example.
+
+| Notebook  | Description  |       |
+|:----------|:-------------|------:|
+| [Entity extraction workflows](https://github.com/neuml/txtai/blob/master/examples/26_Entity_extraction_workflows.ipynb) | Identify entity/label combinations | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/26_Entity_extraction_workflows.ipynb) |
+
+## Configuration-driven example
+
+Pipelines are run with Python or configuration. Pipelines can be instantiated in [configuration](../../../api/configuration/#pipeline) using the lower case name of the pipeline. Configuration-driven pipelines are run with [workflows](../../../workflow/#configuration-driven-example) or the [API](../../../api#local-instance).
+
+### config.yml
+```yaml
+# Create pipeline using lower case class name
+entity:
+
+# Run pipeline with workflow
+workflow:
+  entity:
+    tasks:
+      - action: entity
+```
+
+### Run with Workflows
+
+```python
+from txtai.app import Application
+
+# Create and run pipeline with workflow
+app = Application("config.yml")
+list(app.workflow("entity", ["Canada's last fully intact ice shelf has suddenly collapsed, forming a Manhattan-sized iceberg"]))
+```
+
+### Run with API
+
+```bash
+CONFIG=config.yml uvicorn "txtai.api:app" &
+
+curl \
+  -X POST "http://localhost:8000/workflow" \
+  -H "Content-Type: application/json" \
+  -d '{"name":"entity", "elements": ["Canadas last fully intact ice shelf has suddenly collapsed, forming a Manhattan-sized iceberg"]}'
+```
+
+## Methods
+
+Python documentation for the pipeline.
+
+### ::: txtai.pipeline.Entity.__init__
+### ::: txtai.pipeline.Entity.__call__

docs/pipeline/text/extractor.md

@@ -0,0 +1,71 @@
+# Extractor
+
+![pipeline](../../images/pipeline.png#only-light)
+![pipeline](../../images/pipeline-dark.png#only-dark)
+
+The Extractor pipeline is a combination of an embeddings query and an Extractive QA model. Filtering the context for a QA model helps maximize performance of the model.
+
+## Example
+
+The following shows a simple example using this pipeline.
+
+```python
+from txtai.embeddings import Embeddings
+from txtai.pipeline import Extractor
+
+# Embeddings model ranks candidates before passing to QA pipeline
+embeddings = Embeddings({"path": "sentence-transformers/nli-mpnet-base-v2"})
+
+# Create and run pipeline
+extractor = Extractor(embeddings, "distilbert-base-cased-distilled-squad")
+extractor([["What was won"] * 3 + [False]],
+          ["Maine man wins $1M from $25 lottery ticket"])
+```
+
+See the links below for more detailed examples.
+
+| Notebook  | Description  |       |
+|:----------|:-------------|------:|
+| [Extractive QA with txtai](https://github.com/neuml/txtai/blob/master/examples/05_Extractive_QA_with_txtai.ipynb) | Introduction to extractive question-answering with txtai | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/05_Extractive_QA_with_txtai.ipynb) |
+| [Extractive QA with Elasticsearch](https://github.com/neuml/txtai/blob/master/examples/06_Extractive_QA_with_Elasticsearch.ipynb) | Run extractive question-answering queries with Elasticsearch | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/06_Extractive_QA_with_Elasticsearch.ipynb) |
+| [Extractive QA to build structured data](https://github.com/neuml/txtai/blob/master/examples/20_Extractive_QA_to_build_structured_data.ipynb) | Build structured datasets using extractive question-answering | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/20_Extractive_QA_to_build_structured_data.ipynb) |
+
+## Configuration-driven example
+
+Pipelines are run with Python or configuration. Pipelines can be instantiated in [configuration](../../../api/configuration/#pipeline) using the lower case name of the pipeline. Configuration-driven pipelines are run with [workflows](../../../workflow/#configuration-driven-example) or the [API](../../../api#local-instance).
+
+### config.yml
+```yaml
+# Create pipeline using lower case class name
+extractor:
+```
+
+### Run with Workflows
+
+```python
+from txtai.app import Application
+
+# Create and run pipeline with workflow
+app = Application("config.yml")
+list(app.extract([{"name": "What was won", "query": "What was won",
+                   "question", "What was won", "snippet": False}], 
+                 ["Maine man wins $1M from $25 lottery ticket"]))
+```
+
+### Run with API
+
+```bash
+CONFIG=config.yml uvicorn "txtai.api:app" &
+
+curl \
+  -X POST "http://localhost:8000/extract" \
+  -H "Content-Type: application/json" \
+  -d '{"queue": [{"name":"What was won", "query": "What was won", "question": "What was won", "snippet": false}], "texts": ["Maine man wins $1M from $25 lottery ticket"]}'
+```
+
+## Methods
+
+Python documentation for the pipeline.
+
+### ::: txtai.pipeline.Extractor.__init__
+### ::: txtai.pipeline.Extractor.__call__

docs/pipeline/text/labels.md

@@ -0,0 +1,72 @@
+# Labels
+
+![pipeline](../../images/pipeline.png#only-light)
+![pipeline](../../images/pipeline-dark.png#only-dark)
+
+The Labels pipeline uses a text classification model to apply labels to input text. This pipeline can classify text using either a zero shot model (dynamic labeling) or a standard text classification model (fixed labeling).
+
+## Example
+
+The following shows a simple example using this pipeline.
+
+```python
+from txtai.pipeline import Labels
+
+# Create and run pipeline
+labels = Labels()
+labels(
+    ["Great news", "That's rough"],
+    ["positive", "negative"]
+)
+```
+
+See the link below for a more detailed example.
+
+| Notebook  | Description  |       |
+|:----------|:-------------|------:|
+| [Apply labels with zero shot classification](https://github.com/neuml/txtai/blob/master/examples/07_Apply_labels_with_zero_shot_classification.ipynb) | Use zero shot learning for labeling, classification and topic modeling | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/07_Apply_labels_with_zero_shot_classification.ipynb) |
+
+## Configuration-driven example
+
+Pipelines are run with Python or configuration. Pipelines can be instantiated in [configuration](../../../api/configuration/#pipeline) using the lower case name of the pipeline. Configuration-driven pipelines are run with [workflows](../../../workflow/#configuration-driven-example) or the [API](../../../api#local-instance).
+
+### config.yml
+```yaml
+# Create pipeline using lower case class name
+labels:
+
+# Run pipeline with workflow
+workflow:
+  labels:
+    tasks:
+      - action: labels
+        args: [["positive", "negative"]]
+```
+
+### Run with Workflows
+
+```python
+from txtai.app import Application
+
+# Create and run pipeline with workflow
+app = Application("config.yml")
+list(app.workflow("labels", ["Great news", "That's rough"]))
+```
+
+### Run with API
+
+```bash
+CONFIG=config.yml uvicorn "txtai.api:app" &
+
+curl \
+  -X POST "http://localhost:8000/workflow" \
+  -H "Content-Type: application/json" \
+  -d '{"name":"labels", "elements": ["Great news", "Thats rough"]}'
+```
+
+## Methods
+
+Python documentation for the pipeline.
+
+### ::: txtai.pipeline.Labels.__init__
+### ::: txtai.pipeline.Labels.__call__

docs/pipeline/text/sequences.md

@@ -0,0 +1,69 @@
+# Sequences
+
+![pipeline](../../images/pipeline.png#only-light)
+![pipeline](../../images/pipeline-dark.png#only-dark)
+
+The Sequences pipeline runs text through a sequence-sequence model and generates output text.
+
+## Example
+
+The following shows a simple example using this pipeline.
+
+```python
+from txtai.pipeline import Sequences
+
+# Create and run pipeline
+sequences = Sequences()
+sequences("Hello, how are you?", "translate English to French: ")
+```
+
+See the link below for a more detailed example.
+
+| Notebook  | Description  |       |
+|:----------|:-------------|------:|
+| [Query translation](https://github.com/neuml/txtai/blob/master/examples/33_Query_translation.ipynb) | Domain-specific natural language queries with query translation | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/33_Query_translation.ipynb) |
+
+## Configuration-driven example
+
+Pipelines are run with Python or configuration. Pipelines can be instantiated in [configuration](../../../api/configuration/#pipeline) using the lower case name of the pipeline. Configuration-driven pipelines are run with [workflows](../../../workflow/#configuration-driven-example) or the [API](../../../api#local-instance).
+
+### config.yml
+```yaml
+# Create pipeline using lower case class name
+sequences:
+
+# Run pipeline with workflow
+workflow:
+  sequences:
+    tasks:
+      - action: sequences
+        args: ["translate English to French: "]
+```
+
+### Run with Workflows
+
+```python
+from txtai.app import Application
+
+# Create and run pipeline with workflow
+app = Application("config.yml")
+list(app.workflow("sequences", ["Hello, how are you?"]))
+```
+
+### Run with API
+
+```bash
+CONFIG=config.yml uvicorn "txtai.api:app" &
+
+curl \
+  -X POST "http://localhost:8000/workflow" \
+  -H "Content-Type: application/json" \
+  -d '{"name":"sequences", "elements": ["Hello, how are you?"]}'
+```
+
+## Methods
+
+Python documentation for the pipeline.
+
+### ::: txtai.pipeline.Sequences.__init__
+### ::: txtai.pipeline.Sequences.__call__

docs/pipeline/text/similarity.md

@@ -0,0 +1,72 @@
+# Similarity
+
+![pipeline](../../images/pipeline.png#only-light)
+![pipeline](../../images/pipeline-dark.png#only-dark)
+
+The Similarity pipeline computes similarity between queries and list of text using a text classifier.
+
+This pipeline supports both standard text classification models and zero-shot classification models. The pipeline uses the queries as labels for the input text. The results are transposed to get scores per query/label vs scores per input text. 
+
+Cross-encoder models are supported via the `crossencode=True` constructor parameter. These models are loaded with a CrossEncoder pipeline that can also be instantiated directly. The CrossEncoder pipeline has the same methods and functionality as described below.
+
+## Example
+
+The following shows a simple example using this pipeline.
+
+```python
+from txtai.pipeline import Similarity
+
+# Create and run pipeline
+similarity = Similarity()
+similarity("feel good story", [
+    "Maine man wins $1M from $25 lottery ticket", 
+    "Don't sacrifice slower friends in a bear attack"
+])
+```
+
+See the link below for a more detailed example.
+
+| Notebook  | Description  |       |
+|:----------|:-------------|------:|
+| [Add semantic search to Elasticsearch](https://github.com/neuml/txtai/blob/master/examples/04_Add_semantic_search_to_Elasticsearch.ipynb)  | Add semantic search to existing search systems | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/04_Add_semantic_search_to_Elasticsearch.ipynb) |
+
+## Configuration-driven example
+
+Pipelines are run with Python or configuration. Pipelines can be instantiated in [configuration](../../../api/configuration/#pipeline) using the lower case name of the pipeline. Configuration-driven pipelines are run with [workflows](../../../workflow/#configuration-driven-example) or the [API](../../../api#local-instance).
+
+### config.yml
+```yaml
+# Create pipeline using lower case class name
+similarity:
+```
+
+### Run with Workflows
+
+```python
+from txtai.app import Application
+
+# Create and run pipeline with workflow
+app = Application("config.yml")
+app.similarity("feel good story", [
+    "Maine man wins $1M from $25 lottery ticket", 
+    "Don't sacrifice slower friends in a bear attack"
+])
+```
+
+### Run with API
+
+```bash
+CONFIG=config.yml uvicorn "txtai.api:app" &
+
+curl \
+  -X POST "http://localhost:8000/similarity" \
+  -H "Content-Type: application/json" \
+  -d '{"query": "feel good story", "texts": ["Maine man wins $1M from $25 lottery ticket", "Dont sacrifice slower friends in a bear attack"]}'
+```
+
+## Methods
+
+Python documentation for the pipeline.
+
+### ::: txtai.pipeline.Similarity.__init__
+### ::: txtai.pipeline.Similarity.__call__

docs/pipeline/text/summary.md

@@ -0,0 +1,68 @@
+# Summary
+
+![pipeline](../../images/pipeline.png#only-light)
+![pipeline](../../images/pipeline-dark.png#only-dark)
+
+The Summary pipeline summarizes text. This pipeline runs a text2text model that abstractively creates a summary of the input text.
+
+## Example
+
+The following shows a simple example using this pipeline.
+
+```python
+from txtai.pipeline import Summary
+
+# Create and run pipeline
+summary = Summary()
+summary("Enter long, detailed text to summarize here")
+```
+
+See the link below for a more detailed example.
+
+| Notebook  | Description  |       |
+|:----------|:-------------|------:|
+| [Building abstractive text summaries](https://github.com/neuml/txtai/blob/master/examples/09_Building_abstractive_text_summaries.ipynb) | Run abstractive text summarization | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/09_Building_abstractive_text_summaries.ipynb) |
+
+## Configuration-driven example
+
+Pipelines are run with Python or configuration. Pipelines can be instantiated in [configuration](../../../api/configuration/#pipeline) using the lower case name of the pipeline. Configuration-driven pipelines are run with [workflows](../../../workflow/#configuration-driven-example) or the [API](../../../api#local-instance).
+
+### config.yml
+```yaml
+# Create pipeline using lower case class name
+summary:
+
+# Run pipeline with workflow
+workflow:
+  summary:
+    tasks:
+      - action: summary
+```
+
+### Run with Workflows
+
+```python
+from txtai.app import Application
+
+# Create and run pipeline with workflow
+app = Application("config.yml")
+list(app.workflow("summary", ["Enter long, detailed text to summarize here"]))
+```
+
+### Run with API
+
+```bash
+CONFIG=config.yml uvicorn "txtai.api:app" &
+
+curl \
+  -X POST "http://localhost:8000/workflow" \
+  -H "Content-Type: application/json" \
+  -d '{"name":"summary", "elements":["Enter long, detailed text to summarize here"]}'
+```
+
+## Methods
+
+Python documentation for the pipeline.
+
+### ::: txtai.pipeline.Summary.__init__
+### ::: txtai.pipeline.Summary.__call__

docs/pipeline/text/translation.md

@@ -0,0 +1,69 @@
+# Translation
+
+![pipeline](../../images/pipeline.png#only-light)
+![pipeline](../../images/pipeline-dark.png#only-dark)
+
+The Translation pipeline translates text between languages. It supports over 100+ languages. Automatic source language detection is built-in. This pipeline detects the language of each input text row, loads a model for the source-target combination and translates text to the target language.
+
+## Example
+
+The following shows a simple example using this pipeline.
+
+```python
+from txtai.pipeline import Translation
+
+# Create and run pipeline
+translate = Translation()
+translate("This is a test translation into Spanish", "es")
+```
+
+See the link below for a more detailed example.
+
+| Notebook  | Description  |       |
+|:----------|:-------------|------:|
+| [Translate text between languages](https://github.com/neuml/txtai/blob/master/examples/12_Translate_text_between_languages.ipynb) | Streamline machine translation and language detection | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/12_Translate_text_between_languages.ipynb) |
+
+## Configuration-driven example
+
+Pipelines are run with Python or configuration. Pipelines can be instantiated in [configuration](../../../api/configuration/#pipeline) using the lower case name of the pipeline. Configuration-driven pipelines are run with [workflows](../../../workflow/#configuration-driven-example) or the [API](../../../api#local-instance).
+
+### config.yml
+```yaml
+# Create pipeline using lower case class name
+translation:
+
+# Run pipeline with workflow
+workflow:
+  translate:
+    tasks:
+      - action: translation
+        args: ["es"]
+```
+
+### Run with Workflows
+
+```python
+from txtai.app import Application
+
+# Create and run pipeline with workflow
+app = Application("config.yml")
+list(app.workflow("translate", ["This is a test translation into Spanish"]))
+```
+
+### Run with API
+
+```bash
+CONFIG=config.yml uvicorn "txtai.api:app" &
+
+curl \
+  -X POST "http://localhost:8000/workflow" \
+  -H "Content-Type: application/json" \
+  -d '{"name":"translate", "elements":["This is a test translation into Spanish"]}'
+```
+
+## Methods 
+
+Python documentation for the pipeline.
+
+### ::: txtai.pipeline.Translation.__init__
+### ::: txtai.pipeline.Translation.__call__

docs/pipeline/train/hfonnx.md

@@ -0,0 +1,38 @@
+# HFOnnx
+
+![pipeline](../../images/pipeline.png#only-light)
+![pipeline](../../images/pipeline-dark.png#only-dark)
+
+Exports a Hugging Face Transformer model to ONNX. Currently, this works best with classification/pooling/qa models. Work is ongoing for sequence to
+sequence models (summarization, transcription, translation).
+
+## Example
+
+The following shows a simple example using this pipeline.
+
+```python
+from txtai.pipeline import HFOnnx, Labels
+
+# Model path
+path = "distilbert-base-uncased-finetuned-sst-2-english"
+
+# Export model to ONNX
+onnx = HFOnnx()
+model = onnx(path, "text-classification", "model.onnx", True)
+
+# Run inference and validate
+labels = Labels((model, path), dynamic=False)
+labels("I am happy")
+```
+
+See the link below for a more detailed example.
+
+| Notebook  | Description  |       |
+|:----------|:-------------|------:|
+| [Export and run models with ONNX](https://github.com/neuml/txtai/blob/master/examples/18_Export_and_run_models_with_ONNX.ipynb) | Export models with ONNX, run natively in JavaScript, Java and Rust | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/18_Export_and_run_models_with_ONNX.ipynb) |
+
+## Methods 
+
+Python documentation for the pipeline.
+
+### ::: txtai.pipeline.HFOnnx.__call__

docs/pipeline/train/mlonnx.md

@@ -0,0 +1,20 @@
+# MLOnnx
+
+![pipeline](../../images/pipeline.png#only-light)
+![pipeline](../../images/pipeline-dark.png#only-dark)
+
+Exports a traditional machine learning model (i.e. scikit-learn) to ONNX.
+
+## Example
+
+See the link below for a detailed example.
+
+| Notebook  | Description  |       |
+|:----------|:-------------|------:|
+| [Export and run other machine learning models](https://github.com/neuml/txtai/blob/master/examples/21_Export_and_run_other_machine_learning_models.ipynb) | Export and run models from scikit-learn, PyTorch and more | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/21_Export_and_run_other_machine_learning_models.ipynb) |
+
+## Methods 
+
+Python documentation for the pipeline.
+
+### ::: txtai.pipeline.MLOnnx.__call__

docs/pipeline/train/trainer.md

@@ -0,0 +1,52 @@
+# HFTrainer
+
+![pipeline](../../images/pipeline.png#only-light)
+![pipeline](../../images/pipeline-dark.png#only-dark)
+
+Trains a new Hugging Face Transformer model using the Trainer framework.
+
+## Example
+
+The following shows a simple example using this pipeline.
+
+```python
+import pandas as pd
+
+from datasets import load_dataset
+
+from txtai.pipeline import HFTrainer
+
+trainer = HFTrainer()
+
+# Pandas DataFrame
+df = pd.read_csv("training.csv")
+model, tokenizer = trainer("bert-base-uncased", df)
+
+# Hugging Face dataset
+ds = load_dataset("glue", "sst2")
+model, tokenizer = trainer("bert-base-uncased", ds["train"], columns=("sentence", "label"))
+
+# List of dicts
+dt = [{"text": "sentence 1", "label": 0}, {"text": "sentence 2", "label": 1}]]
+model, tokenizer = trainer("bert-base-uncased", dt)
+
+# Support additional TrainingArguments
+model, tokenizer = trainer("bert-base-uncased", dt, 
+                            learning_rate=3e-5, num_train_epochs=5)
+```
+
+All [TrainingArguments](https://huggingface.co/transformers/main_classes/trainer.html#transformers.TrainingArguments) are supported as function arguments to the trainer call.
+
+See the links below for more detailed examples.
+
+| Notebook  | Description  |       |
+|:----------|:-------------|------:|
+| [Train a text labeler](https://github.com/neuml/txtai/blob/master/examples/16_Train_a_text_labeler.ipynb) | Build text sequence classification models | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/16_Train_a_text_labeler.ipynb) |
+| [Train without labels](https://github.com/neuml/txtai/blob/master/examples/17_Train_without_labels.ipynb) | Use zero-shot classifiers to train new models | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/17_Train_without_labels.ipynb) |
+| [Train a QA model](https://github.com/neuml/txtai/blob/master/examples/19_Train_a_QA_model.ipynb) | Build and fine-tune question-answering models | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/neuml/txtai/blob/master/examples/19_Train_a_QA_model.ipynb) |
+
+## Methods 
+
+Python documentation for the pipeline.
+
+### ::: txtai.pipeline.HFTrainer.__call__