deepset-ai/haystack
1
0
Fork 0
mirror of https://github.com/deepset-ai/haystack.git synced 2026-10-12 02:12:50 +00:00
Code Issues Packages Projects Releases Wiki Activity

chore: Merge main into v3 (#11558)

Browse Source 
Signed-off-by: dependabot[bot] <[email protected]>
Signed-off-by: NIK-TIGER-BILL <[email protected]>
Co-authored-by: Pragnyan Ramtha <[email protected]>
Co-authored-by: Julian Risch <[email protected]>
Co-authored-by: Claude Sonnet 4.6 <[email protected]>
Co-authored-by: Aarkin Karnik <[email protected]>
Co-authored-by: Haystack Bot <[email protected]>
Co-authored-by: Stefano Fiorucci <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: davidsbatista <[email protected]>
Co-authored-by: bogdankostic <[email protected]>
Co-authored-by: bogdankostic <[email protected]>
Co-authored-by: Sachin Yadav <[email protected]>
Co-authored-by: Copilot Autofix powered by AI <[email protected]>
Co-authored-by: anakin87 <[email protected]>
Co-authored-by: Abdulhamid Onawole <[email protected]>
Co-authored-by: David S. Batista <[email protected]>
Co-authored-by: Aditya Raut <[email protected]>
Co-authored-by: Nhat Huy Vu <[email protected]>
Co-authored-by: julian-risch <[email protected]>
Co-authored-by: kota <[email protected]>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Co-authored-by: devteamaegis <[email protected]>
Co-authored-by: Ritwij Aryan Parmar <[email protected]>
Co-authored-by: Srivatsa Kamballa <[email protected]>
Co-authored-by: vidigoat <[email protected]>
Co-authored-by: Syed Shahmeer Ali <[email protected]>
Co-authored-by: Mohamed Arbi <[email protected]>
Co-authored-by: Zamuldinov Nikita <[email protected]>
Co-authored-by: Bilge Yücel <[email protected]>
Co-authored-by: jdoughty04 <[email protected]>
Co-authored-by: Govindh Kishore <[email protected]>
This commit is contained in:
Sebastian Husch Lee
2026-06-09 10:53:22 +02:00
committed by GitHub
co-authored by Pragnyan Ramtha Julian Risch Claude Sonnet 4.6 Aarkin Karnik Haystack Bot Stefano Fiorucci dependabot[bot] davidsbatista bogdankostic bogdankostic Sachin Yadav Copilot Autofix powered by AI anakin87 Abdulhamid Onawole David S. Batista Aditya Raut Nhat Huy Vu julian-risch kota Copilot Autofix powered by AI devteamaegis Ritwij Aryan Parmar Srivatsa Kamballa vidigoat Syed Shahmeer Ali Mohamed Arbi Zamuldinov Nikita Bilge Yücel jdoughty04 Govindh Kishore
parent 8d7d179828
commit 515980d50b
332 changed files with 43510 additions and 5662 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/workflows/branch_off.yml
+1 -1
View File
@@ -20,7 +20,7 @@ jobs:
steps:
- name: Checkout this repo
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
with:
ref: main
# Persist the bot token so the branch/tag pushes below authenticate as
.github/workflows/check_api_ref.yml
+1 -1
View File
@@ -13,7 +13,7 @@ jobs:
test-api-reference-build:
runs-on: ubuntu-slim
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
with:
fetch-depth: 0
.github/workflows/claude-code-review.yml
+2 -2
View File
@@ -29,13 +29,13 @@ jobs:
id-token: write
steps:
- name: Checkout repository
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
with:
fetch-depth: 1
- name: Run Claude Code Review
id: claude-review
uses: anthropics/claude-code-action@787c5a0ce96a9a6cfb050ea0c8f4c05f2447c251 # v1.0.133
uses: anthropics/claude-code-action@fbda2eb1bdc90d319b8d853f5deb53bca199a7c1 # v1.0.140
with:
anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY_PR_REVIEWS_PUBLIC_REPOS }}
plugin_marketplaces: 'https://github.com/anthropics/claude-code.git'
.github/workflows/claude.yml
+2 -2
View File
@@ -50,13 +50,13 @@ jobs:
actions: read
steps:
- name: Checkout repository
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
with:
fetch-depth: 1
- name: Run Claude Code
id: claude
uses: anthropics/claude-code-action@787c5a0ce96a9a6cfb050ea0c8f4c05f2447c251 # v1.0.133
uses: anthropics/claude-code-action@fbda2eb1bdc90d319b8d853f5deb53bca199a7c1 # v1.0.140
with:
anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY_PR_REVIEWS_PUBLIC_REPOS }}
additional_permissions: |
.github/workflows/docker_release.yml
+1 -1
View File
@@ -28,7 +28,7 @@ jobs:
steps:
- name: Checkout
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
- name: Set up QEMU
uses: docker/setup-qemu-action@06116385d9baf250c9f4dcb4858b16962ea869c3 # v4.1.0
.github/workflows/docs-website-test-docs-snippets.yml
+1 -1
View File
@@ -33,7 +33,7 @@ jobs:
steps:
- name: Checkout
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
- name: Setup Python
uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
.github/workflows/docs_search_sync.yml
+1 -1
View File
@@ -19,7 +19,7 @@ jobs:
steps:
- name: Checkout Haystack repo
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
- name: Install Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
.github/workflows/docstring_labeler.yml
+2 -2
View File
@@ -23,7 +23,7 @@ jobs:
steps:
- name: Checkout base commit
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
with:
ref: ${{ github.base_ref }}
@@ -39,7 +39,7 @@ jobs:
echo "checksum=$CHECKSUM" >> "$GITHUB_OUTPUT"
- name: Checkout HEAD commit
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
with:
ref: ${{ github.event.pull_request.head.sha }}
.github/workflows/docusaurus_sync.yml
+1 -1
View File
@@ -25,7 +25,7 @@ jobs:
steps:
- name: Checkout Haystack repo
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
- name: Set up Python
uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
.github/workflows/e2e.yml
+1 -1
View File
@@ -29,7 +29,7 @@ jobs:
timeout-minutes: 60
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
with:
.github/workflows/github_release.yml
+1 -1
View File
@@ -19,7 +19,7 @@ jobs:
steps:
- name: Checkout
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
with:
fetch-tags: true
fetch-depth: 0 # slow but needed by reno
.github/workflows/license_compliance.yml
+1 -1
View File
@@ -23,7 +23,7 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Checkout the code
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
- name: Setup Python
uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
.github/workflows/nightly_testpypi_release.yml
+1 -1
View File
@@ -21,7 +21,7 @@ jobs:
# Always build from main for consistency (scheduled and manual runs)
steps:
- name: Checkout main
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
with:
ref: main
fetch-depth: 1
.github/workflows/promote_unstable_docs.yml
+1 -1
View File
@@ -19,7 +19,7 @@ jobs:
runs-on: ubuntu-slim
steps:
- name: Checkout this repo
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
# use VERSION.txt file from main branch
with:
ref: main
.github/workflows/push_release_notes_to_website.yml
+1 -1
View File
@@ -21,7 +21,7 @@ jobs:
runs-on: ubuntu-slim
steps:
- name: Checkout Haystack home repository
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
with:
repository: deepset-ai/haystack-home
.github/workflows/pypi_release.yml
+1 -1
View File
@@ -22,7 +22,7 @@ jobs:
steps:
- name: Checkout
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
- name: Install Hatch
run: |
.github/workflows/release.yml
+7 -7
View File
@@ -50,7 +50,7 @@ jobs:
is_first_rc: ${{ steps.parse-validate.outputs.is_first_rc }}
steps:
- name: Checkout repository
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
with:
fetch-depth: 0 # needed to fetch tags and branches
- name: Parse and validate version
@@ -74,7 +74,7 @@ jobs:
contents: write
steps:
- name: Checkout repository
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
with:
fetch-depth: 0 # needed to fetch tags and branches
# use this token so the created tag triggers workflows (does not happen with the default github.token)
@@ -111,7 +111,7 @@ jobs:
VERSION: ${{ needs.parse-validate-version.outputs.version }}
steps:
- name: Checkout repository
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
with:
fetch-depth: 0 # needed to fetch tags and branches
@@ -151,7 +151,7 @@ jobs:
VERSION: ${{ needs.parse-validate-version.outputs.version }}
steps:
- name: Checkout dc-pipeline-templates
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
with:
repository: deepset-ai/dc-pipeline-templates
token: ${{ secrets.HAYSTACK_BOT_TOKEN }}
@@ -182,14 +182,14 @@ jobs:
VERSION: ${{ needs.parse-validate-version.outputs.version }}
steps:
- name: Checkout haystack (for utils)
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
with:
path: haystack
sparse-checkout: .github/utils/update_haystack_dc_custom_nodes.py
sparse-checkout-cone-mode: false
- name: Checkout deepset-cloud-custom-nodes
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
with:
repository: deepset-ai/deepset-cloud-custom-nodes
token: ${{ secrets.HAYSTACK_BOT_TOKEN }}
@@ -273,7 +273,7 @@ jobs:
runs-on: ubuntu-slim
steps:
- name: Checkout repository
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
- name: Prepare release notification
env:
.github/workflows/release_notes.yml
+1 -1
View File
@@ -25,7 +25,7 @@ jobs:
PYTHON_VERSION: "3.10"
steps:
- name: Checkout
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
with:
# With the default value of 1, there are corner cases where tj-actions/changed-files
# fails with a `no merge base` error
.github/workflows/slow.yml
+2 -2
View File
@@ -49,7 +49,7 @@ jobs:
outputs:
changes: ${{ steps.changes.outputs.changes }}
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
- name: Check for changed code
id: changes
uses: dorny/paths-filter@fbd0ab8f3e69293af611ebaee6363fc25e6d187d # v4.0.1
@@ -123,7 +123,7 @@ jobs:
install_cmd: "echo 'No additional dependencies needed'"
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
with:
.github/workflows/tests.yml
+8 -8
View File
@@ -50,7 +50,7 @@ jobs:
outputs:
changes: ${{ steps.changes.outputs.changes }}
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
- name: Check for changed code
id: changes
uses: dorny/paths-filter@fbd0ab8f3e69293af611ebaee6363fc25e6d187d # v4.0.1
@@ -72,7 +72,7 @@ jobs:
(needs.check-if-changed.outputs.changes == 'true')
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
with:
@@ -93,7 +93,7 @@ jobs:
needs: format
runs-on: ubuntu-slim
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
with:
@@ -123,7 +123,7 @@ jobs:
- macos-latest
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
with:
@@ -171,7 +171,7 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
with:
# With the default value of 1, there are corner cases where tj-actions/changed-files
# fails with a `no merge base` error
@@ -212,7 +212,7 @@ jobs:
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
with:
@@ -238,7 +238,7 @@ jobs:
HAYSTACK_MPS_ENABLED: false
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
with:
@@ -270,7 +270,7 @@ jobs:
HAYSTACK_XPU_ENABLED: false
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
with:
.github/workflows/v3_pypi_release.yml
+1 -1
View File
@@ -17,7 +17,7 @@ jobs:
id-token: write
steps:
- name: Checkout v3
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
with:
ref: v3
fetch-depth: 1
.github/workflows/workflows_linting.yml
+1 -1
View File
@@ -13,7 +13,7 @@ jobs:
runs-on: ubuntu-slim
steps:
- name: Checkout
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
- uses: actions/setup-go@4a3601121dd01d1626a1e23e37211e3254c1c06c # v6.4.0
with:
README.md
+1 -1
View File
@@ -5,7 +5,7 @@
| ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| CI/CD | [![Tests](https://github.com/deepset-ai/haystack/actions/workflows/tests.yml/badge.svg)](https://github.com/deepset-ai/haystack/actions/workflows/tests.yml) [![types - Mypy](https://img.shields.io/badge/types-Mypy-blue.svg)](https://github.com/python/mypy) [![Coverage badge](https://raw.githubusercontent.com/deepset-ai/haystack/python-coverage-comment-action-data/badge.svg)](https://htmlpreview.github.io/?https://github.com/deepset-ai/haystack/blob/python-coverage-comment-action-data/htmlcov/index.html) [![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff) |
| Docs | [![Website](https://img.shields.io/website?label=documentation&up_message=online&url=https%3A%2F%2Fdocs.haystack.deepset.ai)](https://docs.haystack.deepset.ai) |
| Package | [![PyPI](https://img.shields.io/pypi/v/haystack-ai)](https://pypi.org/project/haystack-ai/) ![PyPI - Downloads](https://img.shields.io/pypi/dm/haystack-ai?color=blue&logo=pypi&logoColor=gold) ![PyPI - Python Version](https://img.shields.io/pypi/pyversions/haystack-ai?logo=python&logoColor=gold) [![Conda Version](https://img.shields.io/conda/vn/conda-forge/haystack-ai.svg)](https://anaconda.org/conda-forge/haystack-ai) [![GitHub](https://img.shields.io/github/license/deepset-ai/haystack?color=blue)](LICENSE) [![License Compliance](https://github.com/deepset-ai/haystack/actions/workflows/license_compliance.yml/badge.svg)](https://github.com/deepset-ai/haystack/actions/workflows/license_compliance.yml) [![OpenSSF Best Practices](https://www.bestpractices.dev/projects/13067/badge)](https://www.bestpractices.dev/projects/13067)|
| Package | [![PyPI](https://img.shields.io/pypi/v/haystack-ai)](https://pypi.org/project/haystack-ai/) ![PyPI - Downloads](https://img.shields.io/pypi/dm/haystack-ai?color=blue&logo=pypi&logoColor=gold) ![PyPI - Python Version](https://img.shields.io/pypi/pyversions/haystack-ai?logo=python&logoColor=gold) [![Conda Version](https://img.shields.io/conda/vn/conda-forge/haystack-ai.svg)](https://anaconda.org/conda-forge/haystack-ai) [![GitHub](https://img.shields.io/github/license/deepset-ai/haystack?color=blue)](LICENSE) [![License Compliance](https://github.com/deepset-ai/haystack/actions/workflows/license_compliance.yml/badge.svg)](https://github.com/deepset-ai/haystack/actions/workflows/license_compliance.yml) [![HVTrust](https://hvtracker.net/badge/haystack.svg)](https://hvtracker.net/agents/haystack/) [![Evidence Grade](https://hvtracker.net/badge/haystack-grade.svg)](https://hvtracker.net/agents/haystack/) [![OpenSSF Best Practices](https://www.bestpractices.dev/projects/13067/badge)](https://www.bestpractices.dev/projects/13067)|
| Meta | [![Discord](https://img.shields.io/discord/993534733298450452?logo=discord)](https://discord.com/invite/qZxjM4bAHU) [![Twitter Follow](https://img.shields.io/twitter/follow/haystack_ai)](https://twitter.com/haystack_ai) |
</div>
docs-website/docs/concepts/data-classes.mdx
+7 -1
View File
@@ -9,7 +9,7 @@ description: "In Haystack, there are a handful of core classes that are regularl
In Haystack, there are a handful of core classes that are regularly used in many different places. These are classes that carry data through the system and you are likely to interact with these as either the input or output of your pipeline.
Haystack uses data classes to help components communicate with each other in a simple and modular way. By doing this, data flows seamlessly through the Haystack pipelines. This page goes over the available data classes in Haystack: ByteStream, Answer (along with its variants ExtractedAnswer and GeneratedAnswer), ChatMessage, Document, and StreamingChunk, explaining how they contribute to the Haystack ecosystem.
Haystack uses data classes to help components communicate with each other in a simple and modular way. By doing this, data flows seamlessly through the Haystack pipelines. This page goes over the available data classes in Haystack: ByteStream, Answer (along with its variants ExtractedAnswer and GeneratedAnswer), ChatMessage, FileContent, Document, and StreamingChunk, explaining how they contribute to the Haystack ecosystem.
You can check out the detailed parameters in our [Data Classes](/reference/data-classes-api) API reference.
@@ -120,6 +120,12 @@ image = ByteStream.from_file_path("dog.jpg")
Read the detailed documentation for the `ChatMessage` data class on a dedicated [ChatMessage](data-classes/chatmessage.mdx) page.
### FileContent
`FileContent` represents a file payload that can be attached to a `ChatMessage`, including base64 data, MIME type, filename, and provider-specific metadata.
Read the detailed documentation for the `FileContent` data class on a dedicated [FileContent](data-classes/filecontent.mdx) page.
### Document
#### Overview
docs-website/docs/concepts/data-classes/filecontent.mdx
+119
View File
@@ -0,0 +1,119 @@
---
title: "FileContent"
id: filecontent
slug: "/filecontent"
description: "`FileContent` represents file payloads in chat messages, including base64 data, MIME type, filename, and provider-specific metadata."
---
# FileContent
`FileContent` represents a file payload that can be attached to a [`ChatMessage`](chatmessage.mdx). Use it when a chat model accepts file inputs, such as PDFs or other documents, together with the user's text prompt.
If you need the full list of parameters and methods, see the [`FileContent` API reference](/reference/data-classes-api#filecontent).
## Attributes
```python
@dataclass
class FileContent:
base64_data: str
mime_type: str | None = None
filename: str | None = None
extra: dict[str, Any] = field(default_factory=dict)
validation: bool = True
```
- `base64_data` stores the file content as a base64-encoded string.
- `mime_type` identifies the file type, for example `application/pdf`. Providing it explicitly is recommended because many model providers require it.
- `filename` is optional, but some providers use it when processing uploaded files.
- `extra` can store provider-specific metadata. Values should be JSON serializable.
- `validation` checks that `base64_data` is valid and tries to infer the MIME type when one is not provided.
## Create from a file path
Use `from_file_path` to read a local file, base64-encode it, infer the MIME type from the path, and populate the filename.
```python
from haystack.dataclasses import ChatMessage, FileContent
file_content = FileContent.from_file_path("data/attention-is-all-you-need.pdf")
message = ChatMessage.from_user(
content_parts=[
file_content,
"Summarize the key ideas in this paper.",
]
)
```
Pass `filename` or `extra` when a provider expects a specific filename or provider-specific options:
```python
file_content = FileContent.from_file_path(
"data/report.pdf",
filename="quarterly-report.pdf",
extra={"source": "finance"},
)
```
## Create from a URL
Use `from_url` to download a file and convert it into a `FileContent` instance.
```python
from haystack.dataclasses import FileContent
file_content = FileContent.from_url(
"https://example.com/reports/quarterly-report.pdf",
timeout=30,
)
```
If no filename is provided, Haystack uses the final path segment of the URL.
## Create from base64 data
If you already have file bytes, encode them and pass the MIME type explicitly.
```python
import base64
from pathlib import Path
from haystack.dataclasses import FileContent
data = Path("data/manual.pdf").read_bytes()
file_content = FileContent(
base64_data=base64.b64encode(data).decode("utf-8"),
mime_type="application/pdf",
filename="manual.pdf",
)
```
Set `validation=False` only when the base64 data and MIME type are already trusted and you want to skip validation.
## Inspect files in a ChatMessage
After adding `FileContent` to a `ChatMessage`, use the `file` and `files` properties to access file payloads.
```python
from haystack.dataclasses import ChatMessage, FileContent
file_content = FileContent.from_file_path("data/invoice.pdf")
message = ChatMessage.from_user(content_parts=[file_content, "Extract the invoice total."])
print(message.file)
print(message.files)
```
`message.file` returns the first file payload, or `None` if there are no files. `message.files` returns all file payloads.
## Serialization
Use `to_dict` and `from_dict` to serialize and restore file content.
```python
payload = file_content.to_dict()
restored = FileContent.from_dict(payload)
```
For tracing, Haystack replaces the full base64 payload with a placeholder so large files are not sent to the tracing backend.
docs-website/docs/concepts/document-store/choosing-a-document-store.mdx
+2
View File
@@ -79,7 +79,9 @@ Pure vector databases, also known as just “vector databases”, offer efficien
This category is relatively small but growing fast and includes well-known relational databases where vector capabilities were added through plugins or extensions. They are not as performant as the previous categories, but the main advantage of these databases is the opportunity to easily combine vectors with structured data, having a one-stop data shop for your application. You should pick a vector-capable SQL database when the performance trade-off is paid off by the lower cost of maintaining a single database instance for your application or when the structured data plays a more fundamental role in your business logic, with vectors being more of a nice-to-have.
- [Oracle](../../document-stores/oracledocumentstore.mdx)
- [Pgvector](../../document-stores/pgvectordocumentstore.mdx)
- [Supabase](../../document-stores/supabasedocumentstore.mdx)
#### Vector-capable NoSQL databases
docs-website/docs/document-stores/oracledocumentstore.mdx
+191
View File
@@ -0,0 +1,191 @@
---
title: "OracleDocumentStore"
id: oracledocumentstore
slug: "/oracledocumentstore"
description: "Use Oracle AI Vector Search as a document store in Haystack, with vector similarity and keyword search powered by Oracle Database 23ai."
---
# OracleDocumentStore
<div className="key-value-table">
| | |
| --- | --- |
| API reference | [Oracle](/reference/integrations-oracle) |
| GitHub link | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/oracle |
</div>
`OracleDocumentStore` is a Document Store backed by [Oracle AI Vector Search](https://www.oracle.com/database/ai-vector-search/), available in Oracle Database 23ai and later.
It stores documents alongside dense vector embeddings in a native `VECTOR` column, and supports both vector similarity search and keyword search via an automatically managed DBMS_SEARCH index.
## Installation
```shell
pip install oracle-haystack
```
## Connection
`OracleDocumentStore` connects to Oracle using the `OracleConnectionConfig` dataclass, which supports two connection modes:
- **Thin mode** (default): connects directly over TCP. No Oracle Instant Client required.
- **Thick mode**: activated automatically when `wallet_location` is provided. Used for Oracle Autonomous Database (ADB-S) connections.
Set the connection parameters as environment variables:
```shell
export ORACLE_USER="haystack"
export ORACLE_PASSWORD="secret"
export ORACLE_DSN="localhost:1521/freepdb1"
```
## Initialization
```python
from haystack.utils import Secret
from haystack_integrations.document_stores.oracle import (
OracleDocumentStore,
OracleConnectionConfig,
)
document_store = OracleDocumentStore(
connection_config=OracleConnectionConfig(
user=Secret.from_env_var("ORACLE_USER"),
password=Secret.from_env_var("ORACLE_PASSWORD"),
dsn=Secret.from_env_var("ORACLE_DSN"),
),
embedding_dim=768,
)
```
To learn more about the initialization parameters, see the [API docs](/reference/integrations-oracle#oracledocumentstore).
### Connecting to Oracle Autonomous Database
For Oracle Autonomous Database (ADB-S), provide a wallet for authentication. The store automatically activates thick mode when `wallet_location` is set:
```python
document_store = OracleDocumentStore(
connection_config=OracleConnectionConfig(
user=Secret.from_env_var("ORACLE_USER"),
password=Secret.from_env_var("ORACLE_PASSWORD"),
dsn=Secret.from_env_var("ORACLE_DSN"),
wallet_location="/path/to/wallet",
wallet_password=Secret.from_env_var("WALLET_PASSWORD"),
),
embedding_dim=1536,
)
```
### HNSW Vector Index
By default, the store performs exact vector search. To enable approximate nearest-neighbor search (faster on large datasets), create an HNSW index:
```python
document_store = OracleDocumentStore(
connection_config=OracleConnectionConfig(
user=Secret.from_env_var("ORACLE_USER"),
password=Secret.from_env_var("ORACLE_PASSWORD"),
dsn=Secret.from_env_var("ORACLE_DSN"),
),
embedding_dim=768,
distance_metric="COSINE",
create_index=True, # creates the HNSW index on startup
hnsw_neighbors=32,
hnsw_ef_construction=200,
hnsw_accuracy=95,
)
```
## Supported Retrievers
- [`OracleEmbeddingRetriever`](../pipeline-components/retrievers/oracleembeddingretriever.mdx): Retrieves documents from `OracleDocumentStore` based on vector similarity to a query embedding.
- [`OracleKeywordRetriever`](../pipeline-components/retrievers/oraclekeywordretriever.mdx): Retrieves documents matching a keyword query using Oracle's DBMS_SEARCH full-text index.
## Example: RAG pipeline
```python
from haystack import Document, Pipeline
from haystack.document_stores.types import DuplicatePolicy
from haystack.components.embedders import (
SentenceTransformersDocumentEmbedder,
SentenceTransformersTextEmbedder,
)
from haystack.components.builders import ChatPromptBuilder
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.dataclasses import ChatMessage
from haystack.utils import Secret
from haystack_integrations.document_stores.oracle import (
OracleDocumentStore,
OracleConnectionConfig,
)
from haystack_integrations.components.retrievers.oracle import OracleEmbeddingRetriever
document_store = OracleDocumentStore(
connection_config=OracleConnectionConfig(
user=Secret.from_env_var("ORACLE_USER"),
password=Secret.from_env_var("ORACLE_PASSWORD"),
dsn=Secret.from_env_var("ORACLE_DSN"),
),
embedding_dim=768,
)
# Index documents
documents = [
Document(content="There are over 7,000 languages spoken around the world today."),
Document(
content="Elephants have been observed to behave in a way that indicates a high level of self-awareness.",
),
Document(
content="In certain places, you can witness the phenomenon of bioluminescent waves.",
),
]
doc_embedder = SentenceTransformersDocumentEmbedder(
model="sentence-transformers/all-MiniLM-L6-v2",
)
doc_embedder.warm_up()
embedded_docs = doc_embedder.run(documents)["documents"]
document_store.write_documents(embedded_docs, policy=DuplicatePolicy.OVERWRITE)
# Build a RAG pipeline
template = [
ChatMessage.from_user(
"""
Given the following context, answer the question.
Context: {% for doc in documents %}{{ doc.content }}{% endfor %}
Question: {{ query }}
""",
),
]
pipeline = Pipeline()
pipeline.add_component(
"embedder",
SentenceTransformersTextEmbedder(model="sentence-transformers/all-MiniLM-L6-v2"),
)
pipeline.add_component(
"retriever",
OracleEmbeddingRetriever(document_store=document_store, top_k=3),
)
pipeline.add_component("prompt_builder", ChatPromptBuilder(template=template))
pipeline.add_component(
"llm",
OpenAIChatGenerator(api_key=Secret.from_env_var("OPENAI_API_KEY")),
)
pipeline.connect("embedder.embedding", "retriever.query_embedding")
pipeline.connect("retriever.documents", "prompt_builder.documents")
pipeline.connect("prompt_builder.prompt", "llm.messages")
result = pipeline.run(
{
"embedder": {"text": "How many languages are there?"},
"prompt_builder": {"query": "How many languages are there?"},
},
)
print(result["llm"]["replies"][0].text)
```
docs-website/docs/document-stores/qdrant-document-store.mdx
+2 -2
View File
@@ -56,7 +56,7 @@ print(document_store.count_documents())
When you create a `QdrantDocumentStore` instance, Haystack takes care of setting up the collection. In general, you cannot use a Qdrant collection created without Haystack with Haystack. If you want to migrate your existing collection, see the sample script at https://github.com/deepset-ai/haystack-core-integrations/blob/main/integrations/qdrant/src/haystack_integrations/document_stores/qdrant/migrate_to_sparse.py.
:::
You can also connect directly to [Qdrant Cloud](https://cloud.qdrant.io/login) directly. Once you have your API key and your cluster URL from the Qdrant dashboard, you can connect like this:
You can also connect directly to [Qdrant Cloud](https://cloud.qdrant.io/login). Once you have your API key and your cluster URL from the Qdrant dashboard, you can connect like this:
```python
from haystack.dataclasses.document import Document
@@ -66,7 +66,7 @@ from haystack.utils import Secret
document_store = QdrantDocumentStore(
url="https://XXXXXXXXX.us-east4-0.gcp.cloud.qdrant.io:6333",
index="your_index_name",
embedding_dim=1024, # based on the embedding model
embedding_dim=5, # based on the embedding model
recreate_index=True, # enable only to recreate the index and not connect to the existing one
api_key=Secret.from_token("YOUR_TOKEN"),
)
docs-website/docs/document-stores/supabasedocumentstore.mdx
+182
View File
@@ -0,0 +1,182 @@
---
title: "SupabaseDocumentStore"
id: supabasedocumentstore
slug: "/supabasedocumentstore"
description: "Use Supabase as a document store in Haystack, with vector search (pgvector) or full-text search (PGroonga)."
---
# SupabaseDocumentStore
<div className="key-value-table">
| | |
| --- | --- |
| API reference | [Supabase](/reference/integrations-supabase) |
| GitHub link | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/supabase/ |
</div>
[Supabase](https://supabase.com/) is an open-source backend platform built on PostgreSQL. The Supabase integration for Haystack provides two document stores:
- **`SupabasePgvectorDocumentStore`** — vector similarity search using the [pgvector](https://github.com/pgvector/pgvector) PostgreSQL extension, which comes pre-installed on Supabase.
- **`SupabaseGroongaDocumentStore`** — multilingual full-text search using the [PGroonga](https://pgroonga.github.io/) PostgreSQL extension. No embeddings required.
## Installation
```shell
pip install supabase-haystack
```
## SupabasePgvectorDocumentStore
`SupabasePgvectorDocumentStore` is a thin wrapper around [`PgvectorDocumentStore`](./pgvectordocumentstore.mdx) with Supabase-specific defaults:
- Reads the connection string from the `SUPABASE_DB_URL` environment variable.
- Defaults `create_extension` to `False` since pgvector is pre-installed on Supabase.
### Connection
Set the `SUPABASE_DB_URL` environment variable with your Supabase database connection string.
:::tip[Use session mode (port 5432)]
Supabase offers two pooler ports: transaction mode (port 6543) and session mode (port 5432). For best compatibility with pgvector operations, use session mode or a direct connection.
:::
```shell
export SUPABASE_DB_URL="postgresql://postgres.[project-ref]:[password]@aws-0-[region].pooler.supabase.com:5432/postgres"
```
### Initialization
```python
from haystack_integrations.document_stores.supabase import SupabasePgvectorDocumentStore
document_store = SupabasePgvectorDocumentStore(
embedding_dimension=768,
vector_function="cosine_similarity",
recreate_table=True,
)
```
To learn more about the initialization parameters, see the [API docs](/reference/integrations-supabase#supabasepgvectordocumentstore).
### Supported Retrievers
- [`SupabasePgvectorEmbeddingRetriever`](/reference/integrations-supabase#supabasepgvectorembeddingretriever): Fetches documents from the store based on a query embedding.
- [`SupabasePgvectorKeywordRetriever`](/reference/integrations-supabase#supabasepgvectorkeywordretriever): Fetches documents matching a keyword query using PostgreSQL's `ts_rank_cd` ranking.
### Example: RAG pipeline
```python
from haystack import Document, Pipeline
from haystack.document_stores.types.policy import DuplicatePolicy
from haystack.components.embedders import (
SentenceTransformersTextEmbedder,
SentenceTransformersDocumentEmbedder,
)
from haystack.components.builders import ChatPromptBuilder
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.dataclasses import ChatMessage
from haystack.utils import Secret
from haystack_integrations.document_stores.supabase import SupabasePgvectorDocumentStore
from haystack_integrations.components.retrievers.supabase import (
SupabasePgvectorEmbeddingRetriever,
)
document_store = SupabasePgvectorDocumentStore(
embedding_dimension=768,
vector_function="cosine_similarity",
recreate_table=True,
)
# Index documents
documents = [
Document(content="There are over 7,000 languages spoken around the world today."),
Document(
content="Elephants have been observed to behave in a way that indicates a high level of self-awareness.",
),
Document(
content="In certain places, you can witness the phenomenon of bioluminescent waves.",
),
]
embedder = SentenceTransformersDocumentEmbedder()
documents_with_embeddings = embedder.run(documents)
document_store.write_documents(
documents_with_embeddings["documents"],
policy=DuplicatePolicy.OVERWRITE,
)
# Query pipeline
prompt_template = [
ChatMessage.from_system("Answer the question based on the provided context."),
ChatMessage.from_user(
"Query: {{query}}\nDocuments:\n{% for doc in documents %}{{ doc.content }}\n{% endfor %}\nAnswer:",
),
]
query_pipeline = Pipeline()
query_pipeline.add_component("text_embedder", SentenceTransformersTextEmbedder())
query_pipeline.add_component(
"retriever",
SupabasePgvectorEmbeddingRetriever(document_store=document_store),
)
query_pipeline.add_component(
"prompt_builder",
ChatPromptBuilder(
template=prompt_template,
required_variables=["query", "documents"],
),
)
query_pipeline.add_component("generator", OpenAIChatGenerator(model="gpt-4o"))
query_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")
query_pipeline.connect("retriever.documents", "prompt_builder.documents")
query_pipeline.connect("prompt_builder.prompt", "generator.messages")
result = query_pipeline.run(
{
"text_embedder": {"text": "How many languages are there?"},
"prompt_builder": {"query": "How many languages are there?"},
},
)
```
---
## SupabaseGroongaDocumentStore
`SupabaseGroongaDocumentStore` uses [PGroonga](https://pgroonga.github.io/), a PostgreSQL extension for fast, multilingual full-text search. Unlike the pgvector store, it works with plain text queries and requires no embeddings.
### Prerequisites
PGroonga must be enabled in your Supabase project. Run the following SQL in the Supabase SQL editor:
```sql
CREATE EXTENSION IF NOT EXISTS pgroonga;
```
You also need to create a SQL function that PGroonga uses for search. See the [integration README](https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/supabase/) for the required function definition.
### Initialization
```python
from haystack_integrations.document_stores.supabase import SupabaseGroongaDocumentStore
from haystack.utils import Secret
document_store = SupabaseGroongaDocumentStore(
supabase_url="https://<project-ref>.supabase.co",
supabase_key=Secret.from_env_var("SUPABASE_SERVICE_KEY"),
table_name="haystack_groonga_documents",
)
document_store.warm_up()
```
:::note
`warm_up()` must be called before using the store. It initializes the Supabase client and creates the table and PGroonga index if they don't exist.
:::
To learn more about the initialization parameters, see the [API docs](/reference/integrations-supabase).
### Supported Retrievers
- [`SupabaseGroongaBM25Retriever`](/reference/integrations-supabase): Retrieves documents using PGroonga full-text search. Works without embeddings and can be combined with `SupabasePgvectorEmbeddingRetriever` for hybrid search pipelines.
docs-website/docs/memory-stores/cogneememorystore.mdx
+124
View File
@@ -0,0 +1,124 @@
---
title: "CogneeMemoryStore"
id: cogneememorystore
slug: "/cogneememorystore"
description: "A persistent memory store backed by Cognee's knowledge graph API."
---
# CogneeMemoryStore
`CogneeMemoryStore` is a persistent memory store backed by Cognee's knowledge graph API. It is the shared data layer used by [`CogneeRetriever`](../pipeline-components/retrievers/cogneeretriever.mdx) and [`CogneeWriter`](../pipeline-components/writers/cogneewriter.mdx).
<div className="key-value-table">
| | |
| --- | --- |
| **Used by** | [`CogneeRetriever`](../pipeline-components/retrievers/cogneeretriever.mdx), [`CogneeWriter`](../pipeline-components/writers/cogneewriter.mdx) |
| **Optional init variables** | `search_type`, `top_k`, `dataset_name`, `session_id`, `self_improvement`, `timeout` |
| **API reference** | [Cognee](/reference/integrations-cognee#cogneememorystore) |
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/cognee |
| **Package name** | `cognee-haystack` |
</div>
## Overview
`CogneeMemoryStore` wraps Cognee's V2 memory API:
- `add_memories` → `cognee.remember`
- `search_memories` → `cognee.recall`
- `improve` → `cognee.improve`
- `delete_all_memories` → `cognee.forget`
Cognee supports two memory tiers. Set `session_id` to use the **session cache** — fast writes with no LLM extraction, session-aware recall. Leave `session_id` as `None` to write to the **permanent knowledge graph**, which uses LLM extraction during ingestion and supports richer graph-completion queries.
Cognee configuration (LLM provider, database, vector store) is read from environment variables. See the [Cognee documentation](https://docs.cognee.ai) for setup instructions.
### Parameters
- `search_type` is *optional* and defaults to `"GRAPH_COMPLETION"`. Controls which Cognee recall strategy is used. Other useful values include `"CHUNKS"` for raw retrieval and `"SUMMARIES"` for summarized graph nodes.
- `top_k` is *optional* and defaults to `5`. Sets the default maximum number of memories returned per search.
- `dataset_name` is *optional* and defaults to `"haystack_memory"`. Names the Cognee dataset backing this store.
- `session_id` is *optional* and defaults to `None`. When set, reads and writes target the session-cache tier. When `None`, the permanent knowledge graph is used.
- `self_improvement` is *optional* and defaults to `True`. When `True`, Cognee runs graph improvement inline after every write. Set to `False` when you want `improve()` to be the sole improvement trigger.
- `timeout` is *optional* and defaults to `300`. Per-call timeout in seconds for any Cognee operation.
### Installation
Install the Cognee integration:
```bash
pip install cognee-haystack
```
Set your LLM API key (used by Cognee for graph extraction and queries):
```bash
export LLM_API_KEY="your-llm-api-key"
```
Optionally, set a separate embedding API key (defaults to `LLM_API_KEY` when unset):
```bash
export EMBEDDING_API_KEY="your-embedding-api-key"
```
## Usage
### On its own
```python
from haystack.dataclasses import ChatMessage
from haystack_integrations.memory_stores.cognee import CogneeMemoryStore
store = CogneeMemoryStore(search_type="GRAPH_COMPLETION", top_k=5)
store.add_memories(
messages=[ChatMessage.from_user("Alice enjoys hiking and outdoor activities.")],
user_id="a1b2c3d4-e5f6-7890-abcd-ef1234567890",
)
memories = store.search_memories(
query="What does Alice like?",
user_id="a1b2c3d4-e5f6-7890-abcd-ef1234567890",
)
print([msg.text for msg in memories])
```
### Session tier vs permanent graph
Use `session_id` to control which memory tier is targeted. A single store can serve both tiers — the writer's `session_id` overrides the store's `session_id` per call.
```python
from haystack.dataclasses import ChatMessage
from haystack_integrations.memory_stores.cognee import CogneeMemoryStore
store = CogneeMemoryStore(dataset_name="my_agent_memory", self_improvement=False)
# Write long-lived facts to the permanent graph (no session_id).
store.add_memories(
messages=[ChatMessage.from_user("Alice is a senior data scientist at Acme Corp.")],
)
# Write transient session context to the session cache.
store.add_memories(
messages=[ChatMessage.from_user("Alice is currently debugging a vector store issue.")],
session_id="alice_session_1",
)
# Promote the session cache into the permanent graph.
store.improve(session_id="alice_session_1")
```
### Delete all memories
```python
# Delete only this store's dataset (session cache is unaffected).
store.delete_all_memories()
# To wipe everything including the session cache, call cognee directly:
import asyncio
import cognee
asyncio.run(cognee.forget(everything=True))
```
docs-website/docs/memory-stores/mem0memorystore.mdx
+103
View File
@@ -0,0 +1,103 @@
---
title: "Mem0MemoryStore"
id: mem0memorystore
slug: "/mem0memorystore"
description: "A memory store backed by the Mem0 cloud API."
---
# Mem0MemoryStore
`Mem0MemoryStore` is a memory store backed by the Mem0 cloud API. It is the shared data layer used by [`Mem0MemoryRetriever`](../pipeline-components/retrievers/mem0memoryretriever.mdx), [`Mem0MemoryWriter`](../pipeline-components/writers/mem0memorywriter.mdx), and the [Mem0 Memory Tools](../tools/ready-made-tools/mem0memorytools.mdx).
<div className="key-value-table">
| | |
| --- | --- |
| **Used by** | [`Mem0MemoryRetriever`](../pipeline-components/retrievers/mem0memoryretriever.mdx), [`Mem0MemoryWriter`](../pipeline-components/writers/mem0memorywriter.mdx), [`Mem0MemoryRetrieverTool`, `Mem0MemoryWriterTool`](../tools/ready-made-tools/mem0memorytools.mdx) |
| **Optional init variables** | `api_key`: Defaults to `MEM0_API_KEY` environment variable |
| **API reference** | [Mem0](/reference/integrations-mem0#mem0memorystore) |
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/mem0 |
| **Package name** | `mem0-haystack` |
</div>
## Overview
`Mem0MemoryStore` wraps the Mem0 cloud API and provides two core methods:
- `add_memories` — stores a list of `ChatMessage` objects as memories in Mem0.
- `search_memories` — retrieves memories from Mem0 that are relevant to a query.
Scope memories with at least one Mem0 entity ID: `user_id`, `run_id`, `agent_id`, or `app_id`. These are runtime parameters, so a single store instance can serve multiple users or sessions.
The `infer` parameter on `add_memories` controls how Mem0 processes incoming messages:
- `infer=True` lets Mem0 extract memories from the messages automatically. This is useful when storing a full Agent turn.
- `infer=False` stores the supplied message text as-is. This is useful when the exact memory text has already been selected upstream.
### Installation
Install the Mem0 integration:
```bash
pip install mem0-haystack
```
Set your Mem0 API key:
```bash
export MEM0_API_KEY="your-mem0-api-key"
```
## Usage
### On its own
```python
from haystack.dataclasses import ChatMessage
from haystack_integrations.memory_stores.mem0 import Mem0MemoryStore
store = Mem0MemoryStore()
store.add_memories(
messages=[ChatMessage.from_user("Alice prefers concise Python examples.")],
user_id="alice",
infer=False,
)
memories = store.search_memories(
query="What does Alice prefer?",
user_id="alice",
top_k=3,
)
print([msg.text for msg in memories])
```
### Scoping with multiple entity IDs
Mem0 supports narrowing the scope of reads and writes with `user_id`, `run_id`, `agent_id`, and `app_id`. Pass any combination at call time:
```python
store.add_memories(
messages=[ChatMessage.from_user("Alice is working on a documentation search system.")],
user_id="alice",
run_id="docs-assistant-session-1",
infer=True,
)
memories = store.search_memories(
query="What project is Alice working on?",
user_id="alice",
run_id="docs-assistant-session-1",
)
print([msg.text for msg in memories])
```
### Retrieving all memories in scope
Pass `query=None` to return all memories matching the provided scope without a relevance search:
```python
all_memories = store.search_memories(query=None, user_id="alice")
print([msg.text for msg in all_memories])
```
docs-website/docs/pipeline-components/audio.mdx
+1
View File
@@ -11,5 +11,6 @@ Use these components to work with audio in Haystack by transcribing files or con
| Name | Description |
| --- | --- |
| [FunASRTranscriber](audio/funasrtranscriber.mdx) | Transcribe audio files using FunASR — a local, open-source speech recognition toolkit supporting 50+ languages. |
| [LocalWhisperTranscriber](audio/localwhispertranscriber.mdx) | Transcribe audio files using OpenAI's Whisper model using your local installation of Whisper. |
| [RemoteWhisperTranscriber](audio/remotewhispertranscriber.mdx) | Transcribe audio files using OpenAI's Whisper model. |
docs-website/docs/pipeline-components/audio/funasrtranscriber.mdx
+67
View File
@@ -0,0 +1,67 @@
---
title: "FunASRTranscriber"
id: funasrtranscriber
slug: "/funasrtranscriber"
description: "Transcribe audio files to Documents using FunASR — a local, open-source speech recognition toolkit supporting 50+ languages."
---
# FunASRTranscriber
Transcribe audio files to Haystack Documents using FunASR — a local, open-source speech recognition toolkit supporting 50+ languages.
<div className="key-value-table">
| | |
| --- | --- |
| **Most common position in a pipeline** | As the first component in an indexing pipeline |
| **Mandatory run variables** | `sources`: A list of audio file paths (`str` or `Path`) or `ByteStream` objects |
| **Output variables** | `documents`: A list of Haystack Documents, one per source, with transcript text in `content` |
| **API reference** | [FunASR integration](/reference/integrations-funasr) |
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/blob/main/integrations/funasr/src/haystack_integrations/components/audio/funasr/transcriber.py |
</div>
## Overview
`FunASRTranscriber` uses [FunASR](https://github.com/modelscope/FunASR), an open-source speech recognition toolkit from Alibaba DAMO Academy, to transcribe audio files into Haystack `Document` objects. It runs entirely locally — no API key required.
The default model is `iic/SenseVoiceSmall`, a multilingual model supporting 50+ languages that is 5–10x faster than Whisper. Models are downloaded from ModelScope on first use and cached in `~/.cache/modelscope`.
The component accepts audio file paths (`str` or `Path`) as well as `ByteStream` objects. Call `warm_up()` before running in a pipeline to load the model into memory.
## Usage
### On its own
```python
from haystack_integrations.components.audio.funasr import FunASRTranscriber
transcriber = FunASRTranscriber()
transcriber.warm_up()
result = transcriber.run(sources=["speech.wav"])
print(result["documents"][0].content)
```
### In a pipeline
```python
from haystack import Pipeline
from haystack.components.fetchers import LinkContentFetcher
from haystack_integrations.components.audio.funasr import FunASRTranscriber
pipe = Pipeline()
pipe.add_component("fetcher", LinkContentFetcher())
pipe.add_component("transcriber", FunASRTranscriber())
pipe.connect("fetcher", "transcriber")
result = pipe.run(
data={
"fetcher": {
"urls": ["https://example.com/interview.wav"],
},
}
)
print(result["transcriber"]["documents"][0].content)
```
docs-website/docs/pipeline-components/embedders/huggingfaceapidocumentembedder.mdx
+46 -19
View File
@@ -17,9 +17,9 @@ Use this component to compute document embeddings using various Hugging Face API
| **Mandatory init variables** | `api_type`: The type of Hugging Face API to use <br /> <br />`api_params`: A dictionary with one of the following keys: <br /> <br />- `model`: Hugging Face model ID. Required when `api_type` is `SERVERLESS_INFERENCE_API`.**OR** - `url`: URL of the inference endpoint. Required when `api_type` is `INFERENCE_ENDPOINTS` or `TEXT_EMBEDDINGS_INFERENCE`. <br /> <br />`token`: The Hugging Face API token. Can be set with `HF_API_TOKEN` or `HF_TOKEN` env var. |
| **Mandatory run variables** | `documents`: A list of documents to be embedded |
| **Output variables** | `documents`: A list of documents to be embedded (enriched with embeddings) |
| **API reference** | [Embedders](/reference/embedders-api) |
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/embedders/hugging_face_api_document_embedder.py |
| **Package name** | `haystack-ai` |
| **API reference** | [Hugging Face API](/reference/integrations-huggingface-api) |
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/huggingface_api |
| **Package name** | `huggingface-api-haystack` |
</div>
@@ -43,8 +43,14 @@ The token is needed:
## Usage
Install the `huggingface-api-haystack` package to use the `HuggingFaceAPIDocumentEmbedder`:
```shell
pip install huggingface-api-haystack
```
Similarly to other Document Embedders, this component allows adding prefixes (and postfixes) to include instruction and embedding metadata.
For more fine-grained details, refer to the component’s [API reference](/reference/embedders-api#huggingfaceapidocumentembedder).
For more fine-grained details, refer to the component’s [API reference](/reference/integrations-huggingface-api#huggingfaceapidocumentembedder).
### On its own
@@ -56,7 +62,9 @@ To use this API, you need a [free Hugging Face token](https://huggingface.co/set
The Embedder expects the `model` in `api_params`.
```python
from haystack.components.embedders import HuggingFaceAPIDocumentEmbedder
from haystack_integrations.components.embedders.huggingface_api import (
HuggingFaceAPIDocumentEmbedder,
)
from haystack.utils import Secret
from haystack.dataclasses import Document
@@ -84,7 +92,9 @@ Additionally, in this case, you need to provide your Hugging Face token.
The Embedder expects the `url` of your endpoint in `api_params`.
```python
from haystack.components.embedders import HuggingFaceAPIDocumentEmbedder
from haystack_integrations.components.embedders.huggingface_api import (
HuggingFaceAPIDocumentEmbedder,
)
from haystack.utils import Secret
from haystack.dataclasses import Document
@@ -123,7 +133,9 @@ For more information, refer to the [official TEI repository](https://github.com/
The Embedder expects the `url` of your TEI instance in `api_params`.
```python
from haystack.components.embedders import HuggingFaceAPIDocumentEmbedder
from haystack_integrations.components.embedders.huggingface_api import (
HuggingFaceAPIDocumentEmbedder,
)
from haystack.dataclasses import Document
doc = Document(content="I love pizza!")
@@ -145,38 +157,53 @@ print(result["documents"][0].embedding)
from haystack import Document
from haystack import Pipeline
from haystack.document_stores.in_memory import InMemoryDocumentStore
from haystack.components.embedders import HuggingFaceAPITextEmbedder, HuggingFaceAPIDocumentEmbedder
from haystack_integrations.components.embedders.huggingface_api import (
HuggingFaceAPITextEmbedder,
HuggingFaceAPIDocumentEmbedder,
)
from haystack.components.writers import DocumentWriter
from haystack.components.retrievers.in_memory import InMemoryEmbeddingRetriever
document_store = InMemoryDocumentStore(embedding_similarity_function="cosine")
documents = [Document(content="My name is Wolfgang and I live in Berlin"),
Document(content="I saw a black horse running"),
Document(content="Germany has many big cities")]
documents = [
Document(content="My name is Wolfgang and I live in Berlin"),
Document(content="I saw a black horse running"),
Document(content="Germany has many big cities"),
]
document_embedder = HuggingFaceAPIDocumentEmbedder(api_type="serverless_inference_api",
api_params={"model": "BAAI/bge-small-en-v1.5"})
document_embedder = HuggingFaceAPIDocumentEmbedder(
api_type="serverless_inference_api",
api_params={"model": "BAAI/bge-small-en-v1.5"},
)
indexing_pipeline = Pipeline()
indexing_pipeline.add_component("document_embedder", document_embedder)
indexing_pipeline.add_component("doc_writer", DocumentWriter(document_store=document_store)
indexing_pipeline.add_component(
"doc_writer",
DocumentWriter(document_store=document_store),
)
indexing_pipeline.connect("document_embedder", "doc_writer")
indexing_pipeline.run({"document_embedder": {"documents": documents}})
text_embedder = HuggingFaceAPITextEmbedder(api_type="serverless_inference_api",
api_params={"model": "BAAI/bge-small-en-v1.5"})
text_embedder = HuggingFaceAPITextEmbedder(
api_type="serverless_inference_api",
api_params={"model": "BAAI/bge-small-en-v1.5"},
)
query_pipeline = Pipeline()
query_pipeline.add_component("text_embedder", text_embedder)
query_pipeline.add_component("retriever", InMemoryEmbeddingRetriever(document_store=document_store))
query_pipeline.add_component(
"retriever",
InMemoryEmbeddingRetriever(document_store=document_store),
)
query_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")
query = "Who lives in Berlin?"
result = query_pipeline.run({"text_embedder":{"text": query}})
result = query_pipeline.run({"text_embedder": {"text": query}})
print(result['retriever']['documents'][0])
print(result["retriever"]["documents"][0])
# Document(id=..., content: 'My name is Wolfgang and I live in Berlin', ...)
```
docs-website/docs/pipeline-components/embedders/huggingfaceapitextembedder.mdx
+20 -8
View File
@@ -17,9 +17,9 @@ Use this component to embed strings using various Hugging Face APIs.
| **Mandatory init variables** | `api_type`: The type of Hugging Face API to use <br /> <br />`api_params`: A dictionary with one of the following keys: <br /> <br />- `model`: Hugging Face model ID. Required when `api_type` is `SERVERLESS_INFERENCE_API`.**OR** - `url`: URL of the inference endpoint. Required when `api_type` is `INFERENCE_ENDPOINTS` or `TEXT_EMBEDDINGS_INFERENCE`. <br /> <br />`token`: The Hugging Face API token. Can be set with `HF_API_TOKEN` or `HF_TOKEN` env var. |
| **Mandatory run variables** | `text`: A string |
| **Output variables** | `embedding`: A list of float numbers |
| **API reference** | [Embedders](/reference/embedders-api) |
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/embedders/hugging_face_api_text_embedder.py |
| **Package name** | `haystack-ai` |
| **API reference** | [Hugging Face API](/reference/integrations-huggingface-api) |
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/huggingface_api |
| **Package name** | `huggingface-api-haystack` |
</div>
@@ -43,8 +43,14 @@ The token is needed:
## Usage
Install the `huggingface-api-haystack` package to use the `HuggingFaceAPITextEmbedder`:
```shell
pip install huggingface-api-haystack
```
Similarly to other text Embedders, this component allows adding prefixes (and postfixes) to include instructions.
For more fine-grained details, refer to the component’s [API reference](/reference/embedders-api#huggingfaceapitextembedder).
For more fine-grained details, refer to the component’s [API reference](/reference/integrations-huggingface-api#huggingfaceapitextembedder).
### On its own
@@ -56,7 +62,9 @@ To use this API, you need a [free Hugging Face token](https://huggingface.co/set
The Embedder expects the `model` in `api_params`.
```python
from haystack.components.embedders import HuggingFaceAPITextEmbedder
from haystack_integrations.components.embedders.huggingface_api import (
HuggingFaceAPITextEmbedder,
)
from haystack.utils import Secret
text_embedder = HuggingFaceAPITextEmbedder(
@@ -80,7 +88,9 @@ Additionally, in this case, you need to provide your Hugging Face token.
The Embedder expects the `url` of your endpoint in `api_params`.
```python
from haystack.components.embedders import HuggingFaceAPITextEmbedder
from haystack_integrations.components.embedders.huggingface_api import (
HuggingFaceAPITextEmbedder,
)
from haystack.utils import Secret
text_embedder = HuggingFaceAPITextEmbedder(
@@ -115,7 +125,9 @@ For more information, refer to the [official TEI repository](https://github.com/
The Embedder expects the `url` of your TEI instance in `api_params`.
```python
from haystack.components.embedders import HuggingFaceAPITextEmbedder
from haystack_integrations.components.embedders.huggingface_api import (
HuggingFaceAPITextEmbedder,
)
from haystack.utils import Secret
text_embedder = HuggingFaceAPITextEmbedder(
@@ -134,7 +146,7 @@ print(text_embedder.run("I love pizza!"))
from haystack import Document
from haystack import Pipeline
from haystack.document_stores.in_memory import InMemoryDocumentStore
from haystack.components.embedders import (
from haystack_integrations.components.embedders.huggingface_api import (
HuggingFaceAPITextEmbedder,
HuggingFaceAPIDocumentEmbedder,
)
docs-website/docs/pipeline-components/generators/huggingfaceapichatgenerator.mdx
+33 -11
View File
@@ -17,9 +17,9 @@ This generator enables chat completion using various Hugging Face APIs.
| **Mandatory init variables** | `api_type`: The type of Hugging Face API to use <br /> <br />`api_params`: A dictionary with one of the following keys: <br /> <br />- `model`: Hugging Face model ID. Required when `api_type` is `SERVERLESS_INFERENCE_API`.**OR** - `url`: URL of the inference endpoint. Required when `api_type` is `INFERENCE_ENDPOINTS` or `TEXT_EMBEDDINGS_INFERENCE`.`token`: The Hugging Face API token. Can be set with `HF_API_TOKEN` or `HF_TOKEN` env var. |
| **Mandatory run variables** | `messages`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects representing the chat or a plain string |
| **Output variables** | `replies`: A list of replies of the LLM to the input chat |
| **API reference** | [Generators](/reference/generators-api) |
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/generators/chat/hugging_face_api.py |
| **Package name** | `haystack-ai` |
| **API reference** | [Hugging Face API](/reference/integrations-huggingface-api) |
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/huggingface_api |
| **Package name** | `huggingface-api-haystack` |
</div>
@@ -49,6 +49,12 @@ This Generator supports [streaming](guides-to-generators/choosing-the-right-gene
## Usage
Install the `huggingface-api-haystack` package to use the `HuggingFaceAPIChatGenerator`:
```shell
pip install huggingface-api-haystack
```
### On its own
#### Using Serverless Inference API (Inference Providers) - Free Tier Available
@@ -59,10 +65,14 @@ To use this API, you need a [free Hugging Face token](https://huggingface.co/set
The Generator expects the `model` in `api_params`. It's also recommended to specify a `provider` for better performance and reliability.
```python
from haystack.components.generators.chat import HuggingFaceAPIChatGenerator
from haystack_integrations.components.generators.huggingface_api import (
HuggingFaceAPIChatGenerator,
)
from haystack.dataclasses import ChatMessage
from haystack.utils import Secret
from haystack.utils.hf import HFGenerationAPIType
from haystack_integrations.components.common.huggingface_api.utils import (
HFGenerationAPIType,
)
messages = [
ChatMessage.from_system("\\nYou are a helpful, respectful and honest assistant"),
@@ -93,7 +103,9 @@ Additionally, in this case, you need to provide your Hugging Face token.
The Generator expects the `url` of your endpoint in `api_params`.
```python
from haystack.components.generators.chat import HuggingFaceAPIChatGenerator
from haystack_integrations.components.generators.huggingface_api import (
HuggingFaceAPIChatGenerator,
)
from haystack.dataclasses import ChatMessage
from haystack.utils import Secret
@@ -117,10 +129,14 @@ print(result)
You can also use this component with multimodal models that support both text and image input:
```python
from haystack.components.generators.chat import HuggingFaceAPIChatGenerator
from haystack_integrations.components.generators.huggingface_api import (
HuggingFaceAPIChatGenerator,
)
from haystack.dataclasses import ChatMessage, ImageContent
from haystack.utils import Secret
from haystack.utils.hf import HFGenerationAPIType
from haystack_integrations.components.common.huggingface_api.utils import (
HFGenerationAPIType,
)
# Create an image from file path, URL, or base64
image = ImageContent.from_file_path("path/to/your/image.jpg")
@@ -163,7 +179,9 @@ For more information, refer to the [official TGI repository](https://github.com/
The Generator expects the `url` of your TGI instance in `api_params`.
```python
from haystack.components.generators.chat import HuggingFaceAPIChatGenerator
from haystack_integrations.components.generators.huggingface_api import (
HuggingFaceAPIChatGenerator,
)
from haystack.dataclasses import ChatMessage
messages = [
@@ -184,11 +202,15 @@ print(result)
```python
from haystack.components.builders import ChatPromptBuilder
from haystack.components.generators.chat import HuggingFaceAPIChatGenerator
from haystack_integrations.components.generators.huggingface_api import (
HuggingFaceAPIChatGenerator,
)
from haystack.dataclasses import ChatMessage
from haystack import Pipeline
from haystack.utils import Secret
from haystack.utils.hf import HFGenerationAPIType
from haystack_integrations.components.common.huggingface_api.utils import (
HFGenerationAPIType,
)
# no parameter init, we don't use any runtime template variables
prompt_builder = ChatPromptBuilder()
docs-website/docs/pipeline-components/rankers/huggingfaceteiranker.mdx
+13 -5
View File
@@ -17,9 +17,9 @@ Use this component to rank documents based on their similarity to the query usin
| **Mandatory init variables** | `url`: Base URL of the TEI reranking service (for example, "https://api.example.com"). |
| **Mandatory run variables** | `query`: A query string <br /> <br />`documents`: A list of document objects |
| **Output variables** | `documents`: A grouped list of documents |
| **API reference** | [Rankers](/reference/rankers-api) |
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/rankers/hugging_face_tei.py |
| **Package name** | `haystack-ai` |
| **API reference** | [Hugging Face API](/reference/integrations-huggingface-api) |
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/huggingface_api |
| **Package name** | `huggingface-api-haystack` |
</div>
@@ -38,6 +38,12 @@ Depending on your TEI server configuration, you may also require a Hugging Face
## Usage
Install the `huggingface-api-haystack` package to use the `HuggingFaceTEIRanker`:
```shell
pip install huggingface-api-haystack
```
### On its own
You can use `HuggingFaceTEIRanker` outside of a pipeline to order documents based on your query.
@@ -46,7 +52,7 @@ This example uses the `HuggingFaceTEIRanker` to rank two simple documents. To ru
```python
from haystack import Document
from haystack.components.rankers import HuggingFaceTEIRanker
from haystack_integrations.components.rankers.huggingface_api import HuggingFaceTEIRanker
from haystack.utils import Secret
reranker = HuggingFaceTEIRanker(
@@ -76,7 +82,9 @@ Below is an example of a pipeline that retrieves documents from an `InMemoryDocu
from haystack import Document, Pipeline
from haystack.document_stores.in_memory import InMemoryDocumentStore
from haystack.components.retrievers.in_memory import InMemoryBM25Retriever
from haystack.components.rankers import HuggingFaceTEIRanker
from haystack_integrations.components.rankers.huggingface_api import (
HuggingFaceTEIRanker,
)
docs = [
Document(content="Paris is in France"),
docs-website/docs/pipeline-components/retrievers.mdx
+5
View File
@@ -177,6 +177,8 @@ For details on how to initialize and use a Retriever in a pipeline, see the docu
| [OpenSearchBM25Retriever](retrievers/opensearchbm25retriever.mdx) | A keyword-based Retriever that fetches Documents matching a query from an OpenSearch Document Store. |
| [OpenSearchEmbeddingRetriever](retrievers/opensearchembeddingretriever.mdx) | An embedding-based Retriever compatible with the OpenSearch Document Store. |
| [OpenSearchHybridRetriever](retrievers/opensearchhybridretriever.mdx) | A SuperComponent that implements a Hybrid Retriever in a single component, relying on OpenSearch as the backend Document Store. |
| [OracleEmbeddingRetriever](retrievers/oracleembeddingretriever.mdx) | An embedding-based Retriever compatible with the Oracle Document Store. |
| [OracleKeywordRetriever](retrievers/oraclekeywordretriever.mdx) | A keyword-based Retriever that fetches Documents matching a query from the Oracle Document Store. |
| [PgvectorEmbeddingRetriever](retrievers/pgvectorembeddingretriever.mdx) | An embedding-based Retriever compatible with the Pgvector Document Store. |
| [PgvectorKeywordRetriever](retrievers/pgvectorkeywordretriever.mdx) | A keyword-based Retriever that fetches documents matching a query from the Pgvector Document Store. |
| [PineconeEmbeddingRetriever](retrievers/pineconedenseretriever.mdx) | An embedding-based Retriever compatible with the Pinecone Document Store. |
@@ -185,6 +187,9 @@ For details on how to initialize and use a Retriever in a pipeline, see the docu
| [QdrantHybridRetriever](retrievers/qdranthybridretriever.mdx) | A Retriever based both on dense and sparse embeddings, compatible with the Qdrant Document Store. |
| [SentenceWindowRetriever](retrievers/sentencewindowretrieval.mdx) | Retrieves neighboring sentences around relevant sentences to get the full context. |
| [SnowflakeTableRetriever](retrievers/snowflaketableretriever.mdx) | Connects to a Snowflake database to execute an SQL query. |
| [SupabaseGroongaBM25Retriever](retrievers/supabasegroongabm25retriever.mdx) | A full-text Retriever that fetches documents from the SupabaseGroongaDocumentStore using PGroonga search. |
| [SupabasePgvectorEmbeddingRetriever](retrievers/supabasepgvectorembeddingretriever.mdx) | An embedding-based Retriever compatible with the SupabasePgvectorDocumentStore. |
| [SupabasePgvectorKeywordRetriever](retrievers/supabasepgvectorkeywordretriever.mdx) | A keyword-based Retriever that fetches documents matching a query from the SupabasePgvectorDocumentStore. |
| [TextEmbeddingRetriever](retrievers/textembeddingretriever.mdx) | Wraps an embedding-based retriever with a text embedder into a single component that accepts a text query. |
| [VespaEmbeddingRetriever](retrievers/vespaembeddingretriever.mdx) | An embedding-based Retriever compatible with the Vespa Document Store. |
| [VespaKeywordRetriever](retrievers/vespakeywordretriever.mdx) | A keyword-based Retriever that fetches Documents matching a query from the Vespa Document Store. |
docs-website/docs/pipeline-components/retrievers/cogneeretriever.mdx
+139
View File
@@ -0,0 +1,139 @@
---
title: "CogneeRetriever"
id: cogneeretriever
slug: "/cogneeretriever"
description: "Retrieves memories from a CogneeMemoryStore and returns them as system ChatMessage objects."
---
# CogneeRetriever
Retrieves memories from a `CogneeMemoryStore` and returns them as system `ChatMessage` objects.
<div className="key-value-table">
| | |
| --- | --- |
| **Most common position in a pipeline** | Before an [`Agent`](../agents-1/agent.mdx) or Chat Generator in memory-augmented pipelines |
| **Mandatory init variables** | `memory_store`: A `CogneeMemoryStore` instance |
| **Optional init variables** | `top_k`: Maximum number of memories to return (defaults to the store's `top_k`) |
| **Mandatory run variables** | `query`: A text query to search memories |
| **Optional run variables** | `user_id`: Cognee user ID to scope the retrieval; pass `None` to use Cognee's default user |
| **Output variables** | `messages`: A list of system `ChatMessage` objects |
| **API reference** | [Cognee](/reference/integrations-cognee#cogneeretriever) |
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/cognee |
| **Package name** | `cognee-haystack` |
</div>
## Overview
`CogneeRetriever` retrieves memories from a `CogneeMemoryStore` and returns them as system `ChatMessage` objects.
Use it to inject long-term memory into an Agent or a chat generation pipeline before the model produces a response.
Search behavior — including the search strategy (`search_type`), dataset, and session tier — is configured on the `CogneeMemoryStore`. The retriever is a thin pipeline adapter over `search_memories`.
The `user_id` parameter scopes the retrieval to a specific Cognee user. Pass `None` to use Cognee's default user.
## Installation
Install the Cognee integration:
```bash
pip install cognee-haystack
```
Set your LLM API key (used by Cognee for graph extraction and queries):
```bash
export LLM_API_KEY="your-llm-api-key"
```
Optionally, set a separate embedding API key (defaults to `LLM_API_KEY` when unset):
```bash
export EMBEDDING_API_KEY="your-embedding-api-key"
```
## Usage
### On its own
```python
from haystack.dataclasses import ChatMessage
from haystack_integrations.components.retrievers.cognee import CogneeRetriever
from haystack_integrations.memory_stores.cognee import CogneeMemoryStore
store = CogneeMemoryStore(search_type="GRAPH_COMPLETION", top_k=5)
# Write some memories first
store.add_memories(
messages=[ChatMessage.from_user("Alice prefers concise Python examples.")],
user_id="a1b2c3d4-e5f6-7890-abcd-ef1234567890",
)
retriever = CogneeRetriever(memory_store=store, top_k=3)
result = retriever.run(
query="What does Alice prefer?",
user_id="a1b2c3d4-e5f6-7890-abcd-ef1234567890",
)
memories = result["messages"]
print([message.text for message in memories])
```
### In a Pipeline
This example retrieves memories, prepends them to the current user message, and passes the combined message list to an Agent.
```python
from haystack import Pipeline
from haystack.components.agents import Agent
from haystack.components.converters import OutputAdapter
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.dataclasses import ChatMessage
from haystack_integrations.components.retrievers.cognee import CogneeRetriever
from haystack_integrations.memory_stores.cognee import CogneeMemoryStore
store = CogneeMemoryStore(dataset_name="my_agent_memory", session_id="alice_session_1")
pipeline = Pipeline()
pipeline.add_component("retriever", CogneeRetriever(memory_store=store, top_k=5))
pipeline.add_component(
"memory_context",
OutputAdapter(
template="{{ memories + user_messages }}",
output_type=list[ChatMessage],
unsafe=True,
),
)
pipeline.add_component(
"agent",
Agent(
chat_generator=OpenAIChatGenerator(model="gpt-4o-mini"),
system_prompt=(
"Use any system messages at the start of the conversation as long-term memory. "
"Answer concisely."
),
),
)
pipeline.connect("retriever.messages", "memory_context.memories")
pipeline.connect("memory_context.output", "agent.messages")
query = "Give me a short implementation tip."
pipeline.run(
{
"retriever": {
"query": query,
"user_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
},
"memory_context": {
"user_messages": [ChatMessage.from_user(query)],
},
}
)
```
docs-website/docs/pipeline-components/retrievers/oracleembeddingretriever.mdx
+142
View File
@@ -0,0 +1,142 @@
---
title: "OracleEmbeddingRetriever"
id: oracleembeddingretriever
slug: "/oracleembeddingretriever"
description: "An embedding-based Retriever compatible with the Oracle Document Store."
---
# OracleEmbeddingRetriever
An embedding-based Retriever compatible with the Oracle Document Store.
<div className="key-value-table">
| | |
| --- | --- |
| **Most common position in a pipeline** | 1. After a Text Embedder and before a [`PromptBuilder`](../builders/promptbuilder.mdx) in a RAG pipeline 2. The last component in a semantic search pipeline 3. After a Text Embedder and before an [`ExtractiveReader`](../readers/extractivereader.mdx) in an extractive QA pipeline |
| **Mandatory init variables** | `document_store`: An instance of an [OracleDocumentStore](../../document-stores/oracledocumentstore.mdx) |
| **Mandatory run variables** | `query_embedding`: A vector representing the query (a list of floats) |
| **Output variables** | `documents`: A list of documents |
| **API reference** | [Oracle](/reference/integrations-oracle) |
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/oracle |
| **Package name** | `oracle-haystack` |
</div>
## Overview
The `OracleEmbeddingRetriever` is an embedding-based Retriever compatible with `OracleDocumentStore`. It uses Oracle AI Vector Search to compare query and document embeddings, fetching the most relevant documents based on vector similarity.
When using `OracleEmbeddingRetriever` in a pipeline, make sure embeddings are available for both documents (at index time) and queries (at query time). Use a Document Embedder in your indexing pipeline and a Text Embedder in your query pipeline.
The distance metric (COSINE, EUCLIDEAN, or DOT) is configured on the `OracleDocumentStore`. In addition to `query_embedding`, the retriever accepts `top_k` (maximum documents to return) and `filters` to narrow the search space.
## Installation
To run Oracle Database 23ai locally with Docker:
```shell
docker run -d --name oracle23ai \
-p 1521:1521 \
-e ORACLE_PASSWORD=oracle \
container-registry.oracle.com/database/free:latest
```
Install the Oracle integration for Haystack:
```shell
pip install oracle-haystack
```
## Usage
### On its own
This Retriever needs an `OracleDocumentStore` and indexed documents with embeddings to run.
```python
from haystack.utils import Secret
from haystack_integrations.document_stores.oracle import (
OracleDocumentStore,
OracleConnectionConfig,
)
from haystack_integrations.components.retrievers.oracle import OracleEmbeddingRetriever
document_store = OracleDocumentStore(
connection_config=OracleConnectionConfig(
user=Secret.from_env_var("ORACLE_USER"),
password=Secret.from_env_var("ORACLE_PASSWORD"),
dsn=Secret.from_env_var("ORACLE_DSN"),
),
embedding_dim=768,
)
retriever = OracleEmbeddingRetriever(document_store=document_store)
# using a fake vector to keep the example simple
retriever.run(query_embedding=[0.1] * 768)
```
### In a Pipeline
```python
from haystack import Document, Pipeline
from haystack.document_stores.types import DuplicatePolicy
from haystack.components.embedders import (
SentenceTransformersDocumentEmbedder,
SentenceTransformersTextEmbedder,
)
from haystack.utils import Secret
from haystack_integrations.document_stores.oracle import (
OracleDocumentStore,
OracleConnectionConfig,
)
from haystack_integrations.components.retrievers.oracle import OracleEmbeddingRetriever
document_store = OracleDocumentStore(
connection_config=OracleConnectionConfig(
user=Secret.from_env_var("ORACLE_USER"),
password=Secret.from_env_var("ORACLE_PASSWORD"),
dsn=Secret.from_env_var("ORACLE_DSN"),
),
embedding_dim=768,
)
documents = [
Document(content="There are over 7,000 languages spoken around the world today."),
Document(
content="Elephants have been observed to behave in a way that indicates a high level of self-awareness, such as recognizing themselves in mirrors.",
),
Document(
content="In certain parts of the world, like the Maldives, Puerto Rico, and San Diego, you can witness the phenomenon of bioluminescent waves.",
),
]
document_embedder = SentenceTransformersDocumentEmbedder(
model="sentence-transformers/all-MiniLM-L6-v2",
)
document_embedder.warm_up()
documents_with_embeddings = document_embedder.run(documents)
document_store.write_documents(
documents_with_embeddings["documents"],
policy=DuplicatePolicy.OVERWRITE,
)
query_pipeline = Pipeline()
query_pipeline.add_component(
"text_embedder",
SentenceTransformersTextEmbedder(model="sentence-transformers/all-MiniLM-L6-v2"),
)
query_pipeline.add_component(
"retriever",
OracleEmbeddingRetriever(document_store=document_store),
)
query_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")
query = "How many languages are there?"
result = query_pipeline.run({"text_embedder": {"text": query}})
print(result["retriever"]["documents"][0])
```
docs-website/docs/pipeline-components/retrievers/oraclekeywordretriever.mdx
+150
View File
@@ -0,0 +1,150 @@
---
title: "OracleKeywordRetriever"
id: oraclekeywordretriever
slug: "/oraclekeywordretriever"
description: "A keyword-based Retriever that fetches documents matching a query from the Oracle Document Store using Oracle's DBMS_SEARCH full-text index."
---
# OracleKeywordRetriever
A keyword-based Retriever that fetches documents matching a query from the Oracle Document Store.
<div className="key-value-table">
| | |
| --- | --- |
| **Most common position in a pipeline** | 1. Before a [`PromptBuilder`](../builders/promptbuilder.mdx) in a RAG pipeline 2. The last component in a keyword search pipeline 3. Before an [`ExtractiveReader`](../readers/extractivereader.mdx) in an extractive QA pipeline |
| **Mandatory init variables** | `document_store`: An instance of an [OracleDocumentStore](../../document-stores/oracledocumentstore.mdx) |
| **Mandatory run variables** | `query`: A string |
| **Output variables** | `documents`: A list of documents matching the query |
| **API reference** | [Oracle](/reference/integrations-oracle) |
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/oracle |
| **Package name** | `oracle-haystack` |
</div>
## Overview
The `OracleKeywordRetriever` is a keyword-based Retriever compatible with `OracleDocumentStore`. It uses Oracle's DBMS_SEARCH full-text index — automatically created when the document store is initialized — to search documents by keyword relevance.
This retriever works without embeddings, making it suitable for keyword-only pipelines or as the keyword branch of a hybrid search pipeline.
In addition to `query`, the retriever accepts `top_k` (maximum documents to return) and `filters` to narrow the search space.
## Installation
To run Oracle Database 23ai locally with Docker:
```shell
docker run -d --name oracle23ai \
-p 1521:1521 \
-e ORACLE_PASSWORD=oracle \
container-registry.oracle.com/database/free:latest
```
Install the Oracle integration for Haystack:
```shell
pip install oracle-haystack
```
## Usage
### On its own
This Retriever needs an `OracleDocumentStore` and indexed documents to run.
```python
from haystack.utils import Secret
from haystack_integrations.document_stores.oracle import (
OracleDocumentStore,
OracleConnectionConfig,
)
from haystack_integrations.components.retrievers.oracle import OracleKeywordRetriever
document_store = OracleDocumentStore(
connection_config=OracleConnectionConfig(
user=Secret.from_env_var("ORACLE_USER"),
password=Secret.from_env_var("ORACLE_PASSWORD"),
dsn=Secret.from_env_var("ORACLE_DSN"),
),
embedding_dim=768,
)
retriever = OracleKeywordRetriever(document_store=document_store)
retriever.run(query="my keyword query")
```
### In a RAG pipeline
```python
from haystack import Document, Pipeline
from haystack.components.builders import ChatPromptBuilder
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.dataclasses import ChatMessage
from haystack.document_stores.types import DuplicatePolicy
from haystack.utils import Secret
from haystack_integrations.document_stores.oracle import (
OracleDocumentStore,
OracleConnectionConfig,
)
from haystack_integrations.components.retrievers.oracle import OracleKeywordRetriever
prompt_template = [
ChatMessage.from_user(
"""
Given these documents, answer the question.\nDocuments:
{% for doc in documents %}
{{ doc.content }}
{% endfor %}
\nQuestion: {{question}}
\nAnswer:
""",
),
]
document_store = OracleDocumentStore(
connection_config=OracleConnectionConfig(
user=Secret.from_env_var("ORACLE_USER"),
password=Secret.from_env_var("ORACLE_PASSWORD"),
dsn=Secret.from_env_var("ORACLE_DSN"),
),
embedding_dim=768,
)
documents = [
Document(content="There are over 7,000 languages spoken around the world today."),
Document(
content="Elephants have been observed to behave in a way that indicates a high level of self-awareness, such as recognizing themselves in mirrors.",
),
Document(
content="In certain parts of the world, like the Maldives, Puerto Rico, and San Diego, you can witness the phenomenon of bioluminescent waves.",
),
]
document_store.write_documents(documents=documents, policy=DuplicatePolicy.SKIP)
retriever = OracleKeywordRetriever(document_store=document_store)
rag_pipeline = Pipeline()
rag_pipeline.add_component(name="retriever", instance=retriever)
rag_pipeline.add_component(
instance=ChatPromptBuilder(template=prompt_template, required_variables="*"),
name="prompt_builder",
)
rag_pipeline.add_component(instance=OpenAIChatGenerator(), name="llm")
rag_pipeline.connect("retriever", "prompt_builder.documents")
rag_pipeline.connect("prompt_builder.prompt", "llm.messages")
question = "How many languages are there?"
result = rag_pipeline.run(
{
"retriever": {"query": question},
"prompt_builder": {"question": question},
},
)
print(result["llm"]["replies"][0].text)
```
docs-website/docs/pipeline-components/retrievers/qdrantembeddingretriever.mdx
+1 -1
View File
@@ -31,7 +31,7 @@ When using the `QdrantEmbeddingRetriever` in your NLP system, make sure it has t
In addition to the `query_embedding`, the `QdrantEmbeddingRetriever` accepts other optional parameters, including `top_k` (the maximum number of Documents to retrieve) and `filters` to narrow down the search space.
Some relevant parameters that impact the embedding retrieval must be defined when the corresponding `QdrantDocumentStore` is initialized: these include the embedding dimension (`embedding_dim`), the `similarity` function to use when comparing embeddings and the HNWS configuration (`hnsw_config`).
Some relevant parameters that impact the embedding retrieval must be defined when the corresponding `QdrantDocumentStore` is initialized: these include the embedding dimension (`embedding_dim`), the `similarity` function to use when comparing embeddings and the HNSW configuration (`hnsw_config`).
### Installation
docs-website/docs/pipeline-components/retrievers/qdranthybridretriever.mdx
+3 -3
View File
@@ -16,7 +16,7 @@ A Retriever based both on dense and sparse embeddings, compatible with the Qdran
| **Most common position in a pipeline** | 1\. After a Text Embedder and before a [`PromptBuilder`](../builders/promptbuilder.mdx) in a RAG pipeline <br /> <br />2. The last component in a hybrid search pipeline <br /> 3. After a Text Embedder and before an [`ExtractiveReader`](../readers/extractivereader.mdx) in an extractive QA pipeline |
| **Mandatory init variables** | `document_store`: An instance of a [QdrantDocumentStore](../../document-stores/qdrant-document-store.mdx) |
| **Mandatory run variables** | `query_embedding`: A dense vector representing the query (a list of floats) <br /> <br />`query_sparse_embedding`: A [`SparseEmbedding`](../../concepts/data-classes.mdx#sparseembedding) object containing a vectorial representation of the query |
| **Output variables** | `document`: A list of documents |
| **Output variables** | `documents`: A list of documents |
| **API reference** | [Qdrant](/reference/integrations-qdrant) |
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/qdrant |
| **Package name** | `qdrant-haystack` |
@@ -170,14 +170,14 @@ querying.connect("dense_text_embedder.embedding", "retriever.query_embedding")
question = "Who supports fastembed?"
results = query_mix.run(
results = querying.run(
{
"dense_text_embedder": {"text": question},
"sparse_text_embedder": {"text": question},
},
)
print(result["retriever"]["documents"][0])
print(results["retriever"]["documents"][0])
# Document(id=...,
# content: 'fastembed is supported by and maintained by Qdrant.',
docs-website/docs/pipeline-components/retrievers/qdrantsparseembeddingretriever.mdx
+2 -2
View File
@@ -100,8 +100,8 @@ from haystack_integrations.components.retrievers.qdrant import (
from haystack_integrations.document_stores.qdrant import QdrantDocumentStore
from haystack.document_stores.types import DuplicatePolicy
from haystack_integrations.components.embedders.fastembed import (
FastembedDocumentEmbedder,
FastembedTextEmbedder,
FastembedSparseDocumentEmbedder,
FastembedSparseTextEmbedder,
)
document_store = QdrantDocumentStore(
docs-website/docs/pipeline-components/retrievers/supabasegroongabm25retriever.mdx
+152
View File
@@ -0,0 +1,152 @@
---
title: "SupabaseGroongaBM25Retriever"
id: supabasegroongabm25retriever
slug: "/supabasegroongabm25retriever"
description: "A full-text Retriever that fetches documents from the SupabaseGroongaDocumentStore using PGroonga search."
---
# SupabaseGroongaBM25Retriever
A full-text Retriever that fetches documents from the SupabaseGroongaDocumentStore using PGroonga search.
<div className="key-value-table">
| | |
| --- | --- |
| **Most common position in a pipeline** | 1. Before a [`PromptBuilder`](../builders/promptbuilder.mdx) in a RAG pipeline 2. The last component in the full-text search pipeline |
| **Mandatory init variables** | `document_store`: An instance of a [SupabaseGroongaDocumentStore](../../document-stores/supabasedocumentstore.mdx) |
| **Mandatory run variables** | `query`: A string |
| **Output variables** | `documents`: A list of documents (matching the query) |
| **API reference** | [Supabase](/reference/integrations-supabase) |
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/supabase |
| **Package name** | `supabase-haystack` |
</div>
## Overview
`SupabaseGroongaBM25Retriever` retrieves Documents from the `SupabaseGroongaDocumentStore` using [PGroonga](https://pgroonga.github.io/), a PostgreSQL extension for fast, multilingual full-text search.
Unlike embedding-based retrievers, this Retriever works with plain text queries and requires no embeddings. It supports a wide range of languages out of the box through PGroonga's multilingual indexing capabilities.
The Retriever can be combined with `SupabasePgvectorEmbeddingRetriever` and a [`DocumentJoiner`](../joiners/documentjoiner.mdx) for hybrid search pipelines that take advantage of both keyword and semantic retrieval.
You can also use of the [Smart Pipeline Connections](https://docs.haystack.deepset.ai/docs/smart-pipeline-connections) and skip the `DocumentJoiner` if you want to combine the results of both retrievers in a RAG pipeline.
In addition to `query`, the Retriever accepts optional parameters including `top_k` (the maximum number of Documents to retrieve) and `filters` to narrow the search space.
## Prerequisites
PGroonga must be enabled in your Supabase project. Run the following SQL in the Supabase SQL editor:
```sql
CREATE EXTENSION IF NOT EXISTS pgroonga;
```
You also need to create a SQL function that PGroonga uses for search. See the [integration README](https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/supabase/) for the required function definition.
## Installation
```shell
pip install supabase-haystack
```
## Usage
### On its own
This Retriever needs the `SupabaseGroongaDocumentStore` and indexed Documents to run.
Set the `SUPABASE_URL` and `SUPABASE_SERVICE_KEY` environment variables for your Supabase project.
```python
from haystack_integrations.document_stores.supabase import SupabaseGroongaDocumentStore
from haystack_integrations.components.retrievers.supabase import (
SupabaseGroongaBM25Retriever,
)
from haystack.utils import Secret
document_store = SupabaseGroongaDocumentStore(
supabase_url="https://<project-ref>.supabase.co",
supabase_key=Secret.from_env_var("SUPABASE_SERVICE_KEY"),
table_name="haystack_groonga_documents",
)
retriever = SupabaseGroongaBM25Retriever(document_store=document_store)
retriever.run(query="my nice query")
```
### In a RAG pipeline
The prerequisites for running this code are:
- Set an environment variable `OPENAI_API_KEY` with your OpenAI API key.
- Set an environment variable `SUPABASE_SERVICE_KEY` with your Supabase service role key.
```python
from haystack import Document, Pipeline
from haystack.components.builders.answer_builder import AnswerBuilder
from haystack.components.builders import ChatPromptBuilder
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.dataclasses import ChatMessage
from haystack.document_stores.types import DuplicatePolicy
from haystack.utils import Secret
from haystack_integrations.document_stores.supabase import SupabaseGroongaDocumentStore
from haystack_integrations.components.retrievers.supabase import (
SupabaseGroongaBM25Retriever,
)
document_store = SupabaseGroongaDocumentStore(
supabase_url="https://<project-ref>.supabase.co",
supabase_key=Secret.from_env_var("SUPABASE_SERVICE_KEY"),
table_name="haystack_groonga_documents",
)
documents = [
Document(content="There are over 7,000 languages spoken around the world today."),
Document(
content="Elephants have been observed to behave in a way that indicates a high level of self-awareness, such as recognizing themselves in mirrors.",
),
Document(
content="In certain parts of the world, like the Maldives, Puerto Rico, and San Diego, you can witness the phenomenon of bioluminescent waves.",
),
]
document_store.write_documents(documents=documents, policy=DuplicatePolicy.SKIP)
prompt_template = [
ChatMessage.from_user(
"Given these documents, answer the question.\nDocuments:\n"
"{% for doc in documents %}{{ doc.content }}{% endfor %}\n"
"Question: {{question}}\nAnswer:",
),
]
retriever = SupabaseGroongaBM25Retriever(document_store=document_store)
rag_pipeline = Pipeline()
rag_pipeline.add_component(name="retriever", instance=retriever)
rag_pipeline.add_component(
instance=ChatPromptBuilder(
template=prompt_template,
required_variables={"question", "documents"},
),
name="prompt_builder",
)
rag_pipeline.add_component(instance=OpenAIChatGenerator(), name="llm")
rag_pipeline.add_component(instance=AnswerBuilder(), name="answer_builder")
rag_pipeline.connect("retriever", "prompt_builder.documents")
rag_pipeline.connect("prompt_builder.prompt", "llm.messages")
rag_pipeline.connect("llm.replies", "answer_builder.replies")
rag_pipeline.connect("retriever", "answer_builder.documents")
question = "languages spoken around the world today"
result = rag_pipeline.run(
{
"retriever": {"query": question},
"prompt_builder": {"question": question},
"answer_builder": {"query": question},
},
)
print(result["answer_builder"])
```
docs-website/docs/pipeline-components/retrievers/supabasepgvectorembeddingretriever.mdx
+115
View File
@@ -0,0 +1,115 @@
---
title: "SupabasePgvectorEmbeddingRetriever"
id: supabasepgvectorembeddingretriever
slug: "/supabasepgvectorembeddingretriever"
description: "An embedding-based Retriever compatible with the SupabasePgvectorDocumentStore."
---
# SupabasePgvectorEmbeddingRetriever
An embedding-based Retriever compatible with the SupabasePgvectorDocumentStore.
<div className="key-value-table">
| | |
| --- | --- |
| **Most common position in a pipeline** | 1. After a Text Embedder and before a [`PromptBuilder`](../builders/promptbuilder.mdx) in a RAG pipeline 2. The last component in the semantic search pipeline 3. After a Text Embedder and before an [`ExtractiveReader`](../readers/extractivereader.mdx) in an extractive QA pipeline |
| **Mandatory init variables** | `document_store`: An instance of a [SupabasePgvectorDocumentStore](../../document-stores/supabasedocumentstore.mdx) |
| **Mandatory run variables** | `query_embedding`: A vector representing the query (a list of floats) |
| **Output variables** | `documents`: A list of documents |
| **API reference** | [Supabase](/reference/integrations-supabase) |
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/supabase |
| **Package name** | `supabase-haystack` |
</div>
## Overview
`SupabasePgvectorEmbeddingRetriever` is a thin wrapper around [`PgvectorEmbeddingRetriever`](pgvectorembeddingretriever.mdx), adapted for use with `SupabasePgvectorDocumentStore`. It compares the query and Document embeddings and fetches the Documents most relevant to the query based on vector similarity.
When using this Retriever in your pipeline, make sure embeddings are available. Add a Document Embedder to your indexing pipeline and a Text Embedder to your query pipeline.
In addition to `query_embedding`, the Retriever accepts optional parameters including `top_k` (the maximum number of Documents to retrieve), `filters` to narrow down the search space, and `vector_function` to override the similarity function set on the Document Store.
Some relevant parameters that impact embedding retrieval must be defined when the `SupabasePgvectorDocumentStore` is initialized: `embedding_dimension`, `vector_function`, and `search_strategy` (`"exact_nearest_neighbor"` or `"hnsw"`).
## Installation
```shell
pip install supabase-haystack
```
## Usage
### On its own
This Retriever needs the `SupabasePgvectorDocumentStore` and indexed Documents to run.
Set the `SUPABASE_DB_URL` environment variable with your Supabase database connection string.
```python
from haystack_integrations.document_stores.supabase import SupabasePgvectorDocumentStore
from haystack_integrations.components.retrievers.supabase import (
SupabasePgvectorEmbeddingRetriever,
)
document_store = SupabasePgvectorDocumentStore(embedding_dimension=768)
retriever = SupabasePgvectorEmbeddingRetriever(document_store=document_store)
# using a fake vector to keep the example simple
retriever.run(query_embedding=[0.1] * 768)
```
### In a Pipeline
```python
from haystack import Document, Pipeline
from haystack.document_stores.types import DuplicatePolicy
from haystack.components.embedders import (
SentenceTransformersTextEmbedder,
SentenceTransformersDocumentEmbedder,
)
from haystack_integrations.document_stores.supabase import SupabasePgvectorDocumentStore
from haystack_integrations.components.retrievers.supabase import (
SupabasePgvectorEmbeddingRetriever,
)
document_store = SupabasePgvectorDocumentStore(
embedding_dimension=768,
vector_function="cosine_similarity",
recreate_table=True,
)
documents = [
Document(content="There are over 7,000 languages spoken around the world today."),
Document(
content="Elephants have been observed to behave in a way that indicates a high level of self-awareness, such as recognizing themselves in mirrors.",
),
Document(
content="In certain parts of the world, like the Maldives, Puerto Rico, and San Diego, you can witness the phenomenon of bioluminescent waves.",
),
]
document_embedder = SentenceTransformersDocumentEmbedder()
documents_with_embeddings = document_embedder.run(documents)
document_store.write_documents(
documents_with_embeddings.get("documents"),
policy=DuplicatePolicy.OVERWRITE,
)
query_pipeline = Pipeline()
query_pipeline.add_component("text_embedder", SentenceTransformersTextEmbedder())
query_pipeline.add_component(
"retriever",
SupabasePgvectorEmbeddingRetriever(document_store=document_store),
)
query_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")
query = "How many languages are there?"
result = query_pipeline.run({"text_embedder": {"text": query}})
print(result["retriever"]["documents"][0])
```
docs-website/docs/pipeline-components/retrievers/supabasepgvectorkeywordretriever.mdx
+135
View File
@@ -0,0 +1,135 @@
---
title: "SupabasePgvectorKeywordRetriever"
id: supabasepgvectorkeywordretriever
slug: "/supabasepgvectorkeywordretriever"
description: "A keyword-based Retriever that fetches documents matching a query from the SupabasePgvectorDocumentStore."
---
# SupabasePgvectorKeywordRetriever
A keyword-based Retriever that fetches documents matching a query from the SupabasePgvectorDocumentStore.
<div className="key-value-table">
| | |
| --- | --- |
| **Most common position in a pipeline** | 1. Before a [`PromptBuilder`](../builders/promptbuilder.mdx) in a RAG pipeline 2. The last component in the semantic search pipeline 3. Before an [`ExtractiveReader`](../readers/extractivereader.mdx) in an extractive QA pipeline |
| **Mandatory init variables** | `document_store`: An instance of a [SupabasePgvectorDocumentStore](../../document-stores/supabasedocumentstore.mdx) |
| **Mandatory run variables** | `query`: A string |
| **Output variables** | `documents`: A list of documents (matching the query) |
| **API reference** | [Supabase](/reference/integrations-supabase) |
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/supabase |
| **Package name** | `supabase-haystack` |
</div>
## Overview
`SupabasePgvectorKeywordRetriever` is a thin wrapper around [`PgvectorKeywordRetriever`](pgvectorkeywordretriever.mdx), adapted for use with `SupabasePgvectorDocumentStore`.
It uses PostgreSQL full-text search (`to_tsvector` / `plainto_tsquery`) to find Documents and ranks them with the `ts_rank_cd` function. The ranking considers how often the query terms appear in the Document, how close together the terms are, and how important the part of the Document is where they occur. For more details, see the [PostgreSQL documentation](https://www.postgresql.org/docs/current/textsearch-controls.html#TEXTSEARCH-RANKING).
Keep in mind that, unlike similar components such as `ElasticsearchBM25Retriever`, this Retriever does not apply fuzzy search out of the box, so it's necessary to carefully formulate the query in order to avoid getting zero results.
The language used to parse query and Document content for keyword retrieval is set via the `language` parameter on the `SupabasePgvectorDocumentStore` (defaults to `"english"`).
In addition to the `query`, the Retriever accepts optional parameters including `top_k` (the maximum number of Documents to retrieve) and `filters` to narrow the search space.
## Installation
```shell
pip install supabase-haystack
```
## Usage
### On its own
This Retriever needs the `SupabasePgvectorDocumentStore` and indexed Documents to run.
Set the `SUPABASE_DB_URL` environment variable with your Supabase database connection string.
```python
from haystack_integrations.document_stores.supabase import SupabasePgvectorDocumentStore
from haystack_integrations.components.retrievers.supabase import (
SupabasePgvectorKeywordRetriever,
)
document_store = SupabasePgvectorDocumentStore()
retriever = SupabasePgvectorKeywordRetriever(document_store=document_store)
retriever.run(query="my nice query")
```
### In a RAG pipeline
The prerequisites for running this code are:
- Set an environment variable `OPENAI_API_KEY` with your OpenAI API key.
- Set an environment variable `SUPABASE_DB_URL` with the connection string to your Supabase database.
```python
from haystack import Document, Pipeline
from haystack.components.builders.answer_builder import AnswerBuilder
from haystack.components.builders import ChatPromptBuilder
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.dataclasses import ChatMessage
from haystack.document_stores.types import DuplicatePolicy
from haystack_integrations.document_stores.supabase import SupabasePgvectorDocumentStore
from haystack_integrations.components.retrievers.supabase import (
SupabasePgvectorKeywordRetriever,
)
prompt_template = [
ChatMessage.from_user(
"Given these documents, answer the question.\nDocuments:\n"
"{% for doc in documents %}{{ doc.content }}{% endfor %}\n"
"Question: {{question}}\nAnswer:",
),
]
document_store = SupabasePgvectorDocumentStore(
language="english",
recreate_table=True,
)
documents = [
Document(content="There are over 7,000 languages spoken around the world today."),
Document(
content="Elephants have been observed to behave in a way that indicates a high level of self-awareness, such as recognizing themselves in mirrors.",
),
Document(
content="In certain parts of the world, like the Maldives, Puerto Rico, and San Diego, you can witness the phenomenon of bioluminescent waves.",
),
]
document_store.write_documents(documents=documents, policy=DuplicatePolicy.SKIP)
retriever = SupabasePgvectorKeywordRetriever(document_store=document_store)
rag_pipeline = Pipeline()
rag_pipeline.add_component(name="retriever", instance=retriever)
rag_pipeline.add_component(
instance=ChatPromptBuilder(
template=prompt_template,
required_variables={"question", "documents"},
),
name="prompt_builder",
)
rag_pipeline.add_component(instance=OpenAIChatGenerator(), name="llm")
rag_pipeline.add_component(instance=AnswerBuilder(), name="answer_builder")
rag_pipeline.connect("retriever", "prompt_builder.documents")
rag_pipeline.connect("prompt_builder.prompt", "llm.messages")
rag_pipeline.connect("llm.replies", "answer_builder.replies")
rag_pipeline.connect("retriever", "answer_builder.documents")
question = "languages spoken around the world today"
result = rag_pipeline.run(
{
"retriever": {"query": question},
"prompt_builder": {"question": question},
"answer_builder": {"query": question},
},
)
print(result["answer_builder"])
```
docs-website/docs/pipeline-components/writers/cogneewriter.mdx
+136
View File
@@ -0,0 +1,136 @@
---
title: "CogneeWriter"
id: cogneewriter
slug: "/cogneewriter"
description: "Writes ChatMessage objects to a CogneeMemoryStore as long-term memories."
---
# CogneeWriter
Writes `ChatMessage` objects to a `CogneeMemoryStore` as long-term memories.
<div className="key-value-table">
| | |
| --- | --- |
| **Most common position in a pipeline** | After an [`Agent`](../agents-1/agent.mdx) or Chat Generator in memory-augmented pipelines |
| **Mandatory init variables** | `memory_store`: A `CogneeMemoryStore` instance |
| **Optional init variables** | `session_id`: When set, writes target the session-cache tier; when `None`, writes go to the permanent knowledge graph |
| **Mandatory run variables** | `messages`: A list of `ChatMessage` objects |
| **Optional run variables** | `user_id`: Cognee user ID to scope the write; pass `None` to use Cognee's default user |
| **Output variables** | `messages_written`: The list of `ChatMessage` objects that were written (passed through unchanged) |
| **API reference** | [Cognee](/reference/integrations-cognee#cogneewriter) |
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/cognee |
| **Package name** | `cognee-haystack` |
</div>
## Overview
`CogneeWriter` persists a list of `ChatMessage` objects into a `CogneeMemoryStore`. Use it in a Haystack Pipeline to store conversation facts or user preferences after an Agent turn.
Messages are passed through unchanged to the pipeline output (`messages_written`), making this component easy to chain after an Agent or generator without breaking the pipeline flow.
The `session_id` init parameter controls which Cognee memory tier is targeted:
- Omit `session_id` (or set it to `None`) to write to the **permanent knowledge graph** — Cognee runs LLM extraction during ingestion, producing rich graph-completion-ready nodes.
- Set `session_id` to write to the **session cache** — fast writes with no LLM extraction, scoped to that session. Session content can later be promoted to the permanent graph via `CogneeMemoryStore.improve()`.
The writer's `session_id` overrides the store's `session_id` per call, so a single store can back multiple writers targeting different memory tiers.
## Installation
Install the Cognee integration:
```bash
pip install cognee-haystack
```
Set your LLM API key (used by Cognee for graph extraction):
```bash
export LLM_API_KEY="your-llm-api-key"
```
Optionally, set a separate embedding API key (defaults to `LLM_API_KEY` when unset):
```bash
export EMBEDDING_API_KEY="your-embedding-api-key"
```
## Usage
### On its own
```python
from haystack.dataclasses import ChatMessage
from haystack_integrations.components.writers.cognee import CogneeWriter
from haystack_integrations.memory_stores.cognee import CogneeMemoryStore
store = CogneeMemoryStore()
writer = CogneeWriter(memory_store=store)
result = writer.run(
messages=[ChatMessage.from_user("Alice prefers concise Python examples.")],
user_id="a1b2c3d4-e5f6-7890-abcd-ef1234567890",
)
print(result["messages_written"])
```
To write to the session cache instead of the permanent graph, pass a `session_id`:
```python
session_writer = CogneeWriter(memory_store=store, session_id="alice_session_1")
session_writer.run(
messages=[ChatMessage.from_user("Alice is currently debugging a vector store issue.")],
user_id="a1b2c3d4-e5f6-7890-abcd-ef1234567890",
)
```
### In a Pipeline
This example connects an Agent's full `messages` output to `CogneeWriter`, so Cognee stores the conversation turn in the permanent graph.
```python
from haystack import Pipeline
from haystack.components.agents import Agent
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.dataclasses import ChatMessage
from haystack_integrations.components.writers.cognee import CogneeWriter
from haystack_integrations.memory_stores.cognee import CogneeMemoryStore
store = CogneeMemoryStore(dataset_name="my_agent_memory")
pipeline = Pipeline()
pipeline.add_component(
"agent",
Agent(
chat_generator=OpenAIChatGenerator(model="gpt-4o-mini"),
system_prompt=(
"Answer the user and preserve durable user facts or preferences for future conversations."
),
),
)
pipeline.add_component("writer", CogneeWriter(memory_store=store))
pipeline.connect("agent.messages", "writer.messages")
result = pipeline.run(
{
"agent": {
"messages": [
ChatMessage.from_user(
"My name is Alice and I prefer concise Python examples.",
),
],
},
"writer": {
"user_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
},
},
)
print(result["writer"]["messages_written"])
```
docs-website/reference/integrations-api/amazon_bedrock.md
+6 -4
View File
@@ -1311,7 +1311,7 @@ Deserializes the component from a dictionary.
```python
run(
messages: list[ChatMessage],
messages: list[ChatMessage] | str,
streaming_callback: StreamingCallbackT | None = None,
generation_kwargs: dict[str, Any] | None = None,
tools: ToolsType | None = None,
@@ -1324,7 +1324,8 @@ Supports both standard and streaming responses depending on whether a streaming
**Parameters:**
- **messages** (<code>list\[ChatMessage\]</code>) – A list of `ChatMessage` objects forming the chat history.
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of `ChatMessage` objects forming the chat history.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – Optional callback for handling streaming outputs.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Optional dictionary of generation parameters. Some common parameters are:
- `maxTokens`: Maximum number of tokens to generate.
@@ -1347,7 +1348,7 @@ Supports both standard and streaming responses depending on whether a streaming
```python
run_async(
messages: list[ChatMessage],
messages: list[ChatMessage] | str,
streaming_callback: StreamingCallbackT | None = None,
generation_kwargs: dict[str, Any] | None = None,
tools: ToolsType | None = None,
@@ -1360,7 +1361,8 @@ Designed for use cases where non-blocking or concurrent execution is desired.
**Parameters:**
- **messages** (<code>list\[ChatMessage\]</code>) – A list of `ChatMessage` objects forming the chat history.
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of `ChatMessage` objects forming the chat history.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – Optional async-compatible callback for handling streaming outputs.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Optional dictionary of generation parameters. Some common parameters are:
- `maxTokens`: Maximum number of tokens to generate.
docs-website/reference/integrations-api/anthropic.md
+12 -8
View File
@@ -167,7 +167,7 @@ Deserialize this component from a dictionary.
```python
run(
messages: list[ChatMessage],
messages: list[ChatMessage] | str,
streaming_callback: StreamingCallbackT | None = None,
generation_kwargs: dict[str, Any] | None = None,
tools: ToolsType | None = None,
@@ -178,7 +178,8 @@ Invokes the Anthropic API with the given messages and generation kwargs.
**Parameters:**
- **messages** (<code>list\[ChatMessage\]</code>) – A list of ChatMessage instances representing the input messages.
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of ChatMessage instances representing the input messages.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Optional arguments to pass to the Anthropic generation endpoint.
- **tools** (<code>ToolsType | None</code>) – A list of Tool and/or Toolset objects, or a single Toolset, that the model can use.
@@ -194,7 +195,7 @@ Invokes the Anthropic API with the given messages and generation kwargs.
```python
run_async(
messages: list[ChatMessage],
messages: list[ChatMessage] | str,
streaming_callback: StreamingCallbackT | None = None,
generation_kwargs: dict[str, Any] | None = None,
tools: ToolsType | None = None,
@@ -205,7 +206,8 @@ Async version of the run method. Invokes the Anthropic API with the given messag
**Parameters:**
- **messages** (<code>list\[ChatMessage\]</code>) – A list of ChatMessage instances representing the input messages.
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of ChatMessage instances representing the input messages.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Optional arguments to pass to the Anthropic generation endpoint.
- **tools** (<code>ToolsType | None</code>) – A list of Tool and/or Toolset objects, or a single Toolset, that the model can use.
@@ -360,7 +362,7 @@ This method is idempotent — it only creates clients once.
```python
run(
messages: list[ChatMessage],
messages: list[ChatMessage] | str,
streaming_callback: StreamingCallbackT | None = None,
generation_kwargs: dict[str, Any] | None = None,
tools: ToolsType | None = None,
@@ -371,7 +373,8 @@ Invokes the AnthropicFoundry API with the given messages and generation kwargs.
**Parameters:**
- **messages** (<code>list\[ChatMessage\]</code>) – A list of ChatMessage instances representing the input messages.
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of ChatMessage instances representing the input messages.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Optional arguments to pass to the Anthropic generation endpoint.
- **tools** (<code>ToolsType | None</code>) – A list of Tool and/or Toolset objects, or a single Toolset, that the model can use.
@@ -387,7 +390,7 @@ Invokes the AnthropicFoundry API with the given messages and generation kwargs.
```python
run_async(
messages: list[ChatMessage],
messages: list[ChatMessage] | str,
streaming_callback: StreamingCallbackT | None = None,
generation_kwargs: dict[str, Any] | None = None,
tools: ToolsType | None = None,
@@ -398,7 +401,8 @@ Async version of the run method. Invokes the AnthropicFoundry API with the given
**Parameters:**
- **messages** (<code>list\[ChatMessage\]</code>) – A list of ChatMessage instances representing the input messages.
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of ChatMessage instances representing the input messages.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Optional arguments to pass to the Anthropic generation endpoint.
- **tools** (<code>ToolsType | None</code>) – A list of Tool and/or Toolset objects, or a single Toolset, that the model can use.
docs-website/reference/integrations-api/arangodb.md
+244
View File
@@ -0,0 +1,244 @@
---
title: "Arangodb"
id: integrations-arangodb
description: "Arangodb integration for Haystack"
slug: "/integrations-arangodb"
---
## haystack_integrations.components.retrievers.arangodb.embedding_retriever
### ArangoEmbeddingRetriever
Retrieves documents from an `ArangoDocumentStore` using vector similarity on embeddings.
The similarity function is configured on the `ArangoDocumentStore` (cosine, dot product, or L2).
Example usage:
```python
from haystack_integrations.document_stores.arangodb import ArangoDocumentStore
from haystack_integrations.components.retrievers.arangodb import ArangoEmbeddingRetriever
store = ArangoDocumentStore(host="http://localhost:8529", database="haystack",
username="root", collection_name="docs", embedding_dimension=768)
retriever = ArangoEmbeddingRetriever(document_store=store, top_k=5)
result = retriever.run(query_embedding=[0.1, 0.2, ...])
```
#### __init__
```python
__init__(
*,
document_store: ArangoDocumentStore,
top_k: int = 10,
filters: dict[str, Any] | None = None
) -> None
```
Creates a new ArangoEmbeddingRetriever.
**Parameters:**
- **document_store** (<code>ArangoDocumentStore</code>) – The `ArangoDocumentStore` to retrieve documents from.
- **top_k** (<code>int</code>) – Maximum number of documents to return.
- **filters** (<code>dict\[str, Any\] | None</code>) – Optional Haystack metadata filters applied at retrieval time.
#### run
```python
run(
query_embedding: list[float],
top_k: int | None = None,
filters: dict[str, Any] | None = None,
) -> dict[str, list[Document]]
```
Retrieves documents most similar to `query_embedding`.
**Parameters:**
- **query_embedding** (<code>list\[float\]</code>) – The query vector.
- **top_k** (<code>int | None</code>) – Overrides the instance-level `top_k` for this call.
- **filters** (<code>dict\[str, Any\] | None</code>) – Overrides the instance-level `filters` for this call.
**Returns:**
- <code>dict\[str, list\[Document\]\]</code> – A dictionary with `documents` — a list of `Document` objects sorted by score.
#### to_dict
```python
to_dict() -> dict[str, Any]
```
Serializes the component to a dictionary.
**Returns:**
- <code>dict\[str, Any\]</code> – Dictionary with serialized data.
#### from_dict
```python
from_dict(data: dict[str, Any]) -> ArangoEmbeddingRetriever
```
Deserializes the component from a dictionary.
**Parameters:**
- **data** (<code>dict\[str, Any\]</code>) – Dictionary to deserialize from.
**Returns:**
- <code>ArangoEmbeddingRetriever</code> – Deserialized component.
## haystack_integrations.document_stores.arangodb.document_store
### ArangoDocumentStore
A Haystack DocumentStore backed by [ArangoDB](https://www.arangodb.com/).
Documents are stored in an ArangoDB collection and support vector similarity search
via AQL vector functions (requires ArangoDB 3.12+).
Example usage:
```python
from haystack_integrations.document_stores.arangodb import ArangoDocumentStore
from haystack.utils import Secret
store = ArangoDocumentStore(
host="http://localhost:8529",
database="haystack",
username=Secret.from_env_var("ARANGO_USERNAME", strict=False),
password=Secret.from_env_var("ARANGO_PASSWORD"),
collection_name="documents",
embedding_dimension=768,
)
```
#### __init__
```python
__init__(
*,
host: str = "http://localhost:8529",
database: str = "haystack",
username: Secret = Secret.from_env_var("ARANGO_USERNAME", strict=False),
password: Secret = Secret.from_env_var("ARANGO_PASSWORD"),
collection_name: str = "haystack_documents",
embedding_dimension: int = 768,
recreate_collection: bool = False,
similarity_function: Literal["cosine", "dot_product", "l2"] = "cosine"
) -> None
```
Creates a new ArangoDocumentStore instance.
**Parameters:**
- **host** (<code>str</code>) – ArangoDB server URL, e.g. `http://localhost:8529`.
- **database** (<code>str</code>) – Name of the ArangoDB database to use. Created if it does not exist.
- **username** (<code>Secret</code>) – ArangoDB username as a `Secret`. Defaults to `ARANGO_USERNAME` env var,
falling back to `root` if the variable is not set.
- **password** (<code>Secret</code>) – ArangoDB password as a `Secret`. Defaults to `ARANGO_PASSWORD` env var.
- **collection_name** (<code>str</code>) – Name of the collection to store documents in.
- **embedding_dimension** (<code>int</code>) – Dimensionality of document embeddings.
- **recreate_collection** (<code>bool</code>) – If `True`, drop and recreate the collection on startup.
- **similarity_function** (<code>Literal['cosine', 'dot_product', 'l2']</code>) – Vector similarity function to use for embedding retrieval.
One of `"cosine"` (default), `"dot_product"`, or `"l2"`.
#### count_documents
```python
count_documents() -> int
```
Returns the number of documents in the store.
**Returns:**
- <code>int</code> – Document count.
#### filter_documents
```python
filter_documents(filters: dict[str, Any] | None = None) -> list[Document]
```
Returns documents matching the provided filters.
**Parameters:**
- **filters** (<code>dict\[str, Any\] | None</code>) – Haystack metadata filters. If `None`, all documents are returned.
**Returns:**
- <code>list\[Document\]</code> – List of matching `Document` objects.
#### write_documents
```python
write_documents(
documents: list[Document], policy: DuplicatePolicy = DuplicatePolicy.NONE
) -> int
```
Writes documents to the store.
**Parameters:**
- **documents** (<code>list\[Document\]</code>) – Documents to write.
- **policy** (<code>DuplicatePolicy</code>) – How to handle duplicates — `OVERWRITE`, `SKIP`, or `FAIL` (default).
**Returns:**
- <code>int</code> – Number of documents written.
**Raises:**
- <code>ValueError</code> – If `documents` contains non-`Document` objects.
- <code>DuplicateDocumentError</code> – If a duplicate is found and policy is `FAIL`.
#### delete_documents
```python
delete_documents(document_ids: list[str]) -> None
```
Deletes documents by their IDs.
**Parameters:**
- **document_ids** (<code>list\[str\]</code>) – List of document IDs to delete.
#### to_dict
```python
to_dict() -> dict[str, Any]
```
Serializes the component to a dictionary.
**Returns:**
- <code>dict\[str, Any\]</code> – Dictionary with serialized data.
#### from_dict
```python
from_dict(data: dict[str, Any]) -> ArangoDocumentStore
```
Deserializes the component from a dictionary.
**Parameters:**
- **data** (<code>dict\[str, Any\]</code>) – Dictionary to deserialize from.
**Returns:**
- <code>ArangoDocumentStore</code> – Deserialized component.
docs-website/reference/integrations-api/cohere.md
+6 -4
View File
@@ -729,7 +729,7 @@ Deserializes the component from a dictionary.
```python
run(
messages: list[ChatMessage],
messages: list[ChatMessage] | str,
generation_kwargs: dict[str, Any] | None = None,
tools: ToolsType | None = None,
streaming_callback: StreamingCallbackT | None = None,
@@ -740,7 +740,8 @@ Invoke the chat endpoint based on the provided messages and generation parameter
**Parameters:**
- **messages** (<code>list\[ChatMessage\]</code>) – list of `ChatMessage` instances representing the input messages.
- **messages** (<code>list\[ChatMessage\] | str</code>) – list of `ChatMessage` instances representing the input messages.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – additional keyword arguments for chat generation. These parameters will
potentially override the parameters passed in the __init__ method.
For more details on the parameters supported by the Cohere API, refer to the
@@ -759,7 +760,7 @@ Invoke the chat endpoint based on the provided messages and generation parameter
```python
run_async(
messages: list[ChatMessage],
messages: list[ChatMessage] | str,
generation_kwargs: dict[str, Any] | None = None,
tools: ToolsType | None = None,
streaming_callback: StreamingCallbackT | None = None,
@@ -770,7 +771,8 @@ Asynchronously invoke the chat endpoint based on the provided messages and gener
**Parameters:**
- **messages** (<code>list\[ChatMessage\]</code>) – list of `ChatMessage` instances representing the input messages.
- **messages** (<code>list\[ChatMessage\] | str</code>) – list of `ChatMessage` instances representing the input messages.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – additional keyword arguments for chat generation. These parameters will
potentially override the parameters passed in the __init__ method.
For more details on the parameters supported by the Cohere API, refer to the
docs-website/reference/integrations-api/docling_serve.md
+30 -1
View File
@@ -18,6 +18,27 @@ Enumeration of export formats supported by DoclingServe.
- `TEXT`: Extracts plain text.
- `JSON`: Returns the full Docling document as a JSON string.
### ConversionMode
Bases: <code>str</code>, <code>Enum</code>
Execution mode for DoclingServe conversions.
- `SYNC`: Uses DoclingServe's synchronous conversion endpoints.
- `ASYNC`: Uses DoclingServe's async job endpoints and polls for completion.
### DoclingServeConversionError
Bases: <code>Exception</code>
Raised when DoclingServe reports an async task or conversion failure.
### DoclingServeTimeoutError
Bases: <code>DoclingServeConversionError</code>
Raised when a DoclingServe async task exceeds job_timeout.
### DoclingServeConverter
Converts documents to Haystack Documents using a DoclingServe server.
@@ -54,7 +75,10 @@ __init__(
timeout: float = 120.0,
api_key: Secret | None = Secret.from_env_var(
"DOCLING_SERVE_API_KEY", strict=False
)
),
mode: ConversionMode | str = ConversionMode.SYNC,
poll_interval: float = 2.0,
job_timeout: float = 600.0
) -> None
```
@@ -73,6 +97,11 @@ Initializes the DoclingServeConverter.
- **api_key** (<code>Secret | None</code>) – API key for authenticating with a secured DoclingServe instance. Reads from the
`DOCLING_SERVE_API_KEY` environment variable by default. Set to `None` to disable
authentication.
- **mode** (<code>ConversionMode | str</code>) – Conversion mode. `sync` uses DoclingServe's synchronous endpoints. `async` submits
conversion jobs to DoclingServe's async endpoints and polls until completion.
- **poll_interval** (<code>float</code>) – Controls both the server-side long-poll wait (?wait= parameter) and the maximum local sleep between polls.
A higher value reduces round-trips; a lower value increases polling frequency.
- **job_timeout** (<code>float</code>) – Maximum time in seconds to wait for each async conversion job.
#### to_dict
docs-website/reference/integrations-api/funasr.md
+157
View File
@@ -0,0 +1,157 @@
---
title: "FunASR"
id: integrations-funasr
description: "FunASR speech-to-text integration for Haystack"
slug: "/integrations-funasr"
---
## haystack_integrations.components.audio.funasr.transcriber
### FunASRTranscriber
Transcribes audio files to Documents using [FunASR](https://github.com/modelscope/FunASR).
FunASR is an open-source speech recognition toolkit from Alibaba DAMO Academy.
It supports 50+ languages, speaker diarization, and timestamp extraction, and runs
entirely locally — no API key required.
Models are downloaded from ModelScope on first use and cached in `~/.cache/modelscope`.
**Usage Example:**
```python
from haystack_integrations.components.audio.funasr import FunASRTranscriber
transcriber = FunASRTranscriber()
result = transcriber.run(sources=["speech.wav", "interview.mp3"])
documents = result["documents"]
```
**Speaker diarization and punctuation:**
```python
from haystack.utils import ComponentDevice
transcriber = FunASRTranscriber(
model="paraformer-zh",
vad_model="fsmn-vad",
punc_model="ct-punc",
spk_model="cam++",
device=ComponentDevice.from_str("cuda"),
)
```
**SenseVoice with inverse text normalisation:**
```python
transcriber = FunASRTranscriber(
model="iic/SenseVoiceSmall",
generation_kwargs={"use_itn": True, "merge_vad": True, "language": "auto"},
)
```
#### __init__
```python
__init__(
*,
model: str = "iic/SenseVoiceSmall",
vad_model: str | None = "fsmn-vad",
punc_model: str | None = "ct-punc",
spk_model: str | None = None,
device: ComponentDevice | None = None,
batch_size_s: int = 300,
store_full_path: bool = False,
generation_kwargs: dict[str, Any] | None = None
) -> None
```
Create a FunASRTranscriber component.
**Parameters:**
- **model** (<code>str</code>) – FunASR model name or local path. Defaults to `"iic/SenseVoiceSmall"`,
a multilingual model supporting 50+ languages that is 5-10x faster than Whisper.
Alternatives include `"paraformer-zh"` (Chinese) or `"paraformer-en"` (English).
Browse available models at https://modelscope.github.io/FunASR/model-selection.html.
- **vad_model** (<code>str | None</code>) – Voice activity detection model used to split long audio into segments.
Set to `None` to process the audio as a single stream.
Browse available VAD models at https://www.modelscope.cn/models.
- **punc_model** (<code>str | None</code>) – Punctuation restoration model. Set to `None` to disable punctuation.
Browse available punctuation models at https://www.modelscope.cn/models.
- **spk_model** (<code>str | None</code>) – Speaker diarization model (e.g. `"cam++"`). When set, a `"speakers"`
key is included in the Document metadata. Defaults to `None` (diarization disabled).
Browse available speaker diarization models at https://www.modelscope.cn/models.
- **device** (<code>ComponentDevice | None</code>) – The device to run inference on. If `None`, the default device is selected
automatically. Use `ComponentDevice.from_str("cuda")` for GPU inference.
- **batch_size_s** (<code>int</code>) – Batch size in seconds for VAD-segmented audio. Larger values
improve throughput at the cost of memory.
- **store_full_path** (<code>bool</code>) – If `True`, store the full audio file path in Document metadata.
If `False` (default), store only the file name.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Extra keyword arguments forwarded to `AutoModel.generate()`.
Use this for model-specific options such as `use_itn=True` or `merge_vad=True`
for SenseVoice, or `hotword="..."` for contextual recognition.
#### warm_up
```python
warm_up() -> None
```
Load the FunASR model into memory.
Models are downloaded from ModelScope on first call and cached locally.
This method is idempotent — calling it multiple times is safe.
#### to_dict
```python
to_dict() -> dict[str, Any]
```
Serialize the component to a dictionary.
**Returns:**
- <code>dict\[str, Any\]</code> – Dictionary with serialized data.
#### from_dict
```python
from_dict(data: dict[str, Any]) -> FunASRTranscriber
```
Deserialize the component from a dictionary.
**Parameters:**
- **data** (<code>dict\[str, Any\]</code>) – Dictionary to deserialize from.
**Returns:**
- <code>FunASRTranscriber</code> – Deserialized component.
#### run
```python
run(
sources: list[str | Path | ByteStream],
meta: dict[str, Any] | list[dict[str, Any]] | None = None,
) -> dict[str, list[Document]]
```
Transcribe audio sources to Documents.
**Parameters:**
- **sources** (<code>list\[str | Path | ByteStream\]</code>) – Audio file paths (`str` or `Path`) or `ByteStream` objects.
Supported formats: WAV, MP3, FLAC, OGG, M4A, AAC, and any format that
FunASR's underlying audio backend (soundfile/ffmpeg) can decode.
- **meta** (<code>dict\[str, Any\] | list\[dict\[str, Any\]\] | None</code>) – Metadata to attach to the produced Documents. Pass a single dict
to apply the same metadata to all Documents, or a list aligned with `sources`.
**Returns:**
- <code>dict\[str, list\[Document\]\]</code> – Dictionary with key `"documents"` — one `Document` per source whose
`content` holds the full transcript text.
docs-website/reference/integrations-api/google_genai.md
+49 -7
View File
@@ -81,7 +81,9 @@ __init__(
progress_bar: bool = True,
meta_fields_to_embed: list[str] | None = None,
embedding_separator: str = "\n",
config: dict[str, Any] | None = None
config: dict[str, Any] | None = None,
timeout: float | None = None,
max_retries: int | None = None
) -> None
```
@@ -114,6 +116,8 @@ Creates an GoogleGenAIDocumentEmbedder component.
Specifying task types in `config` does not take effect for `gemini-embedding-2`.
See [Gemini documentation](https://ai.google.dev/gemini-api/docs/embeddings#task-types) for more
information.
- **timeout** (<code>float | None</code>) – The timeout in seconds for the underlying Google GenAI client network requests.
- **max_retries** (<code>int | None</code>) – The maximum number of retries for the underlying Google GenAI client network requests.
#### to_dict
@@ -260,7 +264,9 @@ __init__(
model: str = "gemini-embedding-2",
batch_size: int = 6,
progress_bar: bool = True,
config: dict[str, Any] | None = None
config: dict[str, Any] | None = None,
timeout: float | None = None,
max_retries: int | None = None
) -> None
```
@@ -293,6 +299,36 @@ Creates an GoogleGenAIMultimodalDocumentEmbedder component.
You can for example set the output dimensionality of the embedding: `{"output_dimensionality": 768}`.
See [Google API documentation](https://googleapis.github.io/python-genai/genai.html#genai.types.EmbedContentConfig)
for the available options.
- **timeout** (<code>float | None</code>) – The timeout in seconds for the underlying Google GenAI client network requests.
- **max_retries** (<code>int | None</code>) – The maximum number of retries for the underlying Google GenAI client network requests.
#### to_dict
```python
to_dict() -> dict[str, Any]
```
Serializes the component to a dictionary.
**Returns:**
- <code>dict\[str, Any\]</code> – Dictionary with serialized data.
#### from_dict
```python
from_dict(data: dict[str, Any]) -> GoogleGenAIMultimodalDocumentEmbedder
```
Deserializes the component from a dictionary.
**Parameters:**
- **data** (<code>dict\[str, Any\]</code>) – Dictionary to deserialize from.
**Returns:**
- <code>GoogleGenAIMultimodalDocumentEmbedder</code> – Deserialized component.
#### run
@@ -405,7 +441,9 @@ __init__(
model: str = "gemini-embedding-001",
prefix: str = "",
suffix: str = "",
config: dict[str, Any] | None = None
config: dict[str, Any] | None = None,
timeout: float | None = None,
max_retries: int | None = None
) -> None
```
@@ -434,6 +472,8 @@ Creates an GoogleGenAITextEmbedder component.
Specifying task types in `config` does not take effect for `gemini-embedding-2`.
See [Gemini documentation](https://ai.google.dev/gemini-api/docs/embeddings#task-types) for more
information.
- **timeout** (<code>float | None</code>) – The timeout in seconds for the underlying Google GenAI client network requests.
- **max_retries** (<code>int | None</code>) – The maximum number of retries for the underlying Google GenAI client network requests.
#### to_dict
@@ -759,7 +799,7 @@ Deserializes the component from a dictionary.
```python
run(
messages: list[ChatMessage],
messages: list[ChatMessage] | str,
generation_kwargs: dict[str, Any] | None = None,
safety_settings: list[dict[str, Any]] | None = None,
streaming_callback: StreamingCallbackT | None = None,
@@ -771,7 +811,8 @@ Run the Google Gen AI chat generator on the given input data.
**Parameters:**
- **messages** (<code>list\[ChatMessage\]</code>) – A list of ChatMessage instances representing the input messages.
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of ChatMessage instances representing the input messages.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Configuration for generation. If provided, it will override
the default config. Supports `thinking_budget` for Gemini 2.5 series thinking configuration.
- **safety_settings** (<code>list\[dict\[str, Any\]\] | None</code>) – Safety settings for content filtering. If provided, it will override the
@@ -796,7 +837,7 @@ Run the Google Gen AI chat generator on the given input data.
```python
run_async(
messages: list[ChatMessage],
messages: list[ChatMessage] | str,
generation_kwargs: dict[str, Any] | None = None,
safety_settings: list[dict[str, Any]] | None = None,
streaming_callback: StreamingCallbackT | None = None,
@@ -808,7 +849,8 @@ Async version of the run method. Run the Google Gen AI chat generator on the giv
**Parameters:**
- **messages** (<code>list\[ChatMessage\]</code>) – A list of ChatMessage instances representing the input messages.
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of ChatMessage instances representing the input messages.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Configuration for generation. If provided, it will override
the default config. Supports `thinking_budget` for Gemini 2.5 series thinking configuration.
See https://ai.google.dev/gemini-api/docs/thinking for possible values.
docs-website/reference/integrations-api/huggingface_api.md
+796
View File
@@ -0,0 +1,796 @@
---
title: "Hugging Face API"
id: integrations-huggingface-api
description: "Hugging Face API integration for Haystack"
slug: "/integrations-huggingface-api"
---
## haystack_integrations.components.embedders.huggingface_api.document_embedder
### HuggingFaceAPIDocumentEmbedder
Embeds documents using Hugging Face APIs.
Use it with the following Hugging Face APIs:
- [Free Serverless Inference API](https://huggingface.co/inference-api)
- [Paid Inference Endpoints](https://huggingface.co/inference-endpoints)
- [Self-hosted Text Embeddings Inference](https://github.com/huggingface/text-embeddings-inference)
### Usage examples
#### With free serverless inference API
```python
from haystack_integrations.components.embedders.huggingface_api import HuggingFaceAPIDocumentEmbedder
from haystack.utils import Secret
from haystack.dataclasses import Document
doc = Document(content="I love pizza!")
doc_embedder = HuggingFaceAPIDocumentEmbedder(api_type="serverless_inference_api",
api_params={"model": "BAAI/bge-small-en-v1.5"},
token=Secret.from_token("<your-api-key>"))
result = document_embedder.run([doc])
print(result["documents"][0].embedding)
# [0.017020374536514282, -0.023255806416273117, ...]
```
#### With paid inference endpoints
```python
from haystack_integrations.components.embedders.huggingface_api import HuggingFaceAPIDocumentEmbedder
from haystack.utils import Secret
from haystack.dataclasses import Document
doc = Document(content="I love pizza!")
doc_embedder = HuggingFaceAPIDocumentEmbedder(api_type="inference_endpoints",
api_params={"url": "<your-inference-endpoint-url>"},
token=Secret.from_token("<your-api-key>"))
result = document_embedder.run([doc])
print(result["documents"][0].embedding)
# [0.017020374536514282, -0.023255806416273117, ...]
```
#### With self-hosted text embeddings inference
```python
from haystack_integrations.components.embedders.huggingface_api import HuggingFaceAPIDocumentEmbedder
from haystack.dataclasses import Document
doc = Document(content="I love pizza!")
doc_embedder = HuggingFaceAPIDocumentEmbedder(api_type="text_embeddings_inference",
api_params={"url": "http://localhost:8080"})
result = document_embedder.run([doc])
print(result["documents"][0].embedding)
# [0.017020374536514282, -0.023255806416273117, ...]
```
#### __init__
```python
__init__(
api_type: HFEmbeddingAPIType | str,
api_params: dict[str, str],
token: Secret | None = Secret.from_env_var(
["HF_API_TOKEN", "HF_TOKEN"], strict=False
),
prefix: str = "",
suffix: str = "",
truncate: bool | None = True,
normalize: bool | None = False,
batch_size: int = 32,
progress_bar: bool = True,
meta_fields_to_embed: list[str] | None = None,
embedding_separator: str = "\n",
concurrency_limit: int = 4,
) -> None
```
Creates a HuggingFaceAPIDocumentEmbedder component.
**Parameters:**
- **api_type** (<code>HFEmbeddingAPIType | str</code>) – The type of Hugging Face API to use.
- **api_params** (<code>dict\[str, str\]</code>) – A dictionary with the following keys:
- `model`: Hugging Face model ID. Required when `api_type` is `SERVERLESS_INFERENCE_API`.
- `url`: URL of the inference endpoint. Required when `api_type` is `INFERENCE_ENDPOINTS` or
`TEXT_EMBEDDINGS_INFERENCE`.
- **token** (<code>Secret | None</code>) – The Hugging Face token to use as HTTP bearer authorization.
Check your HF token in your [account settings](https://huggingface.co/settings/tokens).
- **prefix** (<code>str</code>) – A string to add at the beginning of each text.
- **suffix** (<code>str</code>) – A string to add at the end of each text.
- **truncate** (<code>bool | None</code>) – Truncates the input text to the maximum length supported by the model.
Applicable when `api_type` is `TEXT_EMBEDDINGS_INFERENCE`, or `INFERENCE_ENDPOINTS`
if the backend uses Text Embeddings Inference.
If `api_type` is `SERVERLESS_INFERENCE_API`, this parameter is ignored.
- **normalize** (<code>bool | None</code>) – Normalizes the embeddings to unit length.
Applicable when `api_type` is `TEXT_EMBEDDINGS_INFERENCE`, or `INFERENCE_ENDPOINTS`
if the backend uses Text Embeddings Inference.
If `api_type` is `SERVERLESS_INFERENCE_API`, this parameter is ignored.
- **batch_size** (<code>int</code>) – Number of documents to process at once.
- **progress_bar** (<code>bool</code>) – If `True`, shows a progress bar when running.
- **meta_fields_to_embed** (<code>list\[str\] | None</code>) – List of metadata fields to embed along with the document text.
- **embedding_separator** (<code>str</code>) – Separator used to concatenate the metadata fields to the document text.
- **concurrency_limit** (<code>int</code>) – The maximum number of requests that should be allowed to run concurrently.
This parameter is only used in the `run_async` method.
**Raises:**
- <code>ValueError</code> – If the required `model` or `url` is missing from `api_params`, the `url` is invalid,
or the `api_type` is unknown.
#### to_dict
```python
to_dict() -> dict[str, Any]
```
Serializes the component to a dictionary.
**Returns:**
- <code>dict\[str, Any\]</code> – Dictionary with serialized data.
#### from_dict
```python
from_dict(data: dict[str, Any]) -> HuggingFaceAPIDocumentEmbedder
```
Deserializes the component from a dictionary.
**Parameters:**
- **data** (<code>dict\[str, Any\]</code>) – Dictionary to deserialize from.
**Returns:**
- <code>HuggingFaceAPIDocumentEmbedder</code> – Deserialized component.
#### run
```python
run(documents: list[Document]) -> dict[str, list[Document]]
```
Embeds a list of documents.
**Parameters:**
- **documents** (<code>list\[Document\]</code>) – Documents to embed.
**Returns:**
- <code>dict\[str, list\[Document\]\]</code> – A dictionary with the following keys:
- `documents`: A list of documents with embeddings.
**Raises:**
- <code>TypeError</code> – If `documents` is not a list of Documents.
- <code>ValueError</code> – If the embeddings returned by the API have an unexpected shape.
#### run_async
```python
run_async(documents: list[Document]) -> dict[str, list[Document]]
```
Embeds a list of documents asynchronously.
**Parameters:**
- **documents** (<code>list\[Document\]</code>) – Documents to embed.
**Returns:**
- <code>dict\[str, list\[Document\]\]</code> – A dictionary with the following keys:
- `documents`: A list of documents with embeddings.
**Raises:**
- <code>TypeError</code> – If `documents` is not a list of Documents.
- <code>ValueError</code> – If the embeddings returned by the API have an unexpected shape.
## haystack_integrations.components.embedders.huggingface_api.text_embedder
### HuggingFaceAPITextEmbedder
Embeds strings using Hugging Face APIs.
Use it with the following Hugging Face APIs:
- [Free Serverless Inference API](https://huggingface.co/inference-api)
- [Paid Inference Endpoints](https://huggingface.co/inference-endpoints)
- [Self-hosted Text Embeddings Inference](https://github.com/huggingface/text-embeddings-inference)
### Usage examples
#### With free serverless inference API
```python
from haystack_integrations.components.embedders.huggingface_api import HuggingFaceAPITextEmbedder
from haystack.utils import Secret
text_embedder = HuggingFaceAPITextEmbedder(api_type="serverless_inference_api",
api_params={"model": "BAAI/bge-small-en-v1.5"},
token=Secret.from_token("<your-api-key>"))
print(text_embedder.run("I love pizza!"))
# {'embedding': [0.017020374536514282, -0.023255806416273117, ...],
```
#### With paid inference endpoints
```python
from haystack_integrations.components.embedders.huggingface_api import HuggingFaceAPITextEmbedder
from haystack.utils import Secret
text_embedder = HuggingFaceAPITextEmbedder(api_type="inference_endpoints",
api_params={"model": "BAAI/bge-small-en-v1.5"},
token=Secret.from_token("<your-api-key>"))
print(text_embedder.run("I love pizza!"))
# {'embedding': [0.017020374536514282, -0.023255806416273117, ...],
```
#### With self-hosted text embeddings inference
```python
from haystack_integrations.components.embedders.huggingface_api import HuggingFaceAPITextEmbedder
from haystack.utils import Secret
text_embedder = HuggingFaceAPITextEmbedder(api_type="text_embeddings_inference",
api_params={"url": "http://localhost:8080"})
print(text_embedder.run("I love pizza!"))
# {'embedding': [0.017020374536514282, -0.023255806416273117, ...],
```
#### __init__
```python
__init__(
api_type: HFEmbeddingAPIType | str,
api_params: dict[str, str],
token: Secret | None = Secret.from_env_var(
["HF_API_TOKEN", "HF_TOKEN"], strict=False
),
prefix: str = "",
suffix: str = "",
truncate: bool | None = True,
normalize: bool | None = False,
) -> None
```
Creates a HuggingFaceAPITextEmbedder component.
**Parameters:**
- **api_type** (<code>HFEmbeddingAPIType | str</code>) – The type of Hugging Face API to use.
- **api_params** (<code>dict\[str, str\]</code>) – A dictionary with the following keys:
- `model`: Hugging Face model ID. Required when `api_type` is `SERVERLESS_INFERENCE_API`.
- `url`: URL of the inference endpoint. Required when `api_type` is `INFERENCE_ENDPOINTS` or
`TEXT_EMBEDDINGS_INFERENCE`.
- **token** (<code>Secret | None</code>) – The Hugging Face token to use as HTTP bearer authorization.
Check your HF token in your [account settings](https://huggingface.co/settings/tokens).
- **prefix** (<code>str</code>) – A string to add at the beginning of each text.
- **suffix** (<code>str</code>) – A string to add at the end of each text.
- **truncate** (<code>bool | None</code>) – Truncates the input text to the maximum length supported by the model.
Applicable when `api_type` is `TEXT_EMBEDDINGS_INFERENCE`, or `INFERENCE_ENDPOINTS`
if the backend uses Text Embeddings Inference.
If `api_type` is `SERVERLESS_INFERENCE_API`, this parameter is ignored.
- **normalize** (<code>bool | None</code>) – Normalizes the embeddings to unit length.
Applicable when `api_type` is `TEXT_EMBEDDINGS_INFERENCE`, or `INFERENCE_ENDPOINTS`
if the backend uses Text Embeddings Inference.
If `api_type` is `SERVERLESS_INFERENCE_API`, this parameter is ignored.
**Raises:**
- <code>ValueError</code> – If the required `model` or `url` is missing from `api_params`, the `url` is invalid,
or the `api_type` is unknown.
#### to_dict
```python
to_dict() -> dict[str, Any]
```
Serializes the component to a dictionary.
**Returns:**
- <code>dict\[str, Any\]</code> – Dictionary with serialized data.
#### from_dict
```python
from_dict(data: dict[str, Any]) -> HuggingFaceAPITextEmbedder
```
Deserializes the component from a dictionary.
**Parameters:**
- **data** (<code>dict\[str, Any\]</code>) – Dictionary to deserialize from.
**Returns:**
- <code>HuggingFaceAPITextEmbedder</code> – Deserialized component.
#### run
```python
run(text: str) -> dict[str, Any]
```
Embeds a single string.
**Parameters:**
- **text** (<code>str</code>) – Text to embed.
**Returns:**
- <code>dict\[str, Any\]</code> – A dictionary with the following keys:
- `embedding`: The embedding of the input text.
**Raises:**
- <code>TypeError</code> – If `text` is not a string.
- <code>ValueError</code> – If the embedding returned by the API has an unexpected shape.
#### run_async
```python
run_async(text: str) -> dict[str, Any]
```
Embeds a single string asynchronously.
**Parameters:**
- **text** (<code>str</code>) – Text to embed.
**Returns:**
- <code>dict\[str, Any\]</code> – A dictionary with the following keys:
- `embedding`: The embedding of the input text.
**Raises:**
- <code>TypeError</code> – If `text` is not a string.
- <code>ValueError</code> – If the embedding returned by the API has an unexpected shape.
## haystack_integrations.components.generators.huggingface_api.chat.chat_generator
### HuggingFaceAPIChatGenerator
Completes chats using Hugging Face APIs.
HuggingFaceAPIChatGenerator uses the [ChatMessage](https://docs.haystack.deepset.ai/docs/chatmessage)
format for input and output. Use it to generate text with Hugging Face APIs:
- [Serverless Inference API (Inference Providers)](https://huggingface.co/docs/inference-providers)
- [Paid Inference Endpoints](https://huggingface.co/inference-endpoints)
- [Self-hosted Text Generation Inference](https://github.com/huggingface/text-generation-inference)
### Usage examples
#### With the serverless inference API (Inference Providers) - free tier available
```python
from haystack_integrations.components.generators.huggingface_api import HuggingFaceAPIChatGenerator
from haystack.dataclasses import ChatMessage
from haystack.utils import Secret
from haystack_integrations.components.common.huggingface_api.utils import HFGenerationAPIType
messages = [ChatMessage.from_system("\nYou are a helpful, respectful and honest assistant"),
ChatMessage.from_user("What's Natural Language Processing?")]
# the api_type can be expressed using the HFGenerationAPIType enum or as a string
api_type = HFGenerationAPIType.SERVERLESS_INFERENCE_API
api_type = "serverless_inference_api" # this is equivalent to the above
generator = HuggingFaceAPIChatGenerator(api_type=api_type,
api_params={"model": "Qwen/Qwen2.5-7B-Instruct",
"provider": "together"},
token=Secret.from_token("<your-api-key>"))
result = generator.run(messages)
print(result)
```
#### With the serverless inference API (Inference Providers) and text+image input
```python
from haystack_integrations.components.generators.huggingface_api import HuggingFaceAPIChatGenerator
from haystack.dataclasses import ChatMessage, ImageContent
from haystack.utils import Secret
from haystack_integrations.components.common.huggingface_api.utils import HFGenerationAPIType
# Create an image from file path, URL, or base64
image = ImageContent.from_file_path("path/to/your/image.jpg")
# Create a multimodal message with both text and image
messages = [ChatMessage.from_user(content_parts=["Describe this image in detail", image])]
generator = HuggingFaceAPIChatGenerator(
api_type=HFGenerationAPIType.SERVERLESS_INFERENCE_API,
api_params={
"model": "Qwen/Qwen2.5-VL-7B-Instruct", # Vision Language Model
"provider": "hyperbolic"
},
token=Secret.from_token("<your-api-key>")
)
result = generator.run(messages)
print(result)
```
#### With paid inference endpoints
```python
from haystack_integrations.components.generators.huggingface_api import HuggingFaceAPIChatGenerator
from haystack.dataclasses import ChatMessage
from haystack.utils import Secret
messages = [ChatMessage.from_system("\nYou are a helpful, respectful and honest assistant"),
ChatMessage.from_user("What's Natural Language Processing?")]
generator = HuggingFaceAPIChatGenerator(api_type="inference_endpoints",
api_params={"url": "<your-inference-endpoint-url>"},
token=Secret.from_token("<your-api-key>"))
result = generator.run(messages)
print(result)
```
#### With self-hosted text generation inference
```python
from haystack_integrations.components.generators.huggingface_api import HuggingFaceAPIChatGenerator
from haystack.dataclasses import ChatMessage
messages = [ChatMessage.from_system("\nYou are a helpful, respectful and honest assistant"),
ChatMessage.from_user("What's Natural Language Processing?")]
generator = HuggingFaceAPIChatGenerator(api_type="text_generation_inference",
api_params={"url": "http://localhost:8080"})
result = generator.run(messages)
print(result)
```
#### __init__
```python
__init__(
api_type: HFGenerationAPIType | str,
api_params: dict[str, str],
token: Secret | None = Secret.from_env_var(
["HF_API_TOKEN", "HF_TOKEN"], strict=False
),
generation_kwargs: dict[str, Any] | None = None,
stop_words: list[str] | None = None,
streaming_callback: StreamingCallbackT | None = None,
tools: ToolsType | None = None,
) -> None
```
Initialize the HuggingFaceAPIChatGenerator instance.
**Parameters:**
- **api_type** (<code>HFGenerationAPIType | str</code>) – The type of Hugging Face API to use. Available types:
- `text_generation_inference`: See [TGI](https://github.com/huggingface/text-generation-inference).
- `inference_endpoints`: See [Inference Endpoints](https://huggingface.co/inference-endpoints).
- `serverless_inference_api`: See
[Serverless Inference API - Inference Providers](https://huggingface.co/docs/inference-providers).
- **api_params** (<code>dict\[str, str\]</code>) – A dictionary with the following keys:
- `model`: Hugging Face model ID. Required when `api_type` is `SERVERLESS_INFERENCE_API`.
- `provider`: Provider name. Recommended when `api_type` is `SERVERLESS_INFERENCE_API`.
- `url`: URL of the inference endpoint. Required when `api_type` is `INFERENCE_ENDPOINTS` or
`TEXT_GENERATION_INFERENCE`.
- Other parameters specific to the chosen API type, such as `timeout`, `headers`, etc.
- **token** (<code>Secret | None</code>) – The Hugging Face token to use as HTTP bearer authorization.
Check your HF token in your [account settings](https://huggingface.co/settings/tokens).
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – A dictionary with keyword arguments to customize text generation.
Some examples: `max_tokens`, `temperature`, `top_p`.
For details, see [Hugging Face chat_completion documentation](https://huggingface.co/docs/huggingface_hub/package_reference/inference_client#huggingface_hub.InferenceClient.chat_completion).
- **stop_words** (<code>list\[str\] | None</code>) – An optional list of strings representing the stop words.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – An optional callable for handling streaming responses.
- **tools** (<code>ToolsType | None</code>) – A list of Tool and/or Toolset objects, or a single Toolset for which the model can prepare calls.
The chosen model should support tool/function calling, according to the model card.
Support for tools in the Hugging Face API and TGI is not yet fully refined and you may experience
unexpected behavior.
**Raises:**
- <code>ValueError</code> – If the required `model` or `url` is missing from `api_params`, the `url` is invalid, the `api_type`
is unknown, `tools` and `streaming_callback` are used together, or duplicate tool names are provided.
#### warm_up
```python
warm_up() -> None
```
Warm up the Hugging Face API chat generator.
This will warm up the tools registered in the chat generator.
This method is idempotent and will only warm up the tools once.
#### to_dict
```python
to_dict() -> dict[str, Any]
```
Serialize this component to a dictionary.
**Returns:**
- <code>dict\[str, Any\]</code> – A dictionary containing the serialized component.
#### from_dict
```python
from_dict(data: dict[str, Any]) -> HuggingFaceAPIChatGenerator
```
Deserialize this component from a dictionary.
#### run
```python
run(
messages: list[ChatMessage] | str,
generation_kwargs: dict[str, Any] | None = None,
tools: ToolsType | None = None,
streaming_callback: StreamingCallbackT | None = None,
) -> dict[str, list[ChatMessage]]
```
Invoke the text generation inference based on the provided messages and generation parameters.
**Parameters:**
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of ChatMessage objects representing the input messages. If a string is provided, it is converted
to a list containing a ChatMessage with user role.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Additional keyword arguments for text generation.
- **tools** (<code>ToolsType | None</code>) – A list of tools or a Toolset for which the model can prepare calls. If set, it will override
the `tools` parameter set during component initialization. This parameter can accept either a
list of `Tool` objects or a `Toolset` instance.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – An optional callable for handling streaming responses. If set, it will override the `streaming_callback`
parameter set during component initialization.
**Returns:**
- <code>dict\[str, list\[ChatMessage\]\]</code> – A dictionary with the following keys:
- `replies`: A list containing the generated responses as ChatMessage objects.
**Raises:**
- <code>ValueError</code> – If `tools` and a streaming callback are used together, or if duplicate tool names are provided.
#### run_async
```python
run_async(
messages: list[ChatMessage] | str,
generation_kwargs: dict[str, Any] | None = None,
tools: ToolsType | None = None,
streaming_callback: StreamingCallbackT | None = None,
) -> dict[str, list[ChatMessage]]
```
Asynchronously invokes the text generation inference based on the provided messages and generation parameters.
This is the asynchronous version of the `run` method. It has the same parameters
and return values but can be used with `await` in an async code.
**Parameters:**
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of ChatMessage objects representing the input messages. If a string is provided, it is converted
to a list containing a ChatMessage with user role.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Additional keyword arguments for text generation.
- **tools** (<code>ToolsType | None</code>) – A list of tools or a Toolset for which the model can prepare calls. If set, it will override the `tools`
parameter set during component initialization. This parameter can accept either a list of `Tool` objects
or a `Toolset` instance.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – An optional callable for handling streaming responses. If set, it will override the `streaming_callback`
parameter set during component initialization.
**Returns:**
- <code>dict\[str, list\[ChatMessage\]\]</code> – A dictionary with the following keys:
- `replies`: A list containing the generated responses as ChatMessage objects.
**Raises:**
- <code>ValueError</code> – If `tools` and a streaming callback are used together, or if duplicate tool names are provided.
## haystack_integrations.components.rankers.huggingface_api.ranker
### TruncationDirection
Bases: <code>str</code>, <code>Enum</code>
Defines the direction to truncate text when input length exceeds the model's limit.
Attributes:
LEFT: Truncate text from the left side (start of text).
RIGHT: Truncate text from the right side (end of text).
### HuggingFaceTEIRanker
Ranks documents based on their semantic similarity to the query.
It can be used with a Text Embeddings Inference (TEI) API endpoint:
- [Self-hosted Text Embeddings Inference](https://github.com/huggingface/text-embeddings-inference)
- [Hugging Face Inference Endpoints](https://huggingface.co/inference-endpoints)
Usage example:
```python
from haystack import Document
from haystack.utils import Secret
from haystack_integrations.components.rankers.huggingface_api import HuggingFaceTEIRanker
reranker = HuggingFaceTEIRanker(
url="http://localhost:8080",
top_k=5,
timeout=30,
token=Secret.from_token("my_api_token")
)
docs = [Document(content="The capital of France is Paris"), Document(content="The capital of Germany is Berlin")]
result = reranker.run(query="What is the capital of France?", documents=docs)
ranked_docs = result["documents"]
print(ranked_docs)
# >> {'documents': [Document(id=..., content: 'the capital of France is Paris', score: 0.9979767),
# >> Document(id=..., content: 'the capital of Germany is Berlin', score: 0.13982213)]}
```
#### __init__
```python
__init__(
*,
url: str,
top_k: int = 10,
raw_scores: bool = False,
timeout: int | None = 30,
max_retries: int = 3,
retry_status_codes: list[int] | None = None,
token: Secret | None = Secret.from_env_var(
["HF_API_TOKEN", "HF_TOKEN"], strict=False
)
) -> None
```
Initializes the TEI reranker component.
**Parameters:**
- **url** (<code>str</code>) – Base URL of the TEI reranking service (for example, "https://api.example.com").
- **top_k** (<code>int</code>) – Maximum number of top documents to return.
- **raw_scores** (<code>bool</code>) – If True, include raw relevance scores in the API payload.
- **timeout** (<code>int | None</code>) – Request timeout in seconds.
- **max_retries** (<code>int</code>) – Maximum number of retry attempts for failed requests.
- **retry_status_codes** (<code>list\[int\] | None</code>) – List of HTTP status codes that will trigger a retry.
When None, HTTP 408, 418, 429 and 503 will be retried (default: None).
- **token** (<code>Secret | None</code>) – The Hugging Face token to use as HTTP bearer authorization. Not always required
depending on your TEI server configuration.
Check your HF token in your [account settings](https://huggingface.co/settings/tokens).
#### to_dict
```python
to_dict() -> dict[str, Any]
```
Serializes the component to a dictionary.
**Returns:**
- <code>dict\[str, Any\]</code> – Dictionary with serialized data.
#### from_dict
```python
from_dict(data: dict[str, Any]) -> HuggingFaceTEIRanker
```
Deserializes the component from a dictionary.
**Parameters:**
- **data** (<code>dict\[str, Any\]</code>) – Dictionary to deserialize from.
**Returns:**
- <code>HuggingFaceTEIRanker</code> – Deserialized component.
#### run
```python
run(
query: str,
documents: list[Document],
top_k: int | None = None,
truncation_direction: TruncationDirection | None = None,
) -> dict[str, list[Document]]
```
Reranks the provided documents by relevance to the query using the TEI API.
Before ranking, documents are deduplicated by their id, retaining only the document with the highest score
if a score is present.
**Parameters:**
- **query** (<code>str</code>) – The user query string to guide reranking.
- **documents** (<code>list\[Document\]</code>) – List of `Document` objects to rerank.
- **top_k** (<code>int | None</code>) – Optional override for the maximum number of documents to return.
- **truncation_direction** (<code>TruncationDirection | None</code>) – If set, enables text truncation in the specified direction.
**Returns:**
- <code>dict\[str, list\[Document\]\]</code> – A dictionary with the following keys:
- `documents`: A list of reranked documents.
**Raises:**
- <code>RuntimeError</code> – - If the API request fails.
- <code>RuntimeError</code> – - If the API returns an error response.
- <code>TypeError</code> – - If the API response is not in the expected list format.
#### run_async
```python
run_async(
query: str,
documents: list[Document],
top_k: int | None = None,
truncation_direction: TruncationDirection | None = None,
) -> dict[str, list[Document]]
```
Asynchronously reranks the provided documents by relevance to the query using the TEI API.
Before ranking, documents are deduplicated by their id, retaining only the document with the highest score
if a score is present.
**Parameters:**
- **query** (<code>str</code>) – The user query string to guide reranking.
- **documents** (<code>list\[Document\]</code>) – List of `Document` objects to rerank.
- **top_k** (<code>int | None</code>) – Optional override for the maximum number of documents to return.
- **truncation_direction** (<code>TruncationDirection | None</code>) – If set, enables text truncation in the specified direction.
**Returns:**
- <code>dict\[str, list\[Document\]\]</code> – A dictionary with the following keys:
- `documents`: A list of reranked documents.
**Raises:**
- <code>httpx.RequestError</code> – - If the API request fails.
- <code>RuntimeError</code> – - If the API returns an error response.
- <code>TypeError</code> – - If the API response is not in the expected list format.
docs-website/reference/integrations-api/litellm.md
+17 -4
View File
@@ -73,7 +73,7 @@ Create a LiteLLMChatGenerator instance.
```python
run(
messages: list[ChatMessage],
messages: list[ChatMessage] | str,
streaming_callback: StreamingCallbackT | None = None,
generation_kwargs: dict[str, Any] | None = None,
*,
@@ -85,7 +85,8 @@ Invoke chat completion via LiteLLM.
**Parameters:**
- **messages** (<code>list\[ChatMessage\]</code>) – Input messages as ChatMessage instances.
- **messages** (<code>list\[ChatMessage\] | str</code>) – Input messages as ChatMessage instances.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – Override the streaming callback for this call.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Override generation parameters for this call.
- **tools** (<code>ToolsType | None</code>) – Override tools for this call.
@@ -98,7 +99,7 @@ Invoke chat completion via LiteLLM.
```python
run_async(
messages: list[ChatMessage],
messages: list[ChatMessage] | str,
streaming_callback: StreamingCallbackT | None = None,
generation_kwargs: dict[str, Any] | None = None,
*,
@@ -106,7 +107,19 @@ run_async(
) -> dict[str, list[ChatMessage]]
```
Async version of run().
Async version of run(). Invoke chat completion via LiteLLM.
**Parameters:**
- **messages** (<code>list\[ChatMessage\] | str</code>) – Input messages as ChatMessage instances.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – Override the streaming callback for this call.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Override generation parameters for this call.
- **tools** (<code>ToolsType | None</code>) – Override tools for this call.
**Returns:**
- <code>dict\[str, list\[ChatMessage\]\]</code> – A dict with key `replies` containing ChatMessage instances.
#### to_dict
docs-website/reference/integrations-api/llama_cpp.md
+131 -130
View File
@@ -5,11 +5,8 @@ description: "Llama.cpp integration for Haystack"
slug: "/integrations-llama-cpp"
---
<a id="haystack_integrations.components.generators.llama_cpp.chat.chat_generator"></a>
## Module haystack\_integrations.components.generators.llama\_cpp.chat.chat\_generator
<a id="haystack_integrations.components.generators.llama_cpp.chat.chat_generator.LlamaCppChatGenerator"></a>
## haystack_integrations.components.generators.llama_cpp.chat.chat_generator
### LlamaCppChatGenerator
@@ -20,6 +17,7 @@ It employs the quantized GGUF format, suitable for running these models on stand
Supports both text-only and multimodal (text + image) models like LLaVA.
Usage example:
```python
from haystack_integrations.components.generators.llama_cpp import LlamaCppChatGenerator
user_message = [ChatMessage.from_user("Who is the best American actor?")]
@@ -30,6 +28,7 @@ print(generator.run(user_message, generation_kwargs={"max_tokens": 128}))
```
Usage example with multimodal (image + text):
```python
from haystack.dataclasses import ChatMessage, ImageContent
@@ -46,94 +45,95 @@ generator = LlamaCppChatGenerator(
model_clip_path="mmproj-model-f16.gguf", # CLIP model
n_ctx=4096 # Larger context for image processing
)
generator.warm_up()
result = generator.run(messages)
print(result)
```
<a id="haystack_integrations.components.generators.llama_cpp.chat.chat_generator.LlamaCppChatGenerator.__init__"></a>
#### LlamaCppChatGenerator.\_\_init\_\_
#### __init__
```python
def __init__(model: str,
n_ctx: int | None = 0,
n_batch: int | None = 512,
model_kwargs: dict[str, Any] | None = None,
generation_kwargs: dict[str, Any] | None = None,
*,
tools: ToolsType | None = None,
streaming_callback: StreamingCallbackT | None = None,
chat_handler_name: str | None = None,
model_clip_path: str | None = None) -> None
__init__(
model: str,
n_ctx: int | None = 0,
n_batch: int | None = 512,
model_kwargs: dict[str, Any] | None = None,
generation_kwargs: dict[str, Any] | None = None,
*,
tools: ToolsType | None = None,
streaming_callback: StreamingCallbackT | None = None,
chat_handler_name: str | None = None,
model_clip_path: str | None = None
) -> None
```
**Arguments**:
Initialize LlamaCppChatGenerator.
- `model`: The path of a quantized model for text generation, for example, "zephyr-7b-beta.Q4_0.gguf".
If the model path is also specified in the `model_kwargs`, this parameter will be ignored.
- `n_ctx`: The number of tokens in the context. When set to 0, the context will be taken from the model.
- `n_batch`: Prompt processing maximum batch size.
- `model_kwargs`: Dictionary containing keyword arguments used to initialize the LLM for text generation.
These keyword arguments provide fine-grained control over the model loading.
In case of duplication, these kwargs override `model`, `n_ctx`, and `n_batch` init parameters.
For more information on the available kwargs, see
[llama.cpp documentation](https://llama-cpp-python.readthedocs.io/en/latest/api-reference/`llama_cpp.Llama.__init__`).
- `generation_kwargs`: A dictionary containing keyword arguments to customize text generation.
For more information on the available kwargs, see
[llama.cpp documentation](https://llama-cpp-python.readthedocs.io/en/latest/api-reference/`llama_cpp.Llama.create_chat_completion`).
- `tools`: A list of Tool and/or Toolset objects, or a single Toolset for which the model can prepare calls.
Each tool should have a unique name.
- `streaming_callback`: A callback function that is called when a new token is received from the stream.
- `chat_handler_name`: Name of the chat handler for multimodal models.
Common options include: "Llava16ChatHandler", "MoondreamChatHandler", "Qwen25VLChatHandler".
For other handlers, check
[llama-cpp-python documentation](https://llama-cpp-python.readthedocs.io/en/latest/`multi`-modal-models).
- `model_clip_path`: Path to the CLIP model for vision processing (e.g., "mmproj.bin").
Required when chat_handler_name is provided for multimodal models.
**Parameters:**
<a id="haystack_integrations.components.generators.llama_cpp.chat.chat_generator.LlamaCppChatGenerator.to_dict"></a>
- **model** (<code>str</code>) – The path of a quantized model for text generation, for example, "zephyr-7b-beta.Q4_0.gguf".
If the model path is also specified in the `model_kwargs`, this parameter will be ignored.
- **n_ctx** (<code>int | None</code>) – The number of tokens in the context. When set to 0, the context will be taken from the model.
- **n_batch** (<code>int | None</code>) – Prompt processing maximum batch size.
- **model_kwargs** (<code>dict\[str, Any\] | None</code>) – Dictionary containing keyword arguments used to initialize the LLM for text generation.
These keyword arguments provide fine-grained control over the model loading.
In case of duplication, these kwargs override `model`, `n_ctx`, and `n_batch` init parameters.
For more information on the available kwargs, see
[llama.cpp documentation](https://llama-cpp-python.readthedocs.io/en/latest/api-reference/#llama_cpp.Llama.__init__).
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – A dictionary containing keyword arguments to customize text generation.
For more information on the available kwargs, see
[llama.cpp documentation](https://llama-cpp-python.readthedocs.io/en/latest/api-reference/#llama_cpp.Llama.create_chat_completion).
- **tools** (<code>ToolsType | None</code>) – A list of Tool and/or Toolset objects, or a single Toolset for which the model can prepare calls.
Each tool should have a unique name.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
- **chat_handler_name** (<code>str | None</code>) – Name of the chat handler for multimodal models.
Common options include: "Llava16ChatHandler", "MoondreamChatHandler", "Qwen25VLChatHandler".
For other handlers, check
[llama-cpp-python documentation](https://llama-cpp-python.readthedocs.io/en/latest/#multi-modal-models).
- **model_clip_path** (<code>str | None</code>) – Path to the CLIP model for vision processing (e.g., "mmproj.bin").
Required when chat_handler_name is provided for multimodal models.
#### LlamaCppChatGenerator.to\_dict
#### warm_up
```python
def to_dict() -> dict[str, Any]
warm_up() -> None
```
Load and initialize the llama.cpp model.
#### to_dict
```python
to_dict() -> dict[str, Any]
```
Serializes the component to a dictionary.
**Returns**:
**Returns:**
Dictionary with serialized data.
- <code>dict\[str, Any\]</code> – Dictionary with serialized data.
<a id="haystack_integrations.components.generators.llama_cpp.chat.chat_generator.LlamaCppChatGenerator.from_dict"></a>
#### LlamaCppChatGenerator.from\_dict
#### from_dict
```python
@classmethod
def from_dict(cls, data: dict[str, Any]) -> "LlamaCppChatGenerator"
from_dict(data: dict[str, Any]) -> LlamaCppChatGenerator
```
Deserializes the component from a dictionary.
**Arguments**:
**Parameters:**
- `data`: Dictionary to deserialize from.
- **data** (<code>dict\[str, Any\]</code>) – Dictionary to deserialize from.
**Returns**:
**Returns:**
Deserialized component.
- <code>LlamaCppChatGenerator</code> – Deserialized component.
<a id="haystack_integrations.components.generators.llama_cpp.chat.chat_generator.LlamaCppChatGenerator.run"></a>
#### LlamaCppChatGenerator.run
#### run
```python
@component.output_types(replies=list[ChatMessage])
def run(
messages: list[ChatMessage],
run(
messages: list[ChatMessage] | str,
generation_kwargs: dict[str, Any] | None = None,
*,
tools: ToolsType | None = None,
@@ -143,31 +143,29 @@ def run(
Run the text generation model on the given list of ChatMessages.
**Arguments**:
**Parameters:**
- `messages`: A list of ChatMessage instances representing the input messages.
- `generation_kwargs`: A dictionary containing keyword arguments to customize text generation.
For more information on the available kwargs, see
[llama.cpp documentation](https://llama-cpp-python.readthedocs.io/en/latest/api-reference/`llama_cpp.Llama.create_chat_completion`).
- `tools`: A list of Tool and/or Toolset objects, or a single Toolset for which the model can prepare calls.
Each tool should have a unique name. If set, it will override the `tools` parameter set during
component initialization.
- `streaming_callback`: A callback function that is called when a new token is received from the stream.
If set, it will override the `streaming_callback` parameter set during component initialization.
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of ChatMessage instances representing the input messages.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – A dictionary containing keyword arguments to customize text generation.
For more information on the available kwargs, see
[llama.cpp documentation](https://llama-cpp-python.readthedocs.io/en/latest/api-reference/#llama_cpp.Llama.create_chat_completion).
- **tools** (<code>ToolsType | None</code>) – A list of Tool and/or Toolset objects, or a single Toolset for which the model can prepare calls.
Each tool should have a unique name. If set, it will override the `tools` parameter set during
component initialization.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
If set, it will override the `streaming_callback` parameter set during component initialization.
**Returns**:
**Returns:**
A dictionary with the following keys:
- <code>dict\[str, list\[ChatMessage\]\]</code> – A dictionary with the following keys:
- `replies`: The responses from the model
<a id="haystack_integrations.components.generators.llama_cpp.chat.chat_generator.LlamaCppChatGenerator.run_async"></a>
#### LlamaCppChatGenerator.run\_async
#### run_async
```python
@component.output_types(replies=list[ChatMessage])
async def run_async(
messages: list[ChatMessage],
run_async(
messages: list[ChatMessage] | str,
generation_kwargs: dict[str, Any] | None = None,
*,
tools: ToolsType | None = None,
@@ -180,28 +178,25 @@ Async version of run. Runs the text generation model on the given list of ChatMe
Uses a thread pool to avoid blocking the event loop, since llama-cpp-python provides
only synchronous inference.
**Arguments**:
**Parameters:**
- `messages`: A list of ChatMessage instances representing the input messages.
- `generation_kwargs`: A dictionary containing keyword arguments to customize text generation.
For more information on the available kwargs, see
[llama.cpp documentation](https://llama-cpp-python.readthedocs.io/en/latest/api-reference/`llama_cpp.Llama.create_chat_completion`).
- `tools`: A list of Tool and/or Toolset objects, or a single Toolset for which the model can prepare calls.
Each tool should have a unique name. If set, it will override the `tools` parameter set during
component initialization.
- `streaming_callback`: A callback function that is called when a new token is received from the stream.
If set, it will override the `streaming_callback` parameter set during component initialization.
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of ChatMessage instances representing the input messages.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – A dictionary containing keyword arguments to customize text generation.
For more information on the available kwargs, see
[llama.cpp documentation](https://llama-cpp-python.readthedocs.io/en/latest/api-reference/#llama_cpp.Llama.create_chat_completion).
- **tools** (<code>ToolsType | None</code>) – A list of Tool and/or Toolset objects, or a single Toolset for which the model can prepare calls.
Each tool should have a unique name. If set, it will override the `tools` parameter set during
component initialization.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
If set, it will override the `streaming_callback` parameter set during component initialization.
**Returns**:
**Returns:**
A dictionary with the following keys:
- <code>dict\[str, list\[ChatMessage\]\]</code> – A dictionary with the following keys:
- `replies`: The responses from the model
<a id="haystack_integrations.components.generators.llama_cpp.generator"></a>
## Module haystack\_integrations.components.generators.llama\_cpp.generator
<a id="haystack_integrations.components.generators.llama_cpp.generator.LlamaCppGenerator"></a>
## haystack_integrations.components.generators.llama_cpp.generator
### LlamaCppGenerator
@@ -211,6 +206,7 @@ Provides an interface to generate text using LLM via llama.cpp.
It employs the quantized GGUF format, suitable for running these models on standard machines (even without GPUs).
Usage example:
```python
from haystack_integrations.components.generators.llama_cpp import LlamaCppGenerator
generator = LlamaCppGenerator(model="zephyr-7b-beta.Q4_0.gguf", n_ctx=2048, n_batch=512)
@@ -219,57 +215,62 @@ print(generator.run("Who is the best American actor?", generation_kwargs={"max_t
# {'replies': ['John Cusack'], 'meta': [{"object": "text_completion", ...}]}
```
<a id="haystack_integrations.components.generators.llama_cpp.generator.LlamaCppGenerator.__init__"></a>
#### LlamaCppGenerator.\_\_init\_\_
#### __init__
```python
def __init__(model: str,
n_ctx: int | None = 0,
n_batch: int | None = 512,
model_kwargs: dict[str, Any] | None = None,
generation_kwargs: dict[str, Any] | None = None) -> None
__init__(
model: str,
n_ctx: int | None = 0,
n_batch: int | None = 512,
model_kwargs: dict[str, Any] | None = None,
generation_kwargs: dict[str, Any] | None = None,
) -> None
```
**Arguments**:
Initialize LlamaCppGenerator.
- `model`: The path of a quantized model for text generation, for example, "zephyr-7b-beta.Q4_0.gguf".
If the model path is also specified in the `model_kwargs`, this parameter will be ignored.
- `n_ctx`: The number of tokens in the context. When set to 0, the context will be taken from the model.
- `n_batch`: Prompt processing maximum batch size.
- `model_kwargs`: Dictionary containing keyword arguments used to initialize the LLM for text generation.
These keyword arguments provide fine-grained control over the model loading.
In case of duplication, these kwargs override `model`, `n_ctx`, and `n_batch` init parameters.
For more information on the available kwargs, see
[llama.cpp documentation](https://llama-cpp-python.readthedocs.io/en/latest/api-reference/`llama_cpp.Llama.__init__`).
- `generation_kwargs`: A dictionary containing keyword arguments to customize text generation.
For more information on the available kwargs, see
[llama.cpp documentation](https://llama-cpp-python.readthedocs.io/en/latest/api-reference/`llama_cpp.Llama.create_completion`).
**Parameters:**
<a id="haystack_integrations.components.generators.llama_cpp.generator.LlamaCppGenerator.run"></a>
- **model** (<code>str</code>) – The path of a quantized model for text generation, for example, "zephyr-7b-beta.Q4_0.gguf".
If the model path is also specified in the `model_kwargs`, this parameter will be ignored.
- **n_ctx** (<code>int | None</code>) – The number of tokens in the context. When set to 0, the context will be taken from the model.
- **n_batch** (<code>int | None</code>) – Prompt processing maximum batch size.
- **model_kwargs** (<code>dict\[str, Any\] | None</code>) – Dictionary containing keyword arguments used to initialize the LLM for text generation.
These keyword arguments provide fine-grained control over the model loading.
In case of duplication, these kwargs override `model`, `n_ctx`, and `n_batch` init parameters.
For more information on the available kwargs, see
[llama.cpp documentation](https://llama-cpp-python.readthedocs.io/en/latest/api-reference/#llama_cpp.Llama.__init__).
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – A dictionary containing keyword arguments to customize text generation.
For more information on the available kwargs, see
[llama.cpp documentation](https://llama-cpp-python.readthedocs.io/en/latest/api-reference/#llama_cpp.Llama.create_completion).
#### LlamaCppGenerator.run
#### warm_up
```python
@component.output_types(replies=list[str], meta=list[dict[str, Any]])
def run(
prompt: str,
generation_kwargs: dict[str, Any] | None = None
warm_up() -> None
```
Load and initialize the llama.cpp model.
#### run
```python
run(
prompt: str, generation_kwargs: dict[str, Any] | None = None
) -> dict[str, list[str] | list[dict[str, Any]]]
```
Run the text generation model on the given prompt.
**Arguments**:
**Parameters:**
- `prompt`: the prompt to be sent to the generative model.
- `generation_kwargs`: A dictionary containing keyword arguments to customize text generation.
For more information on the available kwargs, see
[llama.cpp documentation](https://llama-cpp-python.readthedocs.io/en/latest/api-reference/`llama_cpp.Llama.create_completion`).
- **prompt** (<code>str</code>) – the prompt to be sent to the generative model.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – A dictionary containing keyword arguments to customize text generation.
For more information on the available kwargs, see
[llama.cpp documentation](https://llama-cpp-python.readthedocs.io/en/latest/api-reference/#llama_cpp.Llama.create_completion).
**Returns**:
**Returns:**
A dictionary with the following keys:
- <code>dict\[str, list\[str\] | list\[dict\[str, Any\]\]\]</code> – A dictionary with the following keys:
- `replies`: the list of replies generated by the model.
- `meta`: metadata about the request.
docs-website/reference/integrations-api/mistral.md
+6 -4
View File
@@ -576,7 +576,7 @@ Unless specified otherwise in the `model`, this is for Mistral's `mistral-small-
```python
run(
messages: list[ChatMessage],
messages: list[ChatMessage] | str,
streaming_callback: StreamingCallbackT | None = None,
generation_kwargs: dict[str, Any] | None = None,
*,
@@ -589,7 +589,8 @@ Invokes chat completion on the Mistral API.
**Parameters:**
- **messages** (<code>list\[ChatMessage\]</code>) – A list of ChatMessage instances representing the input messages.
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of ChatMessage instances representing the input messages.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Additional keyword arguments for text generation. These parameters will
override the parameters passed during component initialization.
@@ -608,7 +609,7 @@ Invokes chat completion on the Mistral API.
```python
run_async(
messages: list[ChatMessage],
messages: list[ChatMessage] | str,
streaming_callback: StreamingCallbackT | None = None,
generation_kwargs: dict[str, Any] | None = None,
*,
@@ -621,7 +622,8 @@ Asynchronously invokes chat completion on the Mistral API.
**Parameters:**
- **messages** (<code>list\[ChatMessage\]</code>) – A list of ChatMessage instances representing the input messages.
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of ChatMessage instances representing the input messages.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
Must be a coroutine.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Additional keyword arguments for text generation.
docs-website/reference/integrations-api/ollama.md
+6 -4
View File
@@ -319,7 +319,7 @@ Deserializes the component from a dictionary.
```python
run(
messages: list[ChatMessage],
messages: list[ChatMessage] | str,
generation_kwargs: dict[str, Any] | None = None,
tools: ToolsType | None = None,
*,
@@ -331,7 +331,8 @@ Runs an Ollama Model on a given chat history.
**Parameters:**
- **messages** (<code>list\[ChatMessage\]</code>) – A list of ChatMessage instances representing the input messages.
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of ChatMessage instances representing the input messages. If a string is provided, it is converted
to a list containing a ChatMessage with user role.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Per-call overrides for Ollama inference options.
These are merged on top of the instance-level `generation_kwargs`.
Optional arguments to pass to the Ollama generation endpoint, such as temperature, top_p, etc. See the
@@ -351,7 +352,7 @@ Runs an Ollama Model on a given chat history.
```python
run_async(
messages: list[ChatMessage],
messages: list[ChatMessage] | str,
generation_kwargs: dict[str, Any] | None = None,
tools: ToolsType | None = None,
*,
@@ -363,7 +364,8 @@ Async version of run. Runs an Ollama Model on a given chat history.
**Parameters:**
- **messages** (<code>list\[ChatMessage\]</code>) – A list of ChatMessage instances representing the input messages.
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of ChatMessage instances representing the input messages. If a string is provided, it is converted
to a list containing a ChatMessage with user role.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Per-call overrides for Ollama inference options.
These are merged on top of the instance-level `generation_kwargs`.
- **tools** (<code>ToolsType | None</code>) – A list of Tool and/or Toolset objects, or a single Toolset for which the model can prepare calls.
docs-website/reference/integrations-api/openrouter.md
+128 -68
View File
@@ -5,15 +5,15 @@ description: "OpenRouter integration for Haystack"
slug: "/integrations-openrouter"
---
<a id="haystack_integrations.components.generators.openrouter.chat.chat_generator"></a>
## Module haystack\_integrations.components.generators.openrouter.chat.chat\_generator
<a id="haystack_integrations.components.generators.openrouter.chat.chat_generator.OpenRouterChatGenerator"></a>
## haystack_integrations.components.generators.openrouter.chat.chat_generator
### OpenRouterChatGenerator
Bases: <code>OpenAIChatGenerator</code>
Enables text generation using OpenRouter generative models.
For supported models, see [OpenRouter docs](https://openrouter.ai/models).
Users can pass any text generation parameters valid for the OpenRouter chat completion API
@@ -21,9 +21,13 @@ directly to this component using the `generation_kwargs` parameter in `__init__`
parameter in `run` method.
Key Features and Compatibility:
- **Primary Compatibility**: Designed to work seamlessly with the OpenRouter chat completion endpoint.
- **Primary Compatibility**: Compatible with the OpenRouter chat completion endpoint.
- **Streaming Support**: Supports streaming responses from the OpenRouter chat completion endpoint.
- **Customizability**: Supports all parameters supported by the OpenRouter chat completion endpoint.
- **Reasoning Support**: Extracts reasoning/thinking content from models that support it
(e.g., DeepSeek R1, Claude with extended thinking) and stores it in the `ReasoningContent`
field on `ChatMessage`. Reasoning content is only captured for non-streaming requests.
This component uses the ChatMessage format for structuring both input and output,
ensuring coherent and contextually relevant responses in chat-based text generation scenarios.
@@ -34,96 +38,152 @@ For more details on the parameters supported by the OpenRouter API, refer to the
[OpenRouter API Docs](https://openrouter.ai/docs/quickstart).
Usage example:
```python
from haystack_integrations.components.generators.openrouter import OpenRouterChatGenerator
from haystack_integrations.components.generators.openrouter import (
OpenRouterChatGenerator,
)
from haystack.dataclasses import ChatMessage
messages = [ChatMessage.from_user("What's Natural Language Processing?")]
client = OpenRouterChatGenerator()
client = OpenRouterChatGenerator(
model="deepseek/deepseek-r1",
generation_kwargs={"reasoning": {"effort": "high"}},
)
response = client.run(messages)
print(response)
>>{'replies': [ChatMessage(_content='Natural Language Processing (NLP) is a branch of artificial intelligence
>>that focuses on enabling computers to understand, interpret, and generate human language in a way that is
>>meaningful and useful.', _role=<ChatRole.ASSISTANT: 'assistant'>, _name=None,
>>_meta={'model': 'openai/gpt-5-mini', 'index': 0, 'finish_reason': 'stop',
>>'usage': {'prompt_tokens': 15, 'completion_tokens': 36, 'total_tokens': 51}})]}
print(response["replies"][0].reasoning) # Access reasoning content
print(response["replies"][0].text) # Access final answer
```
<a id="haystack_integrations.components.generators.openrouter.chat.chat_generator.OpenRouterChatGenerator.__init__"></a>
#### OpenRouterChatGenerator.\_\_init\_\_
#### __init__
```python
def __init__(*,
api_key: Secret = Secret.from_env_var("OPENROUTER_API_KEY"),
model: str = "openai/gpt-5-mini",
streaming_callback: StreamingCallbackT | None = None,
api_base_url: str | None = "https://openrouter.ai/api/v1",
generation_kwargs: dict[str, Any] | None = None,
tools: ToolsType | None = None,
timeout: float | None = None,
extra_headers: dict[str, Any] | None = None,
max_retries: int | None = None,
http_client_kwargs: dict[str, Any] | None = None)
__init__(
*,
api_key: Secret = Secret.from_env_var("OPENROUTER_API_KEY"),
model: str = "openai/gpt-5-mini",
streaming_callback: StreamingCallbackT | None = None,
api_base_url: str | None = "https://openrouter.ai/api/v1",
generation_kwargs: dict[str, Any] | None = None,
tools: ToolsType | None = None,
timeout: float | None = None,
extra_headers: dict[str, Any] | None = None,
max_retries: int | None = None,
http_client_kwargs: dict[str, Any] | None = None
) -> None
```
Creates an instance of OpenRouterChatGenerator. Unless specified otherwise,
Creates an instance of OpenRouterChatGenerator.
the default model is `openai/gpt-5-mini`.
**Parameters:**
**Arguments**:
- `api_key`: The OpenRouter API key.
- `model`: The name of the OpenRouter chat completion model to use.
- `streaming_callback`: A callback function that is called when a new token is received from the stream.
The callback function accepts StreamingChunk as an argument.
- `api_base_url`: The OpenRouter API Base url.
For more details, see OpenRouter [docs](https://openrouter.ai/docs/quickstart).
- `generation_kwargs`: Other parameters to use for the model. These parameters are all sent directly to
the OpenRouter endpoint. See [OpenRouter API docs](https://openrouter.ai/docs/quickstart) for more details.
Some of the supported parameters:
- **api_key** (<code>Secret</code>) – The OpenRouter API key.
- **model** (<code>str</code>) – The name of the OpenRouter chat completion model to use.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
The callback function accepts StreamingChunk as an argument.
- **api_base_url** (<code>str | None</code>) – The OpenRouter API Base url.
For more details, see OpenRouter [docs](https://openrouter.ai/docs/quickstart).
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Other parameters to use for the model. These parameters are all sent directly to
the OpenRouter endpoint. See [OpenRouter API docs](https://openrouter.ai/docs/quickstart) for more details.
Some of the supported parameters:
- `max_tokens`: The maximum number of tokens the output text can have.
- `temperature`: What sampling temperature to use. Higher values mean the model will take more risks.
Try 0.9 for more creative applications and 0 (argmax sampling) for ones with a well-defined answer.
Try 0.9 for more creative applications and 0 (argmax sampling) for ones with a well-defined answer.
- `top_p`: An alternative to sampling with temperature, called nucleus sampling, where the model
considers the results of the tokens with top_p probability mass. So 0.1 means only the tokens
comprising the top 10% probability mass are considered.
considers the results of the tokens with top_p probability mass. So 0.1 means only the tokens
comprising the top 10% probability mass are considered.
- `stream`: Whether to stream back partial progress. If set, tokens will be sent as data-only server-sent
events as they become available, with the stream terminated by a data: [DONE] message.
events as they become available, with the stream terminated by a data: [DONE] message.
- `safe_prompt`: Whether to inject a safety prompt before all conversations.
- `random_seed`: The seed to use for random sampling.
- `reasoning`: A dict to configure reasoning/thinking tokens for models that support it.
Example: `{"effort": "high"}` or `{"max_tokens": 2000}`.
Reasoning content is only captured for non-streaming requests.
See [OpenRouter reasoning docs](https://openrouter.ai/docs/use-cases/reasoning-tokens).
- `response_format`: A JSON schema or a Pydantic model that enforces the structure of the model's response.
If provided, the output will always be validated against this
format (unless the model returns a tool call).
For details, see the [OpenAI Structured Outputs documentation](https://platform.openai.com/docs/guides/structured-outputs).
Notes:
- This parameter accepts Pydantic models and JSON schemas for latest models starting from GPT-4o.
- For structured outputs with streaming,
the `response_format` must be a JSON schema and not a Pydantic model.
- `tools`: A list of tools or a Toolset for which the model can prepare calls. This parameter can accept either a
list of `Tool` objects or a `Toolset` instance.
- `timeout`: The timeout for the OpenRouter API call.
- `extra_headers`: Additional HTTP headers to include in requests to the OpenRouter API.
This can be useful for adding site URL or title for rankings on openrouter.ai
For more details, see OpenRouter [docs](https://openrouter.ai/docs/quickstart).
- `max_retries`: Maximum number of retries to contact OpenAI after an internal error.
If not set, it defaults to either the `OPENAI_MAX_RETRIES` environment variable, or set to 5.
- `http_client_kwargs`: A dictionary of keyword arguments to configure a custom `httpx.Client`or `httpx.AsyncClient`.
For more information, see the [HTTPX documentation](https://www.python-httpx.org/api/`client`).
- **tools** (<code>ToolsType | None</code>) – A list of tools or a Toolset for which the model can prepare calls. This parameter can accept either a
list of `Tool` objects or a `Toolset` instance.
- **timeout** (<code>float | None</code>) – The timeout for the OpenRouter API call.
- **extra_headers** (<code>dict\[str, Any\] | None</code>) – Additional HTTP headers to include in requests to the OpenRouter API.
This can be useful for adding site URL or title for rankings on openrouter.ai
For more details, see OpenRouter [docs](https://openrouter.ai/docs/quickstart).
- **max_retries** (<code>int | None</code>) – Maximum number of retries to contact OpenAI after an internal error.
If not set, it defaults to either the `OPENAI_MAX_RETRIES` environment variable, or set to 5.
- **http_client_kwargs** (<code>dict\[str, Any\] | None</code>) – A dictionary of keyword arguments to configure a custom `httpx.Client`or `httpx.AsyncClient`.
For more information, see the [HTTPX documentation](https://www.python-httpx.org/api/#client).
<a id="haystack_integrations.components.generators.openrouter.chat.chat_generator.OpenRouterChatGenerator.to_dict"></a>
#### OpenRouterChatGenerator.to\_dict
#### to_dict
```python
def to_dict() -> dict[str, Any]
to_dict() -> dict[str, Any]
```
Serialize this component to a dictionary.
**Returns**:
**Returns:**
The serialized component as a dictionary.
- <code>dict\[str, Any\]</code> – The serialized component as a dictionary.
#### run
```python
run(
messages: list[ChatMessage] | str,
streaming_callback: StreamingCallbackT | None = None,
generation_kwargs: dict[str, Any] | None = None,
*,
tools: ToolsType | None = None,
tools_strict: bool | None = None
) -> dict[str, list[ChatMessage]]
```
Invokes chat completion on the OpenRouter API.
**Parameters:**
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of ChatMessage instances representing the input messages.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Additional keyword arguments for text generation. These parameters will
override the parameters passed during component initialization.
For details on OpenRouter API parameters, see
[OpenRouter docs](https://openrouter.ai/docs/quickstart).
- **tools** (<code>ToolsType | None</code>) – A list of Tool and/or Toolset objects, or a single Toolset for which the model can prepare calls.
If set, it will override the `tools` parameter provided during initialization.
- **tools_strict** (<code>bool | None</code>) – Whether to enable strict schema adherence for tool calls.
**Returns:**
- <code>dict\[str, list\[ChatMessage\]\]</code> – A dictionary with the following key:
- `replies`: A list containing the generated responses as ChatMessage instances.
#### run_async
```python
run_async(
messages: list[ChatMessage] | str,
streaming_callback: StreamingCallbackT | None = None,
generation_kwargs: dict[str, Any] | None = None,
*,
tools: ToolsType | None = None,
tools_strict: bool | None = None
) -> dict[str, list[ChatMessage]]
```
Asynchronously invokes chat completion on the OpenRouter API.
**Parameters:**
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of ChatMessage instances representing the input messages.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
Must be a coroutine.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Additional keyword arguments for text generation.
- **tools** (<code>ToolsType | None</code>) – A list of Tool and/or Toolset objects, or a single Toolset.
- **tools_strict** (<code>bool | None</code>) – Whether to enable strict schema adherence for tool calls.
**Returns:**
- <code>dict\[str, list\[ChatMessage\]\]</code> – A dictionary with the following key:
- `replies`: A list containing the generated responses as ChatMessage instances.
docs-website/reference/integrations-api/togetherai.md
+137 -149
View File
@@ -5,15 +5,15 @@ description: "Together AI integration for Haystack"
slug: "/integrations-togetherai"
---
<a id="haystack_integrations.components.generators.togetherai.chat.chat_generator"></a>
## Module haystack\_integrations.components.generators.togetherai.chat.chat\_generator
<a id="haystack_integrations.components.generators.togetherai.chat.chat_generator.TogetherAIChatGenerator"></a>
## haystack_integrations.components.generators.togetherai.chat.chat_generator
### TogetherAIChatGenerator
Bases: <code>OpenAIChatGenerator</code>
Enables text generation using Together AI generative models.
For supported models, see [Together AI docs](https://docs.together.ai/docs).
Users can pass any text generation parameters valid for the Together AI chat completion API
@@ -21,6 +21,7 @@ directly to this component using the `generation_kwargs` parameter in `__init__`
parameter in `run` method.
Key Features and Compatibility:
- **Primary Compatibility**: Designed to work seamlessly with the Together AI chat completion endpoint.
- **Streaming Support**: Supports streaming responses from the Together AI chat completion endpoint.
- **Customizability**: Supports all parameters supported by the Together AI chat completion endpoint.
@@ -34,6 +35,7 @@ For more details on the parameters supported by the Together AI API, refer to th
[Together AI API Docs](https://docs.together.ai/reference/chat-completions-1).
Usage example:
```python
from haystack_integrations.components.generators.togetherai import TogetherAIChatGenerator
from haystack.dataclasses import ChatMessage
@@ -51,89 +53,84 @@ print(response)
>>'usage': {'prompt_tokens': 15, 'completion_tokens': 36, 'total_tokens': 51}})]}
```
<a id="haystack_integrations.components.generators.togetherai.chat.chat_generator.TogetherAIChatGenerator.__init__"></a>
#### TogetherAIChatGenerator.\_\_init\_\_
#### __init__
```python
def __init__(*,
api_key: Secret = Secret.from_env_var("TOGETHER_API_KEY"),
model: str = "meta-llama/Llama-3.3-70B-Instruct-Turbo",
streaming_callback: StreamingCallbackT | None = None,
api_base_url: str | None = "https://api.together.xyz/v1",
generation_kwargs: dict[str, Any] | None = None,
tools: ToolsType | None = None,
timeout: float | None = None,
max_retries: int | None = None,
http_client_kwargs: dict[str, Any] | None = None)
__init__(
*,
api_key: Secret = Secret.from_env_var("TOGETHER_API_KEY"),
model: str = "meta-llama/Llama-3.3-70B-Instruct-Turbo",
streaming_callback: StreamingCallbackT | None = None,
api_base_url: str | None = "https://api.together.xyz/v1",
generation_kwargs: dict[str, Any] | None = None,
tools: ToolsType | None = None,
timeout: float | None = None,
max_retries: int | None = None,
http_client_kwargs: dict[str, Any] | None = None
) -> None
```
Creates an instance of TogetherAIChatGenerator. Unless specified otherwise,
Creates an instance of TogetherAIChatGenerator.
the default model is `meta-llama/Llama-3.3-70B-Instruct-Turbo`.
**Parameters:**
**Arguments**:
- `api_key`: The Together API key.
- `model`: The name of the Together AI chat completion model to use.
- `streaming_callback`: A callback function that is called when a new token is received from the stream.
The callback function accepts StreamingChunk as an argument.
- `api_base_url`: The Together AI API Base url.
For more details, see Together AI [docs](https://docs.together.ai/docs/openai-api-compatibility).
- `generation_kwargs`: Other parameters to use for the model. These parameters are all sent directly to
the Together AI endpoint. See [Together AI API docs](https://docs.together.ai/reference/chat-completions-1)
for more details.
Some of the supported parameters:
- **api_key** (<code>Secret</code>) – The Together API key.
- **model** (<code>str</code>) – The name of the Together AI chat completion model to use.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
The callback function accepts StreamingChunk as an argument.
- **api_base_url** (<code>str | None</code>) – The Together AI API Base url.
For more details, see Together AI [docs](https://docs.together.ai/docs/openai-api-compatibility).
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Other parameters to use for the model. These parameters are all sent directly to
the Together AI endpoint. See [Together AI API docs](https://docs.together.ai/reference/chat-completions-1)
for more details.
Some of the supported parameters:
- `max_tokens`: The maximum number of tokens the output text can have.
- `temperature`: What sampling temperature to use. Higher values mean the model will take more risks.
Try 0.9 for more creative applications and 0 (argmax sampling) for ones with a well-defined answer.
Try 0.9 for more creative applications and 0 (argmax sampling) for ones with a well-defined answer.
- `top_p`: An alternative to sampling with temperature, called nucleus sampling, where the model
considers the results of the tokens with top_p probability mass. So 0.1 means only the tokens
comprising the top 10% probability mass are considered.
considers the results of the tokens with top_p probability mass. So 0.1 means only the tokens
comprising the top 10% probability mass are considered.
- `stream`: Whether to stream back partial progress. If set, tokens will be sent as data-only server-sent
events as they become available, with the stream terminated by a data: [DONE] message.
events as they become available, with the stream terminated by a data: [DONE] message.
- `safe_prompt`: Whether to inject a safety prompt before all conversations.
- `random_seed`: The seed to use for random sampling.
- `response_format`: A JSON schema or a Pydantic model that enforces the structure of the model's response.
If provided, the output will always be validated against this
format (unless the model returns a tool call).
For details, see the [OpenAI Structured Outputs documentation](https://platform.openai.com/docs/guides/structured-outputs).
Notes:
- For structured outputs with streaming,
the `response_format` must be a JSON schema and not a Pydantic model.
- `tools`: A list of Tool and/or Toolset objects, or a single Toolset for which the model can prepare calls.
Each tool should have a unique name.
- `timeout`: The timeout for the Together AI API call.
- `max_retries`: Maximum number of retries to contact Together AI after an internal error.
If not set, it defaults to either the `OPENAI_MAX_RETRIES` environment variable, or set to 5.
- `http_client_kwargs`: A dictionary of keyword arguments to configure a custom `httpx.Client`or `httpx.AsyncClient`.
For more information, see the [HTTPX documentation](https://www.python-httpx.org/api/`client`).
If provided, the output will always be validated against this
format (unless the model returns a tool call).
For details, see the [OpenAI Structured Outputs documentation](https://platform.openai.com/docs/guides/structured-outputs).
Notes:
- For structured outputs with streaming,
the `response_format` must be a JSON schema and not a Pydantic model.
- **tools** (<code>ToolsType | None</code>) – A list of Tool and/or Toolset objects, or a single Toolset for which the model can prepare calls.
Each tool should have a unique name.
- **timeout** (<code>float | None</code>) – The timeout for the Together AI API call.
- **max_retries** (<code>int | None</code>) – Maximum number of retries to contact Together AI after an internal error.
If not set, it defaults to either the `OPENAI_MAX_RETRIES` environment variable, or set to 5.
- **http_client_kwargs** (<code>dict\[str, Any\] | None</code>) – A dictionary of keyword arguments to configure a custom `httpx.Client`or `httpx.AsyncClient`.
For more information, see the [HTTPX documentation](https://www.python-httpx.org/api/#client).
<a id="haystack_integrations.components.generators.togetherai.chat.chat_generator.TogetherAIChatGenerator.to_dict"></a>
#### TogetherAIChatGenerator.to\_dict
#### to_dict
```python
def to_dict() -> dict[str, Any]
to_dict() -> dict[str, Any]
```
Serialize this component to a dictionary.
**Returns**:
**Returns:**
The serialized component as a dictionary.
- <code>dict\[str, Any\]</code> – The serialized component as a dictionary.
<a id="haystack_integrations.components.generators.togetherai.generator"></a>
## Module haystack\_integrations.components.generators.togetherai.generator
<a id="haystack_integrations.components.generators.togetherai.generator.TogetherAIGenerator"></a>
## haystack_integrations.components.generators.togetherai.generator
### TogetherAIGenerator
Bases: <code>TogetherAIChatGenerator</code>
Provides an interface to generate text using an LLM running on Together AI.
Usage example:
```python
from haystack_integrations.components.generators.togetherai import TogetherAIGenerator
@@ -145,150 +142,141 @@ generator = TogetherAIGenerator(model="deepseek-ai/DeepSeek-R1",
print(generator.run("Who is the best Italian actor?"))
```
<a id="haystack_integrations.components.generators.togetherai.generator.TogetherAIGenerator.__init__"></a>
#### TogetherAIGenerator.\_\_init\_\_
#### __init__
```python
def __init__(api_key: Secret = Secret.from_env_var("TOGETHER_API_KEY"),
model: str = "meta-llama/Llama-3.3-70B-Instruct-Turbo",
api_base_url: str | None = "https://api.together.xyz/v1",
streaming_callback: StreamingCallbackT | None = None,
system_prompt: str | None = None,
generation_kwargs: dict[str, Any] | None = None,
timeout: float | None = None,
max_retries: int | None = None)
__init__(
api_key: Secret = Secret.from_env_var("TOGETHER_API_KEY"),
model: str = "meta-llama/Llama-3.3-70B-Instruct-Turbo",
api_base_url: str | None = "https://api.together.xyz/v1",
streaming_callback: StreamingCallbackT | None = None,
system_prompt: str | None = None,
generation_kwargs: dict[str, Any] | None = None,
timeout: float | None = None,
max_retries: int | None = None,
) -> None
```
Initialize the TogetherAIGenerator.
**Arguments**:
**Parameters:**
- `api_key`: The Together API key.
- `model`: The name of the model to use.
- `api_base_url`: The base URL of the Together AI API.
- `streaming_callback`: A callback function that is called when a new token is received from the stream.
The callback function accepts StreamingChunk as an argument.
- `system_prompt`: The system prompt to use for text generation. If not provided, the system prompt is
omitted, and the default system prompt of the model is used.
- `generation_kwargs`: Other parameters to use for the model. These parameters are all sent directly to
the Together AI endpoint. See Together AI
[documentation](https://docs.together.ai/reference/chat-completions-1) for more details.
Some of the supported parameters:
- **api_key** (<code>Secret</code>) – The Together API key.
- **model** (<code>str</code>) – The name of the model to use.
- **api_base_url** (<code>str | None</code>) – The base URL of the Together AI API.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
The callback function accepts StreamingChunk as an argument.
- **system_prompt** (<code>str | None</code>) – The system prompt to use for text generation. If not provided, the system prompt is
omitted, and the default system prompt of the model is used.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Other parameters to use for the model. These parameters are all sent directly to
the Together AI endpoint. See Together AI
[documentation](https://docs.together.ai/reference/chat-completions-1) for more details.
Some of the supported parameters:
- `max_tokens`: The maximum number of tokens the output text can have.
- `temperature`: What sampling temperature to use. Higher values mean the model will take more risks.
Try 0.9 for more creative applications and 0 (argmax sampling) for ones with a well-defined answer.
Try 0.9 for more creative applications and 0 (argmax sampling) for ones with a well-defined answer.
- `top_p`: An alternative to sampling with temperature, called nucleus sampling, where the model
considers the results of the tokens with top_p probability mass. So, 0.1 means only the tokens
comprising the top 10% probability mass are considered.
considers the results of the tokens with top_p probability mass. So, 0.1 means only the tokens
comprising the top 10% probability mass are considered.
- `n`: How many completions to generate for each prompt. For example, if the LLM gets 3 prompts and n is 2,
it will generate two completions for each of the three prompts, ending up with 6 completions in total.
it will generate two completions for each of the three prompts, ending up with 6 completions in total.
- `stop`: One or more sequences after which the LLM should stop generating tokens.
- `presence_penalty`: What penalty to apply if a token is already present at all. Bigger values mean
the model will be less likely to repeat the same token in the text.
the model will be less likely to repeat the same token in the text.
- `frequency_penalty`: What penalty to apply if a token has already been generated in the text.
Bigger values mean the model will be less likely to repeat the same token in the text.
Bigger values mean the model will be less likely to repeat the same token in the text.
- `logit_bias`: Add a logit bias to specific tokens. The keys of the dictionary are tokens, and the
values are the bias to add to that token.
- `timeout`: Timeout for together.ai Client calls, if not set it is inferred from the `OPENAI_TIMEOUT` environment
variable or set to 30.
- `max_retries`: Maximum retries to establish contact with Together AI if it returns an internal error, if not set it is
inferred from the `OPENAI_MAX_RETRIES` environment variable or set to 5.
values are the bias to add to that token.
- **timeout** (<code>float | None</code>) – Timeout for together.ai Client calls, if not set it is inferred from the `OPENAI_TIMEOUT` environment
variable or set to 30.
- **max_retries** (<code>int | None</code>) – Maximum retries to establish contact with Together AI if it returns an internal error, if not set it is
inferred from the `OPENAI_MAX_RETRIES` environment variable or set to 5.
<a id="haystack_integrations.components.generators.togetherai.generator.TogetherAIGenerator.to_dict"></a>
#### TogetherAIGenerator.to\_dict
#### to_dict
```python
def to_dict() -> dict[str, Any]
to_dict() -> dict[str, Any]
```
Serialize this component to a dictionary.
**Returns**:
**Returns:**
The serialized component as a dictionary.
- <code>dict\[str, Any\]</code> – The serialized component as a dictionary.
<a id="haystack_integrations.components.generators.togetherai.generator.TogetherAIGenerator.from_dict"></a>
#### TogetherAIGenerator.from\_dict
#### from_dict
```python
@classmethod
def from_dict(cls, data: dict[str, Any]) -> "TogetherAIGenerator"
from_dict(data: dict[str, Any]) -> TogetherAIGenerator
```
Deserialize this component from a dictionary.
**Arguments**:
**Parameters:**
- `data`: The dictionary representation of this component.
- **data** (<code>dict\[str, Any\]</code>) – The dictionary representation of this component.
**Returns**:
**Returns:**
The deserialized component instance.
- <code>TogetherAIGenerator</code> – The deserialized component instance.
<a id="haystack_integrations.components.generators.togetherai.generator.TogetherAIGenerator.run"></a>
#### TogetherAIGenerator.run
#### run
```python
@component.output_types(replies=list[str], meta=list[dict[str, Any]])
def run(*,
prompt: str,
system_prompt: str | None = None,
streaming_callback: StreamingCallbackT | None = None,
generation_kwargs: dict[str, Any] | None = None) -> dict[str, Any]
run(
*,
prompt: str,
system_prompt: str | None = None,
streaming_callback: StreamingCallbackT | None = None,
generation_kwargs: dict[str, Any] | None = None
) -> dict[str, Any]
```
Generate text completions synchronously.
**Arguments**:
**Parameters:**
- `prompt`: The input prompt string for text generation.
- `system_prompt`: An optional system prompt to provide context or instructions for the generation.
If not provided, the system prompt set in the `__init__` method will be used.
- `streaming_callback`: A callback function that is called when a new token is received from the stream.
If provided, this will override the `streaming_callback` set in the `__init__` method.
- `generation_kwargs`: Additional keyword arguments for text generation. These parameters will potentially override the parameters
passed in the `__init__` method. Supported parameters include temperature, max_new_tokens, top_p, etc.
- **prompt** (<code>str</code>) – The input prompt string for text generation.
- **system_prompt** (<code>str | None</code>) – An optional system prompt to provide context or instructions for the generation.
If not provided, the system prompt set in the `__init__` method will be used.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
If provided, this will override the `streaming_callback` set in the `__init__` method.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Additional keyword arguments for text generation. These parameters will potentially override the parameters
passed in the `__init__` method. Supported parameters include temperature, max_new_tokens, top_p, etc.
**Returns**:
**Returns:**
A dictionary with the following keys:
- <code>dict\[str, Any\]</code> – A dictionary with the following keys:
- `replies`: A list of generated text completions as strings.
- `meta`: A list of metadata dictionaries containing information about each generation,
including model name, finish reason, and token usage statistics.
including model name, finish reason, and token usage statistics.
<a id="haystack_integrations.components.generators.togetherai.generator.TogetherAIGenerator.run_async"></a>
#### TogetherAIGenerator.run\_async
#### run_async
```python
@component.output_types(replies=list[str], meta=list[dict[str, Any]])
async def run_async(
*,
prompt: str,
system_prompt: str | None = None,
streaming_callback: StreamingCallbackT | None = None,
generation_kwargs: dict[str, Any] | None = None) -> dict[str, Any]
run_async(
*,
prompt: str,
system_prompt: str | None = None,
streaming_callback: StreamingCallbackT | None = None,
generation_kwargs: dict[str, Any] | None = None
) -> dict[str, Any]
```
Generate text completions asynchronously.
**Arguments**:
**Parameters:**
- `prompt`: The input prompt string for text generation.
- `system_prompt`: An optional system prompt to provide context or instructions for the generation.
- `streaming_callback`: A callback function that is called when a new token is received from the stream.
If provided, this will override the `streaming_callback` set in the `__init__` method.
- `generation_kwargs`: Additional keyword arguments for text generation. These parameters will potentially override the parameters
passed in the `__init__` method. Supported parameters include temperature, max_new_tokens, top_p, etc.
- **prompt** (<code>str</code>) – The input prompt string for text generation.
- **system_prompt** (<code>str | None</code>) – An optional system prompt to provide context or instructions for the generation.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
If provided, this will override the `streaming_callback` set in the `__init__` method.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Additional keyword arguments for text generation. These parameters will potentially override the parameters
passed in the `__init__` method. Supported parameters include temperature, max_new_tokens, top_p, etc.
**Returns**:
**Returns:**
A dictionary with the following keys:
- <code>dict\[str, Any\]</code> – A dictionary with the following keys:
- `replies`: A list of generated text completions as strings.
- `meta`: A list of metadata dictionaries containing information about each generation,
including model name, finish reason, and token usage statistics.
including model name, finish reason, and token usage statistics.
docs-website/reference/integrations-api/transformers.md
+1061
View File
File diff suppressed because it is too large Load Diff
docs-website/reference/integrations-api/vllm.md
+6 -4
View File
@@ -478,7 +478,7 @@ Deserialize this component from a dictionary.
```python
run(
messages: list[ChatMessage],
messages: list[ChatMessage] | str,
streaming_callback: StreamingCallbackT | None = None,
generation_kwargs: dict[str, Any] | None = None,
*,
@@ -490,7 +490,8 @@ Run the VLLM chat generator on the given input data.
**Parameters:**
- **messages** (<code>list\[ChatMessage\]</code>) – A list of ChatMessage instances representing the input messages.
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of ChatMessage instances representing the input messages.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Additional keyword arguments for text generation. These parameters will
override the parameters passed during component initialization.
@@ -508,7 +509,7 @@ Run the VLLM chat generator on the given input data.
```python
run_async(
messages: list[ChatMessage],
messages: list[ChatMessage] | str,
streaming_callback: StreamingCallbackT | None = None,
generation_kwargs: dict[str, Any] | None = None,
*,
@@ -520,7 +521,8 @@ Run the VLLM chat generator on the given input data asynchronously.
**Parameters:**
- **messages** (<code>list\[ChatMessage\]</code>) – A list of ChatMessage instances representing the input messages.
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of ChatMessage instances representing the input messages.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
Must be a coroutine.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Additional keyword arguments for text generation. These parameters will
docs-website/reference/integrations-api/watsonx.md
+8 -6
View File
@@ -54,7 +54,7 @@ __init__(
max_retries: int | None = None,
meta_fields_to_embed: list[str] | None = None,
embedding_separator: str = "\n"
)
) -> None
```
Creates a WatsonxDocumentEmbedder component.
@@ -166,7 +166,7 @@ __init__(
suffix: str = "",
timeout: float | None = None,
max_retries: int | None = None
)
) -> None
```
Creates an WatsonxTextEmbedder component.
@@ -411,7 +411,7 @@ Deserialize this component from a dictionary.
```python
run(
*,
messages: list[ChatMessage],
messages: list[ChatMessage] | str,
generation_kwargs: dict[str, Any] | None = None,
streaming_callback: StreamingCallbackT | None = None,
tools: ToolsType | None = None
@@ -422,7 +422,8 @@ Generate chat completions synchronously.
**Parameters:**
- **messages** (<code>list\[ChatMessage\]</code>) – A list of ChatMessage instances representing the input messages.
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of ChatMessage instances representing the input messages.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Additional keyword arguments for text generation. These parameters will potentially override the parameters
passed in the `__init__` method.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
@@ -440,7 +441,7 @@ Generate chat completions synchronously.
```python
run_async(
*,
messages: list[ChatMessage],
messages: list[ChatMessage] | str,
generation_kwargs: dict[str, Any] | None = None,
streaming_callback: StreamingCallbackT | None = None,
tools: ToolsType | None = None
@@ -451,7 +452,8 @@ Generate chat completions asynchronously.
**Parameters:**
- **messages** (<code>list\[ChatMessage\]</code>) – A list of ChatMessage instances representing the input messages.
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of ChatMessage instances representing the input messages.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Additional keyword arguments for text generation. These parameters will potentially override the parameters
passed in the `__init__` method.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
docs-website/reference_versioned_docs/version-2.18/integrations-api/amazon_bedrock.md
+6 -4
View File
@@ -1311,7 +1311,7 @@ Deserializes the component from a dictionary.
```python
run(
messages: list[ChatMessage],
messages: list[ChatMessage] | str,
streaming_callback: StreamingCallbackT | None = None,
generation_kwargs: dict[str, Any] | None = None,
tools: ToolsType | None = None,
@@ -1324,7 +1324,8 @@ Supports both standard and streaming responses depending on whether a streaming
**Parameters:**
- **messages** (<code>list\[ChatMessage\]</code>) – A list of `ChatMessage` objects forming the chat history.
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of `ChatMessage` objects forming the chat history.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – Optional callback for handling streaming outputs.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Optional dictionary of generation parameters. Some common parameters are:
- `maxTokens`: Maximum number of tokens to generate.
@@ -1347,7 +1348,7 @@ Supports both standard and streaming responses depending on whether a streaming
```python
run_async(
messages: list[ChatMessage],
messages: list[ChatMessage] | str,
streaming_callback: StreamingCallbackT | None = None,
generation_kwargs: dict[str, Any] | None = None,
tools: ToolsType | None = None,
@@ -1360,7 +1361,8 @@ Designed for use cases where non-blocking or concurrent execution is desired.
**Parameters:**
- **messages** (<code>list\[ChatMessage\]</code>) – A list of `ChatMessage` objects forming the chat history.
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of `ChatMessage` objects forming the chat history.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – Optional async-compatible callback for handling streaming outputs.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Optional dictionary of generation parameters. Some common parameters are:
- `maxTokens`: Maximum number of tokens to generate.
docs-website/reference_versioned_docs/version-2.18/integrations-api/anthropic.md
+12 -8
View File
@@ -167,7 +167,7 @@ Deserialize this component from a dictionary.
```python
run(
messages: list[ChatMessage],
messages: list[ChatMessage] | str,
streaming_callback: StreamingCallbackT | None = None,
generation_kwargs: dict[str, Any] | None = None,
tools: ToolsType | None = None,
@@ -178,7 +178,8 @@ Invokes the Anthropic API with the given messages and generation kwargs.
**Parameters:**
- **messages** (<code>list\[ChatMessage\]</code>) – A list of ChatMessage instances representing the input messages.
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of ChatMessage instances representing the input messages.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Optional arguments to pass to the Anthropic generation endpoint.
- **tools** (<code>ToolsType | None</code>) – A list of Tool and/or Toolset objects, or a single Toolset, that the model can use.
@@ -194,7 +195,7 @@ Invokes the Anthropic API with the given messages and generation kwargs.
```python
run_async(
messages: list[ChatMessage],
messages: list[ChatMessage] | str,
streaming_callback: StreamingCallbackT | None = None,
generation_kwargs: dict[str, Any] | None = None,
tools: ToolsType | None = None,
@@ -205,7 +206,8 @@ Async version of the run method. Invokes the Anthropic API with the given messag
**Parameters:**
- **messages** (<code>list\[ChatMessage\]</code>) – A list of ChatMessage instances representing the input messages.
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of ChatMessage instances representing the input messages.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Optional arguments to pass to the Anthropic generation endpoint.
- **tools** (<code>ToolsType | None</code>) – A list of Tool and/or Toolset objects, or a single Toolset, that the model can use.
@@ -360,7 +362,7 @@ This method is idempotent — it only creates clients once.
```python
run(
messages: list[ChatMessage],
messages: list[ChatMessage] | str,
streaming_callback: StreamingCallbackT | None = None,
generation_kwargs: dict[str, Any] | None = None,
tools: ToolsType | None = None,
@@ -371,7 +373,8 @@ Invokes the AnthropicFoundry API with the given messages and generation kwargs.
**Parameters:**
- **messages** (<code>list\[ChatMessage\]</code>) – A list of ChatMessage instances representing the input messages.
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of ChatMessage instances representing the input messages.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Optional arguments to pass to the Anthropic generation endpoint.
- **tools** (<code>ToolsType | None</code>) – A list of Tool and/or Toolset objects, or a single Toolset, that the model can use.
@@ -387,7 +390,7 @@ Invokes the AnthropicFoundry API with the given messages and generation kwargs.
```python
run_async(
messages: list[ChatMessage],
messages: list[ChatMessage] | str,
streaming_callback: StreamingCallbackT | None = None,
generation_kwargs: dict[str, Any] | None = None,
tools: ToolsType | None = None,
@@ -398,7 +401,8 @@ Async version of the run method. Invokes the AnthropicFoundry API with the given
**Parameters:**
- **messages** (<code>list\[ChatMessage\]</code>) – A list of ChatMessage instances representing the input messages.
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of ChatMessage instances representing the input messages.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Optional arguments to pass to the Anthropic generation endpoint.
- **tools** (<code>ToolsType | None</code>) – A list of Tool and/or Toolset objects, or a single Toolset, that the model can use.
docs-website/reference_versioned_docs/version-2.18/integrations-api/arangodb.md
+244
View File
@@ -0,0 +1,244 @@
---
title: "Arangodb"
id: integrations-arangodb
description: "Arangodb integration for Haystack"
slug: "/integrations-arangodb"
---
## haystack_integrations.components.retrievers.arangodb.embedding_retriever
### ArangoEmbeddingRetriever
Retrieves documents from an `ArangoDocumentStore` using vector similarity on embeddings.
The similarity function is configured on the `ArangoDocumentStore` (cosine, dot product, or L2).
Example usage:
```python
from haystack_integrations.document_stores.arangodb import ArangoDocumentStore
from haystack_integrations.components.retrievers.arangodb import ArangoEmbeddingRetriever
store = ArangoDocumentStore(host="http://localhost:8529", database="haystack",
username="root", collection_name="docs", embedding_dimension=768)
retriever = ArangoEmbeddingRetriever(document_store=store, top_k=5)
result = retriever.run(query_embedding=[0.1, 0.2, ...])
```
#### __init__
```python
__init__(
*,
document_store: ArangoDocumentStore,
top_k: int = 10,
filters: dict[str, Any] | None = None
) -> None
```
Creates a new ArangoEmbeddingRetriever.
**Parameters:**
- **document_store** (<code>ArangoDocumentStore</code>) – The `ArangoDocumentStore` to retrieve documents from.
- **top_k** (<code>int</code>) – Maximum number of documents to return.
- **filters** (<code>dict\[str, Any\] | None</code>) – Optional Haystack metadata filters applied at retrieval time.
#### run
```python
run(
query_embedding: list[float],
top_k: int | None = None,
filters: dict[str, Any] | None = None,
) -> dict[str, list[Document]]
```
Retrieves documents most similar to `query_embedding`.
**Parameters:**
- **query_embedding** (<code>list\[float\]</code>) – The query vector.
- **top_k** (<code>int | None</code>) – Overrides the instance-level `top_k` for this call.
- **filters** (<code>dict\[str, Any\] | None</code>) – Overrides the instance-level `filters` for this call.
**Returns:**
- <code>dict\[str, list\[Document\]\]</code> – A dictionary with `documents` — a list of `Document` objects sorted by score.
#### to_dict
```python
to_dict() -> dict[str, Any]
```
Serializes the component to a dictionary.
**Returns:**
- <code>dict\[str, Any\]</code> – Dictionary with serialized data.
#### from_dict
```python
from_dict(data: dict[str, Any]) -> ArangoEmbeddingRetriever
```
Deserializes the component from a dictionary.
**Parameters:**
- **data** (<code>dict\[str, Any\]</code>) – Dictionary to deserialize from.
**Returns:**
- <code>ArangoEmbeddingRetriever</code> – Deserialized component.
## haystack_integrations.document_stores.arangodb.document_store
### ArangoDocumentStore
A Haystack DocumentStore backed by [ArangoDB](https://www.arangodb.com/).
Documents are stored in an ArangoDB collection and support vector similarity search
via AQL vector functions (requires ArangoDB 3.12+).
Example usage:
```python
from haystack_integrations.document_stores.arangodb import ArangoDocumentStore
from haystack.utils import Secret
store = ArangoDocumentStore(
host="http://localhost:8529",
database="haystack",
username=Secret.from_env_var("ARANGO_USERNAME", strict=False),
password=Secret.from_env_var("ARANGO_PASSWORD"),
collection_name="documents",
embedding_dimension=768,
)
```
#### __init__
```python
__init__(
*,
host: str = "http://localhost:8529",
database: str = "haystack",
username: Secret = Secret.from_env_var("ARANGO_USERNAME", strict=False),
password: Secret = Secret.from_env_var("ARANGO_PASSWORD"),
collection_name: str = "haystack_documents",
embedding_dimension: int = 768,
recreate_collection: bool = False,
similarity_function: Literal["cosine", "dot_product", "l2"] = "cosine"
) -> None
```
Creates a new ArangoDocumentStore instance.
**Parameters:**
- **host** (<code>str</code>) – ArangoDB server URL, e.g. `http://localhost:8529`.
- **database** (<code>str</code>) – Name of the ArangoDB database to use. Created if it does not exist.
- **username** (<code>Secret</code>) – ArangoDB username as a `Secret`. Defaults to `ARANGO_USERNAME` env var,
falling back to `root` if the variable is not set.
- **password** (<code>Secret</code>) – ArangoDB password as a `Secret`. Defaults to `ARANGO_PASSWORD` env var.
- **collection_name** (<code>str</code>) – Name of the collection to store documents in.
- **embedding_dimension** (<code>int</code>) – Dimensionality of document embeddings.
- **recreate_collection** (<code>bool</code>) – If `True`, drop and recreate the collection on startup.
- **similarity_function** (<code>Literal['cosine', 'dot_product', 'l2']</code>) – Vector similarity function to use for embedding retrieval.
One of `"cosine"` (default), `"dot_product"`, or `"l2"`.
#### count_documents
```python
count_documents() -> int
```
Returns the number of documents in the store.
**Returns:**
- <code>int</code> – Document count.
#### filter_documents
```python
filter_documents(filters: dict[str, Any] | None = None) -> list[Document]
```
Returns documents matching the provided filters.
**Parameters:**
- **filters** (<code>dict\[str, Any\] | None</code>) – Haystack metadata filters. If `None`, all documents are returned.
**Returns:**
- <code>list\[Document\]</code> – List of matching `Document` objects.
#### write_documents
```python
write_documents(
documents: list[Document], policy: DuplicatePolicy = DuplicatePolicy.NONE
) -> int
```
Writes documents to the store.
**Parameters:**
- **documents** (<code>list\[Document\]</code>) – Documents to write.
- **policy** (<code>DuplicatePolicy</code>) – How to handle duplicates — `OVERWRITE`, `SKIP`, or `FAIL` (default).
**Returns:**
- <code>int</code> – Number of documents written.
**Raises:**
- <code>ValueError</code> – If `documents` contains non-`Document` objects.
- <code>DuplicateDocumentError</code> – If a duplicate is found and policy is `FAIL`.
#### delete_documents
```python
delete_documents(document_ids: list[str]) -> None
```
Deletes documents by their IDs.
**Parameters:**
- **document_ids** (<code>list\[str\]</code>) – List of document IDs to delete.
#### to_dict
```python
to_dict() -> dict[str, Any]
```
Serializes the component to a dictionary.
**Returns:**
- <code>dict\[str, Any\]</code> – Dictionary with serialized data.
#### from_dict
```python
from_dict(data: dict[str, Any]) -> ArangoDocumentStore
```
Deserializes the component from a dictionary.
**Parameters:**
- **data** (<code>dict\[str, Any\]</code>) – Dictionary to deserialize from.
**Returns:**
- <code>ArangoDocumentStore</code> – Deserialized component.
docs-website/reference_versioned_docs/version-2.18/integrations-api/cohere.md
+6 -4
View File
@@ -729,7 +729,7 @@ Deserializes the component from a dictionary.
```python
run(
messages: list[ChatMessage],
messages: list[ChatMessage] | str,
generation_kwargs: dict[str, Any] | None = None,
tools: ToolsType | None = None,
streaming_callback: StreamingCallbackT | None = None,
@@ -740,7 +740,8 @@ Invoke the chat endpoint based on the provided messages and generation parameter
**Parameters:**
- **messages** (<code>list\[ChatMessage\]</code>) – list of `ChatMessage` instances representing the input messages.
- **messages** (<code>list\[ChatMessage\] | str</code>) – list of `ChatMessage` instances representing the input messages.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – additional keyword arguments for chat generation. These parameters will
potentially override the parameters passed in the __init__ method.
For more details on the parameters supported by the Cohere API, refer to the
@@ -759,7 +760,7 @@ Invoke the chat endpoint based on the provided messages and generation parameter
```python
run_async(
messages: list[ChatMessage],
messages: list[ChatMessage] | str,
generation_kwargs: dict[str, Any] | None = None,
tools: ToolsType | None = None,
streaming_callback: StreamingCallbackT | None = None,
@@ -770,7 +771,8 @@ Asynchronously invoke the chat endpoint based on the provided messages and gener
**Parameters:**
- **messages** (<code>list\[ChatMessage\]</code>) – list of `ChatMessage` instances representing the input messages.
- **messages** (<code>list\[ChatMessage\] | str</code>) – list of `ChatMessage` instances representing the input messages.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – additional keyword arguments for chat generation. These parameters will
potentially override the parameters passed in the __init__ method.
For more details on the parameters supported by the Cohere API, refer to the
docs-website/reference_versioned_docs/version-2.18/integrations-api/docling_serve.md
+30 -1
View File
@@ -18,6 +18,27 @@ Enumeration of export formats supported by DoclingServe.
- `TEXT`: Extracts plain text.
- `JSON`: Returns the full Docling document as a JSON string.
### ConversionMode
Bases: <code>str</code>, <code>Enum</code>
Execution mode for DoclingServe conversions.
- `SYNC`: Uses DoclingServe's synchronous conversion endpoints.
- `ASYNC`: Uses DoclingServe's async job endpoints and polls for completion.
### DoclingServeConversionError
Bases: <code>Exception</code>
Raised when DoclingServe reports an async task or conversion failure.
### DoclingServeTimeoutError
Bases: <code>DoclingServeConversionError</code>
Raised when a DoclingServe async task exceeds job_timeout.
### DoclingServeConverter
Converts documents to Haystack Documents using a DoclingServe server.
@@ -54,7 +75,10 @@ __init__(
timeout: float = 120.0,
api_key: Secret | None = Secret.from_env_var(
"DOCLING_SERVE_API_KEY", strict=False
)
),
mode: ConversionMode | str = ConversionMode.SYNC,
poll_interval: float = 2.0,
job_timeout: float = 600.0
) -> None
```
@@ -73,6 +97,11 @@ Initializes the DoclingServeConverter.
- **api_key** (<code>Secret | None</code>) – API key for authenticating with a secured DoclingServe instance. Reads from the
`DOCLING_SERVE_API_KEY` environment variable by default. Set to `None` to disable
authentication.
- **mode** (<code>ConversionMode | str</code>) – Conversion mode. `sync` uses DoclingServe's synchronous endpoints. `async` submits
conversion jobs to DoclingServe's async endpoints and polls until completion.
- **poll_interval** (<code>float</code>) – Controls both the server-side long-poll wait (?wait= parameter) and the maximum local sleep between polls.
A higher value reduces round-trips; a lower value increases polling frequency.
- **job_timeout** (<code>float</code>) – Maximum time in seconds to wait for each async conversion job.
#### to_dict
docs-website/reference_versioned_docs/version-2.18/integrations-api/funasr.md
+157
View File
@@ -0,0 +1,157 @@
---
title: "FunASR"
id: integrations-funasr
description: "FunASR speech-to-text integration for Haystack"
slug: "/integrations-funasr"
---
## haystack_integrations.components.audio.funasr.transcriber
### FunASRTranscriber
Transcribes audio files to Documents using [FunASR](https://github.com/modelscope/FunASR).
FunASR is an open-source speech recognition toolkit from Alibaba DAMO Academy.
It supports 50+ languages, speaker diarization, and timestamp extraction, and runs
entirely locally — no API key required.
Models are downloaded from ModelScope on first use and cached in `~/.cache/modelscope`.
**Usage Example:**
```python
from haystack_integrations.components.audio.funasr import FunASRTranscriber
transcriber = FunASRTranscriber()
result = transcriber.run(sources=["speech.wav", "interview.mp3"])
documents = result["documents"]
```
**Speaker diarization and punctuation:**
```python
from haystack.utils import ComponentDevice
transcriber = FunASRTranscriber(
model="paraformer-zh",
vad_model="fsmn-vad",
punc_model="ct-punc",
spk_model="cam++",
device=ComponentDevice.from_str("cuda"),
)
```
**SenseVoice with inverse text normalisation:**
```python
transcriber = FunASRTranscriber(
model="iic/SenseVoiceSmall",
generation_kwargs={"use_itn": True, "merge_vad": True, "language": "auto"},
)
```
#### __init__
```python
__init__(
*,
model: str = "iic/SenseVoiceSmall",
vad_model: str | None = "fsmn-vad",
punc_model: str | None = "ct-punc",
spk_model: str | None = None,
device: ComponentDevice | None = None,
batch_size_s: int = 300,
store_full_path: bool = False,
generation_kwargs: dict[str, Any] | None = None
) -> None
```
Create a FunASRTranscriber component.
**Parameters:**
- **model** (<code>str</code>) – FunASR model name or local path. Defaults to `"iic/SenseVoiceSmall"`,
a multilingual model supporting 50+ languages that is 5-10x faster than Whisper.
Alternatives include `"paraformer-zh"` (Chinese) or `"paraformer-en"` (English).
Browse available models at https://modelscope.github.io/FunASR/model-selection.html.
- **vad_model** (<code>str | None</code>) – Voice activity detection model used to split long audio into segments.
Set to `None` to process the audio as a single stream.
Browse available VAD models at https://www.modelscope.cn/models.
- **punc_model** (<code>str | None</code>) – Punctuation restoration model. Set to `None` to disable punctuation.
Browse available punctuation models at https://www.modelscope.cn/models.
- **spk_model** (<code>str | None</code>) – Speaker diarization model (e.g. `"cam++"`). When set, a `"speakers"`
key is included in the Document metadata. Defaults to `None` (diarization disabled).
Browse available speaker diarization models at https://www.modelscope.cn/models.
- **device** (<code>ComponentDevice | None</code>) – The device to run inference on. If `None`, the default device is selected
automatically. Use `ComponentDevice.from_str("cuda")` for GPU inference.
- **batch_size_s** (<code>int</code>) – Batch size in seconds for VAD-segmented audio. Larger values
improve throughput at the cost of memory.
- **store_full_path** (<code>bool</code>) – If `True`, store the full audio file path in Document metadata.
If `False` (default), store only the file name.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Extra keyword arguments forwarded to `AutoModel.generate()`.
Use this for model-specific options such as `use_itn=True` or `merge_vad=True`
for SenseVoice, or `hotword="..."` for contextual recognition.
#### warm_up
```python
warm_up() -> None
```
Load the FunASR model into memory.
Models are downloaded from ModelScope on first call and cached locally.
This method is idempotent — calling it multiple times is safe.
#### to_dict
```python
to_dict() -> dict[str, Any]
```
Serialize the component to a dictionary.
**Returns:**
- <code>dict\[str, Any\]</code> – Dictionary with serialized data.
#### from_dict
```python
from_dict(data: dict[str, Any]) -> FunASRTranscriber
```
Deserialize the component from a dictionary.
**Parameters:**
- **data** (<code>dict\[str, Any\]</code>) – Dictionary to deserialize from.
**Returns:**
- <code>FunASRTranscriber</code> – Deserialized component.
#### run
```python
run(
sources: list[str | Path | ByteStream],
meta: dict[str, Any] | list[dict[str, Any]] | None = None,
) -> dict[str, list[Document]]
```
Transcribe audio sources to Documents.
**Parameters:**
- **sources** (<code>list\[str | Path | ByteStream\]</code>) – Audio file paths (`str` or `Path`) or `ByteStream` objects.
Supported formats: WAV, MP3, FLAC, OGG, M4A, AAC, and any format that
FunASR's underlying audio backend (soundfile/ffmpeg) can decode.
- **meta** (<code>dict\[str, Any\] | list\[dict\[str, Any\]\] | None</code>) – Metadata to attach to the produced Documents. Pass a single dict
to apply the same metadata to all Documents, or a list aligned with `sources`.
**Returns:**
- <code>dict\[str, list\[Document\]\]</code> – Dictionary with key `"documents"` — one `Document` per source whose
`content` holds the full transcript text.
docs-website/reference_versioned_docs/version-2.18/integrations-api/google_genai.md
+49 -7
View File
@@ -81,7 +81,9 @@ __init__(
progress_bar: bool = True,
meta_fields_to_embed: list[str] | None = None,
embedding_separator: str = "\n",
config: dict[str, Any] | None = None
config: dict[str, Any] | None = None,
timeout: float | None = None,
max_retries: int | None = None
) -> None
```
@@ -114,6 +116,8 @@ Creates an GoogleGenAIDocumentEmbedder component.
Specifying task types in `config` does not take effect for `gemini-embedding-2`.
See [Gemini documentation](https://ai.google.dev/gemini-api/docs/embeddings#task-types) for more
information.
- **timeout** (<code>float | None</code>) – The timeout in seconds for the underlying Google GenAI client network requests.
- **max_retries** (<code>int | None</code>) – The maximum number of retries for the underlying Google GenAI client network requests.
#### to_dict
@@ -260,7 +264,9 @@ __init__(
model: str = "gemini-embedding-2",
batch_size: int = 6,
progress_bar: bool = True,
config: dict[str, Any] | None = None
config: dict[str, Any] | None = None,
timeout: float | None = None,
max_retries: int | None = None
) -> None
```
@@ -293,6 +299,36 @@ Creates an GoogleGenAIMultimodalDocumentEmbedder component.
You can for example set the output dimensionality of the embedding: `{"output_dimensionality": 768}`.
See [Google API documentation](https://googleapis.github.io/python-genai/genai.html#genai.types.EmbedContentConfig)
for the available options.
- **timeout** (<code>float | None</code>) – The timeout in seconds for the underlying Google GenAI client network requests.
- **max_retries** (<code>int | None</code>) – The maximum number of retries for the underlying Google GenAI client network requests.
#### to_dict
```python
to_dict() -> dict[str, Any]
```
Serializes the component to a dictionary.
**Returns:**
- <code>dict\[str, Any\]</code> – Dictionary with serialized data.
#### from_dict
```python
from_dict(data: dict[str, Any]) -> GoogleGenAIMultimodalDocumentEmbedder
```
Deserializes the component from a dictionary.
**Parameters:**
- **data** (<code>dict\[str, Any\]</code>) – Dictionary to deserialize from.
**Returns:**
- <code>GoogleGenAIMultimodalDocumentEmbedder</code> – Deserialized component.
#### run
@@ -405,7 +441,9 @@ __init__(
model: str = "gemini-embedding-001",
prefix: str = "",
suffix: str = "",
config: dict[str, Any] | None = None
config: dict[str, Any] | None = None,
timeout: float | None = None,
max_retries: int | None = None
) -> None
```
@@ -434,6 +472,8 @@ Creates an GoogleGenAITextEmbedder component.
Specifying task types in `config` does not take effect for `gemini-embedding-2`.
See [Gemini documentation](https://ai.google.dev/gemini-api/docs/embeddings#task-types) for more
information.
- **timeout** (<code>float | None</code>) – The timeout in seconds for the underlying Google GenAI client network requests.
- **max_retries** (<code>int | None</code>) – The maximum number of retries for the underlying Google GenAI client network requests.
#### to_dict
@@ -759,7 +799,7 @@ Deserializes the component from a dictionary.
```python
run(
messages: list[ChatMessage],
messages: list[ChatMessage] | str,
generation_kwargs: dict[str, Any] | None = None,
safety_settings: list[dict[str, Any]] | None = None,
streaming_callback: StreamingCallbackT | None = None,
@@ -771,7 +811,8 @@ Run the Google Gen AI chat generator on the given input data.
**Parameters:**
- **messages** (<code>list\[ChatMessage\]</code>) – A list of ChatMessage instances representing the input messages.
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of ChatMessage instances representing the input messages.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Configuration for generation. If provided, it will override
the default config. Supports `thinking_budget` for Gemini 2.5 series thinking configuration.
- **safety_settings** (<code>list\[dict\[str, Any\]\] | None</code>) – Safety settings for content filtering. If provided, it will override the
@@ -796,7 +837,7 @@ Run the Google Gen AI chat generator on the given input data.
```python
run_async(
messages: list[ChatMessage],
messages: list[ChatMessage] | str,
generation_kwargs: dict[str, Any] | None = None,
safety_settings: list[dict[str, Any]] | None = None,
streaming_callback: StreamingCallbackT | None = None,
@@ -808,7 +849,8 @@ Async version of the run method. Run the Google Gen AI chat generator on the giv
**Parameters:**
- **messages** (<code>list\[ChatMessage\]</code>) – A list of ChatMessage instances representing the input messages.
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of ChatMessage instances representing the input messages.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Configuration for generation. If provided, it will override
the default config. Supports `thinking_budget` for Gemini 2.5 series thinking configuration.
See https://ai.google.dev/gemini-api/docs/thinking for possible values.
docs-website/reference_versioned_docs/version-2.18/integrations-api/huggingface_api.md
+796
View File
@@ -0,0 +1,796 @@
---
title: "Hugging Face API"
id: integrations-huggingface-api
description: "Hugging Face API integration for Haystack"
slug: "/integrations-huggingface-api"
---
## haystack_integrations.components.embedders.huggingface_api.document_embedder
### HuggingFaceAPIDocumentEmbedder
Embeds documents using Hugging Face APIs.
Use it with the following Hugging Face APIs:
- [Free Serverless Inference API](https://huggingface.co/inference-api)
- [Paid Inference Endpoints](https://huggingface.co/inference-endpoints)
- [Self-hosted Text Embeddings Inference](https://github.com/huggingface/text-embeddings-inference)
### Usage examples
#### With free serverless inference API
```python
from haystack_integrations.components.embedders.huggingface_api import HuggingFaceAPIDocumentEmbedder
from haystack.utils import Secret
from haystack.dataclasses import Document
doc = Document(content="I love pizza!")
doc_embedder = HuggingFaceAPIDocumentEmbedder(api_type="serverless_inference_api",
api_params={"model": "BAAI/bge-small-en-v1.5"},
token=Secret.from_token("<your-api-key>"))
result = document_embedder.run([doc])
print(result["documents"][0].embedding)
# [0.017020374536514282, -0.023255806416273117, ...]
```
#### With paid inference endpoints
```python
from haystack_integrations.components.embedders.huggingface_api import HuggingFaceAPIDocumentEmbedder
from haystack.utils import Secret
from haystack.dataclasses import Document
doc = Document(content="I love pizza!")
doc_embedder = HuggingFaceAPIDocumentEmbedder(api_type="inference_endpoints",
api_params={"url": "<your-inference-endpoint-url>"},
token=Secret.from_token("<your-api-key>"))
result = document_embedder.run([doc])
print(result["documents"][0].embedding)
# [0.017020374536514282, -0.023255806416273117, ...]
```
#### With self-hosted text embeddings inference
```python
from haystack_integrations.components.embedders.huggingface_api import HuggingFaceAPIDocumentEmbedder
from haystack.dataclasses import Document
doc = Document(content="I love pizza!")
doc_embedder = HuggingFaceAPIDocumentEmbedder(api_type="text_embeddings_inference",
api_params={"url": "http://localhost:8080"})
result = document_embedder.run([doc])
print(result["documents"][0].embedding)
# [0.017020374536514282, -0.023255806416273117, ...]
```
#### __init__
```python
__init__(
api_type: HFEmbeddingAPIType | str,
api_params: dict[str, str],
token: Secret | None = Secret.from_env_var(
["HF_API_TOKEN", "HF_TOKEN"], strict=False
),
prefix: str = "",
suffix: str = "",
truncate: bool | None = True,
normalize: bool | None = False,
batch_size: int = 32,
progress_bar: bool = True,
meta_fields_to_embed: list[str] | None = None,
embedding_separator: str = "\n",
concurrency_limit: int = 4,
) -> None
```
Creates a HuggingFaceAPIDocumentEmbedder component.
**Parameters:**
- **api_type** (<code>HFEmbeddingAPIType | str</code>) – The type of Hugging Face API to use.
- **api_params** (<code>dict\[str, str\]</code>) – A dictionary with the following keys:
- `model`: Hugging Face model ID. Required when `api_type` is `SERVERLESS_INFERENCE_API`.
- `url`: URL of the inference endpoint. Required when `api_type` is `INFERENCE_ENDPOINTS` or
`TEXT_EMBEDDINGS_INFERENCE`.
- **token** (<code>Secret | None</code>) – The Hugging Face token to use as HTTP bearer authorization.
Check your HF token in your [account settings](https://huggingface.co/settings/tokens).
- **prefix** (<code>str</code>) – A string to add at the beginning of each text.
- **suffix** (<code>str</code>) – A string to add at the end of each text.
- **truncate** (<code>bool | None</code>) – Truncates the input text to the maximum length supported by the model.
Applicable when `api_type` is `TEXT_EMBEDDINGS_INFERENCE`, or `INFERENCE_ENDPOINTS`
if the backend uses Text Embeddings Inference.
If `api_type` is `SERVERLESS_INFERENCE_API`, this parameter is ignored.
- **normalize** (<code>bool | None</code>) – Normalizes the embeddings to unit length.
Applicable when `api_type` is `TEXT_EMBEDDINGS_INFERENCE`, or `INFERENCE_ENDPOINTS`
if the backend uses Text Embeddings Inference.
If `api_type` is `SERVERLESS_INFERENCE_API`, this parameter is ignored.
- **batch_size** (<code>int</code>) – Number of documents to process at once.
- **progress_bar** (<code>bool</code>) – If `True`, shows a progress bar when running.
- **meta_fields_to_embed** (<code>list\[str\] | None</code>) – List of metadata fields to embed along with the document text.
- **embedding_separator** (<code>str</code>) – Separator used to concatenate the metadata fields to the document text.
- **concurrency_limit** (<code>int</code>) – The maximum number of requests that should be allowed to run concurrently.
This parameter is only used in the `run_async` method.
**Raises:**
- <code>ValueError</code> – If the required `model` or `url` is missing from `api_params`, the `url` is invalid,
or the `api_type` is unknown.
#### to_dict
```python
to_dict() -> dict[str, Any]
```
Serializes the component to a dictionary.
**Returns:**
- <code>dict\[str, Any\]</code> – Dictionary with serialized data.
#### from_dict
```python
from_dict(data: dict[str, Any]) -> HuggingFaceAPIDocumentEmbedder
```
Deserializes the component from a dictionary.
**Parameters:**
- **data** (<code>dict\[str, Any\]</code>) – Dictionary to deserialize from.
**Returns:**
- <code>HuggingFaceAPIDocumentEmbedder</code> – Deserialized component.
#### run
```python
run(documents: list[Document]) -> dict[str, list[Document]]
```
Embeds a list of documents.
**Parameters:**
- **documents** (<code>list\[Document\]</code>) – Documents to embed.
**Returns:**
- <code>dict\[str, list\[Document\]\]</code> – A dictionary with the following keys:
- `documents`: A list of documents with embeddings.
**Raises:**
- <code>TypeError</code> – If `documents` is not a list of Documents.
- <code>ValueError</code> – If the embeddings returned by the API have an unexpected shape.
#### run_async
```python
run_async(documents: list[Document]) -> dict[str, list[Document]]
```
Embeds a list of documents asynchronously.
**Parameters:**
- **documents** (<code>list\[Document\]</code>) – Documents to embed.
**Returns:**
- <code>dict\[str, list\[Document\]\]</code> – A dictionary with the following keys:
- `documents`: A list of documents with embeddings.
**Raises:**
- <code>TypeError</code> – If `documents` is not a list of Documents.
- <code>ValueError</code> – If the embeddings returned by the API have an unexpected shape.
## haystack_integrations.components.embedders.huggingface_api.text_embedder
### HuggingFaceAPITextEmbedder
Embeds strings using Hugging Face APIs.
Use it with the following Hugging Face APIs:
- [Free Serverless Inference API](https://huggingface.co/inference-api)
- [Paid Inference Endpoints](https://huggingface.co/inference-endpoints)
- [Self-hosted Text Embeddings Inference](https://github.com/huggingface/text-embeddings-inference)
### Usage examples
#### With free serverless inference API
```python
from haystack_integrations.components.embedders.huggingface_api import HuggingFaceAPITextEmbedder
from haystack.utils import Secret
text_embedder = HuggingFaceAPITextEmbedder(api_type="serverless_inference_api",
api_params={"model": "BAAI/bge-small-en-v1.5"},
token=Secret.from_token("<your-api-key>"))
print(text_embedder.run("I love pizza!"))
# {'embedding': [0.017020374536514282, -0.023255806416273117, ...],
```
#### With paid inference endpoints
```python
from haystack_integrations.components.embedders.huggingface_api import HuggingFaceAPITextEmbedder
from haystack.utils import Secret
text_embedder = HuggingFaceAPITextEmbedder(api_type="inference_endpoints",
api_params={"model": "BAAI/bge-small-en-v1.5"},
token=Secret.from_token("<your-api-key>"))
print(text_embedder.run("I love pizza!"))
# {'embedding': [0.017020374536514282, -0.023255806416273117, ...],
```
#### With self-hosted text embeddings inference
```python
from haystack_integrations.components.embedders.huggingface_api import HuggingFaceAPITextEmbedder
from haystack.utils import Secret
text_embedder = HuggingFaceAPITextEmbedder(api_type="text_embeddings_inference",
api_params={"url": "http://localhost:8080"})
print(text_embedder.run("I love pizza!"))
# {'embedding': [0.017020374536514282, -0.023255806416273117, ...],
```
#### __init__
```python
__init__(
api_type: HFEmbeddingAPIType | str,
api_params: dict[str, str],
token: Secret | None = Secret.from_env_var(
["HF_API_TOKEN", "HF_TOKEN"], strict=False
),
prefix: str = "",
suffix: str = "",
truncate: bool | None = True,
normalize: bool | None = False,
) -> None
```
Creates a HuggingFaceAPITextEmbedder component.
**Parameters:**
- **api_type** (<code>HFEmbeddingAPIType | str</code>) – The type of Hugging Face API to use.
- **api_params** (<code>dict\[str, str\]</code>) – A dictionary with the following keys:
- `model`: Hugging Face model ID. Required when `api_type` is `SERVERLESS_INFERENCE_API`.
- `url`: URL of the inference endpoint. Required when `api_type` is `INFERENCE_ENDPOINTS` or
`TEXT_EMBEDDINGS_INFERENCE`.
- **token** (<code>Secret | None</code>) – The Hugging Face token to use as HTTP bearer authorization.
Check your HF token in your [account settings](https://huggingface.co/settings/tokens).
- **prefix** (<code>str</code>) – A string to add at the beginning of each text.
- **suffix** (<code>str</code>) – A string to add at the end of each text.
- **truncate** (<code>bool | None</code>) – Truncates the input text to the maximum length supported by the model.
Applicable when `api_type` is `TEXT_EMBEDDINGS_INFERENCE`, or `INFERENCE_ENDPOINTS`
if the backend uses Text Embeddings Inference.
If `api_type` is `SERVERLESS_INFERENCE_API`, this parameter is ignored.
- **normalize** (<code>bool | None</code>) – Normalizes the embeddings to unit length.
Applicable when `api_type` is `TEXT_EMBEDDINGS_INFERENCE`, or `INFERENCE_ENDPOINTS`
if the backend uses Text Embeddings Inference.
If `api_type` is `SERVERLESS_INFERENCE_API`, this parameter is ignored.
**Raises:**
- <code>ValueError</code> – If the required `model` or `url` is missing from `api_params`, the `url` is invalid,
or the `api_type` is unknown.
#### to_dict
```python
to_dict() -> dict[str, Any]
```
Serializes the component to a dictionary.
**Returns:**
- <code>dict\[str, Any\]</code> – Dictionary with serialized data.
#### from_dict
```python
from_dict(data: dict[str, Any]) -> HuggingFaceAPITextEmbedder
```
Deserializes the component from a dictionary.
**Parameters:**
- **data** (<code>dict\[str, Any\]</code>) – Dictionary to deserialize from.
**Returns:**
- <code>HuggingFaceAPITextEmbedder</code> – Deserialized component.
#### run
```python
run(text: str) -> dict[str, Any]
```
Embeds a single string.
**Parameters:**
- **text** (<code>str</code>) – Text to embed.
**Returns:**
- <code>dict\[str, Any\]</code> – A dictionary with the following keys:
- `embedding`: The embedding of the input text.
**Raises:**
- <code>TypeError</code> – If `text` is not a string.
- <code>ValueError</code> – If the embedding returned by the API has an unexpected shape.
#### run_async
```python
run_async(text: str) -> dict[str, Any]
```
Embeds a single string asynchronously.
**Parameters:**
- **text** (<code>str</code>) – Text to embed.
**Returns:**
- <code>dict\[str, Any\]</code> – A dictionary with the following keys:
- `embedding`: The embedding of the input text.
**Raises:**
- <code>TypeError</code> – If `text` is not a string.
- <code>ValueError</code> – If the embedding returned by the API has an unexpected shape.
## haystack_integrations.components.generators.huggingface_api.chat.chat_generator
### HuggingFaceAPIChatGenerator
Completes chats using Hugging Face APIs.
HuggingFaceAPIChatGenerator uses the [ChatMessage](https://docs.haystack.deepset.ai/docs/chatmessage)
format for input and output. Use it to generate text with Hugging Face APIs:
- [Serverless Inference API (Inference Providers)](https://huggingface.co/docs/inference-providers)
- [Paid Inference Endpoints](https://huggingface.co/inference-endpoints)
- [Self-hosted Text Generation Inference](https://github.com/huggingface/text-generation-inference)
### Usage examples
#### With the serverless inference API (Inference Providers) - free tier available
```python
from haystack_integrations.components.generators.huggingface_api import HuggingFaceAPIChatGenerator
from haystack.dataclasses import ChatMessage
from haystack.utils import Secret
from haystack_integrations.components.common.huggingface_api.utils import HFGenerationAPIType
messages = [ChatMessage.from_system("\nYou are a helpful, respectful and honest assistant"),
ChatMessage.from_user("What's Natural Language Processing?")]
# the api_type can be expressed using the HFGenerationAPIType enum or as a string
api_type = HFGenerationAPIType.SERVERLESS_INFERENCE_API
api_type = "serverless_inference_api" # this is equivalent to the above
generator = HuggingFaceAPIChatGenerator(api_type=api_type,
api_params={"model": "Qwen/Qwen2.5-7B-Instruct",
"provider": "together"},
token=Secret.from_token("<your-api-key>"))
result = generator.run(messages)
print(result)
```
#### With the serverless inference API (Inference Providers) and text+image input
```python
from haystack_integrations.components.generators.huggingface_api import HuggingFaceAPIChatGenerator
from haystack.dataclasses import ChatMessage, ImageContent
from haystack.utils import Secret
from haystack_integrations.components.common.huggingface_api.utils import HFGenerationAPIType
# Create an image from file path, URL, or base64
image = ImageContent.from_file_path("path/to/your/image.jpg")
# Create a multimodal message with both text and image
messages = [ChatMessage.from_user(content_parts=["Describe this image in detail", image])]
generator = HuggingFaceAPIChatGenerator(
api_type=HFGenerationAPIType.SERVERLESS_INFERENCE_API,
api_params={
"model": "Qwen/Qwen2.5-VL-7B-Instruct", # Vision Language Model
"provider": "hyperbolic"
},
token=Secret.from_token("<your-api-key>")
)
result = generator.run(messages)
print(result)
```
#### With paid inference endpoints
```python
from haystack_integrations.components.generators.huggingface_api import HuggingFaceAPIChatGenerator
from haystack.dataclasses import ChatMessage
from haystack.utils import Secret
messages = [ChatMessage.from_system("\nYou are a helpful, respectful and honest assistant"),
ChatMessage.from_user("What's Natural Language Processing?")]
generator = HuggingFaceAPIChatGenerator(api_type="inference_endpoints",
api_params={"url": "<your-inference-endpoint-url>"},
token=Secret.from_token("<your-api-key>"))
result = generator.run(messages)
print(result)
```
#### With self-hosted text generation inference
```python
from haystack_integrations.components.generators.huggingface_api import HuggingFaceAPIChatGenerator
from haystack.dataclasses import ChatMessage
messages = [ChatMessage.from_system("\nYou are a helpful, respectful and honest assistant"),
ChatMessage.from_user("What's Natural Language Processing?")]
generator = HuggingFaceAPIChatGenerator(api_type="text_generation_inference",
api_params={"url": "http://localhost:8080"})
result = generator.run(messages)
print(result)
```
#### __init__
```python
__init__(
api_type: HFGenerationAPIType | str,
api_params: dict[str, str],
token: Secret | None = Secret.from_env_var(
["HF_API_TOKEN", "HF_TOKEN"], strict=False
),
generation_kwargs: dict[str, Any] | None = None,
stop_words: list[str] | None = None,
streaming_callback: StreamingCallbackT | None = None,
tools: ToolsType | None = None,
) -> None
```
Initialize the HuggingFaceAPIChatGenerator instance.
**Parameters:**
- **api_type** (<code>HFGenerationAPIType | str</code>) – The type of Hugging Face API to use. Available types:
- `text_generation_inference`: See [TGI](https://github.com/huggingface/text-generation-inference).
- `inference_endpoints`: See [Inference Endpoints](https://huggingface.co/inference-endpoints).
- `serverless_inference_api`: See
[Serverless Inference API - Inference Providers](https://huggingface.co/docs/inference-providers).
- **api_params** (<code>dict\[str, str\]</code>) – A dictionary with the following keys:
- `model`: Hugging Face model ID. Required when `api_type` is `SERVERLESS_INFERENCE_API`.
- `provider`: Provider name. Recommended when `api_type` is `SERVERLESS_INFERENCE_API`.
- `url`: URL of the inference endpoint. Required when `api_type` is `INFERENCE_ENDPOINTS` or
`TEXT_GENERATION_INFERENCE`.
- Other parameters specific to the chosen API type, such as `timeout`, `headers`, etc.
- **token** (<code>Secret | None</code>) – The Hugging Face token to use as HTTP bearer authorization.
Check your HF token in your [account settings](https://huggingface.co/settings/tokens).
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – A dictionary with keyword arguments to customize text generation.
Some examples: `max_tokens`, `temperature`, `top_p`.
For details, see [Hugging Face chat_completion documentation](https://huggingface.co/docs/huggingface_hub/package_reference/inference_client#huggingface_hub.InferenceClient.chat_completion).
- **stop_words** (<code>list\[str\] | None</code>) – An optional list of strings representing the stop words.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – An optional callable for handling streaming responses.
- **tools** (<code>ToolsType | None</code>) – A list of Tool and/or Toolset objects, or a single Toolset for which the model can prepare calls.
The chosen model should support tool/function calling, according to the model card.
Support for tools in the Hugging Face API and TGI is not yet fully refined and you may experience
unexpected behavior.
**Raises:**
- <code>ValueError</code> – If the required `model` or `url` is missing from `api_params`, the `url` is invalid, the `api_type`
is unknown, `tools` and `streaming_callback` are used together, or duplicate tool names are provided.
#### warm_up
```python
warm_up() -> None
```
Warm up the Hugging Face API chat generator.
This will warm up the tools registered in the chat generator.
This method is idempotent and will only warm up the tools once.
#### to_dict
```python
to_dict() -> dict[str, Any]
```
Serialize this component to a dictionary.
**Returns:**
- <code>dict\[str, Any\]</code> – A dictionary containing the serialized component.
#### from_dict
```python
from_dict(data: dict[str, Any]) -> HuggingFaceAPIChatGenerator
```
Deserialize this component from a dictionary.
#### run
```python
run(
messages: list[ChatMessage] | str,
generation_kwargs: dict[str, Any] | None = None,
tools: ToolsType | None = None,
streaming_callback: StreamingCallbackT | None = None,
) -> dict[str, list[ChatMessage]]
```
Invoke the text generation inference based on the provided messages and generation parameters.
**Parameters:**
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of ChatMessage objects representing the input messages. If a string is provided, it is converted
to a list containing a ChatMessage with user role.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Additional keyword arguments for text generation.
- **tools** (<code>ToolsType | None</code>) – A list of tools or a Toolset for which the model can prepare calls. If set, it will override
the `tools` parameter set during component initialization. This parameter can accept either a
list of `Tool` objects or a `Toolset` instance.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – An optional callable for handling streaming responses. If set, it will override the `streaming_callback`
parameter set during component initialization.
**Returns:**
- <code>dict\[str, list\[ChatMessage\]\]</code> – A dictionary with the following keys:
- `replies`: A list containing the generated responses as ChatMessage objects.
**Raises:**
- <code>ValueError</code> – If `tools` and a streaming callback are used together, or if duplicate tool names are provided.
#### run_async
```python
run_async(
messages: list[ChatMessage] | str,
generation_kwargs: dict[str, Any] | None = None,
tools: ToolsType | None = None,
streaming_callback: StreamingCallbackT | None = None,
) -> dict[str, list[ChatMessage]]
```
Asynchronously invokes the text generation inference based on the provided messages and generation parameters.
This is the asynchronous version of the `run` method. It has the same parameters
and return values but can be used with `await` in an async code.
**Parameters:**
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of ChatMessage objects representing the input messages. If a string is provided, it is converted
to a list containing a ChatMessage with user role.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Additional keyword arguments for text generation.
- **tools** (<code>ToolsType | None</code>) – A list of tools or a Toolset for which the model can prepare calls. If set, it will override the `tools`
parameter set during component initialization. This parameter can accept either a list of `Tool` objects
or a `Toolset` instance.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – An optional callable for handling streaming responses. If set, it will override the `streaming_callback`
parameter set during component initialization.
**Returns:**
- <code>dict\[str, list\[ChatMessage\]\]</code> – A dictionary with the following keys:
- `replies`: A list containing the generated responses as ChatMessage objects.
**Raises:**
- <code>ValueError</code> – If `tools` and a streaming callback are used together, or if duplicate tool names are provided.
## haystack_integrations.components.rankers.huggingface_api.ranker
### TruncationDirection
Bases: <code>str</code>, <code>Enum</code>
Defines the direction to truncate text when input length exceeds the model's limit.
Attributes:
LEFT: Truncate text from the left side (start of text).
RIGHT: Truncate text from the right side (end of text).
### HuggingFaceTEIRanker
Ranks documents based on their semantic similarity to the query.
It can be used with a Text Embeddings Inference (TEI) API endpoint:
- [Self-hosted Text Embeddings Inference](https://github.com/huggingface/text-embeddings-inference)
- [Hugging Face Inference Endpoints](https://huggingface.co/inference-endpoints)
Usage example:
```python
from haystack import Document
from haystack.utils import Secret
from haystack_integrations.components.rankers.huggingface_api import HuggingFaceTEIRanker
reranker = HuggingFaceTEIRanker(
url="http://localhost:8080",
top_k=5,
timeout=30,
token=Secret.from_token("my_api_token")
)
docs = [Document(content="The capital of France is Paris"), Document(content="The capital of Germany is Berlin")]
result = reranker.run(query="What is the capital of France?", documents=docs)
ranked_docs = result["documents"]
print(ranked_docs)
# >> {'documents': [Document(id=..., content: 'the capital of France is Paris', score: 0.9979767),
# >> Document(id=..., content: 'the capital of Germany is Berlin', score: 0.13982213)]}
```
#### __init__
```python
__init__(
*,
url: str,
top_k: int = 10,
raw_scores: bool = False,
timeout: int | None = 30,
max_retries: int = 3,
retry_status_codes: list[int] | None = None,
token: Secret | None = Secret.from_env_var(
["HF_API_TOKEN", "HF_TOKEN"], strict=False
)
) -> None
```
Initializes the TEI reranker component.
**Parameters:**
- **url** (<code>str</code>) – Base URL of the TEI reranking service (for example, "https://api.example.com").
- **top_k** (<code>int</code>) – Maximum number of top documents to return.
- **raw_scores** (<code>bool</code>) – If True, include raw relevance scores in the API payload.
- **timeout** (<code>int | None</code>) – Request timeout in seconds.
- **max_retries** (<code>int</code>) – Maximum number of retry attempts for failed requests.
- **retry_status_codes** (<code>list\[int\] | None</code>) – List of HTTP status codes that will trigger a retry.
When None, HTTP 408, 418, 429 and 503 will be retried (default: None).
- **token** (<code>Secret | None</code>) – The Hugging Face token to use as HTTP bearer authorization. Not always required
depending on your TEI server configuration.
Check your HF token in your [account settings](https://huggingface.co/settings/tokens).
#### to_dict
```python
to_dict() -> dict[str, Any]
```
Serializes the component to a dictionary.
**Returns:**
- <code>dict\[str, Any\]</code> – Dictionary with serialized data.
#### from_dict
```python
from_dict(data: dict[str, Any]) -> HuggingFaceTEIRanker
```
Deserializes the component from a dictionary.
**Parameters:**
- **data** (<code>dict\[str, Any\]</code>) – Dictionary to deserialize from.
**Returns:**
- <code>HuggingFaceTEIRanker</code> – Deserialized component.
#### run
```python
run(
query: str,
documents: list[Document],
top_k: int | None = None,
truncation_direction: TruncationDirection | None = None,
) -> dict[str, list[Document]]
```
Reranks the provided documents by relevance to the query using the TEI API.
Before ranking, documents are deduplicated by their id, retaining only the document with the highest score
if a score is present.
**Parameters:**
- **query** (<code>str</code>) – The user query string to guide reranking.
- **documents** (<code>list\[Document\]</code>) – List of `Document` objects to rerank.
- **top_k** (<code>int | None</code>) – Optional override for the maximum number of documents to return.
- **truncation_direction** (<code>TruncationDirection | None</code>) – If set, enables text truncation in the specified direction.
**Returns:**
- <code>dict\[str, list\[Document\]\]</code> – A dictionary with the following keys:
- `documents`: A list of reranked documents.
**Raises:**
- <code>RuntimeError</code> – - If the API request fails.
- <code>RuntimeError</code> – - If the API returns an error response.
- <code>TypeError</code> – - If the API response is not in the expected list format.
#### run_async
```python
run_async(
query: str,
documents: list[Document],
top_k: int | None = None,
truncation_direction: TruncationDirection | None = None,
) -> dict[str, list[Document]]
```
Asynchronously reranks the provided documents by relevance to the query using the TEI API.
Before ranking, documents are deduplicated by their id, retaining only the document with the highest score
if a score is present.
**Parameters:**
- **query** (<code>str</code>) – The user query string to guide reranking.
- **documents** (<code>list\[Document\]</code>) – List of `Document` objects to rerank.
- **top_k** (<code>int | None</code>) – Optional override for the maximum number of documents to return.
- **truncation_direction** (<code>TruncationDirection | None</code>) – If set, enables text truncation in the specified direction.
**Returns:**
- <code>dict\[str, list\[Document\]\]</code> – A dictionary with the following keys:
- `documents`: A list of reranked documents.
**Raises:**
- <code>httpx.RequestError</code> – - If the API request fails.
- <code>RuntimeError</code> – - If the API returns an error response.
- <code>TypeError</code> – - If the API response is not in the expected list format.
docs-website/reference_versioned_docs/version-2.18/integrations-api/litellm.md
+17 -4
View File
@@ -73,7 +73,7 @@ Create a LiteLLMChatGenerator instance.
```python
run(
messages: list[ChatMessage],
messages: list[ChatMessage] | str,
streaming_callback: StreamingCallbackT | None = None,
generation_kwargs: dict[str, Any] | None = None,
*,
@@ -85,7 +85,8 @@ Invoke chat completion via LiteLLM.
**Parameters:**
- **messages** (<code>list\[ChatMessage\]</code>) – Input messages as ChatMessage instances.
- **messages** (<code>list\[ChatMessage\] | str</code>) – Input messages as ChatMessage instances.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – Override the streaming callback for this call.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Override generation parameters for this call.
- **tools** (<code>ToolsType | None</code>) – Override tools for this call.
@@ -98,7 +99,7 @@ Invoke chat completion via LiteLLM.
```python
run_async(
messages: list[ChatMessage],
messages: list[ChatMessage] | str,
streaming_callback: StreamingCallbackT | None = None,
generation_kwargs: dict[str, Any] | None = None,
*,
@@ -106,7 +107,19 @@ run_async(
) -> dict[str, list[ChatMessage]]
```
Async version of run().
Async version of run(). Invoke chat completion via LiteLLM.
**Parameters:**
- **messages** (<code>list\[ChatMessage\] | str</code>) – Input messages as ChatMessage instances.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – Override the streaming callback for this call.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Override generation parameters for this call.
- **tools** (<code>ToolsType | None</code>) – Override tools for this call.
**Returns:**
- <code>dict\[str, list\[ChatMessage\]\]</code> – A dict with key `replies` containing ChatMessage instances.
#### to_dict
docs-website/reference_versioned_docs/version-2.18/integrations-api/llama_cpp.md
+131 -129
View File
@@ -5,11 +5,8 @@ description: "Llama.cpp integration for Haystack"
slug: "/integrations-llama-cpp"
---
<a id="haystack_integrations.components.generators.llama_cpp.chat.chat_generator"></a>
## Module haystack\_integrations.components.generators.llama\_cpp.chat.chat\_generator
<a id="haystack_integrations.components.generators.llama_cpp.chat.chat_generator.LlamaCppChatGenerator"></a>
## haystack_integrations.components.generators.llama_cpp.chat.chat_generator
### LlamaCppChatGenerator
@@ -20,6 +17,7 @@ It employs the quantized GGUF format, suitable for running these models on stand
Supports both text-only and multimodal (text + image) models like LLaVA.
Usage example:
```python
from haystack_integrations.components.generators.llama_cpp import LlamaCppChatGenerator
user_message = [ChatMessage.from_user("Who is the best American actor?")]
@@ -30,6 +28,7 @@ print(generator.run(user_message, generation_kwargs={"max_tokens": 128}))
```
Usage example with multimodal (image + text):
```python
from haystack.dataclasses import ChatMessage, ImageContent
@@ -46,94 +45,95 @@ generator = LlamaCppChatGenerator(
model_clip_path="mmproj-model-f16.gguf", # CLIP model
n_ctx=4096 # Larger context for image processing
)
generator.warm_up()
result = generator.run(messages)
print(result)
```
<a id="haystack_integrations.components.generators.llama_cpp.chat.chat_generator.LlamaCppChatGenerator.__init__"></a>
#### LlamaCppChatGenerator.\_\_init\_\_
#### __init__
```python
def __init__(model: str,
n_ctx: int | None = 0,
n_batch: int | None = 512,
model_kwargs: dict[str, Any] | None = None,
generation_kwargs: dict[str, Any] | None = None,
*,
tools: ToolsType | None = None,
streaming_callback: StreamingCallbackT | None = None,
chat_handler_name: str | None = None,
model_clip_path: str | None = None) -> None
__init__(
model: str,
n_ctx: int | None = 0,
n_batch: int | None = 512,
model_kwargs: dict[str, Any] | None = None,
generation_kwargs: dict[str, Any] | None = None,
*,
tools: ToolsType | None = None,
streaming_callback: StreamingCallbackT | None = None,
chat_handler_name: str | None = None,
model_clip_path: str | None = None
) -> None
```
**Arguments**:
Initialize LlamaCppChatGenerator.
- `model`: The path of a quantized model for text generation, for example, "zephyr-7b-beta.Q4_0.gguf".
If the model path is also specified in the `model_kwargs`, this parameter will be ignored.
- `n_ctx`: The number of tokens in the context. When set to 0, the context will be taken from the model.
- `n_batch`: Prompt processing maximum batch size.
- `model_kwargs`: Dictionary containing keyword arguments used to initialize the LLM for text generation.
These keyword arguments provide fine-grained control over the model loading.
In case of duplication, these kwargs override `model`, `n_ctx`, and `n_batch` init parameters.
For more information on the available kwargs, see
[llama.cpp documentation](https://llama-cpp-python.readthedocs.io/en/latest/api-reference/`llama_cpp.Llama.__init__`).
- `generation_kwargs`: A dictionary containing keyword arguments to customize text generation.
For more information on the available kwargs, see
[llama.cpp documentation](https://llama-cpp-python.readthedocs.io/en/latest/api-reference/`llama_cpp.Llama.create_chat_completion`).
- `tools`: A list of Tool and/or Toolset objects, or a single Toolset for which the model can prepare calls.
Each tool should have a unique name.
- `streaming_callback`: A callback function that is called when a new token is received from the stream.
- `chat_handler_name`: Name of the chat handler for multimodal models.
Common options include: "Llava16ChatHandler", "MoondreamChatHandler", "Qwen25VLChatHandler".
For other handlers, check
[llama-cpp-python documentation](https://llama-cpp-python.readthedocs.io/en/latest/`multi`-modal-models).
- `model_clip_path`: Path to the CLIP model for vision processing (e.g., "mmproj.bin").
Required when chat_handler_name is provided for multimodal models.
**Parameters:**
<a id="haystack_integrations.components.generators.llama_cpp.chat.chat_generator.LlamaCppChatGenerator.to_dict"></a>
- **model** (<code>str</code>) – The path of a quantized model for text generation, for example, "zephyr-7b-beta.Q4_0.gguf".
If the model path is also specified in the `model_kwargs`, this parameter will be ignored.
- **n_ctx** (<code>int | None</code>) – The number of tokens in the context. When set to 0, the context will be taken from the model.
- **n_batch** (<code>int | None</code>) – Prompt processing maximum batch size.
- **model_kwargs** (<code>dict\[str, Any\] | None</code>) – Dictionary containing keyword arguments used to initialize the LLM for text generation.
These keyword arguments provide fine-grained control over the model loading.
In case of duplication, these kwargs override `model`, `n_ctx`, and `n_batch` init parameters.
For more information on the available kwargs, see
[llama.cpp documentation](https://llama-cpp-python.readthedocs.io/en/latest/api-reference/#llama_cpp.Llama.__init__).
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – A dictionary containing keyword arguments to customize text generation.
For more information on the available kwargs, see
[llama.cpp documentation](https://llama-cpp-python.readthedocs.io/en/latest/api-reference/#llama_cpp.Llama.create_chat_completion).
- **tools** (<code>ToolsType | None</code>) – A list of Tool and/or Toolset objects, or a single Toolset for which the model can prepare calls.
Each tool should have a unique name.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
- **chat_handler_name** (<code>str | None</code>) – Name of the chat handler for multimodal models.
Common options include: "Llava16ChatHandler", "MoondreamChatHandler", "Qwen25VLChatHandler".
For other handlers, check
[llama-cpp-python documentation](https://llama-cpp-python.readthedocs.io/en/latest/#multi-modal-models).
- **model_clip_path** (<code>str | None</code>) – Path to the CLIP model for vision processing (e.g., "mmproj.bin").
Required when chat_handler_name is provided for multimodal models.
#### LlamaCppChatGenerator.to\_dict
#### warm_up
```python
def to_dict() -> dict[str, Any]
warm_up() -> None
```
Load and initialize the llama.cpp model.
#### to_dict
```python
to_dict() -> dict[str, Any]
```
Serializes the component to a dictionary.
**Returns**:
**Returns:**
Dictionary with serialized data.
- <code>dict\[str, Any\]</code> – Dictionary with serialized data.
<a id="haystack_integrations.components.generators.llama_cpp.chat.chat_generator.LlamaCppChatGenerator.from_dict"></a>
#### LlamaCppChatGenerator.from\_dict
#### from_dict
```python
@classmethod
def from_dict(cls, data: dict[str, Any]) -> "LlamaCppChatGenerator"
from_dict(data: dict[str, Any]) -> LlamaCppChatGenerator
```
Deserializes the component from a dictionary.
**Arguments**:
**Parameters:**
- `data`: Dictionary to deserialize from.
- **data** (<code>dict\[str, Any\]</code>) – Dictionary to deserialize from.
**Returns**:
**Returns:**
Deserialized component.
- <code>LlamaCppChatGenerator</code> – Deserialized component.
<a id="haystack_integrations.components.generators.llama_cpp.chat.chat_generator.LlamaCppChatGenerator.run"></a>
#### LlamaCppChatGenerator.run
#### run
```python
@component.output_types(replies=list[ChatMessage])
def run(
messages: list[ChatMessage],
run(
messages: list[ChatMessage] | str,
generation_kwargs: dict[str, Any] | None = None,
*,
tools: ToolsType | None = None,
@@ -143,31 +143,29 @@ def run(
Run the text generation model on the given list of ChatMessages.
**Arguments**:
**Parameters:**
- `messages`: A list of ChatMessage instances representing the input messages.
- `generation_kwargs`: A dictionary containing keyword arguments to customize text generation.
For more information on the available kwargs, see
[llama.cpp documentation](https://llama-cpp-python.readthedocs.io/en/latest/api-reference/`llama_cpp.Llama.create_chat_completion`).
- `tools`: A list of Tool and/or Toolset objects, or a single Toolset for which the model can prepare calls.
Each tool should have a unique name. If set, it will override the `tools` parameter set during
component initialization.
- `streaming_callback`: A callback function that is called when a new token is received from the stream.
If set, it will override the `streaming_callback` parameter set during component initialization.
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of ChatMessage instances representing the input messages.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – A dictionary containing keyword arguments to customize text generation.
For more information on the available kwargs, see
[llama.cpp documentation](https://llama-cpp-python.readthedocs.io/en/latest/api-reference/#llama_cpp.Llama.create_chat_completion).
- **tools** (<code>ToolsType | None</code>) – A list of Tool and/or Toolset objects, or a single Toolset for which the model can prepare calls.
Each tool should have a unique name. If set, it will override the `tools` parameter set during
component initialization.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
If set, it will override the `streaming_callback` parameter set during component initialization.
**Returns**:
**Returns:**
A dictionary with the following keys:
- <code>dict\[str, list\[ChatMessage\]\]</code> – A dictionary with the following keys:
- `replies`: The responses from the model
<a id="haystack_integrations.components.generators.llama_cpp.chat.chat_generator.LlamaCppChatGenerator.run_async"></a>
#### LlamaCppChatGenerator.run\_async
#### run_async
```python
@component.output_types(replies=list[ChatMessage])
async def run_async(
messages: list[ChatMessage],
run_async(
messages: list[ChatMessage] | str,
generation_kwargs: dict[str, Any] | None = None,
*,
tools: ToolsType | None = None,
@@ -180,28 +178,25 @@ Async version of run. Runs the text generation model on the given list of ChatMe
Uses a thread pool to avoid blocking the event loop, since llama-cpp-python provides
only synchronous inference.
**Arguments**:
**Parameters:**
- `messages`: A list of ChatMessage instances representing the input messages.
- `generation_kwargs`: A dictionary containing keyword arguments to customize text generation.
For more information on the available kwargs, see
[llama.cpp documentation](https://llama-cpp-python.readthedocs.io/en/latest/api-reference/`llama_cpp.Llama.create_chat_completion`).
- `tools`: A list of Tool and/or Toolset objects, or a single Toolset for which the model can prepare calls.
Each tool should have a unique name. If set, it will override the `tools` parameter set during
component initialization.
- `streaming_callback`: A callback function that is called when a new token is received from the stream.
If set, it will override the `streaming_callback` parameter set during component initialization.
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of ChatMessage instances representing the input messages.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – A dictionary containing keyword arguments to customize text generation.
For more information on the available kwargs, see
[llama.cpp documentation](https://llama-cpp-python.readthedocs.io/en/latest/api-reference/#llama_cpp.Llama.create_chat_completion).
- **tools** (<code>ToolsType | None</code>) – A list of Tool and/or Toolset objects, or a single Toolset for which the model can prepare calls.
Each tool should have a unique name. If set, it will override the `tools` parameter set during
component initialization.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
If set, it will override the `streaming_callback` parameter set during component initialization.
**Returns**:
**Returns:**
A dictionary with the following keys:
- <code>dict\[str, list\[ChatMessage\]\]</code> – A dictionary with the following keys:
- `replies`: The responses from the model
<a id="haystack_integrations.components.generators.llama_cpp.generator"></a>
## Module haystack\_integrations.components.generators.llama\_cpp.generator
<a id="haystack_integrations.components.generators.llama_cpp.generator.LlamaCppGenerator"></a>
## haystack_integrations.components.generators.llama_cpp.generator
### LlamaCppGenerator
@@ -211,6 +206,7 @@ Provides an interface to generate text using LLM via llama.cpp.
It employs the quantized GGUF format, suitable for running these models on standard machines (even without GPUs).
Usage example:
```python
from haystack_integrations.components.generators.llama_cpp import LlamaCppGenerator
generator = LlamaCppGenerator(model="zephyr-7b-beta.Q4_0.gguf", n_ctx=2048, n_batch=512)
@@ -219,56 +215,62 @@ print(generator.run("Who is the best American actor?", generation_kwargs={"max_t
# {'replies': ['John Cusack'], 'meta': [{"object": "text_completion", ...}]}
```
<a id="haystack_integrations.components.generators.llama_cpp.generator.LlamaCppGenerator.__init__"></a>
#### LlamaCppGenerator.\_\_init\_\_
#### __init__
```python
def __init__(model: str,
n_ctx: int | None = 0,
n_batch: int | None = 512,
model_kwargs: dict[str, Any] | None = None,
generation_kwargs: dict[str, Any] | None = None) -> None
__init__(
model: str,
n_ctx: int | None = 0,
n_batch: int | None = 512,
model_kwargs: dict[str, Any] | None = None,
generation_kwargs: dict[str, Any] | None = None,
) -> None
```
**Arguments**:
Initialize LlamaCppGenerator.
- `model`: The path of a quantized model for text generation, for example, "zephyr-7b-beta.Q4_0.gguf".
If the model path is also specified in the `model_kwargs`, this parameter will be ignored.
- `n_ctx`: The number of tokens in the context. When set to 0, the context will be taken from the model.
- `n_batch`: Prompt processing maximum batch size.
- `model_kwargs`: Dictionary containing keyword arguments used to initialize the LLM for text generation.
These keyword arguments provide fine-grained control over the model loading.
In case of duplication, these kwargs override `model`, `n_ctx`, and `n_batch` init parameters.
For more information on the available kwargs, see
[llama.cpp documentation](https://llama-cpp-python.readthedocs.io/en/latest/api-reference/`llama_cpp.Llama.__init__`).
- `generation_kwargs`: A dictionary containing keyword arguments to customize text generation.
For more information on the available kwargs, see
[llama.cpp documentation](https://llama-cpp-python.readthedocs.io/en/latest/api-reference/`llama_cpp.Llama.create_completion`).
**Parameters:**
<a id="haystack_integrations.components.generators.llama_cpp.generator.LlamaCppGenerator.run"></a>
- **model** (<code>str</code>) – The path of a quantized model for text generation, for example, "zephyr-7b-beta.Q4_0.gguf".
If the model path is also specified in the `model_kwargs`, this parameter will be ignored.
- **n_ctx** (<code>int | None</code>) – The number of tokens in the context. When set to 0, the context will be taken from the model.
- **n_batch** (<code>int | None</code>) – Prompt processing maximum batch size.
- **model_kwargs** (<code>dict\[str, Any\] | None</code>) – Dictionary containing keyword arguments used to initialize the LLM for text generation.
These keyword arguments provide fine-grained control over the model loading.
In case of duplication, these kwargs override `model`, `n_ctx`, and `n_batch` init parameters.
For more information on the available kwargs, see
[llama.cpp documentation](https://llama-cpp-python.readthedocs.io/en/latest/api-reference/#llama_cpp.Llama.__init__).
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – A dictionary containing keyword arguments to customize text generation.
For more information on the available kwargs, see
[llama.cpp documentation](https://llama-cpp-python.readthedocs.io/en/latest/api-reference/#llama_cpp.Llama.create_completion).
#### LlamaCppGenerator.run
#### warm_up
```python
@component.output_types(replies=list[str], meta=list[dict[str, Any]])
def run(
prompt: str,
generation_kwargs: dict[str, Any] | None = None
warm_up() -> None
```
Load and initialize the llama.cpp model.
#### run
```python
run(
prompt: str, generation_kwargs: dict[str, Any] | None = None
) -> dict[str, list[str] | list[dict[str, Any]]]
```
Run the text generation model on the given prompt.
**Arguments**:
**Parameters:**
- `prompt`: the prompt to be sent to the generative model.
- `generation_kwargs`: A dictionary containing keyword arguments to customize text generation.
For more information on the available kwargs, see
[llama.cpp documentation](https://llama-cpp-python.readthedocs.io/en/latest/api-reference/`llama_cpp.Llama.create_completion`).
- **prompt** (<code>str</code>) – the prompt to be sent to the generative model.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – A dictionary containing keyword arguments to customize text generation.
For more information on the available kwargs, see
[llama.cpp documentation](https://llama-cpp-python.readthedocs.io/en/latest/api-reference/#llama_cpp.Llama.create_completion).
**Returns**:
**Returns:**
A dictionary with the following keys:
- <code>dict\[str, list\[str\] | list\[dict\[str, Any\]\]\]</code> – A dictionary with the following keys:
- `replies`: the list of replies generated by the model.
- `meta`: metadata about the request.
docs-website/reference_versioned_docs/version-2.18/integrations-api/mistral.md
+6 -4
View File
@@ -576,7 +576,7 @@ Unless specified otherwise in the `model`, this is for Mistral's `mistral-small-
```python
run(
messages: list[ChatMessage],
messages: list[ChatMessage] | str,
streaming_callback: StreamingCallbackT | None = None,
generation_kwargs: dict[str, Any] | None = None,
*,
@@ -589,7 +589,8 @@ Invokes chat completion on the Mistral API.
**Parameters:**
- **messages** (<code>list\[ChatMessage\]</code>) – A list of ChatMessage instances representing the input messages.
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of ChatMessage instances representing the input messages.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Additional keyword arguments for text generation. These parameters will
override the parameters passed during component initialization.
@@ -608,7 +609,7 @@ Invokes chat completion on the Mistral API.
```python
run_async(
messages: list[ChatMessage],
messages: list[ChatMessage] | str,
streaming_callback: StreamingCallbackT | None = None,
generation_kwargs: dict[str, Any] | None = None,
*,
@@ -621,7 +622,8 @@ Asynchronously invokes chat completion on the Mistral API.
**Parameters:**
- **messages** (<code>list\[ChatMessage\]</code>) – A list of ChatMessage instances representing the input messages.
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of ChatMessage instances representing the input messages.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
Must be a coroutine.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Additional keyword arguments for text generation.
docs-website/reference_versioned_docs/version-2.18/integrations-api/ollama.md
+6 -4
View File
@@ -319,7 +319,7 @@ Deserializes the component from a dictionary.
```python
run(
messages: list[ChatMessage],
messages: list[ChatMessage] | str,
generation_kwargs: dict[str, Any] | None = None,
tools: ToolsType | None = None,
*,
@@ -331,7 +331,8 @@ Runs an Ollama Model on a given chat history.
**Parameters:**
- **messages** (<code>list\[ChatMessage\]</code>) – A list of ChatMessage instances representing the input messages.
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of ChatMessage instances representing the input messages. If a string is provided, it is converted
to a list containing a ChatMessage with user role.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Per-call overrides for Ollama inference options.
These are merged on top of the instance-level `generation_kwargs`.
Optional arguments to pass to the Ollama generation endpoint, such as temperature, top_p, etc. See the
@@ -351,7 +352,7 @@ Runs an Ollama Model on a given chat history.
```python
run_async(
messages: list[ChatMessage],
messages: list[ChatMessage] | str,
generation_kwargs: dict[str, Any] | None = None,
tools: ToolsType | None = None,
*,
@@ -363,7 +364,8 @@ Async version of run. Runs an Ollama Model on a given chat history.
**Parameters:**
- **messages** (<code>list\[ChatMessage\]</code>) – A list of ChatMessage instances representing the input messages.
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of ChatMessage instances representing the input messages. If a string is provided, it is converted
to a list containing a ChatMessage with user role.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Per-call overrides for Ollama inference options.
These are merged on top of the instance-level `generation_kwargs`.
- **tools** (<code>ToolsType | None</code>) – A list of Tool and/or Toolset objects, or a single Toolset for which the model can prepare calls.
docs-website/reference_versioned_docs/version-2.18/integrations-api/openrouter.md
+129 -68
View File
@@ -5,15 +5,15 @@ description: "OpenRouter integration for Haystack"
slug: "/integrations-openrouter"
---
<a id="haystack_integrations.components.generators.openrouter.chat.chat_generator"></a>
## Module haystack\_integrations.components.generators.openrouter.chat.chat\_generator
<a id="haystack_integrations.components.generators.openrouter.chat.chat_generator.OpenRouterChatGenerator"></a>
## haystack_integrations.components.generators.openrouter.chat.chat_generator
### OpenRouterChatGenerator
Bases: <code>OpenAIChatGenerator</code>
Enables text generation using OpenRouter generative models.
For supported models, see [OpenRouter docs](https://openrouter.ai/models).
Users can pass any text generation parameters valid for the OpenRouter chat completion API
@@ -21,9 +21,13 @@ directly to this component using the `generation_kwargs` parameter in `__init__`
parameter in `run` method.
Key Features and Compatibility:
- **Primary Compatibility**: Designed to work seamlessly with the OpenRouter chat completion endpoint.
- **Primary Compatibility**: Compatible with the OpenRouter chat completion endpoint.
- **Streaming Support**: Supports streaming responses from the OpenRouter chat completion endpoint.
- **Customizability**: Supports all parameters supported by the OpenRouter chat completion endpoint.
- **Reasoning Support**: Extracts reasoning/thinking content from models that support it
(e.g., DeepSeek R1, Claude with extended thinking) and stores it in the `ReasoningContent`
field on `ChatMessage`. Reasoning content is only captured for non-streaming requests.
This component uses the ChatMessage format for structuring both input and output,
ensuring coherent and contextually relevant responses in chat-based text generation scenarios.
@@ -34,95 +38,152 @@ For more details on the parameters supported by the OpenRouter API, refer to the
[OpenRouter API Docs](https://openrouter.ai/docs/quickstart).
Usage example:
```python
from haystack_integrations.components.generators.openrouter import OpenRouterChatGenerator
from haystack_integrations.components.generators.openrouter import (
OpenRouterChatGenerator,
)
from haystack.dataclasses import ChatMessage
messages = [ChatMessage.from_user("What's Natural Language Processing?")]
client = OpenRouterChatGenerator()
client = OpenRouterChatGenerator(
model="deepseek/deepseek-r1",
generation_kwargs={"reasoning": {"effort": "high"}},
)
response = client.run(messages)
print(response)
>>{'replies': [ChatMessage(_content='Natural Language Processing (NLP) is a branch of artificial intelligence
>>that focuses on enabling computers to understand, interpret, and generate human language in a way that is
>>meaningful and useful.', _role=<ChatRole.ASSISTANT: 'assistant'>, _name=None,
>>_meta={'model': 'openai/gpt-5-mini', 'index': 0, 'finish_reason': 'stop',
>>'usage': {'prompt_tokens': 15, 'completion_tokens': 36, 'total_tokens': 51}})]}
print(response["replies"][0].reasoning) # Access reasoning content
print(response["replies"][0].text) # Access final answer
```
<a id="haystack_integrations.components.generators.openrouter.chat.chat_generator.OpenRouterChatGenerator.__init__"></a>
#### OpenRouterChatGenerator.\_\_init\_\_
#### __init__
```python
def __init__(*,
api_key: Secret = Secret.from_env_var("OPENROUTER_API_KEY"),
model: str = "openai/gpt-5-mini",
streaming_callback: StreamingCallbackT | None = None,
api_base_url: str | None = "https://openrouter.ai/api/v1",
generation_kwargs: dict[str, Any] | None = None,
tools: ToolsType | None = None,
timeout: float | None = None,
extra_headers: dict[str, Any] | None = None,
max_retries: int | None = None,
http_client_kwargs: dict[str, Any] | None = None)
__init__(
*,
api_key: Secret = Secret.from_env_var("OPENROUTER_API_KEY"),
model: str = "openai/gpt-5-mini",
streaming_callback: StreamingCallbackT | None = None,
api_base_url: str | None = "https://openrouter.ai/api/v1",
generation_kwargs: dict[str, Any] | None = None,
tools: ToolsType | None = None,
timeout: float | None = None,
extra_headers: dict[str, Any] | None = None,
max_retries: int | None = None,
http_client_kwargs: dict[str, Any] | None = None
) -> None
```
Creates an instance of OpenRouterChatGenerator. Unless specified otherwise,
Creates an instance of OpenRouterChatGenerator.
the default model is `openai/gpt-5-mini`.
**Parameters:**
**Arguments**:
- `api_key`: The OpenRouter API key.
- `model`: The name of the OpenRouter chat completion model to use.
- `streaming_callback`: A callback function that is called when a new token is received from the stream.
The callback function accepts StreamingChunk as an argument.
- `api_base_url`: The OpenRouter API Base url.
For more details, see OpenRouter [docs](https://openrouter.ai/docs/quickstart).
- `generation_kwargs`: Other parameters to use for the model. These parameters are all sent directly to
the OpenRouter endpoint. See [OpenRouter API docs](https://openrouter.ai/docs/quickstart) for more details.
Some of the supported parameters:
- **api_key** (<code>Secret</code>) – The OpenRouter API key.
- **model** (<code>str</code>) – The name of the OpenRouter chat completion model to use.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
The callback function accepts StreamingChunk as an argument.
- **api_base_url** (<code>str | None</code>) – The OpenRouter API Base url.
For more details, see OpenRouter [docs](https://openrouter.ai/docs/quickstart).
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Other parameters to use for the model. These parameters are all sent directly to
the OpenRouter endpoint. See [OpenRouter API docs](https://openrouter.ai/docs/quickstart) for more details.
Some of the supported parameters:
- `max_tokens`: The maximum number of tokens the output text can have.
- `temperature`: What sampling temperature to use. Higher values mean the model will take more risks.
Try 0.9 for more creative applications and 0 (argmax sampling) for ones with a well-defined answer.
Try 0.9 for more creative applications and 0 (argmax sampling) for ones with a well-defined answer.
- `top_p`: An alternative to sampling with temperature, called nucleus sampling, where the model
considers the results of the tokens with top_p probability mass. So 0.1 means only the tokens
comprising the top 10% probability mass are considered.
considers the results of the tokens with top_p probability mass. So 0.1 means only the tokens
comprising the top 10% probability mass are considered.
- `stream`: Whether to stream back partial progress. If set, tokens will be sent as data-only server-sent
events as they become available, with the stream terminated by a data: [DONE] message.
events as they become available, with the stream terminated by a data: [DONE] message.
- `safe_prompt`: Whether to inject a safety prompt before all conversations.
- `random_seed`: The seed to use for random sampling.
- `reasoning`: A dict to configure reasoning/thinking tokens for models that support it.
Example: `{"effort": "high"}` or `{"max_tokens": 2000}`.
Reasoning content is only captured for non-streaming requests.
See [OpenRouter reasoning docs](https://openrouter.ai/docs/use-cases/reasoning-tokens).
- `response_format`: A JSON schema or a Pydantic model that enforces the structure of the model's response.
If provided, the output will always be validated against this
format (unless the model returns a tool call).
For details, see the [OpenAI Structured Outputs documentation](https://platform.openai.com/docs/guides/structured-outputs).
Notes:
- This parameter accepts Pydantic models and JSON schemas for latest models starting from GPT-4o.
- For structured outputs with streaming,
the `response_format` must be a JSON schema and not a Pydantic model.
- `tools`: A list of tools or a Toolset for which the model can prepare calls. This parameter can accept either a
list of `Tool` objects or a `Toolset` instance.
- `timeout`: The timeout for the OpenRouter API call.
- `extra_headers`: Additional HTTP headers to include in requests to the OpenRouter API.
This can be useful for adding site URL or title for rankings on openrouter.ai
For more details, see OpenRouter [docs](https://openrouter.ai/docs/quickstart).
- `max_retries`: Maximum number of retries to contact OpenAI after an internal error.
If not set, it defaults to either the `OPENAI_MAX_RETRIES` environment variable, or set to 5.
- `http_client_kwargs`: A dictionary of keyword arguments to configure a custom `httpx.Client`or `httpx.AsyncClient`.
For more information, see the [HTTPX documentation](https://www.python-httpx.org/api/`client`).
- **tools** (<code>ToolsType | None</code>) – A list of tools or a Toolset for which the model can prepare calls. This parameter can accept either a
list of `Tool` objects or a `Toolset` instance.
- **timeout** (<code>float | None</code>) – The timeout for the OpenRouter API call.
- **extra_headers** (<code>dict\[str, Any\] | None</code>) – Additional HTTP headers to include in requests to the OpenRouter API.
This can be useful for adding site URL or title for rankings on openrouter.ai
For more details, see OpenRouter [docs](https://openrouter.ai/docs/quickstart).
- **max_retries** (<code>int | None</code>) – Maximum number of retries to contact OpenAI after an internal error.
If not set, it defaults to either the `OPENAI_MAX_RETRIES` environment variable, or set to 5.
- **http_client_kwargs** (<code>dict\[str, Any\] | None</code>) – A dictionary of keyword arguments to configure a custom `httpx.Client`or `httpx.AsyncClient`.
For more information, see the [HTTPX documentation](https://www.python-httpx.org/api/#client).
<a id="haystack_integrations.components.generators.openrouter.chat.chat_generator.OpenRouterChatGenerator.to_dict"></a>
#### OpenRouterChatGenerator.to\_dict
#### to_dict
```python
def to_dict() -> dict[str, Any]
to_dict() -> dict[str, Any]
```
Serialize this component to a dictionary.
**Returns**:
**Returns:**
The serialized component as a dictionary.
- <code>dict\[str, Any\]</code> – The serialized component as a dictionary.
#### run
```python
run(
messages: list[ChatMessage] | str,
streaming_callback: StreamingCallbackT | None = None,
generation_kwargs: dict[str, Any] | None = None,
*,
tools: ToolsType | None = None,
tools_strict: bool | None = None
) -> dict[str, list[ChatMessage]]
```
Invokes chat completion on the OpenRouter API.
**Parameters:**
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of ChatMessage instances representing the input messages.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Additional keyword arguments for text generation. These parameters will
override the parameters passed during component initialization.
For details on OpenRouter API parameters, see
[OpenRouter docs](https://openrouter.ai/docs/quickstart).
- **tools** (<code>ToolsType | None</code>) – A list of Tool and/or Toolset objects, or a single Toolset for which the model can prepare calls.
If set, it will override the `tools` parameter provided during initialization.
- **tools_strict** (<code>bool | None</code>) – Whether to enable strict schema adherence for tool calls.
**Returns:**
- <code>dict\[str, list\[ChatMessage\]\]</code> – A dictionary with the following key:
- `replies`: A list containing the generated responses as ChatMessage instances.
#### run_async
```python
run_async(
messages: list[ChatMessage] | str,
streaming_callback: StreamingCallbackT | None = None,
generation_kwargs: dict[str, Any] | None = None,
*,
tools: ToolsType | None = None,
tools_strict: bool | None = None
) -> dict[str, list[ChatMessage]]
```
Asynchronously invokes chat completion on the OpenRouter API.
**Parameters:**
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of ChatMessage instances representing the input messages.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
Must be a coroutine.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Additional keyword arguments for text generation.
- **tools** (<code>ToolsType | None</code>) – A list of Tool and/or Toolset objects, or a single Toolset.
- **tools_strict** (<code>bool | None</code>) – Whether to enable strict schema adherence for tool calls.
**Returns:**
- <code>dict\[str, list\[ChatMessage\]\]</code> – A dictionary with the following key:
- `replies`: A list containing the generated responses as ChatMessage instances.
docs-website/reference_versioned_docs/version-2.18/integrations-api/togetherai.md
+137 -148
View File
@@ -5,15 +5,15 @@ description: "Together AI integration for Haystack"
slug: "/integrations-togetherai"
---
<a id="haystack_integrations.components.generators.togetherai.chat.chat_generator"></a>
## Module haystack\_integrations.components.generators.togetherai.chat.chat\_generator
<a id="haystack_integrations.components.generators.togetherai.chat.chat_generator.TogetherAIChatGenerator"></a>
## haystack_integrations.components.generators.togetherai.chat.chat_generator
### TogetherAIChatGenerator
Bases: <code>OpenAIChatGenerator</code>
Enables text generation using Together AI generative models.
For supported models, see [Together AI docs](https://docs.together.ai/docs).
Users can pass any text generation parameters valid for the Together AI chat completion API
@@ -21,6 +21,7 @@ directly to this component using the `generation_kwargs` parameter in `__init__`
parameter in `run` method.
Key Features and Compatibility:
- **Primary Compatibility**: Designed to work seamlessly with the Together AI chat completion endpoint.
- **Streaming Support**: Supports streaming responses from the Together AI chat completion endpoint.
- **Customizability**: Supports all parameters supported by the Together AI chat completion endpoint.
@@ -34,6 +35,7 @@ For more details on the parameters supported by the Together AI API, refer to th
[Together AI API Docs](https://docs.together.ai/reference/chat-completions-1).
Usage example:
```python
from haystack_integrations.components.generators.togetherai import TogetherAIChatGenerator
from haystack.dataclasses import ChatMessage
@@ -51,89 +53,84 @@ print(response)
>>'usage': {'prompt_tokens': 15, 'completion_tokens': 36, 'total_tokens': 51}})]}
```
<a id="haystack_integrations.components.generators.togetherai.chat.chat_generator.TogetherAIChatGenerator.__init__"></a>
#### TogetherAIChatGenerator.\_\_init\_\_
#### __init__
```python
def __init__(*,
api_key: Secret = Secret.from_env_var("TOGETHER_API_KEY"),
model: str = "meta-llama/Llama-3.3-70B-Instruct-Turbo",
streaming_callback: StreamingCallbackT | None = None,
api_base_url: str | None = "https://api.together.xyz/v1",
generation_kwargs: dict[str, Any] | None = None,
tools: ToolsType | None = None,
timeout: float | None = None,
max_retries: int | None = None,
http_client_kwargs: dict[str, Any] | None = None)
__init__(
*,
api_key: Secret = Secret.from_env_var("TOGETHER_API_KEY"),
model: str = "meta-llama/Llama-3.3-70B-Instruct-Turbo",
streaming_callback: StreamingCallbackT | None = None,
api_base_url: str | None = "https://api.together.xyz/v1",
generation_kwargs: dict[str, Any] | None = None,
tools: ToolsType | None = None,
timeout: float | None = None,
max_retries: int | None = None,
http_client_kwargs: dict[str, Any] | None = None
) -> None
```
Creates an instance of TogetherAIChatGenerator. Unless specified otherwise,
Creates an instance of TogetherAIChatGenerator.
the default model is `meta-llama/Llama-3.3-70B-Instruct-Turbo`.
**Parameters:**
**Arguments**:
- `api_key`: The Together API key.
- `model`: The name of the Together AI chat completion model to use.
- `streaming_callback`: A callback function that is called when a new token is received from the stream.
The callback function accepts StreamingChunk as an argument.
- `api_base_url`: The Together AI API Base url.
For more details, see Together AI [docs](https://docs.together.ai/docs/openai-api-compatibility).
- `generation_kwargs`: Other parameters to use for the model. These parameters are all sent directly to
the Together AI endpoint. See [Together AI API docs](https://docs.together.ai/reference/chat-completions-1)
for more details.
Some of the supported parameters:
- **api_key** (<code>Secret</code>) – The Together API key.
- **model** (<code>str</code>) – The name of the Together AI chat completion model to use.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
The callback function accepts StreamingChunk as an argument.
- **api_base_url** (<code>str | None</code>) – The Together AI API Base url.
For more details, see Together AI [docs](https://docs.together.ai/docs/openai-api-compatibility).
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Other parameters to use for the model. These parameters are all sent directly to
the Together AI endpoint. See [Together AI API docs](https://docs.together.ai/reference/chat-completions-1)
for more details.
Some of the supported parameters:
- `max_tokens`: The maximum number of tokens the output text can have.
- `temperature`: What sampling temperature to use. Higher values mean the model will take more risks.
Try 0.9 for more creative applications and 0 (argmax sampling) for ones with a well-defined answer.
Try 0.9 for more creative applications and 0 (argmax sampling) for ones with a well-defined answer.
- `top_p`: An alternative to sampling with temperature, called nucleus sampling, where the model
considers the results of the tokens with top_p probability mass. So 0.1 means only the tokens
comprising the top 10% probability mass are considered.
considers the results of the tokens with top_p probability mass. So 0.1 means only the tokens
comprising the top 10% probability mass are considered.
- `stream`: Whether to stream back partial progress. If set, tokens will be sent as data-only server-sent
events as they become available, with the stream terminated by a data: [DONE] message.
events as they become available, with the stream terminated by a data: [DONE] message.
- `safe_prompt`: Whether to inject a safety prompt before all conversations.
- `random_seed`: The seed to use for random sampling.
- `response_format`: A JSON schema or a Pydantic model that enforces the structure of the model's response.
If provided, the output will always be validated against this
format (unless the model returns a tool call).
For details, see the [OpenAI Structured Outputs documentation](https://platform.openai.com/docs/guides/structured-outputs).
Notes:
- For structured outputs with streaming,
the `response_format` must be a JSON schema and not a Pydantic model.
- `tools`: A list of Tool and/or Toolset objects, or a single Toolset for which the model can prepare calls.
Each tool should have a unique name.
- `timeout`: The timeout for the Together AI API call.
- `max_retries`: Maximum number of retries to contact Together AI after an internal error.
If not set, it defaults to either the `OPENAI_MAX_RETRIES` environment variable, or set to 5.
- `http_client_kwargs`: A dictionary of keyword arguments to configure a custom `httpx.Client`or `httpx.AsyncClient`.
For more information, see the [HTTPX documentation](https://www.python-httpx.org/api/`client`).
If provided, the output will always be validated against this
format (unless the model returns a tool call).
For details, see the [OpenAI Structured Outputs documentation](https://platform.openai.com/docs/guides/structured-outputs).
Notes:
- For structured outputs with streaming,
the `response_format` must be a JSON schema and not a Pydantic model.
- **tools** (<code>ToolsType | None</code>) – A list of Tool and/or Toolset objects, or a single Toolset for which the model can prepare calls.
Each tool should have a unique name.
- **timeout** (<code>float | None</code>) – The timeout for the Together AI API call.
- **max_retries** (<code>int | None</code>) – Maximum number of retries to contact Together AI after an internal error.
If not set, it defaults to either the `OPENAI_MAX_RETRIES` environment variable, or set to 5.
- **http_client_kwargs** (<code>dict\[str, Any\] | None</code>) – A dictionary of keyword arguments to configure a custom `httpx.Client`or `httpx.AsyncClient`.
For more information, see the [HTTPX documentation](https://www.python-httpx.org/api/#client).
<a id="haystack_integrations.components.generators.togetherai.chat.chat_generator.TogetherAIChatGenerator.to_dict"></a>
#### TogetherAIChatGenerator.to\_dict
#### to_dict
```python
def to_dict() -> dict[str, Any]
to_dict() -> dict[str, Any]
```
Serialize this component to a dictionary.
**Returns**:
**Returns:**
The serialized component as a dictionary.
- <code>dict\[str, Any\]</code> – The serialized component as a dictionary.
<a id="haystack_integrations.components.generators.togetherai.generator"></a>
## Module haystack\_integrations.components.generators.togetherai.generator
<a id="haystack_integrations.components.generators.togetherai.generator.TogetherAIGenerator"></a>
## haystack_integrations.components.generators.togetherai.generator
### TogetherAIGenerator
Bases: <code>TogetherAIChatGenerator</code>
Provides an interface to generate text using an LLM running on Together AI.
Usage example:
```python
from haystack_integrations.components.generators.togetherai import TogetherAIGenerator
@@ -145,149 +142,141 @@ generator = TogetherAIGenerator(model="deepseek-ai/DeepSeek-R1",
print(generator.run("Who is the best Italian actor?"))
```
<a id="haystack_integrations.components.generators.togetherai.generator.TogetherAIGenerator.__init__"></a>
#### TogetherAIGenerator.\_\_init\_\_
#### __init__
```python
def __init__(api_key: Secret = Secret.from_env_var("TOGETHER_API_KEY"),
model: str = "meta-llama/Llama-3.3-70B-Instruct-Turbo",
api_base_url: str | None = "https://api.together.xyz/v1",
streaming_callback: StreamingCallbackT | None = None,
system_prompt: str | None = None,
generation_kwargs: dict[str, Any] | None = None,
timeout: float | None = None,
max_retries: int | None = None)
__init__(
api_key: Secret = Secret.from_env_var("TOGETHER_API_KEY"),
model: str = "meta-llama/Llama-3.3-70B-Instruct-Turbo",
api_base_url: str | None = "https://api.together.xyz/v1",
streaming_callback: StreamingCallbackT | None = None,
system_prompt: str | None = None,
generation_kwargs: dict[str, Any] | None = None,
timeout: float | None = None,
max_retries: int | None = None,
) -> None
```
Initialize the TogetherAIGenerator.
**Arguments**:
**Parameters:**
- `api_key`: The Together API key.
- `model`: The name of the model to use.
- `api_base_url`: The base URL of the Together AI API.
- `streaming_callback`: A callback function that is called when a new token is received from the stream.
The callback function accepts StreamingChunk as an argument.
- `system_prompt`: The system prompt to use for text generation. If not provided, the system prompt is
omitted, and the default system prompt of the model is used.
- `generation_kwargs`: Other parameters to use for the model. These parameters are all sent directly to
the Together AI endpoint. See Together AI
[documentation](https://docs.together.ai/reference/chat-completions-1) for more details.
Some of the supported parameters:
- **api_key** (<code>Secret</code>) – The Together API key.
- **model** (<code>str</code>) – The name of the model to use.
- **api_base_url** (<code>str | None</code>) – The base URL of the Together AI API.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
The callback function accepts StreamingChunk as an argument.
- **system_prompt** (<code>str | None</code>) – The system prompt to use for text generation. If not provided, the system prompt is
omitted, and the default system prompt of the model is used.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Other parameters to use for the model. These parameters are all sent directly to
the Together AI endpoint. See Together AI
[documentation](https://docs.together.ai/reference/chat-completions-1) for more details.
Some of the supported parameters:
- `max_tokens`: The maximum number of tokens the output text can have.
- `temperature`: What sampling temperature to use. Higher values mean the model will take more risks.
Try 0.9 for more creative applications and 0 (argmax sampling) for ones with a well-defined answer.
Try 0.9 for more creative applications and 0 (argmax sampling) for ones with a well-defined answer.
- `top_p`: An alternative to sampling with temperature, called nucleus sampling, where the model
considers the results of the tokens with top_p probability mass. So, 0.1 means only the tokens
comprising the top 10% probability mass are considered.
considers the results of the tokens with top_p probability mass. So, 0.1 means only the tokens
comprising the top 10% probability mass are considered.
- `n`: How many completions to generate for each prompt. For example, if the LLM gets 3 prompts and n is 2,
it will generate two completions for each of the three prompts, ending up with 6 completions in total.
it will generate two completions for each of the three prompts, ending up with 6 completions in total.
- `stop`: One or more sequences after which the LLM should stop generating tokens.
- `presence_penalty`: What penalty to apply if a token is already present at all. Bigger values mean
the model will be less likely to repeat the same token in the text.
the model will be less likely to repeat the same token in the text.
- `frequency_penalty`: What penalty to apply if a token has already been generated in the text.
Bigger values mean the model will be less likely to repeat the same token in the text.
Bigger values mean the model will be less likely to repeat the same token in the text.
- `logit_bias`: Add a logit bias to specific tokens. The keys of the dictionary are tokens, and the
values are the bias to add to that token.
- `timeout`: Timeout for together.ai Client calls, if not set it is inferred from the `OPENAI_TIMEOUT` environment
variable or set to 30.
- `max_retries`: Maximum retries to establish contact with Together AI if it returns an internal error, if not set it is
inferred from the `OPENAI_MAX_RETRIES` environment variable or set to 5.
values are the bias to add to that token.
- **timeout** (<code>float | None</code>) – Timeout for together.ai Client calls, if not set it is inferred from the `OPENAI_TIMEOUT` environment
variable or set to 30.
- **max_retries** (<code>int | None</code>) – Maximum retries to establish contact with Together AI if it returns an internal error, if not set it is
inferred from the `OPENAI_MAX_RETRIES` environment variable or set to 5.
<a id="haystack_integrations.components.generators.togetherai.generator.TogetherAIGenerator.to_dict"></a>
#### TogetherAIGenerator.to\_dict
#### to_dict
```python
def to_dict() -> dict[str, Any]
to_dict() -> dict[str, Any]
```
Serialize this component to a dictionary.
**Returns**:
**Returns:**
The serialized component as a dictionary.
- <code>dict\[str, Any\]</code> – The serialized component as a dictionary.
<a id="haystack_integrations.components.generators.togetherai.generator.TogetherAIGenerator.from_dict"></a>
#### TogetherAIGenerator.from\_dict
#### from_dict
```python
@classmethod
def from_dict(cls, data: dict[str, Any]) -> "TogetherAIGenerator"
from_dict(data: dict[str, Any]) -> TogetherAIGenerator
```
Deserialize this component from a dictionary.
**Arguments**:
**Parameters:**
- `data`: The dictionary representation of this component.
- **data** (<code>dict\[str, Any\]</code>) – The dictionary representation of this component.
**Returns**:
**Returns:**
The deserialized component instance.
- <code>TogetherAIGenerator</code> – The deserialized component instance.
<a id="haystack_integrations.components.generators.togetherai.generator.TogetherAIGenerator.run"></a>
#### TogetherAIGenerator.run
#### run
```python
@component.output_types(replies=list[str], meta=list[dict[str, Any]])
def run(*,
prompt: str,
system_prompt: str | None = None,
streaming_callback: StreamingCallbackT | None = None,
generation_kwargs: dict[str, Any] | None = None) -> dict[str, Any]
run(
*,
prompt: str,
system_prompt: str | None = None,
streaming_callback: StreamingCallbackT | None = None,
generation_kwargs: dict[str, Any] | None = None
) -> dict[str, Any]
```
Generate text completions synchronously.
**Arguments**:
**Parameters:**
- `prompt`: The input prompt string for text generation.
- `system_prompt`: An optional system prompt to provide context or instructions for the generation.
If not provided, the system prompt set in the `__init__` method will be used.
- `streaming_callback`: A callback function that is called when a new token is received from the stream.
If provided, this will override the `streaming_callback` set in the `__init__` method.
- `generation_kwargs`: Additional keyword arguments for text generation. These parameters will potentially override the parameters
passed in the `__init__` method. Supported parameters include temperature, max_new_tokens, top_p, etc.
- **prompt** (<code>str</code>) – The input prompt string for text generation.
- **system_prompt** (<code>str | None</code>) – An optional system prompt to provide context or instructions for the generation.
If not provided, the system prompt set in the `__init__` method will be used.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
If provided, this will override the `streaming_callback` set in the `__init__` method.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Additional keyword arguments for text generation. These parameters will potentially override the parameters
passed in the `__init__` method. Supported parameters include temperature, max_new_tokens, top_p, etc.
**Returns**:
**Returns:**
A dictionary with the following keys:
- <code>dict\[str, Any\]</code> – A dictionary with the following keys:
- `replies`: A list of generated text completions as strings.
- `meta`: A list of metadata dictionaries containing information about each generation,
including model name, finish reason, and token usage statistics.
including model name, finish reason, and token usage statistics.
<a id="haystack_integrations.components.generators.togetherai.generator.TogetherAIGenerator.run_async"></a>
#### TogetherAIGenerator.run\_async
#### run_async
```python
@component.output_types(replies=list[str], meta=list[dict[str, Any]])
async def run_async(
*,
prompt: str,
system_prompt: str | None = None,
streaming_callback: StreamingCallbackT | None = None,
generation_kwargs: dict[str, Any] | None = None) -> dict[str, Any]
run_async(
*,
prompt: str,
system_prompt: str | None = None,
streaming_callback: StreamingCallbackT | None = None,
generation_kwargs: dict[str, Any] | None = None
) -> dict[str, Any]
```
Generate text completions asynchronously.
**Arguments**:
**Parameters:**
- `prompt`: The input prompt string for text generation.
- `system_prompt`: An optional system prompt to provide context or instructions for the generation.
- `streaming_callback`: A callback function that is called when a new token is received from the stream.
If provided, this will override the `streaming_callback` set in the `__init__` method.
- `generation_kwargs`: Additional keyword arguments for text generation. These parameters will potentially override the parameters
passed in the `__init__` method. Supported parameters include temperature, max_new_tokens, top_p, etc.
- **prompt** (<code>str</code>) – The input prompt string for text generation.
- **system_prompt** (<code>str | None</code>) – An optional system prompt to provide context or instructions for the generation.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
If provided, this will override the `streaming_callback` set in the `__init__` method.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Additional keyword arguments for text generation. These parameters will potentially override the parameters
passed in the `__init__` method. Supported parameters include temperature, max_new_tokens, top_p, etc.
**Returns**:
**Returns:**
A dictionary with the following keys:
- <code>dict\[str, Any\]</code> – A dictionary with the following keys:
- `replies`: A list of generated text completions as strings.
- `meta`: A list of metadata dictionaries containing information about each generation,
including model name, finish reason, and token usage statistics.
including model name, finish reason, and token usage statistics.
docs-website/reference_versioned_docs/version-2.18/integrations-api/transformers.md
+1061
View File
File diff suppressed because it is too large Load Diff
docs-website/reference_versioned_docs/version-2.18/integrations-api/vllm.md
+6 -4
View File
@@ -478,7 +478,7 @@ Deserialize this component from a dictionary.
```python
run(
messages: list[ChatMessage],
messages: list[ChatMessage] | str,
streaming_callback: StreamingCallbackT | None = None,
generation_kwargs: dict[str, Any] | None = None,
*,
@@ -490,7 +490,8 @@ Run the VLLM chat generator on the given input data.
**Parameters:**
- **messages** (<code>list\[ChatMessage\]</code>) – A list of ChatMessage instances representing the input messages.
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of ChatMessage instances representing the input messages.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Additional keyword arguments for text generation. These parameters will
override the parameters passed during component initialization.
@@ -508,7 +509,7 @@ Run the VLLM chat generator on the given input data.
```python
run_async(
messages: list[ChatMessage],
messages: list[ChatMessage] | str,
streaming_callback: StreamingCallbackT | None = None,
generation_kwargs: dict[str, Any] | None = None,
*,
@@ -520,7 +521,8 @@ Run the VLLM chat generator on the given input data asynchronously.
**Parameters:**
- **messages** (<code>list\[ChatMessage\]</code>) – A list of ChatMessage instances representing the input messages.
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of ChatMessage instances representing the input messages.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
Must be a coroutine.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Additional keyword arguments for text generation. These parameters will
docs-website/reference_versioned_docs/version-2.18/integrations-api/watsonx.md
+8 -6
View File
@@ -54,7 +54,7 @@ __init__(
max_retries: int | None = None,
meta_fields_to_embed: list[str] | None = None,
embedding_separator: str = "\n"
)
) -> None
```
Creates a WatsonxDocumentEmbedder component.
@@ -166,7 +166,7 @@ __init__(
suffix: str = "",
timeout: float | None = None,
max_retries: int | None = None
)
) -> None
```
Creates an WatsonxTextEmbedder component.
@@ -411,7 +411,7 @@ Deserialize this component from a dictionary.
```python
run(
*,
messages: list[ChatMessage],
messages: list[ChatMessage] | str,
generation_kwargs: dict[str, Any] | None = None,
streaming_callback: StreamingCallbackT | None = None,
tools: ToolsType | None = None
@@ -422,7 +422,8 @@ Generate chat completions synchronously.
**Parameters:**
- **messages** (<code>list\[ChatMessage\]</code>) – A list of ChatMessage instances representing the input messages.
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of ChatMessage instances representing the input messages.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Additional keyword arguments for text generation. These parameters will potentially override the parameters
passed in the `__init__` method.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
@@ -440,7 +441,7 @@ Generate chat completions synchronously.
```python
run_async(
*,
messages: list[ChatMessage],
messages: list[ChatMessage] | str,
generation_kwargs: dict[str, Any] | None = None,
streaming_callback: StreamingCallbackT | None = None,
tools: ToolsType | None = None
@@ -451,7 +452,8 @@ Generate chat completions asynchronously.
**Parameters:**
- **messages** (<code>list\[ChatMessage\]</code>) – A list of ChatMessage instances representing the input messages.
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of ChatMessage instances representing the input messages.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Additional keyword arguments for text generation. These parameters will potentially override the parameters
passed in the `__init__` method.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
docs-website/reference_versioned_docs/version-2.19/integrations-api/amazon_bedrock.md
+6 -4
View File
@@ -1311,7 +1311,7 @@ Deserializes the component from a dictionary.
```python
run(
messages: list[ChatMessage],
messages: list[ChatMessage] | str,
streaming_callback: StreamingCallbackT | None = None,
generation_kwargs: dict[str, Any] | None = None,
tools: ToolsType | None = None,
@@ -1324,7 +1324,8 @@ Supports both standard and streaming responses depending on whether a streaming
**Parameters:**
- **messages** (<code>list\[ChatMessage\]</code>) – A list of `ChatMessage` objects forming the chat history.
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of `ChatMessage` objects forming the chat history.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – Optional callback for handling streaming outputs.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Optional dictionary of generation parameters. Some common parameters are:
- `maxTokens`: Maximum number of tokens to generate.
@@ -1347,7 +1348,7 @@ Supports both standard and streaming responses depending on whether a streaming
```python
run_async(
messages: list[ChatMessage],
messages: list[ChatMessage] | str,
streaming_callback: StreamingCallbackT | None = None,
generation_kwargs: dict[str, Any] | None = None,
tools: ToolsType | None = None,
@@ -1360,7 +1361,8 @@ Designed for use cases where non-blocking or concurrent execution is desired.
**Parameters:**
- **messages** (<code>list\[ChatMessage\]</code>) – A list of `ChatMessage` objects forming the chat history.
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of `ChatMessage` objects forming the chat history.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – Optional async-compatible callback for handling streaming outputs.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Optional dictionary of generation parameters. Some common parameters are:
- `maxTokens`: Maximum number of tokens to generate.
docs-website/reference_versioned_docs/version-2.19/integrations-api/anthropic.md
+12 -8
View File
@@ -167,7 +167,7 @@ Deserialize this component from a dictionary.
```python
run(
messages: list[ChatMessage],
messages: list[ChatMessage] | str,
streaming_callback: StreamingCallbackT | None = None,
generation_kwargs: dict[str, Any] | None = None,
tools: ToolsType | None = None,
@@ -178,7 +178,8 @@ Invokes the Anthropic API with the given messages and generation kwargs.
**Parameters:**
- **messages** (<code>list\[ChatMessage\]</code>) – A list of ChatMessage instances representing the input messages.
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of ChatMessage instances representing the input messages.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Optional arguments to pass to the Anthropic generation endpoint.
- **tools** (<code>ToolsType | None</code>) – A list of Tool and/or Toolset objects, or a single Toolset, that the model can use.
@@ -194,7 +195,7 @@ Invokes the Anthropic API with the given messages and generation kwargs.
```python
run_async(
messages: list[ChatMessage],
messages: list[ChatMessage] | str,
streaming_callback: StreamingCallbackT | None = None,
generation_kwargs: dict[str, Any] | None = None,
tools: ToolsType | None = None,
@@ -205,7 +206,8 @@ Async version of the run method. Invokes the Anthropic API with the given messag
**Parameters:**
- **messages** (<code>list\[ChatMessage\]</code>) – A list of ChatMessage instances representing the input messages.
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of ChatMessage instances representing the input messages.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Optional arguments to pass to the Anthropic generation endpoint.
- **tools** (<code>ToolsType | None</code>) – A list of Tool and/or Toolset objects, or a single Toolset, that the model can use.
@@ -360,7 +362,7 @@ This method is idempotent — it only creates clients once.
```python
run(
messages: list[ChatMessage],
messages: list[ChatMessage] | str,
streaming_callback: StreamingCallbackT | None = None,
generation_kwargs: dict[str, Any] | None = None,
tools: ToolsType | None = None,
@@ -371,7 +373,8 @@ Invokes the AnthropicFoundry API with the given messages and generation kwargs.
**Parameters:**
- **messages** (<code>list\[ChatMessage\]</code>) – A list of ChatMessage instances representing the input messages.
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of ChatMessage instances representing the input messages.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Optional arguments to pass to the Anthropic generation endpoint.
- **tools** (<code>ToolsType | None</code>) – A list of Tool and/or Toolset objects, or a single Toolset, that the model can use.
@@ -387,7 +390,7 @@ Invokes the AnthropicFoundry API with the given messages and generation kwargs.
```python
run_async(
messages: list[ChatMessage],
messages: list[ChatMessage] | str,
streaming_callback: StreamingCallbackT | None = None,
generation_kwargs: dict[str, Any] | None = None,
tools: ToolsType | None = None,
@@ -398,7 +401,8 @@ Async version of the run method. Invokes the AnthropicFoundry API with the given
**Parameters:**
- **messages** (<code>list\[ChatMessage\]</code>) – A list of ChatMessage instances representing the input messages.
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of ChatMessage instances representing the input messages.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Optional arguments to pass to the Anthropic generation endpoint.
- **tools** (<code>ToolsType | None</code>) – A list of Tool and/or Toolset objects, or a single Toolset, that the model can use.
docs-website/reference_versioned_docs/version-2.19/integrations-api/arangodb.md
+244
View File
@@ -0,0 +1,244 @@
---
title: "Arangodb"
id: integrations-arangodb
description: "Arangodb integration for Haystack"
slug: "/integrations-arangodb"
---
## haystack_integrations.components.retrievers.arangodb.embedding_retriever
### ArangoEmbeddingRetriever
Retrieves documents from an `ArangoDocumentStore` using vector similarity on embeddings.
The similarity function is configured on the `ArangoDocumentStore` (cosine, dot product, or L2).
Example usage:
```python
from haystack_integrations.document_stores.arangodb import ArangoDocumentStore
from haystack_integrations.components.retrievers.arangodb import ArangoEmbeddingRetriever
store = ArangoDocumentStore(host="http://localhost:8529", database="haystack",
username="root", collection_name="docs", embedding_dimension=768)
retriever = ArangoEmbeddingRetriever(document_store=store, top_k=5)
result = retriever.run(query_embedding=[0.1, 0.2, ...])
```
#### __init__
```python
__init__(
*,
document_store: ArangoDocumentStore,
top_k: int = 10,
filters: dict[str, Any] | None = None
) -> None
```
Creates a new ArangoEmbeddingRetriever.
**Parameters:**
- **document_store** (<code>ArangoDocumentStore</code>) – The `ArangoDocumentStore` to retrieve documents from.
- **top_k** (<code>int</code>) – Maximum number of documents to return.
- **filters** (<code>dict\[str, Any\] | None</code>) – Optional Haystack metadata filters applied at retrieval time.
#### run
```python
run(
query_embedding: list[float],
top_k: int | None = None,
filters: dict[str, Any] | None = None,
) -> dict[str, list[Document]]
```
Retrieves documents most similar to `query_embedding`.
**Parameters:**
- **query_embedding** (<code>list\[float\]</code>) – The query vector.
- **top_k** (<code>int | None</code>) – Overrides the instance-level `top_k` for this call.
- **filters** (<code>dict\[str, Any\] | None</code>) – Overrides the instance-level `filters` for this call.
**Returns:**
- <code>dict\[str, list\[Document\]\]</code> – A dictionary with `documents` — a list of `Document` objects sorted by score.
#### to_dict
```python
to_dict() -> dict[str, Any]
```
Serializes the component to a dictionary.
**Returns:**
- <code>dict\[str, Any\]</code> – Dictionary with serialized data.
#### from_dict
```python
from_dict(data: dict[str, Any]) -> ArangoEmbeddingRetriever
```
Deserializes the component from a dictionary.
**Parameters:**
- **data** (<code>dict\[str, Any\]</code>) – Dictionary to deserialize from.
**Returns:**
- <code>ArangoEmbeddingRetriever</code> – Deserialized component.
## haystack_integrations.document_stores.arangodb.document_store
### ArangoDocumentStore
A Haystack DocumentStore backed by [ArangoDB](https://www.arangodb.com/).
Documents are stored in an ArangoDB collection and support vector similarity search
via AQL vector functions (requires ArangoDB 3.12+).
Example usage:
```python
from haystack_integrations.document_stores.arangodb import ArangoDocumentStore
from haystack.utils import Secret
store = ArangoDocumentStore(
host="http://localhost:8529",
database="haystack",
username=Secret.from_env_var("ARANGO_USERNAME", strict=False),
password=Secret.from_env_var("ARANGO_PASSWORD"),
collection_name="documents",
embedding_dimension=768,
)
```
#### __init__
```python
__init__(
*,
host: str = "http://localhost:8529",
database: str = "haystack",
username: Secret = Secret.from_env_var("ARANGO_USERNAME", strict=False),
password: Secret = Secret.from_env_var("ARANGO_PASSWORD"),
collection_name: str = "haystack_documents",
embedding_dimension: int = 768,
recreate_collection: bool = False,
similarity_function: Literal["cosine", "dot_product", "l2"] = "cosine"
) -> None
```
Creates a new ArangoDocumentStore instance.
**Parameters:**
- **host** (<code>str</code>) – ArangoDB server URL, e.g. `http://localhost:8529`.
- **database** (<code>str</code>) – Name of the ArangoDB database to use. Created if it does not exist.
- **username** (<code>Secret</code>) – ArangoDB username as a `Secret`. Defaults to `ARANGO_USERNAME` env var,
falling back to `root` if the variable is not set.
- **password** (<code>Secret</code>) – ArangoDB password as a `Secret`. Defaults to `ARANGO_PASSWORD` env var.
- **collection_name** (<code>str</code>) – Name of the collection to store documents in.
- **embedding_dimension** (<code>int</code>) – Dimensionality of document embeddings.
- **recreate_collection** (<code>bool</code>) – If `True`, drop and recreate the collection on startup.
- **similarity_function** (<code>Literal['cosine', 'dot_product', 'l2']</code>) – Vector similarity function to use for embedding retrieval.
One of `"cosine"` (default), `"dot_product"`, or `"l2"`.
#### count_documents
```python
count_documents() -> int
```
Returns the number of documents in the store.
**Returns:**
- <code>int</code> – Document count.
#### filter_documents
```python
filter_documents(filters: dict[str, Any] | None = None) -> list[Document]
```
Returns documents matching the provided filters.
**Parameters:**
- **filters** (<code>dict\[str, Any\] | None</code>) – Haystack metadata filters. If `None`, all documents are returned.
**Returns:**
- <code>list\[Document\]</code> – List of matching `Document` objects.
#### write_documents
```python
write_documents(
documents: list[Document], policy: DuplicatePolicy = DuplicatePolicy.NONE
) -> int
```
Writes documents to the store.
**Parameters:**
- **documents** (<code>list\[Document\]</code>) – Documents to write.
- **policy** (<code>DuplicatePolicy</code>) – How to handle duplicates — `OVERWRITE`, `SKIP`, or `FAIL` (default).
**Returns:**
- <code>int</code> – Number of documents written.
**Raises:**
- <code>ValueError</code> – If `documents` contains non-`Document` objects.
- <code>DuplicateDocumentError</code> – If a duplicate is found and policy is `FAIL`.
#### delete_documents
```python
delete_documents(document_ids: list[str]) -> None
```
Deletes documents by their IDs.
**Parameters:**
- **document_ids** (<code>list\[str\]</code>) – List of document IDs to delete.
#### to_dict
```python
to_dict() -> dict[str, Any]
```
Serializes the component to a dictionary.
**Returns:**
- <code>dict\[str, Any\]</code> – Dictionary with serialized data.
#### from_dict
```python
from_dict(data: dict[str, Any]) -> ArangoDocumentStore
```
Deserializes the component from a dictionary.
**Parameters:**
- **data** (<code>dict\[str, Any\]</code>) – Dictionary to deserialize from.
**Returns:**
- <code>ArangoDocumentStore</code> – Deserialized component.
docs-website/reference_versioned_docs/version-2.19/integrations-api/cohere.md
+6 -4
View File
@@ -729,7 +729,7 @@ Deserializes the component from a dictionary.
```python
run(
messages: list[ChatMessage],
messages: list[ChatMessage] | str,
generation_kwargs: dict[str, Any] | None = None,
tools: ToolsType | None = None,
streaming_callback: StreamingCallbackT | None = None,
@@ -740,7 +740,8 @@ Invoke the chat endpoint based on the provided messages and generation parameter
**Parameters:**
- **messages** (<code>list\[ChatMessage\]</code>) – list of `ChatMessage` instances representing the input messages.
- **messages** (<code>list\[ChatMessage\] | str</code>) – list of `ChatMessage` instances representing the input messages.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – additional keyword arguments for chat generation. These parameters will
potentially override the parameters passed in the __init__ method.
For more details on the parameters supported by the Cohere API, refer to the
@@ -759,7 +760,7 @@ Invoke the chat endpoint based on the provided messages and generation parameter
```python
run_async(
messages: list[ChatMessage],
messages: list[ChatMessage] | str,
generation_kwargs: dict[str, Any] | None = None,
tools: ToolsType | None = None,
streaming_callback: StreamingCallbackT | None = None,
@@ -770,7 +771,8 @@ Asynchronously invoke the chat endpoint based on the provided messages and gener
**Parameters:**
- **messages** (<code>list\[ChatMessage\]</code>) – list of `ChatMessage` instances representing the input messages.
- **messages** (<code>list\[ChatMessage\] | str</code>) – list of `ChatMessage` instances representing the input messages.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – additional keyword arguments for chat generation. These parameters will
potentially override the parameters passed in the __init__ method.
For more details on the parameters supported by the Cohere API, refer to the
docs-website/reference_versioned_docs/version-2.19/integrations-api/docling_serve.md
+30 -1
View File
@@ -18,6 +18,27 @@ Enumeration of export formats supported by DoclingServe.
- `TEXT`: Extracts plain text.
- `JSON`: Returns the full Docling document as a JSON string.
### ConversionMode
Bases: <code>str</code>, <code>Enum</code>
Execution mode for DoclingServe conversions.
- `SYNC`: Uses DoclingServe's synchronous conversion endpoints.
- `ASYNC`: Uses DoclingServe's async job endpoints and polls for completion.
### DoclingServeConversionError
Bases: <code>Exception</code>
Raised when DoclingServe reports an async task or conversion failure.
### DoclingServeTimeoutError
Bases: <code>DoclingServeConversionError</code>
Raised when a DoclingServe async task exceeds job_timeout.
### DoclingServeConverter
Converts documents to Haystack Documents using a DoclingServe server.
@@ -54,7 +75,10 @@ __init__(
timeout: float = 120.0,
api_key: Secret | None = Secret.from_env_var(
"DOCLING_SERVE_API_KEY", strict=False
)
),
mode: ConversionMode | str = ConversionMode.SYNC,
poll_interval: float = 2.0,
job_timeout: float = 600.0
) -> None
```
@@ -73,6 +97,11 @@ Initializes the DoclingServeConverter.
- **api_key** (<code>Secret | None</code>) – API key for authenticating with a secured DoclingServe instance. Reads from the
`DOCLING_SERVE_API_KEY` environment variable by default. Set to `None` to disable
authentication.
- **mode** (<code>ConversionMode | str</code>) – Conversion mode. `sync` uses DoclingServe's synchronous endpoints. `async` submits
conversion jobs to DoclingServe's async endpoints and polls until completion.
- **poll_interval** (<code>float</code>) – Controls both the server-side long-poll wait (?wait= parameter) and the maximum local sleep between polls.
A higher value reduces round-trips; a lower value increases polling frequency.
- **job_timeout** (<code>float</code>) – Maximum time in seconds to wait for each async conversion job.
#### to_dict
docs-website/reference_versioned_docs/version-2.19/integrations-api/funasr.md
+157
View File
@@ -0,0 +1,157 @@
---
title: "FunASR"
id: integrations-funasr
description: "FunASR speech-to-text integration for Haystack"
slug: "/integrations-funasr"
---
## haystack_integrations.components.audio.funasr.transcriber
### FunASRTranscriber
Transcribes audio files to Documents using [FunASR](https://github.com/modelscope/FunASR).
FunASR is an open-source speech recognition toolkit from Alibaba DAMO Academy.
It supports 50+ languages, speaker diarization, and timestamp extraction, and runs
entirely locally — no API key required.
Models are downloaded from ModelScope on first use and cached in `~/.cache/modelscope`.
**Usage Example:**
```python
from haystack_integrations.components.audio.funasr import FunASRTranscriber
transcriber = FunASRTranscriber()
result = transcriber.run(sources=["speech.wav", "interview.mp3"])
documents = result["documents"]
```
**Speaker diarization and punctuation:**
```python
from haystack.utils import ComponentDevice
transcriber = FunASRTranscriber(
model="paraformer-zh",
vad_model="fsmn-vad",
punc_model="ct-punc",
spk_model="cam++",
device=ComponentDevice.from_str("cuda"),
)
```
**SenseVoice with inverse text normalisation:**
```python
transcriber = FunASRTranscriber(
model="iic/SenseVoiceSmall",
generation_kwargs={"use_itn": True, "merge_vad": True, "language": "auto"},
)
```
#### __init__
```python
__init__(
*,
model: str = "iic/SenseVoiceSmall",
vad_model: str | None = "fsmn-vad",
punc_model: str | None = "ct-punc",
spk_model: str | None = None,
device: ComponentDevice | None = None,
batch_size_s: int = 300,
store_full_path: bool = False,
generation_kwargs: dict[str, Any] | None = None
) -> None
```
Create a FunASRTranscriber component.
**Parameters:**
- **model** (<code>str</code>) – FunASR model name or local path. Defaults to `"iic/SenseVoiceSmall"`,
a multilingual model supporting 50+ languages that is 5-10x faster than Whisper.
Alternatives include `"paraformer-zh"` (Chinese) or `"paraformer-en"` (English).
Browse available models at https://modelscope.github.io/FunASR/model-selection.html.
- **vad_model** (<code>str | None</code>) – Voice activity detection model used to split long audio into segments.
Set to `None` to process the audio as a single stream.
Browse available VAD models at https://www.modelscope.cn/models.
- **punc_model** (<code>str | None</code>) – Punctuation restoration model. Set to `None` to disable punctuation.
Browse available punctuation models at https://www.modelscope.cn/models.
- **spk_model** (<code>str | None</code>) – Speaker diarization model (e.g. `"cam++"`). When set, a `"speakers"`
key is included in the Document metadata. Defaults to `None` (diarization disabled).
Browse available speaker diarization models at https://www.modelscope.cn/models.
- **device** (<code>ComponentDevice | None</code>) – The device to run inference on. If `None`, the default device is selected
automatically. Use `ComponentDevice.from_str("cuda")` for GPU inference.
- **batch_size_s** (<code>int</code>) – Batch size in seconds for VAD-segmented audio. Larger values
improve throughput at the cost of memory.
- **store_full_path** (<code>bool</code>) – If `True`, store the full audio file path in Document metadata.
If `False` (default), store only the file name.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Extra keyword arguments forwarded to `AutoModel.generate()`.
Use this for model-specific options such as `use_itn=True` or `merge_vad=True`
for SenseVoice, or `hotword="..."` for contextual recognition.
#### warm_up
```python
warm_up() -> None
```
Load the FunASR model into memory.
Models are downloaded from ModelScope on first call and cached locally.
This method is idempotent — calling it multiple times is safe.
#### to_dict
```python
to_dict() -> dict[str, Any]
```
Serialize the component to a dictionary.
**Returns:**
- <code>dict\[str, Any\]</code> – Dictionary with serialized data.
#### from_dict
```python
from_dict(data: dict[str, Any]) -> FunASRTranscriber
```
Deserialize the component from a dictionary.
**Parameters:**
- **data** (<code>dict\[str, Any\]</code>) – Dictionary to deserialize from.
**Returns:**
- <code>FunASRTranscriber</code> – Deserialized component.
#### run
```python
run(
sources: list[str | Path | ByteStream],
meta: dict[str, Any] | list[dict[str, Any]] | None = None,
) -> dict[str, list[Document]]
```
Transcribe audio sources to Documents.
**Parameters:**
- **sources** (<code>list\[str | Path | ByteStream\]</code>) – Audio file paths (`str` or `Path`) or `ByteStream` objects.
Supported formats: WAV, MP3, FLAC, OGG, M4A, AAC, and any format that
FunASR's underlying audio backend (soundfile/ffmpeg) can decode.
- **meta** (<code>dict\[str, Any\] | list\[dict\[str, Any\]\] | None</code>) – Metadata to attach to the produced Documents. Pass a single dict
to apply the same metadata to all Documents, or a list aligned with `sources`.
**Returns:**
- <code>dict\[str, list\[Document\]\]</code> – Dictionary with key `"documents"` — one `Document` per source whose
`content` holds the full transcript text.
docs-website/reference_versioned_docs/version-2.19/integrations-api/google_genai.md
+49 -7
View File
@@ -81,7 +81,9 @@ __init__(
progress_bar: bool = True,
meta_fields_to_embed: list[str] | None = None,
embedding_separator: str = "\n",
config: dict[str, Any] | None = None
config: dict[str, Any] | None = None,
timeout: float | None = None,
max_retries: int | None = None
) -> None
```
@@ -114,6 +116,8 @@ Creates an GoogleGenAIDocumentEmbedder component.
Specifying task types in `config` does not take effect for `gemini-embedding-2`.
See [Gemini documentation](https://ai.google.dev/gemini-api/docs/embeddings#task-types) for more
information.
- **timeout** (<code>float | None</code>) – The timeout in seconds for the underlying Google GenAI client network requests.
- **max_retries** (<code>int | None</code>) – The maximum number of retries for the underlying Google GenAI client network requests.
#### to_dict
@@ -260,7 +264,9 @@ __init__(
model: str = "gemini-embedding-2",
batch_size: int = 6,
progress_bar: bool = True,
config: dict[str, Any] | None = None
config: dict[str, Any] | None = None,
timeout: float | None = None,
max_retries: int | None = None
) -> None
```
@@ -293,6 +299,36 @@ Creates an GoogleGenAIMultimodalDocumentEmbedder component.
You can for example set the output dimensionality of the embedding: `{"output_dimensionality": 768}`.
See [Google API documentation](https://googleapis.github.io/python-genai/genai.html#genai.types.EmbedContentConfig)
for the available options.
- **timeout** (<code>float | None</code>) – The timeout in seconds for the underlying Google GenAI client network requests.
- **max_retries** (<code>int | None</code>) – The maximum number of retries for the underlying Google GenAI client network requests.
#### to_dict
```python
to_dict() -> dict[str, Any]
```
Serializes the component to a dictionary.
**Returns:**
- <code>dict\[str, Any\]</code> – Dictionary with serialized data.
#### from_dict
```python
from_dict(data: dict[str, Any]) -> GoogleGenAIMultimodalDocumentEmbedder
```
Deserializes the component from a dictionary.
**Parameters:**
- **data** (<code>dict\[str, Any\]</code>) – Dictionary to deserialize from.
**Returns:**
- <code>GoogleGenAIMultimodalDocumentEmbedder</code> – Deserialized component.
#### run
@@ -405,7 +441,9 @@ __init__(
model: str = "gemini-embedding-001",
prefix: str = "",
suffix: str = "",
config: dict[str, Any] | None = None
config: dict[str, Any] | None = None,
timeout: float | None = None,
max_retries: int | None = None
) -> None
```
@@ -434,6 +472,8 @@ Creates an GoogleGenAITextEmbedder component.
Specifying task types in `config` does not take effect for `gemini-embedding-2`.
See [Gemini documentation](https://ai.google.dev/gemini-api/docs/embeddings#task-types) for more
information.
- **timeout** (<code>float | None</code>) – The timeout in seconds for the underlying Google GenAI client network requests.
- **max_retries** (<code>int | None</code>) – The maximum number of retries for the underlying Google GenAI client network requests.
#### to_dict
@@ -759,7 +799,7 @@ Deserializes the component from a dictionary.
```python
run(
messages: list[ChatMessage],
messages: list[ChatMessage] | str,
generation_kwargs: dict[str, Any] | None = None,
safety_settings: list[dict[str, Any]] | None = None,
streaming_callback: StreamingCallbackT | None = None,
@@ -771,7 +811,8 @@ Run the Google Gen AI chat generator on the given input data.
**Parameters:**
- **messages** (<code>list\[ChatMessage\]</code>) – A list of ChatMessage instances representing the input messages.
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of ChatMessage instances representing the input messages.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Configuration for generation. If provided, it will override
the default config. Supports `thinking_budget` for Gemini 2.5 series thinking configuration.
- **safety_settings** (<code>list\[dict\[str, Any\]\] | None</code>) – Safety settings for content filtering. If provided, it will override the
@@ -796,7 +837,7 @@ Run the Google Gen AI chat generator on the given input data.
```python
run_async(
messages: list[ChatMessage],
messages: list[ChatMessage] | str,
generation_kwargs: dict[str, Any] | None = None,
safety_settings: list[dict[str, Any]] | None = None,
streaming_callback: StreamingCallbackT | None = None,
@@ -808,7 +849,8 @@ Async version of the run method. Run the Google Gen AI chat generator on the giv
**Parameters:**
- **messages** (<code>list\[ChatMessage\]</code>) – A list of ChatMessage instances representing the input messages.
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of ChatMessage instances representing the input messages.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Configuration for generation. If provided, it will override
the default config. Supports `thinking_budget` for Gemini 2.5 series thinking configuration.
See https://ai.google.dev/gemini-api/docs/thinking for possible values.
docs-website/reference_versioned_docs/version-2.19/integrations-api/huggingface_api.md
+796
View File
@@ -0,0 +1,796 @@
---
title: "Hugging Face API"
id: integrations-huggingface-api
description: "Hugging Face API integration for Haystack"
slug: "/integrations-huggingface-api"
---
## haystack_integrations.components.embedders.huggingface_api.document_embedder
### HuggingFaceAPIDocumentEmbedder
Embeds documents using Hugging Face APIs.
Use it with the following Hugging Face APIs:
- [Free Serverless Inference API](https://huggingface.co/inference-api)
- [Paid Inference Endpoints](https://huggingface.co/inference-endpoints)
- [Self-hosted Text Embeddings Inference](https://github.com/huggingface/text-embeddings-inference)
### Usage examples
#### With free serverless inference API
```python
from haystack_integrations.components.embedders.huggingface_api import HuggingFaceAPIDocumentEmbedder
from haystack.utils import Secret
from haystack.dataclasses import Document
doc = Document(content="I love pizza!")
doc_embedder = HuggingFaceAPIDocumentEmbedder(api_type="serverless_inference_api",
api_params={"model": "BAAI/bge-small-en-v1.5"},
token=Secret.from_token("<your-api-key>"))
result = document_embedder.run([doc])
print(result["documents"][0].embedding)
# [0.017020374536514282, -0.023255806416273117, ...]
```
#### With paid inference endpoints
```python
from haystack_integrations.components.embedders.huggingface_api import HuggingFaceAPIDocumentEmbedder
from haystack.utils import Secret
from haystack.dataclasses import Document
doc = Document(content="I love pizza!")
doc_embedder = HuggingFaceAPIDocumentEmbedder(api_type="inference_endpoints",
api_params={"url": "<your-inference-endpoint-url>"},
token=Secret.from_token("<your-api-key>"))
result = document_embedder.run([doc])
print(result["documents"][0].embedding)
# [0.017020374536514282, -0.023255806416273117, ...]
```
#### With self-hosted text embeddings inference
```python
from haystack_integrations.components.embedders.huggingface_api import HuggingFaceAPIDocumentEmbedder
from haystack.dataclasses import Document
doc = Document(content="I love pizza!")
doc_embedder = HuggingFaceAPIDocumentEmbedder(api_type="text_embeddings_inference",
api_params={"url": "http://localhost:8080"})
result = document_embedder.run([doc])
print(result["documents"][0].embedding)
# [0.017020374536514282, -0.023255806416273117, ...]
```
#### __init__
```python
__init__(
api_type: HFEmbeddingAPIType | str,
api_params: dict[str, str],
token: Secret | None = Secret.from_env_var(
["HF_API_TOKEN", "HF_TOKEN"], strict=False
),
prefix: str = "",
suffix: str = "",
truncate: bool | None = True,
normalize: bool | None = False,
batch_size: int = 32,
progress_bar: bool = True,
meta_fields_to_embed: list[str] | None = None,
embedding_separator: str = "\n",
concurrency_limit: int = 4,
) -> None
```
Creates a HuggingFaceAPIDocumentEmbedder component.
**Parameters:**
- **api_type** (<code>HFEmbeddingAPIType | str</code>) – The type of Hugging Face API to use.
- **api_params** (<code>dict\[str, str\]</code>) – A dictionary with the following keys:
- `model`: Hugging Face model ID. Required when `api_type` is `SERVERLESS_INFERENCE_API`.
- `url`: URL of the inference endpoint. Required when `api_type` is `INFERENCE_ENDPOINTS` or
`TEXT_EMBEDDINGS_INFERENCE`.
- **token** (<code>Secret | None</code>) – The Hugging Face token to use as HTTP bearer authorization.
Check your HF token in your [account settings](https://huggingface.co/settings/tokens).
- **prefix** (<code>str</code>) – A string to add at the beginning of each text.
- **suffix** (<code>str</code>) – A string to add at the end of each text.
- **truncate** (<code>bool | None</code>) – Truncates the input text to the maximum length supported by the model.
Applicable when `api_type` is `TEXT_EMBEDDINGS_INFERENCE`, or `INFERENCE_ENDPOINTS`
if the backend uses Text Embeddings Inference.
If `api_type` is `SERVERLESS_INFERENCE_API`, this parameter is ignored.
- **normalize** (<code>bool | None</code>) – Normalizes the embeddings to unit length.
Applicable when `api_type` is `TEXT_EMBEDDINGS_INFERENCE`, or `INFERENCE_ENDPOINTS`
if the backend uses Text Embeddings Inference.
If `api_type` is `SERVERLESS_INFERENCE_API`, this parameter is ignored.
- **batch_size** (<code>int</code>) – Number of documents to process at once.
- **progress_bar** (<code>bool</code>) – If `True`, shows a progress bar when running.
- **meta_fields_to_embed** (<code>list\[str\] | None</code>) – List of metadata fields to embed along with the document text.
- **embedding_separator** (<code>str</code>) – Separator used to concatenate the metadata fields to the document text.
- **concurrency_limit** (<code>int</code>) – The maximum number of requests that should be allowed to run concurrently.
This parameter is only used in the `run_async` method.
**Raises:**
- <code>ValueError</code> – If the required `model` or `url` is missing from `api_params`, the `url` is invalid,
or the `api_type` is unknown.
#### to_dict
```python
to_dict() -> dict[str, Any]
```
Serializes the component to a dictionary.
**Returns:**
- <code>dict\[str, Any\]</code> – Dictionary with serialized data.
#### from_dict
```python
from_dict(data: dict[str, Any]) -> HuggingFaceAPIDocumentEmbedder
```
Deserializes the component from a dictionary.
**Parameters:**
- **data** (<code>dict\[str, Any\]</code>) – Dictionary to deserialize from.
**Returns:**
- <code>HuggingFaceAPIDocumentEmbedder</code> – Deserialized component.
#### run
```python
run(documents: list[Document]) -> dict[str, list[Document]]
```
Embeds a list of documents.
**Parameters:**
- **documents** (<code>list\[Document\]</code>) – Documents to embed.
**Returns:**
- <code>dict\[str, list\[Document\]\]</code> – A dictionary with the following keys:
- `documents`: A list of documents with embeddings.
**Raises:**
- <code>TypeError</code> – If `documents` is not a list of Documents.
- <code>ValueError</code> – If the embeddings returned by the API have an unexpected shape.
#### run_async
```python
run_async(documents: list[Document]) -> dict[str, list[Document]]
```
Embeds a list of documents asynchronously.
**Parameters:**
- **documents** (<code>list\[Document\]</code>) – Documents to embed.
**Returns:**
- <code>dict\[str, list\[Document\]\]</code> – A dictionary with the following keys:
- `documents`: A list of documents with embeddings.
**Raises:**
- <code>TypeError</code> – If `documents` is not a list of Documents.
- <code>ValueError</code> – If the embeddings returned by the API have an unexpected shape.
## haystack_integrations.components.embedders.huggingface_api.text_embedder
### HuggingFaceAPITextEmbedder
Embeds strings using Hugging Face APIs.
Use it with the following Hugging Face APIs:
- [Free Serverless Inference API](https://huggingface.co/inference-api)
- [Paid Inference Endpoints](https://huggingface.co/inference-endpoints)
- [Self-hosted Text Embeddings Inference](https://github.com/huggingface/text-embeddings-inference)
### Usage examples
#### With free serverless inference API
```python
from haystack_integrations.components.embedders.huggingface_api import HuggingFaceAPITextEmbedder
from haystack.utils import Secret
text_embedder = HuggingFaceAPITextEmbedder(api_type="serverless_inference_api",
api_params={"model": "BAAI/bge-small-en-v1.5"},
token=Secret.from_token("<your-api-key>"))
print(text_embedder.run("I love pizza!"))
# {'embedding': [0.017020374536514282, -0.023255806416273117, ...],
```
#### With paid inference endpoints
```python
from haystack_integrations.components.embedders.huggingface_api import HuggingFaceAPITextEmbedder
from haystack.utils import Secret
text_embedder = HuggingFaceAPITextEmbedder(api_type="inference_endpoints",
api_params={"model": "BAAI/bge-small-en-v1.5"},
token=Secret.from_token("<your-api-key>"))
print(text_embedder.run("I love pizza!"))
# {'embedding': [0.017020374536514282, -0.023255806416273117, ...],
```
#### With self-hosted text embeddings inference
```python
from haystack_integrations.components.embedders.huggingface_api import HuggingFaceAPITextEmbedder
from haystack.utils import Secret
text_embedder = HuggingFaceAPITextEmbedder(api_type="text_embeddings_inference",
api_params={"url": "http://localhost:8080"})
print(text_embedder.run("I love pizza!"))
# {'embedding': [0.017020374536514282, -0.023255806416273117, ...],
```
#### __init__
```python
__init__(
api_type: HFEmbeddingAPIType | str,
api_params: dict[str, str],
token: Secret | None = Secret.from_env_var(
["HF_API_TOKEN", "HF_TOKEN"], strict=False
),
prefix: str = "",
suffix: str = "",
truncate: bool | None = True,
normalize: bool | None = False,
) -> None
```
Creates a HuggingFaceAPITextEmbedder component.
**Parameters:**
- **api_type** (<code>HFEmbeddingAPIType | str</code>) – The type of Hugging Face API to use.
- **api_params** (<code>dict\[str, str\]</code>) – A dictionary with the following keys:
- `model`: Hugging Face model ID. Required when `api_type` is `SERVERLESS_INFERENCE_API`.
- `url`: URL of the inference endpoint. Required when `api_type` is `INFERENCE_ENDPOINTS` or
`TEXT_EMBEDDINGS_INFERENCE`.
- **token** (<code>Secret | None</code>) – The Hugging Face token to use as HTTP bearer authorization.
Check your HF token in your [account settings](https://huggingface.co/settings/tokens).
- **prefix** (<code>str</code>) – A string to add at the beginning of each text.
- **suffix** (<code>str</code>) – A string to add at the end of each text.
- **truncate** (<code>bool | None</code>) – Truncates the input text to the maximum length supported by the model.
Applicable when `api_type` is `TEXT_EMBEDDINGS_INFERENCE`, or `INFERENCE_ENDPOINTS`
if the backend uses Text Embeddings Inference.
If `api_type` is `SERVERLESS_INFERENCE_API`, this parameter is ignored.
- **normalize** (<code>bool | None</code>) – Normalizes the embeddings to unit length.
Applicable when `api_type` is `TEXT_EMBEDDINGS_INFERENCE`, or `INFERENCE_ENDPOINTS`
if the backend uses Text Embeddings Inference.
If `api_type` is `SERVERLESS_INFERENCE_API`, this parameter is ignored.
**Raises:**
- <code>ValueError</code> – If the required `model` or `url` is missing from `api_params`, the `url` is invalid,
or the `api_type` is unknown.
#### to_dict
```python
to_dict() -> dict[str, Any]
```
Serializes the component to a dictionary.
**Returns:**
- <code>dict\[str, Any\]</code> – Dictionary with serialized data.
#### from_dict
```python
from_dict(data: dict[str, Any]) -> HuggingFaceAPITextEmbedder
```
Deserializes the component from a dictionary.
**Parameters:**
- **data** (<code>dict\[str, Any\]</code>) – Dictionary to deserialize from.
**Returns:**
- <code>HuggingFaceAPITextEmbedder</code> – Deserialized component.
#### run
```python
run(text: str) -> dict[str, Any]
```
Embeds a single string.
**Parameters:**
- **text** (<code>str</code>) – Text to embed.
**Returns:**
- <code>dict\[str, Any\]</code> – A dictionary with the following keys:
- `embedding`: The embedding of the input text.
**Raises:**
- <code>TypeError</code> – If `text` is not a string.
- <code>ValueError</code> – If the embedding returned by the API has an unexpected shape.
#### run_async
```python
run_async(text: str) -> dict[str, Any]
```
Embeds a single string asynchronously.
**Parameters:**
- **text** (<code>str</code>) – Text to embed.
**Returns:**
- <code>dict\[str, Any\]</code> – A dictionary with the following keys:
- `embedding`: The embedding of the input text.
**Raises:**
- <code>TypeError</code> – If `text` is not a string.
- <code>ValueError</code> – If the embedding returned by the API has an unexpected shape.
## haystack_integrations.components.generators.huggingface_api.chat.chat_generator
### HuggingFaceAPIChatGenerator
Completes chats using Hugging Face APIs.
HuggingFaceAPIChatGenerator uses the [ChatMessage](https://docs.haystack.deepset.ai/docs/chatmessage)
format for input and output. Use it to generate text with Hugging Face APIs:
- [Serverless Inference API (Inference Providers)](https://huggingface.co/docs/inference-providers)
- [Paid Inference Endpoints](https://huggingface.co/inference-endpoints)
- [Self-hosted Text Generation Inference](https://github.com/huggingface/text-generation-inference)
### Usage examples
#### With the serverless inference API (Inference Providers) - free tier available
```python
from haystack_integrations.components.generators.huggingface_api import HuggingFaceAPIChatGenerator
from haystack.dataclasses import ChatMessage
from haystack.utils import Secret
from haystack_integrations.components.common.huggingface_api.utils import HFGenerationAPIType
messages = [ChatMessage.from_system("\nYou are a helpful, respectful and honest assistant"),
ChatMessage.from_user("What's Natural Language Processing?")]
# the api_type can be expressed using the HFGenerationAPIType enum or as a string
api_type = HFGenerationAPIType.SERVERLESS_INFERENCE_API
api_type = "serverless_inference_api" # this is equivalent to the above
generator = HuggingFaceAPIChatGenerator(api_type=api_type,
api_params={"model": "Qwen/Qwen2.5-7B-Instruct",
"provider": "together"},
token=Secret.from_token("<your-api-key>"))
result = generator.run(messages)
print(result)
```
#### With the serverless inference API (Inference Providers) and text+image input
```python
from haystack_integrations.components.generators.huggingface_api import HuggingFaceAPIChatGenerator
from haystack.dataclasses import ChatMessage, ImageContent
from haystack.utils import Secret
from haystack_integrations.components.common.huggingface_api.utils import HFGenerationAPIType
# Create an image from file path, URL, or base64
image = ImageContent.from_file_path("path/to/your/image.jpg")
# Create a multimodal message with both text and image
messages = [ChatMessage.from_user(content_parts=["Describe this image in detail", image])]
generator = HuggingFaceAPIChatGenerator(
api_type=HFGenerationAPIType.SERVERLESS_INFERENCE_API,
api_params={
"model": "Qwen/Qwen2.5-VL-7B-Instruct", # Vision Language Model
"provider": "hyperbolic"
},
token=Secret.from_token("<your-api-key>")
)
result = generator.run(messages)
print(result)
```
#### With paid inference endpoints
```python
from haystack_integrations.components.generators.huggingface_api import HuggingFaceAPIChatGenerator
from haystack.dataclasses import ChatMessage
from haystack.utils import Secret
messages = [ChatMessage.from_system("\nYou are a helpful, respectful and honest assistant"),
ChatMessage.from_user("What's Natural Language Processing?")]
generator = HuggingFaceAPIChatGenerator(api_type="inference_endpoints",
api_params={"url": "<your-inference-endpoint-url>"},
token=Secret.from_token("<your-api-key>"))
result = generator.run(messages)
print(result)
```
#### With self-hosted text generation inference
```python
from haystack_integrations.components.generators.huggingface_api import HuggingFaceAPIChatGenerator
from haystack.dataclasses import ChatMessage
messages = [ChatMessage.from_system("\nYou are a helpful, respectful and honest assistant"),
ChatMessage.from_user("What's Natural Language Processing?")]
generator = HuggingFaceAPIChatGenerator(api_type="text_generation_inference",
api_params={"url": "http://localhost:8080"})
result = generator.run(messages)
print(result)
```
#### __init__
```python
__init__(
api_type: HFGenerationAPIType | str,
api_params: dict[str, str],
token: Secret | None = Secret.from_env_var(
["HF_API_TOKEN", "HF_TOKEN"], strict=False
),
generation_kwargs: dict[str, Any] | None = None,
stop_words: list[str] | None = None,
streaming_callback: StreamingCallbackT | None = None,
tools: ToolsType | None = None,
) -> None
```
Initialize the HuggingFaceAPIChatGenerator instance.
**Parameters:**
- **api_type** (<code>HFGenerationAPIType | str</code>) – The type of Hugging Face API to use. Available types:
- `text_generation_inference`: See [TGI](https://github.com/huggingface/text-generation-inference).
- `inference_endpoints`: See [Inference Endpoints](https://huggingface.co/inference-endpoints).
- `serverless_inference_api`: See
[Serverless Inference API - Inference Providers](https://huggingface.co/docs/inference-providers).
- **api_params** (<code>dict\[str, str\]</code>) – A dictionary with the following keys:
- `model`: Hugging Face model ID. Required when `api_type` is `SERVERLESS_INFERENCE_API`.
- `provider`: Provider name. Recommended when `api_type` is `SERVERLESS_INFERENCE_API`.
- `url`: URL of the inference endpoint. Required when `api_type` is `INFERENCE_ENDPOINTS` or
`TEXT_GENERATION_INFERENCE`.
- Other parameters specific to the chosen API type, such as `timeout`, `headers`, etc.
- **token** (<code>Secret | None</code>) – The Hugging Face token to use as HTTP bearer authorization.
Check your HF token in your [account settings](https://huggingface.co/settings/tokens).
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – A dictionary with keyword arguments to customize text generation.
Some examples: `max_tokens`, `temperature`, `top_p`.
For details, see [Hugging Face chat_completion documentation](https://huggingface.co/docs/huggingface_hub/package_reference/inference_client#huggingface_hub.InferenceClient.chat_completion).
- **stop_words** (<code>list\[str\] | None</code>) – An optional list of strings representing the stop words.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – An optional callable for handling streaming responses.
- **tools** (<code>ToolsType | None</code>) – A list of Tool and/or Toolset objects, or a single Toolset for which the model can prepare calls.
The chosen model should support tool/function calling, according to the model card.
Support for tools in the Hugging Face API and TGI is not yet fully refined and you may experience
unexpected behavior.
**Raises:**
- <code>ValueError</code> – If the required `model` or `url` is missing from `api_params`, the `url` is invalid, the `api_type`
is unknown, `tools` and `streaming_callback` are used together, or duplicate tool names are provided.
#### warm_up
```python
warm_up() -> None
```
Warm up the Hugging Face API chat generator.
This will warm up the tools registered in the chat generator.
This method is idempotent and will only warm up the tools once.
#### to_dict
```python
to_dict() -> dict[str, Any]
```
Serialize this component to a dictionary.
**Returns:**
- <code>dict\[str, Any\]</code> – A dictionary containing the serialized component.
#### from_dict
```python
from_dict(data: dict[str, Any]) -> HuggingFaceAPIChatGenerator
```
Deserialize this component from a dictionary.
#### run
```python
run(
messages: list[ChatMessage] | str,
generation_kwargs: dict[str, Any] | None = None,
tools: ToolsType | None = None,
streaming_callback: StreamingCallbackT | None = None,
) -> dict[str, list[ChatMessage]]
```
Invoke the text generation inference based on the provided messages and generation parameters.
**Parameters:**
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of ChatMessage objects representing the input messages. If a string is provided, it is converted
to a list containing a ChatMessage with user role.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Additional keyword arguments for text generation.
- **tools** (<code>ToolsType | None</code>) – A list of tools or a Toolset for which the model can prepare calls. If set, it will override
the `tools` parameter set during component initialization. This parameter can accept either a
list of `Tool` objects or a `Toolset` instance.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – An optional callable for handling streaming responses. If set, it will override the `streaming_callback`
parameter set during component initialization.
**Returns:**
- <code>dict\[str, list\[ChatMessage\]\]</code> – A dictionary with the following keys:
- `replies`: A list containing the generated responses as ChatMessage objects.
**Raises:**
- <code>ValueError</code> – If `tools` and a streaming callback are used together, or if duplicate tool names are provided.
#### run_async
```python
run_async(
messages: list[ChatMessage] | str,
generation_kwargs: dict[str, Any] | None = None,
tools: ToolsType | None = None,
streaming_callback: StreamingCallbackT | None = None,
) -> dict[str, list[ChatMessage]]
```
Asynchronously invokes the text generation inference based on the provided messages and generation parameters.
This is the asynchronous version of the `run` method. It has the same parameters
and return values but can be used with `await` in an async code.
**Parameters:**
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of ChatMessage objects representing the input messages. If a string is provided, it is converted
to a list containing a ChatMessage with user role.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Additional keyword arguments for text generation.
- **tools** (<code>ToolsType | None</code>) – A list of tools or a Toolset for which the model can prepare calls. If set, it will override the `tools`
parameter set during component initialization. This parameter can accept either a list of `Tool` objects
or a `Toolset` instance.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – An optional callable for handling streaming responses. If set, it will override the `streaming_callback`
parameter set during component initialization.
**Returns:**
- <code>dict\[str, list\[ChatMessage\]\]</code> – A dictionary with the following keys:
- `replies`: A list containing the generated responses as ChatMessage objects.
**Raises:**
- <code>ValueError</code> – If `tools` and a streaming callback are used together, or if duplicate tool names are provided.
## haystack_integrations.components.rankers.huggingface_api.ranker
### TruncationDirection
Bases: <code>str</code>, <code>Enum</code>
Defines the direction to truncate text when input length exceeds the model's limit.
Attributes:
LEFT: Truncate text from the left side (start of text).
RIGHT: Truncate text from the right side (end of text).
### HuggingFaceTEIRanker
Ranks documents based on their semantic similarity to the query.
It can be used with a Text Embeddings Inference (TEI) API endpoint:
- [Self-hosted Text Embeddings Inference](https://github.com/huggingface/text-embeddings-inference)
- [Hugging Face Inference Endpoints](https://huggingface.co/inference-endpoints)
Usage example:
```python
from haystack import Document
from haystack.utils import Secret
from haystack_integrations.components.rankers.huggingface_api import HuggingFaceTEIRanker
reranker = HuggingFaceTEIRanker(
url="http://localhost:8080",
top_k=5,
timeout=30,
token=Secret.from_token("my_api_token")
)
docs = [Document(content="The capital of France is Paris"), Document(content="The capital of Germany is Berlin")]
result = reranker.run(query="What is the capital of France?", documents=docs)
ranked_docs = result["documents"]
print(ranked_docs)
# >> {'documents': [Document(id=..., content: 'the capital of France is Paris', score: 0.9979767),
# >> Document(id=..., content: 'the capital of Germany is Berlin', score: 0.13982213)]}
```
#### __init__
```python
__init__(
*,
url: str,
top_k: int = 10,
raw_scores: bool = False,
timeout: int | None = 30,
max_retries: int = 3,
retry_status_codes: list[int] | None = None,
token: Secret | None = Secret.from_env_var(
["HF_API_TOKEN", "HF_TOKEN"], strict=False
)
) -> None
```
Initializes the TEI reranker component.
**Parameters:**
- **url** (<code>str</code>) – Base URL of the TEI reranking service (for example, "https://api.example.com").
- **top_k** (<code>int</code>) – Maximum number of top documents to return.
- **raw_scores** (<code>bool</code>) – If True, include raw relevance scores in the API payload.
- **timeout** (<code>int | None</code>) – Request timeout in seconds.
- **max_retries** (<code>int</code>) – Maximum number of retry attempts for failed requests.
- **retry_status_codes** (<code>list\[int\] | None</code>) – List of HTTP status codes that will trigger a retry.
When None, HTTP 408, 418, 429 and 503 will be retried (default: None).
- **token** (<code>Secret | None</code>) – The Hugging Face token to use as HTTP bearer authorization. Not always required
depending on your TEI server configuration.
Check your HF token in your [account settings](https://huggingface.co/settings/tokens).
#### to_dict
```python
to_dict() -> dict[str, Any]
```
Serializes the component to a dictionary.
**Returns:**
- <code>dict\[str, Any\]</code> – Dictionary with serialized data.
#### from_dict
```python
from_dict(data: dict[str, Any]) -> HuggingFaceTEIRanker
```
Deserializes the component from a dictionary.
**Parameters:**
- **data** (<code>dict\[str, Any\]</code>) – Dictionary to deserialize from.
**Returns:**
- <code>HuggingFaceTEIRanker</code> – Deserialized component.
#### run
```python
run(
query: str,
documents: list[Document],
top_k: int | None = None,
truncation_direction: TruncationDirection | None = None,
) -> dict[str, list[Document]]
```
Reranks the provided documents by relevance to the query using the TEI API.
Before ranking, documents are deduplicated by their id, retaining only the document with the highest score
if a score is present.
**Parameters:**
- **query** (<code>str</code>) – The user query string to guide reranking.
- **documents** (<code>list\[Document\]</code>) – List of `Document` objects to rerank.
- **top_k** (<code>int | None</code>) – Optional override for the maximum number of documents to return.
- **truncation_direction** (<code>TruncationDirection | None</code>) – If set, enables text truncation in the specified direction.
**Returns:**
- <code>dict\[str, list\[Document\]\]</code> – A dictionary with the following keys:
- `documents`: A list of reranked documents.
**Raises:**
- <code>RuntimeError</code> – - If the API request fails.
- <code>RuntimeError</code> – - If the API returns an error response.
- <code>TypeError</code> – - If the API response is not in the expected list format.
#### run_async
```python
run_async(
query: str,
documents: list[Document],
top_k: int | None = None,
truncation_direction: TruncationDirection | None = None,
) -> dict[str, list[Document]]
```
Asynchronously reranks the provided documents by relevance to the query using the TEI API.
Before ranking, documents are deduplicated by their id, retaining only the document with the highest score
if a score is present.
**Parameters:**
- **query** (<code>str</code>) – The user query string to guide reranking.
- **documents** (<code>list\[Document\]</code>) – List of `Document` objects to rerank.
- **top_k** (<code>int | None</code>) – Optional override for the maximum number of documents to return.
- **truncation_direction** (<code>TruncationDirection | None</code>) – If set, enables text truncation in the specified direction.
**Returns:**
- <code>dict\[str, list\[Document\]\]</code> – A dictionary with the following keys:
- `documents`: A list of reranked documents.
**Raises:**
- <code>httpx.RequestError</code> – - If the API request fails.
- <code>RuntimeError</code> – - If the API returns an error response.
- <code>TypeError</code> – - If the API response is not in the expected list format.
docs-website/reference_versioned_docs/version-2.19/integrations-api/litellm.md
+17 -4
View File
@@ -73,7 +73,7 @@ Create a LiteLLMChatGenerator instance.
```python
run(
messages: list[ChatMessage],
messages: list[ChatMessage] | str,
streaming_callback: StreamingCallbackT | None = None,
generation_kwargs: dict[str, Any] | None = None,
*,
@@ -85,7 +85,8 @@ Invoke chat completion via LiteLLM.
**Parameters:**
- **messages** (<code>list\[ChatMessage\]</code>) – Input messages as ChatMessage instances.
- **messages** (<code>list\[ChatMessage\] | str</code>) – Input messages as ChatMessage instances.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – Override the streaming callback for this call.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Override generation parameters for this call.
- **tools** (<code>ToolsType | None</code>) – Override tools for this call.
@@ -98,7 +99,7 @@ Invoke chat completion via LiteLLM.
```python
run_async(
messages: list[ChatMessage],
messages: list[ChatMessage] | str,
streaming_callback: StreamingCallbackT | None = None,
generation_kwargs: dict[str, Any] | None = None,
*,
@@ -106,7 +107,19 @@ run_async(
) -> dict[str, list[ChatMessage]]
```
Async version of run().
Async version of run(). Invoke chat completion via LiteLLM.
**Parameters:**
- **messages** (<code>list\[ChatMessage\] | str</code>) – Input messages as ChatMessage instances.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – Override the streaming callback for this call.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Override generation parameters for this call.
- **tools** (<code>ToolsType | None</code>) – Override tools for this call.
**Returns:**
- <code>dict\[str, list\[ChatMessage\]\]</code> – A dict with key `replies` containing ChatMessage instances.
#### to_dict
docs-website/reference_versioned_docs/version-2.19/integrations-api/llama_cpp.md
+131 -130
View File
@@ -5,11 +5,8 @@ description: "Llama.cpp integration for Haystack"
slug: "/integrations-llama-cpp"
---
<a id="haystack_integrations.components.generators.llama_cpp.chat.chat_generator"></a>
## Module haystack\_integrations.components.generators.llama\_cpp.chat.chat\_generator
<a id="haystack_integrations.components.generators.llama_cpp.chat.chat_generator.LlamaCppChatGenerator"></a>
## haystack_integrations.components.generators.llama_cpp.chat.chat_generator
### LlamaCppChatGenerator
@@ -20,6 +17,7 @@ It employs the quantized GGUF format, suitable for running these models on stand
Supports both text-only and multimodal (text + image) models like LLaVA.
Usage example:
```python
from haystack_integrations.components.generators.llama_cpp import LlamaCppChatGenerator
user_message = [ChatMessage.from_user("Who is the best American actor?")]
@@ -30,6 +28,7 @@ print(generator.run(user_message, generation_kwargs={"max_tokens": 128}))
```
Usage example with multimodal (image + text):
```python
from haystack.dataclasses import ChatMessage, ImageContent
@@ -46,94 +45,95 @@ generator = LlamaCppChatGenerator(
model_clip_path="mmproj-model-f16.gguf", # CLIP model
n_ctx=4096 # Larger context for image processing
)
generator.warm_up()
result = generator.run(messages)
print(result)
```
<a id="haystack_integrations.components.generators.llama_cpp.chat.chat_generator.LlamaCppChatGenerator.__init__"></a>
#### LlamaCppChatGenerator.\_\_init\_\_
#### __init__
```python
def __init__(model: str,
n_ctx: int | None = 0,
n_batch: int | None = 512,
model_kwargs: dict[str, Any] | None = None,
generation_kwargs: dict[str, Any] | None = None,
*,
tools: ToolsType | None = None,
streaming_callback: StreamingCallbackT | None = None,
chat_handler_name: str | None = None,
model_clip_path: str | None = None) -> None
__init__(
model: str,
n_ctx: int | None = 0,
n_batch: int | None = 512,
model_kwargs: dict[str, Any] | None = None,
generation_kwargs: dict[str, Any] | None = None,
*,
tools: ToolsType | None = None,
streaming_callback: StreamingCallbackT | None = None,
chat_handler_name: str | None = None,
model_clip_path: str | None = None
) -> None
```
**Arguments**:
Initialize LlamaCppChatGenerator.
- `model`: The path of a quantized model for text generation, for example, "zephyr-7b-beta.Q4_0.gguf".
If the model path is also specified in the `model_kwargs`, this parameter will be ignored.
- `n_ctx`: The number of tokens in the context. When set to 0, the context will be taken from the model.
- `n_batch`: Prompt processing maximum batch size.
- `model_kwargs`: Dictionary containing keyword arguments used to initialize the LLM for text generation.
These keyword arguments provide fine-grained control over the model loading.
In case of duplication, these kwargs override `model`, `n_ctx`, and `n_batch` init parameters.
For more information on the available kwargs, see
[llama.cpp documentation](https://llama-cpp-python.readthedocs.io/en/latest/api-reference/`llama_cpp.Llama.__init__`).
- `generation_kwargs`: A dictionary containing keyword arguments to customize text generation.
For more information on the available kwargs, see
[llama.cpp documentation](https://llama-cpp-python.readthedocs.io/en/latest/api-reference/`llama_cpp.Llama.create_chat_completion`).
- `tools`: A list of Tool and/or Toolset objects, or a single Toolset for which the model can prepare calls.
Each tool should have a unique name.
- `streaming_callback`: A callback function that is called when a new token is received from the stream.
- `chat_handler_name`: Name of the chat handler for multimodal models.
Common options include: "Llava16ChatHandler", "MoondreamChatHandler", "Qwen25VLChatHandler".
For other handlers, check
[llama-cpp-python documentation](https://llama-cpp-python.readthedocs.io/en/latest/`multi`-modal-models).
- `model_clip_path`: Path to the CLIP model for vision processing (e.g., "mmproj.bin").
Required when chat_handler_name is provided for multimodal models.
**Parameters:**
<a id="haystack_integrations.components.generators.llama_cpp.chat.chat_generator.LlamaCppChatGenerator.to_dict"></a>
- **model** (<code>str</code>) – The path of a quantized model for text generation, for example, "zephyr-7b-beta.Q4_0.gguf".
If the model path is also specified in the `model_kwargs`, this parameter will be ignored.
- **n_ctx** (<code>int | None</code>) – The number of tokens in the context. When set to 0, the context will be taken from the model.
- **n_batch** (<code>int | None</code>) – Prompt processing maximum batch size.
- **model_kwargs** (<code>dict\[str, Any\] | None</code>) – Dictionary containing keyword arguments used to initialize the LLM for text generation.
These keyword arguments provide fine-grained control over the model loading.
In case of duplication, these kwargs override `model`, `n_ctx`, and `n_batch` init parameters.
For more information on the available kwargs, see
[llama.cpp documentation](https://llama-cpp-python.readthedocs.io/en/latest/api-reference/#llama_cpp.Llama.__init__).
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – A dictionary containing keyword arguments to customize text generation.
For more information on the available kwargs, see
[llama.cpp documentation](https://llama-cpp-python.readthedocs.io/en/latest/api-reference/#llama_cpp.Llama.create_chat_completion).
- **tools** (<code>ToolsType | None</code>) – A list of Tool and/or Toolset objects, or a single Toolset for which the model can prepare calls.
Each tool should have a unique name.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
- **chat_handler_name** (<code>str | None</code>) – Name of the chat handler for multimodal models.
Common options include: "Llava16ChatHandler", "MoondreamChatHandler", "Qwen25VLChatHandler".
For other handlers, check
[llama-cpp-python documentation](https://llama-cpp-python.readthedocs.io/en/latest/#multi-modal-models).
- **model_clip_path** (<code>str | None</code>) – Path to the CLIP model for vision processing (e.g., "mmproj.bin").
Required when chat_handler_name is provided for multimodal models.
#### LlamaCppChatGenerator.to\_dict
#### warm_up
```python
def to_dict() -> dict[str, Any]
warm_up() -> None
```
Load and initialize the llama.cpp model.
#### to_dict
```python
to_dict() -> dict[str, Any]
```
Serializes the component to a dictionary.
**Returns**:
**Returns:**
Dictionary with serialized data.
- <code>dict\[str, Any\]</code> – Dictionary with serialized data.
<a id="haystack_integrations.components.generators.llama_cpp.chat.chat_generator.LlamaCppChatGenerator.from_dict"></a>
#### LlamaCppChatGenerator.from\_dict
#### from_dict
```python
@classmethod
def from_dict(cls, data: dict[str, Any]) -> "LlamaCppChatGenerator"
from_dict(data: dict[str, Any]) -> LlamaCppChatGenerator
```
Deserializes the component from a dictionary.
**Arguments**:
**Parameters:**
- `data`: Dictionary to deserialize from.
- **data** (<code>dict\[str, Any\]</code>) – Dictionary to deserialize from.
**Returns**:
**Returns:**
Deserialized component.
- <code>LlamaCppChatGenerator</code> – Deserialized component.
<a id="haystack_integrations.components.generators.llama_cpp.chat.chat_generator.LlamaCppChatGenerator.run"></a>
#### LlamaCppChatGenerator.run
#### run
```python
@component.output_types(replies=list[ChatMessage])
def run(
messages: list[ChatMessage],
run(
messages: list[ChatMessage] | str,
generation_kwargs: dict[str, Any] | None = None,
*,
tools: ToolsType | None = None,
@@ -143,31 +143,29 @@ def run(
Run the text generation model on the given list of ChatMessages.
**Arguments**:
**Parameters:**
- `messages`: A list of ChatMessage instances representing the input messages.
- `generation_kwargs`: A dictionary containing keyword arguments to customize text generation.
For more information on the available kwargs, see
[llama.cpp documentation](https://llama-cpp-python.readthedocs.io/en/latest/api-reference/`llama_cpp.Llama.create_chat_completion`).
- `tools`: A list of Tool and/or Toolset objects, or a single Toolset for which the model can prepare calls.
Each tool should have a unique name. If set, it will override the `tools` parameter set during
component initialization.
- `streaming_callback`: A callback function that is called when a new token is received from the stream.
If set, it will override the `streaming_callback` parameter set during component initialization.
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of ChatMessage instances representing the input messages.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – A dictionary containing keyword arguments to customize text generation.
For more information on the available kwargs, see
[llama.cpp documentation](https://llama-cpp-python.readthedocs.io/en/latest/api-reference/#llama_cpp.Llama.create_chat_completion).
- **tools** (<code>ToolsType | None</code>) – A list of Tool and/or Toolset objects, or a single Toolset for which the model can prepare calls.
Each tool should have a unique name. If set, it will override the `tools` parameter set during
component initialization.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
If set, it will override the `streaming_callback` parameter set during component initialization.
**Returns**:
**Returns:**
A dictionary with the following keys:
- <code>dict\[str, list\[ChatMessage\]\]</code> – A dictionary with the following keys:
- `replies`: The responses from the model
<a id="haystack_integrations.components.generators.llama_cpp.chat.chat_generator.LlamaCppChatGenerator.run_async"></a>
#### LlamaCppChatGenerator.run\_async
#### run_async
```python
@component.output_types(replies=list[ChatMessage])
async def run_async(
messages: list[ChatMessage],
run_async(
messages: list[ChatMessage] | str,
generation_kwargs: dict[str, Any] | None = None,
*,
tools: ToolsType | None = None,
@@ -180,28 +178,25 @@ Async version of run. Runs the text generation model on the given list of ChatMe
Uses a thread pool to avoid blocking the event loop, since llama-cpp-python provides
only synchronous inference.
**Arguments**:
**Parameters:**
- `messages`: A list of ChatMessage instances representing the input messages.
- `generation_kwargs`: A dictionary containing keyword arguments to customize text generation.
For more information on the available kwargs, see
[llama.cpp documentation](https://llama-cpp-python.readthedocs.io/en/latest/api-reference/`llama_cpp.Llama.create_chat_completion`).
- `tools`: A list of Tool and/or Toolset objects, or a single Toolset for which the model can prepare calls.
Each tool should have a unique name. If set, it will override the `tools` parameter set during
component initialization.
- `streaming_callback`: A callback function that is called when a new token is received from the stream.
If set, it will override the `streaming_callback` parameter set during component initialization.
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of ChatMessage instances representing the input messages.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – A dictionary containing keyword arguments to customize text generation.
For more information on the available kwargs, see
[llama.cpp documentation](https://llama-cpp-python.readthedocs.io/en/latest/api-reference/#llama_cpp.Llama.create_chat_completion).
- **tools** (<code>ToolsType | None</code>) – A list of Tool and/or Toolset objects, or a single Toolset for which the model can prepare calls.
Each tool should have a unique name. If set, it will override the `tools` parameter set during
component initialization.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
If set, it will override the `streaming_callback` parameter set during component initialization.
**Returns**:
**Returns:**
A dictionary with the following keys:
- <code>dict\[str, list\[ChatMessage\]\]</code> – A dictionary with the following keys:
- `replies`: The responses from the model
<a id="haystack_integrations.components.generators.llama_cpp.generator"></a>
## Module haystack\_integrations.components.generators.llama\_cpp.generator
<a id="haystack_integrations.components.generators.llama_cpp.generator.LlamaCppGenerator"></a>
## haystack_integrations.components.generators.llama_cpp.generator
### LlamaCppGenerator
@@ -211,6 +206,7 @@ Provides an interface to generate text using LLM via llama.cpp.
It employs the quantized GGUF format, suitable for running these models on standard machines (even without GPUs).
Usage example:
```python
from haystack_integrations.components.generators.llama_cpp import LlamaCppGenerator
generator = LlamaCppGenerator(model="zephyr-7b-beta.Q4_0.gguf", n_ctx=2048, n_batch=512)
@@ -219,57 +215,62 @@ print(generator.run("Who is the best American actor?", generation_kwargs={"max_t
# {'replies': ['John Cusack'], 'meta': [{"object": "text_completion", ...}]}
```
<a id="haystack_integrations.components.generators.llama_cpp.generator.LlamaCppGenerator.__init__"></a>
#### LlamaCppGenerator.\_\_init\_\_
#### __init__
```python
def __init__(model: str,
n_ctx: int | None = 0,
n_batch: int | None = 512,
model_kwargs: dict[str, Any] | None = None,
generation_kwargs: dict[str, Any] | None = None) -> None
__init__(
model: str,
n_ctx: int | None = 0,
n_batch: int | None = 512,
model_kwargs: dict[str, Any] | None = None,
generation_kwargs: dict[str, Any] | None = None,
) -> None
```
**Arguments**:
Initialize LlamaCppGenerator.
- `model`: The path of a quantized model for text generation, for example, "zephyr-7b-beta.Q4_0.gguf".
If the model path is also specified in the `model_kwargs`, this parameter will be ignored.
- `n_ctx`: The number of tokens in the context. When set to 0, the context will be taken from the model.
- `n_batch`: Prompt processing maximum batch size.
- `model_kwargs`: Dictionary containing keyword arguments used to initialize the LLM for text generation.
These keyword arguments provide fine-grained control over the model loading.
In case of duplication, these kwargs override `model`, `n_ctx`, and `n_batch` init parameters.
For more information on the available kwargs, see
[llama.cpp documentation](https://llama-cpp-python.readthedocs.io/en/latest/api-reference/`llama_cpp.Llama.__init__`).
- `generation_kwargs`: A dictionary containing keyword arguments to customize text generation.
For more information on the available kwargs, see
[llama.cpp documentation](https://llama-cpp-python.readthedocs.io/en/latest/api-reference/`llama_cpp.Llama.create_completion`).
**Parameters:**
<a id="haystack_integrations.components.generators.llama_cpp.generator.LlamaCppGenerator.run"></a>
- **model** (<code>str</code>) – The path of a quantized model for text generation, for example, "zephyr-7b-beta.Q4_0.gguf".
If the model path is also specified in the `model_kwargs`, this parameter will be ignored.
- **n_ctx** (<code>int | None</code>) – The number of tokens in the context. When set to 0, the context will be taken from the model.
- **n_batch** (<code>int | None</code>) – Prompt processing maximum batch size.
- **model_kwargs** (<code>dict\[str, Any\] | None</code>) – Dictionary containing keyword arguments used to initialize the LLM for text generation.
These keyword arguments provide fine-grained control over the model loading.
In case of duplication, these kwargs override `model`, `n_ctx`, and `n_batch` init parameters.
For more information on the available kwargs, see
[llama.cpp documentation](https://llama-cpp-python.readthedocs.io/en/latest/api-reference/#llama_cpp.Llama.__init__).
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – A dictionary containing keyword arguments to customize text generation.
For more information on the available kwargs, see
[llama.cpp documentation](https://llama-cpp-python.readthedocs.io/en/latest/api-reference/#llama_cpp.Llama.create_completion).
#### LlamaCppGenerator.run
#### warm_up
```python
@component.output_types(replies=list[str], meta=list[dict[str, Any]])
def run(
prompt: str,
generation_kwargs: dict[str, Any] | None = None
warm_up() -> None
```
Load and initialize the llama.cpp model.
#### run
```python
run(
prompt: str, generation_kwargs: dict[str, Any] | None = None
) -> dict[str, list[str] | list[dict[str, Any]]]
```
Run the text generation model on the given prompt.
**Arguments**:
**Parameters:**
- `prompt`: the prompt to be sent to the generative model.
- `generation_kwargs`: A dictionary containing keyword arguments to customize text generation.
For more information on the available kwargs, see
[llama.cpp documentation](https://llama-cpp-python.readthedocs.io/en/latest/api-reference/`llama_cpp.Llama.create_completion`).
- **prompt** (<code>str</code>) – the prompt to be sent to the generative model.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – A dictionary containing keyword arguments to customize text generation.
For more information on the available kwargs, see
[llama.cpp documentation](https://llama-cpp-python.readthedocs.io/en/latest/api-reference/#llama_cpp.Llama.create_completion).
**Returns**:
**Returns:**
A dictionary with the following keys:
- <code>dict\[str, list\[str\] | list\[dict\[str, Any\]\]\]</code> – A dictionary with the following keys:
- `replies`: the list of replies generated by the model.
- `meta`: metadata about the request.
docs-website/reference_versioned_docs/version-2.19/integrations-api/mistral.md
+6 -4
View File
@@ -576,7 +576,7 @@ Unless specified otherwise in the `model`, this is for Mistral's `mistral-small-
```python
run(
messages: list[ChatMessage],
messages: list[ChatMessage] | str,
streaming_callback: StreamingCallbackT | None = None,
generation_kwargs: dict[str, Any] | None = None,
*,
@@ -589,7 +589,8 @@ Invokes chat completion on the Mistral API.
**Parameters:**
- **messages** (<code>list\[ChatMessage\]</code>) – A list of ChatMessage instances representing the input messages.
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of ChatMessage instances representing the input messages.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Additional keyword arguments for text generation. These parameters will
override the parameters passed during component initialization.
@@ -608,7 +609,7 @@ Invokes chat completion on the Mistral API.
```python
run_async(
messages: list[ChatMessage],
messages: list[ChatMessage] | str,
streaming_callback: StreamingCallbackT | None = None,
generation_kwargs: dict[str, Any] | None = None,
*,
@@ -621,7 +622,8 @@ Asynchronously invokes chat completion on the Mistral API.
**Parameters:**
- **messages** (<code>list\[ChatMessage\]</code>) – A list of ChatMessage instances representing the input messages.
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of ChatMessage instances representing the input messages.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
Must be a coroutine.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Additional keyword arguments for text generation.
docs-website/reference_versioned_docs/version-2.19/integrations-api/ollama.md
+6 -4
View File
@@ -319,7 +319,7 @@ Deserializes the component from a dictionary.
```python
run(
messages: list[ChatMessage],
messages: list[ChatMessage] | str,
generation_kwargs: dict[str, Any] | None = None,
tools: ToolsType | None = None,
*,
@@ -331,7 +331,8 @@ Runs an Ollama Model on a given chat history.
**Parameters:**
- **messages** (<code>list\[ChatMessage\]</code>) – A list of ChatMessage instances representing the input messages.
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of ChatMessage instances representing the input messages. If a string is provided, it is converted
to a list containing a ChatMessage with user role.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Per-call overrides for Ollama inference options.
These are merged on top of the instance-level `generation_kwargs`.
Optional arguments to pass to the Ollama generation endpoint, such as temperature, top_p, etc. See the
@@ -351,7 +352,7 @@ Runs an Ollama Model on a given chat history.
```python
run_async(
messages: list[ChatMessage],
messages: list[ChatMessage] | str,
generation_kwargs: dict[str, Any] | None = None,
tools: ToolsType | None = None,
*,
@@ -363,7 +364,8 @@ Async version of run. Runs an Ollama Model on a given chat history.
**Parameters:**
- **messages** (<code>list\[ChatMessage\]</code>) – A list of ChatMessage instances representing the input messages.
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of ChatMessage instances representing the input messages. If a string is provided, it is converted
to a list containing a ChatMessage with user role.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Per-call overrides for Ollama inference options.
These are merged on top of the instance-level `generation_kwargs`.
- **tools** (<code>ToolsType | None</code>) – A list of Tool and/or Toolset objects, or a single Toolset for which the model can prepare calls.
docs-website/reference_versioned_docs/version-2.19/integrations-api/openrouter.md
+128 -68
View File
@@ -5,15 +5,15 @@ description: "OpenRouter integration for Haystack"
slug: "/integrations-openrouter"
---
<a id="haystack_integrations.components.generators.openrouter.chat.chat_generator"></a>
## Module haystack\_integrations.components.generators.openrouter.chat.chat\_generator
<a id="haystack_integrations.components.generators.openrouter.chat.chat_generator.OpenRouterChatGenerator"></a>
## haystack_integrations.components.generators.openrouter.chat.chat_generator
### OpenRouterChatGenerator
Bases: <code>OpenAIChatGenerator</code>
Enables text generation using OpenRouter generative models.
For supported models, see [OpenRouter docs](https://openrouter.ai/models).
Users can pass any text generation parameters valid for the OpenRouter chat completion API
@@ -21,9 +21,13 @@ directly to this component using the `generation_kwargs` parameter in `__init__`
parameter in `run` method.
Key Features and Compatibility:
- **Primary Compatibility**: Designed to work seamlessly with the OpenRouter chat completion endpoint.
- **Primary Compatibility**: Compatible with the OpenRouter chat completion endpoint.
- **Streaming Support**: Supports streaming responses from the OpenRouter chat completion endpoint.
- **Customizability**: Supports all parameters supported by the OpenRouter chat completion endpoint.
- **Reasoning Support**: Extracts reasoning/thinking content from models that support it
(e.g., DeepSeek R1, Claude with extended thinking) and stores it in the `ReasoningContent`
field on `ChatMessage`. Reasoning content is only captured for non-streaming requests.
This component uses the ChatMessage format for structuring both input and output,
ensuring coherent and contextually relevant responses in chat-based text generation scenarios.
@@ -34,96 +38,152 @@ For more details on the parameters supported by the OpenRouter API, refer to the
[OpenRouter API Docs](https://openrouter.ai/docs/quickstart).
Usage example:
```python
from haystack_integrations.components.generators.openrouter import OpenRouterChatGenerator
from haystack_integrations.components.generators.openrouter import (
OpenRouterChatGenerator,
)
from haystack.dataclasses import ChatMessage
messages = [ChatMessage.from_user("What's Natural Language Processing?")]
client = OpenRouterChatGenerator()
client = OpenRouterChatGenerator(
model="deepseek/deepseek-r1",
generation_kwargs={"reasoning": {"effort": "high"}},
)
response = client.run(messages)
print(response)
>>{'replies': [ChatMessage(_content='Natural Language Processing (NLP) is a branch of artificial intelligence
>>that focuses on enabling computers to understand, interpret, and generate human language in a way that is
>>meaningful and useful.', _role=<ChatRole.ASSISTANT: 'assistant'>, _name=None,
>>_meta={'model': 'openai/gpt-5-mini', 'index': 0, 'finish_reason': 'stop',
>>'usage': {'prompt_tokens': 15, 'completion_tokens': 36, 'total_tokens': 51}})]}
print(response["replies"][0].reasoning) # Access reasoning content
print(response["replies"][0].text) # Access final answer
```
<a id="haystack_integrations.components.generators.openrouter.chat.chat_generator.OpenRouterChatGenerator.__init__"></a>
#### OpenRouterChatGenerator.\_\_init\_\_
#### __init__
```python
def __init__(*,
api_key: Secret = Secret.from_env_var("OPENROUTER_API_KEY"),
model: str = "openai/gpt-5-mini",
streaming_callback: StreamingCallbackT | None = None,
api_base_url: str | None = "https://openrouter.ai/api/v1",
generation_kwargs: dict[str, Any] | None = None,
tools: ToolsType | None = None,
timeout: float | None = None,
extra_headers: dict[str, Any] | None = None,
max_retries: int | None = None,
http_client_kwargs: dict[str, Any] | None = None)
__init__(
*,
api_key: Secret = Secret.from_env_var("OPENROUTER_API_KEY"),
model: str = "openai/gpt-5-mini",
streaming_callback: StreamingCallbackT | None = None,
api_base_url: str | None = "https://openrouter.ai/api/v1",
generation_kwargs: dict[str, Any] | None = None,
tools: ToolsType | None = None,
timeout: float | None = None,
extra_headers: dict[str, Any] | None = None,
max_retries: int | None = None,
http_client_kwargs: dict[str, Any] | None = None
) -> None
```
Creates an instance of OpenRouterChatGenerator. Unless specified otherwise,
Creates an instance of OpenRouterChatGenerator.
the default model is `openai/gpt-5-mini`.
**Parameters:**
**Arguments**:
- `api_key`: The OpenRouter API key.
- `model`: The name of the OpenRouter chat completion model to use.
- `streaming_callback`: A callback function that is called when a new token is received from the stream.
The callback function accepts StreamingChunk as an argument.
- `api_base_url`: The OpenRouter API Base url.
For more details, see OpenRouter [docs](https://openrouter.ai/docs/quickstart).
- `generation_kwargs`: Other parameters to use for the model. These parameters are all sent directly to
the OpenRouter endpoint. See [OpenRouter API docs](https://openrouter.ai/docs/quickstart) for more details.
Some of the supported parameters:
- **api_key** (<code>Secret</code>) – The OpenRouter API key.
- **model** (<code>str</code>) – The name of the OpenRouter chat completion model to use.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
The callback function accepts StreamingChunk as an argument.
- **api_base_url** (<code>str | None</code>) – The OpenRouter API Base url.
For more details, see OpenRouter [docs](https://openrouter.ai/docs/quickstart).
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Other parameters to use for the model. These parameters are all sent directly to
the OpenRouter endpoint. See [OpenRouter API docs](https://openrouter.ai/docs/quickstart) for more details.
Some of the supported parameters:
- `max_tokens`: The maximum number of tokens the output text can have.
- `temperature`: What sampling temperature to use. Higher values mean the model will take more risks.
Try 0.9 for more creative applications and 0 (argmax sampling) for ones with a well-defined answer.
Try 0.9 for more creative applications and 0 (argmax sampling) for ones with a well-defined answer.
- `top_p`: An alternative to sampling with temperature, called nucleus sampling, where the model
considers the results of the tokens with top_p probability mass. So 0.1 means only the tokens
comprising the top 10% probability mass are considered.
considers the results of the tokens with top_p probability mass. So 0.1 means only the tokens
comprising the top 10% probability mass are considered.
- `stream`: Whether to stream back partial progress. If set, tokens will be sent as data-only server-sent
events as they become available, with the stream terminated by a data: [DONE] message.
events as they become available, with the stream terminated by a data: [DONE] message.
- `safe_prompt`: Whether to inject a safety prompt before all conversations.
- `random_seed`: The seed to use for random sampling.
- `reasoning`: A dict to configure reasoning/thinking tokens for models that support it.
Example: `{"effort": "high"}` or `{"max_tokens": 2000}`.
Reasoning content is only captured for non-streaming requests.
See [OpenRouter reasoning docs](https://openrouter.ai/docs/use-cases/reasoning-tokens).
- `response_format`: A JSON schema or a Pydantic model that enforces the structure of the model's response.
If provided, the output will always be validated against this
format (unless the model returns a tool call).
For details, see the [OpenAI Structured Outputs documentation](https://platform.openai.com/docs/guides/structured-outputs).
Notes:
- This parameter accepts Pydantic models and JSON schemas for latest models starting from GPT-4o.
- For structured outputs with streaming,
the `response_format` must be a JSON schema and not a Pydantic model.
- `tools`: A list of tools or a Toolset for which the model can prepare calls. This parameter can accept either a
list of `Tool` objects or a `Toolset` instance.
- `timeout`: The timeout for the OpenRouter API call.
- `extra_headers`: Additional HTTP headers to include in requests to the OpenRouter API.
This can be useful for adding site URL or title for rankings on openrouter.ai
For more details, see OpenRouter [docs](https://openrouter.ai/docs/quickstart).
- `max_retries`: Maximum number of retries to contact OpenAI after an internal error.
If not set, it defaults to either the `OPENAI_MAX_RETRIES` environment variable, or set to 5.
- `http_client_kwargs`: A dictionary of keyword arguments to configure a custom `httpx.Client`or `httpx.AsyncClient`.
For more information, see the [HTTPX documentation](https://www.python-httpx.org/api/`client`).
- **tools** (<code>ToolsType | None</code>) – A list of tools or a Toolset for which the model can prepare calls. This parameter can accept either a
list of `Tool` objects or a `Toolset` instance.
- **timeout** (<code>float | None</code>) – The timeout for the OpenRouter API call.
- **extra_headers** (<code>dict\[str, Any\] | None</code>) – Additional HTTP headers to include in requests to the OpenRouter API.
This can be useful for adding site URL or title for rankings on openrouter.ai
For more details, see OpenRouter [docs](https://openrouter.ai/docs/quickstart).
- **max_retries** (<code>int | None</code>) – Maximum number of retries to contact OpenAI after an internal error.
If not set, it defaults to either the `OPENAI_MAX_RETRIES` environment variable, or set to 5.
- **http_client_kwargs** (<code>dict\[str, Any\] | None</code>) – A dictionary of keyword arguments to configure a custom `httpx.Client`or `httpx.AsyncClient`.
For more information, see the [HTTPX documentation](https://www.python-httpx.org/api/#client).
<a id="haystack_integrations.components.generators.openrouter.chat.chat_generator.OpenRouterChatGenerator.to_dict"></a>
#### OpenRouterChatGenerator.to\_dict
#### to_dict
```python
def to_dict() -> dict[str, Any]
to_dict() -> dict[str, Any]
```
Serialize this component to a dictionary.
**Returns**:
**Returns:**
The serialized component as a dictionary.
- <code>dict\[str, Any\]</code> – The serialized component as a dictionary.
#### run
```python
run(
messages: list[ChatMessage] | str,
streaming_callback: StreamingCallbackT | None = None,
generation_kwargs: dict[str, Any] | None = None,
*,
tools: ToolsType | None = None,
tools_strict: bool | None = None
) -> dict[str, list[ChatMessage]]
```
Invokes chat completion on the OpenRouter API.
**Parameters:**
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of ChatMessage instances representing the input messages.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Additional keyword arguments for text generation. These parameters will
override the parameters passed during component initialization.
For details on OpenRouter API parameters, see
[OpenRouter docs](https://openrouter.ai/docs/quickstart).
- **tools** (<code>ToolsType | None</code>) – A list of Tool and/or Toolset objects, or a single Toolset for which the model can prepare calls.
If set, it will override the `tools` parameter provided during initialization.
- **tools_strict** (<code>bool | None</code>) – Whether to enable strict schema adherence for tool calls.
**Returns:**
- <code>dict\[str, list\[ChatMessage\]\]</code> – A dictionary with the following key:
- `replies`: A list containing the generated responses as ChatMessage instances.
#### run_async
```python
run_async(
messages: list[ChatMessage] | str,
streaming_callback: StreamingCallbackT | None = None,
generation_kwargs: dict[str, Any] | None = None,
*,
tools: ToolsType | None = None,
tools_strict: bool | None = None
) -> dict[str, list[ChatMessage]]
```
Asynchronously invokes chat completion on the OpenRouter API.
**Parameters:**
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of ChatMessage instances representing the input messages.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
Must be a coroutine.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Additional keyword arguments for text generation.
- **tools** (<code>ToolsType | None</code>) – A list of Tool and/or Toolset objects, or a single Toolset.
- **tools_strict** (<code>bool | None</code>) – Whether to enable strict schema adherence for tool calls.
**Returns:**
- <code>dict\[str, list\[ChatMessage\]\]</code> – A dictionary with the following key:
- `replies`: A list containing the generated responses as ChatMessage instances.
docs-website/reference_versioned_docs/version-2.19/integrations-api/togetherai.md
+137 -149
View File
@@ -5,15 +5,15 @@ description: "Together AI integration for Haystack"
slug: "/integrations-togetherai"
---
<a id="haystack_integrations.components.generators.togetherai.chat.chat_generator"></a>
## Module haystack\_integrations.components.generators.togetherai.chat.chat\_generator
<a id="haystack_integrations.components.generators.togetherai.chat.chat_generator.TogetherAIChatGenerator"></a>
## haystack_integrations.components.generators.togetherai.chat.chat_generator
### TogetherAIChatGenerator
Bases: <code>OpenAIChatGenerator</code>
Enables text generation using Together AI generative models.
For supported models, see [Together AI docs](https://docs.together.ai/docs).
Users can pass any text generation parameters valid for the Together AI chat completion API
@@ -21,6 +21,7 @@ directly to this component using the `generation_kwargs` parameter in `__init__`
parameter in `run` method.
Key Features and Compatibility:
- **Primary Compatibility**: Designed to work seamlessly with the Together AI chat completion endpoint.
- **Streaming Support**: Supports streaming responses from the Together AI chat completion endpoint.
- **Customizability**: Supports all parameters supported by the Together AI chat completion endpoint.
@@ -34,6 +35,7 @@ For more details on the parameters supported by the Together AI API, refer to th
[Together AI API Docs](https://docs.together.ai/reference/chat-completions-1).
Usage example:
```python
from haystack_integrations.components.generators.togetherai import TogetherAIChatGenerator
from haystack.dataclasses import ChatMessage
@@ -51,89 +53,84 @@ print(response)
>>'usage': {'prompt_tokens': 15, 'completion_tokens': 36, 'total_tokens': 51}})]}
```
<a id="haystack_integrations.components.generators.togetherai.chat.chat_generator.TogetherAIChatGenerator.__init__"></a>
#### TogetherAIChatGenerator.\_\_init\_\_
#### __init__
```python
def __init__(*,
api_key: Secret = Secret.from_env_var("TOGETHER_API_KEY"),
model: str = "meta-llama/Llama-3.3-70B-Instruct-Turbo",
streaming_callback: StreamingCallbackT | None = None,
api_base_url: str | None = "https://api.together.xyz/v1",
generation_kwargs: dict[str, Any] | None = None,
tools: ToolsType | None = None,
timeout: float | None = None,
max_retries: int | None = None,
http_client_kwargs: dict[str, Any] | None = None)
__init__(
*,
api_key: Secret = Secret.from_env_var("TOGETHER_API_KEY"),
model: str = "meta-llama/Llama-3.3-70B-Instruct-Turbo",
streaming_callback: StreamingCallbackT | None = None,
api_base_url: str | None = "https://api.together.xyz/v1",
generation_kwargs: dict[str, Any] | None = None,
tools: ToolsType | None = None,
timeout: float | None = None,
max_retries: int | None = None,
http_client_kwargs: dict[str, Any] | None = None
) -> None
```
Creates an instance of TogetherAIChatGenerator. Unless specified otherwise,
Creates an instance of TogetherAIChatGenerator.
the default model is `meta-llama/Llama-3.3-70B-Instruct-Turbo`.
**Parameters:**
**Arguments**:
- `api_key`: The Together API key.
- `model`: The name of the Together AI chat completion model to use.
- `streaming_callback`: A callback function that is called when a new token is received from the stream.
The callback function accepts StreamingChunk as an argument.
- `api_base_url`: The Together AI API Base url.
For more details, see Together AI [docs](https://docs.together.ai/docs/openai-api-compatibility).
- `generation_kwargs`: Other parameters to use for the model. These parameters are all sent directly to
the Together AI endpoint. See [Together AI API docs](https://docs.together.ai/reference/chat-completions-1)
for more details.
Some of the supported parameters:
- **api_key** (<code>Secret</code>) – The Together API key.
- **model** (<code>str</code>) – The name of the Together AI chat completion model to use.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
The callback function accepts StreamingChunk as an argument.
- **api_base_url** (<code>str | None</code>) – The Together AI API Base url.
For more details, see Together AI [docs](https://docs.together.ai/docs/openai-api-compatibility).
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Other parameters to use for the model. These parameters are all sent directly to
the Together AI endpoint. See [Together AI API docs](https://docs.together.ai/reference/chat-completions-1)
for more details.
Some of the supported parameters:
- `max_tokens`: The maximum number of tokens the output text can have.
- `temperature`: What sampling temperature to use. Higher values mean the model will take more risks.
Try 0.9 for more creative applications and 0 (argmax sampling) for ones with a well-defined answer.
Try 0.9 for more creative applications and 0 (argmax sampling) for ones with a well-defined answer.
- `top_p`: An alternative to sampling with temperature, called nucleus sampling, where the model
considers the results of the tokens with top_p probability mass. So 0.1 means only the tokens
comprising the top 10% probability mass are considered.
considers the results of the tokens with top_p probability mass. So 0.1 means only the tokens
comprising the top 10% probability mass are considered.
- `stream`: Whether to stream back partial progress. If set, tokens will be sent as data-only server-sent
events as they become available, with the stream terminated by a data: [DONE] message.
events as they become available, with the stream terminated by a data: [DONE] message.
- `safe_prompt`: Whether to inject a safety prompt before all conversations.
- `random_seed`: The seed to use for random sampling.
- `response_format`: A JSON schema or a Pydantic model that enforces the structure of the model's response.
If provided, the output will always be validated against this
format (unless the model returns a tool call).
For details, see the [OpenAI Structured Outputs documentation](https://platform.openai.com/docs/guides/structured-outputs).
Notes:
- For structured outputs with streaming,
the `response_format` must be a JSON schema and not a Pydantic model.
- `tools`: A list of Tool and/or Toolset objects, or a single Toolset for which the model can prepare calls.
Each tool should have a unique name.
- `timeout`: The timeout for the Together AI API call.
- `max_retries`: Maximum number of retries to contact Together AI after an internal error.
If not set, it defaults to either the `OPENAI_MAX_RETRIES` environment variable, or set to 5.
- `http_client_kwargs`: A dictionary of keyword arguments to configure a custom `httpx.Client`or `httpx.AsyncClient`.
For more information, see the [HTTPX documentation](https://www.python-httpx.org/api/`client`).
If provided, the output will always be validated against this
format (unless the model returns a tool call).
For details, see the [OpenAI Structured Outputs documentation](https://platform.openai.com/docs/guides/structured-outputs).
Notes:
- For structured outputs with streaming,
the `response_format` must be a JSON schema and not a Pydantic model.
- **tools** (<code>ToolsType | None</code>) – A list of Tool and/or Toolset objects, or a single Toolset for which the model can prepare calls.
Each tool should have a unique name.
- **timeout** (<code>float | None</code>) – The timeout for the Together AI API call.
- **max_retries** (<code>int | None</code>) – Maximum number of retries to contact Together AI after an internal error.
If not set, it defaults to either the `OPENAI_MAX_RETRIES` environment variable, or set to 5.
- **http_client_kwargs** (<code>dict\[str, Any\] | None</code>) – A dictionary of keyword arguments to configure a custom `httpx.Client`or `httpx.AsyncClient`.
For more information, see the [HTTPX documentation](https://www.python-httpx.org/api/#client).
<a id="haystack_integrations.components.generators.togetherai.chat.chat_generator.TogetherAIChatGenerator.to_dict"></a>
#### TogetherAIChatGenerator.to\_dict
#### to_dict
```python
def to_dict() -> dict[str, Any]
to_dict() -> dict[str, Any]
```
Serialize this component to a dictionary.
**Returns**:
**Returns:**
The serialized component as a dictionary.
- <code>dict\[str, Any\]</code> – The serialized component as a dictionary.
<a id="haystack_integrations.components.generators.togetherai.generator"></a>
## Module haystack\_integrations.components.generators.togetherai.generator
<a id="haystack_integrations.components.generators.togetherai.generator.TogetherAIGenerator"></a>
## haystack_integrations.components.generators.togetherai.generator
### TogetherAIGenerator
Bases: <code>TogetherAIChatGenerator</code>
Provides an interface to generate text using an LLM running on Together AI.
Usage example:
```python
from haystack_integrations.components.generators.togetherai import TogetherAIGenerator
@@ -145,150 +142,141 @@ generator = TogetherAIGenerator(model="deepseek-ai/DeepSeek-R1",
print(generator.run("Who is the best Italian actor?"))
```
<a id="haystack_integrations.components.generators.togetherai.generator.TogetherAIGenerator.__init__"></a>
#### TogetherAIGenerator.\_\_init\_\_
#### __init__
```python
def __init__(api_key: Secret = Secret.from_env_var("TOGETHER_API_KEY"),
model: str = "meta-llama/Llama-3.3-70B-Instruct-Turbo",
api_base_url: str | None = "https://api.together.xyz/v1",
streaming_callback: StreamingCallbackT | None = None,
system_prompt: str | None = None,
generation_kwargs: dict[str, Any] | None = None,
timeout: float | None = None,
max_retries: int | None = None)
__init__(
api_key: Secret = Secret.from_env_var("TOGETHER_API_KEY"),
model: str = "meta-llama/Llama-3.3-70B-Instruct-Turbo",
api_base_url: str | None = "https://api.together.xyz/v1",
streaming_callback: StreamingCallbackT | None = None,
system_prompt: str | None = None,
generation_kwargs: dict[str, Any] | None = None,
timeout: float | None = None,
max_retries: int | None = None,
) -> None
```
Initialize the TogetherAIGenerator.
**Arguments**:
**Parameters:**
- `api_key`: The Together API key.
- `model`: The name of the model to use.
- `api_base_url`: The base URL of the Together AI API.
- `streaming_callback`: A callback function that is called when a new token is received from the stream.
The callback function accepts StreamingChunk as an argument.
- `system_prompt`: The system prompt to use for text generation. If not provided, the system prompt is
omitted, and the default system prompt of the model is used.
- `generation_kwargs`: Other parameters to use for the model. These parameters are all sent directly to
the Together AI endpoint. See Together AI
[documentation](https://docs.together.ai/reference/chat-completions-1) for more details.
Some of the supported parameters:
- **api_key** (<code>Secret</code>) – The Together API key.
- **model** (<code>str</code>) – The name of the model to use.
- **api_base_url** (<code>str | None</code>) – The base URL of the Together AI API.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
The callback function accepts StreamingChunk as an argument.
- **system_prompt** (<code>str | None</code>) – The system prompt to use for text generation. If not provided, the system prompt is
omitted, and the default system prompt of the model is used.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Other parameters to use for the model. These parameters are all sent directly to
the Together AI endpoint. See Together AI
[documentation](https://docs.together.ai/reference/chat-completions-1) for more details.
Some of the supported parameters:
- `max_tokens`: The maximum number of tokens the output text can have.
- `temperature`: What sampling temperature to use. Higher values mean the model will take more risks.
Try 0.9 for more creative applications and 0 (argmax sampling) for ones with a well-defined answer.
Try 0.9 for more creative applications and 0 (argmax sampling) for ones with a well-defined answer.
- `top_p`: An alternative to sampling with temperature, called nucleus sampling, where the model
considers the results of the tokens with top_p probability mass. So, 0.1 means only the tokens
comprising the top 10% probability mass are considered.
considers the results of the tokens with top_p probability mass. So, 0.1 means only the tokens
comprising the top 10% probability mass are considered.
- `n`: How many completions to generate for each prompt. For example, if the LLM gets 3 prompts and n is 2,
it will generate two completions for each of the three prompts, ending up with 6 completions in total.
it will generate two completions for each of the three prompts, ending up with 6 completions in total.
- `stop`: One or more sequences after which the LLM should stop generating tokens.
- `presence_penalty`: What penalty to apply if a token is already present at all. Bigger values mean
the model will be less likely to repeat the same token in the text.
the model will be less likely to repeat the same token in the text.
- `frequency_penalty`: What penalty to apply if a token has already been generated in the text.
Bigger values mean the model will be less likely to repeat the same token in the text.
Bigger values mean the model will be less likely to repeat the same token in the text.
- `logit_bias`: Add a logit bias to specific tokens. The keys of the dictionary are tokens, and the
values are the bias to add to that token.
- `timeout`: Timeout for together.ai Client calls, if not set it is inferred from the `OPENAI_TIMEOUT` environment
variable or set to 30.
- `max_retries`: Maximum retries to establish contact with Together AI if it returns an internal error, if not set it is
inferred from the `OPENAI_MAX_RETRIES` environment variable or set to 5.
values are the bias to add to that token.
- **timeout** (<code>float | None</code>) – Timeout for together.ai Client calls, if not set it is inferred from the `OPENAI_TIMEOUT` environment
variable or set to 30.
- **max_retries** (<code>int | None</code>) – Maximum retries to establish contact with Together AI if it returns an internal error, if not set it is
inferred from the `OPENAI_MAX_RETRIES` environment variable or set to 5.
<a id="haystack_integrations.components.generators.togetherai.generator.TogetherAIGenerator.to_dict"></a>
#### TogetherAIGenerator.to\_dict
#### to_dict
```python
def to_dict() -> dict[str, Any]
to_dict() -> dict[str, Any]
```
Serialize this component to a dictionary.
**Returns**:
**Returns:**
The serialized component as a dictionary.
- <code>dict\[str, Any\]</code> – The serialized component as a dictionary.
<a id="haystack_integrations.components.generators.togetherai.generator.TogetherAIGenerator.from_dict"></a>
#### TogetherAIGenerator.from\_dict
#### from_dict
```python
@classmethod
def from_dict(cls, data: dict[str, Any]) -> "TogetherAIGenerator"
from_dict(data: dict[str, Any]) -> TogetherAIGenerator
```
Deserialize this component from a dictionary.
**Arguments**:
**Parameters:**
- `data`: The dictionary representation of this component.
- **data** (<code>dict\[str, Any\]</code>) – The dictionary representation of this component.
**Returns**:
**Returns:**
The deserialized component instance.
- <code>TogetherAIGenerator</code> – The deserialized component instance.
<a id="haystack_integrations.components.generators.togetherai.generator.TogetherAIGenerator.run"></a>
#### TogetherAIGenerator.run
#### run
```python
@component.output_types(replies=list[str], meta=list[dict[str, Any]])
def run(*,
prompt: str,
system_prompt: str | None = None,
streaming_callback: StreamingCallbackT | None = None,
generation_kwargs: dict[str, Any] | None = None) -> dict[str, Any]
run(
*,
prompt: str,
system_prompt: str | None = None,
streaming_callback: StreamingCallbackT | None = None,
generation_kwargs: dict[str, Any] | None = None
) -> dict[str, Any]
```
Generate text completions synchronously.
**Arguments**:
**Parameters:**
- `prompt`: The input prompt string for text generation.
- `system_prompt`: An optional system prompt to provide context or instructions for the generation.
If not provided, the system prompt set in the `__init__` method will be used.
- `streaming_callback`: A callback function that is called when a new token is received from the stream.
If provided, this will override the `streaming_callback` set in the `__init__` method.
- `generation_kwargs`: Additional keyword arguments for text generation. These parameters will potentially override the parameters
passed in the `__init__` method. Supported parameters include temperature, max_new_tokens, top_p, etc.
- **prompt** (<code>str</code>) – The input prompt string for text generation.
- **system_prompt** (<code>str | None</code>) – An optional system prompt to provide context or instructions for the generation.
If not provided, the system prompt set in the `__init__` method will be used.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
If provided, this will override the `streaming_callback` set in the `__init__` method.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Additional keyword arguments for text generation. These parameters will potentially override the parameters
passed in the `__init__` method. Supported parameters include temperature, max_new_tokens, top_p, etc.
**Returns**:
**Returns:**
A dictionary with the following keys:
- <code>dict\[str, Any\]</code> – A dictionary with the following keys:
- `replies`: A list of generated text completions as strings.
- `meta`: A list of metadata dictionaries containing information about each generation,
including model name, finish reason, and token usage statistics.
including model name, finish reason, and token usage statistics.
<a id="haystack_integrations.components.generators.togetherai.generator.TogetherAIGenerator.run_async"></a>
#### TogetherAIGenerator.run\_async
#### run_async
```python
@component.output_types(replies=list[str], meta=list[dict[str, Any]])
async def run_async(
*,
prompt: str,
system_prompt: str | None = None,
streaming_callback: StreamingCallbackT | None = None,
generation_kwargs: dict[str, Any] | None = None) -> dict[str, Any]
run_async(
*,
prompt: str,
system_prompt: str | None = None,
streaming_callback: StreamingCallbackT | None = None,
generation_kwargs: dict[str, Any] | None = None
) -> dict[str, Any]
```
Generate text completions asynchronously.
**Arguments**:
**Parameters:**
- `prompt`: The input prompt string for text generation.
- `system_prompt`: An optional system prompt to provide context or instructions for the generation.
- `streaming_callback`: A callback function that is called when a new token is received from the stream.
If provided, this will override the `streaming_callback` set in the `__init__` method.
- `generation_kwargs`: Additional keyword arguments for text generation. These parameters will potentially override the parameters
passed in the `__init__` method. Supported parameters include temperature, max_new_tokens, top_p, etc.
- **prompt** (<code>str</code>) – The input prompt string for text generation.
- **system_prompt** (<code>str | None</code>) – An optional system prompt to provide context or instructions for the generation.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
If provided, this will override the `streaming_callback` set in the `__init__` method.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Additional keyword arguments for text generation. These parameters will potentially override the parameters
passed in the `__init__` method. Supported parameters include temperature, max_new_tokens, top_p, etc.
**Returns**:
**Returns:**
A dictionary with the following keys:
- <code>dict\[str, Any\]</code> – A dictionary with the following keys:
- `replies`: A list of generated text completions as strings.
- `meta`: A list of metadata dictionaries containing information about each generation,
including model name, finish reason, and token usage statistics.
including model name, finish reason, and token usage statistics.
docs-website/reference_versioned_docs/version-2.19/integrations-api/transformers.md
+1061
View File
File diff suppressed because it is too large Load Diff
docs-website/reference_versioned_docs/version-2.19/integrations-api/vllm.md
+6 -4
View File
@@ -478,7 +478,7 @@ Deserialize this component from a dictionary.
```python
run(
messages: list[ChatMessage],
messages: list[ChatMessage] | str,
streaming_callback: StreamingCallbackT | None = None,
generation_kwargs: dict[str, Any] | None = None,
*,
@@ -490,7 +490,8 @@ Run the VLLM chat generator on the given input data.
**Parameters:**
- **messages** (<code>list\[ChatMessage\]</code>) – A list of ChatMessage instances representing the input messages.
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of ChatMessage instances representing the input messages.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Additional keyword arguments for text generation. These parameters will
override the parameters passed during component initialization.
@@ -508,7 +509,7 @@ Run the VLLM chat generator on the given input data.
```python
run_async(
messages: list[ChatMessage],
messages: list[ChatMessage] | str,
streaming_callback: StreamingCallbackT | None = None,
generation_kwargs: dict[str, Any] | None = None,
*,
@@ -520,7 +521,8 @@ Run the VLLM chat generator on the given input data asynchronously.
**Parameters:**
- **messages** (<code>list\[ChatMessage\]</code>) – A list of ChatMessage instances representing the input messages.
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of ChatMessage instances representing the input messages.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
Must be a coroutine.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Additional keyword arguments for text generation. These parameters will
docs-website/reference_versioned_docs/version-2.19/integrations-api/watsonx.md
+8 -6
View File
@@ -54,7 +54,7 @@ __init__(
max_retries: int | None = None,
meta_fields_to_embed: list[str] | None = None,
embedding_separator: str = "\n"
)
) -> None
```
Creates a WatsonxDocumentEmbedder component.
@@ -166,7 +166,7 @@ __init__(
suffix: str = "",
timeout: float | None = None,
max_retries: int | None = None
)
) -> None
```
Creates an WatsonxTextEmbedder component.
@@ -411,7 +411,7 @@ Deserialize this component from a dictionary.
```python
run(
*,
messages: list[ChatMessage],
messages: list[ChatMessage] | str,
generation_kwargs: dict[str, Any] | None = None,
streaming_callback: StreamingCallbackT | None = None,
tools: ToolsType | None = None
@@ -422,7 +422,8 @@ Generate chat completions synchronously.
**Parameters:**
- **messages** (<code>list\[ChatMessage\]</code>) – A list of ChatMessage instances representing the input messages.
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of ChatMessage instances representing the input messages.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Additional keyword arguments for text generation. These parameters will potentially override the parameters
passed in the `__init__` method.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
@@ -440,7 +441,7 @@ Generate chat completions synchronously.
```python
run_async(
*,
messages: list[ChatMessage],
messages: list[ChatMessage] | str,
generation_kwargs: dict[str, Any] | None = None,
streaming_callback: StreamingCallbackT | None = None,
tools: ToolsType | None = None
@@ -451,7 +452,8 @@ Generate chat completions asynchronously.
**Parameters:**
- **messages** (<code>list\[ChatMessage\]</code>) – A list of ChatMessage instances representing the input messages.
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of ChatMessage instances representing the input messages.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Additional keyword arguments for text generation. These parameters will potentially override the parameters
passed in the `__init__` method.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – A callback function that is called when a new token is received from the stream.
docs-website/reference_versioned_docs/version-2.20/integrations-api/amazon_bedrock.md
+6 -4
View File
@@ -1311,7 +1311,7 @@ Deserializes the component from a dictionary.
```python
run(
messages: list[ChatMessage],
messages: list[ChatMessage] | str,
streaming_callback: StreamingCallbackT | None = None,
generation_kwargs: dict[str, Any] | None = None,
tools: ToolsType | None = None,
@@ -1324,7 +1324,8 @@ Supports both standard and streaming responses depending on whether a streaming
**Parameters:**
- **messages** (<code>list\[ChatMessage\]</code>) – A list of `ChatMessage` objects forming the chat history.
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of `ChatMessage` objects forming the chat history.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – Optional callback for handling streaming outputs.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Optional dictionary of generation parameters. Some common parameters are:
- `maxTokens`: Maximum number of tokens to generate.
@@ -1347,7 +1348,7 @@ Supports both standard and streaming responses depending on whether a streaming
```python
run_async(
messages: list[ChatMessage],
messages: list[ChatMessage] | str,
streaming_callback: StreamingCallbackT | None = None,
generation_kwargs: dict[str, Any] | None = None,
tools: ToolsType | None = None,
@@ -1360,7 +1361,8 @@ Designed for use cases where non-blocking or concurrent execution is desired.
**Parameters:**
- **messages** (<code>list\[ChatMessage\]</code>) – A list of `ChatMessage` objects forming the chat history.
- **messages** (<code>list\[ChatMessage\] | str</code>) – A list of `ChatMessage` objects forming the chat history.
If a string is provided, it is converted to a list containing a ChatMessage with user role.
- **streaming_callback** (<code>StreamingCallbackT | None</code>) – Optional async-compatible callback for handling streaming outputs.
- **generation_kwargs** (<code>dict\[str, Any\] | None</code>) – Optional dictionary of generation parameters. Some common parameters are:
- `maxTokens`: Maximum number of tokens to generate.

Some files were not shown because too many files have changed in this diff Show More