deepset-ai/haystack
1
0
Fork 0
mirror of https://github.com/deepset-ai/haystack.git synced 2026-10-12 02:12:50 +00:00
Code Issues Packages Projects Releases Wiki Activity

Sync Haystack API reference on Docusaurus (#10243)

Browse Source 
Co-authored-by: vblagoje <[email protected]>
This commit is contained in:
Haystack Bot
2025-12-15 11:16:02 +01:00
committed by GitHub
co-authored by vblagoje
parent 9c52faf6b8
commit 49d1391bca
9 changed files with 2617 additions and 2609 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs-website/reference/haystack-api/builders_api.md
+202 -202
View File
@@ -147,208 +147,6 @@ Example: `\[(\d+)\]` finds "1" in a string "this is an answer[1]".
A dictionary with the following keys:
- `answers`: The answers received from the output of the Generator.
<a id="prompt_builder"></a>
## Module prompt\_builder
<a id="prompt_builder.PromptBuilder"></a>
### PromptBuilder
Renders a prompt filling in any variables so that it can send it to a Generator.
The prompt uses Jinja2 template syntax.
The variables in the default template are used as PromptBuilder's input and are all optional.
If they're not provided, they're replaced with an empty string in the rendered prompt.
To try out different prompts, you can replace the prompt template at runtime by
providing a template for each pipeline run invocation.
### Usage examples
#### On its own
This example uses PromptBuilder to render a prompt template and fill it with `target_language`
and `snippet`. PromptBuilder returns a prompt with the string "Translate the following context to Spanish.
Context: I can't speak Spanish.; Translation:".
```python
from haystack.components.builders import PromptBuilder
template = "Translate the following context to {{ target_language }}. Context: {{ snippet }}; Translation:"
builder = PromptBuilder(template=template)
builder.run(target_language="spanish", snippet="I can't speak spanish.")
```
#### In a Pipeline
This is an example of a RAG pipeline where PromptBuilder renders a custom prompt template and fills it
with the contents of the retrieved documents and a query. The rendered prompt is then sent to a Generator.
```python
from haystack import Pipeline, Document
from haystack.utils import Secret
from haystack.components.generators import OpenAIGenerator
from haystack.components.builders.prompt_builder import PromptBuilder
# in a real world use case documents could come from a retriever, web, or any other source
documents = [Document(content="Joe lives in Berlin"), Document(content="Joe is a software engineer")]
prompt_template = """
Given these documents, answer the question.
Documents:
{% for doc in documents %}
{{ doc.content }}
{% endfor %}
Question: {{query}}
Answer:
"""
p = Pipeline()
p.add_component(instance=PromptBuilder(template=prompt_template), name="prompt_builder")
p.add_component(instance=OpenAIGenerator(api_key=Secret.from_env_var("OPENAI_API_KEY")), name="llm")
p.connect("prompt_builder", "llm")
question = "Where does Joe live?"
result = p.run({"prompt_builder": {"documents": documents, "query": question}})
print(result)
```
#### Changing the template at runtime (prompt engineering)
You can change the prompt template of an existing pipeline, like in this example:
```python
documents = [
Document(content="Joe lives in Berlin", meta={"name": "doc1"}),
Document(content="Joe is a software engineer", meta={"name": "doc1"}),
]
new_template = """
You are a helpful assistant.
Given these documents, answer the question.
Documents:
{% for doc in documents %}
Document {{ loop.index }}:
Document name: {{ doc.meta['name'] }}
{{ doc.content }}
{% endfor %}
Question: {{ query }}
Answer:
"""
p.run({
"prompt_builder": {
"documents": documents,
"query": question,
"template": new_template,
},
})
```
To replace the variables in the default template when testing your prompt,
pass the new variables in the `variables` parameter.
#### Overwriting variables at runtime
To overwrite the values of variables, use `template_variables` during runtime:
```python
language_template = """
You are a helpful assistant.
Given these documents, answer the question.
Documents:
{% for doc in documents %}
Document {{ loop.index }}:
Document name: {{ doc.meta['name'] }}
{{ doc.content }}
{% endfor %}
Question: {{ query }}
Please provide your answer in {{ answer_language | default('English') }}
Answer:
"""
p.run({
"prompt_builder": {
"documents": documents,
"query": question,
"template": language_template,
"template_variables": {"answer_language": "German"},
},
})
```
Note that `language_template` introduces variable `answer_language` which is not bound to any pipeline variable.
If not set otherwise, it will use its default value 'English'.
This example overwrites its value to 'German'.
Use `template_variables` to overwrite pipeline variables (such as documents) as well.
<a id="prompt_builder.PromptBuilder.__init__"></a>
#### PromptBuilder.\_\_init\_\_
```python
def __init__(template: str,
required_variables: Optional[Union[list[str],
Literal["*"]]] = None,
variables: Optional[list[str]] = None)
```
Constructs a PromptBuilder component.
**Arguments**:
- `template`: A prompt template that uses Jinja2 syntax to add variables. For example:
`"Summarize this document: {{ documents[0].content }}\nSummary:"`
It's used to render the prompt.
The variables in the default template are input for PromptBuilder and are all optional,
unless explicitly specified.
If an optional variable is not provided, it's replaced with an empty string in the rendered prompt.
- `required_variables`: List variables that must be provided as input to PromptBuilder.
If a variable listed as required is not provided, an exception is raised.
If set to "*", all variables found in the prompt are required. Optional.
- `variables`: List input variables to use in prompt templates instead of the ones inferred from the
`template` parameter. For example, to use more variables during prompt engineering than the ones present
in the default template, you can provide them here.
<a id="prompt_builder.PromptBuilder.to_dict"></a>
#### PromptBuilder.to\_dict
```python
def to_dict() -> dict[str, Any]
```
Returns a dictionary representation of the component.
**Returns**:
Serialized dictionary representation of the component.
<a id="prompt_builder.PromptBuilder.run"></a>
#### PromptBuilder.run
```python
@component.output_types(prompt=str)
def run(template: Optional[str] = None,
template_variables: Optional[dict[str, Any]] = None,
**kwargs)
```
Renders the prompt template with the provided variables.
It applies the template variables to render the final prompt. You can provide variables via pipeline kwargs.
In order to overwrite the default template, you can set the `template` parameter.
In order to overwrite pipeline kwargs, you can set the `template_variables` parameter.
**Arguments**:
- `template`: An optional string template to overwrite PromptBuilder's default template. If None, the default template
provided at initialization is used.
- `template_variables`: An optional dictionary of template variables to overwrite the pipeline variables.
- `kwargs`: Pipeline variables used for rendering the prompt.
**Raises**:
- `ValueError`: If any of the required template variables is not provided.
**Returns**:
A dictionary with the following keys:
- `prompt`: The updated prompt text after rendering the prompt template.
<a id="chat_prompt_builder"></a>
## Module chat\_prompt\_builder
@@ -553,3 +351,205 @@ Deserialize this component from a dictionary.
The deserialized component.
<a id="prompt_builder"></a>
## Module prompt\_builder
<a id="prompt_builder.PromptBuilder"></a>
### PromptBuilder
Renders a prompt filling in any variables so that it can send it to a Generator.
The prompt uses Jinja2 template syntax.
The variables in the default template are used as PromptBuilder's input and are all optional.
If they're not provided, they're replaced with an empty string in the rendered prompt.
To try out different prompts, you can replace the prompt template at runtime by
providing a template for each pipeline run invocation.
### Usage examples
#### On its own
This example uses PromptBuilder to render a prompt template and fill it with `target_language`
and `snippet`. PromptBuilder returns a prompt with the string "Translate the following context to Spanish.
Context: I can't speak Spanish.; Translation:".
```python
from haystack.components.builders import PromptBuilder
template = "Translate the following context to {{ target_language }}. Context: {{ snippet }}; Translation:"
builder = PromptBuilder(template=template)
builder.run(target_language="spanish", snippet="I can't speak spanish.")
```
#### In a Pipeline
This is an example of a RAG pipeline where PromptBuilder renders a custom prompt template and fills it
with the contents of the retrieved documents and a query. The rendered prompt is then sent to a Generator.
```python
from haystack import Pipeline, Document
from haystack.utils import Secret
from haystack.components.generators import OpenAIGenerator
from haystack.components.builders.prompt_builder import PromptBuilder
# in a real world use case documents could come from a retriever, web, or any other source
documents = [Document(content="Joe lives in Berlin"), Document(content="Joe is a software engineer")]
prompt_template = """
Given these documents, answer the question.
Documents:
{% for doc in documents %}
{{ doc.content }}
{% endfor %}
Question: {{query}}
Answer:
"""
p = Pipeline()
p.add_component(instance=PromptBuilder(template=prompt_template), name="prompt_builder")
p.add_component(instance=OpenAIGenerator(api_key=Secret.from_env_var("OPENAI_API_KEY")), name="llm")
p.connect("prompt_builder", "llm")
question = "Where does Joe live?"
result = p.run({"prompt_builder": {"documents": documents, "query": question}})
print(result)
```
#### Changing the template at runtime (prompt engineering)
You can change the prompt template of an existing pipeline, like in this example:
```python
documents = [
Document(content="Joe lives in Berlin", meta={"name": "doc1"}),
Document(content="Joe is a software engineer", meta={"name": "doc1"}),
]
new_template = """
You are a helpful assistant.
Given these documents, answer the question.
Documents:
{% for doc in documents %}
Document {{ loop.index }}:
Document name: {{ doc.meta['name'] }}
{{ doc.content }}
{% endfor %}
Question: {{ query }}
Answer:
"""
p.run({
"prompt_builder": {
"documents": documents,
"query": question,
"template": new_template,
},
})
```
To replace the variables in the default template when testing your prompt,
pass the new variables in the `variables` parameter.
#### Overwriting variables at runtime
To overwrite the values of variables, use `template_variables` during runtime:
```python
language_template = """
You are a helpful assistant.
Given these documents, answer the question.
Documents:
{% for doc in documents %}
Document {{ loop.index }}:
Document name: {{ doc.meta['name'] }}
{{ doc.content }}
{% endfor %}
Question: {{ query }}
Please provide your answer in {{ answer_language | default('English') }}
Answer:
"""
p.run({
"prompt_builder": {
"documents": documents,
"query": question,
"template": language_template,
"template_variables": {"answer_language": "German"},
},
})
```
Note that `language_template` introduces variable `answer_language` which is not bound to any pipeline variable.
If not set otherwise, it will use its default value 'English'.
This example overwrites its value to 'German'.
Use `template_variables` to overwrite pipeline variables (such as documents) as well.
<a id="prompt_builder.PromptBuilder.__init__"></a>
#### PromptBuilder.\_\_init\_\_
```python
def __init__(template: str,
required_variables: Optional[Union[list[str],
Literal["*"]]] = None,
variables: Optional[list[str]] = None)
```
Constructs a PromptBuilder component.
**Arguments**:
- `template`: A prompt template that uses Jinja2 syntax to add variables. For example:
`"Summarize this document: {{ documents[0].content }}\nSummary:"`
It's used to render the prompt.
The variables in the default template are input for PromptBuilder and are all optional,
unless explicitly specified.
If an optional variable is not provided, it's replaced with an empty string in the rendered prompt.
- `required_variables`: List variables that must be provided as input to PromptBuilder.
If a variable listed as required is not provided, an exception is raised.
If set to "*", all variables found in the prompt are required. Optional.
- `variables`: List input variables to use in prompt templates instead of the ones inferred from the
`template` parameter. For example, to use more variables during prompt engineering than the ones present
in the default template, you can provide them here.
<a id="prompt_builder.PromptBuilder.to_dict"></a>
#### PromptBuilder.to\_dict
```python
def to_dict() -> dict[str, Any]
```
Returns a dictionary representation of the component.
**Returns**:
Serialized dictionary representation of the component.
<a id="prompt_builder.PromptBuilder.run"></a>
#### PromptBuilder.run
```python
@component.output_types(prompt=str)
def run(template: Optional[str] = None,
template_variables: Optional[dict[str, Any]] = None,
**kwargs)
```
Renders the prompt template with the provided variables.
It applies the template variables to render the final prompt. You can provide variables via pipeline kwargs.
In order to overwrite the default template, you can set the `template` parameter.
In order to overwrite pipeline kwargs, you can set the `template_variables` parameter.
**Arguments**:
- `template`: An optional string template to overwrite PromptBuilder's default template. If None, the default template
provided at initialization is used.
- `template_variables`: An optional dictionary of template variables to overwrite the pipeline variables.
- `kwargs`: Pipeline variables used for rendering the prompt.
**Raises**:
- `ValueError`: If any of the required template variables is not provided.
**Returns**:
A dictionary with the following keys:
- `prompt`: The updated prompt text after rendering the prompt template.
docs-website/reference/haystack-api/classifiers_api.md
+9 -1
View File
@@ -36,7 +36,15 @@ document_store = InMemoryDocumentStore()
p = Pipeline()
p.add_component(instance=DocumentLanguageClassifier(languages=["en"]), name="language_classifier")
p.add_component(instance=MetadataRouter(rules={"en": {"language": {"$eq": "en"}}}), name="router")
p.add_component(
instance=MetadataRouter(rules={
"en": {
"field": "meta.language",
"operator": "==",
"value": "en"
}
}),
name="router")
p.add_component(instance=DocumentWriter(document_store=document_store), name="writer")
p.connect("language_classifier.documents", "router.documents")
p.connect("router.en", "writer.documents")
docs-website/reference/haystack-api/connectors_api.md
+99 -99
View File
@@ -5,6 +5,105 @@ description: "Various connectors to integrate with external services."
slug: "/connectors-api"
---
<a id="openapi"></a>
## Module openapi
<a id="openapi.OpenAPIConnector"></a>
### OpenAPIConnector
OpenAPIConnector enables direct invocation of REST endpoints defined in an OpenAPI specification.
The OpenAPIConnector serves as a bridge between Haystack pipelines and any REST API that follows
the OpenAPI(formerly Swagger) specification. It dynamically interprets the API specification and
provides an interface for executing API operations. It is usually invoked by passing input
arguments to it from a Haystack pipeline run method or by other components in a pipeline that
pass input arguments to this component.
**Example**:
```python
from haystack.utils import Secret
from haystack.components.connectors.openapi import OpenAPIConnector
connector = OpenAPIConnector(
openapi_spec="https://bit.ly/serperdev_openapi",
credentials=Secret.from_env_var("SERPERDEV_API_KEY"),
service_kwargs={"config_factory": my_custom_config_factory}
)
response = connector.run(
operation_id="search",
arguments={"q": "Who was Nikola Tesla?"}
)
```
**Notes**:
- The `parameters` argument is required for this component.
- The `service_kwargs` argument is optional, it can be used to pass additional options to the OpenAPIClient.
<a id="openapi.OpenAPIConnector.__init__"></a>
#### OpenAPIConnector.\_\_init\_\_
```python
def __init__(openapi_spec: str,
credentials: Optional[Secret] = None,
service_kwargs: Optional[dict[str, Any]] = None)
```
Initialize the OpenAPIConnector with a specification and optional credentials.
**Arguments**:
- `openapi_spec`: URL, file path, or raw string of the OpenAPI specification
- `credentials`: Optional API key or credentials for the service wrapped in a Secret
- `service_kwargs`: Additional keyword arguments passed to OpenAPIClient.from_spec()
For example, you can pass a custom config_factory or other configuration options.
<a id="openapi.OpenAPIConnector.to_dict"></a>
#### OpenAPIConnector.to\_dict
```python
def to_dict() -> dict[str, Any]
```
Serialize this component to a dictionary.
<a id="openapi.OpenAPIConnector.from_dict"></a>
#### OpenAPIConnector.from\_dict
```python
@classmethod
def from_dict(cls, data: dict[str, Any]) -> "OpenAPIConnector"
```
Deserialize this component from a dictionary.
<a id="openapi.OpenAPIConnector.run"></a>
#### OpenAPIConnector.run
```python
@component.output_types(response=dict[str, Any])
def run(operation_id: str,
arguments: Optional[dict[str, Any]] = None) -> dict[str, Any]
```
Invokes a REST endpoint specified in the OpenAPI specification.
**Arguments**:
- `operation_id`: The operationId from the OpenAPI spec to invoke
- `arguments`: Optional parameters for the endpoint (query, path, or body parameters)
**Returns**:
Dictionary containing the service response
<a id="openapi_service"></a>
## Module openapi\_service
@@ -146,102 +245,3 @@ Deserializes the component from a dictionary.
The deserialized component.
<a id="openapi"></a>
## Module openapi
<a id="openapi.OpenAPIConnector"></a>
### OpenAPIConnector
OpenAPIConnector enables direct invocation of REST endpoints defined in an OpenAPI specification.
The OpenAPIConnector serves as a bridge between Haystack pipelines and any REST API that follows
the OpenAPI(formerly Swagger) specification. It dynamically interprets the API specification and
provides an interface for executing API operations. It is usually invoked by passing input
arguments to it from a Haystack pipeline run method or by other components in a pipeline that
pass input arguments to this component.
**Example**:
```python
from haystack.utils import Secret
from haystack.components.connectors.openapi import OpenAPIConnector
connector = OpenAPIConnector(
openapi_spec="https://bit.ly/serperdev_openapi",
credentials=Secret.from_env_var("SERPERDEV_API_KEY"),
service_kwargs={"config_factory": my_custom_config_factory}
)
response = connector.run(
operation_id="search",
arguments={"q": "Who was Nikola Tesla?"}
)
```
**Notes**:
- The `parameters` argument is required for this component.
- The `service_kwargs` argument is optional, it can be used to pass additional options to the OpenAPIClient.
<a id="openapi.OpenAPIConnector.__init__"></a>
#### OpenAPIConnector.\_\_init\_\_
```python
def __init__(openapi_spec: str,
credentials: Optional[Secret] = None,
service_kwargs: Optional[dict[str, Any]] = None)
```
Initialize the OpenAPIConnector with a specification and optional credentials.
**Arguments**:
- `openapi_spec`: URL, file path, or raw string of the OpenAPI specification
- `credentials`: Optional API key or credentials for the service wrapped in a Secret
- `service_kwargs`: Additional keyword arguments passed to OpenAPIClient.from_spec()
For example, you can pass a custom config_factory or other configuration options.
<a id="openapi.OpenAPIConnector.to_dict"></a>
#### OpenAPIConnector.to\_dict
```python
def to_dict() -> dict[str, Any]
```
Serialize this component to a dictionary.
<a id="openapi.OpenAPIConnector.from_dict"></a>
#### OpenAPIConnector.from\_dict
```python
@classmethod
def from_dict(cls, data: dict[str, Any]) -> "OpenAPIConnector"
```
Deserialize this component from a dictionary.
<a id="openapi.OpenAPIConnector.run"></a>
#### OpenAPIConnector.run
```python
@component.output_types(response=dict[str, Any])
def run(operation_id: str,
arguments: Optional[dict[str, Any]] = None) -> dict[str, Any]
```
Invokes a REST endpoint specified in the OpenAPI specification.
**Arguments**:
- `operation_id`: The operationId from the OpenAPI spec to invoke
- `arguments`: Optional parameters for the endpoint (query, path, or body parameters)
**Returns**:
Dictionary containing the service response
docs-website/reference/haystack-api/embedders_api.md
+228 -228
View File
@@ -696,6 +696,176 @@ Embeds a single string asynchronously.
A dictionary with the following keys:
- `embedding`: The embedding of the input text.
<a id="image/sentence_transformers_doc_image_embedder"></a>
## Module image/sentence\_transformers\_doc\_image\_embedder
<a id="image/sentence_transformers_doc_image_embedder.SentenceTransformersDocumentImageEmbedder"></a>
### SentenceTransformersDocumentImageEmbedder
A component for computing Document embeddings based on images using Sentence Transformers models.
The embedding of each Document is stored in the `embedding` field of the Document.
### Usage example
```python
from haystack import Document
from haystack.components.embedders.image import SentenceTransformersDocumentImageEmbedder
embedder = SentenceTransformersDocumentImageEmbedder(model="sentence-transformers/clip-ViT-B-32")
embedder.warm_up()
documents = [
Document(content="A photo of a cat", meta={"file_path": "cat.jpg"}),
Document(content="A photo of a dog", meta={"file_path": "dog.jpg"}),
]
result = embedder.run(documents=documents)
documents_with_embeddings = result["documents"]
print(documents_with_embeddings)
# [Document(id=...,
# content='A photo of a cat',
# meta={'file_path': 'cat.jpg',
# 'embedding_source': {'type': 'image', 'file_path_meta_field': 'file_path'}},
# embedding=vector of size 512),
# ...]
```
<a id="image/sentence_transformers_doc_image_embedder.SentenceTransformersDocumentImageEmbedder.__init__"></a>
#### SentenceTransformersDocumentImageEmbedder.\_\_init\_\_
```python
def __init__(*,
file_path_meta_field: str = "file_path",
root_path: Optional[str] = None,
model: str = "sentence-transformers/clip-ViT-B-32",
device: Optional[ComponentDevice] = None,
token: Optional[Secret] = Secret.from_env_var(
["HF_API_TOKEN", "HF_TOKEN"], strict=False),
batch_size: int = 32,
progress_bar: bool = True,
normalize_embeddings: bool = False,
trust_remote_code: bool = False,
local_files_only: bool = False,
model_kwargs: Optional[dict[str, Any]] = None,
tokenizer_kwargs: Optional[dict[str, Any]] = None,
config_kwargs: Optional[dict[str, Any]] = None,
precision: Literal["float32", "int8", "uint8", "binary",
"ubinary"] = "float32",
encode_kwargs: Optional[dict[str, Any]] = None,
backend: Literal["torch", "onnx", "openvino"] = "torch") -> None
```
Creates a SentenceTransformersDocumentEmbedder component.
**Arguments**:
- `file_path_meta_field`: The metadata field in the Document that contains the file path to the image or PDF.
- `root_path`: The root directory path where document files are located. If provided, file paths in
document metadata will be resolved relative to this path. If None, file paths are treated as absolute paths.
- `model`: The Sentence Transformers model to use for calculating embeddings. Pass a local path or ID of the model on
Hugging Face. To be used with this component, the model must be able to embed images and text into the same
vector space. Compatible models include:
- "sentence-transformers/clip-ViT-B-32"
- "sentence-transformers/clip-ViT-L-14"
- "sentence-transformers/clip-ViT-B-16"
- "sentence-transformers/clip-ViT-B-32-multilingual-v1"
- "jinaai/jina-embeddings-v4"
- "jinaai/jina-clip-v1"
- "jinaai/jina-clip-v2".
- `device`: The device to use for loading the model.
Overrides the default device.
- `token`: The API token to download private models from Hugging Face.
- `batch_size`: Number of documents to embed at once.
- `progress_bar`: If `True`, shows a progress bar when embedding documents.
- `normalize_embeddings`: If `True`, the embeddings are normalized using L2 normalization, so that each embedding has a norm of 1.
- `trust_remote_code`: If `False`, allows only Hugging Face verified model architectures.
If `True`, allows custom models and scripts.
- `local_files_only`: If `True`, does not attempt to download the model from Hugging Face Hub and only looks at local files.
- `model_kwargs`: Additional keyword arguments for `AutoModelForSequenceClassification.from_pretrained`
when loading the model. Refer to specific model documentation for available kwargs.
- `tokenizer_kwargs`: Additional keyword arguments for `AutoTokenizer.from_pretrained` when loading the tokenizer.
Refer to specific model documentation for available kwargs.
- `config_kwargs`: Additional keyword arguments for `AutoConfig.from_pretrained` when loading the model configuration.
- `precision`: The precision to use for the embeddings.
All non-float32 precisions are quantized embeddings.
Quantized embeddings are smaller and faster to compute, but may have a lower accuracy.
They are useful for reducing the size of the embeddings of a corpus for semantic search, among other tasks.
- `encode_kwargs`: Additional keyword arguments for `SentenceTransformer.encode` when embedding documents.
This parameter is provided for fine customization. Be careful not to clash with already set parameters and
avoid passing parameters that change the output type.
- `backend`: The backend to use for the Sentence Transformers model. Choose from "torch", "onnx", or "openvino".
Refer to the [Sentence Transformers documentation](https://sbert.net/docs/sentence_transformer/usage/efficiency.html)
for more information on acceleration and quantization options.
<a id="image/sentence_transformers_doc_image_embedder.SentenceTransformersDocumentImageEmbedder.to_dict"></a>
#### SentenceTransformersDocumentImageEmbedder.to\_dict
```python
def to_dict() -> dict[str, Any]
```
Serializes the component to a dictionary.
**Returns**:
Dictionary with serialized data.
<a id="image/sentence_transformers_doc_image_embedder.SentenceTransformersDocumentImageEmbedder.from_dict"></a>
#### SentenceTransformersDocumentImageEmbedder.from\_dict
```python
@classmethod
def from_dict(
cls, data: dict[str,
Any]) -> "SentenceTransformersDocumentImageEmbedder"
```
Deserializes the component from a dictionary.
**Arguments**:
- `data`: Dictionary to deserialize from.
**Returns**:
Deserialized component.
<a id="image/sentence_transformers_doc_image_embedder.SentenceTransformersDocumentImageEmbedder.warm_up"></a>
#### SentenceTransformersDocumentImageEmbedder.warm\_up
```python
def warm_up() -> None
```
Initializes the component.
<a id="image/sentence_transformers_doc_image_embedder.SentenceTransformersDocumentImageEmbedder.run"></a>
#### SentenceTransformersDocumentImageEmbedder.run
```python
@component.output_types(documents=list[Document])
def run(documents: list[Document]) -> dict[str, list[Document]]
```
Embed a list of documents.
**Arguments**:
- `documents`: Documents to embed.
**Returns**:
A dictionary with the following keys:
- `documents`: Documents with embeddings.
<a id="openai_document_embedder"></a>
## Module openai\_document\_embedder
@@ -1170,162 +1340,6 @@ Embed a list of documents.
A dictionary with the following keys:
- `documents`: Documents with embeddings.
<a id="sentence_transformers_text_embedder"></a>
## Module sentence\_transformers\_text\_embedder
<a id="sentence_transformers_text_embedder.SentenceTransformersTextEmbedder"></a>
### SentenceTransformersTextEmbedder
Embeds strings using Sentence Transformers models.
You can use it to embed user query and send it to an embedding retriever.
Usage example:
```python
from haystack.components.embedders import SentenceTransformersTextEmbedder
text_to_embed = "I love pizza!"
text_embedder = SentenceTransformersTextEmbedder()
text_embedder.warm_up()
print(text_embedder.run(text_to_embed))
# {'embedding': [-0.07804739475250244, 0.1498992145061493,, ...]}
```
<a id="sentence_transformers_text_embedder.SentenceTransformersTextEmbedder.__init__"></a>
#### SentenceTransformersTextEmbedder.\_\_init\_\_
```python
def __init__(model: str = "sentence-transformers/all-mpnet-base-v2",
device: Optional[ComponentDevice] = None,
token: Optional[Secret] = Secret.from_env_var(
["HF_API_TOKEN", "HF_TOKEN"], strict=False),
prefix: str = "",
suffix: str = "",
batch_size: int = 32,
progress_bar: bool = True,
normalize_embeddings: bool = False,
trust_remote_code: bool = False,
local_files_only: bool = False,
truncate_dim: Optional[int] = None,
model_kwargs: Optional[dict[str, Any]] = None,
tokenizer_kwargs: Optional[dict[str, Any]] = None,
config_kwargs: Optional[dict[str, Any]] = None,
precision: Literal["float32", "int8", "uint8", "binary",
"ubinary"] = "float32",
encode_kwargs: Optional[dict[str, Any]] = None,
backend: Literal["torch", "onnx", "openvino"] = "torch",
revision: Optional[str] = None)
```
Create a SentenceTransformersTextEmbedder component.
**Arguments**:
- `model`: The model to use for calculating embeddings.
Specify the path to a local model or the ID of the model on Hugging Face.
- `device`: Overrides the default device used to load the model.
- `token`: An API token to use private models from Hugging Face.
- `prefix`: A string to add at the beginning of each text to be embedded.
You can use it to prepend the text with an instruction, as required by some embedding models,
such as E5 and bge.
- `suffix`: A string to add at the end of each text to embed.
- `batch_size`: Number of texts to embed at once.
- `progress_bar`: If `True`, shows a progress bar for calculating embeddings.
If `False`, disables the progress bar.
- `normalize_embeddings`: If `True`, the embeddings are normalized using L2 normalization, so that the embeddings have a norm of 1.
- `trust_remote_code`: If `False`, permits only Hugging Face verified model architectures.
If `True`, permits custom models and scripts.
- `local_files_only`: If `True`, does not attempt to download the model from Hugging Face Hub and only looks at local files.
- `truncate_dim`: The dimension to truncate sentence embeddings to. `None` does no truncation.
If the model has not been trained with Matryoshka Representation Learning,
truncation of embeddings can significantly affect performance.
- `model_kwargs`: Additional keyword arguments for `AutoModelForSequenceClassification.from_pretrained`
when loading the model. Refer to specific model documentation for available kwargs.
- `tokenizer_kwargs`: Additional keyword arguments for `AutoTokenizer.from_pretrained` when loading the tokenizer.
Refer to specific model documentation for available kwargs.
- `config_kwargs`: Additional keyword arguments for `AutoConfig.from_pretrained` when loading the model configuration.
- `precision`: The precision to use for the embeddings.
All non-float32 precisions are quantized embeddings.
Quantized embeddings are smaller in size and faster to compute, but may have a lower accuracy.
They are useful for reducing the size of the embeddings of a corpus for semantic search, among other tasks.
- `encode_kwargs`: Additional keyword arguments for `SentenceTransformer.encode` when embedding texts.
This parameter is provided for fine customization. Be careful not to clash with already set parameters and
avoid passing parameters that change the output type.
- `backend`: The backend to use for the Sentence Transformers model. Choose from "torch", "onnx", or "openvino".
Refer to the [Sentence Transformers documentation](https://sbert.net/docs/sentence_transformer/usage/efficiency.html)
for more information on acceleration and quantization options.
- `revision`: The specific model version to use. It can be a branch name, a tag name, or a commit id,
for a stored model on Hugging Face.
<a id="sentence_transformers_text_embedder.SentenceTransformersTextEmbedder.to_dict"></a>
#### SentenceTransformersTextEmbedder.to\_dict
```python
def to_dict() -> dict[str, Any]
```
Serializes the component to a dictionary.
**Returns**:
Dictionary with serialized data.
<a id="sentence_transformers_text_embedder.SentenceTransformersTextEmbedder.from_dict"></a>
#### SentenceTransformersTextEmbedder.from\_dict
```python
@classmethod
def from_dict(cls, data: dict[str, Any]) -> "SentenceTransformersTextEmbedder"
```
Deserializes the component from a dictionary.
**Arguments**:
- `data`: Dictionary to deserialize from.
**Returns**:
Deserialized component.
<a id="sentence_transformers_text_embedder.SentenceTransformersTextEmbedder.warm_up"></a>
#### SentenceTransformersTextEmbedder.warm\_up
```python
def warm_up()
```
Initializes the component.
<a id="sentence_transformers_text_embedder.SentenceTransformersTextEmbedder.run"></a>
#### SentenceTransformersTextEmbedder.run
```python
@component.output_types(embedding=list[float])
def run(text: str)
```
Embed a single string.
**Arguments**:
- `text`: Text to embed.
**Returns**:
A dictionary with the following keys:
- `embedding`: The embedding of the input text.
<a id="sentence_transformers_sparse_document_embedder"></a>
## Module sentence\_transformers\_sparse\_document\_embedder
@@ -1612,95 +1626,81 @@ Embed a single string.
A dictionary with the following keys:
- `sparse_embedding`: The sparse embedding of the input text.
<a id="image/sentence_transformers_doc_image_embedder"></a>
<a id="sentence_transformers_text_embedder"></a>
## Module image/sentence\_transformers\_doc\_image\_embedder
## Module sentence\_transformers\_text\_embedder
<a id="image/sentence_transformers_doc_image_embedder.SentenceTransformersDocumentImageEmbedder"></a>
<a id="sentence_transformers_text_embedder.SentenceTransformersTextEmbedder"></a>
### SentenceTransformersDocumentImageEmbedder
### SentenceTransformersTextEmbedder
A component for computing Document embeddings based on images using Sentence Transformers models.
Embeds strings using Sentence Transformers models.
The embedding of each Document is stored in the `embedding` field of the Document.
You can use it to embed user query and send it to an embedding retriever.
### Usage example
Usage example:
```python
from haystack import Document
from haystack.components.embedders.image import SentenceTransformersDocumentImageEmbedder
from haystack.components.embedders import SentenceTransformersTextEmbedder
embedder = SentenceTransformersDocumentImageEmbedder(model="sentence-transformers/clip-ViT-B-32")
embedder.warm_up()
text_to_embed = "I love pizza!"
documents = [
Document(content="A photo of a cat", meta={"file_path": "cat.jpg"}),
Document(content="A photo of a dog", meta={"file_path": "dog.jpg"}),
]
text_embedder = SentenceTransformersTextEmbedder()
text_embedder.warm_up()
result = embedder.run(documents=documents)
documents_with_embeddings = result["documents"]
print(documents_with_embeddings)
print(text_embedder.run(text_to_embed))
# [Document(id=...,
# content='A photo of a cat',
# meta={'file_path': 'cat.jpg',
# 'embedding_source': {'type': 'image', 'file_path_meta_field': 'file_path'}},
# embedding=vector of size 512),
# ...]
# {'embedding': [-0.07804739475250244, 0.1498992145061493,, ...]}
```
<a id="image/sentence_transformers_doc_image_embedder.SentenceTransformersDocumentImageEmbedder.__init__"></a>
<a id="sentence_transformers_text_embedder.SentenceTransformersTextEmbedder.__init__"></a>
#### SentenceTransformersDocumentImageEmbedder.\_\_init\_\_
#### SentenceTransformersTextEmbedder.\_\_init\_\_
```python
def __init__(*,
file_path_meta_field: str = "file_path",
root_path: Optional[str] = None,
model: str = "sentence-transformers/clip-ViT-B-32",
def __init__(model: str = "sentence-transformers/all-mpnet-base-v2",
device: Optional[ComponentDevice] = None,
token: Optional[Secret] = Secret.from_env_var(
["HF_API_TOKEN", "HF_TOKEN"], strict=False),
prefix: str = "",
suffix: str = "",
batch_size: int = 32,
progress_bar: bool = True,
normalize_embeddings: bool = False,
trust_remote_code: bool = False,
local_files_only: bool = False,
truncate_dim: Optional[int] = None,
model_kwargs: Optional[dict[str, Any]] = None,
tokenizer_kwargs: Optional[dict[str, Any]] = None,
config_kwargs: Optional[dict[str, Any]] = None,
precision: Literal["float32", "int8", "uint8", "binary",
"ubinary"] = "float32",
encode_kwargs: Optional[dict[str, Any]] = None,
backend: Literal["torch", "onnx", "openvino"] = "torch") -> None
backend: Literal["torch", "onnx", "openvino"] = "torch",
revision: Optional[str] = None)
```
Creates a SentenceTransformersDocumentEmbedder component.
Create a SentenceTransformersTextEmbedder component.
**Arguments**:
- `file_path_meta_field`: The metadata field in the Document that contains the file path to the image or PDF.
- `root_path`: The root directory path where document files are located. If provided, file paths in
document metadata will be resolved relative to this path. If None, file paths are treated as absolute paths.
- `model`: The Sentence Transformers model to use for calculating embeddings. Pass a local path or ID of the model on
Hugging Face. To be used with this component, the model must be able to embed images and text into the same
vector space. Compatible models include:
- "sentence-transformers/clip-ViT-B-32"
- "sentence-transformers/clip-ViT-L-14"
- "sentence-transformers/clip-ViT-B-16"
- "sentence-transformers/clip-ViT-B-32-multilingual-v1"
- "jinaai/jina-embeddings-v4"
- "jinaai/jina-clip-v1"
- "jinaai/jina-clip-v2".
- `device`: The device to use for loading the model.
Overrides the default device.
- `token`: The API token to download private models from Hugging Face.
- `batch_size`: Number of documents to embed at once.
- `progress_bar`: If `True`, shows a progress bar when embedding documents.
- `normalize_embeddings`: If `True`, the embeddings are normalized using L2 normalization, so that each embedding has a norm of 1.
- `trust_remote_code`: If `False`, allows only Hugging Face verified model architectures.
If `True`, allows custom models and scripts.
- `model`: The model to use for calculating embeddings.
Specify the path to a local model or the ID of the model on Hugging Face.
- `device`: Overrides the default device used to load the model.
- `token`: An API token to use private models from Hugging Face.
- `prefix`: A string to add at the beginning of each text to be embedded.
You can use it to prepend the text with an instruction, as required by some embedding models,
such as E5 and bge.
- `suffix`: A string to add at the end of each text to embed.
- `batch_size`: Number of texts to embed at once.
- `progress_bar`: If `True`, shows a progress bar for calculating embeddings.
If `False`, disables the progress bar.
- `normalize_embeddings`: If `True`, the embeddings are normalized using L2 normalization, so that the embeddings have a norm of 1.
- `trust_remote_code`: If `False`, permits only Hugging Face verified model architectures.
If `True`, permits custom models and scripts.
- `local_files_only`: If `True`, does not attempt to download the model from Hugging Face Hub and only looks at local files.
- `truncate_dim`: The dimension to truncate sentence embeddings to. `None` does no truncation.
If the model has not been trained with Matryoshka Representation Learning,
truncation of embeddings can significantly affect performance.
- `model_kwargs`: Additional keyword arguments for `AutoModelForSequenceClassification.from_pretrained`
when loading the model. Refer to specific model documentation for available kwargs.
- `tokenizer_kwargs`: Additional keyword arguments for `AutoTokenizer.from_pretrained` when loading the tokenizer.
@@ -1708,18 +1708,20 @@ Refer to specific model documentation for available kwargs.
- `config_kwargs`: Additional keyword arguments for `AutoConfig.from_pretrained` when loading the model configuration.
- `precision`: The precision to use for the embeddings.
All non-float32 precisions are quantized embeddings.
Quantized embeddings are smaller and faster to compute, but may have a lower accuracy.
Quantized embeddings are smaller in size and faster to compute, but may have a lower accuracy.
They are useful for reducing the size of the embeddings of a corpus for semantic search, among other tasks.
- `encode_kwargs`: Additional keyword arguments for `SentenceTransformer.encode` when embedding documents.
- `encode_kwargs`: Additional keyword arguments for `SentenceTransformer.encode` when embedding texts.
This parameter is provided for fine customization. Be careful not to clash with already set parameters and
avoid passing parameters that change the output type.
- `backend`: The backend to use for the Sentence Transformers model. Choose from "torch", "onnx", or "openvino".
Refer to the [Sentence Transformers documentation](https://sbert.net/docs/sentence_transformer/usage/efficiency.html)
for more information on acceleration and quantization options.
- `revision`: The specific model version to use. It can be a branch name, a tag name, or a commit id,
for a stored model on Hugging Face.
<a id="image/sentence_transformers_doc_image_embedder.SentenceTransformersDocumentImageEmbedder.to_dict"></a>
<a id="sentence_transformers_text_embedder.SentenceTransformersTextEmbedder.to_dict"></a>
#### SentenceTransformersDocumentImageEmbedder.to\_dict
#### SentenceTransformersTextEmbedder.to\_dict
```python
def to_dict() -> dict[str, Any]
@@ -1731,15 +1733,13 @@ Serializes the component to a dictionary.
Dictionary with serialized data.
<a id="image/sentence_transformers_doc_image_embedder.SentenceTransformersDocumentImageEmbedder.from_dict"></a>
<a id="sentence_transformers_text_embedder.SentenceTransformersTextEmbedder.from_dict"></a>
#### SentenceTransformersDocumentImageEmbedder.from\_dict
#### SentenceTransformersTextEmbedder.from\_dict
```python
@classmethod
def from_dict(
cls, data: dict[str,
Any]) -> "SentenceTransformersDocumentImageEmbedder"
def from_dict(cls, data: dict[str, Any]) -> "SentenceTransformersTextEmbedder"
```
Deserializes the component from a dictionary.
@@ -1752,33 +1752,33 @@ Deserializes the component from a dictionary.
Deserialized component.
<a id="image/sentence_transformers_doc_image_embedder.SentenceTransformersDocumentImageEmbedder.warm_up"></a>
<a id="sentence_transformers_text_embedder.SentenceTransformersTextEmbedder.warm_up"></a>
#### SentenceTransformersDocumentImageEmbedder.warm\_up
#### SentenceTransformersTextEmbedder.warm\_up
```python
def warm_up() -> None
def warm_up()
```
Initializes the component.
<a id="image/sentence_transformers_doc_image_embedder.SentenceTransformersDocumentImageEmbedder.run"></a>
<a id="sentence_transformers_text_embedder.SentenceTransformersTextEmbedder.run"></a>
#### SentenceTransformersDocumentImageEmbedder.run
#### SentenceTransformersTextEmbedder.run
```python
@component.output_types(documents=list[Document])
def run(documents: list[Document]) -> dict[str, list[Document]]
@component.output_types(embedding=list[float])
def run(text: str)
```
Embed a list of documents.
Embed a single string.
**Arguments**:
- `documents`: Documents to embed.
- `text`: Text to embed.
**Returns**:
A dictionary with the following keys:
- `documents`: Documents with embeddings.
- `embedding`: The embedding of the input text.
docs-website/reference/haystack-api/extractors_api.md
+221 -221
View File
@@ -5,154 +5,95 @@ description: "Components to extract specific elements from textual data."
slug: "/extractors-api"
---
<a id="named_entity_extractor"></a>
<a id="image/llm_document_content_extractor"></a>
## Module named\_entity\_extractor
## Module image/llm\_document\_content\_extractor
<a id="named_entity_extractor.NamedEntityExtractorBackend"></a>
<a id="image/llm_document_content_extractor.LLMDocumentContentExtractor"></a>
### NamedEntityExtractorBackend
### LLMDocumentContentExtractor
NLP backend to use for Named Entity Recognition.
Extracts textual content from image-based documents using a vision-enabled LLM (Large Language Model).
<a id="named_entity_extractor.NamedEntityExtractorBackend.HUGGING_FACE"></a>
This component converts each input document into an image using the DocumentToImageContent component,
uses a prompt to instruct the LLM on how to extract content, and uses a ChatGenerator to extract structured
textual content based on the provided prompt.
#### HUGGING\_FACE
The prompt must not contain variables; it should only include instructions for the LLM. Image data and the prompt
are passed together to the LLM as a chat message.
Uses an Hugging Face model and pipeline.
Documents for which the LLM fails to extract content are returned in a separate `failed_documents` list. These
failed documents will have a `content_extraction_error` entry in their metadata. This metadata can be used for
debugging or for reprocessing the documents later.
<a id="named_entity_extractor.NamedEntityExtractorBackend.SPACY"></a>
#### SPACY
Uses a spaCy model and pipeline.
<a id="named_entity_extractor.NamedEntityExtractorBackend.from_str"></a>
#### NamedEntityExtractorBackend.from\_str
```python
@staticmethod
def from_str(string: str) -> "NamedEntityExtractorBackend"
```
Convert a string to a NamedEntityExtractorBackend enum.
<a id="named_entity_extractor.NamedEntityAnnotation"></a>
### NamedEntityAnnotation
Describes a single NER annotation.
**Arguments**:
- `entity`: Entity label.
- `start`: Start index of the entity in the document.
- `end`: End index of the entity in the document.
- `score`: Score calculated by the model.
<a id="named_entity_extractor.NamedEntityExtractor"></a>
### NamedEntityExtractor
Annotates named entities in a collection of documents.
The component supports two backends: Hugging Face and spaCy. The
former can be used with any sequence classification model from the
[Hugging Face model hub](https://huggingface.co/models), while the
latter can be used with any [spaCy model](https://spacy.io/models)
that contains an NER component. Annotations are stored as metadata
in the documents.
Usage example:
### Usage example
```python
from haystack import Document
from haystack.components.extractors.named_entity_extractor import NamedEntityExtractor
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.components.extractors.image import LLMDocumentContentExtractor
chat_generator = OpenAIChatGenerator()
extractor = LLMDocumentContentExtractor(chat_generator=chat_generator)
documents = [
Document(content="I'm Merlin, the happy pig!"),
Document(content="My name is Clara and I live in Berkeley, California."),
Document(content="", meta={"file_path": "image.jpg"}),
Document(content="", meta={"file_path": "document.pdf", "page_number": 1}),
]
extractor = NamedEntityExtractor(backend="hugging_face", model="dslim/bert-base-NER")
extractor.warm_up()
results = extractor.run(documents=documents)["documents"]
annotations = [NamedEntityExtractor.get_stored_annotations(doc) for doc in results]
print(annotations)
updated_documents = extractor.run(documents=documents)["documents"]
print(updated_documents)
# [Document(content='Extracted text from image.jpg',
# meta={'file_path': 'image.jpg'}),
# ...]
```
<a id="named_entity_extractor.NamedEntityExtractor.__init__"></a>
<a id="image/llm_document_content_extractor.LLMDocumentContentExtractor.__init__"></a>
#### NamedEntityExtractor.\_\_init\_\_
#### LLMDocumentContentExtractor.\_\_init\_\_
```python
def __init__(
*,
backend: Union[str, NamedEntityExtractorBackend],
model: str,
pipeline_kwargs: Optional[dict[str, Any]] = None,
device: Optional[ComponentDevice] = None,
token: Optional[Secret] = Secret.from_env_var(["HF_API_TOKEN", "HF_TOKEN"],
strict=False)
) -> None
def __init__(*,
chat_generator: ChatGenerator,
prompt: str = DEFAULT_PROMPT_TEMPLATE,
file_path_meta_field: str = "file_path",
root_path: Optional[str] = None,
detail: Optional[Literal["auto", "high", "low"]] = None,
size: Optional[tuple[int, int]] = None,
raise_on_failure: bool = False,
max_workers: int = 3)
```
Create a Named Entity extractor component.
Initialize the LLMDocumentContentExtractor component.
**Arguments**:
- `backend`: Backend to use for NER.
- `model`: Name of the model or a path to the model on
the local disk. Dependent on the backend.
- `pipeline_kwargs`: Keyword arguments passed to the pipeline. The
pipeline can override these arguments. Dependent on the backend.
- `device`: The device on which the model is loaded. If `None`,
the default device is automatically selected. If a
device/device map is specified in `pipeline_kwargs`,
it overrides this parameter (only applicable to the
HuggingFace backend).
- `token`: The API token to download private models from Hugging Face.
- `chat_generator`: A ChatGenerator instance representing the LLM used to extract text. This generator must
support vision-based input and return a plain text response.
- `prompt`: Instructional text provided to the LLM. It must not contain Jinja variables.
The prompt should only contain instructions on how to extract the content of the image-based document.
- `file_path_meta_field`: The metadata field in the Document that contains the file path to the image or PDF.
- `root_path`: The root directory path where document files are located. If provided, file paths in
document metadata will be resolved relative to this path. If None, file paths are treated as absolute paths.
- `detail`: Optional detail level of the image (only supported by OpenAI). Can be "auto", "high", or "low".
This will be passed to chat_generator when processing the images.
- `size`: If provided, resizes the image to fit within the specified dimensions (width, height) while
maintaining aspect ratio. This reduces file size, memory usage, and processing time, which is beneficial
when working with models that have resolution constraints or when transmitting images to remote services.
- `raise_on_failure`: If True, exceptions from the LLM are raised. If False, failed documents are logged
and returned.
- `max_workers`: Maximum number of threads used to parallelize LLM calls across documents using a
ThreadPoolExecutor.
<a id="named_entity_extractor.NamedEntityExtractor.warm_up"></a>
<a id="image/llm_document_content_extractor.LLMDocumentContentExtractor.warm_up"></a>
#### NamedEntityExtractor.warm\_up
#### LLMDocumentContentExtractor.warm\_up
```python
def warm_up()
```
Initialize the component.
Warm up the ChatGenerator if it has a warm_up method.
**Raises**:
<a id="image/llm_document_content_extractor.LLMDocumentContentExtractor.to_dict"></a>
- `ComponentError`: If the backend fails to initialize successfully.
<a id="named_entity_extractor.NamedEntityExtractor.run"></a>
#### NamedEntityExtractor.run
```python
@component.output_types(documents=list[Document])
def run(documents: list[Document], batch_size: int = 1) -> dict[str, Any]
```
Annotate named entities in each document and store the annotations in the document's metadata.
**Arguments**:
- `documents`: Documents to process.
- `batch_size`: Batch size used for processing the documents.
**Raises**:
- `ComponentError`: If the backend fails to process a document.
**Returns**:
Processed documents.
<a id="named_entity_extractor.NamedEntityExtractor.to_dict"></a>
#### NamedEntityExtractor.to\_dict
#### LLMDocumentContentExtractor.to\_dict
```python
def to_dict() -> dict[str, Any]
@@ -164,55 +105,50 @@ Serializes the component to a dictionary.
Dictionary with serialized data.
<a id="named_entity_extractor.NamedEntityExtractor.from_dict"></a>
<a id="image/llm_document_content_extractor.LLMDocumentContentExtractor.from_dict"></a>
#### NamedEntityExtractor.from\_dict
#### LLMDocumentContentExtractor.from\_dict
```python
@classmethod
def from_dict(cls, data: dict[str, Any]) -> "NamedEntityExtractor"
def from_dict(cls, data: dict[str, Any]) -> "LLMDocumentContentExtractor"
```
Deserializes the component from a dictionary.
**Arguments**:
- `data`: Dictionary to deserialize from.
- `data`: Dictionary with serialized data.
**Returns**:
Deserialized component.
An instance of the component.
<a id="named_entity_extractor.NamedEntityExtractor.initialized"></a>
<a id="image/llm_document_content_extractor.LLMDocumentContentExtractor.run"></a>
#### NamedEntityExtractor.initialized
#### LLMDocumentContentExtractor.run
```python
@property
def initialized() -> bool
@component.output_types(documents=list[Document],
failed_documents=list[Document])
def run(documents: list[Document]) -> dict[str, list[Document]]
```
Returns if the extractor is ready to annotate text.
Run content extraction on a list of image-based documents using a vision-capable LLM.
<a id="named_entity_extractor.NamedEntityExtractor.get_stored_annotations"></a>
#### NamedEntityExtractor.get\_stored\_annotations
```python
@classmethod
def get_stored_annotations(
cls, document: Document) -> Optional[list[NamedEntityAnnotation]]
```
Returns the document's named entity annotations stored in its metadata, if any.
Each document is passed to the LLM along with a predefined prompt. The response is used to update the document's
content. If the extraction fails, the document is returned in the `failed_documents` list with metadata
describing the failure.
**Arguments**:
- `document`: Document whose annotations are to be fetched.
- `documents`: A list of image-based documents to process. Each must have a valid file path in its metadata.
**Returns**:
The stored annotations.
A dictionary with:
- "documents": Successfully processed documents, updated with extracted content.
- "failed_documents": Documents that failed processing, annotated with failure metadata.
<a id="llm_metadata_extractor"></a>
@@ -431,95 +367,154 @@ A dictionary with the keys:
"metadata_extraction_error" and "metadata_extraction_response" in their metadata. These documents can be
re-run with the extractor to extract metadata.
<a id="image/llm_document_content_extractor"></a>
<a id="named_entity_extractor"></a>
## Module image/llm\_document\_content\_extractor
## Module named\_entity\_extractor
<a id="image/llm_document_content_extractor.LLMDocumentContentExtractor"></a>
<a id="named_entity_extractor.NamedEntityExtractorBackend"></a>
### LLMDocumentContentExtractor
### NamedEntityExtractorBackend
Extracts textual content from image-based documents using a vision-enabled LLM (Large Language Model).
NLP backend to use for Named Entity Recognition.
This component converts each input document into an image using the DocumentToImageContent component,
uses a prompt to instruct the LLM on how to extract content, and uses a ChatGenerator to extract structured
textual content based on the provided prompt.
<a id="named_entity_extractor.NamedEntityExtractorBackend.HUGGING_FACE"></a>
The prompt must not contain variables; it should only include instructions for the LLM. Image data and the prompt
are passed together to the LLM as a chat message.
#### HUGGING\_FACE
Documents for which the LLM fails to extract content are returned in a separate `failed_documents` list. These
failed documents will have a `content_extraction_error` entry in their metadata. This metadata can be used for
debugging or for reprocessing the documents later.
Uses an Hugging Face model and pipeline.
### Usage example
```python
from haystack import Document
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.components.extractors.image import LLMDocumentContentExtractor
chat_generator = OpenAIChatGenerator()
extractor = LLMDocumentContentExtractor(chat_generator=chat_generator)
documents = [
Document(content="", meta={"file_path": "image.jpg"}),
Document(content="", meta={"file_path": "document.pdf", "page_number": 1}),
]
updated_documents = extractor.run(documents=documents)["documents"]
print(updated_documents)
# [Document(content='Extracted text from image.jpg',
# meta={'file_path': 'image.jpg'}),
# ...]
```
<a id="named_entity_extractor.NamedEntityExtractorBackend.SPACY"></a>
<a id="image/llm_document_content_extractor.LLMDocumentContentExtractor.__init__"></a>
#### SPACY
#### LLMDocumentContentExtractor.\_\_init\_\_
Uses a spaCy model and pipeline.
<a id="named_entity_extractor.NamedEntityExtractorBackend.from_str"></a>
#### NamedEntityExtractorBackend.from\_str
```python
def __init__(*,
chat_generator: ChatGenerator,
prompt: str = DEFAULT_PROMPT_TEMPLATE,
file_path_meta_field: str = "file_path",
root_path: Optional[str] = None,
detail: Optional[Literal["auto", "high", "low"]] = None,
size: Optional[tuple[int, int]] = None,
raise_on_failure: bool = False,
max_workers: int = 3)
@staticmethod
def from_str(string: str) -> "NamedEntityExtractorBackend"
```
Initialize the LLMDocumentContentExtractor component.
Convert a string to a NamedEntityExtractorBackend enum.
<a id="named_entity_extractor.NamedEntityAnnotation"></a>
### NamedEntityAnnotation
Describes a single NER annotation.
**Arguments**:
- `chat_generator`: A ChatGenerator instance representing the LLM used to extract text. This generator must
support vision-based input and return a plain text response.
- `prompt`: Instructional text provided to the LLM. It must not contain Jinja variables.
The prompt should only contain instructions on how to extract the content of the image-based document.
- `file_path_meta_field`: The metadata field in the Document that contains the file path to the image or PDF.
- `root_path`: The root directory path where document files are located. If provided, file paths in
document metadata will be resolved relative to this path. If None, file paths are treated as absolute paths.
- `detail`: Optional detail level of the image (only supported by OpenAI). Can be "auto", "high", or "low".
This will be passed to chat_generator when processing the images.
- `size`: If provided, resizes the image to fit within the specified dimensions (width, height) while
maintaining aspect ratio. This reduces file size, memory usage, and processing time, which is beneficial
when working with models that have resolution constraints or when transmitting images to remote services.
- `raise_on_failure`: If True, exceptions from the LLM are raised. If False, failed documents are logged
and returned.
- `max_workers`: Maximum number of threads used to parallelize LLM calls across documents using a
ThreadPoolExecutor.
- `entity`: Entity label.
- `start`: Start index of the entity in the document.
- `end`: End index of the entity in the document.
- `score`: Score calculated by the model.
<a id="image/llm_document_content_extractor.LLMDocumentContentExtractor.warm_up"></a>
<a id="named_entity_extractor.NamedEntityExtractor"></a>
#### LLMDocumentContentExtractor.warm\_up
### NamedEntityExtractor
Annotates named entities in a collection of documents.
The component supports two backends: Hugging Face and spaCy. The
former can be used with any sequence classification model from the
[Hugging Face model hub](https://huggingface.co/models), while the
latter can be used with any [spaCy model](https://spacy.io/models)
that contains an NER component. Annotations are stored as metadata
in the documents.
Usage example:
```python
from haystack import Document
from haystack.components.extractors.named_entity_extractor import NamedEntityExtractor
documents = [
Document(content="I'm Merlin, the happy pig!"),
Document(content="My name is Clara and I live in Berkeley, California."),
]
extractor = NamedEntityExtractor(backend="hugging_face", model="dslim/bert-base-NER")
extractor.warm_up()
results = extractor.run(documents=documents)["documents"]
annotations = [NamedEntityExtractor.get_stored_annotations(doc) for doc in results]
print(annotations)
```
<a id="named_entity_extractor.NamedEntityExtractor.__init__"></a>
#### NamedEntityExtractor.\_\_init\_\_
```python
def __init__(
*,
backend: Union[str, NamedEntityExtractorBackend],
model: str,
pipeline_kwargs: Optional[dict[str, Any]] = None,
device: Optional[ComponentDevice] = None,
token: Optional[Secret] = Secret.from_env_var(["HF_API_TOKEN", "HF_TOKEN"],
strict=False)
) -> None
```
Create a Named Entity extractor component.
**Arguments**:
- `backend`: Backend to use for NER.
- `model`: Name of the model or a path to the model on
the local disk. Dependent on the backend.
- `pipeline_kwargs`: Keyword arguments passed to the pipeline. The
pipeline can override these arguments. Dependent on the backend.
- `device`: The device on which the model is loaded. If `None`,
the default device is automatically selected. If a
device/device map is specified in `pipeline_kwargs`,
it overrides this parameter (only applicable to the
HuggingFace backend).
- `token`: The API token to download private models from Hugging Face.
<a id="named_entity_extractor.NamedEntityExtractor.warm_up"></a>
#### NamedEntityExtractor.warm\_up
```python
def warm_up()
```
Warm up the ChatGenerator if it has a warm_up method.
Initialize the component.
<a id="image/llm_document_content_extractor.LLMDocumentContentExtractor.to_dict"></a>
**Raises**:
#### LLMDocumentContentExtractor.to\_dict
- `ComponentError`: If the backend fails to initialize successfully.
<a id="named_entity_extractor.NamedEntityExtractor.run"></a>
#### NamedEntityExtractor.run
```python
@component.output_types(documents=list[Document])
def run(documents: list[Document], batch_size: int = 1) -> dict[str, Any]
```
Annotate named entities in each document and store the annotations in the document's metadata.
**Arguments**:
- `documents`: Documents to process.
- `batch_size`: Batch size used for processing the documents.
**Raises**:
- `ComponentError`: If the backend fails to process a document.
**Returns**:
Processed documents.
<a id="named_entity_extractor.NamedEntityExtractor.to_dict"></a>
#### NamedEntityExtractor.to\_dict
```python
def to_dict() -> dict[str, Any]
@@ -531,50 +526,55 @@ Serializes the component to a dictionary.
Dictionary with serialized data.
<a id="image/llm_document_content_extractor.LLMDocumentContentExtractor.from_dict"></a>
<a id="named_entity_extractor.NamedEntityExtractor.from_dict"></a>
#### LLMDocumentContentExtractor.from\_dict
#### NamedEntityExtractor.from\_dict
```python
@classmethod
def from_dict(cls, data: dict[str, Any]) -> "LLMDocumentContentExtractor"
def from_dict(cls, data: dict[str, Any]) -> "NamedEntityExtractor"
```
Deserializes the component from a dictionary.
**Arguments**:
- `data`: Dictionary with serialized data.
- `data`: Dictionary to deserialize from.
**Returns**:
An instance of the component.
Deserialized component.
<a id="image/llm_document_content_extractor.LLMDocumentContentExtractor.run"></a>
<a id="named_entity_extractor.NamedEntityExtractor.initialized"></a>
#### LLMDocumentContentExtractor.run
#### NamedEntityExtractor.initialized
```python
@component.output_types(documents=list[Document],
failed_documents=list[Document])
def run(documents: list[Document]) -> dict[str, list[Document]]
@property
def initialized() -> bool
```
Run content extraction on a list of image-based documents using a vision-capable LLM.
Returns if the extractor is ready to annotate text.
Each document is passed to the LLM along with a predefined prompt. The response is used to update the document's
content. If the extraction fails, the document is returned in the `failed_documents` list with metadata
describing the failure.
<a id="named_entity_extractor.NamedEntityExtractor.get_stored_annotations"></a>
#### NamedEntityExtractor.get\_stored\_annotations
```python
@classmethod
def get_stored_annotations(
cls, document: Document) -> Optional[list[NamedEntityAnnotation]]
```
Returns the document's named entity annotations stored in its metadata, if any.
**Arguments**:
- `documents`: A list of image-based documents to process. Each must have a valid file path in its metadata.
- `document`: Document whose annotations are to be fetched.
**Returns**:
A dictionary with:
- "documents": Successfully processed documents, updated with extracted content.
- "failed_documents": Documents that failed processing, annotated with failure metadata.
The stored annotations.
<a id="regex_text_extractor"></a>
docs-website/reference/haystack-api/generators_api.md
+922 -922
View File
File diff suppressed because it is too large Load Diff
docs-website/reference/haystack-api/tools_api.md
+268 -268
View File
@@ -5,274 +5,6 @@ description: "Unified abstractions to represent tools across the framework."
slug: "/tools-api"
---
<a id="tool"></a>
## Module tool
<a id="tool.Tool"></a>
### Tool
Data class representing a Tool that Language Models can prepare a call for.
Accurate definitions of the textual attributes such as `name` and `description`
are important for the Language Model to correctly prepare the call.
For resource-intensive operations like establishing connections to remote services or
loading models, override the `warm_up()` method. This method is called before the Tool
is used and should be idempotent, as it may be called multiple times during
pipeline/agent setup.
**Arguments**:
- `name`: Name of the Tool.
- `description`: Description of the Tool.
- `parameters`: A JSON schema defining the parameters expected by the Tool.
- `function`: The function that will be invoked when the Tool is called.
- `outputs_to_string`: Optional dictionary defining how a tool outputs should be converted into a string.
If the source is provided only the specified output key is sent to the handler.
If the source is omitted the whole tool result is sent to the handler.
Example:
```python
{
"source": "docs", "handler": format_documents
}
```
- `inputs_from_state`: Optional dictionary mapping state keys to tool parameter names.
Example: `{"repository": "repo"}` maps state's "repository" to tool's "repo" parameter.
- `outputs_to_state`: Optional dictionary defining how tool outputs map to keys within state as well as optional handlers.
If the source is provided only the specified output key is sent to the handler.
Example:
```python
{
"documents": {"source": "docs", "handler": custom_handler}
}
```
If the source is omitted the whole tool result is sent to the handler.
Example:
```python
{
"documents": {"handler": custom_handler}
}
```
<a id="tool.Tool.tool_spec"></a>
#### Tool.tool\_spec
```python
@property
def tool_spec() -> dict[str, Any]
```
Return the Tool specification to be used by the Language Model.
<a id="tool.Tool.warm_up"></a>
#### Tool.warm\_up
```python
def warm_up() -> None
```
Prepare the Tool for use.
Override this method to establish connections to remote services, load models,
or perform other resource-intensive initialization. This method should be idempotent,
as it may be called multiple times.
<a id="tool.Tool.invoke"></a>
#### Tool.invoke
```python
def invoke(**kwargs: Any) -> Any
```
Invoke the Tool with the provided keyword arguments.
<a id="tool.Tool.to_dict"></a>
#### Tool.to\_dict
```python
def to_dict() -> dict[str, Any]
```
Serializes the Tool to a dictionary.
**Returns**:
Dictionary with serialized data.
<a id="tool.Tool.from_dict"></a>
#### Tool.from\_dict
```python
@classmethod
def from_dict(cls, data: dict[str, Any]) -> "Tool"
```
Deserializes the Tool from a dictionary.
**Arguments**:
- `data`: Dictionary to deserialize from.
**Returns**:
Deserialized Tool.
<a id="from_function"></a>
## Module from\_function
<a id="from_function.create_tool_from_function"></a>
#### create\_tool\_from\_function
```python
def create_tool_from_function(
function: Callable,
name: Optional[str] = None,
description: Optional[str] = None,
inputs_from_state: Optional[dict[str, str]] = None,
outputs_to_state: Optional[dict[str, dict[str,
Any]]] = None) -> "Tool"
```
Create a Tool instance from a function.
Allows customizing the Tool name and description.
For simpler use cases, consider using the `@tool` decorator.
### Usage example
```python
from typing import Annotated, Literal
from haystack.tools import create_tool_from_function
def get_weather(
city: Annotated[str, "the city for which to get the weather"] = "Munich",
unit: Annotated[Literal["Celsius", "Fahrenheit"], "the unit for the temperature"] = "Celsius"):
'''A simple function to get the current weather for a location.'''
return f"Weather report for {city}: 20 {unit}, sunny"
tool = create_tool_from_function(get_weather)
print(tool)
>>> Tool(name='get_weather', description='A simple function to get the current weather for a location.',
>>> parameters={
>>> 'type': 'object',
>>> 'properties': {
>>> 'city': {'type': 'string', 'description': 'the city for which to get the weather', 'default': 'Munich'},
>>> 'unit': {
>>> 'type': 'string',
>>> 'enum': ['Celsius', 'Fahrenheit'],
>>> 'description': 'the unit for the temperature',
>>> 'default': 'Celsius',
>>> },
>>> }
>>> },
>>> function=<function get_weather at 0x7f7b3a8a9b80>)
```
**Arguments**:
- `function`: The function to be converted into a Tool.
The function must include type hints for all parameters.
The function is expected to have basic python input types (str, int, float, bool, list, dict, tuple).
Other input types may work but are not guaranteed.
If a parameter is annotated using `typing.Annotated`, its metadata will be used as parameter description.
- `name`: The name of the Tool. If not provided, the name of the function will be used.
- `description`: The description of the Tool. If not provided, the docstring of the function will be used.
To intentionally leave the description empty, pass an empty string.
- `inputs_from_state`: Optional dictionary mapping state keys to tool parameter names.
Example: `{"repository": "repo"}` maps state's "repository" to tool's "repo" parameter.
- `outputs_to_state`: Optional dictionary defining how tool outputs map to state and message handling.
Example:
```python
{
"documents": {"source": "docs", "handler": custom_handler},
"message": {"source": "summary", "handler": format_summary}
}
```
**Raises**:
- `ValueError`: If any parameter of the function lacks a type hint.
- `SchemaGenerationError`: If there is an error generating the JSON schema for the Tool.
**Returns**:
The Tool created from the function.
<a id="from_function.tool"></a>
#### tool
```python
def tool(
function: Optional[Callable] = None,
*,
name: Optional[str] = None,
description: Optional[str] = None,
inputs_from_state: Optional[dict[str, str]] = None,
outputs_to_state: Optional[dict[str, dict[str, Any]]] = None
) -> Union[Tool, Callable[[Callable], Tool]]
```
Decorator to convert a function into a Tool.
Can be used with or without parameters:
@tool # without parameters
def my_function(): ...
@tool(name="custom_name") # with parameters
def my_function(): ...
### Usage example
```python
from typing import Annotated, Literal
from haystack.tools import tool
@tool
def get_weather(
city: Annotated[str, "the city for which to get the weather"] = "Munich",
unit: Annotated[Literal["Celsius", "Fahrenheit"], "the unit for the temperature"] = "Celsius"):
'''A simple function to get the current weather for a location.'''
return f"Weather report for {city}: 20 {unit}, sunny"
print(get_weather)
>>> Tool(name='get_weather', description='A simple function to get the current weather for a location.',
>>> parameters={
>>> 'type': 'object',
>>> 'properties': {
>>> 'city': {'type': 'string', 'description': 'the city for which to get the weather', 'default': 'Munich'},
>>> 'unit': {
>>> 'type': 'string',
>>> 'enum': ['Celsius', 'Fahrenheit'],
>>> 'description': 'the unit for the temperature',
>>> 'default': 'Celsius',
>>> },
>>> }
>>> },
>>> function=<function get_weather at 0x7f7b3a8a9b80>)
```
**Arguments**:
- `function`: The function to decorate (when used without parameters)
- `name`: Optional custom name for the tool
- `description`: Optional custom description
- `inputs_from_state`: Optional dictionary mapping state keys to tool parameter names
- `outputs_to_state`: Optional dictionary defining how tool outputs map to state and message handling
**Returns**:
Either a Tool instance or a decorator function that will create one
<a id="component_tool"></a>
## Module component\_tool
@@ -450,6 +182,274 @@ def invoke(**kwargs: Any) -> Any
Invoke the Tool with the provided keyword arguments.
<a id="from_function"></a>
## Module from\_function
<a id="from_function.create_tool_from_function"></a>
#### create\_tool\_from\_function
```python
def create_tool_from_function(
function: Callable,
name: Optional[str] = None,
description: Optional[str] = None,
inputs_from_state: Optional[dict[str, str]] = None,
outputs_to_state: Optional[dict[str, dict[str,
Any]]] = None) -> "Tool"
```
Create a Tool instance from a function.
Allows customizing the Tool name and description.
For simpler use cases, consider using the `@tool` decorator.
### Usage example
```python
from typing import Annotated, Literal
from haystack.tools import create_tool_from_function
def get_weather(
city: Annotated[str, "the city for which to get the weather"] = "Munich",
unit: Annotated[Literal["Celsius", "Fahrenheit"], "the unit for the temperature"] = "Celsius"):
'''A simple function to get the current weather for a location.'''
return f"Weather report for {city}: 20 {unit}, sunny"
tool = create_tool_from_function(get_weather)
print(tool)
>>> Tool(name='get_weather', description='A simple function to get the current weather for a location.',
>>> parameters={
>>> 'type': 'object',
>>> 'properties': {
>>> 'city': {'type': 'string', 'description': 'the city for which to get the weather', 'default': 'Munich'},
>>> 'unit': {
>>> 'type': 'string',
>>> 'enum': ['Celsius', 'Fahrenheit'],
>>> 'description': 'the unit for the temperature',
>>> 'default': 'Celsius',
>>> },
>>> }
>>> },
>>> function=<function get_weather at 0x7f7b3a8a9b80>)
```
**Arguments**:
- `function`: The function to be converted into a Tool.
The function must include type hints for all parameters.
The function is expected to have basic python input types (str, int, float, bool, list, dict, tuple).
Other input types may work but are not guaranteed.
If a parameter is annotated using `typing.Annotated`, its metadata will be used as parameter description.
- `name`: The name of the Tool. If not provided, the name of the function will be used.
- `description`: The description of the Tool. If not provided, the docstring of the function will be used.
To intentionally leave the description empty, pass an empty string.
- `inputs_from_state`: Optional dictionary mapping state keys to tool parameter names.
Example: `{"repository": "repo"}` maps state's "repository" to tool's "repo" parameter.
- `outputs_to_state`: Optional dictionary defining how tool outputs map to state and message handling.
Example:
```python
{
"documents": {"source": "docs", "handler": custom_handler},
"message": {"source": "summary", "handler": format_summary}
}
```
**Raises**:
- `ValueError`: If any parameter of the function lacks a type hint.
- `SchemaGenerationError`: If there is an error generating the JSON schema for the Tool.
**Returns**:
The Tool created from the function.
<a id="from_function.tool"></a>
#### tool
```python
def tool(
function: Optional[Callable] = None,
*,
name: Optional[str] = None,
description: Optional[str] = None,
inputs_from_state: Optional[dict[str, str]] = None,
outputs_to_state: Optional[dict[str, dict[str, Any]]] = None
) -> Union[Tool, Callable[[Callable], Tool]]
```
Decorator to convert a function into a Tool.
Can be used with or without parameters:
@tool # without parameters
def my_function(): ...
@tool(name="custom_name") # with parameters
def my_function(): ...
### Usage example
```python
from typing import Annotated, Literal
from haystack.tools import tool
@tool
def get_weather(
city: Annotated[str, "the city for which to get the weather"] = "Munich",
unit: Annotated[Literal["Celsius", "Fahrenheit"], "the unit for the temperature"] = "Celsius"):
'''A simple function to get the current weather for a location.'''
return f"Weather report for {city}: 20 {unit}, sunny"
print(get_weather)
>>> Tool(name='get_weather', description='A simple function to get the current weather for a location.',
>>> parameters={
>>> 'type': 'object',
>>> 'properties': {
>>> 'city': {'type': 'string', 'description': 'the city for which to get the weather', 'default': 'Munich'},
>>> 'unit': {
>>> 'type': 'string',
>>> 'enum': ['Celsius', 'Fahrenheit'],
>>> 'description': 'the unit for the temperature',
>>> 'default': 'Celsius',
>>> },
>>> }
>>> },
>>> function=<function get_weather at 0x7f7b3a8a9b80>)
```
**Arguments**:
- `function`: The function to decorate (when used without parameters)
- `name`: Optional custom name for the tool
- `description`: Optional custom description
- `inputs_from_state`: Optional dictionary mapping state keys to tool parameter names
- `outputs_to_state`: Optional dictionary defining how tool outputs map to state and message handling
**Returns**:
Either a Tool instance or a decorator function that will create one
<a id="tool"></a>
## Module tool
<a id="tool.Tool"></a>
### Tool
Data class representing a Tool that Language Models can prepare a call for.
Accurate definitions of the textual attributes such as `name` and `description`
are important for the Language Model to correctly prepare the call.
For resource-intensive operations like establishing connections to remote services or
loading models, override the `warm_up()` method. This method is called before the Tool
is used and should be idempotent, as it may be called multiple times during
pipeline/agent setup.
**Arguments**:
- `name`: Name of the Tool.
- `description`: Description of the Tool.
- `parameters`: A JSON schema defining the parameters expected by the Tool.
- `function`: The function that will be invoked when the Tool is called.
- `outputs_to_string`: Optional dictionary defining how a tool outputs should be converted into a string.
If the source is provided only the specified output key is sent to the handler.
If the source is omitted the whole tool result is sent to the handler.
Example:
```python
{
"source": "docs", "handler": format_documents
}
```
- `inputs_from_state`: Optional dictionary mapping state keys to tool parameter names.
Example: `{"repository": "repo"}` maps state's "repository" to tool's "repo" parameter.
- `outputs_to_state`: Optional dictionary defining how tool outputs map to keys within state as well as optional handlers.
If the source is provided only the specified output key is sent to the handler.
Example:
```python
{
"documents": {"source": "docs", "handler": custom_handler}
}
```
If the source is omitted the whole tool result is sent to the handler.
Example:
```python
{
"documents": {"handler": custom_handler}
}
```
<a id="tool.Tool.tool_spec"></a>
#### Tool.tool\_spec
```python
@property
def tool_spec() -> dict[str, Any]
```
Return the Tool specification to be used by the Language Model.
<a id="tool.Tool.warm_up"></a>
#### Tool.warm\_up
```python
def warm_up() -> None
```
Prepare the Tool for use.
Override this method to establish connections to remote services, load models,
or perform other resource-intensive initialization. This method should be idempotent,
as it may be called multiple times.
<a id="tool.Tool.invoke"></a>
#### Tool.invoke
```python
def invoke(**kwargs: Any) -> Any
```
Invoke the Tool with the provided keyword arguments.
<a id="tool.Tool.to_dict"></a>
#### Tool.to\_dict
```python
def to_dict() -> dict[str, Any]
```
Serializes the Tool to a dictionary.
**Returns**:
Dictionary with serialized data.
<a id="tool.Tool.from_dict"></a>
#### Tool.from\_dict
```python
@classmethod
def from_dict(cls, data: dict[str, Any]) -> "Tool"
```
Deserializes the Tool from a dictionary.
**Arguments**:
- `data`: Dictionary to deserialize from.
**Returns**:
Deserialized Tool.
<a id="toolset"></a>
## Module toolset
docs-website/reference/haystack-api/utils_api.md
+562 -562
View File
File diff suppressed because it is too large Load Diff
docs-website/reference/haystack-api/websearch_api.md
+106 -106
View File
@@ -5,6 +5,112 @@ description: "Web search engine for Haystack."
slug: "/websearch-api"
---
<a id="searchapi"></a>
## Module searchapi
<a id="searchapi.SearchApiWebSearch"></a>
### SearchApiWebSearch
Uses [SearchApi](https://www.searchapi.io/) to search the web for relevant documents.
Usage example:
```python
from haystack.components.websearch import SearchApiWebSearch
from haystack.utils import Secret
websearch = SearchApiWebSearch(top_k=10, api_key=Secret.from_token("test-api-key"))
results = websearch.run(query="Who is the boyfriend of Olivia Wilde?")
assert results["documents"]
assert results["links"]
```
<a id="searchapi.SearchApiWebSearch.__init__"></a>
#### SearchApiWebSearch.\_\_init\_\_
```python
def __init__(api_key: Secret = Secret.from_env_var("SEARCHAPI_API_KEY"),
top_k: Optional[int] = 10,
allowed_domains: Optional[list[str]] = None,
search_params: Optional[dict[str, Any]] = None)
```
Initialize the SearchApiWebSearch component.
**Arguments**:
- `api_key`: API key for the SearchApi API
- `top_k`: Number of documents to return.
- `allowed_domains`: List of domains to limit the search to.
- `search_params`: Additional parameters passed to the SearchApi API.
For example, you can set 'num' to 100 to increase the number of search results.
See the [SearchApi website](https://www.searchapi.io/) for more details.
The default search engine is Google, however, users can change it by setting the `engine`
parameter in the `search_params`.
<a id="searchapi.SearchApiWebSearch.to_dict"></a>
#### SearchApiWebSearch.to\_dict
```python
def to_dict() -> dict[str, Any]
```
Serializes the component to a dictionary.
**Returns**:
Dictionary with serialized data.
<a id="searchapi.SearchApiWebSearch.from_dict"></a>
#### SearchApiWebSearch.from\_dict
```python
@classmethod
def from_dict(cls, data: dict[str, Any]) -> "SearchApiWebSearch"
```
Deserializes the component from a dictionary.
**Arguments**:
- `data`: The dictionary to deserialize from.
**Returns**:
The deserialized component.
<a id="searchapi.SearchApiWebSearch.run"></a>
#### SearchApiWebSearch.run
```python
@component.output_types(documents=list[Document], links=list[str])
def run(query: str) -> dict[str, Union[list[Document], list[str]]]
```
Uses [SearchApi](https://www.searchapi.io/) to search the web.
**Arguments**:
- `query`: Search query.
**Raises**:
- `TimeoutError`: If the request to the SearchApi API times out.
- `SearchApiError`: If an error occurs while querying the SearchApi API.
**Returns**:
A dictionary with the following keys:
- "documents": List of documents returned by the search engine.
- "links": List of links returned by the search engine.
<a id="serper_dev"></a>
## Module serper\_dev
@@ -120,109 +226,3 @@ A dictionary with the following keys:
- "documents": List of documents returned by the search engine.
- "links": List of links returned by the search engine.
<a id="searchapi"></a>
## Module searchapi
<a id="searchapi.SearchApiWebSearch"></a>
### SearchApiWebSearch
Uses [SearchApi](https://www.searchapi.io/) to search the web for relevant documents.
Usage example:
```python
from haystack.components.websearch import SearchApiWebSearch
from haystack.utils import Secret
websearch = SearchApiWebSearch(top_k=10, api_key=Secret.from_token("test-api-key"))
results = websearch.run(query="Who is the boyfriend of Olivia Wilde?")
assert results["documents"]
assert results["links"]
```
<a id="searchapi.SearchApiWebSearch.__init__"></a>
#### SearchApiWebSearch.\_\_init\_\_
```python
def __init__(api_key: Secret = Secret.from_env_var("SEARCHAPI_API_KEY"),
top_k: Optional[int] = 10,
allowed_domains: Optional[list[str]] = None,
search_params: Optional[dict[str, Any]] = None)
```
Initialize the SearchApiWebSearch component.
**Arguments**:
- `api_key`: API key for the SearchApi API
- `top_k`: Number of documents to return.
- `allowed_domains`: List of domains to limit the search to.
- `search_params`: Additional parameters passed to the SearchApi API.
For example, you can set 'num' to 100 to increase the number of search results.
See the [SearchApi website](https://www.searchapi.io/) for more details.
The default search engine is Google, however, users can change it by setting the `engine`
parameter in the `search_params`.
<a id="searchapi.SearchApiWebSearch.to_dict"></a>
#### SearchApiWebSearch.to\_dict
```python
def to_dict() -> dict[str, Any]
```
Serializes the component to a dictionary.
**Returns**:
Dictionary with serialized data.
<a id="searchapi.SearchApiWebSearch.from_dict"></a>
#### SearchApiWebSearch.from\_dict
```python
@classmethod
def from_dict(cls, data: dict[str, Any]) -> "SearchApiWebSearch"
```
Deserializes the component from a dictionary.
**Arguments**:
- `data`: The dictionary to deserialize from.
**Returns**:
The deserialized component.
<a id="searchapi.SearchApiWebSearch.run"></a>
#### SearchApiWebSearch.run
```python
@component.output_types(documents=list[Document], links=list[str])
def run(query: str) -> dict[str, Union[list[Document], list[str]]]
```
Uses [SearchApi](https://www.searchapi.io/) to search the web.
**Arguments**:
- `query`: Search query.
**Raises**:
- `TimeoutError`: If the request to the SearchApi API times out.
- `SearchApiError`: If an error occurs while querying the SearchApi API.
**Returns**:
A dictionary with the following keys:
- "documents": List of documents returned by the search engine.
- "links": List of links returned by the search engine.