mirror of
https://github.com/deepset-ai/haystack.git
synced 2026-10-12 02:12:50 +00:00
chore: Merge main into v3 (#11573)
Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: NIK-TIGER-BILL <[email protected]> Co-authored-by: Sebastian Husch Lee <[email protected]> Co-authored-by: Pragnyan Ramtha <[email protected]> Co-authored-by: Julian Risch <[email protected]> Co-authored-by: Claude Sonnet 4.6 <[email protected]> Co-authored-by: Aarkin Karnik <[email protected]> Co-authored-by: Haystack Bot <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: davidsbatista <[email protected]> Co-authored-by: bogdankostic <[email protected]> Co-authored-by: bogdankostic <[email protected]> Co-authored-by: Sachin Yadav <[email protected]> Co-authored-by: Copilot Autofix powered by AI <[email protected]> Co-authored-by: Abdulhamid Onawole <[email protected]> Co-authored-by: David S. Batista <[email protected]> Co-authored-by: Aditya Raut <[email protected]> Co-authored-by: Nhat Huy Vu <[email protected]> Co-authored-by: julian-risch <[email protected]> Co-authored-by: kota <[email protected]> Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> Co-authored-by: devteamaegis <[email protected]> Co-authored-by: Ritwij Aryan Parmar <[email protected]> Co-authored-by: Srivatsa Kamballa <[email protected]> Co-authored-by: vidigoat <[email protected]> Co-authored-by: Syed Shahmeer Ali <[email protected]> Co-authored-by: Mohamed Arbi <[email protected]> Co-authored-by: Zamuldinov Nikita <[email protected]> Co-authored-by: Bilge Yücel <[email protected]> Co-authored-by: jdoughty04 <[email protected]> Co-authored-by: Govindh Kishore <[email protected]>
This commit is contained in:
Stefano Fiorucci
co-authored by
Sebastian Husch Lee
Pragnyan Ramtha
Julian Risch
Claude Sonnet 4.6
Aarkin Karnik
Haystack Bot
dependabot[bot]
davidsbatista
bogdankostic
bogdankostic
Sachin Yadav
Copilot Autofix powered by AI
Abdulhamid Onawole
David S. Batista
Aditya Raut
Nhat Huy Vu
julian-risch
kota
Copilot Autofix powered by AI
devteamaegis
Ritwij Aryan Parmar
Srivatsa Kamballa
vidigoat
Syed Shahmeer Ali
Mohamed Arbi
Zamuldinov Nikita
Bilge Yücel
jdoughty04
Govindh Kishore
Sebastian Husch Lee
Pragnyan Ramtha
Julian Risch
Claude Sonnet 4.6
Aarkin Karnik
Haystack Bot
dependabot[bot]
davidsbatista
bogdankostic
bogdankostic
Sachin Yadav
Copilot Autofix powered by AI
Abdulhamid Onawole
David S. Batista
Aditya Raut
Nhat Huy Vu
julian-risch
kota
Copilot Autofix powered by AI
devteamaegis
Ritwij Aryan Parmar
Srivatsa Kamballa
vidigoat
Syed Shahmeer Ali
Mohamed Arbi
Zamuldinov Nikita
Bilge Yücel
jdoughty04
Govindh Kishore
parent
1771a300f3
commit
f4c02eb523
@@ -35,7 +35,7 @@ jobs:
|
||||
|
||||
- name: Run Claude Code Review
|
||||
id: claude-review
|
||||
uses: anthropics/claude-code-action@fbda2eb1bdc90d319b8d853f5deb53bca199a7c1 # v1.0.140
|
||||
uses: anthropics/claude-code-action@593d7a5c4e0073569f74772c2b7b64c30ec14707 # v1.0.141
|
||||
with:
|
||||
anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY_PR_REVIEWS_PUBLIC_REPOS }}
|
||||
plugin_marketplaces: 'https://github.com/anthropics/claude-code.git'
|
||||
|
||||
@@ -56,7 +56,7 @@ jobs:
|
||||
|
||||
- name: Run Claude Code
|
||||
id: claude
|
||||
uses: anthropics/claude-code-action@fbda2eb1bdc90d319b8d853f5deb53bca199a7c1 # v1.0.140
|
||||
uses: anthropics/claude-code-action@593d7a5c4e0073569f74772c2b7b64c30ec14707 # v1.0.141
|
||||
with:
|
||||
anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY_PR_REVIEWS_PUBLIC_REPOS }}
|
||||
additional_permissions: |
|
||||
|
||||
@@ -35,6 +35,13 @@ repos:
|
||||
- ruff
|
||||
- add-trailing-comma
|
||||
|
||||
- id: release-note-backticks
|
||||
name: release-note-backticks
|
||||
language: python
|
||||
entry: python scripts/release_note_backticks.py
|
||||
files: ^releasenotes/notes/.*\.yaml$
|
||||
types: [text]
|
||||
|
||||
- repo: https://github.com/codespell-project/codespell
|
||||
rev: v2.4.1
|
||||
hooks:
|
||||
|
||||
+17
-7
@@ -17,9 +17,9 @@ Classifies the documents based on the provided labels and adds them to their met
|
||||
| **Mandatory init variables** | `model`: The name or path of a Hugging Face model for zero shot document classification <br /> <br />`labels`: The set of possible class labels to classify each document into, for example, [`positive`, `negative`]. The labels depend on the selected model. |
|
||||
| **Mandatory run variables** | `documents`: A list of documents to classify |
|
||||
| **Output variables** | `documents`: A list of processed documents with an added `classification` metadata field |
|
||||
| **API reference** | [Classifiers](/reference/classifiers-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/classifiers/zero_shot_document_classifier.py |
|
||||
| **Package name** | `haystack-ai` |
|
||||
| **API reference** | [Transformers](/reference/integrations-transformers) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/transformers |
|
||||
| **Package name** | `transformers-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
@@ -42,11 +42,19 @@ Available models for the task of zero-shot-classification are:
|
||||
|
||||
## Usage
|
||||
|
||||
Install the `transformers-haystack` package to use the `TransformersZeroShotDocumentClassifier`:
|
||||
|
||||
```shell
|
||||
pip install transformers-haystack
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack.components.classifiers import TransformersZeroShotDocumentClassifier
|
||||
from haystack_integrations.components.classifiers.transformers import (
|
||||
TransformersZeroShotDocumentClassifier,
|
||||
)
|
||||
|
||||
documents = [
|
||||
Document(id="0", content="Cats don't get teeth cavities."),
|
||||
@@ -71,7 +79,9 @@ from haystack import Document
|
||||
from haystack.components.retrievers.in_memory import InMemoryBM25Retriever
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.core.pipeline import Pipeline
|
||||
from haystack.components.classifiers import TransformersZeroShotDocumentClassifier
|
||||
from haystack_integrations.components.classifiers.transformers import (
|
||||
TransformersZeroShotDocumentClassifier,
|
||||
)
|
||||
|
||||
documents = [
|
||||
Document(id="0", content="Today was a nice day!"),
|
||||
@@ -88,8 +98,8 @@ document_classifier = TransformersZeroShotDocumentClassifier(
|
||||
document_store.write_documents(documents)
|
||||
|
||||
pipeline = Pipeline()
|
||||
pipeline.add_component(retriever, name="retriever")
|
||||
pipeline.add_component(document_classifier, name="document_classifier")
|
||||
pipeline.add_component(name="retriever", instance=retriever)
|
||||
pipeline.add_component(name="document_classifier", instance=document_classifier)
|
||||
pipeline.connect("retriever", "document_classifier")
|
||||
|
||||
queries = ["How was your day today?", "How was your day yesterday?"]
|
||||
|
||||
@@ -9,6 +9,12 @@ description: "This component extracts predefined entities out of a piece of text
|
||||
|
||||
This component extracts predefined entities out of a piece of text and writes them into documents’ meta field.
|
||||
|
||||
:::warning[Deprecated]
|
||||
|
||||
`NamedEntityExtractor` is deprecated and will be removed in Haystack 3.0. It has moved to the `transformers-haystack` package and was renamed to `TransformersNamedEntityExtractor`. See [TransformersNamedEntityExtractor](transformersnamedentityextractor.mdx) for the updated documentation.
|
||||
|
||||
:::
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
|
||||
+102
@@ -0,0 +1,102 @@
|
||||
---
|
||||
title: "TransformersNamedEntityExtractor"
|
||||
id: transformersnamedentityextractor
|
||||
slug: "/transformersnamedentityextractor"
|
||||
description: "This component extracts predefined entities out of a piece of text and writes them into documents’ meta field."
|
||||
---
|
||||
|
||||
# TransformersNamedEntityExtractor
|
||||
|
||||
This component extracts predefined entities out of a piece of text and writes them into documents’ meta field.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | After the [PreProcessor](../preprocessors.mdx) in an indexing pipeline or after a [Retriever](../retrievers.mdx) in a query pipeline |
|
||||
| **Mandatory init variables** | `model`: Name or path of the model to use |
|
||||
| **Mandatory run variables** | `documents`: A list of documents |
|
||||
| **Output variables** | `documents`: A list of documents |
|
||||
| **API reference** | [Transformers](/reference/integrations-transformers) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/transformers |
|
||||
| **Package name** | `transformers-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`TransformersNamedEntityExtractor` looks for entities, which are spans in the text. The extractor automatically recognizes and groups them depending on their class, such as people's names, organizations, locations, and other types. The exact classes are determined by the model that you initialize the component with.
|
||||
|
||||
`TransformersNamedEntityExtractor` takes a list of documents as input and returns a list of the same documents with their `meta` data enriched with `NamedEntityAnnotations`. A `NamedEntityAnnotation` consists of the type of the entity, the start and end of the span, and a score calculated by the model, for example: `NamedEntityAnnotation(entity='PER', start=11, end=16, score=0.9)`.
|
||||
|
||||
When the `TransformersNamedEntityExtractor` is initialized, you need to set a `model`. Optionally, you can set `pipeline_kwargs`, which are then passed on to the Hugging Face pipeline. You can additionally set the `device` that is used to run the component.
|
||||
|
||||
Authentication with a Hugging Face API token is only required to access private or gated models. You can pass the token at initialization with `token`, or set the `HF_API_TOKEN` or `HF_TOKEN` environment variable.
|
||||
|
||||
## Usage
|
||||
|
||||
Install the `transformers-haystack` package to use the `TransformersNamedEntityExtractor`:
|
||||
|
||||
```shell
|
||||
pip install transformers-haystack
|
||||
```
|
||||
|
||||
The component works with any Hugging Face model that supports token classification or NER.
|
||||
|
||||
`TransformersNamedEntityExtractor` accepts a list of `Documents` as its input. The extractor annotates the raw text in the documents and stores the annotations in the document's `meta` dictionary under the `named_entities` key.
|
||||
|
||||
```python
|
||||
from haystack.dataclasses import Document
|
||||
from haystack_integrations.components.extractors.transformers import (
|
||||
TransformersNamedEntityExtractor,
|
||||
)
|
||||
|
||||
extractor = TransformersNamedEntityExtractor(model="dslim/bert-base-NER")
|
||||
|
||||
documents = [
|
||||
Document(content="My name is Clara and I live in Berkeley, California."),
|
||||
Document(content="I'm Merlin, the happy pig!"),
|
||||
Document(content="New York State is home to the Empire State Building."),
|
||||
]
|
||||
|
||||
extractor.run(documents)
|
||||
print(documents)
|
||||
```
|
||||
|
||||
Here is the example result:
|
||||
|
||||
```python
|
||||
[Document(id=aec840d1b6c85609f4f16c3e222a5a25fd8c4c53bd981a40c1268ab9c72cee10, content: 'My name is Clara and I live in Berkeley, California.', meta: {'named_entities': [NamedEntityAnnotation(entity='PER', start=11, end=16, score=0.99641764), NamedEntityAnnotation(entity='LOC', start=31, end=39, score=0.996198), NamedEntityAnnotation(entity='LOC', start=41, end=51, score=0.9990196)]}),
|
||||
Document(id=98f1dc5d0ccd9d9950cd191d1076db0f7af40c401dd7608f11c90cb3fc38c0c2, content: 'I'm Merlin, the happy pig!', meta: {'named_entities': [NamedEntityAnnotation(entity='PER', start=4, end=10, score=0.99054915)]}),
|
||||
Document(id=44948ea0eec018b33aceaaedde4616eb9e93ce075e0090ec1613fc145f84b4a9, content: 'New York State is home to the Empire State Building.', meta: {'named_entities': [NamedEntityAnnotation(entity='LOC', start=0, end=14, score=0.9989541), NamedEntityAnnotation(entity='LOC', start=30, end=51, score=0.95746297)]})]
|
||||
```
|
||||
|
||||
### Get stored annotations
|
||||
|
||||
This component includes the `get_stored_annotations` helper class method that allows you to retrieve the annotations stored in a `Document` transparently:
|
||||
|
||||
```python
|
||||
from haystack.dataclasses import Document
|
||||
from haystack_integrations.components.extractors.transformers import (
|
||||
TransformersNamedEntityExtractor,
|
||||
)
|
||||
|
||||
extractor = TransformersNamedEntityExtractor(model="dslim/bert-base-NER")
|
||||
|
||||
documents = [
|
||||
Document(content="My name is Clara and I live in Berkeley, California."),
|
||||
Document(content="I'm Merlin, the happy pig!"),
|
||||
Document(content="New York State is home to the Empire State Building."),
|
||||
]
|
||||
|
||||
extractor.run(documents)
|
||||
|
||||
annotations = [
|
||||
TransformersNamedEntityExtractor.get_stored_annotations(doc) for doc in documents
|
||||
]
|
||||
print(annotations)
|
||||
|
||||
# If a Document doesn't contain any annotations, this returns None.
|
||||
new_doc = Document(content="In one of many possible worlds...")
|
||||
assert TransformersNamedEntityExtractor.get_stored_annotations(new_doc) is None
|
||||
```
|
||||
@@ -43,6 +43,18 @@ client = AzureOpenAIChatGenerator(
|
||||
We recommend using environment variables instead of initialization parameters.
|
||||
:::
|
||||
|
||||
To switch `azure_endpoint` and `api_version` between environments without editing your pipeline, pass a Secret that resolves them from environment variables at runtime:
|
||||
|
||||
```python
|
||||
from haystack.components.generators.chat import AzureOpenAIChatGenerator
|
||||
from haystack.utils import Secret
|
||||
|
||||
client = AzureOpenAIChatGenerator(
|
||||
azure_endpoint=Secret.from_env_var("AZURE_OPENAI_ENDPOINT"),
|
||||
api_version=Secret.from_env_var("AZURE_OPENAI_API_VERSION"),
|
||||
)
|
||||
```
|
||||
|
||||
Then, the component needs a list of `ChatMessage` objects to operate. `ChatMessage` is a data class that contains a message, a role (who generated the message, such as `user`, `assistant`, `system`, `function`), and optional metadata. If a string is passed, it is converted into a list containing a single `ChatMessage` with the `user` role. See the [usage](#usage) section for an example.
|
||||
|
||||
You can pass any chat completion parameters that are valid for the `openai.ChatCompletion.create` method directly to `AzureOpenAIChatGenerator` using the `generation_kwargs` parameter, both at initialization and to `run()` method. For more details on the supported parameters, refer to the [Azure documentation](https://learn.microsoft.com/en-us/azure/ai-services/openai/reference).
|
||||
|
||||
@@ -9,6 +9,12 @@ description: "Provides an interface for chat completion using a Hugging Face mod
|
||||
|
||||
Provides an interface for chat completion using a Hugging Face model that runs locally.
|
||||
|
||||
:::warning[Deprecated]
|
||||
|
||||
`HuggingFaceLocalChatGenerator` is deprecated and will be removed in Haystack 3.0. It has moved to the `transformers-haystack` package and was renamed to `TransformersChatGenerator`. See [TransformersChatGenerator](transformerschatgenerator.mdx) for the updated documentation.
|
||||
|
||||
:::
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
|
||||
@@ -0,0 +1,101 @@
|
||||
---
|
||||
title: "TransformersChatGenerator"
|
||||
id: transformerschatgenerator
|
||||
slug: "/transformerschatgenerator"
|
||||
description: "Provides an interface for chat completion using a Hugging Face model that runs locally."
|
||||
---
|
||||
|
||||
# TransformersChatGenerator
|
||||
|
||||
Provides an interface for chat completion using a Hugging Face model that runs locally.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | After a [ChatPromptBuilder](../builders/chatpromptbuilder.mdx) |
|
||||
| **Mandatory init variables** | None |
|
||||
| **Mandatory run variables** | `messages`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects representing the chat or a plain string |
|
||||
| **Output variables** | `replies`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects generated by the LLM |
|
||||
| **API reference** | [Transformers](/reference/integrations-transformers) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/transformers |
|
||||
| **Package name** | `transformers-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
Keep in mind that if LLMs run locally, you may need a powerful machine to run them. This depends strongly on the model you select and its parameter count.
|
||||
|
||||
If a string is passed to `messages`, it is converted into a list containing a single `ChatMessage` with the `user` role.
|
||||
|
||||
Authentication with a Hugging Face API token is only required to access private or gated models. You can pass the token at initialization with `token`, or set the `HF_API_TOKEN` or `HF_TOKEN` environment variable:
|
||||
|
||||
```python
|
||||
generator = TransformersChatGenerator(
|
||||
token=Secret.from_token("<your-api-key>"),
|
||||
)
|
||||
```
|
||||
|
||||
### Streaming
|
||||
|
||||
This Generator supports [streaming](guides-to-generators/choosing-the-right-generator.mdx#streaming-support) the tokens from the LLM directly in output. To do so, pass a function to the `streaming_callback` init parameter.
|
||||
|
||||
## Usage
|
||||
|
||||
Install the `transformers-haystack` package to use the `TransformersChatGenerator`:
|
||||
|
||||
```shell
|
||||
pip install transformers-haystack
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.generators.transformers import (
|
||||
TransformersChatGenerator,
|
||||
)
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
generator = TransformersChatGenerator(model="Qwen/Qwen3-0.6B")
|
||||
messages = [ChatMessage.from_user("What's Natural Language Processing? Be brief.")]
|
||||
print(generator.run(messages))
|
||||
```
|
||||
|
||||
### In a Pipeline
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.components.builders.prompt_builder import ChatPromptBuilder
|
||||
from haystack_integrations.components.generators.transformers import (
|
||||
TransformersChatGenerator,
|
||||
)
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack.utils import Secret
|
||||
|
||||
prompt_builder = ChatPromptBuilder()
|
||||
llm = TransformersChatGenerator(
|
||||
model="Qwen/Qwen3-0.6B",
|
||||
token=Secret.from_env_var("HF_API_TOKEN"),
|
||||
)
|
||||
|
||||
pipe = Pipeline()
|
||||
pipe.add_component("prompt_builder", prompt_builder)
|
||||
pipe.add_component("llm", llm)
|
||||
pipe.connect("prompt_builder.prompt", "llm.messages")
|
||||
location = "Berlin"
|
||||
messages = [
|
||||
ChatMessage.from_system(
|
||||
"Always respond in German even if some input data is in other languages.",
|
||||
),
|
||||
ChatMessage.from_user("Tell me about {{location}}"),
|
||||
]
|
||||
pipe.run(
|
||||
data={
|
||||
"prompt_builder": {
|
||||
"template_variables": {"location": location},
|
||||
"template": messages,
|
||||
},
|
||||
},
|
||||
)
|
||||
```
|
||||
@@ -9,6 +9,12 @@ description: "Use this component in extractive question answering pipelines base
|
||||
|
||||
Use this component in extractive question answering pipelines based on a query and a list of documents.
|
||||
|
||||
:::warning[Deprecated]
|
||||
|
||||
`ExtractiveReader` is deprecated and will be removed in Haystack 3.0. It has moved to the `transformers-haystack` package and was renamed to `TransformersExtractiveReader`. See [TransformersExtractiveReader](transformersextractivereader.mdx) for the updated documentation.
|
||||
|
||||
:::
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
|
||||
@@ -0,0 +1,119 @@
|
||||
---
|
||||
title: "TransformersExtractiveReader"
|
||||
id: transformersextractivereader
|
||||
slug: "/transformersextractivereader"
|
||||
description: "Use this component in extractive question answering pipelines based on a query and a list of documents."
|
||||
---
|
||||
|
||||
# TransformersExtractiveReader
|
||||
|
||||
Use this component in extractive question answering pipelines based on a query and a list of documents.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | In query pipelines, after a component that returns a list of documents, such as a [Retriever](../retrievers.mdx) |
|
||||
| **Mandatory init variables** | None |
|
||||
| **Mandatory run variables** | `documents`: A list of documents <br /> <br />`query`: A query string |
|
||||
| **Output variables** | `answers`: A list of [`ExtractedAnswer`](../../concepts/data-classes.mdx#extractedanswer) objects |
|
||||
| **API reference** | [Transformers](/reference/integrations-transformers) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/transformers |
|
||||
| **Package name** | `transformers-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`TransformersExtractiveReader` locates and extracts answers to a given query from the document text. It's used in extractive QA systems where you want to know exactly where the answer is located within the document. It's usually coupled with a Retriever that precedes it, but you can also use it with other components that fetch documents.
|
||||
|
||||
Readers assign a _probability_ to answers. This score ranges from 0 to 1, indicating how well the results the Reader returned match the query. Probability closest to 1 means the model has high confidence in the answer's relevance. The Reader sorts the answers based on their probability scores, with higher probability listed first. You can limit the number of answers the Reader returns in the optional `top_k` parameter.
|
||||
|
||||
You can use the probability to set the quality expectations for your system. To do that, use the `confidence_score` parameter of the Reader to set a minimum probability threshold for answers. For example, setting `confidence_threshold` to `0.7` means only answers with a probability higher than 0.7 will be returned.
|
||||
|
||||
By default, the Reader includes a scenario where no answer to the query is found in the document text (`no_answer=True`). In this case, it returns an additional `ExtractedAnswer` with no text and the probability that none of the `top_k` answers are correct. For example, if `top_k=4` the system will return four answers and an additional empty one. Each answer has a probability assigned. If the empty answer has a probability of 0.5, it means that's the probability that none of the returned answers is correct. To receive only the actual top_k answers, set the `no_answer` parameter to `False` when initializing the component.
|
||||
|
||||
### Models
|
||||
|
||||
Here are the models that we recommend for using with `TransformersExtractiveReader`:
|
||||
|
||||
| | | |
|
||||
| --- | --- | --- |
|
||||
| Model URL | Description | Language |
|
||||
| [deepset/roberta-base-squad2-distilled](https://huggingface.co/deepset/roberta-base-squad2-distilled) (default) | A distilled model, relatively fast and with good performance. | English |
|
||||
| [deepset/roberta-large-squad2](https://huggingface.co/deepset/roberta-large-squad2) | A large model with good performance. Slower than the distilled one. | English |
|
||||
| [deepset/tinyroberta-squad2](https://huggingface.co/deepset/tinyroberta-squad2) | A distilled version of roberta-large-squad2 model, very fast. | English |
|
||||
| [deepset/xlm-roberta-base-squad2](https://huggingface.co/deepset/xlm-roberta-base-squad2) | A base multilingual model with good speed and performance. | Multilingual |
|
||||
|
||||
You can also view other question answering models on [Hugging Face](https://huggingface.co/models?pipeline_tag=question-answering).
|
||||
|
||||
Authentication with a Hugging Face API token is only required to access private or gated models. You can pass the token at initialization with `token`, or set the `HF_API_TOKEN` or `HF_TOKEN` environment variable.
|
||||
|
||||
## Usage
|
||||
|
||||
Install the `transformers-haystack` package to use the `TransformersExtractiveReader`:
|
||||
|
||||
```shell
|
||||
pip install transformers-haystack
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
Below is an example that uses the `TransformersExtractiveReader` outside of a pipeline. The Reader gets the query and the documents at runtime. It should return two answers and an additional third answer with no text and the probability that the `top_k` answers are incorrect.
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack_integrations.components.readers.transformers import (
|
||||
TransformersExtractiveReader,
|
||||
)
|
||||
|
||||
docs = [
|
||||
Document(content="Paris is the capital of France."),
|
||||
Document(content="Berlin is the capital of Germany."),
|
||||
]
|
||||
|
||||
reader = TransformersExtractiveReader()
|
||||
|
||||
reader.run(query="What is the capital of France?", documents=docs, top_k=2)
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
Below is an example of a pipeline that retrieves a document from an `InMemoryDocumentStore` based on keyword search (using `InMemoryBM25Retriever`). It then uses the `TransformersExtractiveReader` to extract the answer to our query from the top retrieved documents.
|
||||
|
||||
With the TransformersExtractiveReader’s `top_k` set to 2, an additional, third answer with no text and the probability that the other `top_k` answers are incorrect is also returned.
|
||||
|
||||
```python
|
||||
from haystack import Document, Pipeline
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.components.retrievers.in_memory import InMemoryBM25Retriever
|
||||
from haystack_integrations.components.readers.transformers import (
|
||||
TransformersExtractiveReader,
|
||||
)
|
||||
|
||||
docs = [
|
||||
Document(content="Paris is the capital of France."),
|
||||
Document(content="Berlin is the capital of Germany."),
|
||||
Document(content="Rome is the capital of Italy."),
|
||||
Document(content="Madrid is the capital of Spain."),
|
||||
]
|
||||
document_store = InMemoryDocumentStore()
|
||||
document_store.write_documents(docs)
|
||||
|
||||
retriever = InMemoryBM25Retriever(document_store=document_store)
|
||||
reader = TransformersExtractiveReader()
|
||||
|
||||
extractive_qa_pipeline = Pipeline()
|
||||
extractive_qa_pipeline.add_component(instance=retriever, name="retriever")
|
||||
extractive_qa_pipeline.add_component(instance=reader, name="reader")
|
||||
|
||||
extractive_qa_pipeline.connect("retriever.documents", "reader.documents")
|
||||
|
||||
query = "What is the capital of France?"
|
||||
extractive_qa_pipeline.run(
|
||||
data={
|
||||
"retriever": {"query": query, "top_k": 3},
|
||||
"reader": {"query": query, "top_k": 2},
|
||||
},
|
||||
)
|
||||
```
|
||||
@@ -17,9 +17,9 @@ Use this component to route text input to various output connections based on a
|
||||
| **Mandatory init variables** | `model`: The name or path of a Hugging Face model for text classification <br /> <br />`token`: The Hugging Face API token. Can be set with `HF_API_TOKEN` or `HF_TOKEN` env var. |
|
||||
| **Mandatory run variables** | `text`: The text to be routed to one of the specified outputs based on which label it has been categorized into |
|
||||
| **Output variables** | `documents`: A dictionary with the label as key and the text as value |
|
||||
| **API reference** | [Routers](/reference/routers-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/routers/transformers_text_router.py |
|
||||
| **Package name** | `haystack-ai` |
|
||||
| **API reference** | [Transformers](/reference/integrations-transformers) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/transformers |
|
||||
| **Package name** | `transformers-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
@@ -31,10 +31,16 @@ First, you need to set a selected model with a `model` parameter when initializi
|
||||
|
||||
You can additionally provide the `labels` parameter – a list of strings of possible class labels to classify each sequence into. If not provided, the component fetches the labels from the model configuration file hosted on the HuggingFace Hub using `transformers.AutoConfig.from_pretrained`.
|
||||
|
||||
To see the full list of parameters, check out our [API reference](/reference/routers-api#transformerstextrouter).
|
||||
To see the full list of parameters, check out our [API reference](/reference/integrations-transformers#transformerstextrouter).
|
||||
|
||||
## Usage
|
||||
|
||||
Install the `transformers-haystack` package to use the `TransformersTextRouter`:
|
||||
|
||||
```shell
|
||||
pip install transformers-haystack
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
The `TransformersTextRouter` isn’t very effective on its own, as its main strength lies in working within a pipeline. The component's true potential is unlocked when it is integrated into a pipeline, where it can efficiently route text to the most appropriate components. Please see the following section for a complete example of usage.
|
||||
@@ -45,9 +51,11 @@ Below is an example of a simple pipeline that routes English queries to a Text G
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.components.routers import TransformersTextRouter
|
||||
from haystack_integrations.components.routers.transformers import TransformersTextRouter
|
||||
from haystack.components.builders.chat_prompt_builder import ChatPromptBuilder
|
||||
from haystack.components.generators.chat import HuggingFaceLocalChatGenerator
|
||||
from haystack_integrations.components.generators.transformers import (
|
||||
TransformersChatGenerator,
|
||||
)
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
p = Pipeline()
|
||||
@@ -73,13 +81,13 @@ p.add_component(
|
||||
name="german_prompt_builder",
|
||||
)
|
||||
p.add_component(
|
||||
instance=HuggingFaceLocalChatGenerator(
|
||||
instance=TransformersChatGenerator(
|
||||
model="DiscoResearch/Llama3-DiscoLeo-Instruct-8B-v0.1",
|
||||
),
|
||||
name="german_llm",
|
||||
)
|
||||
p.add_component(
|
||||
instance=HuggingFaceLocalChatGenerator(model="microsoft/Phi-3-mini-4k-instruct"),
|
||||
instance=TransformersChatGenerator(model="microsoft/Phi-3-mini-4k-instruct"),
|
||||
name="english_llm",
|
||||
)
|
||||
|
||||
|
||||
@@ -17,9 +17,9 @@ Use this component to route text input to various output connections based on it
|
||||
| **Mandatory init variables** | `labels`: A list of labels for classification <br /> <br />`token`: The Hugging Face API token. Can be set with `HF_API_TOKEN` or `HF_TOKEN` env var. |
|
||||
| **Mandatory run variables** | `text`: The text to be routed to one of the specified outputs based on which label it has been categorized into |
|
||||
| **Output variables** | `documents`: A dictionary with the label as key and the text as value |
|
||||
| **API reference** | [Routers](/reference/routers-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/routers/zero_shot_text_router.py |
|
||||
| **Package name** | `haystack-ai` |
|
||||
| **API reference** | [Transformers](/reference/integrations-transformers) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/transformers |
|
||||
| **Package name** | `transformers-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
@@ -31,10 +31,16 @@ Use this component to route text input to various output connections based on it
|
||||
|
||||
To use `TransformersZeroShotTextRouter`, you need to provide the mandatory `labels` parameter – a list of strings of possible class labels to classify each sequence into.
|
||||
|
||||
To see the full list of parameters, check out our [API reference](/reference/routers-api#transformerszeroshottextrouter).
|
||||
To see the full list of parameters, check out our [API reference](/reference/integrations-transformers#transformerszeroshottextrouter).
|
||||
|
||||
## Usage
|
||||
|
||||
Install the `transformers-haystack` package to use the `TransformersZeroShotTextRouter`:
|
||||
|
||||
```shell
|
||||
pip install transformers-haystack
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
The `TransformersZeroShotTextRouter` isn’t very effective on its own, as its main strength lies in working within a pipeline. The component's true potential is unlocked when it is integrated into a pipeline, where it can efficiently route text to the most appropriate components. Please see the following section for a complete example of usage.
|
||||
@@ -53,7 +59,7 @@ Finally, the pipeline is executed with a sample text: "What is the capital of Ge
|
||||
from haystack import Document
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.core.pipeline import Pipeline
|
||||
from haystack.components.routers import TransformersZeroShotTextRouter
|
||||
from haystack_integrations.components.routers.transformers import TransformersZeroShotTextRouter
|
||||
from haystack.components.embedders import SentenceTransformersTextEmbedder, SentenceTransformersDocumentEmbedder
|
||||
from haystack.components.retrievers import InMemoryEmbeddingRetriever
|
||||
|
||||
|
||||
@@ -246,8 +246,8 @@ for the full list.
|
||||
|
||||
```python
|
||||
__init__(
|
||||
azure_endpoint: str | None = None,
|
||||
api_version: str | None = "2024-12-01-preview",
|
||||
azure_endpoint: str | Secret | None = None,
|
||||
api_version: str | Secret | None = "2024-12-01-preview",
|
||||
azure_deployment: str | None = "gpt-4.1-mini",
|
||||
api_key: Secret | None = Secret.from_env_var(
|
||||
"AZURE_OPENAI_API_KEY", strict=False
|
||||
@@ -275,8 +275,15 @@ Initialize the Azure OpenAI Chat Generator component.
|
||||
|
||||
**Parameters:**
|
||||
|
||||
- **azure_endpoint** (<code>str | None</code>) – The endpoint of the deployed model, for example `"https://example-resource.azure.openai.com/"`.
|
||||
- **api_version** (<code>str | None</code>) – The version of the API to use. Defaults to 2024-12-01-preview.
|
||||
- **azure_endpoint** (<code>str | Secret | None</code>) – The endpoint of the deployed model, for example `"https://example-resource.azure.openai.com/"`.
|
||||
Can also be a [Secret](https://docs.haystack.deepset.ai/docs/secret-management), for example
|
||||
`Secret.from_env_var("AZURE_OPENAI_ENDPOINT")`, to resolve the value from an environment variable at
|
||||
runtime. This is useful to switch endpoints between environments (e.g. dev and prod) without changing the
|
||||
serialized pipeline.
|
||||
- **api_version** (<code>str | Secret | None</code>) – The version of the API to use. Defaults to 2024-12-01-preview.
|
||||
Can also be a [Secret](https://docs.haystack.deepset.ai/docs/secret-management), for example
|
||||
`Secret.from_env_var("AZURE_OPENAI_API_VERSION")`, to resolve the value from an environment variable at
|
||||
runtime.
|
||||
- **azure_deployment** (<code>str | None</code>) – The deployment of the model, usually the model name.
|
||||
- **api_key** (<code>Secret | None</code>) – The API key to use for authentication.
|
||||
- **azure_ad_token** (<code>Secret | None</code>) – [Azure Active Directory token](https://www.microsoft.com/en-us/security/business/identity-access/microsoft-entra-id).
|
||||
|
||||
@@ -367,6 +367,7 @@ export default {
|
||||
'pipeline-components/extractors/namedentityextractor',
|
||||
'pipeline-components/extractors/presidioentityextractor',
|
||||
'pipeline-components/extractors/regextextextractor',
|
||||
'pipeline-components/extractors/transformersnamedentityextractor',
|
||||
],
|
||||
},
|
||||
{
|
||||
@@ -437,6 +438,7 @@ export default {
|
||||
'pipeline-components/generators/stackitchatgenerator',
|
||||
'pipeline-components/generators/togetheraichatgenerator',
|
||||
'pipeline-components/generators/togetheraigenerator',
|
||||
'pipeline-components/generators/transformerschatgenerator',
|
||||
'pipeline-components/generators/vertexaicodegenerator',
|
||||
'pipeline-components/generators/vertexaigeminichatgenerator',
|
||||
'pipeline-components/generators/vertexaigeminigenerator',
|
||||
@@ -536,6 +538,7 @@ export default {
|
||||
},
|
||||
items: [
|
||||
'pipeline-components/readers/extractivereader',
|
||||
'pipeline-components/readers/transformersextractivereader',
|
||||
],
|
||||
},
|
||||
{
|
||||
|
||||
+17
-7
@@ -17,9 +17,9 @@ Classifies the documents based on the provided labels and adds them to their met
|
||||
| **Mandatory init variables** | `model`: The name or path of a Hugging Face model for zero shot document classification <br /> <br />`labels`: The set of possible class labels to classify each document into, for example, [`positive`, `negative`]. The labels depend on the selected model. |
|
||||
| **Mandatory run variables** | `documents`: A list of documents to classify |
|
||||
| **Output variables** | `documents`: A list of processed documents with an added `classification` metadata field |
|
||||
| **API reference** | [Classifiers](/reference/classifiers-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/classifiers/zero_shot_document_classifier.py |
|
||||
| **Package name** | `haystack-ai` |
|
||||
| **API reference** | [Transformers](/reference/integrations-transformers) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/transformers |
|
||||
| **Package name** | `transformers-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
@@ -42,11 +42,19 @@ Available models for the task of zero-shot-classification are:
|
||||
|
||||
## Usage
|
||||
|
||||
Install the `transformers-haystack` package to use the `TransformersZeroShotDocumentClassifier`:
|
||||
|
||||
```shell
|
||||
pip install transformers-haystack
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack.components.classifiers import TransformersZeroShotDocumentClassifier
|
||||
from haystack_integrations.components.classifiers.transformers import (
|
||||
TransformersZeroShotDocumentClassifier,
|
||||
)
|
||||
|
||||
documents = [
|
||||
Document(id="0", content="Cats don't get teeth cavities."),
|
||||
@@ -71,7 +79,9 @@ from haystack import Document
|
||||
from haystack.components.retrievers.in_memory import InMemoryBM25Retriever
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.core.pipeline import Pipeline
|
||||
from haystack.components.classifiers import TransformersZeroShotDocumentClassifier
|
||||
from haystack_integrations.components.classifiers.transformers import (
|
||||
TransformersZeroShotDocumentClassifier,
|
||||
)
|
||||
|
||||
documents = [
|
||||
Document(id="0", content="Today was a nice day!"),
|
||||
@@ -88,8 +98,8 @@ document_classifier = TransformersZeroShotDocumentClassifier(
|
||||
document_store.write_documents(documents)
|
||||
|
||||
pipeline = Pipeline()
|
||||
pipeline.add_component(retriever, name="retriever")
|
||||
pipeline.add_component(document_classifier, name="document_classifier")
|
||||
pipeline.add_component(name="retriever", instance=retriever)
|
||||
pipeline.add_component(name="document_classifier", instance=document_classifier)
|
||||
pipeline.connect("retriever", "document_classifier")
|
||||
|
||||
queries = ["How was your day today?", "How was your day yesterday?"]
|
||||
|
||||
+6
@@ -9,6 +9,12 @@ description: "This component extracts predefined entities out of a piece of text
|
||||
|
||||
This component extracts predefined entities out of a piece of text and writes them into documents’ meta field.
|
||||
|
||||
:::warning[Deprecated]
|
||||
|
||||
`NamedEntityExtractor` is deprecated and will be removed in Haystack 3.0. It has moved to the `transformers-haystack` package and was renamed to `TransformersNamedEntityExtractor`. See [TransformersNamedEntityExtractor](transformersnamedentityextractor.mdx) for the updated documentation.
|
||||
|
||||
:::
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
|
||||
+102
@@ -0,0 +1,102 @@
|
||||
---
|
||||
title: "TransformersNamedEntityExtractor"
|
||||
id: transformersnamedentityextractor
|
||||
slug: "/transformersnamedentityextractor"
|
||||
description: "This component extracts predefined entities out of a piece of text and writes them into documents’ meta field."
|
||||
---
|
||||
|
||||
# TransformersNamedEntityExtractor
|
||||
|
||||
This component extracts predefined entities out of a piece of text and writes them into documents’ meta field.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | After the [PreProcessor](../preprocessors.mdx) in an indexing pipeline or after a [Retriever](../retrievers.mdx) in a query pipeline |
|
||||
| **Mandatory init variables** | `model`: Name or path of the model to use |
|
||||
| **Mandatory run variables** | `documents`: A list of documents |
|
||||
| **Output variables** | `documents`: A list of documents |
|
||||
| **API reference** | [Transformers](/reference/integrations-transformers) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/transformers |
|
||||
| **Package name** | `transformers-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`TransformersNamedEntityExtractor` looks for entities, which are spans in the text. The extractor automatically recognizes and groups them depending on their class, such as people's names, organizations, locations, and other types. The exact classes are determined by the model that you initialize the component with.
|
||||
|
||||
`TransformersNamedEntityExtractor` takes a list of documents as input and returns a list of the same documents with their `meta` data enriched with `NamedEntityAnnotations`. A `NamedEntityAnnotation` consists of the type of the entity, the start and end of the span, and a score calculated by the model, for example: `NamedEntityAnnotation(entity='PER', start=11, end=16, score=0.9)`.
|
||||
|
||||
When the `TransformersNamedEntityExtractor` is initialized, you need to set a `model`. Optionally, you can set `pipeline_kwargs`, which are then passed on to the Hugging Face pipeline. You can additionally set the `device` that is used to run the component.
|
||||
|
||||
Authentication with a Hugging Face API token is only required to access private or gated models. You can pass the token at initialization with `token`, or set the `HF_API_TOKEN` or `HF_TOKEN` environment variable.
|
||||
|
||||
## Usage
|
||||
|
||||
Install the `transformers-haystack` package to use the `TransformersNamedEntityExtractor`:
|
||||
|
||||
```shell
|
||||
pip install transformers-haystack
|
||||
```
|
||||
|
||||
The component works with any Hugging Face model that supports token classification or NER.
|
||||
|
||||
`TransformersNamedEntityExtractor` accepts a list of `Documents` as its input. The extractor annotates the raw text in the documents and stores the annotations in the document's `meta` dictionary under the `named_entities` key.
|
||||
|
||||
```python
|
||||
from haystack.dataclasses import Document
|
||||
from haystack_integrations.components.extractors.transformers import (
|
||||
TransformersNamedEntityExtractor,
|
||||
)
|
||||
|
||||
extractor = TransformersNamedEntityExtractor(model="dslim/bert-base-NER")
|
||||
|
||||
documents = [
|
||||
Document(content="My name is Clara and I live in Berkeley, California."),
|
||||
Document(content="I'm Merlin, the happy pig!"),
|
||||
Document(content="New York State is home to the Empire State Building."),
|
||||
]
|
||||
|
||||
extractor.run(documents)
|
||||
print(documents)
|
||||
```
|
||||
|
||||
Here is the example result:
|
||||
|
||||
```python
|
||||
[Document(id=aec840d1b6c85609f4f16c3e222a5a25fd8c4c53bd981a40c1268ab9c72cee10, content: 'My name is Clara and I live in Berkeley, California.', meta: {'named_entities': [NamedEntityAnnotation(entity='PER', start=11, end=16, score=0.99641764), NamedEntityAnnotation(entity='LOC', start=31, end=39, score=0.996198), NamedEntityAnnotation(entity='LOC', start=41, end=51, score=0.9990196)]}),
|
||||
Document(id=98f1dc5d0ccd9d9950cd191d1076db0f7af40c401dd7608f11c90cb3fc38c0c2, content: 'I'm Merlin, the happy pig!', meta: {'named_entities': [NamedEntityAnnotation(entity='PER', start=4, end=10, score=0.99054915)]}),
|
||||
Document(id=44948ea0eec018b33aceaaedde4616eb9e93ce075e0090ec1613fc145f84b4a9, content: 'New York State is home to the Empire State Building.', meta: {'named_entities': [NamedEntityAnnotation(entity='LOC', start=0, end=14, score=0.9989541), NamedEntityAnnotation(entity='LOC', start=30, end=51, score=0.95746297)]})]
|
||||
```
|
||||
|
||||
### Get stored annotations
|
||||
|
||||
This component includes the `get_stored_annotations` helper class method that allows you to retrieve the annotations stored in a `Document` transparently:
|
||||
|
||||
```python
|
||||
from haystack.dataclasses import Document
|
||||
from haystack_integrations.components.extractors.transformers import (
|
||||
TransformersNamedEntityExtractor,
|
||||
)
|
||||
|
||||
extractor = TransformersNamedEntityExtractor(model="dslim/bert-base-NER")
|
||||
|
||||
documents = [
|
||||
Document(content="My name is Clara and I live in Berkeley, California."),
|
||||
Document(content="I'm Merlin, the happy pig!"),
|
||||
Document(content="New York State is home to the Empire State Building."),
|
||||
]
|
||||
|
||||
extractor.run(documents)
|
||||
|
||||
annotations = [
|
||||
TransformersNamedEntityExtractor.get_stored_annotations(doc) for doc in documents
|
||||
]
|
||||
print(annotations)
|
||||
|
||||
# If a Document doesn't contain any annotations, this returns None.
|
||||
new_doc = Document(content="In one of many possible worlds...")
|
||||
assert TransformersNamedEntityExtractor.get_stored_annotations(new_doc) is None
|
||||
```
|
||||
+12
@@ -43,6 +43,18 @@ client = AzureOpenAIChatGenerator(
|
||||
We recommend using environment variables instead of initialization parameters.
|
||||
:::
|
||||
|
||||
To switch `azure_endpoint` and `api_version` between environments without editing your pipeline, pass a Secret that resolves them from environment variables at runtime:
|
||||
|
||||
```python
|
||||
from haystack.components.generators.chat import AzureOpenAIChatGenerator
|
||||
from haystack.utils import Secret
|
||||
|
||||
client = AzureOpenAIChatGenerator(
|
||||
azure_endpoint=Secret.from_env_var("AZURE_OPENAI_ENDPOINT"),
|
||||
api_version=Secret.from_env_var("AZURE_OPENAI_API_VERSION"),
|
||||
)
|
||||
```
|
||||
|
||||
Then, the component needs a list of `ChatMessage` objects to operate. `ChatMessage` is a data class that contains a message, a role (who generated the message, such as `user`, `assistant`, `system`, `function`), and optional metadata. If a string is passed, it is converted into a list containing a single `ChatMessage` with the `user` role. See the [usage](#usage) section for an example.
|
||||
|
||||
You can pass any chat completion parameters that are valid for the `openai.ChatCompletion.create` method directly to `AzureOpenAIChatGenerator` using the `generation_kwargs` parameter, both at initialization and to `run()` method. For more details on the supported parameters, refer to the [Azure documentation](https://learn.microsoft.com/en-us/azure/ai-services/openai/reference).
|
||||
|
||||
+6
@@ -9,6 +9,12 @@ description: "Provides an interface for chat completion using a Hugging Face mod
|
||||
|
||||
Provides an interface for chat completion using a Hugging Face model that runs locally.
|
||||
|
||||
:::warning[Deprecated]
|
||||
|
||||
`HuggingFaceLocalChatGenerator` is deprecated and will be removed in Haystack 3.0. It has moved to the `transformers-haystack` package and was renamed to `TransformersChatGenerator`. See [TransformersChatGenerator](transformerschatgenerator.mdx) for the updated documentation.
|
||||
|
||||
:::
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
|
||||
+101
@@ -0,0 +1,101 @@
|
||||
---
|
||||
title: "TransformersChatGenerator"
|
||||
id: transformerschatgenerator
|
||||
slug: "/transformerschatgenerator"
|
||||
description: "Provides an interface for chat completion using a Hugging Face model that runs locally."
|
||||
---
|
||||
|
||||
# TransformersChatGenerator
|
||||
|
||||
Provides an interface for chat completion using a Hugging Face model that runs locally.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | After a [ChatPromptBuilder](../builders/chatpromptbuilder.mdx) |
|
||||
| **Mandatory init variables** | None |
|
||||
| **Mandatory run variables** | `messages`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects representing the chat or a plain string |
|
||||
| **Output variables** | `replies`: A list of [`ChatMessage`](../../concepts/data-classes/chatmessage.mdx) objects generated by the LLM |
|
||||
| **API reference** | [Transformers](/reference/integrations-transformers) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/transformers |
|
||||
| **Package name** | `transformers-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
Keep in mind that if LLMs run locally, you may need a powerful machine to run them. This depends strongly on the model you select and its parameter count.
|
||||
|
||||
If a string is passed to `messages`, it is converted into a list containing a single `ChatMessage` with the `user` role.
|
||||
|
||||
Authentication with a Hugging Face API token is only required to access private or gated models. You can pass the token at initialization with `token`, or set the `HF_API_TOKEN` or `HF_TOKEN` environment variable:
|
||||
|
||||
```python
|
||||
generator = TransformersChatGenerator(
|
||||
token=Secret.from_token("<your-api-key>"),
|
||||
)
|
||||
```
|
||||
|
||||
### Streaming
|
||||
|
||||
This Generator supports [streaming](guides-to-generators/choosing-the-right-generator.mdx#streaming-support) the tokens from the LLM directly in output. To do so, pass a function to the `streaming_callback` init parameter.
|
||||
|
||||
## Usage
|
||||
|
||||
Install the `transformers-haystack` package to use the `TransformersChatGenerator`:
|
||||
|
||||
```shell
|
||||
pip install transformers-haystack
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
```python
|
||||
from haystack_integrations.components.generators.transformers import (
|
||||
TransformersChatGenerator,
|
||||
)
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
generator = TransformersChatGenerator(model="Qwen/Qwen3-0.6B")
|
||||
messages = [ChatMessage.from_user("What's Natural Language Processing? Be brief.")]
|
||||
print(generator.run(messages))
|
||||
```
|
||||
|
||||
### In a Pipeline
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.components.builders.prompt_builder import ChatPromptBuilder
|
||||
from haystack_integrations.components.generators.transformers import (
|
||||
TransformersChatGenerator,
|
||||
)
|
||||
from haystack.dataclasses import ChatMessage
|
||||
from haystack.utils import Secret
|
||||
|
||||
prompt_builder = ChatPromptBuilder()
|
||||
llm = TransformersChatGenerator(
|
||||
model="Qwen/Qwen3-0.6B",
|
||||
token=Secret.from_env_var("HF_API_TOKEN"),
|
||||
)
|
||||
|
||||
pipe = Pipeline()
|
||||
pipe.add_component("prompt_builder", prompt_builder)
|
||||
pipe.add_component("llm", llm)
|
||||
pipe.connect("prompt_builder.prompt", "llm.messages")
|
||||
location = "Berlin"
|
||||
messages = [
|
||||
ChatMessage.from_system(
|
||||
"Always respond in German even if some input data is in other languages.",
|
||||
),
|
||||
ChatMessage.from_user("Tell me about {{location}}"),
|
||||
]
|
||||
pipe.run(
|
||||
data={
|
||||
"prompt_builder": {
|
||||
"template_variables": {"location": location},
|
||||
"template": messages,
|
||||
},
|
||||
},
|
||||
)
|
||||
```
|
||||
+6
@@ -9,6 +9,12 @@ description: "Use this component in extractive question answering pipelines base
|
||||
|
||||
Use this component in extractive question answering pipelines based on a query and a list of documents.
|
||||
|
||||
:::warning[Deprecated]
|
||||
|
||||
`ExtractiveReader` is deprecated and will be removed in Haystack 3.0. It has moved to the `transformers-haystack` package and was renamed to `TransformersExtractiveReader`. See [TransformersExtractiveReader](transformersextractivereader.mdx) for the updated documentation.
|
||||
|
||||
:::
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
|
||||
+119
@@ -0,0 +1,119 @@
|
||||
---
|
||||
title: "TransformersExtractiveReader"
|
||||
id: transformersextractivereader
|
||||
slug: "/transformersextractivereader"
|
||||
description: "Use this component in extractive question answering pipelines based on a query and a list of documents."
|
||||
---
|
||||
|
||||
# TransformersExtractiveReader
|
||||
|
||||
Use this component in extractive question answering pipelines based on a query and a list of documents.
|
||||
|
||||
<div className="key-value-table">
|
||||
|
||||
| | |
|
||||
| --- | --- |
|
||||
| **Most common position in a pipeline** | In query pipelines, after a component that returns a list of documents, such as a [Retriever](../retrievers.mdx) |
|
||||
| **Mandatory init variables** | None |
|
||||
| **Mandatory run variables** | `documents`: A list of documents <br /> <br />`query`: A query string |
|
||||
| **Output variables** | `answers`: A list of [`ExtractedAnswer`](../../concepts/data-classes.mdx#extractedanswer) objects |
|
||||
| **API reference** | [Transformers](/reference/integrations-transformers) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/transformers |
|
||||
| **Package name** | `transformers-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
## Overview
|
||||
|
||||
`TransformersExtractiveReader` locates and extracts answers to a given query from the document text. It's used in extractive QA systems where you want to know exactly where the answer is located within the document. It's usually coupled with a Retriever that precedes it, but you can also use it with other components that fetch documents.
|
||||
|
||||
Readers assign a _probability_ to answers. This score ranges from 0 to 1, indicating how well the results the Reader returned match the query. Probability closest to 1 means the model has high confidence in the answer's relevance. The Reader sorts the answers based on their probability scores, with higher probability listed first. You can limit the number of answers the Reader returns in the optional `top_k` parameter.
|
||||
|
||||
You can use the probability to set the quality expectations for your system. To do that, use the `confidence_score` parameter of the Reader to set a minimum probability threshold for answers. For example, setting `confidence_threshold` to `0.7` means only answers with a probability higher than 0.7 will be returned.
|
||||
|
||||
By default, the Reader includes a scenario where no answer to the query is found in the document text (`no_answer=True`). In this case, it returns an additional `ExtractedAnswer` with no text and the probability that none of the `top_k` answers are correct. For example, if `top_k=4` the system will return four answers and an additional empty one. Each answer has a probability assigned. If the empty answer has a probability of 0.5, it means that's the probability that none of the returned answers is correct. To receive only the actual top_k answers, set the `no_answer` parameter to `False` when initializing the component.
|
||||
|
||||
### Models
|
||||
|
||||
Here are the models that we recommend for using with `TransformersExtractiveReader`:
|
||||
|
||||
| | | |
|
||||
| --- | --- | --- |
|
||||
| Model URL | Description | Language |
|
||||
| [deepset/roberta-base-squad2-distilled](https://huggingface.co/deepset/roberta-base-squad2-distilled) (default) | A distilled model, relatively fast and with good performance. | English |
|
||||
| [deepset/roberta-large-squad2](https://huggingface.co/deepset/roberta-large-squad2) | A large model with good performance. Slower than the distilled one. | English |
|
||||
| [deepset/tinyroberta-squad2](https://huggingface.co/deepset/tinyroberta-squad2) | A distilled version of roberta-large-squad2 model, very fast. | English |
|
||||
| [deepset/xlm-roberta-base-squad2](https://huggingface.co/deepset/xlm-roberta-base-squad2) | A base multilingual model with good speed and performance. | Multilingual |
|
||||
|
||||
You can also view other question answering models on [Hugging Face](https://huggingface.co/models?pipeline_tag=question-answering).
|
||||
|
||||
Authentication with a Hugging Face API token is only required to access private or gated models. You can pass the token at initialization with `token`, or set the `HF_API_TOKEN` or `HF_TOKEN` environment variable.
|
||||
|
||||
## Usage
|
||||
|
||||
Install the `transformers-haystack` package to use the `TransformersExtractiveReader`:
|
||||
|
||||
```shell
|
||||
pip install transformers-haystack
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
Below is an example that uses the `TransformersExtractiveReader` outside of a pipeline. The Reader gets the query and the documents at runtime. It should return two answers and an additional third answer with no text and the probability that the `top_k` answers are incorrect.
|
||||
|
||||
```python
|
||||
from haystack import Document
|
||||
from haystack_integrations.components.readers.transformers import (
|
||||
TransformersExtractiveReader,
|
||||
)
|
||||
|
||||
docs = [
|
||||
Document(content="Paris is the capital of France."),
|
||||
Document(content="Berlin is the capital of Germany."),
|
||||
]
|
||||
|
||||
reader = TransformersExtractiveReader()
|
||||
|
||||
reader.run(query="What is the capital of France?", documents=docs, top_k=2)
|
||||
```
|
||||
|
||||
### In a pipeline
|
||||
|
||||
Below is an example of a pipeline that retrieves a document from an `InMemoryDocumentStore` based on keyword search (using `InMemoryBM25Retriever`). It then uses the `TransformersExtractiveReader` to extract the answer to our query from the top retrieved documents.
|
||||
|
||||
With the TransformersExtractiveReader’s `top_k` set to 2, an additional, third answer with no text and the probability that the other `top_k` answers are incorrect is also returned.
|
||||
|
||||
```python
|
||||
from haystack import Document, Pipeline
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.components.retrievers.in_memory import InMemoryBM25Retriever
|
||||
from haystack_integrations.components.readers.transformers import (
|
||||
TransformersExtractiveReader,
|
||||
)
|
||||
|
||||
docs = [
|
||||
Document(content="Paris is the capital of France."),
|
||||
Document(content="Berlin is the capital of Germany."),
|
||||
Document(content="Rome is the capital of Italy."),
|
||||
Document(content="Madrid is the capital of Spain."),
|
||||
]
|
||||
document_store = InMemoryDocumentStore()
|
||||
document_store.write_documents(docs)
|
||||
|
||||
retriever = InMemoryBM25Retriever(document_store=document_store)
|
||||
reader = TransformersExtractiveReader()
|
||||
|
||||
extractive_qa_pipeline = Pipeline()
|
||||
extractive_qa_pipeline.add_component(instance=retriever, name="retriever")
|
||||
extractive_qa_pipeline.add_component(instance=reader, name="reader")
|
||||
|
||||
extractive_qa_pipeline.connect("retriever.documents", "reader.documents")
|
||||
|
||||
query = "What is the capital of France?"
|
||||
extractive_qa_pipeline.run(
|
||||
data={
|
||||
"retriever": {"query": query, "top_k": 3},
|
||||
"reader": {"query": query, "top_k": 2},
|
||||
},
|
||||
)
|
||||
```
|
||||
+16
-8
@@ -17,9 +17,9 @@ Use this component to route text input to various output connections based on a
|
||||
| **Mandatory init variables** | `model`: The name or path of a Hugging Face model for text classification <br /> <br />`token`: The Hugging Face API token. Can be set with `HF_API_TOKEN` or `HF_TOKEN` env var. |
|
||||
| **Mandatory run variables** | `text`: The text to be routed to one of the specified outputs based on which label it has been categorized into |
|
||||
| **Output variables** | `documents`: A dictionary with the label as key and the text as value |
|
||||
| **API reference** | [Routers](/reference/routers-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/routers/transformers_text_router.py |
|
||||
| **Package name** | `haystack-ai` |
|
||||
| **API reference** | [Transformers](/reference/integrations-transformers) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/transformers |
|
||||
| **Package name** | `transformers-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
@@ -31,10 +31,16 @@ First, you need to set a selected model with a `model` parameter when initializi
|
||||
|
||||
You can additionally provide the `labels` parameter – a list of strings of possible class labels to classify each sequence into. If not provided, the component fetches the labels from the model configuration file hosted on the HuggingFace Hub using `transformers.AutoConfig.from_pretrained`.
|
||||
|
||||
To see the full list of parameters, check out our [API reference](/reference/routers-api#transformerstextrouter).
|
||||
To see the full list of parameters, check out our [API reference](/reference/integrations-transformers#transformerstextrouter).
|
||||
|
||||
## Usage
|
||||
|
||||
Install the `transformers-haystack` package to use the `TransformersTextRouter`:
|
||||
|
||||
```shell
|
||||
pip install transformers-haystack
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
The `TransformersTextRouter` isn’t very effective on its own, as its main strength lies in working within a pipeline. The component's true potential is unlocked when it is integrated into a pipeline, where it can efficiently route text to the most appropriate components. Please see the following section for a complete example of usage.
|
||||
@@ -45,9 +51,11 @@ Below is an example of a simple pipeline that routes English queries to a Text G
|
||||
|
||||
```python
|
||||
from haystack import Pipeline
|
||||
from haystack.components.routers import TransformersTextRouter
|
||||
from haystack_integrations.components.routers.transformers import TransformersTextRouter
|
||||
from haystack.components.builders.chat_prompt_builder import ChatPromptBuilder
|
||||
from haystack.components.generators.chat import HuggingFaceLocalChatGenerator
|
||||
from haystack_integrations.components.generators.transformers import (
|
||||
TransformersChatGenerator,
|
||||
)
|
||||
from haystack.dataclasses import ChatMessage
|
||||
|
||||
p = Pipeline()
|
||||
@@ -73,13 +81,13 @@ p.add_component(
|
||||
name="german_prompt_builder",
|
||||
)
|
||||
p.add_component(
|
||||
instance=HuggingFaceLocalChatGenerator(
|
||||
instance=TransformersChatGenerator(
|
||||
model="DiscoResearch/Llama3-DiscoLeo-Instruct-8B-v0.1",
|
||||
),
|
||||
name="german_llm",
|
||||
)
|
||||
p.add_component(
|
||||
instance=HuggingFaceLocalChatGenerator(model="microsoft/Phi-3-mini-4k-instruct"),
|
||||
instance=TransformersChatGenerator(model="microsoft/Phi-3-mini-4k-instruct"),
|
||||
name="english_llm",
|
||||
)
|
||||
|
||||
|
||||
+11
-5
@@ -17,9 +17,9 @@ Use this component to route text input to various output connections based on it
|
||||
| **Mandatory init variables** | `labels`: A list of labels for classification <br /> <br />`token`: The Hugging Face API token. Can be set with `HF_API_TOKEN` or `HF_TOKEN` env var. |
|
||||
| **Mandatory run variables** | `text`: The text to be routed to one of the specified outputs based on which label it has been categorized into |
|
||||
| **Output variables** | `documents`: A dictionary with the label as key and the text as value |
|
||||
| **API reference** | [Routers](/reference/routers-api) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack/blob/main/haystack/components/routers/zero_shot_text_router.py |
|
||||
| **Package name** | `haystack-ai` |
|
||||
| **API reference** | [Transformers](/reference/integrations-transformers) |
|
||||
| **GitHub link** | https://github.com/deepset-ai/haystack-core-integrations/tree/main/integrations/transformers |
|
||||
| **Package name** | `transformers-haystack` |
|
||||
|
||||
</div>
|
||||
|
||||
@@ -31,10 +31,16 @@ Use this component to route text input to various output connections based on it
|
||||
|
||||
To use `TransformersZeroShotTextRouter`, you need to provide the mandatory `labels` parameter – a list of strings of possible class labels to classify each sequence into.
|
||||
|
||||
To see the full list of parameters, check out our [API reference](/reference/routers-api#transformerszeroshottextrouter).
|
||||
To see the full list of parameters, check out our [API reference](/reference/integrations-transformers#transformerszeroshottextrouter).
|
||||
|
||||
## Usage
|
||||
|
||||
Install the `transformers-haystack` package to use the `TransformersZeroShotTextRouter`:
|
||||
|
||||
```shell
|
||||
pip install transformers-haystack
|
||||
```
|
||||
|
||||
### On its own
|
||||
|
||||
The `TransformersZeroShotTextRouter` isn’t very effective on its own, as its main strength lies in working within a pipeline. The component's true potential is unlocked when it is integrated into a pipeline, where it can efficiently route text to the most appropriate components. Please see the following section for a complete example of usage.
|
||||
@@ -53,7 +59,7 @@ Finally, the pipeline is executed with a sample text: "What is the capital of Ge
|
||||
from haystack import Document
|
||||
from haystack.document_stores.in_memory import InMemoryDocumentStore
|
||||
from haystack.core.pipeline import Pipeline
|
||||
from haystack.components.routers import TransformersZeroShotTextRouter
|
||||
from haystack_integrations.components.routers.transformers import TransformersZeroShotTextRouter
|
||||
from haystack.components.embedders import SentenceTransformersTextEmbedder, SentenceTransformersDocumentEmbedder
|
||||
from haystack.components.retrievers import InMemoryEmbeddingRetriever
|
||||
|
||||
|
||||
@@ -362,7 +362,8 @@
|
||||
"pipeline-components/extractors/llmmetadataextractor",
|
||||
"pipeline-components/extractors/namedentityextractor",
|
||||
"pipeline-components/extractors/presidioentityextractor",
|
||||
"pipeline-components/extractors/regextextextractor"
|
||||
"pipeline-components/extractors/regextextextractor",
|
||||
"pipeline-components/extractors/transformersnamedentityextractor"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -433,6 +434,7 @@
|
||||
"pipeline-components/generators/stackitchatgenerator",
|
||||
"pipeline-components/generators/togetheraichatgenerator",
|
||||
"pipeline-components/generators/togetheraigenerator",
|
||||
"pipeline-components/generators/transformerschatgenerator",
|
||||
"pipeline-components/generators/vertexaicodegenerator",
|
||||
"pipeline-components/generators/vertexaigeminichatgenerator",
|
||||
"pipeline-components/generators/vertexaigeminigenerator",
|
||||
@@ -531,7 +533,8 @@
|
||||
"id": "pipeline-components/readers"
|
||||
},
|
||||
"items": [
|
||||
"pipeline-components/readers/extractivereader"
|
||||
"pipeline-components/readers/extractivereader",
|
||||
"pipeline-components/readers/transformersextractivereader"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
#
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
import warnings
|
||||
from dataclasses import replace
|
||||
from typing import Any
|
||||
|
||||
@@ -112,6 +113,15 @@ class TransformersZeroShotDocumentClassifier:
|
||||
Dictionary containing keyword arguments used to initialize the
|
||||
Hugging Face pipeline for text classification.
|
||||
"""
|
||||
warnings.warn(
|
||||
"`TransformersZeroShotDocumentClassifier` will be removed from Haystack in version 3.0, as it is moving "
|
||||
"to the `transformers-haystack` package. To continue using it, install that package with "
|
||||
"`pip install transformers-haystack` and update your import to "
|
||||
"`from haystack_integrations.components.classifiers.transformers import "
|
||||
"TransformersZeroShotDocumentClassifier`.",
|
||||
FutureWarning,
|
||||
stacklevel=2,
|
||||
)
|
||||
|
||||
torch_and_transformers_import.check()
|
||||
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
#
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
import warnings
|
||||
from abc import ABC, abstractmethod
|
||||
from collections.abc import Iterator
|
||||
from contextlib import contextmanager
|
||||
@@ -136,6 +137,15 @@ class NamedEntityExtractor:
|
||||
:param token:
|
||||
The API token to download private models from Hugging Face.
|
||||
"""
|
||||
warnings.warn(
|
||||
"`NamedEntityExtractor` will be removed from Haystack in version 3.0, as it is moving to the "
|
||||
"`transformers-haystack` package and being renamed to `TransformersNamedEntityExtractor`. "
|
||||
"To continue using it, install that package with "
|
||||
"`pip install transformers-haystack` and update your import to "
|
||||
"`from haystack_integrations.components.extractors.transformers import TransformersNamedEntityExtractor`.",
|
||||
FutureWarning,
|
||||
stacklevel=2,
|
||||
)
|
||||
|
||||
if isinstance(backend, str):
|
||||
backend = NamedEntityExtractorBackend.from_str(backend)
|
||||
|
||||
@@ -111,8 +111,8 @@ class AzureOpenAIChatGenerator(OpenAIChatGenerator):
|
||||
# ruff: noqa: PLR0913
|
||||
def __init__(
|
||||
self,
|
||||
azure_endpoint: str | None = None,
|
||||
api_version: str | None = "2024-12-01-preview",
|
||||
azure_endpoint: str | Secret | None = None,
|
||||
api_version: str | Secret | None = "2024-12-01-preview",
|
||||
azure_deployment: str | None = "gpt-4.1-mini",
|
||||
api_key: Secret | None = Secret.from_env_var("AZURE_OPENAI_API_KEY", strict=False),
|
||||
azure_ad_token: Secret | None = Secret.from_env_var("AZURE_OPENAI_AD_TOKEN", strict=False),
|
||||
@@ -132,7 +132,14 @@ class AzureOpenAIChatGenerator(OpenAIChatGenerator):
|
||||
Initialize the Azure OpenAI Chat Generator component.
|
||||
|
||||
:param azure_endpoint: The endpoint of the deployed model, for example `"https://example-resource.azure.openai.com/"`.
|
||||
Can also be a [Secret](https://docs.haystack.deepset.ai/docs/secret-management), for example
|
||||
`Secret.from_env_var("AZURE_OPENAI_ENDPOINT")`, to resolve the value from an environment variable at
|
||||
runtime. This is useful to switch endpoints between environments (e.g. dev and prod) without changing the
|
||||
serialized pipeline.
|
||||
:param api_version: The version of the API to use. Defaults to 2024-12-01-preview.
|
||||
Can also be a [Secret](https://docs.haystack.deepset.ai/docs/secret-management), for example
|
||||
`Secret.from_env_var("AZURE_OPENAI_API_VERSION")`, to resolve the value from an environment variable at
|
||||
runtime.
|
||||
:param azure_deployment: The deployment of the model, usually the model name.
|
||||
:param api_key: The API key to use for authentication.
|
||||
:param azure_ad_token: [Azure Active Directory token](https://www.microsoft.com/en-us/security/business/identity-access/microsoft-entra-id).
|
||||
@@ -194,8 +201,14 @@ class AzureOpenAIChatGenerator(OpenAIChatGenerator):
|
||||
# None init parameters. This way we accommodate the use case where env var AZURE_OPENAI_ENDPOINT is set instead
|
||||
# of passing it as a parameter.
|
||||
azure_endpoint = azure_endpoint or os.environ.get("AZURE_OPENAI_ENDPOINT")
|
||||
if not azure_endpoint:
|
||||
# `azure_endpoint` and `api_version` accept either a plain string or a `Secret`. We keep the original value
|
||||
# on the instance for serialization and resolve it to a string only when building the client.
|
||||
resolved_azure_endpoint = (
|
||||
azure_endpoint.resolve_value() if isinstance(azure_endpoint, Secret) else azure_endpoint
|
||||
)
|
||||
if not resolved_azure_endpoint:
|
||||
raise ValueError("Please provide an Azure endpoint or set the environment variable AZURE_OPENAI_ENDPOINT.")
|
||||
resolved_api_version = api_version.resolve_value() if isinstance(api_version, Secret) else api_version
|
||||
|
||||
if api_key is None and azure_ad_token is None:
|
||||
raise ValueError("Please provide an API key or an Azure Active Directory token.")
|
||||
@@ -221,8 +234,8 @@ class AzureOpenAIChatGenerator(OpenAIChatGenerator):
|
||||
self.tools_strict = tools_strict
|
||||
|
||||
client_args: dict[str, Any] = {
|
||||
"api_version": api_version,
|
||||
"azure_endpoint": azure_endpoint,
|
||||
"api_version": resolved_api_version,
|
||||
"azure_endpoint": resolved_azure_endpoint,
|
||||
"azure_deployment": azure_deployment,
|
||||
"api_key": api_key.resolve_value() if api_key is not None else None,
|
||||
"azure_ad_token": azure_ad_token.resolve_value() if azure_ad_token is not None else None,
|
||||
@@ -279,10 +292,12 @@ class AzureOpenAIChatGenerator(OpenAIChatGenerator):
|
||||
generation_kwargs["response_format"] = json_schema
|
||||
return default_to_dict(
|
||||
self,
|
||||
azure_endpoint=self.azure_endpoint,
|
||||
azure_endpoint=self.azure_endpoint.to_dict()
|
||||
if isinstance(self.azure_endpoint, Secret)
|
||||
else self.azure_endpoint,
|
||||
azure_deployment=self.azure_deployment,
|
||||
organization=self.organization,
|
||||
api_version=self.api_version,
|
||||
api_version=self.api_version.to_dict() if isinstance(self.api_version, Secret) else self.api_version,
|
||||
streaming_callback=callback_name,
|
||||
generation_kwargs=generation_kwargs,
|
||||
timeout=self.timeout,
|
||||
|
||||
@@ -6,6 +6,7 @@ import asyncio
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
import warnings
|
||||
from collections.abc import AsyncIterator, Callable
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from contextlib import asynccontextmanager, suppress
|
||||
@@ -190,6 +191,15 @@ class HuggingFaceLocalChatGenerator:
|
||||
Whether to enable thinking mode in the chat template for thinking-capable models.
|
||||
When enabled, the model generates intermediate reasoning before the final response. Defaults to False.
|
||||
"""
|
||||
warnings.warn(
|
||||
"`HuggingFaceLocalChatGenerator` will be removed from Haystack in version 3.0, as it is moving to the "
|
||||
"`transformers-haystack` package and being renamed to `TransformersChatGenerator`. To continue using it, "
|
||||
"install that package with `pip install transformers-haystack` and update your import to "
|
||||
"`from haystack_integrations.components.generators.transformers import TransformersChatGenerator`.",
|
||||
FutureWarning,
|
||||
stacklevel=2,
|
||||
)
|
||||
|
||||
torch_and_transformers_import.check()
|
||||
|
||||
if tools and streaming_callback is not None:
|
||||
|
||||
@@ -3,6 +3,7 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
import math
|
||||
import warnings
|
||||
from dataclasses import replace
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
@@ -107,6 +108,15 @@ class ExtractiveReader:
|
||||
when loading the model specified in `model`. For details on what kwargs you can pass,
|
||||
see the model's documentation.
|
||||
"""
|
||||
warnings.warn(
|
||||
"`ExtractiveReader` will be removed from Haystack in version 3.0, as it is moving to the "
|
||||
"`transformers-haystack` package and being renamed to `TransformersExtractiveReader`. To continue using "
|
||||
"it, install that package with `pip install transformers-haystack` and update your import to "
|
||||
"`from haystack_integrations.components.readers.transformers import TransformersExtractiveReader`.",
|
||||
FutureWarning,
|
||||
stacklevel=2,
|
||||
)
|
||||
|
||||
torch_and_transformers_import.check()
|
||||
self.model_name_or_path = str(model)
|
||||
self.model = None
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
#
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
import warnings
|
||||
from typing import Any
|
||||
|
||||
from haystack import component, default_from_dict, default_to_dict
|
||||
@@ -89,6 +90,15 @@ class TransformersTextRouter:
|
||||
:param huggingface_pipeline_kwargs: A dictionary of keyword arguments for initializing the Hugging Face
|
||||
text classification pipeline.
|
||||
"""
|
||||
warnings.warn(
|
||||
"`TransformersTextRouter` will be removed from Haystack in version 3.0, as it is moving to the "
|
||||
"`transformers-haystack` package. To continue using it, install that package with "
|
||||
"`pip install transformers-haystack` and update your import to "
|
||||
"`from haystack_integrations.components.routers.transformers import TransformersTextRouter`.",
|
||||
FutureWarning,
|
||||
stacklevel=2,
|
||||
)
|
||||
|
||||
torch_and_transformers_import.check()
|
||||
|
||||
self.token = token
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
#
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
import warnings
|
||||
from typing import Any
|
||||
|
||||
from haystack import component, default_from_dict, default_to_dict
|
||||
@@ -116,6 +117,15 @@ class TransformersZeroShotTextRouter:
|
||||
:param huggingface_pipeline_kwargs: A dictionary of keyword arguments for initializing the Hugging Face
|
||||
zero shot text classification.
|
||||
"""
|
||||
warnings.warn(
|
||||
"`TransformersZeroShotTextRouter` will be removed from Haystack in version 3.0, as it is moving to the "
|
||||
"`transformers-haystack` package. To continue using it, install that package with "
|
||||
"`pip install transformers-haystack` and update your import to "
|
||||
"`from haystack_integrations.components.routers.transformers import TransformersZeroShotTextRouter`.",
|
||||
FutureWarning,
|
||||
stacklevel=2,
|
||||
)
|
||||
|
||||
torch_and_transformers_import.check()
|
||||
|
||||
self.token = token
|
||||
|
||||
+8
@@ -0,0 +1,8 @@
|
||||
---
|
||||
enhancements:
|
||||
- |
|
||||
``AzureOpenAIChatGenerator`` now accepts a ``Secret`` for the ``azure_endpoint`` and ``api_version`` parameters in
|
||||
addition to a plain string. This makes it possible to resolve these values from environment variables at
|
||||
runtime, for example with ``Secret.from_env_var("AZURE_OPENAI_ENDPOINT")``, so the same serialized pipeline can
|
||||
switch between environments (e.g. dev and prod) by changing environment variables instead of the pipeline
|
||||
definition.
|
||||
@@ -0,0 +1,20 @@
|
||||
---
|
||||
deprecations:
|
||||
- |
|
||||
``TransformersZeroShotDocumentClassifier``, ``TransformersTextRouter``, ``TransformersZeroShotTextRouter``,
|
||||
``HuggingFaceLocalChatGenerator``, ``NamedEntityExtractor``, and ``ExtractiveReader`` are deprecated and will be
|
||||
removed from Haystack in version 3.0. They are moving to the ``transformers-haystack`` package. Some of them are
|
||||
also being renamed. To continue using them, install the package with ``pip install transformers-haystack`` and
|
||||
update your imports as follows:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from haystack_integrations.components.classifiers.transformers import TransformersZeroShotDocumentClassifier
|
||||
from haystack_integrations.components.routers.transformers import TransformersTextRouter
|
||||
from haystack_integrations.components.routers.transformers import TransformersZeroShotTextRouter
|
||||
# HuggingFaceLocalChatGenerator was renamed to TransformersChatGenerator
|
||||
from haystack_integrations.components.generators.transformers import TransformersChatGenerator
|
||||
# NamedEntityExtractor (Hugging Face backend) was renamed to TransformersNamedEntityExtractor
|
||||
from haystack_integrations.components.extractors.transformers import TransformersNamedEntityExtractor
|
||||
# ExtractiveReader was renamed to TransformersExtractiveReader
|
||||
from haystack_integrations.components.readers.transformers import TransformersExtractiveReader
|
||||
@@ -0,0 +1,60 @@
|
||||
# SPDX-FileCopyrightText: 2022-present deepset GmbH <[email protected]>
|
||||
#
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
"""
|
||||
Pre-commit hook that enforces reStructuredText inline-code syntax in reno release notes.
|
||||
|
||||
Release notes under ``releasenotes/notes/*.yaml`` are rendered as reStructuredText, where inline
|
||||
code must use double backticks (``like_this``). A common mistake is to use Markdown-style single
|
||||
backticks (`like_this`), which RST renders as italic "interpreted text" instead of code.
|
||||
|
||||
By default the hook rewrites single backticks to double backticks and exits non-zero when it
|
||||
changes a file, so the rewrite can be reviewed before re-staging; ``--check`` only reports.
|
||||
It leaves existing double backticks ``code`` untouched and skips RST roles (:func:`x`) and
|
||||
hyperlink references (`text <url>`_).
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import re
|
||||
import sys
|
||||
|
||||
# A single-backtick inline-code span that should use double backticks. The look-arounds skip
|
||||
# valid RST: ``...`` pairs (the backtick-adjacent guards), roles like :func:`x` (no ':' before
|
||||
# the opening backtick), and hyperlink refs like `text <url>`_ (no '_' after the closing one).
|
||||
SINGLE_BACKTICK_RE = re.compile(r"(?<![`:])`(?!`)([^`\n]+?)`(?![`_])")
|
||||
|
||||
|
||||
def fix_text(text: str) -> str:
|
||||
"""Return ``text`` with single-backtick inline code rewritten to double backticks."""
|
||||
return SINGLE_BACKTICK_RE.sub(r"``\1``", text)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
"""Entry point for the pre-commit hook."""
|
||||
parser = argparse.ArgumentParser(description="Enforce double backticks in reno release notes.")
|
||||
parser.add_argument("--check", action="store_true", help="Report problems without modifying files.")
|
||||
parser.add_argument("files", nargs="*")
|
||||
args = parser.parse_args()
|
||||
|
||||
ret = 0
|
||||
for path in args.files:
|
||||
with open(path, encoding="utf-8") as f:
|
||||
original = f.read()
|
||||
fixed = fix_text(original)
|
||||
if fixed == original:
|
||||
continue
|
||||
ret = 1
|
||||
print(f"{'single backtick in' if args.check else 'fixed'}: {path}", file=sys.stderr)
|
||||
if not args.check:
|
||||
with open(path, "w", encoding="utf-8") as f:
|
||||
f.write(fixed)
|
||||
|
||||
if ret:
|
||||
hint = "Run without --check to fix automatically." if args.check else "Review and re-stage the changes."
|
||||
print(f"\nrelease notes need double backticks (``like_this``) for inline code. {hint}", file=sys.stderr)
|
||||
return ret
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -137,6 +137,100 @@ class TestAzureOpenAIChatGenerator:
|
||||
assert component.azure_ad_token_provider is not None
|
||||
assert component.max_retries == 0
|
||||
|
||||
def test_init_with_secret_azure_endpoint_and_api_version(self, monkeypatch):
|
||||
"""`azure_endpoint` and `api_version` accept a Secret that is resolved from an environment variable."""
|
||||
monkeypatch.setenv("AZURE_OPENAI_API_KEY", "test-api-key")
|
||||
monkeypatch.setenv("AZURE_OPENAI_ENDPOINT", "https://test-resource.azure.openai.com/")
|
||||
monkeypatch.setenv("AZURE_OPENAI_API_VERSION", "2024-08-01-preview")
|
||||
component = AzureOpenAIChatGenerator(
|
||||
azure_endpoint=Secret.from_env_var("AZURE_OPENAI_ENDPOINT"),
|
||||
api_version=Secret.from_env_var("AZURE_OPENAI_API_VERSION"),
|
||||
)
|
||||
# The Secret objects are kept on the instance so they can be serialized
|
||||
assert component.azure_endpoint == Secret.from_env_var("AZURE_OPENAI_ENDPOINT")
|
||||
assert component.api_version == Secret.from_env_var("AZURE_OPENAI_API_VERSION")
|
||||
# The clients receive the resolved string values
|
||||
assert str(component.client._azure_endpoint) == "https://test-resource.azure.openai.com/"
|
||||
assert component.client._api_version == "2024-08-01-preview"
|
||||
assert str(component.async_client._azure_endpoint) == "https://test-resource.azure.openai.com/"
|
||||
assert component.async_client._api_version == "2024-08-01-preview"
|
||||
|
||||
def test_init_fail_with_unset_secret_azure_endpoint(self, monkeypatch):
|
||||
"""A Secret azure_endpoint that resolves to nothing raises the same error as a missing endpoint."""
|
||||
monkeypatch.setenv("AZURE_OPENAI_API_KEY", "test-api-key")
|
||||
monkeypatch.delenv("AZURE_OPENAI_ENDPOINT", raising=False)
|
||||
with pytest.raises(ValueError, match="Azure endpoint"):
|
||||
AzureOpenAIChatGenerator(azure_endpoint=Secret.from_env_var("AZURE_OPENAI_ENDPOINT", strict=False))
|
||||
|
||||
def test_to_dict_with_secret_azure_endpoint_and_api_version(self, monkeypatch):
|
||||
"""Secret `azure_endpoint` and `api_version` are serialized as Secret dictionaries."""
|
||||
monkeypatch.setenv("AZURE_OPENAI_API_KEY", "test-api-key")
|
||||
monkeypatch.setenv("AZURE_OPENAI_ENDPOINT", "https://test-resource.azure.openai.com/")
|
||||
monkeypatch.setenv("AZURE_OPENAI_API_VERSION", "2024-08-01-preview")
|
||||
component = AzureOpenAIChatGenerator(
|
||||
azure_endpoint=Secret.from_env_var("AZURE_OPENAI_ENDPOINT"),
|
||||
api_version=Secret.from_env_var("AZURE_OPENAI_API_VERSION"),
|
||||
)
|
||||
init_params = component.to_dict()["init_parameters"]
|
||||
assert init_params["azure_endpoint"] == {
|
||||
"type": "env_var",
|
||||
"env_vars": ["AZURE_OPENAI_ENDPOINT"],
|
||||
"strict": True,
|
||||
}
|
||||
assert init_params["api_version"] == {
|
||||
"type": "env_var",
|
||||
"env_vars": ["AZURE_OPENAI_API_VERSION"],
|
||||
"strict": True,
|
||||
}
|
||||
|
||||
def test_secret_azure_endpoint_and_api_version_roundtrip(self, monkeypatch):
|
||||
"""Serializing and deserializing a component with Secret endpoint/version restores the Secrets."""
|
||||
monkeypatch.setenv("AZURE_OPENAI_API_KEY", "test-api-key")
|
||||
monkeypatch.setenv("AZURE_OPENAI_ENDPOINT", "https://test-resource.azure.openai.com/")
|
||||
monkeypatch.setenv("AZURE_OPENAI_API_VERSION", "2024-08-01-preview")
|
||||
component = AzureOpenAIChatGenerator(
|
||||
azure_endpoint=Secret.from_env_var("AZURE_OPENAI_ENDPOINT"),
|
||||
api_version=Secret.from_env_var("AZURE_OPENAI_API_VERSION"),
|
||||
)
|
||||
deserialized = AzureOpenAIChatGenerator.from_dict(component.to_dict())
|
||||
assert deserialized.azure_endpoint == Secret.from_env_var("AZURE_OPENAI_ENDPOINT")
|
||||
assert deserialized.api_version == Secret.from_env_var("AZURE_OPENAI_API_VERSION")
|
||||
assert str(deserialized.client._azure_endpoint) == "https://test-resource.azure.openai.com/"
|
||||
assert deserialized.client._api_version == "2024-08-01-preview"
|
||||
|
||||
def test_from_dict_with_secret_azure_endpoint_and_api_version(self, monkeypatch):
|
||||
"""from_dict deserializes Secret azure_endpoint/api_version dicts and resolves them for the client."""
|
||||
monkeypatch.setenv("AZURE_OPENAI_API_KEY", "test-api-key")
|
||||
monkeypatch.setenv("AZURE_OPENAI_ENDPOINT", "https://test-resource.azure.openai.com/")
|
||||
monkeypatch.setenv("AZURE_OPENAI_API_VERSION", "2024-08-01-preview")
|
||||
data = {
|
||||
"type": "haystack.components.generators.chat.azure.AzureOpenAIChatGenerator",
|
||||
"init_parameters": {
|
||||
"api_key": {"env_vars": ["AZURE_OPENAI_API_KEY"], "strict": False, "type": "env_var"},
|
||||
"azure_ad_token": {"env_vars": ["AZURE_OPENAI_AD_TOKEN"], "strict": False, "type": "env_var"},
|
||||
"azure_endpoint": {"env_vars": ["AZURE_OPENAI_ENDPOINT"], "strict": True, "type": "env_var"},
|
||||
"api_version": {"env_vars": ["AZURE_OPENAI_API_VERSION"], "strict": True, "type": "env_var"},
|
||||
"azure_deployment": "gpt-4.1-mini",
|
||||
"organization": None,
|
||||
"streaming_callback": None,
|
||||
"generation_kwargs": {},
|
||||
"timeout": 30.0,
|
||||
"max_retries": 5,
|
||||
"default_headers": {},
|
||||
"tools": None,
|
||||
"tools_strict": False,
|
||||
"azure_ad_token_provider": None,
|
||||
"http_client_kwargs": None,
|
||||
},
|
||||
}
|
||||
generator = AzureOpenAIChatGenerator.from_dict(data)
|
||||
# The Secret dicts are deserialized back into Secret objects
|
||||
assert generator.azure_endpoint == Secret.from_env_var("AZURE_OPENAI_ENDPOINT")
|
||||
assert generator.api_version == Secret.from_env_var("AZURE_OPENAI_API_VERSION")
|
||||
# And they are resolved to the string values the client expects
|
||||
assert str(generator.client._azure_endpoint) == "https://test-resource.azure.openai.com/"
|
||||
assert generator.client._api_version == "2024-08-01-preview"
|
||||
|
||||
def test_to_dict_default(self, monkeypatch):
|
||||
monkeypatch.setenv("AZURE_OPENAI_API_KEY", "test-api-key")
|
||||
component = AzureOpenAIChatGenerator(azure_endpoint="some-non-existing-endpoint")
|
||||
|
||||
@@ -0,0 +1,90 @@
|
||||
# SPDX-FileCopyrightText: 2022-present deepset GmbH <[email protected]>
|
||||
#
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
import importlib.util
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
_SCRIPT = Path(__file__).resolve().parents[1] / "scripts" / "release_note_backticks.py"
|
||||
_spec = importlib.util.spec_from_file_location("release_note_backticks", _SCRIPT)
|
||||
_module = importlib.util.module_from_spec(_spec)
|
||||
_spec.loader.exec_module(_module)
|
||||
fix_text = _module.fix_text
|
||||
|
||||
|
||||
class TestFixText:
|
||||
@pytest.mark.parametrize(
|
||||
"text, expected",
|
||||
[
|
||||
("Use `OpenAIChatGenerator` now.", "Use ``OpenAIChatGenerator`` now."),
|
||||
("Set `api_key` and `azure_endpoint`.", "Set ``api_key`` and ``azure_endpoint``."),
|
||||
('Call `Secret.from_env_var("X")`.', 'Call ``Secret.from_env_var("X")``.'),
|
||||
("Leading `code` token.", "Leading ``code`` token."),
|
||||
],
|
||||
)
|
||||
def test_converts_single_to_double(self, text, expected):
|
||||
assert fix_text(text) == expected
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"text",
|
||||
[
|
||||
"Already correct: ``OpenAIChatGenerator``.",
|
||||
"Two literals ``Secret`` and ``api_key``.",
|
||||
"No inline code at all here.",
|
||||
"RST role :func:`do_thing` must stay single.",
|
||||
"Link `Haystack <https://haystack.deepset.ai>`_ must stay single.",
|
||||
],
|
||||
)
|
||||
def test_leaves_valid_rst_untouched(self, text):
|
||||
assert fix_text(text) == text
|
||||
|
||||
def test_only_single_backticks_in_mixed_text_are_converted(self):
|
||||
text = "Use ``Secret`` for `api_key` and `azure_endpoint`."
|
||||
expected = "Use ``Secret`` for ``api_key`` and ``azure_endpoint``."
|
||||
assert fix_text(text) == expected
|
||||
|
||||
def test_is_idempotent(self):
|
||||
once = fix_text("Set `x` and `y`.")
|
||||
assert fix_text(once) == once
|
||||
|
||||
def test_unbalanced_single_backtick_is_left_untouched(self):
|
||||
# A stray, unpaired backtick is ambiguous, so we never rewrite it.
|
||||
text = "An unbalanced `backtick stays as is.\n"
|
||||
assert fix_text(text) == text
|
||||
|
||||
|
||||
class TestCli:
|
||||
def test_fix_rewrites_file_and_is_idempotent(self, tmp_path):
|
||||
note = tmp_path / "note.yaml"
|
||||
note.write_text("enhancements:\n - |\n Use `Foo` and `Bar` now.\n", encoding="utf-8")
|
||||
|
||||
first = subprocess.run([sys.executable, str(_SCRIPT), str(note)], capture_output=True, text=True, check=False)
|
||||
assert first.returncode == 1
|
||||
assert "``Foo``" in note.read_text(encoding="utf-8")
|
||||
assert "``Bar``" in note.read_text(encoding="utf-8")
|
||||
|
||||
# Running again on the now-fixed file is a no-op and succeeds.
|
||||
second = subprocess.run([sys.executable, str(_SCRIPT), str(note)], capture_output=True, text=True, check=False)
|
||||
assert second.returncode == 0
|
||||
|
||||
def test_check_mode_reports_without_modifying(self, tmp_path):
|
||||
note = tmp_path / "note.yaml"
|
||||
content = "enhancements:\n - |\n Use `Foo` now.\n"
|
||||
note.write_text(content, encoding="utf-8")
|
||||
|
||||
result = subprocess.run(
|
||||
[sys.executable, str(_SCRIPT), "--check", str(note)], capture_output=True, text=True, check=False
|
||||
)
|
||||
assert result.returncode == 1
|
||||
assert note.read_text(encoding="utf-8") == content
|
||||
|
||||
def test_clean_file_passes(self, tmp_path):
|
||||
note = tmp_path / "note.yaml"
|
||||
note.write_text("enhancements:\n - |\n Use ``Foo`` now.\n", encoding="utf-8")
|
||||
|
||||
result = subprocess.run([sys.executable, str(_SCRIPT), str(note)], capture_output=True, text=True, check=False)
|
||||
assert result.returncode == 0
|
||||
Reference in New Issue
Block a user