### Motivation and Context
Adding support for AI context behaviors to Semantic Kernel.
These allow creating plugins that are able to listen to messages being
added to the chat history and contribute to the AI context on each
invocation. This PR also integrates support for these with Agents and
AgentThreads.
#10100#10712
---------
Co-authored-by: SergeyMenshykh <[email protected]>
Co-authored-by: Roger Barreto <[email protected]>
### Motivation and Context
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
Multi-agent orchestration ADR. Includes the following orchestrations:
- Concurrent
- Sequential
- Group Chat
- Magentic One
- Handoff
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
An ADR for multi-agent orchestration in Semantic Kernel.
.Net PR: https://github.com/microsoft/semantic-kernel/pull/11542
Python PR: https://github.com/microsoft/semantic-kernel/pull/11993
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄
### Motivation and Context
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
The mongo link in one of the ADR's was gone, replaced.
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [ ] The code builds clean without any errors or warnings
- [ ] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [ ] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄
### Motivation and Context
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
This PR adds the ability to stand up your Kernel as a MCP Server, it
returns the Server object, which you can then run as either a Stdio or
Sse server.
Includes a sample to demonstrate.
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄
---------
Co-authored-by: Nico Möller <[email protected]>
### Motivation and Context
This PR introduces declarative agent support by adding new abstractions
and definitions for configuring agents via YAML and by updating existing
agent constructs.
- New model, connection, tool, metadata, input, factory, definition, and
creation option classes have been added.
- The access modifier for the Template property in Agent has been
updated to public.
- Multiple samples for ChatCompletion, OpenAIAssistant, and Bedrock
agents demonstrate declarative agent creation via YAML.
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [ ] The code builds clean without any errors or warnings
- [ ] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [ ] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄
---------
Co-authored-by: Chris <[email protected]>
Co-authored-by: Dmytro Struk <[email protected]>
Co-authored-by: westey <[email protected]>
Co-authored-by: Copilot <[email protected]>
### Motivation and Context
- Resolves Partially #10099
- Resolves#10733
- Resolves#10734
Developers need a clear example of how to integrate database operations
with Semantic Kernel for AI-powered data interactions. The
StructuredDataConnector demo provides a practical implementation showing
how to leverage SK's plugin architecture with Entity Framework Core for
structured data operations.
### Description
- Implemented `StructuredDataConnector` demo showcasing database-AI
integration
- Added `StructuredDataService<TContext>` for handling basic CRUD
operations
- Created `StructuredDataPluginFactory` for easy plugin creation
- Included example using OpenAI chat completion with function calling
- Added interactive mode for testing database queries through natural
language
The demo provides a reference implementation for developers wanting to
integrate database operations with SK's AI capabilities.
### Motivation and Context
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
This PR fixes links that were found unreachable by `linkspector`:
https://github.com/microsoft/semantic-kernel/actions/runs/13954045797/job/39063140670?pr=11055
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄
### Motivation and Context
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
This change improves readability by fixing typos in decision documents
2. What problem does it solve?
Existing typos in documentation
3. What scenario does it contribute to?
Developer documentation?
4. If it fixes an open issue, please link to the issue here.
### Description
This PR fixes typos across various ADRs that I noticed while reading
about Agent changes 😄
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [X] The code builds clean without any errors or warnings
- [X] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [X] All unit tests pass, and I have added new tests where possible
- [X] I didn't break anyone 😄
### Motivation and Context
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
ADR describing the preliminary design for realtime API's.
As these API's themselves are in preview, we might have to deal with
incompatible changes down the road.
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄
### Motivation and Context
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [ ] The code builds clean without any errors or warnings
- [ ] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [ ] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄
### Motivation and Context
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
We got a report stating that there was still a old limit on the number
of files supplied to the Azure Assistant API.
This PR fixes that and also does some further cleanup of the code.
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄
---------
Co-authored-by: Chris <[email protected]>
### Motivation and Context
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
The current link check action (github-action-markdown-link-check) is no
longer being maintained, because the developer created a new one, this
PR adds the new one side by side to test.
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄
### Motivation and Context
Today, every AI connector in SK that supports function calling has its
own implementation of tool call behavior model classes. These classes
are used to configure the way connectors advertise and invoke functions.
For example, the behavior classes can describe which functions should be
advertised to LLM by a connector, whether the functions should be called
automatically by the connector, or if the connector caller will invoke
them himself manually, etc.
All the tool call behavior classes are the same in terms of describing
the desired function call behavior. However, the classes have a mapping
functionality to map the function call behavior to the
connector-specific model classes, and that's what makes the function
calling classes non-reusable between connectors. For example, [the
constructor of the ToolCallBehavior
class](https://github.com/microsoft/semantic-kernel/blob/0c40031eb917bbf46c9af97897051f45e4084986/dotnet/src/Connectors/Connectors.OpenAI/ToolCallBehavior.cs#L165)
references the
[OpenAIFunction](https://github.com/microsoft/semantic-kernel/blob/main/dotnet/src/Connectors/Connectors.OpenAI/AzureSdk/OpenAIFunction.cs)
class that lives in the Microsoft.SemanticKernel.Connectors.OpenAI
namespace declared in the Connectors.OpenAI project. The classes can't
be reused by, let's say, the Mistral AI connector without introducing an
explicit project dependency from the Connectors.Mistral project to the
Connectors.OpenAI project, a dependency we definitely don't want to
have.
Additionally, today, it's not possible to specify function calling
behavior declaratively in YAML, MD, and Kernel(config.json) prompts.
### Description
This ADR outlines design options (there's only one at the moment) to
address the issues above. Related to
https://github.com/microsoft/semantic-kernel/issues/5253
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [ ] The code builds clean without any errors or warnings
- [ ] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [ ] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄
### Motivation, Context and Description
This PR merges the AOT-related changes to make
`SemanticKernel.Abstractions`, `SemanticKernel.Core`, and
`Connectors.Onnx` packages Native-AOT compatible into the main branch.
Closes: https://github.com/microsoft/semantic-kernel/issues/5411
---------
Co-authored-by: Mark Wallace <[email protected]>
### Motivation and Context
Feature branch for the vector and text search work.
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [ ] The code builds clean without any errors or warnings
- [ ] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [ ] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄
---------
Co-authored-by: Dmytro Struk <[email protected]>
Co-authored-by: westey <[email protected]>
Co-authored-by: Weihan Li <[email protected]>
Co-authored-by: Roger Barreto <[email protected]>
# Connectors Versioning Strategy for Underlying SDKs
## Context and Problem Statement
This week (01-10-2024) OpenAI and Azure OpenAI released their first
stable version and we need to bring some options ahead of us regarding
how to move forward with the versioning strategy for the next releases
of `OpenAI` and `AzureOpenAI` connectors which will also set the path
moving forward with other connectors and providers versioning
strategies.
This ADR brings different options how we can move forward thinking on
the impact on the users and also how to keep a clear message on our
strategy.
Currently, Azure Open AI GA package against what we were expecting
choose remove many of the features previously available in preview
packages from their first GA version.
This also requires us to rethink how we are going to proceed with our
strategy for the following versions of our connectors.
| Name | SDK NameSpace | Semantic Kernel NameSpace |
| ------------------- | --------------- |
----------------------------------------------- |
| OpenAI (OAI) | OpenAI | Microsoft.SemanticKernel.Connectors.OpenAI |
| Azure OpenAI (AOAI) | Azure.AI.OpenAI |
Microsoft.SemanticKernel.Connectors.AzureOpenAI |
## Decision Drivers
- Minimize the impact of customers
- Allow customers to use either GA or Beta versions of OpenAI and
Azure.AI.OpenAI packages
- Keep a clear message on our strategy
- Keep the compatibility with the previous versions
- Our package versioning should make it clear which version of OpenAI or
Azure.AI.OpenAI packages we depend on
- Follow the Semantic Kernel versioning strategy in a way that
accommodates well with other SDK version strategies.
## Considered Options
1. **Keep As-Is** - Target only preview packages.
2. **Preview + GA versioning** (Create a new version (GA + pre-release)
side by side of the Azure OpenAI and OpenAI Connectors).
3. **Stop targeting preview packages**, only target GA packages moving
forward.
## 1. Keep As-Is - Target only preview packages
This option will keep the current strategy of targeting only preview
packages, which will keep the compatibility with the previous versions
and new GA targeting versions and pipelines for our customers. This
option has the least impact on our users and our pipeline strategy.
Today all customers that are already using Azure OpenAI Connector have
their pipelines configured to use the preview packages.
```mermaid
%%{init: { 'logLevel': 'debug', 'theme': 'base', 'gitGraph': {'showBranches': true, 'showCommitLabel':true,'mainBranchName': 'SemanticKernel'}} }%%
gitGraph TB:
checkout SemanticKernel
commit id:"SK 1.21"
branch OpenAI
commit id:"OAI 2.0-beta.12"
branch AzureOpenAI
commit id:"AOAI 2.0-beta.6"
checkout SemanticKernel
merge OpenAI id:"SK OAI 1.22"
merge AzureOpenAI id:"SK AOAI 1.22"
checkout OpenAI
commit id:"OAI 2.0 GA"
checkout AzureOpenAI
merge OpenAI id:"AOAI 2.0 GA"
checkout SemanticKernel
commit id:"Skipped GA's"
checkout OpenAI
commit id:"OAI 2.1-beta.1"
checkout AzureOpenAI
commit id:"AOAI 2.1-beta.1"
checkout SemanticKernel
merge OpenAI id:"SK OAI 1.23"
merge AzureOpenAI id:"SK AOAI 1.23"
```
Pros:
- No changes in strategy. (Least impact on customers)
- Keep the compatibility with the previous versions and new GA targeting
versions and pipelines.
- Compatible with our previous strategy of targeting preview packages.
- Azure and OpenAI SDKs will always be in sync with new GA versions,
allowing us to keep the targeting preview with the latest GA patches.
Cons:
- There won't be a SK connector version that targets a stable GA package
for OpenAI or AzureOpenAI.
- New customers that understand and target GA only available features
and also have a strict requirement for dependent packages to be also GA
will not be able to use the SK connector. (We don't have an estimate but
this could be very small compared to the number of customers that are
already OK on using the preview Azure SDK OpenAI SDK available for the
past 18 months)
- Potential unexpected breaking changes introduced by OpenAI and
Azure.AI.OpenAI beta versions that eventually we might be passing on due
to their dependency.
## 2. Preview + GA versioning
This option we will introduce pre-release versions of the connectors:
1. General Available (GA) versions of the connector will target a GA
version of the SDK.
2. Pre-release versions of the connector will target a pre-release
versions of the SDK.
This option has some impact for customers that were targeting strictly
only GA packages on their pipeline while using preview features that are
not available anymore on underlying SDK GA versions.
All preview only functionalities not available in the SDK will be
Annotate in Semantic kernel connectors with an Experimental `SKEXP0011`
dedicated identifier attribute, to identify and clarify the potential
impact when attempting to move to a `GA` package.
Those annotations will be removed as soon as they are officially
supported on the GA version of the SDK.
```mermaid
%%{init: { 'logLevel': 'debug', 'theme': 'base', 'gitGraph': {'showBranches': true, 'showCommitLabel':true,'mainBranchName': 'SemanticKernel'}} }%%
gitGraph TB:
checkout SemanticKernel
commit id:"SK 1.21"
branch OpenAI
commit id:"OAI 2.0-beta.12"
branch AzureOpenAI
commit id:"AOAI 2.0-beta.6"
checkout SemanticKernel
merge OpenAI id:"SK OAI 1.22-beta"
merge AzureOpenAI id:"SK AOAI 1.22-beta"
checkout OpenAI
commit id:"OAI 2.0 GA"
checkout AzureOpenAI
merge OpenAI id:"AOAI 2.0 GA"
checkout SemanticKernel
merge OpenAI id:"SK OAI 1.23"
merge AzureOpenAI id:"SK AOAI 1.23"
checkout OpenAI
commit id:"OAI 2.1-beta.1"
checkout AzureOpenAI
merge OpenAI id:"AOAI 2.1-beta.1"
checkout SemanticKernel
merge OpenAI id:"SK OAI 1.23-beta"
merge AzureOpenAI id:"SK AOAI 1.23-beta"
checkout OpenAI
commit id:"OAI 2.1-beta.2"
checkout AzureOpenAI
merge OpenAI id:"AOAI 2.1-beta.2"
checkout SemanticKernel
merge OpenAI id:"SK OAI 1.24-beta"
checkout SemanticKernel
merge AzureOpenAI id:"SK AOAI 1.24-beta"
```
Pros:
- We send a clear message moving forward regarding what Azure and OpenAI
consider stable and what is not, exposing only stable features from
those SDKs in what we previously were considering as GA available
features.
- New customers that have a strict requirement for dependent packages to
be also GA will be able to use the SK connector.
- We will be able to have preview versions of Connectors for new
features that are not yet GA without impacting the GA versions of the
Connectors.
Cons:
- This change our strategy for versioning, needing to some clear
clarification and communication for the first releases to mitigate
impact or smooth the transition.
- Customers that were using `OpenAI` and `AzureOpenAI` preview only
features available in previous SK GA packages will need to update their
pipelines to target only future SK pre-release versions.
- Small Overhead to maintain two versions of the connectors.
### Version and Branching Strategy
Create a special release branch for the targeted `GA` version of the
connector, keeping it in the record for that release with all
modifications/removal that all the other projects need to make to work
with the stable release this will be also a important guideline on where
and when to add/remove the `SKEXP0011` exceptions from API's samples.
We will follow our own version cadence with the addition of `beta`
prefix for `beta` versions of the underlying SDKs.
| Seq | OpenAI Version | Azure OpenAI Version | Semantic Kernel
Version<sup>1</sup> | Branch |
| --- | -------------- | -------------------- |
----------------------------------- | --------------- |
| 1 | 2.0.0 | 2.0.0 | 1.25.0 | releases/1.25.0 |
| 2 | 2.1.0-beta.1 | 2.1.0-beta.1 | 1.26.0-beta | main |
| 3 | 2.1.0-beta.3 | 2.1.0-beta.2 | 1.27.0-beta | main |
| 4 | No changes | No changes | 1.27.1-beta<sup>**2**</sup> | main |
| 5 | 2.1.0 | 2.1.0 | 1.28.0 | releases/1.28.0 |
| 6 | 2.2.0-beta.1 | 2.1.0-beta.1 | 1.29.0-beta | main |
1. Versions apply for the **Connectors packages** and the **Semantic
Kernel meta package**.
2. No changes on the SDKs but other minor changes to Semantic Kernel
code base that needed a version update.
### Optional Smoothing Transition
In the intend to smooth the transition and mitigate impact on customers
using preview features on SK GA packages straight away we would provide
a notice period where we give the time for customers adapt to the
`preview` vs `GA` future releases of the connector packages. While for
the notice duration we would maintain our strategy with the **Keep
As-Is** option before shifting to the **Preview + GA versioning**
option.
## 3. Stop targeting preview packages
> [!WARNING]
> This option is not recommended but needs to be considered.
This option will stop targeting preview packages, being strict with our
1.0 GA strategy, not exposing our customers to non-GA SDK features.
As big features like Azure Assistants are still in preview, this option
will have a big impact on our customers if they were targeting Agent
frameworks and other important features that are not yet General
Available. Described in
[here](https://github.com/Azure/azure-sdk-for-net/releases/tag/Azure.AI.OpenAI_2.0.0)
> Assistants, Audio Generation, Batch, Files, Fine-Tuning, and Vector
Stores are not yet included in the GA surface; they will continue to be
available in preview library releases and the originating Azure OpenAI
Service api-version labels.
```mermaid
%%{init: { 'logLevel': 'debug', 'theme': 'base', 'gitGraph': {'showBranches': true, 'showCommitLabel':true,'mainBranchName': 'SemanticKernel'}} }%%
gitGraph TB:
checkout SemanticKernel
commit id:"SK 1.21.1"
branch OpenAI
commit id:"OAI 2.0.0-beta.12"
branch AzureOpenAI
commit id:"AOAI 2.0.0-beta.6"
checkout OpenAI
commit id:"OAI 2.0.0 GA"
checkout SemanticKernel
merge OpenAI id:"SK OAI 1.22.0"
checkout AzureOpenAI
merge OpenAI id:"AOAI 2.0.0 GA"
checkout SemanticKernel
merge AzureOpenAI id:"SK AOAI 1.22.0"
checkout OpenAI
commit id:"OAI 2.1.0-beta.1"
checkout AzureOpenAI
commit id:"AOAI 2.1.0-beta.1"
checkout OpenAI
commit id:"OAI 2.1.0 GA"
checkout SemanticKernel
merge OpenAI id:"SK OAI 1.23.0"
checkout AzureOpenAI
commit id:"AOAI 2.1.0 GA"
checkout SemanticKernel
merge AzureOpenAI id:"SK AOAI 1.23.0"
```
Pros:
- As we have been only deploying GA versions of the connector, strictly
we would be following a responsible GA only approach with GA SK packages
not exposing customers to preview features as GA features at all.
Cons:
- Big impact on customers that are targeting preview features with no
option to resort to a preview version of the connector.
- This strategy will render the use of the Semantic Kernel with
Assistants and any other preview feature in Azure impractical.
## Decision Outcome
Chosen option: **Keep as is**
As the current AI landscape for SDK is a fast changing environment, we
need to be able be update and at the same time avoid as much as possible
mix our current versioning strategy also minimizing the impact on
customers. We decided on **Keep As-Is** option for now, and we may
reconsider **Preview + GA versioning** option in the future when that
decision doesn't bring big impact to our customer base.
### Motivation and Context
This PR introduces the design for supporting OpenAI's Structured Outputs
`response_format` within the Semantic Kernel Python SDK. While OpenAI’s
previous JSON mode improved model reliability, it still lacked strict
schema adherence, which led to inefficiencies and the need for custom
workarounds.
By integrating Structured Outputs, this ADR outlines a design that
ensures generated AI outputs will conform to predefined JSON schemas,
improving consistency, reducing errors, and eliminating the need for
retries or custom tools. This design will provide a more streamlined and
reliable approach for developers building schema-compliant AI-driven
applications.
In the future we will have a separate ADR around OpenAI's structured
outputs for function calling. That is out of scope for this ADR.
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [X] The code builds clean without any errors or warnings
- [X] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [X] All unit tests pass, and I have added new tests where possible
- [X] I didn't break anyone 😄
### Motivation and Context
OpenAI assistants were introduced several versions ago; however, there
was open work to support streaming for OpenAI assistants v2. Work to
support streaming assistants was also needed in Agent Group Chat
scenarios.
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
### Description
This PR introduces:
- Support for streaming with (Azure) OpenAI assistants
- Support for Agent Chat Scenarios that can now leverage a streaming
invoke instead of just a sync invoke.
- Unit tests to increase the test coverage to near 100% for the newly
added code
- Adds a few more OpenAI assistant agent samples
- Closes#7267
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [X] The code builds clean without any errors or warnings
- [X] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [X] All unit tests pass, and I have added new tests where possible
- [X] I didn't break anyone 😄
### Motivation and Context
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
OpenAI recently starts providing token usage information on their
streaming chat completion API. This ADR opens the discussion on how we
should consume that information within our `StreamingChatMessageContent`
data structure.
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
1. An ADR documenting 4 approaches.
2. Implementation of the selected approach.
3. Fix issues where data in the streaming contents are modified after
concatenation (`__add__`).
4. Add streaming to model diagnostics operation names so that we can
distinguish streaming and non-streaming operations.
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄
---------
Co-authored-by: Ben Thomas <[email protected]>
### Motivation and Context
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
Related: https://github.com/microsoft/semantic-kernel/issues/7946
Closes: https://github.com/microsoft/semantic-kernel/issues/8519
[Structured
Outputs](https://platform.openai.com/docs/guides/structured-outputs) is
a feature in OpenAI API that ensures the model will always generate
responses based on provided JSON Schema. This gives more control over
model responses, allows to avoid model hallucinations and write simpler
prompts without a need to be specific about response format.
This PR contains an ADR which describes several options how to enable
this functionality in .NET version of Semantic Kernel and implementation
for Option 1 and Option 2.
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄
---------
Co-authored-by: Eirik Tsarpalis <[email protected]>
# Motivation and Context
This PR brings support for Azure AI Studio Model Catalogs also deployed
thru GitHub Models, this Connector uses the `Azure AI Inference SDK`
library client.
Closes#3992Closes#7958
### Motivation and Context
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
This PR implements:
https://github.com/microsoft/semantic-kernel/blob/main/docs/decisions/0052-python-ai-connector-new-abstract-methods.md
(PR: https://github.com/microsoft/semantic-kernel/pull/8430).
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
1. Add abstract methods `_inner_get_chat_message_content` and
`_inner_get_streaming_chat_message_content` to
`ChatCompletionClientBase`.
2. Implement the abstractions in all chat completion connectors.
3. Add abstract methods `_inner_get_text_contents` and
`_inner_get_streaming_text_contents` to `TextCompletionClientBase`.
4. Implement the abstractions in all text completion connectors.
5. Remove text completion APIs from `OllamaChatCompletion` (breaking
changes).
> No breaking changes on other connectors except Ollama, and public APIs
stay the same.
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄
### Motivation and Context
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
This ADR documents a proposal to introduce four new abstract methods in
`ChatCompletionClientBase` and `TextCompletionClientBase` to optimize
code and further streamline the process of adding new AI connectors.
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄
### Motivation and Context
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
Resolves: https://github.com/microsoft/semantic-kernel/issues/7317
This ADR contains an information about implementing Entity Framework
memory connector, which potentially could work with multiple databases
at the same time and its current limitations.
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄
### Motivation and Context
The current MemoryStore abstractions have limitations around consuming
existing data stores with custom schemas.
This PR adds a new abstraction layer that supports custom schemas and
data models, removes opinionated behaviors from connectors, and aims to
support more of the features available in each database.
This set of abstractions do not yet contain any support for Searching,
and this will be added to the IVectorStoreRecordCollection
interface soon.
See the included ADR document for more information.
### Description
- Adding IVectorStore and IVectorStoreRecordCollection interfaces, plus
associated options classes, attributes and definition classes.
- Adding an AzureAISearch implementation.
- Adding a Qdrant implementation
- Adding a Redis implementation with both JSON and Hashes support
- Adding a Pinecone implementation
- Adding a Volatile (In memory) implementation
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄
---------
Co-authored-by: Maurycy Markowski <[email protected]>
Co-authored-by: Dmytro Struk <[email protected]>
### Motivation and Context
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
This ADR captures agent concepts as the project solidifies. Will be
updated as needed to remain _in sync_.
While the code exists as _ground-truth_, capturing the following
additional vectors can aid with translation to _Python_ and _Java_:
- Conceptual overview
- Objectives / requirements
- Analysis
- Intent
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
This merges the initial _Agent_ ADR with the current direction with the
goal of capturing project essence for translation into _Python_ and
_Java_
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄
---------
Co-authored-by: Evan Mattson <[email protected]>
### Motivation and Context
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
Users of the Agent Framework are unable to store and later retrieve
conversation state when using an AgentChat to coordinate Agent
interactions. This limits the ability for an agent conversation to
single use as it must be maintained with memory of the process that
initiated the conversation.
Formalizing a mechanism that supports serialization and deserialization
of any AgentChat class provides an avenue to capture and restore state
across multiple sessions as well as compute boundaries.
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
Design in ADR form for input, exploration of alternatives, and review.
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [X] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [X] I didn't break anyone 😄
### Motivation and Context
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
Open AI has release the _Assistants V2_ API. This builds on top of the
V1 _assistant_ concept, but also invalidates certain V1 features. In
addition, the _dotnet_ API that supports _Assistant V2_ features is
entirely divergent on the `Azure.AI.OpenAI.Assistants` SDK that is
currently in use.
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
ADR describes changes to `Microsoft.SemanticKernel.Agents.OpenAI`
project / package.
Changes prototyped in this PR:
https://github.com/microsoft/semantic-kernel/pull/7126
PR completion is gated by the V2 connector migration work due to
versioning constraints.
> ADR Reviewed on July 18th, 2024
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [X] The code builds clean without any errors or warnings
- [X] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [X] All unit tests pass, and I have added new tests where possible
- [X] I didn't break anyone 😄
---------
Co-authored-by: Evan Mattson <[email protected]>
### Motivation and Context
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
All builds failing due to spellcheck violation: **serializeable** =>
**serializable**
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
Changed three instances of **serializeable** to **serializable**
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [X] The code builds clean without any errors or warnings
- [X] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [X] All unit tests pass, and I have added new tests where possible
- [X] I didn't break anyone 😄
# OpenAI and Azure Connectors Naming and Structuring
Resolves#6529
## Context and Problem Statement
It has recently been announced that OpenAI and Azure will each have
their own dedicated SDKs for accessing their services. Previously, there
was no official SDK for OpenAI, and our OpenAI Connector relied solely
on the Azure SDK client for access.
With the introduction of the official OpenAI SDK, we now have access to
more up-to-date features provided by OpenAI, making it advantageous to
use this SDK instead of the Azure SDK.
Additionally, it has become clear that we need to separate the OpenAI
connector into two distinct targets: one for OpenAI and another for
Azure OpenAI. This separation will enhance code clarity and facilitate a
better understanding of the usage of each target.
## Decision Drivers
- Avoiding breaking changes whenever possible.
- Develop a new dedicated connector for Azure OpenAI, utilizing the
underlying Azure SDK.
- Minimize or eliminate any breaking changes for developers currently
using the existing OpenAI connector.
## Versioning
Although current `Azure.AI.OpenAI` and `OpenAI` SDKs have being update
on its major versions, that change does not represents a
`SemanticKernel` major breaking change, any options below consider that
he new updated version of `SemanticKernel.Connectors.OpenAI` will be a
minor version bump `1.N+1.0`.
### Meta Package Strategy
Currently the `Microsoft.SemanticKernel` package is a meta package that
includes both `SemanticKernel.Core` and
`SemanticKernel.Connectors.OpenAI`, with the new changes a new project
will be added to the meta package
`SemanticKernel.Connectors.AzureOpenAI` that will include the new Azure
OpenAI connector.
## Documentation (Upgrade Path)
A documentation guidance and samples/examples will be created to guide
on how to upgrade from the current OpenAI connector to the new when
needed.
## OpenAI SDK limitations
The new OpenAI SDK introduce some limitations that need to be considered
and pontentially can introduce breaking changes if not remediated by our
internal implementation.
- #### ⚠️ No support for multiple results (Choices) per request.
**Remediation**: Internally make the multiple requests and combine them.
**No remediation**: Breaking change removing `ResultsPerPrompt` from
`OpenAIPromptExecutionSettings`.
- #### ⚠️ Text Generation modality is not supported.
**Remediation**: Internally provide a HttpClient to be used against
`gpt-3.5-turbo-instruct` for text generation modality. Same way was done
for `TextToImage`, `AudioToText` service modalities.
**No remediation**: Breaking change removing any specific
`TextGeneration` service implementations, this change don't impact
`ChatCompletion` services that may still being used as
`ITextGenerationService` implementations.
## Improvements
This also represents an opportunity to improve the current OpenAI
connector by introducing the `Configuration` pattern to allow more
flexibility and control over the services and their configurations.
```csharp
// Before
builder.AddAzureOpenAIChatCompletion(deploymentName, endpoint, apiKey, httpClient);
// After
builder.AddAzureOpenAIChatCompletion(new
{
DeploymentName = modelId;
Endpoint = endpoint;
ApiKey = apiKey;
});
```
```csharp
// Before
builder.AddAzureOpenAIChatCompletion(deploymentName, openAIClient, serviceId, modelId)
// After
builder.AddAzureOpenAIChatCompletion(new
{
DeploymentName = deploymentName;
ServiceId = serviceId;
ModelId = modelId;
}, openAIClient);
```
## Considered Options
- Option 1 - Slow transition.
- Option 2 - Independent Connectors from Start.
- Option 3 - Update OpenAI with Azure as is.
## Option 1 - Slow Transition
This is the least breaking approach where we keep the current legacy
OpenAI and AzureOpenAI APIs temporarily in the connector using last
Azure SDK `Azure.AI.OpenAI 1.0.0-beta.17` and add new OpenAI specific
APIs using the new `OpenAI 2.0.0-beta.*` SDK package.
This approach also implies that a new connector will be created on a
second moment for Azure OpenAI services specifically fully dependent on
the latest `Azure.AI.OpenAI 2.0.0-beta.*` SDK package.
In a later stage we will deprecate all the OpenAI and Azure legacy APIs
in the `SemanticKernel.Connectors.OpenAI` namespace and remove Azure SDK
`Azure.AI.OpenAI 1.0.0-beta.17` and those APIs in a future release,
making the OpenAI Connector fully dedicated for OpenAI services only
depending on with the `OpenAI 2.0.0-beta.*` dependency.
```mermaid
graph TD
A[SemanticKernel.Connectors.OpenAI] --> B[OpenAI 2.0.0-beta.*]
A --> C[Azure.OpenAI 1.0.0-beta.17]
D[SemanticKernel.Connectors.AzureOpenAI] --> E[Azure.AI.OpenAI 2.0.0-beta.*]
```
The new `Options` pattern we be used as an improvement as well as a
measure to avoid breaking changes with the legacy APIs.
Following this change the the `SemanticKernel.Connectors.OpenAI` a new
connector will be created `SemanticKernel.Connectors.AzureOpenAI` for
Azure OpenAI services, using the Azure SDK `Azure.AI.OpenAI
2.0.0-beta.*` with all new APIs using the options approach.
### Phases of the transition
- **Phase 1**: Add new OpenAI SDK APIs to the current OpenAI connector
and keep the Azure OpenAI APIs using the last Azure SDK.
- **Phase 2**:
- Create a new connector for Azure OpenAI services using the new Azure
SDK
- Deprecate all Azure OpenAI APIs in the `OpenAI` connector pointing to
new `AzureOpenAI` connector
- Remove Azure SDK dependency from the OpenAI connector.
- Add `AzureOpenAI` connector to the `Microsoft.SemanticKernel` meta
package.
- **Phase 3**: Deprecate all legacy `OpenAI APIs` in the `OpenAI`
connector pointing to new `Options` APIs.
- **Phase 4**: Remove all legacy APIs from the OpenAI connector.
### Impact
Pros:
- Minimal breaking changes for developers using the current OpenAI
connector.
- Clear separation of concerns between OpenAI and Azure OpenAI
connectors.
Cons:
- Since `SemanticKernel.Connectors.AzureOpenAI` and
`SemanticKernel.Connectors.OpenAI` share a same dependency of different
versions, both packages cannot be used in the same project and a
strategy will be needed when deploying both connectors:
- Added dependency for both `Azure OpenAI 1.0-beta17` and `OpenAI
2.0-beta1`.
### Dependency Management Strategies
1. Use only one of the connectors in the same project, some
modifications will be needed to accomodate `Concepts` and other projects
that shares OpenAI and AzureOpenAI examples.
2. Hold AzureOpenAI connector implementation until we are ready to break
(exclude) all Azure APIs in OpenAI connector.
3. Deploy a new project with a new namespace for `Azure.AI.OpenAI.Legacy
1.0.0-beta.17` and update our `SemanticKernel.Connectors.OpenAI` to use
this new namespace to avoid version clashing on the `Azure.AI.OpenAI`
namespace.
## Option 2 - Independent Connectors from Start.
This option is focused on creating fully independent connectors for
OpenAI and Azure OpenAI services since the start with all breaking
changes needed to achieve that.
Impact:
- `Azure` extension methods will be removed from
`SemanticKernel.Connectors.OpenAI` namespace to avoid clashing with same
names introduced in the new `SemanticKernel.Connectors.AzureOpenAI`.
- All azure specific services like `AzureOpenAIChatCompletion`,
`AzureOpenAITextGeneration` services will be obsoleted and throw not
supported exceptions when used from `SemanticKernel.Connectors.OpenAI`
connector.
### Impact
Pros:
- Clear separation of concerns between OpenAI and Azure OpenAI
connectors.
- Small breaking changes for developers focused on OpenAI specific APIs.
- Faster transition to the new OpenAI SDK and Azure OpenAI SDK.
Cons:
- Large breaking changes for developers using the current OpenAI
connector for Azure.
- ⚠️ May have the same problem with the dependency management as in
Option 1 when using both connectors in the same project may conflict due
to the shared dependency of different OpenAI SDK versions.
## Option 3 - Update OpenAI with Azure as is.
This option is fully focused in the least impact possible, combining
both Azure and OpenAI SDK dependencies in one single connector following
the same approach as the current connector.
Changes:
1. Update all current OpenAI specific services and client to use new
OpenAI SDK
2. Update Azure specific services and client to use the latest Azure
OpenAI SDK.
3. Optionally add `Options` pattern new APIs to the connector services
and deprecate old ones.
### Impact
Pros:
- Minimal breaking changes for developers using the current OpenAI
connector.
- The breaking changes will be limited on how we tackle the points
mentioned in the [OpenAI SDK limitations](#openai-sdk-limitations)
above.
Cons:
- We will be limited on the OpenAI SDK version that is used by the
latest `Azure.AI.OpenAI` package, which may not be the latest version
available.
- Developers using direct OpenAI services don't expected having any
Azure related dependencies.
## Decision Outcome
#### Chosen option:
TBD
#### OpenAI SDK limitations:
- Multiple results: **Do not remediate**.
- Text Generation modality is not supported: **Do not remediate**.
### Motivation and Context
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
Related to: https://github.com/microsoft/semantic-kernel/issues/6530,
https://github.com/microsoft/semantic-kernel/issues/3992
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [ ] The code builds clean without any errors or warnings
- [ ] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [ ] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄
Corrected the capitalization of the page header/title. I also removed
the manually typed math and replaced it with inline LaTeX display math
to make the formulas easier to read as well as to add a more
professional look.
For some reason, when using the `$$` LaTeX delimiter, GitHub sometimes
cuts off the vertical portion of large square root symbols. I sort of
see that happening here with the second formula, but when I zoom in and
out, it corrects itself, so I'm unsure if it's how the browser is
drawing the formula, or if the issue originates solely with GitHub. If
it's present on the screen of any reviewers, please let me know and I'll
switch the code to use the single `$` LaTeX delimiter which doesn't have
this problem. The only downfall with the single `$` delimiter is that it
defaults to left alignment instead of center alignment, but I'm pretty
sure that can be manually adjusted.
Take Care
### Motivation and Context
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required? The manually typed math looked a tad
sloppy and was a bit difficult to read.
2. What problem does it solve? It makes the math formulas easier to read
and understand because LaTeX formats the formulas properly as well as
providing accessibility options for users who need to tweak how the
formulas are displayed.
3. What scenario does it contribute to? Being able to read the documents
without any confusion.
4. If it fixes an open issue, please link to the issue here.
-->
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄
### Motivation and Context
Closes#6535
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [ ] The code builds clean without any errors or warnings
- [ ] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [ ] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄
---------
Co-authored-by: Roger Barreto <[email protected]>
### Motivation and Context
⚠️ Breaking changes on non-experimental types **ImageContent**
Resolves#5625Resolves#5295
For a brief time this changes will keep the content below as
experimental.
- BinaryContent
- AudioContent
- ImageContent
- FunctionCallContent
- FunctionResultContent
Changes:
### **BinaryContent**
- Removed providers for lazy loading content, simplifying its usage and
APIs.
- Removed `Stream` constructor to avoid IDisposable resource consumption
or bad practices.
- Added `Uri` dedicated for Referenced Uri information
- Added `DataUri` property which can be set or get dynamically (auto
generated if you created the
content using byte array with a mime type)
Setting a `DataUri` will automatically update the `MimeType` property
and add any extra metadata that may be available in the data scheme
definition.
- Added a required `mimeType` property to the ByteArray constructor, to
encourage passing the mimeType when creating BinaryContent directly or
from specializations.
- Added `Data` property which can be set or get dynamically (auto
generated if you created the content using a data uri format)
Setting a Data on an existing BinaryContent will also reflect on the
getter of `DataUri` for the given content.
- Added DataUri and Base64 validation when setting DataUri on the
contents.
- When using DataUri parameters those merge with the current content
metadata.
i.e:
`data:image/jpeg;parameter1=value1;parameter2=value2;base64,binary==`
### ⚠️ **ImageContent** Fixes bugs and inconsistency behavior:
- Setting the Data of an image doesn't change the current data uri and
vice versa, allowing the sema image content instance to have different
binary data to representations.
- When an Image could not have DataUri and Uri in the same instance,
this limits scenarios where you have the image data but want to have a
reference to where that content is from.
- Wasn't possible to create an Image from a data uri greater than the
size limit of .Net System.Uri type here:
[https://github.com/dotnet/runtime/issues/96544](https://github.com/dotnet/runtime/issues/96544).
### **FunctionResultContent**
- Update `Id` property to `CallId`.
### Motivation and Context
1. This change is required to ensure all documentation links are
up-to-date.
2. It solves the problem of users encountering broken or outdated links.
3. It contributes to the scenario of maintaining accurate and helpful
documentation for users.
4. N/A
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
### Description
Updated several obsolete links in the documentation to point to the
current and correct URLs. This ensures users have access to the most
recent and relevant information.
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄
### Motivation and Context
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
Implements dealing with unsafe content, by doing HTML parsing on
variables and function results.
Closes: #5889
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
Adds parameter `allow_dangerously_set_content` to:
- InputVariable
- PromptTemplateConfig
- PromptTemplateBase
The behavior is that if the flag is set to True on the template itself
(KernelPromptTemplate, Jinja2PromptTemplate or HandlebarsPromptTemplate)
the behavior is the same, no encoding is done on inputs.
Otherwise:
- variables are encoded by default, this can be switched off using the
InputVariables class for that variable.
- function output is encoded by default, this can be switched off using
the flag in the PromptTemplateConfig, this is not yet possible to do on
a per function basis.
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄
