mirror of
https://github.com/microsoft/semantic-kernel.git
synced 2026-10-12 02:19:06 +00:00
0195778be25892cba507fe593d5592b71c2adf68
371
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
d84363f477 |
Python: Migrate to new Google GenAI SDK (#13371)
### Motivation and Context <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> Closes #12970, #13352 ### Description <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> 1. Migrate Google AI connector to use the new Google AI SDK. 2. This PR also marks the VertexAI connector as deprecated in favor of the Google AI connector. With this update, if users wants to connect to a VertexAI project instead of a Google AI endpoint, they can do so with the Google AI connector. 3. This PR also adds the ability for users to provide a custom client to the Google AI connector (#13352). 4. This PR also marks `AzureTextCompletion` deprecated. ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [ ] I didn't break anyone 😄 |
||
|
|
145ad9bfd2 |
Python: Upgrade Onnx Connector to use 0.9.0 (#13162)
### Motivation and Context Fixes : #13001 ### Description Package version 0.9.0 introduced very helpful methods like applying chat templates automatically from the tokenizer and also support for multi audio & image files. This PR adds the following functionalities: - [x] Add Inference with multiple Images - [x] Add Inference with multiple Audios - [x] Chat Templates for non Multimodal Models are read via onnx ### Samples Text Sample with ONNX: <img width="1728" height="90" alt="image" src="https://github.com/user-attachments/assets/b93d9fc1-4e38-4fa9-b535-cc316e0900ed" /> Image Sample with ONNX : <img width="1721" height="91" alt="image" src="https://github.com/user-attachments/assets/47761beb-b728-4936-a6b6-fd253e041689" /> ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄 --------- Co-authored-by: Evan Mattson <[email protected]> |
||
|
|
1c11258a32 |
Python: added af_tool bridge for kernel_functions (#13227)
### Motivation and Context <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> Adds a new method to KernelFunction that creates a Agent Framework Tool from that function. Does require the user to install agent-framework-core themselves. ### Description <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄 |
||
|
|
e73446e86e |
Python: [Python] Updated Agents SDK version to latest for CopilotStudio connector (#13144)
### Motivation and Context
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
- The Agents SDK has been released, and the recent release changed the
namespace from `microsoft.agents` to `microsoft_agents`, which breaks
imports.
2. What problem does it solve?
- This updates the `pyproject.toml` and copilot studio related files to
fix the imports
3. What scenario does it contribute to?
- Using a Copilot Studio agent as an Agent in Semantic Kernel
4. If it fixes an open issue, please link to the issue here.
- N/A
-->
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
- Updated pyproject.toml to use the released packages for the Agents
SDK.
- Updated imports in the `copilot_studio_agent` related files to use the
new `microsoft_agents` namespace
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [] I didn't break anyone 😄
|
||
|
|
28ea2f4df8 |
Python: Fix AzureAIAgent readme links (#13138)
### Motivation and Context Current getting started with agents README links for AzureAIAgent are broken. <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> ### Description - Fixes #13129 <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [X] The code builds clean without any errors or warnings - [X] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [X] All unit tests pass, and I have added new tests where possible - [X] I didn't break anyone 😄 |
||
|
|
61d3e6ee8d |
Python: Introduce NvidiaChatCompletion AI Connector (#12952)
### Motivation and Context <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? Enables developers to use NVIDIA's chat models through Semantic Kernel with structured output support. 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> This change adds chat completion support with structured output to the existing NVIDIA connector, enabling developers to use NVIDIA's chat models through Semantic Kernel while building conversational AI applications that require structured, validated responses. The connector will support more model types iteratively, with future PRs introducing VLM models and additional capabilities. ### Description <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> - New Chat Completion Service: `NvidiaChatCompletion`: Chat completion service following same structure as other connectors - Extends existing NVIDIA embedding connector patterns and architecture - Custom logic for handling structured output specific to NVIDIA models - Enhanced Configuration: `NvidiaChatPromptExecutionSettings`: Chat-specific settings following existing connector patterns ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄 |
||
|
|
d7db35dc47 |
Python: Fix Azure AI Inference connector model_extras duplication (#13066)
### Motivation and Context <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> Addresses #13005 ### Description <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> 1. Remove the `extra_parameters` key from `AzureAIInferenceChatPromptExecutionSettings` before calling the chat completion API to avoid unknown parameter error. 2. Allow setting the API version on the `AzureAIInferenceXXX` connectors. ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄 |
||
|
|
f7fdfcd9a9 |
Python: Add AzureAIAgent Deep Research Tool support (#13034)
### Motivation and Context The deep research tool was introduced for the AzureAIAgent; however, we didn't have support yet in SK. Adding support now for using the deep research tool for both streaming and non-streaming. Samples added as well. <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> ### Description - Add support for deep research tool. - Add samples with notes on prerequisites to configure the sample to run properly. - Added support for `AZURE_AI_AGENT_DEEP_RESEARCH_MODEL` env var within `AzureAIAgentSettings` class. - Bumped min allowed `azure-ai-agents` package ver to 1.2.0b3. - Closes #12978 <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [X] The code builds clean without any errors or warnings - [X] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [X] All unit tests pass, and I have added new tests where possible - [X] I didn't break anyone 😄 |
||
|
|
86b10b95d2 |
Python: provide AzureAIAgent thread msg id during streaming (#13032)
### Motivation and Context As part of invoking an AzureAIAgent via streaming, we weren't providing the `thread_id` via the `AgentStreamEvent.THREAD_MESSAGE_CREATED` event. <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> ### Description Supply the caller with the `thread_msg_id` by sticking it into the returned SCMC's `metadata` dictionary. - Closes #12959 Sample output looks like: ``` Sample Output: # User: 'Hello' # AuthorRole.ASSISTANT: (thread message id: msg_HZ2h4Wzbj7GEcnVCjnyEuYWT) Hello! How can I assist you with the menu today? # User: 'What is the special soup?' # AuthorRole.ASSISTANT: (thread message id: msg_TSjkJK6hHJojIkPvF6uUofHD) The special soup today is Clam Chowder. Would you like to know more about it or anything else from the menu? # User: 'How much does that cost?' # AuthorRole.ASSISTANT: (thread message id: msg_liwTpBFrB9JpCM1oM9EXKiwq) The Clam Chowder costs $9.99. Is there anything else you'd like to know? # User: 'Thank you' # AuthorRole.ASSISTANT: (thread message id: msg_K6lpR3gYIHethXq17T6gJcxi) You're welcome! If you have any more questions or need assistance, feel free to ask. Enjoy your meal! ``` <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [X] The code builds clean without any errors or warnings - [X] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [X] All unit tests pass, and I have added new tests where possible - [X] I didn't break anyone 😄 |
||
|
|
5e50e190c4 |
Python: Add reasoning support for OpenAI Responses Agents (GPT-5, o4-mini, o3) (#12881)
## Summary Adds reasoning capabilities to Python OpenAI ResponsesAgent, bringing parity with the C# implementation. This enables fine-grained control over reasoning effort for O-series models (o1, o3-mini, o4-mini) and gpt-5 that support the reasoning parameter. Fixes #12843 ## Changes ### Core Implementation - **`openai_responses_agent.py`**: Added reasoning configuration support - Constructor-level reasoning effort setting - Per-invocation reasoning effort override capability - Proper parameter validation and model compatibility checks - **`responses_agent_thread_actions.py`**: Extended thread actions to support reasoning parameters - Reasoning effort propagation through thread operations - Metadata preservation for reasoning tokens and summaries ### Sample and Tests - **`responses_agent_reasoning.py`**: Comprehensive demonstration sample - Constructor vs per-invocation reasoning configuration - Function calling integration with reasoning - Reasoning comparison scenarios (low/medium/high effort) - Error handling and troubleshooting guidance - **`test_openai_responses_agent_reasoning.py`**: Full unit test coverage - Parameter validation tests - Integration scenarios with function calling - Edge cases and error conditions ## Features ✅ **Constructor-Level Reasoning**: Set default reasoning effort when creating agents ✅ **Per-Invocation Override**: Override reasoning effort per request ✅ **Priority Hierarchy**: per-invocation > constructor > model default ✅ **Function Calling Compatible**: Works seamlessly with existing plugin system ✅ **Azure OpenAI & OpenAI Support**: Compatible with both service providers ✅ **Model Validation**: Automatic compatibility checks for O-series models nad GÜT-5 ✅ **Metadata Access**: Reasoning tokens and summaries available in response metadata ## Usage Example ```python # Constructor-level reasoning # Constructor-level reasoning configuration agent = AzureResponsesAgent( ai_model_id="gpt-5", client=client, reasoning={"effort": "low"} # Default reasoning for all requests ) # Per-invocation override response = await agent.invoke( "Solve this complex problem step by step", reasoning={"effort": "high"} ) # Invoke with reasoning callback to capture intermediate thoughts response = await agent.invoke( "Analyze this data step by step", reasoning={"effort": "high", "summary": "detailed"}, on_intermediate_message=handle_reasoning_message ) # Streaming with reasoning async for response in agent.invoke_stream( "Explain quantum computing in detail", reasoning={"effort": "high", "summary": "detailed"}, on_intermediate_message=handle_reasoning_message ): print(response.content, end="", flush=True) --------- Co-authored-by: Evan Mattson <[email protected]> Co-authored-by: Evan Mattson <[email protected]> |
||
|
|
18d9a8e8b2 |
Python: .Net: Updated encoding logic in prompt templates (#12983)
### Motivation and Context <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> Resolves: https://github.com/microsoft/semantic-kernel/issues/11821 Today, the encoding of template arguments is performed only if argument type is `string`. In case of custom type, anonymous type or collection - the encoding is not performed. This PR contains changes to throw an exception in case if encoding is enabled but complex type is used. In case of complex type, the encoding should be performed manually according to business logic and automatic encoding should be explicitly disabled. This enforces stricter, but more secure template rendering rules. **Note**: this is a breaking change for customers who use Handlebars or Liquid template with complex type arguments. Code changes are required when initializing template arguments: ```diff var arguments = new KernelArguments() { { "customer", new { - firstName = userInput.FirstName, - lastName = userInput.LastName, + firstName = HttpUtility.HtmlEncode(userInput.FirstName), + lastName = HttpUtility.HtmlEncode(userInput.LastName), } } }; var templateFactory = new LiquidPromptTemplateFactory(); var promptTemplateConfig = new PromptTemplateConfig() { TemplateFormat = "liquid" + InputVariables = new() + { + // We set AllowDangerouslySetContent to 'true' because each property of this argument is encoded manually. + new() { Name = "customer", AllowDangerouslySetContent = true }, + } }; var promptTemplate = templateFactory.Create(promptTemplateConfig); var renderedPrompt = await promptTemplate.RenderAsync(kernel, arguments); ``` ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [ ] I didn't break anyone 😄 |
||
|
|
2a47ba9b7d |
Python: Removed usage of DefaultAzureCredential (#12964)
### Motivation and Context <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> The usage of `DefaultAzureCredential` by default is unsafe and raises security issues. Instead, users should provide their own credential types such as `AzureCliCredential`, `ManagedIdentityCredential` and so on. More information here: https://learn.microsoft.com/en-us/python/api/azure-identity/azure.identity.aio?view=azure-python In this PR: 1. Removed the usage of `DefaultAzureCredential` from authentication logic. 2. Replaced `DefaultAzureCredential` with `AzureCliCredential` in all examples. **Note**: this is breaking change for Azure users, who relies on `DefaultAzureCredential`. Users will need to update the code and pass the credential type of their choice by using `credential` parameter when initializing one of the Azure clients. Before: ```python chat_completion = AzureChatCompletion() ``` After: ```python chat_completion = AzureChatCompletion(credential=AzureCliCredential()) ``` ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [ ] I didn't break anyone 😄 |
||
|
|
96bad5ceac |
Python: Update MCP streamable sample (#12950)
### Motivation and Context The MCPStreamableHttpPlugin concept sample pointing to GitHub isn't working. Updating it to use the Microsoft Learn site MCP server instead. <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> ### Description Update sample. <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [X] The code builds clean without any errors or warnings - [X] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [X] All unit tests pass, and I have added new tests where possible - [X] I didn't break anyone 😄 |
||
|
|
6416c58b90 |
Python: Add arguments and results attributes to execute tool span (#12940)
### Motivation and Context <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> We are adding two new attributes to the execute tool span according to this proposal to Otel Semantic Conventions: https://github.com/open-telemetry/semantic-conventions/pull/2528 ### Description <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> The following two attributes are added to the `execute_tool` span: 1. `gen_ai.tool.call.arguments` 2. `gen_ai.tool.call.result` ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄 |
||
|
|
8336c320d9 |
Python: Add bedrock model provider parameter (#12853)
### Motivation and Context <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> Fixes #12833 ### Description <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> Add an optional setting to the Bedrock AI connectors to allow users to specify the Bedrock model providers (i.e. amazon, anthropic, etc) so that they can use application inference profiles. ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄 --------- Co-authored-by: Dmytro Struk <[email protected]> |
||
|
|
e5972e2430 |
Python: Add incomplete details reason to AzureAIAgent error message (#12796)
### Motivation and Context When errors occur processing a run of an AzureAIAgent, we surface the error message but not the incomplete_details reason if it exists. <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> ### Description This PR adds handling for the incomplete details reason, if it exists, so it can be passed back via the error message. - Closes #12752 <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [X] The code builds clean without any errors or warnings - [X] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [X] All unit tests pass, and I have added new tests where possible - [X] I didn't break anyone 😄 |
||
|
|
b7bb2832aa |
Python: Input & output attributes in invoke_agent span (#12834)
### Motivation and Context <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> The OTel workgroup has been thinking about enhancing tracing for agents. This is a proposal from our tracing team: https://github.com/open-telemetry/semantic-conventions/pull/2528. This PR implements part of the proposal so that when the proposal gets merged, we can quickly follow. ### Description <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> Two new attributes are added to the `invoke_agent` span, namely the `gen_ai.agent.invocation_input` and `gen_ai.agent.invocation_output`. <img width="747" height="583" alt="image" src="https://github.com/user-attachments/assets/d8b81f68-915b-48ee-8f86-66a0d4f686cc" /> ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄 --------- Co-authored-by: Evan Mattson <[email protected]> |
||
|
|
c16db27622 |
Python: Update README.md (#12737)
Update links due to renamed files ### Motivation and Context <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> ### Description <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [x] The code builds clean without any errors or warnings - [ ] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [ ] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄 --------- Co-authored-by: Evan Mattson <[email protected]> |
||
|
|
1efc7ad04e |
Python: Pass through arguments when creating agents from specs (#12771)
### Motivation and Context Currently, attempting to pass through KernelArguments through agent creation via declarative spec isn't working properly. The arguments are used to normalize fields, but are then not passed through fully to where the agent is created. <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> ### Description Pass arguments through, and merge them properly with the normalized fields. This allows for underlying args to be used by the agent, like in the case of structured outputs. - Closes #12770 <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [X] The code builds clean without any errors or warnings - [X] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [X] All unit tests pass, and I have added new tests where possible - [X] I didn't break anyone 😄 |
||
|
|
b723fbd5e1 |
Python: Add mix agent orchestration sample (#12749)
### Motivation and Context <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> We don't have a sample showing how to use a mix of agents of various types in an orchestration. This PR adds one. A bug is also discovered in the OpenAIResponsesAgent while creating the sample. ### Description <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> 1. Add a new sample showing how to use a mix of agents in the Handoff orchestration. 2. Fix a bug in the OpenAIResponsesAgent where the tools were not considering the tools in the kernel instance. A new test is added too. 3. Fix a bug in the AgentActorBase where the intermediate message callback was rejecting ChatMeesageContent type. ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄 |
||
|
|
c79f5a5c63 |
Python: improved MCP connect and additional samples (#12696)
### Motivation and Context <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> Adding a sample using StreamableHTTPServer towards Github. Added a demo showing how to connect to a OAuth protected MCP server. Also some improvements in the connect logic for MCP. ### Description <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄 |
||
|
|
48f71b1f2e |
Python: Support AzureAI agent MCP tools for streaming and non-streaming invocations (#12736)
### Motivation and Context MCP tool support was released for the AzureAI Agent and it has yet to be added to the SK Python AzureAIAgent. This PR adds functionality to handle the MCP tools for both streaming and non-streaming invocations. <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> ### Description Adds MCP support for the Python AzureAI Agent. - Upgrades dependent packages to be able to use the mcp tool from the Azure SDK. <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [X] The code builds clean without any errors or warnings - [X] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [X] All unit tests pass, and I have added new tests where possible - [X] I didn't break anyone 😄 |
||
|
|
9e3cdffc6b |
Python: Improve AzureAIAgent get thread msg handling. (#12535)
### Motivation and Context Currently, when fetching thread messages for an AzureAIAgent, it assumes that the agent exists, so we can pull it down and get its id. This may not always be the case - an agent could have been deleted, but the thread still exists. We need to account for this. When creating user messages that we add to the thread, there's no `name` or `id` parameter allowed. We can, however, add metadata to the message. Going forward we are adding the `agent_id` as metadata to the message, so upon retrieval later, we can attempt to get it and return the id with the user's message. This allows one to see which agent_id has responded to the user's message: ``` # Messages in the thread (asc order): # AuthorRole.USER for name=asst_mXwZOwyJLxXGtaYKHizRH6Ip: Hello # AuthorRole.ASSISTANT for name=asst_mXwZOwyJLxXGtaYKHizRH6Ip: Hello! How can I assist you with the menu today? # AuthorRole.USER for name=asst_mXwZOwyJLxXGtaYKHizRH6Ip: What is the special soup? # AuthorRole.ASSISTANT for name=asst_mXwZOwyJLxXGtaYKHizRH6Ip: The special soup today is Clam Chowder. Would you like to know more about it or anything else on the menu? # AuthorRole.USER for name=asst_mXwZOwyJLxXGtaYKHizRH6Ip: How much does that cost? # AuthorRole.ASSISTANT for name=asst_mXwZOwyJLxXGtaYKHizRH6Ip: The Clam Chowder costs $9.99. Would you like to order it or need information on other items? # AuthorRole.USER for name=asst_mXwZOwyJLxXGtaYKHizRH6Ip: Thank you # AuthorRole.ASSISTANT for name=asst_mXwZOwyJLxXGtaYKHizRH6Ip: You're welcome! If you have any more questions or need assistance with the menu, feel free to ask. Enjoy your meal! ``` <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> ### Description - Closes #12435 <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [X] The code builds clean without any errors or warnings - [X] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [X] All unit tests pass, and I have added new tests where possible - [X] I didn't break anyone 😄 |
||
|
|
003cc4c010 |
Python: Feature python vector stores preb (#12271)
### Motivation and Context <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> Overhaul of the VectorStores to get it matching with dotnet and the latest development there. Closes: - #11938 - #11598 - #11597 - #11517 - #10561 - #10391 - #9911 - #9892 - #10867 ### Description <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> Many changes, some highlights: - Moved from different vector fields types, to a single with a enum first VectorStoreField() - Renamed several items - Made embeddinggenerrator a part of the vector field spec and do automatic vectorization from there or from the same param on a collection - Adds hybrid search using the same setup - Removed the intermediate TextSeachVectorSearch object need, by allowing `create_search_function` directly on a collection Will write out full changes in a migration guide in the docs. ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [ ] I didn't break anyone 😄 |
||
|
|
bd4168316d |
Python: add Foundry local sample (#12214)
### Motivation and Context <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> Sample on how to use Foundry Local with SK. ### Description <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄 |
||
|
|
206655adbb |
Python: Add check for not approved and approved for Agent Sample (#12316)
### Motivation and Context I was having a Demo with the Agents, and found that `step06_chat_completion_agent_group_chat.py` sample always terminated after one iteration. The Reviewer Agent answered "not approved", which also triggered the `Termination Strategy`, due to the fact that the word `approved` is also in `not approved`. To only trigger when there is a real approval I added a check if the answer is `approved` **and** the answer also don't include `not approve` ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄 --------- Co-authored-by: Chris <[email protected]> Co-authored-by: Evan Mattson <[email protected]> |
||
|
|
44f1253460 |
Python: Agent response callbacks to provide full context (#12501)
### Motivation and Context <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> Currently, the agent response callbacks in the `Orchestrations` are triggered only when the final responses from the agents become available. This limits developers to create solutions that can expose the internal processing of the agents in multi agent orchestrations. ### Description <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> The callbacks will now be triggered by the intermediate messages of the agents: 1. A new sample 2. Tests P.s. Fix a bug where the name of the intermediate messages is not assigned. ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄 |
||
|
|
9b8f8e22c3 |
Python: Correct chat completion agent history truncation samples (#12474)
### Motivation and Context <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> While addressing the developer in this [discussion](https://github.com/microsoft/semantic-kernel/discussions/12450), I notice that there are a few changes needed to the improve the truncation history reducer samples of the chat completion agent. ### Description <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> Update the samples. ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [ ] The code builds clean without any errors or warnings - [ ] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [ ] All unit tests pass, and I have added new tests where possible - [ ] I didn't break anyone 😄 |
||
|
|
dfbc993d8f |
Python: Remove orchestration persona adaptation messages (#12446)
### Motivation and Context <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> We previously found that agents might get carried away from in long contexts. The persona adaptation messages were designed to steer the agents in an orchestration to respond more closely to the instructions provided (as system message). As we perform more testing with newer models, we believe that the persona adaptation messages are no longer necessary. The additional instability (because they are hardcoded messages that devs can't override) and cost (the additional token cost which is not significant, but they do pollute the context which makes debugging difficult) they introduce is not worth the benefit they provide. Closing #12294 > Ideally, we'd want to approach this more analytically. The lack of a multi-agent evaluation process makes it difficult. All samples were verified to make sure they behave the same before and after the removal of the persona adaptation messages. ### Description <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> 1. Remove the persona adaptation messages. 2. Unit tests. 3. Replace `OpenAIChatCompletion` with `AzureChatCompletion` in some of the samples. ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄 |
||
|
|
4b23389f71 |
Python: Emit token usage with streaming chat completion agent. (#12416)
### Motivation and Context
The chat completion agent was not emitting token using during streaming
invocation because we were only allowing through `response.items`. In
the case of token usage, `response.items` is [] and the usage is
contained as part of the message's `metadata` dict. This PR fixes that
bug and allows for `response.items or response.metadata.get("usage")`.
Two new samples are added to the concepts/agents/chat_completion dir to
show how one can track token use for streaming and non-streaming agent
invocation. Token usage handling is also added to the chat completion
agent integration tests.
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
### Description
- Fixes a bug where we weren't emitting the streaming token usage for
the chat completion agent. Also now includes the `prompt_tokens_details`
and `completion_tokens_details` models that are returned, but not
previously handled.
- Adds new samples
- Updates integration tests to track token usage and make sure they're
non-zero.
- Closes #12411
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [X] The code builds clean without any errors or warnings
- [X] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [X] All unit tests pass, and I have added new tests where possible
- [X] I didn't break anyone 😄
|
||
|
|
3e31c3bdc6 |
Python: Remove planner-related code and samples. (#12417)
### Motivation and Context Per deprecation method, removing planner-related code after June 1, 2025. <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> ### Description - Removing planner-related code. - Closes #12400 <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [X] The code builds clean without any errors or warnings - [X] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [X] All unit tests pass, and I have added new tests where possible - [ ] I didn't break anyone 😄 |
||
|
|
e0221e58ce |
Python: Fix OpenAPI concept sample (#12418)
### Motivation and Context An OpenAPI concept sample that we have is broken. This PR fixes it to run properly based on the current OpenAPI plugin handling. <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> ### Description Fixes the broken OpenAPI sample. <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [X] The code builds clean without any errors or warnings - [X] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [X] All unit tests pass, and I have added new tests where possible - [X] I didn't break anyone 😄 |
||
|
|
0e7556e112 |
Python: Add file handling support to BinaryContent for OpenAI Responses API (#12258)
## Summary
Enhances `BinaryContent` to support file handling for OpenAI Responses
API, enabling file uploads through the responses agent while maintaining
a provider-agnostic design.
## Changes
### BinaryContent Enhancements
- **Add `can_read` property**: Indicates whether content has readable
data available
- **Add `from_file()` class method**: Creates BinaryContent instances
from file paths with automatic base64 encoding
- **Fix Unicode handling**: Prevents decode errors when processing
binary files (PDFs, images, etc.)
### OpenAI Responses Agent Integration
- **Add BinaryContent support**: Pattern matching case for file handling
in `responses_agent_thread_actions.py`
- **Correct OpenAI API format**: Uses proper `filename` and `file_data`
structure with data URI format
- **UUID-based filenames**: Generates appropriate filenames with
mime-type extensions
- **Provider-specific mapping**: File format conversion happens only in
OpenAI agent code
### Testing
- **Test coverage**: New tests for `can_read`, `from_file()`, and binary
data handling
- **Unicode error prevention**: Specific test for binary PDF-like
content
- **Base64 encoding verification**: Ensures proper data format for API
compatibility
## Design Principles
- **Provider-agnostic**: BinaryContent remains completely generic with
no OpenAI-specific dependencies
- **Clean separation**: OpenAI format mapping isolated to OpenAI agent
files
- **No FileContent class**: Enhances existing BinaryContent instead of
introducing new types
- **Follows existing patterns**: Similar approach to ImageContent and
TextContent handling
## Usage Example
```python
response = await self.agent.get_response(
messages=ChatMessageContent(
content="Analyse PDF",
role=AuthorRole("user"),
items=[BinaryContent.from_file(file_path=c":/test.pdf")],
)
)
Co-authored-by: Evan Mattson <[email protected]>
|
||
|
|
2f84e5d1bf |
Python: Add chat completion agent code interpreter sample (#12393)
### Motivation and Context <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> Closing #10962 ### Description <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> 1. Add a sample to demonstrate how to create a chat completion agent with code interpreter capabilities using the Azure Container Apps session pool service. 2. Fix a bug that cause the session tool to throw because the upload API doesn't return the file uploaded. 3. Modify unit tests accordingly. ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄 |
||
|
|
3baaee2ff3 |
Python: streaming agent response callback in agent orchestrations (#12360)
### Motivation and Context <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> Issue: #12306 Discussion: #12302 Currently, we only have a **non-streaming** agent response callback for users to interact with the agents inside an orchestration. Support for a stream one has been requested by developers. ### Description <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> Add support for a streaming agent response callback to all orchestrations. ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄 |
||
|
|
c822511f2c |
Python: Add special notes to samples that are still using AgentGroupchat (#12377)
### Motivation and Context <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> As we move away from `AgentGroupChat` to the new orchestrations, we want to make sure it's clear to developers getting started with SK are aware of the changes. ### Description <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> We are including a special note to all the samples that use `AgentGroupChat` to inform developers that this feature is no longer maintained and they should consider using `GroupChatOrchestration` instead. We are keeping the samples for now so that developers can still refer to them and we don't break any external links to these samples, but we want to make sure they are aware of the changes and can migrate to the new orchestrations. ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄 |
||
|
|
fb0e418a7c |
Python: Fix for AzureAIAgent streaming func call invocations. Other fixes/improvements for tool call content types, and logging. (#12335)
### Motivation and Context
The AzureAIAgent streaming function call invocation is broken when
handling more than one required function call from the service. There is
nested logic to handle passing the tool calls back to the service
because it requires one to pass in a handler.
In this PR, we're now controlling everything from the main loop and are
passing back the handler as the event stream for further processing.
This does require some logic to determine if we have an initial async
context manager (based on the first call to create the run stream) or if
we're now using the handler from submitting tool calls. We can now see
that when we have multiple functions involved, we are able to get the
results as we expect:
```
Sample Output:
# User: 'What is the price of the special drink and then special food item added together?'
Function Call:> MenuPlugin-get_specials with arguments: {}
Function Result:>
Special Soup: Clam Chowder
Special Salad: Cobb Salad
Special Drink: Chai Tea
for function: MenuPlugin-get_specials
Function Call:> MenuPlugin-get_item_price with arguments: {"menu_item": "Chai Tea"}
Function Call:> MenuPlugin-get_item_price with arguments: {"menu_item": "Clam Chowder"}
Function Result:> $9.99 for function: MenuPlugin-get_item_price
Function Result:> $9.99 for function: MenuPlugin-get_item_price
Function Call:> MathPlugin-Add with arguments: {"input":9.99,"amount":9.99}
Function Result:> 19.98 for function: MathPlugin-Add
# AuthorRole.ASSISTANT: The price of the special drink, Chai Tea, is $9.99 and the price of the special food
item, Clam Chowder, is $9.99. Added together, the total price is $19.98.
```
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
### Description
Fix handling multiple function calls for the streaming AzureAIAgent.
- Fixes a missing `quote` from the url citation for streaming annotation
content
- Shows the correct way to retrieve Bing Grounding tool calls in the two
concept samples.
- Adds handling for tool content content for OpenAPI plugins.
- Adds logging for streaming agent invocation tool calls.
- Relaxes the handling of the `FunctionResultContent` id attribute to be
`str | None` similar to that of `FunctionCallContent`. The type hints
for `id` have always been `str | None`, but the actual type hint on the
attribute has been `str`. We don't always get a `FunctionResultContent`
id from the Azure Agent Service during streaming invocations - message
addition via the overide `__add__` method still work properly.
- Move the AzureAIAgent pkg dependencies to be installed by default,
instead of requiring the `azure` extra.
- Closes #12312
- Closes #12328
- Closes #12331
- Closes #12324
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [X] The code builds clean without any errors or warnings
- [X] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [X] All unit tests pass, and I have added new tests where possible
- [X] I didn't break anyone 😄
|
||
|
|
82229cecc3 |
Python: Improve Python README (#12289)
### Motivation and Context The Python README file is a bit stale, and lacking current content related to agent framework and directly calling the AI service (not always needing to invoke via the kernel). Additionally, there are links to samples/notebooks that are relative and thus not resolving from pypi. The [`ALL_SETTINGS.md` guide](https://github.com/microsoft/semantic-kernel/blob/main/python/samples/concepts/setup/ALL_SETTINGS.md#semantic-kernel-settings) is also missing the Agent Framework settings used. Adding those in this PR. <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> ### Description Improve the README to be more relevant. - Closes #12286 <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [X] The code builds clean without any errors or warnings - [X] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [X] All unit tests pass, and I have added new tests where possible - [X] I didn't break anyone 😄 |
||
|
|
ff1997047d |
Python: Support Declarative Spec for OpenAIAssistantAgent & OpenAIResponsesAgent (#12247)
### Motivation and Context This PR builds on the initial work/abstractions from #11982. The `AzureAssistantAgent` / `OpenAIAssistantAgent` as well as the `AzureResponsesAgent` / `OpenAIResponsesAgent` now support declarative specs. Several examples on how to declare the spec are shown in the respective dirs in `samples/concepts/agents`, as well as a simple getting started example for each agent type. To further simplify working with these two agents, the `setup_resources()` methods for each agent type have been marked with a deprecation tag. The new method to use is `OpenAIAssistantAgent.create_client()` or `OpenAIResponsesAgent.create_client()`. Further, when one needs to access the underlying setting's, it's simple to do so with a settings object and accessing the attribute. In the previous PR for the AzureAIAgent, it lacked support for templating - this is included in this current PR. <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> ### Description Bring in support for the declarative single agent spec for these agent types. - Add samples - Add unit tests - Add support for prompt templates for all current agent types - Closes #11979 <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [X] The code builds clean without any errors or warnings - [X] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [X] All unit tests pass, and I have added new tests where possible - [X] I didn't break anyone 😄 |
||
|
|
7243188e92 |
Python: Use BaseModel instead of KernelBaseModel in samples (#12224)
### Motivation and Context <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> It's unnecessary to use `KernelBaseModel` in the samples. ### Description <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> Switch to `BaseModel`. ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄 |
||
|
|
c084b067de |
Python: Support structured outputs with Azure AI inference chat completion. (#12183)
### Motivation and Context The Azure AI inference SDK released support for structured outputs for chat completions (for models that support it). We had not brought this into SK. This PR adds functionality and the ability to run the structured output samples that exist today. <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> ### Description Adds functionality to handle structured outputs for the Azure AI Inference Chat Completion calls. - Fixes a bug within AzureAIInferenceChatCompletion where the wrong isinstance check type is used. This prevented adding tool calls during streaming invocation. Also properly add the `ai_model_id` from the connector during message construction during streaming invocation, - Adds unit tests for AzureAIInferenceChatCompletion structured outputs - Closes #11952 <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [X] The code builds clean without any errors or warnings - [X] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [X] All unit tests pass, and I have added new tests where possible - [X] I didn't break anyone 😄 |
||
|
|
c64e51c40e |
Python: Update the AzureAIAgent packages, types, and APIs based on Azure SDK's new code (#12084)
### Motivation and Context The Azure AI Agent Service is moving to GA soon. As part of that, they've made several changes in the underlying APIs, which require the use of new packages, updated type imports and some new underlying APIs. This PR updates the AzureAIAgent to use the latest Azure SDK's new code. - The client's connection string has been replaced with an endpoint. - The `AzureAIAgentSettings`'s `model_deployment_name` was updated to `deployment_name`. - Tools are now imported from `from azure.ai.agents.models` instead of `from azure.ai.projects.models`. - Declarative spec handling for AzureAIAgent was updated based on the new APIs. - Unit tests were updated. <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> ### Description Update the AzureAIAgent related code, samples and tests based on the Azure SDK's latest APIs. - Closes #12177 <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [X] The code builds clean without any errors or warnings - [X] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [X] All unit tests pass, and I have added new tests where possible - [ ] I didn't break anyone 😄 --------- Co-authored-by: Tao Chen <[email protected]> |
||
|
|
03f0aba633 |
Python: added WebRTC support for Azure OpenAI Realtime (#12078)
### Motivation and Context <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> Added support for WebRTC for Azure OpenAI Realtime models. ### Description <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄 |
||
|
|
0634c30241 |
Python: marked planners deprecated and removed all related items (#12058)
### Motivation and Context <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> In line with our dotnet, marking all planners as deprecated, and removing tests and samples using them. ### Description <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [ ] I didn't break anyone 😄 --------- Co-authored-by: Evan Mattson <[email protected]> |
||
|
|
d67f568d99 |
Python: Multi-agent orchestration: Magentic (#12104)
### Motivation and Context <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> PR for Magentic multi-agent orchestration. ADR: https://github.com/microsoft/semantic-kernel/blob/main/docs/decisions/0071-multi-agent-orchestration.md Initial PR: https://github.com/microsoft/semantic-kernel/pull/11993 ### Description <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> This PR includes the following: 1. Implementations for the magentic orchestrations 2. Unit tests and samples ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄 --------- Co-authored-by: Evan Mattson <[email protected]> |
||
|
|
d7c2489a24 |
Python: Add MultiAgent Orchestration to README. (#12073)
### Motivation and Context The new multi-agent orchestration samples are missing from the getting started with agents README. Add them in with their descriptions. Also, remove two unused intermediate message callback lists from two samples that are no longer used. <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> ### Description Add samples to README. Clean up some samples. <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [X] The code builds clean without any errors or warnings - [X] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [X] All unit tests pass, and I have added new tests where possible - [X] I didn't break anyone 😄 |
||
|
|
50cf782645 |
Python: Remove Copilot Studio Direct link from Pyproject.toml (#12068)
### Motivation and Context The pyproject.toml currently references Microsoft Copilot Studio packages that are only on test PyPI, and not yet available on the Public PyPI repo. This breaks the ability to push the public SK package to PyPI. We will, until public PyPI packages are available, remove the extra from our pyproject.toml and place explicit package install instructions in the Copilot Studio Agent README. <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> ### Description Explicit instructions on how to install the necessary packages to use the CopilotStudioAgent while the packages are on Test PyPI. <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [X] The code builds clean without any errors or warnings - [X] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [X] All unit tests pass, and I have added new tests where possible - [X] I didn't break anyone 😄 |
||
|
|
ee962943d6 |
Python: Multi-agent orchestration: Handoff (#12046)
### Motivation and Context <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> PR for Handoff multi-agent orchestration. ADR: https://github.com/microsoft/semantic-kernel/blob/main/docs/decisions/0071-multi-agent-orchestration.md Initial PR: https://github.com/microsoft/semantic-kernel/pull/11993 ### Description <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> This PR includes the following: 1. Implementations for the handoff orchestrations 2. Unit tests and samples Follow up PRs will add the following: - Magentic One Contribution Check ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄 |
||
|
|
6115e8d7dc |
Python: fixes and updates for MCP (#12039)
### Motivation and Context <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> Removes the two surpressed calls for extra info out of a MCP server, since they seemed to hang every now and then. Updated the package version spec Added one of the samples as a integration test Added: - load_tools flag on the constructor, default = true - load_prompt flag on the constructor, default = true - request_timeout param, optional, default None ### Description <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄 |
||
|
|
d826ad8d5a |
Python: Multi-agent orchestration: Group Chat (#12045)
### Motivation and Context <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> PR for Group Chat multi-agent orchestration. ADR: https://github.com/microsoft/semantic-kernel/blob/main/docs/decisions/0071-multi-agent-orchestration.md Initial PR: https://github.com/microsoft/semantic-kernel/pull/11993 ### Description <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> This PR includes the following: 1. Implementations for the group chat orchestrations 2. Unit tests and samples Follow up PRs will add the following: - Handoff - Magentic One ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄 |