mirror of
https://github.com/microsoft/semantic-kernel.git
synced 2026-10-12 11:19:56 +00:00
dotnet-1.9.0
88
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Eduard van Valkenburg
|
7d2ac133bc |
Python: move Hugging Face tests to integration (#5983)
### Motivation and Context <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> Even though the completion itself does not callout, loading the model is a HTTP call, hence moving the Hugging Face tests to Integration tests. Plus small fix to make them run. ### Description <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄 |
||
|
Stefano LottiniandEduard van Valkenburg
|
57b55af578 |
Python: (Astra DB) Include caller identity as user-agent when issuing HTTP request to Astra DB's Data API (#5921)
### Motivation and Context This PR enriches the HTTP requests to the Astra DB Data API (occurring in the AstraClient) with a User-Agent header specifying the identity of the caller (specifically, "semantic_kernel"). While not mandatory, this is good practice with Astra DB and helps collecting aggregate usage data on the API side. ### Description Using the `importlib.metadata` library the package version is extracted (in a fail-safe way) and used, if present, to enrich the caller information (variable `ASTRA_CALLER_IDENTITY`, later set as the user-agent header for all HTTP requests to the Data API). ### Contribution Checklist - [X] The code builds clean without any errors or warnings - [X] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [X] All unit tests pass, and I have added new tests where possible - [X] I didn't break anyone 😄 --------- Co-authored-by: Eduard van Valkenburg <[email protected]> |
||
|
Eduard van Valkenburg
|
6751f9d819 |
Python: Updated plugins (#5827)
### Motivation and Context
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
The loading of plugins and functions into Kernel was not fully
consistent and clear, this is now simplified.
Adding a plugin can now be done in two ways:
- multiple: `kernel.add_plugins[plugin1, plugin2]`
- single plugin: `kernel.add_plugin(plugin)`
- if the plugin here is a class, with methods that have the
kernel_function decorator then that is parsed into a plugin
Adding a function can now be done in three ways:
- multiple: `kernel.add_functions([func1, func2]` or
`kernel.add_function({'func1': func1})`
- single: `kernel.add_function('plugin_name', func1)`
- from a prompt: `kernel.add_function(function_name='func1',
prompt='test prompt', ...)`
In other words, all the different methods that were available have been
simplified, these methods also return the created plugin (or updated
plugin for the `add_functions` method. The `add_function` (singular)
method has a parameter `return_plugin` to control whether you get the
created or updated plugin, instead of the created function.
**An important callout:**
One big internal change that this introduces is that a function that
gets added to a plugin (new or existing) is automatically copied, the
metadata is deep-copied and the plugin_name in the metadata is updated,
so this sequence is valid now:
```python
@kernel_function(name='test')
def f(...):
etc
func = KernelFunctionFromMethod(method=f)
func2 = kernel.add_function('plugin', func)
assert func != func2
```
Also closes: #5855
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
Removed KernelPluginCollection
Updated KernelPlugin
added from_... method to load from different sources
Updated KernelFunctionFromPrompt
added from_yaml and from_directory methods.
Added and updated tests to 99% coverage of KernelPlugin and
KernelFunctionFromPrompt
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄
|
||
|
|
ca2ad19130 |
Python: Update to AOAI OYD 2024-02-15-preview API version (#5684)
### Motivation and Context <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> ### Description * Updated models to support the new preview API version 2024-02-15-preview, with some backward compatibility with previous API shape through the use of aliases * Updated samples, unit tests and integration tests * Added experimental settings management using pydantic-settings -- not strictly required here, but I thought it was useful for testing to be able to set all values through the .env file, and validation comes for free using Pydantic. Looking forward to feedback on this. :) ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [ ] The code builds clean without any errors or warnings - [ ] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [ ] All unit tests pass, and I have added new tests where possible - [ ] I didn't break anyone 😄 --------- Co-authored-by: Abby Hartman <[email protected]> Co-authored-by: Evan Mattson <[email protected]> |
||
|
Evan Mattson
|
c4c7464f07 |
Python: Introduce operations to handle OpenAI plugins, improve OpenAPI plugins, and allow for auth (#5695)
### Motivation and Context Python had the ability to handle operations for OpenAPI plugins, but at a basic level. It did not have the ability to handle OpenAI plugins, nor did it allow the user to configure an authentication callback. This PR brings in that functionality. <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> ### Description This PR: - Allows a user to handle OpenAI plugins with/without auth. Closes #5355 - Improves the functionality around handling OpenAPI specs. - Introduces an auth callback mechanism to allow for REST API calls to use auth - Introduces OpenAI/OpenAPI function execution settings. - Provides two new kernel examples that show how to handle a simple OpenAI plugin (Klarna) as well as a more complex plugin (Azure Key Vault) that requires auth. - Updates unit tests. <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [X] The code builds clean without any errors or warnings - [X] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [X] All unit tests pass, and I have added new tests where possible - [X] I didn't break anyone 😄 |
||
|
Eduard van Valkenburg
|
ccbeb7b285 |
Python: rebuilt xml creation and parsing (#5550)
### Motivation and Context <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> We got a report stating that a empty chat_history between two other params, didn't work, so had to change some things in the way chat_history xml is done, and subsequent improved the parsing, now using xml fully. ### Description <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> - Added to_element methods to CMC classes, to_prompt uses that. - Redid the chat_history from_prompt method to fully utilize xml parsing. - Added chat_history tags to detect an empty chatHistory in a prompt. - Made sure when only 1 message is put into the ChatHistory after rendering that that is a USER message, instead of SYSTEM, if multiple the first one is still SYSTEM. ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄 |
||
|
Evan Mattson
|
5b127badbf |
Python: Add support for yaml prompt template (#5527)
### Motivation and Context SK Python doesn't have a way to create a prompt template config from a yaml file. This is neeed to further achieve parity with dotnet and java. <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> ### Description This PR introduces: - some kernel methods to load yaml prompts from a directory as well as create a kernel function from a string that contains yaml. Closes #4640 - Unit tests and a syntax example showing how to create a yaml prompt and load it <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [X] The code builds clean without any errors or warnings - [X] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [X] All unit tests pass, and I have added new tests where possible - [X] I didn't break anyone 😄 |
||
|
Eduard van Valkenburg
|
59c3a9fc9a |
Python: planner love (#5477)
### Motivation and Context <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> Based on #5474 went into the `using-the-planner` notebook and made sure it works with the latest code. ### Description <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> Moved all the planners to use '-' in line with the current work on new templates and planners, once that is in several extension classes can be changed. Encountered several small bugs left over from changes over the last months. Got through the whole notebook with results. ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄 |
||
|
Eduard van Valkenburg
|
3af69df7ea |
Python: Handlebars template support (#5466)
### Motivation and Context <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> Introducing Handlebars templating to the Python Semantic Kernel! Closes: #5446 Prework for: #4641 ### Description <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> Adds HandlebarsPromptTemplate as a subclass of PromptTemplateBase Adds second literal to the PromptTemplateConfig definition Adds helpers for Handlebars (system_helpers) and `create_helper_from_function` function Moves fully_qualified_name into KernelFunctionMetadata instead of in different extensions. Defaults are still 'semantic-kernel' templates, and without making explicit changes that still is the norm. Adds a azure_chat_gpt_handlebars.py sample ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄 |
||
|
Evan Mattson
|
0636b17061 |
Python: Introduce the function calling stepwise planner (#5350)
### Motivation and Context To further align with the SK dotnet SDK and its functionality, the Python SDK needs a function calling stepwise planner. Now that we have auto tool calling available, it makes sense to introduce this planner. <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> ### Description This PR: - Introduces the function calling stepwise planner, its planner options, and planner result. Closes #4642 - Introduces a kernel example `function_calling_stepwise_planner.py` to exercise the planner and show how it works. - Adds unit tests and an integration test. <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [ ] The code builds clean without any errors or warnings - [ ] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [ ] All unit tests pass, and I have added new tests where possible - [ ] I didn't break anyone 😄 |
||
|
Eduard van Valkenburg
|
275db86fd1 |
Python: add py3.12 to unit tests (#5191)
### Motivation and Context <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> Since Python 3.12 is now released we should test with it. In the fall 3.8 will be out of support so then we should remove support for that. Since Numpy no longer support py 3.8 as of [1.25](https://numpy.org/doc/stable/release/1.25.0-notes.html) we pin that for py3.8, including other packages that depend on older version of numpy for py3.8, and allow the new versions to work against newer version of python. Closes #5144 ### Description <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄 |
||
|
Eduard van Valkenburg
|
e884e81776 |
Python: update usearch to 2.9 (#5420)
### Motivation and Context <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> Updates the usearch package to 2.9+, and fixes the tests for that. Closes #2676 ### Description <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄 |
||
|
Eduard van Valkenburg
|
555a7c8436 |
Python: fix for streaming openai responses, and first parts of fixes for Chat With Your Data (#5387)
### Motivation and Context <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> ### Description <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [ ] The code builds clean without any errors or warnings - [ ] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [ ] All unit tests pass, and I have added new tests where possible - [ ] I didn't break anyone 😄 |
||
|
Evan Mattson
|
bf5d21c69a |
Python: Properly handle exceptions in the kernel using the exception metadata (#5391)
### Motivation and Context Two things: 1) Add the exception to the on function invoked args if an exception exists in the function result. 2) Set the stepwise planner's goal to the plan's description, not the name. <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> ### Description The PR addresses: - Properly handling exceptions to raise them from the kernel if a function result contains an exception in the metadata. Closes #5385 - In the stepwise planner, the goal was getting set as the plan's name, instead of the description. Closes #5316 <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [X] The code builds clean without any errors or warnings - [X] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [X] All unit tests pass, and I have added new tests where possible - [X] I didn't break anyone 😄 |
||
|
Eduard van Valkenburg
|
9005c7cc2c |
Python: improved chat history parsing from xml (#5315)
### Motivation and Context <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> I noticed that there was a slight bug in the way the chat_history is parsed from a prompt when you have a message without content (like a tool_call), fixed that and added a test for that case. ### Description <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄 |
||
|
|
d956799df7 |
Python: fix: qdrant upsert on same id (#5090)
### Motivation and Context This bug prevent us from using `QdrantMemoryStore` as memory. ### Description <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄 Co-authored-by: Evan Mattson <[email protected]> Co-authored-by: Eduard van Valkenburg <[email protected]> |
||
|
Eduard van Valkenburg
|
2493e293fb |
Python: Update kernel tests (#5341)
### Motivation and Context <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> Adding a whole bunch of tests to better cover Kernel.py. Also reviewed and updated the conftest and tweaked the scope, now runs faster! Skipped efforts for handlers since work is being done there. ### Description <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄 |
||
|
Evan MattsonandEduard van Valkenburg
|
b93bab90ac |
Python: Support Auto tool calling for OpenAI models (#5267)
### Motivation and Context Auto tool calling hasn't been implemented in Python for OpenAI models. This is a much needed feature, which helps open the possibilities and extends being able to use the OpenAI models for just pure back and forth chats. <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> ### Description This PR: - introduces auto tool calls for OpenAI models. If the prompt function is of an OpenAI chat completion type, then we pass the kernel as an argument to the OpenAI Chat Completion Base class. The kernel is used to auto-invoke tools/functions seamlessly as part of the chat completion request. Closes #4553 - splits the current chat completion/chat completion stream methods into smaller, more readable/functional methods in the OpenAI Chat Completion Base class. - Adds an integration test to cover auto invoke tool calls - Updates a kernel example to be able to work for a streaming/non-streaming tool call. Other Kernel Examples that don't use auto invoke tools still work as they did before. - Removes old code in the utils file that was related to the manual form of invoking function calls. <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [X] The code builds clean without any errors or warnings - [X] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [X] All unit tests pass, and I have added new tests where possible - [X] I didn't break anyone 😄 --------- Co-authored-by: Eduard van Valkenburg <[email protected]> |
||
|
Jib
|
d6c92480b0 |
Python: bugfix: min relevance score calculation filtering on mongodbatlas semantic search (#5224)
### Motivation and Context #4433 <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> ### Description `$match` operation queries need a different structure than provided. As a result users receive `OperationFailure`s when trying to run queries. This introduces a fix to provide to correct query for filtering results by the min_relevance_score. <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄 |
||
|
Eduard van Valkenburg
|
739e7f0e2a |
Python: rebuilt exceptions structure; pythonic version (#5231)
### Motivation and Context <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> The existing Exceptions structure was very much inspired by dotnet, this is now replaced with a pythonic implementations. This means all Exceptions derive from KernelException and then specialise for different purposes. Closes #2194 ### Description <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> Added folder for all exception types, all can be imported through `from semantic_kernel.exceptions import ...` no need for a user to know which file the relevant one is in, but keeps things tidy for developers. Removed old KernelException, added back subtypes for the errorcodes. Went through everything to make sure the `raise ... from exc` pattern is used as much as possible as that returns a better stacktrace. ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄 |
||
|
david-polo
|
079371d8b6 |
.Net: Python: remove _async from methods (#5160)
### Motivation and Context Fix error #5092. ### Description - rename methods of AstraMemoryStore ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [ ] The code builds clean without any errors or warnings - [ ] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [ ] All unit tests pass, and I have added new tests where possible - [ ] I didn't break anyone 😄 |
||
|
|
a6f66ec098 |
Python: split kernelfunction into fromprompt and frommethod classes (#5159)
### Motivation and Context <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> Closes #4633 and brings this part of the python sdk in sync with dotnet, also makes for a much more readable set of classes. ### Description <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> Added: - KernelFunctionFromPrompt class - KernelFunctionFromMethod class - tests for both. Changed: - removed parts of KernelFunction that are specific to one of the two subtypes - removed expose attribute of KernelParameterMetadata as that is no longer needed ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [ ] I didn't break anyone 😄 --------- Co-authored-by: Evan Mattson <[email protected]> Co-authored-by: Evan Mattson <[email protected]> |
||
|
Evan Mattson
|
8b16af69c0 |
Python: Remove the memory tied to the kernel. Update kernel examples and notebooks. (#5143)
### Motivation and Context As we move toward v1, it is necessary to remove the single kernel memory attribute so that memory connectors can be handled via plugins. This change also refactors the `import_plugin` method name to be similar with dotnet as `import_plugin_from_object`, and introduces a simple `kernel.invoke_prompt` method. <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> ### Description Updates are: - The memory connector is now passed to the `SemanticTextMemory` class, along with the embedding generator. This allows the user to add memory to multiple plugins/services as needed. Closes #4632 - Kernel Examples and Notebooks were updated to reflect these changes. Now using Service Enum for all notebooks. Closes #4557 - There were planners using memory related to the kernel, to store the functions, and many of these planners are non-existent in dotnet's v1 SDK. Python has pending work to update the planner/remove deprecated planners before getting to v1. - A Kernel method called `invoke_prompt` was introduced to invoke a simple prompt, and return the `FunctionResult` when complete. - All unit/integration tests are passing. <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [X] The code builds clean without any errors or warnings - [X] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [X] All unit tests pass, and I have added new tests where possible - [ ] I didn't break anyone 😄 |
||
|
|
5817c2b927 |
Python: major features for the Kernel Arguments, Function Result and new Prompt Templating (#5077)
### Motivation and Context <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> Major work to replace the context with Kernel Arguments on the front-end and Function Results on the back. Updated the function decorator to a new approach, in line with dotnet. Revamps the way function are called, allowing native functions to be completely ignorant of SK, other then the decorator. This also moves things into the new folder structure in sync with dotnet. ### Description <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> Adds: - KernelArguments, a dict-like class that replaces the variables in KernelContext. Closes #4565 - FunctionResult, a class to hold the results of function executions, includes the function, the value and metadata, as well as two convenient function to get the value out of it (str and get_inner_content) the first is generic, the second specifically for KernelContent responses (from AI services). - AI Service Selector class, has a default, which is based on the order in the arguments followed by the order in the functions, can be overridden to implement your own strategy. Closes #4631 - Introduces ChatHistory and refactors the PromptTemplateConfig. Closes #4856, #4630 - Improves parsing of templates, will now all validate during creation and throw an error then, instead of some that do not check for validaty until used. - Introduces named_args block and thereby the ability to have multiple arguments for a function call in a template. Closes #5003 Updates: - kernel_function decorators, the parameter decorator was removed and instead we now use Annotated to add a description of a field and we get the type and required from the function definition. - core plugins, use the new approach to kernel_function decorators. - planners, template engines have all been updated to use the kernel and kernelarguments instead of Context. - Events have been updated, now use kernelarguments and function_result - Tokenizers support for named_args and improvements on parsing and checking. - Kernel examples and notebooks to use the latest code. - All unit and integration tests. There is more code coverage now than before. Removed: - kernelContext - kernel_function_parameter_decorator - delegate handling code for native functions - file_io_plugin and tests - SemanticFunctionConfig - ChatPromptTemplate ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [ ] I didn't break anyone 😄 --------- Co-authored-by: Evan Mattson <[email protected]> Co-authored-by: Evan Mattson <[email protected]> Co-authored-by: Evan Mattson <[email protected]> |
||
|
|
5b8deb6745 |
Python: Fixing python document pipeline (#3744)
Fixing python document pipeline --------- Co-authored-by: Mark Wallace <[email protected]> Co-authored-by: Evan Mattson <[email protected]> |
||
|
Eduard van Valkenburg
|
9ec32347ed |
Python: fixes for hugging face and working integration tests (#4927)
### Motivation and Context <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> Fixes #4652 and reenabled integration tests for hugging face found we never added the prompt execution settings class into the right place in the HF class, fixed now. ### Description <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄 |
||
|
Evan MattsonandEvan Mattson
|
2b6bd03b03 |
Python: Add kernel plugin collection and remove old plugin collection classes (#4764)
### Motivation and Context This PR solves two works items: #4560 and #4667. The Python SDK had different ways of handling plugins and collections. The work in the PR helps align the SDK with dotnet nomenclature and structure. <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> ### Description A `KernelPluginCollection` is introduced with an underlying `DefaultKernelPlugin` and a `KernelPlugin` base class. Other code is refactored to use this new `KernelPluginCollection` model which should simplify how plugins are handled. Unit tests are added to test the new functionality. Existing unit tests and integration tests, along with kernel examples and notebooks were updated based on these changes. Unlike before, we're requiring plugins to be defined with a name, which removes the GLOBAL_PLUGIN constant name of "__GLOBAL_PLUGIN__". In some senses, while creating a plugin or a function, a name may not be provided and so a random name in the form of a plugin (p_<random_ascii_chars>) or function (f_<random_ascii_chars>) is used. <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [X] The code builds clean without any errors or warnings - [X] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [X] All unit tests pass, and I have added new tests where possible - [ ] I didn't break anyone 😄 --------- Co-authored-by: Evan Mattson <[email protected]> |
||
|
Evan Mattson
|
7170f0d6a1 |
Python: Remove sync function invoke. Update other code to use async invoke. (#4796)
### Motivation and Context Completes #4794. Most of SK Python is written with async code; however, there were some code paths that masked the underlying async code and made it look like a sync function invoke. In reality, asyncio spun up event loops and attempted to manage them properly. This becomes tricky when running a Jupyter notebook as they have their own event loop. <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> ### Description This PR removes the ability to call a kernel function synchronously. Since the majority of the code base is written with async methods, and to match dotnet, which is async, we're removing the sync function invoke and its dependent code. Some Planners were also run synchronously -- these were updated to run asynchronously. Kernel examples, notebooks, docs, and tests were updated accordingly. <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [X] The code builds clean without any errors or warnings - [X] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [X] All unit tests pass, and I have added new tests where possible - [ ] I didn't break anyone 😄 |
||
|
Julio Carlos Menendez
|
5303126c0d |
Python: Renames request settings to prompt execution settings (#4782)
### Motivation and Context Following the C# implementation, renames all classes and variables of `*RequestSettings` to `*PromptExecutionSettings`. Fixes #4676 ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [X] The code builds clean without any errors or warnings - [X] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [X] All unit tests pass, and I have added new tests where possible - [X] I didn't break anyone 😄 |
||
|
|
9dbbf56c96 |
Python: Add astra memory (#4374)
### Motivation and Context We are going to add [Astra](https://docs.datastax.com/en/home/docs/index.html) as a memory store. DataStax Astra DB is a serverless vector database that’s perfect for managing mission-critical AI workloads. It’s built on Apache Cassandra®, making Astra DB a highly scalable, reliable database technology. It’s a powerful all-in-one data storage solution, ideal for Generative AI projects. ### Description - Add astra into python/semantic_kernel/connectors/memory/astradb - Add test into python/tests/integration/connectors/memory/test_astradb.py ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [ ] The code builds clean without any errors or warnings - [ ] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [ ] All unit tests pass, and I have added new tests where possible - [ ] I didn't break anyone 😄 --------- Co-authored-by: jsbilgi <[email protected]> Co-authored-by: Obioma Anomnachi <[email protected]> |
||
|
Eduard van Valkenburg
|
11cf8eacd4 |
Python: implement KernelContent and all related classes (#4491)
### Motivation and Context <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> As we are preparing to move towards v1 where the context no longer is used, we need a better way of handling the responses from models and capture them in a more structured way, also in preparation of multi-modal models and the like. Closes #4598 ### Description <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> Added KernelContent and StreamingKernelContent base classs, consistent with the dotnet versions of those. Subclasses of this class for specific services, for instance, the OpenAI class has parsing for the content, function_calls, tool_calls in it, and the Azure OpenAI one has parsing for the context in it. ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [ ] I didn't break anyone 😄 |
||
|
Julio Carlos MenendezandEvan Mattson
|
d97f9536c4 |
Python: Removes the _async suffix where is not needed (#4735)
### Motivation and Context Delete all *_async where *_sync doesn't exist to make the library more Pythonic. Fixes #4629 ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [X] The code builds clean without any errors or warnings - [X] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [X] All unit tests pass, and I have added new tests where possible - [X] I didn't break anyone 😄 --------- Co-authored-by: Evan Mattson <[email protected]> |
||
|
Julio Carlos Menendez
|
08710373a4 |
Python: Renames SK prefix to Kernel (#4716)
### Motivation and Context The .NET SDK uses the `Kernel` prefix instead of `SK` for the context, function, model, etc. classes. This PR aligns the Python package with that policy. Fixes #4561 ### Contribution Checklist - [X] The code builds clean without any errors or warnings - [X] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [X] All unit tests pass, and I have added new tests where possible - [X] I didn't break anyone 😄 |
||
|
Evan MattsonandEvan Mattson
|
674af5468d |
Python: Rename skills to plugins. Update prompt template config to use execution settings to align with dotnet. (#4595)
### Motivation and Context To better align with SK dotnet v1, Python skills have been renamed to plugins. This removes the root samples skills directory as dotnet supports the plugins and their configs. This the beginning of more work to consolidate other files/names, per the backlog. I've run several greps for files or file names containing "skill" (case insensitive), and no results appear. Do alert if you find anything else that includes "skill." Fixes #3319 <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> ### Description This is a large, breaking change PR as the replacement of "skills" to "plugins" affects many files. It was difficult to break this up into smaller chunks without leaving the SK Python repo in a bad state for some time. Other updates include: - Prompt template has been updated to use `execution_settings`, which aligns with dotnet, from the previous `completion` that Python used. - AzureOpenAI/OpenAI function calling has been updated to match the current versions of tools/tool_choice. Parsing messages in the open_ai utils respects this as it now looks for tool_calls and it configures the function_call with the id (new), name, and argument). - File renames, where the filename used to contain "skill" will look like new code, but it's the same code, just with "plugin" in the name. - Kernel examples and notebooks have been tested and are working. Unit tests and integration tests are passing. <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [X] The code builds clean without any errors or warnings - [X] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [X] All unit tests pass, and I have added new tests where possible - [X] I didn't break anyone 😄 --------- Co-authored-by: Evan Mattson <[email protected]> |
||
|
Sarah WidderandSarah Widder
|
4778b0fd30 |
Python: Fix issues with chat OYD, update samples and tests (#4570)
### Motivation and Context <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> Fixes #4538 - data source parameters in ExtraBody were not serialized correctly by Pydantic, so the call to the chat completions extensions API was failing. ### Description <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> - Add Pydantic annotation `SerializeAsAny` to `AzureDataSources` to ensure all parameters of subclasses are serialized, not just the ones in the base class. - Update with-data samples to use `AzureChatRequestSettings` - Update unit tests and unskip integration test ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄 --------- Co-authored-by: Sarah Widder <[email protected]> |
||
|
Eduard van Valkenburg
|
fa3e32bb16 |
Python: set line-length for black in sync with Ruff, run black. (#4396)
### Motivation and Context <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> As we are tightening the formatting and linting setups, this was missing, now the line-length for black and ruff are the same, thereby no longer running into black issues as often. ### Description <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄 |
||
|
Eduard van Valkenburg
|
b91529a62e |
Python: Implement AI request settings (#4097)
### Motivation and Context <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> This PR implements the new approach to AI Request settings as described here: https://github.com/microsoft/semantic-kernel/blob/main/docs/decisions/0008-support-generic-llm-request-settings.md and completes #3312 ### Description <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> Does this: - Creates AI Request Settings with service_id (referring to the registered AI services in a kernel) and extension_data, a dict that can hold any number of settings - has several methods: - `from_ai_request_settings` -> constructor to create a new request_settings object based on another, supports creating specific ai_request_settings from generic ones. - `update_from_ai_request_settings` -> updating the fields of a request setting from another ones. - `prepare_settings_dict` -> method that is used when actually creating the dict that get's passed to the handler of the AI service. - Created subclasses like OpenAIChatRequestSettings which has all the required fields of a OpenAI Chat completion, same for OpenAIText, AzureChat, GooglePalm, HuggingFace, etc. - added `get_request_settings_class` to ai_services to return the class used for the settings for that service. The goal is that when running, this settings object get's updated and a single operations (called `prepare_settings_dict`) is used to return a dict that can be passed to the api call directly, thereby reducing the need for custom logic everywhere, this will also negate the need for things like `complete_chat_with_functions_async` and others. So also removed complete_chat_..._async methods from openai and azure openai. It has also changed the prompt_template_config with the generic type for the AIRequestSettings or subclasses, and some other changes. ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [ ] I didn't break anyone 😄 |
||
|
Evan MattsonandEvan Mattson
|
c4852aa2bd |
Python: Mark flaky test with xfail until we can fix it. (#4467)
### Motivation and Context The `test_can_execute_stepwise_plan` integration test succeeds sometimes and fails others. Marking it with `xfail` until we can root-cause the issue. It's blocking PRs from completing successfully. Underlying issue is #4466. <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> ### Description See above. <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [X] The code builds clean without any errors or warnings - [X] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [X] All unit tests pass, and I have added new tests where possible - [X] I didn't break anyone 😄 Co-authored-by: Evan Mattson <[email protected]> |
||
|
Sarah WidderandSarah Widder
|
a11c9951c0 |
Python: Add python connector for Azure OpenAI on your data (#3919)
### Motivation and Context <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> This change adds a python connector for Azure OpenAI chat completions with data, which was already supported in the .NET SDK. ### Description <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> Add connector for Azure OpenAI chat completions with data and associated sample code for how to use it. Some minor changes were made to the SK function to handle the "tool" message that contains the citations and other information returned from the chat completions extensions API. ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [ ] The code builds clean without any errors or warnings - [ ] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [ ] All unit tests pass, and I have added new tests where possible - [ ] I didn't break anyone 😄 --------- Co-authored-by: Sarah Widder <[email protected]> |
||
|
Evan MattsonandEvan Mattson
|
15f13b285d |
Python: allow default_headers configuration and users to pass in custom AzureOpenAI/OpenAI clients. (#3903)
### Motivation and Context Currently, users in Python have no ability to configure default_headers for the AzureOpenAI/OpenAI clients we use. Additionally, if a user wants to set up their own AzureOpenAI/OpenAI client, this isn't possible. This PR addresses #2895. <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> ### Description Allow users to set default_headers for Chat/Text/Embedding Completion classes for both OpenAI and AzureOpenAI. Additionally, allow users to pass in a customer OpenAI/AzureOpenAI client, if desired. Update tests accordingly. <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [ ] The code builds clean without any errors or warnings - [ ] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [ ] All unit tests pass, and I have added new tests where possible - [ ] I didn't break anyone 😄 --------- Co-authored-by: Evan Mattson <[email protected]> |
||
|
Eduard van Valkenburg
|
0868068f9b |
Python: Upgrade to Pydantic 2+ (#3723)
### Motivation and Context <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> Upgraded pydantic to 2.0+, much cleaner, less complexity in different types of base models. ### Description <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [ ] The code builds clean without any errors or warnings - [ ] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [ ] All unit tests pass, and I have added new tests where possible - [ ] I didn't break anyone 😄 |
||
|
+17Alex ChaoJoowonRoger BarretoDmytro StrukMark WallaceBen ThomasBen ThomasDevis LucatoAayush KatariaAbby HarrisonAbby HarrisonStephen ToubTeresa HoangChrisGil LaHayeGina TrioloTao ChenLee MillerJennifer MarsmanWeihan LiSergeyMenshykhDiego Colombodependabot[bot]Alex ChaoEduard van ValkenburgEvan MattsonEvan Mattson |
c765cb1109 |
Python: Make SK compatible with OpenAI 1.0 (#3555)
Need to update SK Python to integrate the breaking changes from openai-python 1.0 https://github.com/microsoft/semantic-kernel/issues/3330 This PR replaces the old one that was merging to main https://github.com/microsoft/semantic-kernel/pull/3417 --------- ### Motivation and Context <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> ### Description <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [ ] The code builds clean without any errors or warnings - [ ] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [ ] All unit tests pass, and I have added new tests where possible - [ ] I didn't break anyone 😄 --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Joowon <[email protected]> Co-authored-by: Roger Barreto <[email protected]> Co-authored-by: Dmytro Struk <[email protected]> Co-authored-by: Mark Wallace <[email protected]> Co-authored-by: Ben Thomas <[email protected]> Co-authored-by: Ben Thomas <[email protected]> Co-authored-by: Devis Lucato <[email protected]> Co-authored-by: Aayush Kataria <[email protected]> Co-authored-by: Abby Harrison <[email protected]> Co-authored-by: Abby Harrison <[email protected]> Co-authored-by: Stephen Toub <[email protected]> Co-authored-by: Teresa Hoang <[email protected]> Co-authored-by: Chris <[email protected]> Co-authored-by: Gil LaHaye <[email protected]> Co-authored-by: Gina Triolo <[email protected]> Co-authored-by: Tao Chen <[email protected]> Co-authored-by: Lee Miller <[email protected]> Co-authored-by: Jennifer Marsman <[email protected]> Co-authored-by: Weihan Li <[email protected]> Co-authored-by: SergeyMenshykh <[email protected]> Co-authored-by: Diego Colombo <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Alex Chao <achao@achao> Co-authored-by: Eduard van Valkenburg <[email protected]> Co-authored-by: Evan Mattson <[email protected]> Co-authored-by: Evan Mattson <[email protected]> |
||
|
|
dcad9f7ba5 |
Python: Adding Azure CosmosDB Mongo vCore as a datastore. (#2990)
### Motivation and Context I have added Azure CosmosDB MongoDB vCore as a data store. MongoDB vCore now supports vector search on embeddings, and it could be used to seamlessly integrate your AI-based applications with your data stored in the Azure CosmosDB. More details about Mongo vCore can be found here: https://learn.microsoft.com/en-us/azure/cosmos-db/mongodb/vcore/vector-search. Issue https://github.com/microsoft/semantic-kernel/issues/2375 <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, by providing the following information: --> 1. Why is this change required? **MongoDB vCore now supports vector search on embeddings, and it could be used to seamlessly integrate your AI-based applications with your data stored in the Azure CosmosDB.** 2. What problem does it solve? This adds a new memory store MongoDB vCore(Azure CosmosDB) ### Description <!-- Describe your changes, the overall approach, and the underlying design. These notes will help you understand how your code works. Thanks! --> ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [ ] The code builds clean without any errors or warnings - [ ] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [ ] All unit tests pass, and I have added new tests where possible - [ ] I didn't break anyone 😄 --------- Co-authored-by: Abby Harrison <[email protected]> Co-authored-by: Abby Harrison <[email protected]> Co-authored-by: Mark Wallace <[email protected]> |
||
|
|
7d31513d3f |
Python: Context Variables like dict (#3039)
### Motivation and Context <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> It's shame sometimes it shines through that some of the earliest code in the python code base was a direct port from C#, like with the .get method on the context variables, made the change here to have the exact some signature as the default python dict. Also replaces contains_key with __contains__ so that 'in' keyword can be used, just like a normal dict. Fixes #2962 ### Description <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> Changed the .get method Replaced the contains_key method with __contains__ Made changes in samples, tests and planner to deal with that. ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [ ] I didn't break anyone 😄 --------- Co-authored-by: Devis Lucato <[email protected]> Co-authored-by: Devis Lucato <[email protected]> |
||
|
|
8ce66fe9a9 |
Python: Adding MongoDB Atlas Vector Search Connector (#2818)
### Motivation and Context Resolves: #2591 <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> ### Description <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> * Added in the `MongoDBAtlasMemoryStore`. A memory store abstraction that facilitates connections between a MongoDB Atlas cluster to conduct [Atlas Vector Search](https://www.mongodb.com/products/platform/atlas-vector-search) on the Microsoft Semantic Kernel. * Leverages our async python driver [motor](https://motor.readthedocs.io/en/stable/) to keep with the heuristic of asynchronous code ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄 ### Callout * When writing my test, I noticed there's a habit of other tests conducting a `try` on imports. Grepping through those implementations, I see some of them [encapsulate their respective libraries in function calls rather than doing direct imports in the module.](https://github.com/microsoft/semantic-kernel/blob/main/python/semantic_kernel/connectors/memory/chroma/chroma_memory_store.py#L55-L63) Is this the general code pattern we should follow? Or is it fine to have our imports at the top of the module fine? --------- Co-authored-by: Steven Silvester <[email protected]> Co-authored-by: Abby Harrison <[email protected]> Co-authored-by: Abby Harrison <[email protected]> |
||
|
Abby Harrison
|
393277bf8f |
Python: skip redis tests in CI if the platform is not linux (#2727)
### Motivation and Context The python integration tests use the public redis docker image. This image is not available to our macOS and windows test agents. This PR ensures that the tests are skipped on non-linux platforms (CI only). ### Description <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [ ] The code builds clean without any errors or warnings - [ ] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [ ] All unit tests pass, and I have added new tests where possible - [ ] I didn't break anyone 😄 |
||
|
|
73afea986a |
Python: added settings function to load direct into constructor (#2069)
### Motivation and Context <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> Noticed that another commonly edited environment setting for api_version was missing from the settings util. And want to pass the env to the constructor. ### Description <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> Added api_version to the settings util so that that can be controlled through .env as well. Added convenience function that outputs a dict that can be passed directly into Completion constructors. Updated sample to use that. ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#dev-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [ ] I didn't break anyone 😄 --------- Co-authored-by: Abby Harrison <[email protected]> Co-authored-by: Lee Miller <[email protected]> |
||
|
|
edf35de85d |
Python: Redis memory connector (#2132)
### Motivation and Context <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> 1. .NET supports Redis memory while Python does not 2. Match up Python features w/ .NET 3. Users can utilize a Redis backend for memories 4. Closes #1981 ### Description <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> Implemented a memory connector for Redis and made unit tests for it. Also added .env configuration for a connection string. ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [X] The code builds clean without any errors or warnings - [X] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#dev-scripts) raises no violations - [X] All unit tests pass, and I have added new tests where possible - [X] I didn't break anyone 😄 --------- Co-authored-by: Dmytro Struk <[email protected]> Co-authored-by: Abby Harrison <[email protected]> Co-authored-by: Abby Harrison <[email protected]> |
||
|
Karol ŻakandAbby Harrison
|
601c9de124 |
Python: Add HF model and pipeline kwargs to allow model loading customization (#2547)
### Motivation and Context <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> I was trying to use this library to load and use [HF Falcon model](https://huggingface.co/tiiuae/falcon-7b-instruct). Unfortunately in order to make it work I needed to pass in some additional arguments such as `torch_dtype`, `cache_dir` or `tokenizer`. It was not possible with current implementation so I decided to add `model_kwargs` and `pipeline_kwargs` to the `HuggingFaceTextCompletion` class init function so now, with the changes I introduced I can successfully load any `falcon` models by using this code: ```python import semantic_kernel as sk import semantic_kernel.connectors.ai.hugging_face as sk_hf import torch from transformers import AutoTokenizer cache_dir = "/mnt/tmp" model_name = "tiiuae/falcon-7b-instruct" tokenizer = AutoTokenizer.from_pretrained( pretrained_model_name_or_path=model_name, cache_dir=cache_dir, trust_remote_code=True) hf_model = sk_hf.HuggingFaceTextCompletion( model_name, task="text-generation", model_kwargs={ "cache_dir": cache_dir, "torch_dtype": torch.bfloat16 }, pipeline_kwargs={ "tokenizer": tokenizer, "trust_remote_code": True, "device": 0, } ) ``` ### Description <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> The changes I introduced add `model_kwargs` and `pipeline_kwargs` to `HuggingFaceTextCompletion` class so that these arguments are then passed to the underlying `transformers.pipeline()` function and thus allow more customization options. ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄 --------- Co-authored-by: Abby Harrison <[email protected]> |
||
|
|
3881a3137f |
Python: Adding USearch memory connector (#2358)
### Motivation and Context The integration of [USearch](https://github.com/unum-cloud/usearch) as a memory connector to Semantic Kernel (SK). ### Description The USearch `Index` does not natively have the ability to store different collections, and it only stores embeddings without other attributes like `MemoryRecord`. The `USearchMemoryStore` class encapsulates these capabilities. It uses the USearch `Index` to store a collection of embeddings under unique IDs, with original collection names mapped to those IDs. Other `MemoryRecord ` attributes are stored in a `pyarrow.Table`, which is mapped to each collection. It's important to note the current behavior when a user removes a record or upserts a new one with an existing ID: the old row is not removed from the `pyarrow.Table`. This is done for performance reasons but could lead to the table growing in size. By default, `USearchMemoryStore` operates as an in-memory store. To enable persistence, you must set the persist mode with calling appropriate `__init__ `, supplying a path to the directory for the persist files. For each collection, two files will be created: `{collection_name}.usearch` and `{collection_name}.parquet`. Changes will only be dumped to the disk when `close_async` is called. Due to the interface provided by the base class `MemoryStoreBase`, this happens implicitly when using a context manager, or it may be called explicitly. Since collection names are used to store files on disk, all names are converted to lowercase. To ensure efficient use of memory, you should call `close_async`. --------- Co-authored-by: Abby Harrison <[email protected]> Co-authored-by: Abby Harrison <[email protected]> Co-authored-by: Devis Lucato <[email protected]> |