### Motivation and Context
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
Fixes#4652
and reenabled integration tests for hugging face
found we never added the prompt execution settings class into the right
place in the HF class, fixed now.
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄
### Motivation and Context
This PR solves two works items: #4560 and #4667. The Python SDK had
different ways of handling plugins and collections. The work in the PR
helps align the SDK with dotnet nomenclature and structure.
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
### Description
A `KernelPluginCollection` is introduced with an underlying
`DefaultKernelPlugin` and a `KernelPlugin` base class. Other code is
refactored to use this new `KernelPluginCollection` model which should
simplify how plugins are handled. Unit tests are added to test the new
functionality. Existing unit tests and integration tests, along with
kernel examples and notebooks were updated based on these changes.
Unlike before, we're requiring plugins to be defined with a name, which
removes the GLOBAL_PLUGIN constant name of "__GLOBAL_PLUGIN__". In some
senses, while creating a plugin or a function, a name may not be
provided and so a random name in the form of a plugin
(p_<random_ascii_chars>) or function (f_<random_ascii_chars>) is used.
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [X] The code builds clean without any errors or warnings
- [X] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [X] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄
---------
Co-authored-by: Evan Mattson <[email protected]>
### Motivation and Context
Completes #4794.
Most of SK Python is written with async code; however, there were some
code paths that masked the underlying async code and made it look like a
sync function invoke. In reality, asyncio spun up event loops and
attempted to manage them properly. This becomes tricky when running a
Jupyter notebook as they have their own event loop.
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
### Description
This PR removes the ability to call a kernel function synchronously.
Since the majority of the code base is written with async methods, and
to match dotnet, which is async, we're removing the sync function invoke
and its dependent code. Some Planners were also run synchronously --
these were updated to run asynchronously. Kernel examples, notebooks,
docs, and tests were updated accordingly.
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [X] The code builds clean without any errors or warnings
- [X] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [X] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄
### Motivation and Context
Following the C# implementation, renames all classes and variables of
`*RequestSettings` to `*PromptExecutionSettings`.
Fixes#4676
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [X] The code builds clean without any errors or warnings
- [X] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [X] All unit tests pass, and I have added new tests where possible
- [X] I didn't break anyone 😄
### Motivation and Context
We are going to add
[Astra](https://docs.datastax.com/en/home/docs/index.html) as a memory
store.
DataStax Astra DB is a serverless vector database that’s perfect for
managing mission-critical AI workloads. It’s built on Apache Cassandra®,
making Astra DB a highly scalable, reliable database technology. It’s a
powerful all-in-one data storage solution, ideal for Generative AI
projects.
### Description
- Add astra into python/semantic_kernel/connectors/memory/astradb
- Add test into
python/tests/integration/connectors/memory/test_astradb.py
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [ ] The code builds clean without any errors or warnings
- [ ] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [ ] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄
---------
Co-authored-by: jsbilgi <[email protected]>
Co-authored-by: Obioma Anomnachi <[email protected]>
### Motivation and Context
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
As we are preparing to move towards v1 where the context no longer is
used, we need a better way of handling the responses from models and
capture them in a more structured way, also in preparation of
multi-modal models and the like.
Closes#4598
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
Added KernelContent and StreamingKernelContent base classs, consistent
with the dotnet versions of those.
Subclasses of this class for specific services, for instance, the OpenAI
class has parsing for the content, function_calls, tool_calls in it, and
the Azure OpenAI one has parsing for the context in it.
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄
### Motivation and Context
To better align with SK dotnet v1, Python skills have been renamed to
plugins. This removes the root samples skills directory as dotnet
supports the plugins and their configs. This the beginning of more work
to consolidate other files/names, per the backlog. I've run several
greps for files or file names containing "skill" (case insensitive), and
no results appear. Do alert if you find anything else that includes
"skill."
Fixes#3319
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
### Description
This is a large, breaking change PR as the replacement of "skills" to
"plugins" affects many files. It was difficult to break this up into
smaller chunks without leaving the SK Python repo in a bad state for
some time. Other updates include:
- Prompt template has been updated to use `execution_settings`, which
aligns with dotnet, from the previous `completion` that Python used.
- AzureOpenAI/OpenAI function calling has been updated to match the
current versions of tools/tool_choice. Parsing messages in the open_ai
utils respects this as it now looks for tool_calls and it configures the
function_call with the id (new), name, and argument).
- File renames, where the filename used to contain "skill" will look
like new code, but it's the same code, just with "plugin" in the name.
- Kernel examples and notebooks have been tested and are working. Unit
tests and integration tests are passing.
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [X] The code builds clean without any errors or warnings
- [X] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [X] All unit tests pass, and I have added new tests where possible
- [X] I didn't break anyone 😄
---------
Co-authored-by: Evan Mattson <[email protected]>
### Motivation and Context
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
Fixes#4538 - data source parameters in ExtraBody were not serialized
correctly by Pydantic, so the call to the chat completions extensions
API was failing.
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
- Add Pydantic annotation `SerializeAsAny` to `AzureDataSources` to
ensure all parameters of subclasses are serialized, not just the ones in
the base class.
- Update with-data samples to use `AzureChatRequestSettings`
- Update unit tests and unskip integration test
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄
---------
Co-authored-by: Sarah Widder <[email protected]>
### Motivation and Context
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
As we are tightening the formatting and linting setups, this was
missing, now the line-length for black and ruff are the same, thereby no
longer running into black issues as often.
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄
### Motivation and Context
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
This PR implements the new approach to AI Request settings as described
here:
https://github.com/microsoft/semantic-kernel/blob/main/docs/decisions/0008-support-generic-llm-request-settings.md
and completes #3312
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
Does this:
- Creates AI Request Settings with service_id (referring to the
registered AI services in a kernel) and extension_data, a dict that can
hold any number of settings
- has several methods:
- `from_ai_request_settings` -> constructor to create a new
request_settings object based on another, supports creating specific
ai_request_settings from generic ones.
- `update_from_ai_request_settings` -> updating the fields of a request
setting from another ones.
- `prepare_settings_dict` -> method that is used when actually creating
the dict that get's passed to the handler of the AI service.
- Created subclasses like OpenAIChatRequestSettings which has all the
required fields of a OpenAI Chat completion, same for OpenAIText,
AzureChat, GooglePalm, HuggingFace, etc.
- added `get_request_settings_class` to ai_services to return the class
used for the settings for that service.
The goal is that when running, this settings object get's updated and a
single operations (called `prepare_settings_dict`) is used to return a
dict that can be passed to the api call directly, thereby reducing the
need for custom logic everywhere, this will also negate the need for
things like `complete_chat_with_functions_async` and others. So also
removed complete_chat_..._async methods from openai and azure openai.
It has also changed the prompt_template_config with the generic type for
the AIRequestSettings or subclasses, and some other changes.
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄
### Motivation and Context
The `test_can_execute_stepwise_plan` integration test succeeds sometimes
and fails others. Marking it with `xfail` until we can root-cause the
issue. It's blocking PRs from completing successfully.
Underlying issue is #4466.
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
### Description
See above.
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [X] The code builds clean without any errors or warnings
- [X] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [X] All unit tests pass, and I have added new tests where possible
- [X] I didn't break anyone 😄
Co-authored-by: Evan Mattson <[email protected]>
### Motivation and Context
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
This change adds a python connector for Azure OpenAI chat completions
with data, which was already supported in the .NET SDK.
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
Add connector for Azure OpenAI chat completions with data and associated
sample code for how to use it. Some minor changes were made to the SK
function to handle the "tool" message that contains the citations and
other information returned from the chat completions extensions API.
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [ ] The code builds clean without any errors or warnings
- [ ] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [ ] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄
---------
Co-authored-by: Sarah Widder <[email protected]>
### Motivation and Context
Currently, users in Python have no ability to configure default_headers
for the AzureOpenAI/OpenAI clients we use. Additionally, if a user wants
to set up their own AzureOpenAI/OpenAI client, this isn't possible. This
PR addresses #2895.
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
### Description
Allow users to set default_headers for Chat/Text/Embedding Completion
classes for both OpenAI and AzureOpenAI. Additionally, allow users to
pass in a customer OpenAI/AzureOpenAI client, if desired. Update tests
accordingly.
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [ ] The code builds clean without any errors or warnings
- [ ] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [ ] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄
---------
Co-authored-by: Evan Mattson <[email protected]>
### Motivation and Context
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
Upgraded pydantic to 2.0+, much cleaner, less complexity in different
types of base models.
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [ ] The code builds clean without any errors or warnings
- [ ] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [ ] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄
### Motivation and Context
I have added Azure CosmosDB MongoDB vCore as a data store. MongoDB vCore
now supports vector search on embeddings, and it could be used to
seamlessly integrate your AI-based applications with your data stored in
the Azure CosmosDB. More details about Mongo vCore can be found here:
https://learn.microsoft.com/en-us/azure/cosmos-db/mongodb/vcore/vector-search.
Issue https://github.com/microsoft/semantic-kernel/issues/2375
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, by providing the following
information:
-->
1. Why is this change required? **MongoDB vCore now supports vector
search on embeddings, and it could be used to seamlessly integrate your
AI-based applications with your data stored in the Azure CosmosDB.**
2. What problem does it solve? This adds a new memory store MongoDB
vCore(Azure CosmosDB)
### Description
<!-- Describe your changes, the overall approach, and the underlying
design.
These notes will help you understand how your code works. Thanks! -->
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [ ] The code builds clean without any errors or warnings
- [ ] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [ ] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄
---------
Co-authored-by: Abby Harrison <[email protected]>
Co-authored-by: Abby Harrison <[email protected]>
Co-authored-by: Mark Wallace <[email protected]>
### Motivation and Context
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
It's shame sometimes it shines through that some of the earliest code in
the python code base was a direct port from C#, like with the .get
method on the context variables, made the change here to have the exact
some signature as the default python dict.
Also replaces contains_key with __contains__ so that 'in' keyword can be
used, just like a normal dict.
Fixes#2962
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
Changed the .get method
Replaced the contains_key method with __contains__
Made changes in samples, tests and planner to deal with that.
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄
---------
Co-authored-by: Devis Lucato <[email protected]>
Co-authored-by: Devis Lucato <[email protected]>
### Motivation and Context
Resolves: #2591
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
* Added in the `MongoDBAtlasMemoryStore`. A memory store abstraction
that facilitates connections between a MongoDB Atlas cluster to conduct
[Atlas Vector
Search](https://www.mongodb.com/products/platform/atlas-vector-search)
on the Microsoft Semantic Kernel.
* Leverages our async python driver
[motor](https://motor.readthedocs.io/en/stable/) to keep with the
heuristic of asynchronous code
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄
### Callout
* When writing my test, I noticed there's a habit of other tests
conducting a `try` on imports. Grepping through those implementations, I
see some of them [encapsulate their respective libraries in function
calls rather than doing direct imports in the
module.](https://github.com/microsoft/semantic-kernel/blob/main/python/semantic_kernel/connectors/memory/chroma/chroma_memory_store.py#L55-L63)
Is this the general code pattern we should follow? Or is it fine to have
our imports at the top of the module fine?
---------
Co-authored-by: Steven Silvester <[email protected]>
Co-authored-by: Abby Harrison <[email protected]>
Co-authored-by: Abby Harrison <[email protected]>
### Motivation and Context
The python integration tests use the public redis docker image. This
image is not available to our macOS and windows test agents. This PR
ensures that the tests are skipped on non-linux platforms (CI only).
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [ ] The code builds clean without any errors or warnings
- [ ] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [ ] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄
### Motivation and Context
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
Noticed that another commonly edited environment setting for api_version
was missing from the settings util. And want to pass the env to the
constructor.
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
Added api_version to the settings util so that that can be controlled
through .env as well.
Added convenience function that outputs a dict that can be passed
directly into Completion constructors.
Updated sample to use that.
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#dev-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄
---------
Co-authored-by: Abby Harrison <[email protected]>
Co-authored-by: Lee Miller <[email protected]>
### Motivation and Context
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
1. .NET supports Redis memory while Python does not
2. Match up Python features w/ .NET
3. Users can utilize a Redis backend for memories
4. Closes#1981
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
Implemented a memory connector for Redis and made unit tests for it.
Also added .env configuration for a connection string.
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [X] The code builds clean without any errors or warnings
- [X] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#dev-scripts)
raises no violations
- [X] All unit tests pass, and I have added new tests where possible
- [X] I didn't break anyone 😄
---------
Co-authored-by: Dmytro Struk <[email protected]>
Co-authored-by: Abby Harrison <[email protected]>
Co-authored-by: Abby Harrison <[email protected]>
### Motivation and Context
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
I was trying to use this library to load and use [HF Falcon
model](https://huggingface.co/tiiuae/falcon-7b-instruct). Unfortunately
in order to make it work I needed to pass in some additional arguments
such as `torch_dtype`, `cache_dir` or `tokenizer`. It was not possible
with current implementation so I decided to add `model_kwargs` and
`pipeline_kwargs` to the `HuggingFaceTextCompletion` class init function
so now, with the changes I introduced I can successfully load any
`falcon` models by using this code:
```python
import semantic_kernel as sk
import semantic_kernel.connectors.ai.hugging_face as sk_hf
import torch
from transformers import AutoTokenizer
cache_dir = "/mnt/tmp"
model_name = "tiiuae/falcon-7b-instruct"
tokenizer = AutoTokenizer.from_pretrained(
pretrained_model_name_or_path=model_name,
cache_dir=cache_dir,
trust_remote_code=True)
hf_model = sk_hf.HuggingFaceTextCompletion(
model_name,
task="text-generation",
model_kwargs={
"cache_dir": cache_dir,
"torch_dtype": torch.bfloat16
},
pipeline_kwargs={
"tokenizer": tokenizer,
"trust_remote_code": True,
"device": 0,
}
)
```
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
The changes I introduced add `model_kwargs` and `pipeline_kwargs` to
`HuggingFaceTextCompletion` class so that these arguments are then
passed to the underlying `transformers.pipeline()` function and thus
allow more customization options.
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄
---------
Co-authored-by: Abby Harrison <[email protected]>
### Motivation and Context
The integration of [USearch](https://github.com/unum-cloud/usearch) as a
memory connector to Semantic Kernel (SK).
### Description
The USearch `Index` does not natively have the ability to store
different collections, and it only stores embeddings without other
attributes like `MemoryRecord`.
The `USearchMemoryStore` class encapsulates these capabilities. It uses
the USearch `Index` to store a collection of embeddings under unique
IDs, with original collection names mapped to those IDs. Other
`MemoryRecord ` attributes are stored in a `pyarrow.Table`, which is
mapped to each collection.
It's important to note the current behavior when a user removes a record
or upserts a new one with an existing ID: the old row is not removed
from the `pyarrow.Table`. This is done for performance reasons but could
lead to the table growing in size.
By default, `USearchMemoryStore` operates as an in-memory store. To
enable persistence, you must set the persist mode with calling
appropriate `__init__ `, supplying a path to the directory for the
persist files. For each collection, two files will be created:
`{collection_name}.usearch` and `{collection_name}.parquet`. Changes
will only be dumped to the disk when `close_async` is called. Due to the
interface provided by the base class `MemoryStoreBase`, this happens
implicitly when using a context manager, or it may be called explicitly.
Since collection names are used to store files on disk, all names are
converted to lowercase.
To ensure efficient use of memory, you should call `close_async`.
---------
Co-authored-by: Abby Harrison <[email protected]>
Co-authored-by: Abby Harrison <[email protected]>
Co-authored-by: Devis Lucato <[email protected]>
### Motivation and Context
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
implement stepwise planner #2221
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [ ] The code builds clean without any errors or warnings
- [ ] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [ ] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄
---------
Co-authored-by: Abby Harrison <[email protected]>
Co-authored-by: Lee Miller <[email protected]>
Co-authored-by: Abby Harrison <[email protected]>
Fix failing integration tests introduced in #2474 (May be filing follow
up issue on that)
- Update WriterSkillFake to include a context parameter for the end
marker
- Update test_sequential_planner to use the new end marker parameter in
WriterSkillFake
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄
### Motivation and Context
Implementation of Google PaLM connector with chat completion, text
embedding and three example files to demonstrate their functionality.
Closes#2098
I also opened a PR for text completion #2076
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
### Description
**What's new:**
1. Implemented Google Palm connector with chat completion and text
embedding
2. Added 3 example files to `python/samples/kernel-syntax-examples`
3. The example files show PaLM chat in action with skills, memory, and
just with user messages. The memory example uses PaLM embedding as well
5. Added integration tests to test chat with skills and embedding with
kernel.memory functions
6. Added unit tests to verify successful class initialization and
correct behavior from class functions
**There are some important differences between google palm chat and open
ai chat:**
1. PaLM has two functions for chatting, `chat` and `reply`. The chat
function in google's genai library starts a new conversation, and reply
continues the conversation. Reply is an attribute of the response object
returned by chat. So an instance of the `GooglePalmChatCompletion` class
needs a way to determine which function to use, which is why I
introduced a private attribute to store the response object. See
https://developers.generativeai.google/api/python/google/generativeai/types/ChatResponse
2. PaLM does not use system messages. Instead, the chat function takes a
parameter called context. It serves the same purpose, to prime the
assistant with certain behaviors and information. So when the user
passes a system message to `complete_chat_async`, it is passed to `chat`
as the context parameter.
3. Semantic memory works with the chat service as long as the user
creates a chat prompt template. The prompt containing the memory needs
to be added to the chat prompt template as a system message. See
`python\samples\kernel-syntax-examples\google_palm_chat_with_memory.py`
for more details. If the only purpose of `complete_async` in
`GooglePalmChatCompletion` is to send memories + user messages to the
chat service as a text prompt, then `complete_async` is not fulfilling
its intended purpose. A possible solution would be to send the text
prompt as a request to the text service within `complete_async`.
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄
Currently no warnings, there was 1 warning when first installing genai
with `poetry add google.generativeai==v0.1.0rc2` from within poetry
shell: "The locked version 0.1.0rc2 for google-generativeai is a yanked
version. Reason for being yanked: Release is marked as supporting Py3.8,
but in practice it requires 3.9". We would need to require later
versions of python to fix it.
---------
Co-authored-by: Abby Harrison <[email protected]>
Co-authored-by: Abby Harrison <[email protected]>
Co-authored-by: Abby Harrison <[email protected]>
### Motivation and Context
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
Implementation of Google PaLM connector with text completion and an
example file to demonstrate its functionality.
Closes#1979
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
1. Implemented Google Palm connector with text completion
2. Added example file to ```python/samples/kernel-syntax-examples```
3. Added integration tests with different inputs to kernel.run_async
4. Added unit tests to ensure successful initialization of the class and
successful API calls
5. 3 optional arguments (top_k, safety_settings, client) for
google.generativeai.generate_text were not included. See more
information about the function and its arguments:
https://developers.generativeai.google/api/python/google/generativeai/generate_text
I also opened a PR for text embedding and chat completion #2258
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#dev-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄
Currently no warnings, there was 1 warning when first installing genai
with `poetry add google.generativeai==v0.1.0rc2` from within poetry
shell: "The locked version 0.1.0rc2 for google-generativeai is a yanked
version. Reason for being yanked: Release is marked as supporting Py3.8,
but in practice it requires 3.9". We would need to require later
versions of python to fix it.
---------
Co-authored-by: Abby Harrison <[email protected]>
Co-authored-by: Abby Harrison <[email protected]>
### Motivation and Context
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
Adding optional batch size parameter for when Azure OpenAI limits number
of inputs for embeddings requests. Fixes
https://github.com/microsoft/semantic-kernel/issues/2099
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [ ] The code builds clean without any errors or warnings
- [ ] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [ ] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄
---------
Co-authored-by: Abby Harrison <[email protected]>
Co-authored-by: Abby Harrison <[email protected]>
### Motivation and Context
This PR addresses multiple issues with the integration tests. Fixes
integration test breaks related to qdrant and chroma package updates.
Labels known flaky tests as xfail. Adds merge queue gates. Adds push
notifications to teams for test results.
### Description
- integration tests are now 1x daily down from 2x daily
- adds a run condition for `merge_group`
- if run for merge_group, a subset of the tests will gate the merge
- ends the results of each test matrix to a dedicated teams channel
- update qdrant client requirement in the pyproject.toml
- remove `persist_directory` from chroma, as the default is now
persistent
- default with_embedding to false in qdrant_memory_store
- modify chroma test setup and teardown to create the database in a
known location and clear the database of entries at the end of each test
(I would have just deleted the database at the end of each test, but
there are test agent permissions issues with this in CI)
- marked known flaky tests (pinecone and sequential_planner) as allowed
to fail
Fixes#2284
---------
Co-authored-by: Abby Harrison <[email protected]>
Co-authored-by: Dmytro Struk <[email protected]>
### Motivation and Context
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
fixes#1915
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [ ] The code builds clean without any errors or warnings
- [ ] The PR follows SK Contribution Guidelines
(https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
- [ ] The code follows the .NET coding conventions
(https://learn.microsoft.com/dotnet/csharp/fundamentals/coding-style/coding-conventions)
verified with `dotnet format`
- [ ] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄
---------
Co-authored-by: Abby Harrison <[email protected]>
Co-authored-by: Abby Harrison <[email protected]>
Co-authored-by: Shawn Callegari <[email protected]>
### Motivation and Context
This PR adds Qdrant support for SK memory to the Python SK
### Description
- added `qdrant_memory_store.py`
- added `test_qdrant_memory_store.py`
- added dependency group `qdrant` to the pyproject.toml
- added dependency group `qdrant` to the `--without` list for python
unit tests
- updated poetry
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [ ] The code builds clean without any errors or warnings
- [ ] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#dev-scripts)
raises no violations
- [ ] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄
---------
Co-authored-by: Tara W <[email protected]>
Co-authored-by: Dmytro Struk <[email protected]>
### Motivation and Context
The milvus integration tests are failing on windows systems only - get
calls seem to be timing out. The integration tests rely on milvus-lite,
a local, lightweight milvus application for python. Milvus is not
officially supported for windows, so this PR disables the tests for any
windows platform.
See milvus requirements documentation at:
https://github.com/milvus-io/milvus-lite/blob/main/README.md#requirements
### Description
- mark the milvus tests as _skip_ if the local platform is windows.
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [ ] The code builds clean without any errors or warnings
- [ ] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#dev-scripts)
raises no violations
- [ ] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄
### Motivation and Context
The service was renamed to Azure Cognitive Search in 2019, so we should
follow the new convention. The only thing still named "Azure Search" is
the old Azure SDK, out of scope.
### Description
* Revert nuget package name to the previous name (we'll delete the new
package, which has been out only for few days)
* Rename C# and Python classes and namespaces
This change brings in Milvus to the semantic kernel connectors.
---------
Signed-off-by: Filip Haltmayer <[email protected]>
Co-authored-by: Abby Harrison <[email protected]>
Co-authored-by: Abby Harrison <[email protected]>
* Move Azure Search dependencies to own group
* Improve Azure Search integration tests: try to delete collections when
tests fail
* Wait 1 sec after upsert to reduce random failures
---------
Co-authored-by: Abby Harrison <[email protected]>
### Motivation and Context
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
Temporal solution for #1606
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [ ] The code builds clean without any errors or warnings
- [ ] The PR follows SK Contribution Guidelines
(https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
- [ ] The code follows the .NET coding conventions
(https://learn.microsoft.com/dotnet/csharp/fundamentals/coding-style/coding-conventions)
verified with `dotnet format`
- [ ] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄
---------
Co-authored-by: Shawn Callegari <[email protected]>
Co-authored-by: Abby Harrison <[email protected]>
Co-authored-by: Abby Harrison <[email protected]>
* .NET, Python: Add new Memory Storage based on Azure Cognitive Search
with custom embeddings
* .NET: Delete previous Azure Search semantic memory component based on old
Azure Semantic Search w/ Lucene
### Motivation and Context
This PR addresses a number of failing integration tests. Chroma,
ConversationSummary Skill, and Pinecone.
- Chroma memory store tests broke due to a chroma dependency update
where the metadata checks for booleans (which are not allowed).
Previously, boolean metadata was not supported, but failed to validate
correctly. This PR gets around this by converting booleans to strings
when storing metadata, and converting string "True" to a boolean upon
metadata retrieval.
- ConversationSummarySkill added a callback parameter to all calls to
estimate the token count for the LLM call. This PR ensures that all
these calls have a default callback function to ensure backwards
compatibility with applications.
- This PR introduces retry logic to the Pinecone integration tests. The
tests are backed by the free tier of Pinecone which is a shared
instance. Traffic is high and unreliable causing the integration tests
to fail consistently. Retry logic addresses a lot of the unreliability,
but it is not a perfect fix. **We should address the backing service in
parallel to this change.**
- Add new dependencies to postgres to account for different install
needs for linux/macos/windows
### Description
- converts chroma boolean params to strings upon metadata save
- converts chroma boolean params from strings upon metadata retrieve
- all conversationsummaryskill methods have a default callback for token
count
- Adds a 1 second delay between each pinecone test
- Adds retry logic with progressive increases in waits between calls
should a pinecone request fail on known errors caused by service
availability
Action:
https://github.com/microsoft/semantic-kernel/actions/runs/5547243155
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [ ] The code builds clean without any errors or warnings
- [ ] The PR follows SK Contribution Guidelines
(https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
- [ ] The code follows the .NET coding conventions
(https://learn.microsoft.com/dotnet/csharp/fundamentals/coding-style/coding-conventions)
verified with `dotnet format`
- [ ] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄
---------
Co-authored-by: Dmytro Struk <[email protected]>
Co-authored-by: Shawn Callegari <[email protected]>
### Motivation and Context
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
Brings parity to python implementation
2. What problem does it solve?
With the pgvector extension, Postgres is a viable option for the storing
of embeddings for memory in Semantic Kernel.
The C# implementation is already capable of this, but the feature has
yet to be added to python.
3. What scenario does it contribute to?
Additional options for memory storage
4. If it fixes an open issue, please link to the issue here.
-->
### Description
Implemention of the MemoryStore to use a Postgresql (with pgvector
extension) to resolve#1270
Taking a different approach from the .net implementation:
Collections as postgres schemas as opposed to collection being another
column in the table, willing to change to match but would love to have a
discussion comparing the options.
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [x] The code builds clean without any errors or warnings
- [x] The PR follows SK Contribution Guidelines
(https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
- [x] The code follows the .NET coding conventions
(https://learn.microsoft.com/dotnet/csharp/fundamentals/coding-style/coding-conventions)
verified with `dotnet format`
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄
---------
Co-authored-by: Dmytro Struk <[email protected]>
Co-authored-by: Shawn Callegari <[email protected]>
Co-authored-by: Abby Harrison <[email protected]>
Co-authored-by: Abby Harrison <[email protected]>
### Motivation and Context
The pinecone integration tests can run locally, but not in CI. CI has no
way of picking up the pinecone environment.
Pinecone tests rely on the following environment variables in CI:
- Pinecone__ApiKey
- Pinecone__Environment
`get_pinecone_config` in `test_pinecone.py` currently only accounts for
Pinecone__ApiKey
### Description
This PR changes the method `get_pinecone_config` in `test_pinecone.py`
to return the pinecone environment from an environment variable if the
tests are being run in CI.
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [ ] The code builds clean without any errors or warnings
- [ ] The PR follows SK Contribution Guidelines
(https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
- [ ] The code follows the .NET coding conventions
(https://learn.microsoft.com/dotnet/csharp/fundamentals/coding-style/coding-conventions)
verified with `dotnet format`
- [ ] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄
Co-authored-by: Devis Lucato <[email protected]>
### Motivation and Context
Adding a Pinecone Memory store using the Pinecone vector database client
for SK python
Add Pinecone vector db memory to SK Python
Memory storage for SK and vectors
### Description
- Using the existing pinecone python module/client
- Reviewing with console exceptions responses that come back from
pinecone, will update if any were omitted
- Currently investigating and would like feedback if I should respond to
throttling errors in a different manner
- Currently testing each API response, slight delay may occur due to
throttling of my starter account in Pinecone.
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [ ] The code builds clean without any errors or warnings
- [X] The PR follows SK Contribution Guidelines
(https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
- [ ] The code follows the .NET coding conventions
(https://learn.microsoft.com/dotnet/csharp/fundamentals/coding-style/coding-conventions)
verified with `dotnet format`
- [ ] All unit tests pass, and I have added new tests where possible
- [X] I didn't break anyone 😄
---------
Co-authored-by: Lee Miller <[email protected]>
Co-authored-by: Casey Schadewitz <[email protected]>
Co-authored-by: cschadewitz <[email protected]>
Co-authored-by: Abby Harrison <[email protected]>
Co-authored-by: Devis Lucato <[email protected]>
Co-authored-by: Dmytro Struk <[email protected]>
Co-authored-by: Shawn Callegari <[email protected]>
### Motivation and Context
The addition of the field `additional metadata` to `MemoryRecord` was
not propagated to the weaviate integration tests. This PR ensures that
the fields are properly filled and aligned.
### Description
- Added an entry for `"additional info"` to the memory record test
fixtures for weaviate.
### Motivation and Context
Closes#1271 and addresses several TODOs in ChromaMemoryStore
### Description
- added field `additional_metadata` to `MemoryRecord` and all downstream
methods/classes (see `SemanticTextMemoryBase`, `SemanticTextMemory`,
`MemoryQueryResult`, and `ChromaMemoryStore`)
- addressed ChromaMemoryStore TODO 'what is key used for?' For chromadb,
key and id are effectively the same value, but the two fields are
differentiated for memory stores like Qdrant which require that key be a
uuid. I updated the logic so that this is more apparent. Note that this
results in no change in end-user functionality.
- Updated ChromaMemoryStore TODO 'get_async say embedding is optional
but Record constructor requires it.` MemoryRecord constructor no longer
requires an embedding. Logic is updated and a test added to show that
embedding retrieval is functionally optional.
### Motivation and Context
Python integration tests were failing consistently for 2 reasons. First,
all ubuntu system tests were failing due to missing shared object files
related to cuda. Second, macOS + python 3.11 tests were failing due to
hsnwlib unable to install on native hardware.
See https://github.com/microsoft/semantic-kernel/pull/1066 for
description of changes to integration test code to condense test code,
make better use of pytest fixtures, and reduce the number of calls to
external APIs (OpenAI, AOAI)
### Description
- Integration tests were failing on Linux due to a regression in python
torch=2.0.1. Downgrading the version to 2.0.0 resolved these failures.
- For MacOS + 3.11 specifically, setting the environment variable
`HNSWLIB_NO_NATIVE=1` ensures that the hnswlib wheel can be built and
installed properly on M1 hardware.
- Python integration tests reliably and consistently passing:
https://github.com/microsoft/semantic-kernel/actions/runs/5035748596
Eduard van Valkenburg