### Motivation and Context
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
Overhaul of the VectorStores to get it matching with dotnet and the
latest development there.
Closes:
- #11938
- #11598
- #11597
- #11517
- #10561
- #10391
- #9911
- #9892
- #10867
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
Many changes, some highlights:
- Moved from different vector fields types, to a single with a enum
first VectorStoreField()
- Renamed several items
- Made embeddinggenerrator a part of the vector field spec and do
automatic vectorization from there or from the same param on a
collection
- Adds hybrid search using the same setup
- Removed the intermediate TextSeachVectorSearch object need, by
allowing `create_search_function` directly on a collection
Will write out full changes in a migration guide in the docs.
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄
### Motivation and Context
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
Introducing the Pinecone Vector Store and Collection
This also upgrades the dependencies to v6
Adding the ability to run with asyncio or GRPC
TODO:
- [x] Filtering
- [x] Tests
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
Adds PineconeStore and PineconeCollection
Closes: #9928
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄
Implement a new version of the MongoDB Atlas Memory Connector with
MongoDBAtlasStore and MongoDBAtlasCollection classes.
* **MongoDBAtlasStore**:
- Add
`python/semantic_kernel/connectors/memory/mongodb_atlas/mongodb_atlas_store.py`
to implement the MongoDBAtlasStore class.
- Add unit tests in
`python/tests/unit/connectors/memory/mongodb_atlas/test_mongodb_atlas_store.py`
to test the MongoDBAtlasStore class.
* **MongoDBAtlasCollection**:
- Add
`python/semantic_kernel/connectors/memory/mongodb_atlas/mongodb_atlas_collection.py`
to implement the MongoDBAtlasCollection class.
- Add unit tests in
`python/tests/unit/connectors/memory/mongodb_atlas/test_mongodb_atlas_collection.py`
to test the MongoDBAtlasCollection class.
---
For more details, open the [Copilot Workspace
session](https://copilot-workspace.githubnext.com/microsoft/semantic-kernel/pull/10177?shareId=e9e73180-0fff-4e60-bbb5-5c3493b27e61).
closes#6764
---------
Co-authored-by: Evan Mattson <[email protected]>
### Motivation and Context
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
Add Weaviate DB to Semantic Kernel with the new vector store and vector
collection interfaces.
Address: https://github.com/microsoft/semantic-kernel/issues/6553
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄
---------
Co-authored-by: Evan Mattson <[email protected]>
### Motivation and Context
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
This PR adds new vector store and vector store record collections as
well as implementations for:
- Azure AI Search
- Redis
- Qdrant
- Volatile (in-memory)
It also adds samples, tests, and unit-tests for these.
Next it adds the vector store record fields, definition and decorator,
with tests and samples.
All marked experimental. Existing Redis, Azure AI Search, Qdrant and
Volatile will be marked as deprecated in the future, once the new
collections are feature complete.
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄
### Motivation and Context
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
Updated coveragerc and mypy files in prep of spike
Moved Text tests into single integration test setup
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄
---------
Co-authored-by: Evan Mattson <[email protected]>
### Motivation and Context
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
Working through improving test coverage
Small changes introduced:
- ai_service_selector: select_ai_service method, now has extra optional
variable for type_, when not set the behavior is the same as before,
otherwise allows any type of AI service present to be selected,
including embeddings (which was not possible)
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄
---------
Co-authored-by: Evan Mattson <[email protected]>
### Motivation and Context
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
Got a report of a bug in the action planner creation, fixed here.
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄
### Motivation and Context
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
Updates the name of the app info key to Semantic-Kernel-Version to match
dotnet, closes#5038
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄
### Motivation and Context
<!-- Thank you for your contribution to the semantic-kernel repo! Please
help reviewers and future users, providing the following information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here. -->
Major work to replace the context with Kernel Arguments on the front-end
and Function Results on the back.
Updated the function decorator to a new approach, in line with dotnet.
Revamps the way function are called, allowing native functions to be
completely ignorant of SK, other then the decorator.
This also moves things into the new folder structure in sync with
dotnet.
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
Adds:
- KernelArguments, a dict-like class that replaces the variables in
KernelContext. Closes#4565
- FunctionResult, a class to hold the results of function executions,
includes the function, the value and metadata, as well as two convenient
function to get the value out of it (str and get_inner_content) the
first is generic, the second specifically for KernelContent responses
(from AI services).
- AI Service Selector class, has a default, which is based on the order
in the arguments followed by the order in the functions, can be
overridden to implement your own strategy. Closes#4631
- Introduces ChatHistory and refactors the PromptTemplateConfig. Closes
#4856, #4630
- Improves parsing of templates, will now all validate during creation
and throw an error then, instead of some that do not check for validaty
until used.
- Introduces named_args block and thereby the ability to have multiple
arguments for a function call in a template. Closes#5003
Updates:
- kernel_function decorators, the parameter decorator was removed and
instead we now use Annotated to add a description of a field and we get
the type and required from the function definition.
- core plugins, use the new approach to kernel_function decorators.
- planners, template engines have all been updated to use the kernel and
kernelarguments instead of Context.
- Events have been updated, now use kernelarguments and function_result
- Tokenizers support for named_args and improvements on parsing and
checking.
- Kernel examples and notebooks to use the latest code.
- All unit and integration tests. There is more code coverage now than
before.
Removed:
- kernelContext
- kernel_function_parameter_decorator
- delegate handling code for native functions
- file_io_plugin and tests
- SemanticFunctionConfig
- ChatPromptTemplate
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄
---------
Co-authored-by: Evan Mattson <[email protected]>
Co-authored-by: Evan Mattson <[email protected]>
Co-authored-by: Evan Mattson <[email protected]>
This reverts commit ce1ebe1611.
### Motivation and Context
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [ ] The code builds clean without any errors or warnings
- [ ] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [ ] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄
### Motivation and Context
This PR is to add a [Python coverage
comment](https://github.com/marketplace/actions/python-coverage-comment)
GitHub action to the CICD workflow, so that comments related to coverage
change will be posted to a PR automatically.
In this PR:
1. I updated the `DEV_SETUP.md` to include some instructions on how to
enable `pre-commit`
2. I added the `coverage` package as the dev dependency
3. I updated the `.coveragerc` file so that `.coverage` is generated in
relative path
4. I added the workflow for generating and posting coverage comments.
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
5. What problem does it solve?
6. What scenario does it contribute to?
7. If it fixes an open issue, please link to the issue here.
-->
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄
---------
Co-authored-by: Huijing Huang <[email protected]>
Co-authored-by: Evan Mattson <[email protected]>
### Motivation and Context
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
This PR implements the new approach to AI Request settings as described
here:
https://github.com/microsoft/semantic-kernel/blob/main/docs/decisions/0008-support-generic-llm-request-settings.md
and completes #3312
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
Does this:
- Creates AI Request Settings with service_id (referring to the
registered AI services in a kernel) and extension_data, a dict that can
hold any number of settings
- has several methods:
- `from_ai_request_settings` -> constructor to create a new
request_settings object based on another, supports creating specific
ai_request_settings from generic ones.
- `update_from_ai_request_settings` -> updating the fields of a request
setting from another ones.
- `prepare_settings_dict` -> method that is used when actually creating
the dict that get's passed to the handler of the AI service.
- Created subclasses like OpenAIChatRequestSettings which has all the
required fields of a OpenAI Chat completion, same for OpenAIText,
AzureChat, GooglePalm, HuggingFace, etc.
- added `get_request_settings_class` to ai_services to return the class
used for the settings for that service.
The goal is that when running, this settings object get's updated and a
single operations (called `prepare_settings_dict`) is used to return a
dict that can be passed to the api call directly, thereby reducing the
need for custom logic everywhere, this will also negate the need for
things like `complete_chat_with_functions_async` and others. So also
removed complete_chat_..._async methods from openai and azure openai.
It has also changed the prompt_template_config with the generic type for
the AIRequestSettings or subclasses, and some other changes.
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄
### Motivation and Context
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
To improve the code and PR quality I want to add code coverage to the
dev dependencies, and setup. I then also added tasks into the python
folder for both the pre-commit config and running unit tests with code
coverage. Fixes#3980
Initially I excluded code coverage for all connectors, that needs to be
refined and this can also be added to the GH actions once we have it
setup well.
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄
Eduard van Valkenburg