### Motivation and Context
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
extend mypy coverage for prompt_templates, template_engine and some
other smaller folders.
still todo: connectors, memory and planner folders.
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄
### Motivation and Context
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
Added a section on documentation to the DEV_SETUP
Added a check for the copyright row to pre-commit
Closes#5629
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄
### Motivation and Context
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
- Added static code checking to pre-commit (check-ast and
nbqa-check-ast)
- Added Bandit security checking to pre-commit
- Added pre-commit step to python-lint workflow, if it works, can delete
seperate mypy, ruff and black tests
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄
### Motivation and Context
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
Working through improving test coverage
Small changes introduced:
- ai_service_selector: select_ai_service method, now has extra optional
variable for type_, when not set the behavior is the same as before,
otherwise allows any type of AI service present to be selected,
including embeddings (which was not possible)
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄
---------
Co-authored-by: Evan Mattson <[email protected]>
### Motivation and Context
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
Now that we only support 310 and up, we could unify all files, to use
the new style typing and annotations.
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄
### Motivation and Context
During the dev of the feature branch, the service_id was added as an arg
to the import plugin from prompt directory; however, it was later
removed but not removed from the notebooks.
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
### Description
This PR:
- Removes the service_id from the import_plugin_from_prompt_directory
method as it is no longer present. Closes#5233
- Uses the `default` service_id which is included as part of the stock
plugins.
- Add the function_name and plugin_name that were missing from
create_function_from_prompt.
- Run the latest ruff (0.3.0) and black (24.2.0) formatting on all
python files. This is why other files show -- their formatting was
updated.
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [X] The code builds clean without any errors or warnings
- [X] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [X] All unit tests pass, and I have added new tests where possible
- [X] I didn't break anyone 😄
### Motivation and Context
<!-- Thank you for your contribution to the semantic-kernel repo! Please
help reviewers and future users, providing the following information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here. -->
Major work to replace the context with Kernel Arguments on the front-end
and Function Results on the back.
Updated the function decorator to a new approach, in line with dotnet.
Revamps the way function are called, allowing native functions to be
completely ignorant of SK, other then the decorator.
This also moves things into the new folder structure in sync with
dotnet.
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
Adds:
- KernelArguments, a dict-like class that replaces the variables in
KernelContext. Closes#4565
- FunctionResult, a class to hold the results of function executions,
includes the function, the value and metadata, as well as two convenient
function to get the value out of it (str and get_inner_content) the
first is generic, the second specifically for KernelContent responses
(from AI services).
- AI Service Selector class, has a default, which is based on the order
in the arguments followed by the order in the functions, can be
overridden to implement your own strategy. Closes#4631
- Introduces ChatHistory and refactors the PromptTemplateConfig. Closes
#4856, #4630
- Improves parsing of templates, will now all validate during creation
and throw an error then, instead of some that do not check for validaty
until used.
- Introduces named_args block and thereby the ability to have multiple
arguments for a function call in a template. Closes#5003
Updates:
- kernel_function decorators, the parameter decorator was removed and
instead we now use Annotated to add a description of a field and we get
the type and required from the function definition.
- core plugins, use the new approach to kernel_function decorators.
- planners, template engines have all been updated to use the kernel and
kernelarguments instead of Context.
- Events have been updated, now use kernelarguments and function_result
- Tokenizers support for named_args and improvements on parsing and
checking.
- Kernel examples and notebooks to use the latest code.
- All unit and integration tests. There is more code coverage now than
before.
Removed:
- kernelContext
- kernel_function_parameter_decorator
- delegate handling code for native functions
- file_io_plugin and tests
- SemanticFunctionConfig
- ChatPromptTemplate
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄
---------
Co-authored-by: Evan Mattson <[email protected]>
Co-authored-by: Evan Mattson <[email protected]>
Co-authored-by: Evan Mattson <[email protected]>
### Motivation and Context
Completes #4794.
Most of SK Python is written with async code; however, there were some
code paths that masked the underlying async code and made it look like a
sync function invoke. In reality, asyncio spun up event loops and
attempted to manage them properly. This becomes tricky when running a
Jupyter notebook as they have their own event loop.
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
### Description
This PR removes the ability to call a kernel function synchronously.
Since the majority of the code base is written with async methods, and
to match dotnet, which is async, we're removing the sync function invoke
and its dependent code. Some Planners were also run synchronously --
these were updated to run asynchronously. Kernel examples, notebooks,
docs, and tests were updated accordingly.
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [X] The code builds clean without any errors or warnings
- [X] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [X] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄
### Motivation and Context
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
As we are tightening the formatting and linting setups, this was
missing, now the line-length for black and ruff are the same, thereby no
longer running into black issues as often.
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄
### Motivation and Context
This PR addresses a number of failing integration tests. Chroma,
ConversationSummary Skill, and Pinecone.
- Chroma memory store tests broke due to a chroma dependency update
where the metadata checks for booleans (which are not allowed).
Previously, boolean metadata was not supported, but failed to validate
correctly. This PR gets around this by converting booleans to strings
when storing metadata, and converting string "True" to a boolean upon
metadata retrieval.
- ConversationSummarySkill added a callback parameter to all calls to
estimate the token count for the LLM call. This PR ensures that all
these calls have a default callback function to ensure backwards
compatibility with applications.
- This PR introduces retry logic to the Pinecone integration tests. The
tests are backed by the free tier of Pinecone which is a shared
instance. Traffic is high and unreliable causing the integration tests
to fail consistently. Retry logic addresses a lot of the unreliability,
but it is not a perfect fix. **We should address the backing service in
parallel to this change.**
- Add new dependencies to postgres to account for different install
needs for linux/macos/windows
### Description
- converts chroma boolean params to strings upon metadata save
- converts chroma boolean params from strings upon metadata retrieve
- all conversationsummaryskill methods have a default callback for token
count
- Adds a 1 second delay between each pinecone test
- Adds retry logic with progressive increases in waits between calls
should a pinecone request fail on known errors caused by service
availability
Action:
https://github.com/microsoft/semantic-kernel/actions/runs/5547243155
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [ ] The code builds clean without any errors or warnings
- [ ] The PR follows SK Contribution Guidelines
(https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
- [ ] The code follows the .NET coding conventions
(https://learn.microsoft.com/dotnet/csharp/fundamentals/coding-style/coding-conventions)
verified with `dotnet format`
- [ ] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄
---------
Co-authored-by: Dmytro Struk <[email protected]>
Co-authored-by: Shawn Callegari <[email protected]>
### Motivation and Context
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
addressing issue #1387
chunking text should allow use of an external tokenizer
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
added pass-through of an token counting function, defaulting to the
existing _token_counter() method
while fixing a type hint bug I got sucked into making a few changes to
clean up the code.
future work would be nice to add chunk overlap functionality similar to
langchain's [TextSplitter](
https://github.com/hwchase17/langchain/blob/a6ebffb69504576a805f3b9f09732ad344751b89/langchain/text_splitter.py#L58)
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [X] The code builds clean without any errors or warnings
- [X] The PR follows SK Contribution Guidelines
(https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
- [X] The code follows the .NET coding conventions
(https://learn.microsoft.com/dotnet/csharp/fundamentals/coding-style/coding-conventions)
verified with `dotnet format`
- [X] All unit tests pass, and I have added new tests where possible
- [X] I didn't break anyone 😄
---------
Co-authored-by: Scott Graham <[email protected]>
Co-authored-by: Lee Miller <[email protected]>
Co-authored-by: Mark Karle <[email protected]>
Co-authored-by: Shawn Callegari <[email protected]>
### Motivation and Context
This PR introduces native python support for Hugging Face models that
can: complete text, generate new text, summarize, and that generate
embeddings. Currently only supports downloading the models locally from
the HF model hub. Future plans include supporting the HF inference API
as well.
### Description
- Added 2 services: `hf_text_completion` and `hf_text_embedding`
- `hf_text_completion` supports the following tasks: _text-generation_,
_text2text-generation_, and _summarization_
- `hf_text_embedding` supports any model supported by the
sentence-transformers pip package
- Added dependencies: pytorch, transformers, sentence-transformers to
`requirements.txt` and `poetry.lock`
- fixed typo: `get_embedding_service_service_id` ->
`get_embedding_service_id`
- Added a number of integration tests for supported HF models
### Motivation and Context
Porting the text partitioning module to python
(Reopening of PR #427 )
### Description
- This is adding the Text partitioning module in
`semantic_kernel/semantic_functions/semantic_text_partitioner.py`
and the function_extention in
`semantic_kernel/semantic_functions/function_extension.py`
- Compared to the C# version the files were added directly into the
`semantic_functions` directory instead of the
`semantic_functions/partitioning` to not have too many nested
directories.
Eduard van Valkenburg