### Motivation and Context
Completes #4794.
Most of SK Python is written with async code; however, there were some
code paths that masked the underlying async code and made it look like a
sync function invoke. In reality, asyncio spun up event loops and
attempted to manage them properly. This becomes tricky when running a
Jupyter notebook as they have their own event loop.
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
### Description
This PR removes the ability to call a kernel function synchronously.
Since the majority of the code base is written with async methods, and
to match dotnet, which is async, we're removing the sync function invoke
and its dependent code. Some Planners were also run synchronously --
these were updated to run asynchronously. Kernel examples, notebooks,
docs, and tests were updated accordingly.
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [X] The code builds clean without any errors or warnings
- [X] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [X] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄
### Motivation and Context
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
As we are tightening the formatting and linting setups, this was
missing, now the line-length for black and ruff are the same, thereby no
longer running into black issues as often.
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄
### Motivation and Context
This PR addresses a number of failing integration tests. Chroma,
ConversationSummary Skill, and Pinecone.
- Chroma memory store tests broke due to a chroma dependency update
where the metadata checks for booleans (which are not allowed).
Previously, boolean metadata was not supported, but failed to validate
correctly. This PR gets around this by converting booleans to strings
when storing metadata, and converting string "True" to a boolean upon
metadata retrieval.
- ConversationSummarySkill added a callback parameter to all calls to
estimate the token count for the LLM call. This PR ensures that all
these calls have a default callback function to ensure backwards
compatibility with applications.
- This PR introduces retry logic to the Pinecone integration tests. The
tests are backed by the free tier of Pinecone which is a shared
instance. Traffic is high and unreliable causing the integration tests
to fail consistently. Retry logic addresses a lot of the unreliability,
but it is not a perfect fix. **We should address the backing service in
parallel to this change.**
- Add new dependencies to postgres to account for different install
needs for linux/macos/windows
### Description
- converts chroma boolean params to strings upon metadata save
- converts chroma boolean params from strings upon metadata retrieve
- all conversationsummaryskill methods have a default callback for token
count
- Adds a 1 second delay between each pinecone test
- Adds retry logic with progressive increases in waits between calls
should a pinecone request fail on known errors caused by service
availability
Action:
https://github.com/microsoft/semantic-kernel/actions/runs/5547243155
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [ ] The code builds clean without any errors or warnings
- [ ] The PR follows SK Contribution Guidelines
(https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
- [ ] The code follows the .NET coding conventions
(https://learn.microsoft.com/dotnet/csharp/fundamentals/coding-style/coding-conventions)
verified with `dotnet format`
- [ ] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄
---------
Co-authored-by: Dmytro Struk <[email protected]>
Co-authored-by: Shawn Callegari <[email protected]>
### Motivation and Context
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
addressing issue #1387
chunking text should allow use of an external tokenizer
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
added pass-through of an token counting function, defaulting to the
existing _token_counter() method
while fixing a type hint bug I got sucked into making a few changes to
clean up the code.
future work would be nice to add chunk overlap functionality similar to
langchain's [TextSplitter](
https://github.com/hwchase17/langchain/blob/a6ebffb69504576a805f3b9f09732ad344751b89/langchain/text_splitter.py#L58)
### Contribution Checklist
<!-- Before submitting this PR, please make sure: -->
- [X] The code builds clean without any errors or warnings
- [X] The PR follows SK Contribution Guidelines
(https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
- [X] The code follows the .NET coding conventions
(https://learn.microsoft.com/dotnet/csharp/fundamentals/coding-style/coding-conventions)
verified with `dotnet format`
- [X] All unit tests pass, and I have added new tests where possible
- [X] I didn't break anyone 😄
---------
Co-authored-by: Scott Graham <[email protected]>
Co-authored-by: Lee Miller <[email protected]>
Co-authored-by: Mark Karle <[email protected]>
Co-authored-by: Shawn Callegari <[email protected]>
### Motivation and Context
This PR introduces native python support for Hugging Face models that
can: complete text, generate new text, summarize, and that generate
embeddings. Currently only supports downloading the models locally from
the HF model hub. Future plans include supporting the HF inference API
as well.
### Description
- Added 2 services: `hf_text_completion` and `hf_text_embedding`
- `hf_text_completion` supports the following tasks: _text-generation_,
_text2text-generation_, and _summarization_
- `hf_text_embedding` supports any model supported by the
sentence-transformers pip package
- Added dependencies: pytorch, transformers, sentence-transformers to
`requirements.txt` and `poetry.lock`
- fixed typo: `get_embedding_service_service_id` ->
`get_embedding_service_id`
- Added a number of integration tests for supported HF models
### Motivation and Context
Porting the text partitioning module to python
(Reopening of PR #427 )
### Description
- This is adding the Text partitioning module in
`semantic_kernel/semantic_functions/semantic_text_partitioner.py`
and the function_extention in
`semantic_kernel/semantic_functions/function_extension.py`
- Compared to the C# version the files were added directly into the
`semantic_functions` directory instead of the
`semantic_functions/partitioning` to not have too many nested
directories.
Evan Mattson