### Motivation and Context
In the Python `BasicPlanner` one of the examples has a 'translate'
function, but doesn't quite reference it correctly in the JSON output.
### Description
The overall approach is to make the various parts of the few-shot
example in the prompt self-consistent.
### Motivation and Context
Python integration tests were failing consistently for 2 reasons. First,
all ubuntu system tests were failing due to missing shared object files
related to cuda. Second, macOS + python 3.11 tests were failing due to
hsnwlib unable to install on native hardware.
See https://github.com/microsoft/semantic-kernel/pull/1066 for
description of changes to integration test code to condense test code,
make better use of pytest fixtures, and reduce the number of calls to
external APIs (OpenAI, AOAI)
### Description
- Integration tests were failing on Linux due to a regression in python
torch=2.0.1. Downgrading the version to 2.0.0 resolved these failures.
- For MacOS + 3.11 specifically, setting the environment variable
`HNSWLIB_NO_NATIVE=1` ensures that the hnswlib wheel can be built and
installed properly on M1 hardware.
- Python integration tests reliably and consistently passing:
https://github.com/microsoft/semantic-kernel/actions/runs/5035748596
### Motivation and Context
Use of abstraction was not consistent in the python SK. This PR makes
the import and use of ABC consistent and removes unused file kernel.py.
### Description
- MemoryStoreBase derives from ABC - this properly sets its metaclass.
MemoryStoreBase itself should never be instantiated, only derived.
- RetryMechanism derives ABC, so should be named 'Base' like all other
derivatives of ABC in this repo.
- removed kernel_base.py, was made obsolete by
https://github.com/microsoft/semantic-kernel/pull/1015
### Motivation and Context
#691#1053
### Description
1. Added an argument to prevent TypeError when inferring an unknown
type.
2. Fixed a bug in the process of checking parameters when inferring the
InStringAndContext type.
### Motivation and Context
The separation of Kernel, KernelBase, and KernelExtensions is several
layers of unnecessary abstraction. This PR does not change any API or
kernel behavior - it just reduces the number of files and namespaces to
accomplish the same functionality.
### Description
- Removed the `semantic_kernel/kernel_extensions` namespace. Methods
that existed in this namespace now exist in kernel.py.
- Removed KernelBase - this abstract class implies that there are
multiple ways to instantiate SK. For Python, we expect only a single
type of Kernel to instantiated for all instances.
- Updated tests and imports to accommodate the above changes.
Add support for Chroma https://docs.trychroma.com/
> Chroma is the open-source embedding database. Chroma makes it easy to
build LLM apps by making knowledge, facts, and skills pluggable for
LLMs.
### Motivation and Context
* #403 Support for Chroma embedding database
* #426 Python: feat: add chroma memory store
* #449 Python: feat: add chroma memory store
---------
Co-authored-by: Abby Harrison <[email protected]>
Co-authored-by: Abby Harrison <[email protected]>
Co-authored-by: Devis Lucato <[email protected]>
This commit fixes a typo in the e2e_text_completion.py test file, where
the word "signularity" was misspelled as "singularity". This could cause
confusion or errors in the test output. The typo was corrected to
"singularity" to match the intended meaning of the poem. No other
changes were made to the test file or the functionality of the chatbot.
### Motivation and Context
Porting conversation summary skill. Integration test is added as well.
Action item and topic will be ported later.
---------
Co-authored-by: Po-Wei Huang <[email protected]>
Co-authored-by: Abby Harrison <[email protected]>
Co-authored-by: Abby Harrison <[email protected]>
Co-authored-by: Devis Lucato <[email protected]>
Co-authored-by: Devis Lucato <[email protected]>
### Motivation and Context
Using both OpenAI and Azure OpenAI services would not work due to the
way they were configured. The `openai` module would be configured
globally with keys and endpoints for one type of service and then
overwritten when adding another service. Instead, we can configure the
settings when making the API calls.
Also, some integration tests are failing because of throttling by GPT4.
### Description
- Use the API settings when calling the openai API instead of
configuring gloabally
- Increased delay to 3 seconds for some integration tests to avoid rate
limit errors
- Relaxed the assertion strings for some integration tests
- Ran pre-commit linting
- Switch some tests to GPT Turbo 3.5
---------
Co-authored-by: Devis Lucato <[email protected]>
### Motivation and Context
Port MathSkill from C# to python3. This is essential for advanced skill
like plan
### Description
The PR includes two part. The first part is an implementation of
MathSkill. The implementation follows C# version.
It does several things:
1. Parse initial value
2. Get value from context
3. Sum or subtract
The unit test covers these scenarios:
1. Add/subtract value
2. Error should throw if string is not correct. Not able to parse.
Last part is the init.py and feature matrix.
Unit test pass locally.
Co-authored-by: Po-Wei Huang <[email protected]>
Co-authored-by: Mark Karle <[email protected]>
Co-authored-by: Devis Lucato <[email protected]>
### Motivation and Context
* Removing the requirements.txt and just pinning each notebook to a pip
package that should work.
Doing this allows users to run notebooks without error even if the code
in the core repo updates.
When a new pip package is released, we will bump the version number in
each notebook.
* Fix usage of context variables. Apps should avoid using anonymous
context variables and prefer using the context, to retain access to the
kernel internals during the orchestration.
---------
Co-authored-by: Alex Chao <[email protected]>
Co-authored-by: Devis Lucato <[email protected]>
* Fix lint errors
* Bump version number
* Add retry loop to integration tests
* Enable all integration tests: some tests will fail if OpenAI/Azure are
throttling.
* Fix#735: broken links in pip README. Sync nuget and pip readmes.
* Update planning doc, use plural for "planners".
* Add RAG to list of patterns supported by SK.
### Motivation and Context
KernelConfig isn't needed as it's less pythonic and is not used
independently from Kernel. This moves all its methods and members into
Kernel.
### Description
- Removed KernelConfig
- Put KernelConfig's methods and variables into Kernel
- Ran unit and end-to-end tests
- Did not run notebooks because they are out of date with the current
repo.
---------
Co-authored-by: Shawn Callegari <[email protected]>
calling the memory record's embedding property throws
`RecursionError: maximum recursion depth exceeded`
return the attribute instead of the property
Signed-off-by: hsm207 <[email protected]>
### Motivation and Context
The release of Python SK from 0.2.4 -> 0.2.6 significantly increased the
size of the pip package. The dependencies causing this are not core to
the kernel, so this PR establishes a pattern for SK connector
dependencies (ex: Hugging Face, memory storage)
### Description
- Added a poetry group called hugging_face - this is the same name as
the connectors namespace in the code.
- Added dependency installs to the Hugging Face sample notebook.
- removed Hugging Face dependencies from requirements.txt.
- Added import errors for torch, transformers, and sentence-transformers
in the hugging_face connector classes.
- Added command `--with hugging_face` to integration test setup.
### Motivation and Context
<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
1. Why is this change required?
2. What problem does it solve?
3. What scenario does it contribute to?
4. If it fixes an open issue, please link to the issue here.
-->
python: fix the issue of `No module named 'semantic_kernel.planning'`.
### Description
<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
Add __init__ file for the planning package.
### Motivation and Context
Many users and customers have expressed interest in having planning
capabilities in Python. This PR seeks to provide a basic version of the
planner so that people can learn and play around with the concepts and
can be comfortable with implementing their own versions.
A Jupyter notebook is included.
### Description
Most of the logic of the plan can be found under the `PROMPT` in
`basic_planner.py`.
There is a known limitation that this currently doesn't explicit parse
out arguments that are defined in the semantic function prompts, so when
creating the [AVAILABLE FUNCTIONS] string, it just uses the name of the
function and the description.
The execution of the plan also happens from start to finish. There is
currently no support for piecewise execution.
---------
Co-authored-by: Alex Chao <[email protected]>
Co-authored-by: Alex Chao <[email protected]>
Co-authored-by: Abby Harrison <[email protected]>
Co-authored-by: Mark Karle <[email protected]>
### Motivation and Context
Run python integration tests on every push to main, and every 12 hours.
These integration tests will run on every push to main as well as twice
a day. At the time of this PR, we are seeing ~3 PRs to the python SK per
day. Each Integration test run will make ~32 AI requests. For each PR,
12 runs will be made: 3 each for python 3.8, 3.9, 3.10, and 3.11 for a
total of ~400 calls per push to main.
We should re-evaluate the trigger on pushes to main if PRs to the python
SK significantly increase.
### Description
- add workflow definition for integration tests
- Example run:
https://github.com/microsoft/semantic-kernel/actions/runs/4874609026/jobs/8695771345?pr=763
- For push triggers, add path checks for dotnet/** or python/**
depending on the type of workflow
- pinned requirements.txt dependencies to the versions pinned in the
pyproject.toml file to ensure parity between pip package dependencies
and local code
**Note**: Integration tests are failing on windows with python 3.8 -
already captured a task to address the problem
### Motivation and Context
The poetry dependency file used black formatter version 23.1.0 and the
pre-commit config used version 23.3.0. I updated black to 23.3.0 in the
poetry dependency file.
### Motivation and Context
New release version 0.2.6
Fix bug in 0.2.5.dev0 pip package
Update notebooks to the latest pip package (0.2.6)
### Description of Changes
- fixed a param name error in the creation of a reference memory_record
- renamed text_service -> text_completion_service (consistency with .NET)
- renamed text_embedding_service -> text_embedding_generation_service
(consistency with .NET)
- Updated notebooks accordingly
- bumped pip versions to 0.2.6.dev0
### Motivation and Context
This PR introduces the ability to run formal integration tests for
python using pytest - separated from unit tests for easier automation.
### Description
- moved `python/tests/end-to-end` directory to
`python/tests/integration`
- renamed integration tests to test_* pattern so pytest will detect them
- import `GenerationConfig` for hugging face completions - according to
logs, this is the recommended practice
### Motivation and Context
This PR significantly reduces the complexity of implementing a new
memory store. Previously, memory stores needed to implement
EmbeddingIndexBase, DataStoreBase, and MemoryStoreBase. Now, memory
stores only need to implement MemoryStoreBase. This PR also brings
MemoryStoreBase to parity with the .NET IMemoryStore interface.
### Description
- Removed `data_entry.py`, `data_store_base.py`,
`volatile_data_store.py`, and `embedding_index_base.py`
- Added `with_embedding` parameter to all get/search methods. Passing
the embedding between the application and storage is not optional and
defaults to False
- `MemoryStoreBase` (Python SK) == `IMemoryStore` (.NET SK)
- Updated `volatile_memory_store` to implement `MemoryStoreBase`
- Removed `ABC` from `MemoryStoreBase`
- Added documentation to memory files
Closes https://github.com/microsoft/semantic-kernel/issues/670
### Motivation and Context
This PR introduces native python support for Hugging Face models that
can: complete text, generate new text, summarize, and that generate
embeddings. Currently only supports downloading the models locally from
the HF model hub. Future plans include supporting the HF inference API
as well.
### Description
- Added 2 services: `hf_text_completion` and `hf_text_embedding`
- `hf_text_completion` supports the following tasks: _text-generation_,
_text2text-generation_, and _summarization_
- `hf_text_embedding` supports any model supported by the
sentence-transformers pip package
- Added dependencies: pytorch, transformers, sentence-transformers to
`requirements.txt` and `poetry.lock`
- fixed typo: `get_embedding_service_service_id` ->
`get_embedding_service_id`
- Added a number of integration tests for supported HF models
### Motivation and Context
Porting the text partitioning module to python
(Reopening of PR #427 )
### Description
- This is adding the Text partitioning module in
`semantic_kernel/semantic_functions/semantic_text_partitioner.py`
and the function_extention in
`semantic_kernel/semantic_functions/function_extension.py`
- Compared to the C# version the files were added directly into the
`semantic_functions` directory instead of the
`semantic_functions/partitioning` to not have too many nested
directories.
### Motivation and Context
ve_skill_from_directory`, I've noticed that if
there are 2 or more similar directories(see dir tree below), it doesn't
work correctly.
```
(base) joowonkim ~/Repo/sk-consultings/example-thinkpool/poc1-architecture/skills main ± tree
.
├── handler_skills
│ ├── AccountHandlerSkill
│ │ └── native_function.py
│ ├── PortfolioHandlerSkill
│ │ └── native_function.py
│ └── StockHandlerSkill
│ └── native_function.py
├── intent_skills
│ └── IntentDetectionSkill
│ └── ParseIntent
│ ├── config.json
│ └── skprompt.txt
```
Importing using absolute paths rather than sys.paths reliably worked as
intended.
### Description
remove `sys.path` related codes and change them into using absolute path
of `native_function.py`
### Motivation and Context
Users want to use ChatGPT as the service for their semantic functions.
This feature exists with the C# SK. ChatGPT is perfectly capable of
doing this. Additionally added more end-to-end tests with the intention
of being able to run them as integration tests for the Python SK.
### Description
- `OpenAIChatCompletion` now additionally inherits from
`TextCompletionClientBase`. This extends to `AzureOpenAIChatCompletion`.
- If a chat service is also a `TextCompletionClientBase`, it will be
added to both `text_services` and `chat_services`.
- Created 4 end-to-end tests that target the combination of Azure,
OpenAI, text service, and chat service. Abstracted common behavior for
these tests.
- Added asserts to these end-to-end tests. These should be run as
integration tests, but that will be handled in a separate PR.
### Motivation and Context
This PR is a consistency update between the Python and .NET Semantic
Kernels.
**This PR does not contain any feature enhancements or improvements!**
### Description
- Replace terminology 'backend' with 'service'
- Move semantic_kernel.ai (semantic_kernel/ai) to
semantic_kernel.connectors.ai (semantic_kernel/connectors/ai)
### Motivation and Context
the method Today() in TimeSkill.cs and today() in TimeSkill.py was
missing.
Today must be an alias for Date.
### Description
Add the method Today.
Co-authored-by: Abby Harrison <[email protected]>
### Motivation and Context
It is confusing to have multiple skfunction invoke_* calls. This PR
combines invoke and invoke_with_vars to enable calls with the input
pattern: with SKContext, or with SKContextVariables.
### Description
- Removed `sk_function` methods `invoke_with_vars` and
`invoke_with_vars_async`
- Added the following optional paramters to `sk_function` methods
`invoke_` and `invoke_async`: `variables` and `memory`
- Added log warnings for when an both SKContext and variables/memory are
passed to `invoke` methods - this design ensures that the SDContext and
it's associated variables and memory are prioritized
- Updated tests and notebooks where applicable
### Motivation and Context
HttpSkill python version
### Description
- Porting the HttpSkill from C# to Python.
- Adding corresponding unit tests.
Co-authored-by: Abby Harrison <[email protected]>
### Motivation and Context
To make the Python SK more pythonic, combined all kernel.run_* methods
into a single method with optional parameters.
### Description
- Removed `kernel.run_on_vars_async()`
- Removed `kernel.run_on_str_async()`
- Added `SKContext`, `ContextVariables`, and string params to
`kernel.run_async()`
- Moved `run_on_vars_async()` logic to `run_async()` method
- Arranged method params so that required params come before optional
params.
- Updated tests and notebooks where applicable.
### Motivation and Context
With the official publishing of the Python Semantic Kernel, we should be
informing the public of feature parity between Python and C#. Python is
currently playing catch up.
### Description
Removed the FEATURE_PARITY.md file from the python directory and added a
FEATURE_MATRIX.md file to the top level of the repo. The main REAME.md
now links to this file. Added a stronger breakdown of features that we
are tracking for SK.
### Motivation and Context
It's better to avoid nested classes in Python, so in SKFunction, I move
RunThread out of the class. RunThread is needed because we need to wait
for the result of an async function that's being run in an existing
event loop, and subclassing Thread seems to be the best way to do that.
This also adds python-test.yml to the github workflows.
For python-test.yml:
- In this PR, the checks were run because python files were changed.
- In this test PR: https://github.com/mkarle/semantic-kernel/pull/2, the
python tests were skipped because files in the python directory were not
changed.
### Motivation and Context
We don't need to specify asyncio in dependency lists because it is
native to python.
### Description
- Remove asyncio from pyproject.toml and requirements.txt
### Motivation and Context
1. Why is this change required? prompts are not semantic functions with
input <-> other way round
2. What problem does it solve? clearer understanding
3. What scenario does it contribute to? documentation
### Description
- corrected wording
- added a table for Q&A
### Contribution Checklist
- [x] I didn't break anyone 😄
---------
Co-authored-by: Devis Lucato <[email protected]>
Co-authored-by: Devis Lucato <[email protected]>
### Motivation and Context
This PR makes creating a kernel instance more pythonic. The builder
pattern is removed and the kernel is constructed using Kernel(...).
Work originally started with #137
### Description
- Removed kernel_builder.py.
- Instance creation pattern `kernel = sk.create_kernel() -> kernel =
sk.Kernel()`
- For notebook 6, there is no need to create a separate kernel for the
GitHub files. Changed to use a separate collection.
Co-authored-by: Devis Lucato <[email protected]>
Richard Edgar
