microsoft/semantic-kernel
1
0
Fork 0
mirror of https://github.com/microsoft/semantic-kernel.git synced 2026-10-12 02:19:06 +00:00
Code Issues Packages Projects Releases Wiki Activity

Python: switch to uv (#8320)

Browse Source 
### Motivation and Context

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->
Switches from poetry to uv
much faster and easier lock management
easier maintenance of pyproject.toml

fixes: #6584
fixes: #8326

### Description

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄
This commit is contained in:
Eduard van Valkenburg
2024-08-26 13:34:29 +02:00
committed by GitHub
parent 163d512753
commit f90538b9c0
24 changed files with 5836 additions and 8379 deletions
Download Patch File Download Diff File Expand all files Collapse all files
python/DEV_SETUP.md
+115 -157
View File
@@ -1,9 +1,88 @@
# Dev Setup
This document describes how to setup your environment with Python and Poetry,
This document describes how to setup your environment with Python and uv,
if you're working on new features or a bug fix for Semantic Kernel, or simply
want to run the tests included.
## System setup
## If you're on WSL
Check that you've cloned the repository to `~/workspace` or a similar folder.
Avoid `/mnt/c/` and prefer using your WSL user's home directory.
Ensure you have the WSL extension for VSCode installed.
## Using uv
uv allows us to use SK from the local files, without worrying about paths, as
if you had SK pip package installed.
To install SK and all the required tools in your system, first, navigate to the directory containing
this DEV_SETUP using your chosen shell.
### For windows (non-WSL)
Check the [uv documentation](https://docs.astral.sh/uv/getting-started/installation/) for the installation instructions. At the time of writing this is the command to install uv:
```powershell
powershell -c "irm https://astral.sh/uv/install.ps1 | iex"
```
You can then run the following commands manually:
```powershell
# Install Python 3.10, 3.11, and 3.12
uv python install 3.10 3.11 3.12
# Create a virtual environment with Python 3.10 (you can change this to 3.11 or 3.12)
$PYTHON_VERSION = "3.10"
uv venv --python $PYTHON_VERSION
# Install SK and all dependencies
uv sync --all-extras --dev
# Install pre-commit hooks
uv run pre-commit install -c python/.pre-commit-config.yaml
```
Or you can then either install [`make`](https://gnuwin32.sourceforge.net/packages/make.htm) and then follow the guide for Mac and Linux, or run the following commands, the commands are shown as bash but should work in powershell as well.
### For Mac and Linux (both native and WSL)
It is super simple to get started, run the following commands:
```bash
make install
```
This will install uv, python, Semantic Kernel and all dependencies and the pre-commit config. It uses python 3.10 by default, if you want to change that set the `PYTHON_VERSION` environment variable to the desired version (currently supported are 3.10, 3.11, 3.12). For instance for 3.12"
```bash
make install PYTHON_VERSION=3.12
```
If you want to change python version (without installing uv, python and pre-commit), you can use the same parameter, but do:
```bash
make install-sk PYTHON_VERSION=3.12
```
ℹ️ **Note**: Running the install or install-sk command will wipe away your existing virtual environment and create a new one.
Alternatively you can run the VSCode task `Python: Install` to run the same command.
## VSCode Setup
Open the workspace in [VSCode](https://code.visualstudio.com/docs/editor/workspaces).
> The workspace for python should be rooted in the `./python` folder.
Open any of the `.py` files in the project and run the `Python: Select Interpreter`
command from the command palette. Make sure the virtual env (default path is `.venv`) created by
`uv` is selected.
If prompted, install `ruff`. (It should have been installed as part of `uv sync --dev`).
You also need to install the `ruff` extension in VSCode so that auto-formatting uses the `ruff` formatter on save.
Read more about the extension [here](https://github.com/astral-sh/ruff-vscode).
## LLM setup
Make sure you have an
@@ -12,153 +91,36 @@ Make sure you have an
There are two methods to manage keys, secrets, and endpoints:
1. Store them in environment variables. SK Python leverages pydantic settings to load keys, secrets, and endpoints. This means that there is a first attempt to load them from environment variables. The `.env` file naming applies to how the names should be stored as environment variables.
1. Store them in environment variables. SK Python leverages pydantic settings to load keys, secrets, and endpoints from the environment.
> When you are using VSCode and have the python extension setup, it automatically loads environment variables from a `.env` file, so you don't have to manually set them in the terminal.
> During runtime on different platforms, environment settings set as part of the deployments should be used.
2. If you'd like to use the `.env` file, you will need to configure the `.env` file with the following keys into a `.env` file (see the `.env.example` file):
2. Store them in a separate `.env` file, like `dev.env`, you can then pass that name into the constructor for most services, to the `env_file_path` parameter, see below.
> Do not store `*.env` files in your repository, and make sure to add them to your `.gitignore` file.
```
There are a lot of settings, for a more extensive list of settings, see [ALL_SETTINGS.md](./samples/concepts/setup/ALL_SETTINGS.md).
### Example for file-based setup with OpenAI Chat Completions
To configure a `.env` file with just the keys needed for OpenAI Chat Completions, you can create a `openai.env` (this name is just as an example, a single `.env` with all required keys is more common) file in the root of the `python` folder with the following content:
Content of `openai.env`:
```env
OPENAI_API_KEY=""
OPENAI_ORG_ID=""
AZURE_OPENAI_CHAT_DEPLOYMENT_NAME=""
AZURE_OPENAI_TEXT_DEPLOYMENT_NAME=""
AZURE_OPENAI_EMBEDDING_DEPLOYMENT_NAME=""
AZURE_OPENAI_ENDPOINT=""
AZURE_OPENAI_API_KEY=""
OPENAI_CHAT_MODEL_ID="gpt-4o-mini"
```
You will then configure the Text/ChatCompletion class with the keyword argument `env_file_path`:
You will then configure the ChatCompletion class with the keyword argument `env_file_path`:
```python
chat_completion = OpenAIChatCompletion(service_id="test", env_file_path=<path_to_file>)
chat_completion = OpenAIChatCompletion(service_id="test", env_file_path="openai.env")
```
This optional `env_file_path` parameter will allow pydantic settings to use the `.env` file as a fallback to read the settings.
If using the second method, we suggest adding a copy of the `.env` file under these folders:
- [./tests](./tests)
- [./samples/getting_started](./samples/getting_started).
## System setup
To get started, you'll need VSCode and a local installation of Python 3.8+.
You can run:
```python
python3 --version ; pip3 --version ; code -v
```
to verify that you have the required dependencies.
## If you're on WSL
Check that you've cloned the repository to `~/workspace` or a similar folder.
Avoid `/mnt/c/` and prefer using your WSL user's home directory.
Ensure you have the WSL extension for VSCode installed (and the Python extension
for VSCode installed).
You'll also need `pip3` installed. If you don't yet have a `python3` install in WSL,
you can run:
```bash
sudo apt-get update && sudo apt-get install python3 python3-pip
```
ℹ️ **Note**: if you don't have your PATH setup to find executables installed by `pip3`,
you may need to run `~/.local/bin/poetry install` and `~/.local/bin/poetry shell`
instead. You can fix this by adding `export PATH="$HOME/.local/bin:$PATH"` to
your `~/.bashrc` and closing/re-opening the terminal.\_
## Using Poetry
Poetry allows to use SK from the local files, without worrying about paths, as
if you had SK pip package installed.
To install Poetry in your system, first, navigate to the directory containing
this README using your chosen shell. You will need to have Python 3.10, 3.11, or 3.12
installed.
Install the Poetry package manager and create a project virtual environment.
Note: SK requires at least Poetry 1.2.0.
### Note for MacOS Users
It is best to install Poetry using their
[official installer](https://python-poetry.org/docs/#installing-with-the-official-installer).
On MacOS, you might find that `python` commands are not recognized by default,
and you can only use `python3`. To make it easier to run `python ...` commands
(which Poetry requires), you can create an alias in your shell configuration file.
Follow these steps:
1. **Open your shell configuration file**:
- For **Bash**: `nano ~/.bash_profile` or `nano ~/.bashrc`
- For **Zsh** (default on macOS Catalina and later): `nano ~/.zshrc`
2. **Add the alias**:
```sh
alias python='python3'
```
3. **Save the file and exit**:
- In `nano`, press `CTRL + X`, then `Y`, and hit `Enter`.
4. **Apply the changes**:
- For **Bash**: `source ~/.bash_profile` or `source ~/.bashrc`
- For **Zsh**: `source ~/.zshrc`
After these steps, you should be able to use `python` in your terminal to run
Python 3 commands.
### Poetry Installation
```bash
# Install poetry package if not choosing to install via their official installer
pip3 install poetry
# optionally, define which python version you want to use
poetry env use python3.11
# Use poetry to install base project dependencies
poetry install
# If you want to get all dependencies for tests installed, use
# poetry install --with tests
# example: poetry install --with hugging_face
# Use poetry to activate project venv
poetry shell
# Optionally, you can install the pre-commit hooks
poetry run pre-commit install
# this will run linters and mypy checks on all the changed code.
```
## VSCode Setup
Open the [workspace](https://code.visualstudio.com/docs/editor/workspaces) in VSCode.
> The Python workspace is the `./python` folder if you are at the root of the repository.
Open any of the `.py` files in the project and run the `Python: Select Interpreter`
command from the command palette. Make sure the virtual env (venv) created by
`poetry` is selected.
The python you're looking for should be under `~/.cache/pypoetry/virtualenvs/semantic-kernel-.../bin/python`.
If prompted, install `ruff`. (It should have been installed as part of `poetry install`).
You also need to install the `ruff` extension in VSCode so that auto-formatting uses the `ruff` formatter on save.
Read more about the extension here: https://github.com/astral-sh/ruff-vscode
## Tests
You can run the unit tests under the [tests/unit](tests/unit/) folder.
```bash
poetry install --with unit-tests
poetry run pytest tests/unit
uv run pytest tests/unit
```
Alternatively, you can run them using VSCode Tasks. Open the command palette
@@ -167,21 +129,18 @@ Alternatively, you can run them using VSCode Tasks. Open the command palette
You can run the integration tests under the [tests/integration](tests/integration/) folder.
```bash
poetry install --with tests
poetry run pytest tests/integration
uv run pytest tests/integration
```
You can also run all the tests together under the [tests](tests/) folder.
```bash
poetry install
poetry run pytest tests
uv run pytest tests
```
Alternatively, you can run them using VSCode Tasks. Open the command palette
(`Ctrl+Shift+P`) and type `Tasks: Run Task`. Select `Python: Tests - All` from the list.
## Tools and scripts
## Implementation Decisions
@@ -203,9 +162,9 @@ They should contain:
- If necessary to further explain the logic a newline follows the first line and then the explanation is given.
- The following three sections are optional, and if used should be separated by a single empty line.
- Arguments are then specified after a header called `Args:`, with each argument being specified in the following format:
- `arg_name` (`arg_type`): Explanation of the argument, arg_type is optional, as long as you are consistent.
- `arg_name`: Explanation of the argument.
- if a longer explanation is needed for a argument, it should be placed on the next line, indented by 4 spaces.
- Default values do not have to be specified, they will be pulled from the definition.
- Type and default values do not have to be specified, they will be pulled from the definition.
- Returns are specified after a header called `Returns:` or `Yields:`, with the return type and explanation of the return value.
- Finally, a header for exceptions can be added, called `Raises:`, with each exception being specified in the following format:
- `ExceptionType`: Explanation of the exception.
@@ -227,12 +186,12 @@ def equal(arg1: str, arg2: str) -> bool:
Here is extra explanation of the logic involved.
Args:
arg1 (str): The first string to compare.
arg2 (str): The second string to compare.
arg1: The first string to compare.
arg2: The second string to compare.
This string requires extra explanation.
Returns:
bool: True if the strings are the same, False otherwise.
True if the strings are the same, False otherwise.
Raises:
ValueError: If one of the strings is empty.
@@ -244,11 +203,8 @@ If in doubt, use the link above to read much more considerations of what to do a
## Pydantic and Serialization
[Pydantic Documentation](https://docs.pydantic.dev/1.10/)
### Overview
This section describes how one can enable serialization for their class using Pydantic.
For more info you can refer to the [Pydantic Documentation](https://docs.pydantic.dev/latest/).
### Upgrading existing classes to use Pydantic
@@ -263,7 +219,7 @@ class A:
self.d = d
```
You would convert this to a Pydantic class by subclassing from the `KernelBaseModel` class.
You would convert this to a Pydantic class by sub-classing from the `KernelBaseModel` class.
```python
from pydantic import Field
@@ -298,10 +254,13 @@ class A:
You can use the `KernelBaseModel` to convert these to pydantic serializable classes.
```python
from typing import Generic
from typing import Generic, TypeVar
from semantic_kernel.kernel_pydantic import KernelBaseModel
T1 = TypeVar("T1")
T2 = TypeVar("T2", bound=<some class>)
class A(KernelBaseModel, Generic[T1, T2]):
# T1 and T2 must be specified in the Generic argument otherwise, pydantic will
# NOT be able to serialize this class
@@ -310,32 +269,31 @@ class A(KernelBaseModel, Generic[T1, T2]):
c: T2
```
## Pipeline checks
## Code quality checks
To run the same checks that run during the GitHub Action build, you can use
this command, from the [python](../python) folder:
To run the same checks that run during a commit and the GitHub Action `Python Code Quality Checks`, you can use this command, from the [python](../python) folder:
```bash
poetry run pre-commit run -a
uv run pre-commit run -a
```
or use the following task (using `Ctrl+Shift+P`):
- `Python - Run Checks` to run the checks on the whole project.
- `Python - Run Checks - Staged` to run the checks on the currently staged files only.
Ideally you should run these checks before committing any changes, use `poetry run pre-commit install` to set that up.
Ideally you should run these checks before committing any changes, when you install using the instructions above the pre-commit hooks should be installed already.
## Code Coverage
We try to maintain a high code coverage for the project. To run the code coverage on the unit tests, you can use the following command:
```bash
poetry run pytest --cov=semantic_kernel --cov-report=term-missing:skip-covered tests/unit/
uv run pytest --cov=semantic_kernel --cov-report=term-missing:skip-covered tests/unit/
```
or use the following task (using `Ctrl+Shift+P`):
- `Python: Tests - Code Coverage` to run the code coverage on the whole project.
This will show you which files are not covered by the tests, including the specific lines not covered.
This will show you which files are not covered by the tests, including the specific lines not covered. Make sure to consider the untested lines from the code you are working on, but feel free to add other tests as well, that is always welcome!
## Catching up with the latest changes
There are many people committing to Semantic Kernel, so it is important to keep your local repository up to date. To do this, you can run the following commands: