microsoft/semantic-kernel
1
0
Fork 0
mirror of https://github.com/microsoft/semantic-kernel.git synced 2026-10-12 11:19:56 +00:00
Code Issues Packages Projects Releases Wiki Activity
5,101 Commits 64 Branches 1,077 Tags
main
Commit Graph
82 Commits
Author SHA1 Message Date
Giulio Leonegiulio-leoneEvan MattsonCopilot
 		f034ed1d4c Python: fix: ChatHistoryTruncationReducer orphans TOOL role messages (#13608) 
## Summary

Fixes #12708

`ChatHistoryTruncationReducer.reduce()` can orphan `TOOL` role messages
by truncating the preceding `assistant` message that contains the
`tool_calls`, causing OpenAI to reject the history with:

```
messages with role 'tool' must be a response to a preceding message with 'tool_calls'
```

## Root Cause

`locate_safe_reduction_index()` uses
`contains_function_call_or_result()` to detect tool-related messages
during its backward scan. This function only checks `msg.items` for
`FunctionCallContent`/`FunctionResultContent` instances.

However, `TOOL` role messages can contain only text content (no
`FunctionResultContent` in `items`), which causes the backward scan to
treat them as regular messages. The truncation point then lands between
the `tool_calls` assistant message and its `TOOL` responses, orphaning
the tool results.

## Fix

Added `AuthorRole.TOOL` check to `contains_function_call_or_result()`:

```python
if msg.role == AuthorRole.TOOL:
    return True
```

This ensures **any** `TOOL` role message is recognized as part of a tool
call/result pair, regardless of whether it has `FunctionResultContent`
in its `items`.

## Test

Added
`test_locate_safe_reduction_index_tool_role_without_function_result_content`
that verifies TOOL role messages with only text content are not
separated from their tool call.

---------

Co-authored-by: giulio-leone <[email protected]>
Co-authored-by: Evan Mattson <[email protected]>
Co-authored-by: Copilot <[email protected]>
 2026-03-24 02:00:04 +00:00
Roli BoschClaude Opus 4.6Evan Mattson
 		3c1b688dcf Python: Fix ChatHistoryTruncationReducer deleting system prompt (#13610) 
## Summary

`ChatHistoryTruncationReducer.reduce()` silently deletes
system/developer messages when truncating chat history. This is because
it calls `extract_range()`, which unconditionally filters out
system/developer messages — a function designed for the summarization
use case, not truncation.

Fixes #12612.

## Approach

Port the .NET SDK fix (PR #10344) to Python:

1. Detect the first system/developer message before truncation
2. Pass `has_system_message=True` to `locate_safe_reduction_index` so
`target_count` accounts for the preserved message (matches .NET's
`targetCount -= hasSystemMessage ? 1 : 0`)
3. Use a simple `history[truncation_index:]` slice instead of
`extract_range` (which strips system messages)
4. Prepend the system message if it was truncated away

Also adds a guard for `target_count <= 0` after the system message
adjustment to prevent `IndexError` when `target_count=1`.

## Changes

| File | Change |
|------|--------|
| `chat_history_reducer_utils.py` | Add `has_system_message` parameter,
adjust `target_count`, guard against `<= 0` |
| `chat_history_truncation_reducer.py` | Detect system message, bypass
`extract_range`, prepend if truncated |
| `test_chat_history_truncation_reducer.py` | Update 2 existing tests +
4 new tests (system, developer, no-system, target_count=1) |

## Note

The summarization reducer (`ChatHistorySummarizationReducer`) has the
same bug — it also uses `extract_range` and loses system messages during
summarization. The .NET summarization reducer preserves system messages
via `AssemblySummarizedHistory`. This should be addressed in a follow-up
PR to keep this change focused.

## Test plan

- [x] 11/11 truncation reducer tests pass (was 7, now 11)
- [x] 9/9 utils tests pass (no regressions from new parameter)
- [x] 11/11 summarization reducer tests pass (unaffected)
- [x] Reproduces and fixes the exact scenario from #12612
- [x] Edge case: `target_count=1` with system message does not crash

---------

Co-authored-by: Claude Opus 4.6 <[email protected]>
Co-authored-by: Evan Mattson <[email protected]>
 2026-03-19 12:28:52 +00:00
Weiguang LiandEvan Mattson e500f727ed Python: fix prompt template engine blocking HTML tags in content (#13633) 
### Motivation and Context

Fixes #13632

When a prompt contains HTML tags such as `<p>`, `<div>`, `<b>`, etc.,
the
`ChatHistory.from_rendered_prompt()` method wraps the rendered prompt in
a
`<root>` element and parses it as XML. Any child elements whose tag
names do
not match the known SK template tags (`message`, `chat_history`) were
silently
skipped, causing their text content to be lost. This meant a prompt
like:

```
Translate: "<p>What is your name?</p>"
```

would arrive at the LLM as an empty message, because `<p>What is your
name?</p>`
was discarded during XML traversal.

### Description

In `ChatHistory.from_rendered_prompt`, when an XML child element has an
unrecognized tag (i.e. it is not `message` or `chat_history`), the
element
is now serialized back to its original text representation and appended
to
the preceding message content. The element's tail text is likewise kept
in
the same message, preserving the full original prompt text.

**Changed files:**

- `python/semantic_kernel/contents/chat_history.py` -- handle unknown
XML
  tags as literal text in `from_rendered_prompt()`
- `python/tests/unit/contents/test_chat_history.py` -- added four
regression
tests covering HTML `<p>`, multiple HTML tags, HTML surrounded by text,
  and SK tags mixed with HTML

### Contribution Checklist

- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
- [x] The existing tests pass and new tests have been added
- [x] I didn't break any existing functionality

---------

Co-authored-by: Evan Mattson <[email protected]>
 2026-03-18 11:01:25 +00:00
Tao Chen d5c2b9d889 Python: Improves the robustness of filename handling (#13643) 
### Motivation and Context

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->
Improves the robustness of filename handling in the Bedrock agent
integration. Ensures that filenames sourced from external responses are
properly sanitized before use in file operations, and adds safer
defaults for file write behavior.

### Description

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
1. Sanitize filenames received from Bedrock agent responses.
2. Add overwrite parameter to BinaryContent.write_to_file (defaults to
False).
3. Add tests.
4. Update samples.

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄
 2026-03-12 17:02:02 +00:00
Christian GlessnerEvan MattsonEvan Mattson
 		5e50e190c4 Python: Add reasoning support for OpenAI Responses Agents (GPT-5, o4-mini, o3) (#12881) 
## Summary

Adds reasoning capabilities to Python OpenAI ResponsesAgent, bringing
parity with the C# implementation. This enables fine-grained control
over reasoning effort for O-series models (o1, o3-mini, o4-mini) and
gpt-5 that support the reasoning parameter.

Fixes #12843

## Changes

### Core Implementation
- **`openai_responses_agent.py`**: Added reasoning configuration support
  - Constructor-level reasoning effort setting
  - Per-invocation reasoning effort override capability
  - Proper parameter validation and model compatibility checks
- **`responses_agent_thread_actions.py`**: Extended thread actions to
support reasoning parameters
  - Reasoning effort propagation through thread operations
  - Metadata preservation for reasoning tokens and summaries

### Sample and Tests
- **`responses_agent_reasoning.py`**: Comprehensive demonstration sample
  - Constructor vs per-invocation reasoning configuration
  - Function calling integration with reasoning
  - Reasoning comparison scenarios (low/medium/high effort)
  - Error handling and troubleshooting guidance
- **`test_openai_responses_agent_reasoning.py`**: Full unit test
coverage
  - Parameter validation tests
  - Integration scenarios with function calling
  - Edge cases and error conditions

## Features

✅ **Constructor-Level Reasoning**: Set default reasoning effort when
creating agents
✅ **Per-Invocation Override**: Override reasoning effort per request  
✅ **Priority Hierarchy**: per-invocation > constructor > model default  
✅ **Function Calling Compatible**: Works seamlessly with existing plugin
system
✅ **Azure OpenAI & OpenAI Support**: Compatible with both service
providers
✅ **Model Validation**: Automatic compatibility checks for O-series
models nad GÜT-5
✅ **Metadata Access**: Reasoning tokens and summaries available in
response metadata

## Usage Example

```python
# Constructor-level reasoning
# Constructor-level reasoning configuration
agent = AzureResponsesAgent(
    ai_model_id="gpt-5",
    client=client,
    reasoning={"effort": "low"}  # Default reasoning for all requests
)

# Per-invocation override
response = await agent.invoke(
    "Solve this complex problem step by step",
    reasoning={"effort": "high"}  
)


# Invoke with reasoning callback to capture intermediate thoughts
response = await agent.invoke(
    "Analyze this data step by step",
    reasoning={"effort": "high", "summary": "detailed"},
    on_intermediate_message=handle_reasoning_message
)

# Streaming with reasoning
async for response in agent.invoke_stream(
    "Explain quantum computing in detail",
    reasoning={"effort": "high", "summary": "detailed"},
    on_intermediate_message=handle_reasoning_message
):
    print(response.content, end="", flush=True)

---------

Co-authored-by: Evan Mattson <[email protected]>
Co-authored-by: Evan Mattson <[email protected]>
 2025-08-28 23:25:26 +00:00
ふぁーandEvan Mattson d3318adfc2 Python: Add container_id and filename fields to AnnotationContent class (#12985) 
### Motivation and Context

I confirmed that Code Interpreter can run via Azure Responses API in
Semantic Kernel, but I found two issues: an error occurs when a file is
created inside the Code Interpreter, and the response lacks parameters
required to access the created file.

### Description

#### Bug 1: ValidationError for AnnotationContent

Semantic Kernel uses the following enum for
`AnnotationContent.content_type`:

```python
@experimental
class CitationType(str, Enum):
    """Citation type."""

    URL_CITATION = "url_citation"
    FILE_PATH = "file_path"
    FILE_CITATION = "file_citation"
```

However, the Responses API returns file citations with a `type` like
this:

```json
{"container_id": "cntr_68a6c50135d88190b6fdc062051155b50847ddec73d3be1b", "end_index": 94, "file_id": "cfile_68a6c67a7fcc8190889d7e1789d485e2", "filename": "sample.txt", "start_index": 66, "type": "container_file_citation"}
```

There is no matching enum member on the Semantic Kernel side. The
`CitationType` should include `CONTAINER_FILE_CITATION`, for example:

```python
@experimental
class CitationType(str, Enum):
    """Citation type."""

    URL_CITATION = "url_citation"
    FILE_PATH = "file_path"
    FILE_CITATION = "file_citation"
    CONTAINER_FILE_CITATION = "container_file_citation"
```

#### Bug 2: Missing parameters to access generated files

The response returned by the agent shows a `file_id`, but lacks the
`container_id` and `filename` required to download the file (the
filename is needed to replace `sandbox:/mnt/data/sample.txt`):

Example call:

```python
r = await agent.get_response(cast(list, history.messages), thread=thread)
print(r.content.items)
```

Example output:

```python
[
  TextContent(inner_content=None, ai_model_id=None, metadata={}, content_type='text', text='<user_prompt>', encoding=None),
  TextContent(inner_content=None, ai_model_id=None, metadata={}, content_type='text', text='<assistant_message>\n\n[sample.txt](sandbox:/mnt/data/sample.txt)', encoding=None), 
  AnnotationContent(inner_content=None, ai_model_id=None, metadata={}, content_type='annotation', file_id='cfile_68a6c67a7fcc8190889d7e1789d485e2', quote=None, start_index=88, end_index=116, url=None, title=None, citation_type=<CitationType.CONTAINER_FILE_CITATION: 'container_file_citation'>)
]
```

To include these values in the parsed result, `AnnotationContent` in
`semantic_kernel/contents/annotation_content.py` should expose
`container_id` and `filename`, and map the incoming `type` to
`citation_type`. Example suggested structure:

```python
@experimental
class AnnotationContent(KernelContent):
    """Annotation content."""

    content_type: Literal[ContentTypes.ANNOTATION_CONTENT] = Field(ANNOTATION_CONTENT_TAG, init=False)  # type: ignore
    tag: ClassVar[str] = ANNOTATION_CONTENT_TAG
    file_id: str | None = None
    quote: str | None = None
    start_index: int | None = None
    end_index: int | None = None
    url: str | None = None
    title: str | None = None
    # added
    container_id: str | None = None
    filename: str | None = None
    citation_type: CitationType | None = Field(None, alias="type")

    model_config = ConfigDict(
        extra="ignore",
        populate_by_name=True,
    )

    def __str__(self) -> str:
        ...

    def to_element(self) -> Element:
        ...

    @classmethod
    def from_element(cls: type[_T], element: Element) -> _T:
        ...

    def to_dict(self) -> dict[str, Any]:
        ...
```

Summary: add the missing enum value `container_file_citation` and extend
`AnnotationContent` to include `container_id` and `filename` (and map
`type` to `citation_type`) so file references returned by the Azure
Responses API are fully usable.

#### Code

```python
client: AsyncAzureOpenAI = AsyncAzureOpenAI(
    base_url=urljoin(server_settings.AZURE_OPENAI_COMPLETION_ENDPOINT.rstrip("/") + "/", "openai/v1/"),
    api_key=server_settings.AZURE_OPENAI_COMPLETION_API_KEY,
    api_version="preview",
)

async def run(history: ChatHistory) -> dict[str, Any]:
    with open("sample.txt", "rb") as fp:
        f = await client.files.create(file=fp, purpose="assistants")

    agent = AzureResponsesAgent(
        ai_model_id=server_settings.AZURE_OPENAI_COMPLETION_DEPLOYMENT_NAME,
        client=client,
        name="name",
        function_choice_behavior=FunctionChoiceBehavior.Required(),
        tools=[
            CodeInterpreter(
                type="code_interpreter",
                container=CodeInterpreterContainerCodeInterpreterToolAuto(type="auto", file_ids=[f.id]),
            )
        ],
    )

    thread = ResponsesAgentThread(client, history, previous_response_id=None, enable_store=True)
    r = await agent.get_response(cast(list, history.messages), thread=thread)
    print(thread.id, r.content.items)

    annotation_content = [item for item in r.content.items if isinstance(item, AnnotationContent)]

    for ac in annotation_content:
        print(ac.file_id, ac.filename, ac.container_id)
```

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄

Co-authored-by: Evan Mattson <[email protected]>
 2025-08-28 00:01:39 +00:00
AllaouaAllaoua BenchikhEvan MattsonEvan Mattson
 		45fe60545c Python: Fix issue with orphaned tool/tool_calls (#12923) 
### Motivation and Context

Tool/tool_calls are getting orphaned.

Issue : https://github.com/microsoft/semantic-kernel/issues/12708

How to reproduce :

```python
from semantic_kernel.contents import (
    AuthorRole,
    ChatMessageContent,
    FunctionCallContent,
    FunctionResultContent,
    ChatHistoryTruncationReducer,
)

messages = [
    ChatMessageContent(role=AuthorRole.SYSTEM, content="sys"),
    ChatMessageContent(role=AuthorRole.USER, content="user A"),
    ChatMessageContent(
        role=AuthorRole.ASSISTANT,
        items=[
            FunctionCallContent(
                id="123", function_name="search", plugin_name="plugin", arguments={"q": "x"}
            )
        ],
    ),
    ChatMessageContent(
        role=AuthorRole.TOOL,
        items=[
            FunctionResultContent(
                id="123", function_name="search", plugin_name="plugin", result="RESULT"
            )
        ],
    ),
    ChatMessageContent(role=AuthorRole.USER, content="user B"),
    ChatMessageContent(role=AuthorRole.ASSISTANT, content="done"),
]

reducer = ChatHistoryTruncationReducer(target_count=3, threshold_count=0)
reducer.messages = messages

print("Before reduce:", [m.role for m in reducer.messages])
await reducer.reduce()
print("After reduce:", [m.role for m in reducer.messages])

# After reduce: [<AuthorRole.TOOL: 'tool'>, <AuthorRole.USER: 'user'>, <AuthorRole.ASSISTANT: 'assistant'>]
```

### Description

Move backward as long as we are in a tool/tool_call island when reducing
the messages.

---------

Co-authored-by: Allaoua Benchikh <[email protected]>
Co-authored-by: Evan Mattson <[email protected]>
Co-authored-by: Evan Mattson <[email protected]>
 2025-08-14 00:23:04 +00:00
Evan Mattson 5332c2e330 Python: Require tool_call_id parameter for string-based tool messages in ChatHistory (#12753) 
### Motivation and Context

The `add_tool_message` method of `ChatHistory` accepts string content
without requiring a `tool_call_id` parameter. This design is flawed from
an API design perspective. It created tool messages that:

1. Violated chat completion protocols: most LLM APIs (OpenAI, etc.)
require tool messages to have a `tool_call_id` that references a
previous function call
2. Broke conversation flow: tool messages should always be responses to
specific tool calls, maintaining the call-response relationship
3. Created malformed message sequences: messages without proper tool
call IDs would fail when sent to language models

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->

### Description

This PR modifies the `add_tool_message` string overload to require a
`tool_call_id` parameter, which makes sure we have proper tool call
protocol compliance.

1. The string overload now validates that `tool_call_id` is provided and
raises a clear error if missing
2. Tool messages now create `FunctionResultContent` objects with both
`id` and `call_id` set to the provided `tool_call_id`
3. Added support for an optional `function_name` parameter for better
bookkeeping
4. Provides informative error messages explaining why `tool_call_id` is
required
- Closes #12744 

Before:

```python
# This would create an invalid tool message without tool_call_id
# when the `.to_dict()` method is called on `ChatMessageContent`
chat_history.add_tool_message("Function result")  # Creates broken message and losing pairing to the corresponding `FunctionCallContent`.
```

After: 

```python
# Now requires tool_call_id for proper protocol compliance
chat_history.add_tool_message("Function result")  # raises clear error due to missing `tool_call_id`
chat_history.add_tool_message("Function result", tool_call_id="call_123")  # works correctly

# Optional function name for better bookkeeping
chat_history.add_tool_message("Function result", tool_call_id="call_123", function_name="get_weather") # works as well
```

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [X] The code builds clean without any errors or warnings
- [X] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [X] All unit tests pass, and I have added new tests where possible
- [X] I didn't break anyone 😄
 2025-07-23 04:07:20 +00:00
Christian GlessnerandEvan Mattson 0e7556e112 Python: Add file handling support to BinaryContent for OpenAI Responses API (#12258) 
## Summary

Enhances `BinaryContent` to support file handling for OpenAI Responses
API, enabling file uploads through the responses agent while maintaining
a provider-agnostic design.

  ## Changes

  ### BinaryContent Enhancements
- **Add `can_read` property**: Indicates whether content has readable
data available
- **Add `from_file()` class method**: Creates BinaryContent instances
from file paths with automatic base64 encoding
- **Fix Unicode handling**: Prevents decode errors when processing
binary files (PDFs, images, etc.)

  ### OpenAI Responses Agent Integration
- **Add BinaryContent support**: Pattern matching case for file handling
in `responses_agent_thread_actions.py`
- **Correct OpenAI API format**: Uses proper `filename` and `file_data`
structure with data URI format
- **UUID-based filenames**: Generates appropriate filenames with
mime-type extensions
- **Provider-specific mapping**: File format conversion happens only in
OpenAI agent code

  ### Testing
- **Test coverage**: New tests for `can_read`, `from_file()`, and binary
data handling
- **Unicode error prevention**: Specific test for binary PDF-like
content
- **Base64 encoding verification**: Ensures proper data format for API
compatibility

  ## Design Principles

- **Provider-agnostic**: BinaryContent remains completely generic with
no OpenAI-specific dependencies
- **Clean separation**: OpenAI format mapping isolated to OpenAI agent
files
- **No FileContent class**: Enhances existing BinaryContent instead of
introducing new types
- **Follows existing patterns**: Similar approach to ImageContent and
TextContent handling

  ## Usage Example

  ```python
 response = await self.agent.get_response(
        messages=ChatMessageContent(
            content="Analyse PDF",
            role=AuthorRole("user"),
            items=[BinaryContent.from_file(file_path=c":/test.pdf")],
        )
    )

Co-authored-by: Evan Mattson <[email protected]>
 2025-06-09 21:15:39 +00:00
Evan Mattson fb0e418a7c Python: Fix for AzureAIAgent streaming func call invocations. Other fixes/improvements for tool call content types, and logging. (#12335) 
### Motivation and Context

The AzureAIAgent streaming function call invocation is broken when
handling more than one required function call from the service. There is
nested logic to handle passing the tool calls back to the service
because it requires one to pass in a handler.

In this PR, we're now controlling everything from the main loop and are
passing back the handler as the event stream for further processing.
This does require some logic to determine if we have an initial async
context manager (based on the first call to create the run stream) or if
we're now using the handler from submitting tool calls. We can now see
that when we have multiple functions involved, we are able to get the
results as we expect:

```
Sample Output:

# User: 'What is the price of the special drink and then special food item added together?'
Function Call:> MenuPlugin-get_specials with arguments: {}
Function Result:> 
        Special Soup: Clam Chowder
        Special Salad: Cobb Salad
        Special Drink: Chai Tea
        for function: MenuPlugin-get_specials
Function Call:> MenuPlugin-get_item_price with arguments: {"menu_item": "Chai Tea"}
Function Call:> MenuPlugin-get_item_price with arguments: {"menu_item": "Clam Chowder"}
Function Result:> $9.99 for function: MenuPlugin-get_item_price
Function Result:> $9.99 for function: MenuPlugin-get_item_price
Function Call:> MathPlugin-Add with arguments: {"input":9.99,"amount":9.99}
Function Result:> 19.98 for function: MathPlugin-Add
# AuthorRole.ASSISTANT: The price of the special drink, Chai Tea, is $9.99 and the price of the special food 
    item, Clam Chowder, is $9.99. Added together, the total price is $19.98.
```

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->

### Description

Fix handling multiple function calls for the streaming AzureAIAgent.
- Fixes a missing `quote` from the url citation for streaming annotation
content
- Shows the correct way to retrieve Bing Grounding tool calls in the two
concept samples.
- Adds handling for tool content content for OpenAPI plugins.
- Adds logging for streaming agent invocation tool calls.
- Relaxes the handling of the `FunctionResultContent` id attribute to be
`str | None` similar to that of `FunctionCallContent`. The type hints
for `id` have always been `str | None`, but the actual type hint on the
attribute has been `str`. We don't always get a `FunctionResultContent`
id from the Azure Agent Service during streaming invocations - message
addition via the overide `__add__` method still work properly.
- Move the AzureAIAgent pkg dependencies to be installed by default,
instead of requiring the `azure` extra.
- Closes #12312
- Closes #12328
- Closes #12331
- Closes #12324

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [X] The code builds clean without any errors or warnings
- [X] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [X] All unit tests pass, and I have added new tests where possible
- [X] I didn't break anyone 😄
 2025-06-03 01:11:56 +00:00
Evan Mattson b88f05287a Python: Preserve Citation Title in AnnotationContent from Azure AI Foundry Annotations (#12152) 
### Motivation and Context

When invoking an Azure AI Foundry agent via AzureAIAgent, url_citation
metadata such as the document filename (title) is omitted in the
resulting AnnotationContent. Only the url field is populated; the
filename (title) is lost, and no structured metadata is retained. This
breaks traceability and rich citation in downstream responses.

This PR updates the annotation mapping to include the title from
url_citation into the resulting AnnotationContent.

This PR also updates the content types to properly show the present
information either in the `to_dict()` call or `to_str` call, as well as
handling the attributes for the `from_element` class method.

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->

### Description

Citations in responses returned via AzureAIAgent now retain the full
provenance information (url, title, etc.) as delivered by the underlying
agent, matching the behavior observed with direct AIProjectClient calls.
- Closes #12133 
- Adds further unit tests to validate functionality

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [X] The code builds clean without any errors or warnings
- [X] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [X] All unit tests pass, and I have added new tests where possible
- [X] I didn't break anyone 😄
 2025-05-19 17:29:31 +00:00
kanchi 966fa8ca04 Python: Replace mutable default arguments and prevent unintended side effects (#11849) 
### Motivation and Context

Some functions and methods were using mutable objects (typically `dict`)
as default argument values.
In Python, default arguments are evaluated only once at the time of
function definition.
This can lead to unexpected behavior when the default value is modified
within the function, as subsequent calls to the function may reuse the
modified object.

As a result, calling such functions or methods multiple times can lead
to unintended behavior.

Reference:  
[8.7. Function definitions (Official Python
Documentation)](https://docs.python.org/3.12/reference/compound_stmts.html#function-definitions)
> Default parameter values are evaluated from left to right when the
function definition is executed. This means that the expression is
evaluated once, when the function is defined, and that the same
“pre-computed” value is used for each call. This is especially important
to understand when a default parameter value is a mutable object, such
as a list or a dictionary: if the function modifies the object (e.g. by
appending an item to a list), the default parameter value is in effect
modified. This is generally not what was intended. A way around this is
to use None as the default, and explicitly test for it in the body of
the function, e.g.:


### Description

- Replaced mutable types (e.g., `dict`, `list`) used as default argument
values with `None`, and properly initialized them within the function
body.
- Reviewed and updated the parts of the code that performed destructive
modifications on the arguments, to avoid unintentionally modifying
variables from the caller’s scope.
- ~Added the `flake8-bugbear` rule in Ruff to detect this kind of
issue.~
- ~For now, errors in `tests` and `samples` directories are ignored due
to the large number of violations.~

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [ ] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄
 2025-05-06 19:03:05 +00:00
Evan Mattson 329a35edea Python: Introduce the OpenAI Responses Agent (#11240) 
### Motivation and Context

OpenAI recently released their Responses API, which is their newest core
API and an agentic API primitive, combining the simplicity of Chat
Completions with the ability to do more agentic tasks. Azure OpenAI also
recently released the Responses API, with close-to feature parity with
OpenAI's SDK. It gives us enough to introduce an `AzureResponsesAgent`
and an `OpenAIResponsesAgent` to Semantic Kernel.

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->

### Description

This PR introduces:
- The `AzureResponsesAgent` and `OpenAIResponsesAgent`
- Allows use of the web search tool with `OpenAI` (`Azure OpenAI`
currently doesn't support this).
- Allows use of the file search tool with both `OpenAI` and `Azure
OpenAI`.
- Provides getting started samples for both agent types.

**TODO:**
- Add unit test coverage for the `ResponsesAgentThreadActions` class.
- Add support for the computer user agent tool.
- Improve typing in the `ResponsesAgentThreadActions` class.
- Closes #11026

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [X] The code builds clean without any errors or warnings
- [X] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [X] All unit tests pass, and I have added new tests where possible
- [X] I didn't break anyone 😄
 2025-04-02 23:18:57 +00:00
Evan Mattson 87bc3d6bc1 Python: Support AzureAIAgent agent_id instead of assistant_id (#10908) 
### Motivation and Context

The Azure Python SDK made an underlying change to move away from
`assistant_id` keyword arguments to now support `agent_id` keyword
arguments. Additionally, they now support a
`MessageTextUrlCitationAnnotation` and
`MessageDeltaTextUrlCitationAnnotation` for annotations.

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->

### Description

Update the `AzureAIAgent` to pass in `agent_id` in place of
`assistant_id`, and bring in support for the new annotation types.
- Updates samples to simpler patterns where needed.
- Updates unit tests.
- Closes #10852

Validated that our [AzureAIAgent + Azure AI Search
sample](https://github.com/microsoft/semantic-kernel/blob/main/python/samples/concepts/agents/azure_ai_agent/azure_ai_agent_azure_ai_search.py)
works properly:

---

```markdown
# User: 'Which hotels are available with full-sized kitchens in Nashville, TN?'

# Agent: In Nashville, TN, here are some hotel options that feature full-sized kitchens:

1. **Extended Stay Suites** - All suites include full-sized kitchens stocked with cookware, separate living and sleeping areas, and sofa beds. Some larger rooms have fireplaces and patios or balconies【3:0†source】.

2. **New Premium Rooms** - These newly furnished rooms are fully equipped with premium amenities, including a full kitchen, washer & dryer, and access to a fitness center. Units also include fireplaces and small outdoor balconies【3:1†source】.

3. **Conveniently Located Extended Stay** - This option offers a full kitchen along with a washer & dryer, and amenities such as free WiFi and a fitness center, located near downtown Nashville【3:3†source】.

4. **Inn with Functional Kitchens** - This inn provides living room suites and functional kitchens, situated conveniently near a lake and welcomes pets【3:4†source】.

These options provide a range of amenities for travelers looking for the convenience of a full-sized kitchen during their stay.

# User: 'Fun hotels with free WiFi.'

# Agent: Here are some fun hotels that offer free WiFi:

1. **Hotel with Rooftop Plaza** - Offers complimentary continental breakfast in the lobby and free Wi-Fi throughout the hotel. Its penthouse suites provide stunning views【7:0†source】.

2. **Lakeside Hotel** - Features newly redesigned rooms, an airport shuttle, and lakeside amenities including a resort-style pool. The hotel provides stylish new guestrooms with Internet TVs and free WiFi【7:3†source】.

3. **Downtown Hotel** - Located near the vibrant downtown area and the business district. This hotel offers free WiFi, local transportation, and a special treat of milk and cookies for guests【7:4†source】.

These hotels combine fun experiences with the convenience of free internet access for guests.
```

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [X] The code builds clean without any errors or warnings
- [X] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [X] All unit tests pass, and I have added new tests where possible
- [X] I didn't break anyone 😄
 2025-03-11 13:21:23 +00:00
Eduard van ValkenburgandEvan Mattson de54193a0f Python: Introducing Realtime Clients for OpenAI and Azure OpenAI (#10127) 
### Motivation and Context

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->
Implements the OpenAI Realtime API with Semantic Kernel

### Description

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->

Adds a RealtimeClientBase
Adds RealtimeEvents
Adds OpenAI and Azure Realtime clients

Closes: #10108

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄

---------

Co-authored-by: Evan Mattson <[email protected]>
 2025-03-04 07:44:10 +00:00
Eduard van ValkenburgandEvan Mattson 4eb44930e9 Python: improved chat history samples (#10737) 
### Motivation and Context

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->
Improve the serialization sample
Added a more complex sample for using ChatHistory with a VectorStore to
store in a backend.

Also improves the typing of the vectorstoremodel decorator

### Description

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄

---------

Co-authored-by: Evan Mattson <[email protected]>
 2025-03-04 06:58:32 +00:00
Evan Mattson 174bdabf38 Python: Introduce feature decorator to allow for experimental and release candidate decorator usage (#10691) 
### Motivation and Context

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->

This change is required to improve the flexibility and maintainability
of our feature annotation system. Previously, separate decorators (e.g.,
`experimental_function` and `experimental_class`) were used to mark
experimental features, resulting in code duplication and limiting our
ability to handle additional feature stages. As our SDK evolves, we need
a unified approach that can support multiple stages—such as
experimental, release candidate, and future states—while also allowing
version information for release candidate features to be centrally
managed.

### Description

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->

This PR refactors our feature decorators by introducing a unified
`stage` decorator that updates the docstring and attaches metadata for
both functions and classes. Two convenience decorators, `experimental`
and `release_candidate`, are built on top of `stage`:
- The `experimental` decorator marks features as experimental and sets
an `is_experimental` attribute.
- The `release_candidate` decorator supports multiple usage patterns
(with or without parentheses and with an optional version parameter) to
mark features as release candidate and sets an `is_release_candidate`
attribute.

This unified approach reduces duplication, simplifies the codebase, and
lays the groundwork for easily extending feature stages in the future.

This decorator supports the following usage patterns:
- `@experimental` (for both classes and functions)
- `@release_candidate`  (no parentheses)
- `@release_candidate()`  (empty parentheses)
- `@release_candidate("1.21.3-rc1")`  (positional version)
- `@release_candidate(version="1.21.3-rc1")`  (keyword version)

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [X] The code builds clean without any errors or warnings
- [X] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [X] All unit tests pass, and I have added new tests where possible
- [X] I didn't break anyone 😄
 2025-02-27 00:17:54 +00:00
Evan Mattson 5cad2864df Python: Fix Infinite Recursion in Hashing Logic by Tracking Visited Objects (#10598) 
### Motivation and Context

We encountered an issue where our hashing utility (`make_hashable(...)`)
could enter an infinite recursion loop whenever it encountered cyclical
references. This was particularly evident with Pydantic models that
reference each other (or themselves), or with dictionaries/lists that
recursively include a reference to an already-seen object. In such
scenarios, the function continued to expand and re-expand the same
objects indefinitely, triggering a RecursionError.

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->

### Description

We introduced a visited dictionary in `make_hashable`. Before diving
deeper into any object -- whether it’s a dict, a list, a set, a tuple,
or a Pydantic model -- we record its id() in visited. The very next time
we see that same object ID, we immediately return its already-computed
placeholder or final representation, short-circuiting the cycle.
- Fixes #10592

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [X] The code builds clean without any errors or warnings
- [X] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [X] All unit tests pass, and I have added new tests where possible
- [X] I didn't break anyone 😄
 2025-02-18 22:55:04 +00:00
Eduard van Valkenburg cbfd7e9330 Python: improved content inits, added ndarray support for binary content and small fixes to defaults (#10469) 
### Motivation and Context

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->
This PR adds support for ndarray's as the content carrier for all binary
content types (binary, image, audio) as that is more optimized for
larger content.

It also does some fixes to the initialization of those content types and
the underlying data uri type.

Also some fixes for unspecified default param in pydantic Field,
language servers do not recognize `Field("default value")` as having a
default, so changed those occurances to `Field(default="default value")`

### Description

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄
 2025-02-12 16:59:08 +00:00
Eduard van Valkenburg 7bdf55028c Python: experimental py3.13 support (#10276) 
### Motivation and Context

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->
Some small fixes in uv setup within actions
Added experimental unit tests runs for py3.13
First typing improvement from pyright and pyright config.

### Description

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄
 2025-02-12 13:05:39 +00:00
Evan Mattson c3ab713322 Python: Add support for Azure AI Agent Service (#10414) 
### Motivation and Context

Azure AI Agents are a managed offering, and are built on top of the
OpenAI assistant v2 APIs. Until now we have not had support for them in
SK Python. This PR adds support for Azure AI Agents in SK Python. The
current abstractions allow one to run managed Azure AI Agents using SK
constructs as is demonstrated via the getting started samples and/or the
`azure_ai_agent` concept code samples. The developer must first follow
the required Azure AI Agent deployment steps, and bring their connection
string as well as their model deployment name.

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->

### Description

Add support for Azure AI Agent Service in SK
- Add sample code
- Add unit tests
- Restructures the `getting_started_with_agents` samples to move the
respective agent code into folders like `chat_completion`,
`azure_ai_agent`, etc. Updates the README as well.
- Closes #10187 

**TODO**: add more test coverage either in this PR, or a subsequent one.

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [X] The code builds clean without any errors or warnings
- [X] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [X] All unit tests pass, and I have added new tests where possible
- [X] I didn't break anyone 😄
 2025-02-07 02:11:10 +00:00
ふぁーTao ChenEvan Mattson
 		4d2b2be2d0 Python: Bug: Type Definition Error ImageContent (#10395) 
### Motivation and Context  
1. This change is required because IDE type-checking throws an error
when manually passing values (e.g., Base64 strings) to ImageContent’s
constructor.
2. It fixes the missing argument definitions (data, data_uri,
data_format, mime_type), which caused the type error in IDEs like VS
Code.
3. It resolves the issue reported where users observed IDE-type errors
despite code functioning at runtime.
   
### Description  
• Added explicit type definitions (data, data_uri, data_format,
mime_type) in the ImageContent class constructor and fields.
• Updated the docstring and __init__ to reflect the new arguments.    
• Ensured the code maintains backward compatibility by preserving
existing methods like from_image_file.
• Verified that type-checking in major Python IDEs now passes without
warnings.

### Example

```
import asyncio
import base64

from semantic_kernel.connectors.ai.open_ai.services.azure_chat_completion import AzureChatCompletion,
from semantic_kernel.contents import AuthorRole, ChatHistory, ImageContent
from semantic_kernel.contents.chat_message_content import ChatMessageContent
from semantic_kernel.contents.text_content import TextContent


async def main():
    service = AzureChatCompletion(
        api_key="************************",
        deployment_name="gpt-4o-mini",
        endpoint="https://************************.openai.azure.com",
        api_version="2024-12-01-preview",
    )

    chat_history = ChatHistory()

    # example of base64 image
    # or `ImageContent.from_image_file`
    with open("weather_icon.png", "rb") as image_file:
        b64_string = base64.b64encode(image_file.read()).decode("utf-8")

    chat_history.add_message(
        ChatMessageContent(
            role=AuthorRole.USER,
            items=[
                ImageContent(
                    data_uri=f"data:image/png;base64,{b64_string}",
                    data_format="base64",
                    mime_type="image/png",
                ),
                TextContent(text="what is the weather"),
            ],
        )
    )
    response = await service.get_chat_message_contents(
        chat_history=chat_history,
        settings=service.get_prompt_execution_settings_class()(),
    )

    print(response[0].content)
    # >> The icon you provided typically represents sunny or clear weather.
    # >> If you need specific weather information for a location,
    # >> please provide the location or check a reliable weather service for the latest updates.


asyncio.run(main())
```   

### Contribution Checklist  
- [x] The code builds clean without any errors or warnings    
- [x] The PR follows the SK Contribution Guidelines and the
pre-submission formatting script raises no violations
- [x] All unit tests pass, and I have added new tests where possible    
- [x] I didn’t break anyone ☺

---------

Co-authored-by: Tao Chen <[email protected]>
Co-authored-by: Evan Mattson <[email protected]>
 2025-02-05 02:44:24 +00:00
Tao Chen 7789c5fe55 Python: Bedrock agent (#10307) 
### Motivation and Context

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->
Address https://github.com/microsoft/semantic-kernel/issues/10284

Integration of AWS Bedrock agent to SK Python

### Description
1. Implementation of Bedrock agent in SK (essentially an agent connector
to the Bedrock agent service).
2. A dedicated channel that works with Bedrock agents.
3. Samples on how to use Bedrock agents.

Unit and integration tests will be included in another PR to prevent
this PR from becoming too big.

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄
 2025-01-31 19:44:39 +00:00
Evan Mattson 44034ebd23 Python: Improve hashing of cmc and scmc items. Add tests. (#10332) 
### Motivation and Context

We have some content types that can't be hashed as-is. This PR makes
sure that if there are unhashable types, that we turn them into hashable
types. Add unit tests to exercise the scenarios.

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->

### Description

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [X] The code builds clean without any errors or warnings
- [X] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [X] All unit tests pass, and I have added new tests where possible
- [X] I didn't break anyone 😄
 2025-01-29 07:54:42 +00:00
Eduard van ValkenburgEvan MattsonEvan Mattson
 		ec9b980e30 Python: improve agent samples and chat history handling (#10301) 
### Motivation and Context

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->
When passing the same object to `agent.reduce_history` as is present in
the `history_reducer` attribute of the agent that the function doesn't
accurately behave. This fixes that.

Also updates the sample to be a bit more concise.

Also fixes the way the single_dispatch is setup in ChatHistory.

Also ensures system/developer messages are not reduced away as that
might impact performance.

The `reduce_history` method was removed from the agent base class, in
favor of having the caller manage the change history and reduction as
needed. The `reduce_history` was added to the agent group chat, as the
chat history is managed internally as agents are invoked.

### Description

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄

---------

Co-authored-by: Evan Mattson <[email protected]>
Co-authored-by: Evan Mattson <[email protected]>
 2025-01-29 00:48:26 +00:00
Evan Mattson 7e206318dd Python: Allow enums to be passed in to on_function_result. Improve handling FRC result so it can be hashed. (#10316) 
### Motivation and Context

Two topics in this PR:
- A request to be able to pass in Enums to the Python process
`on_function_result`, instead of just strings.
- There were some custom types (Pydantic models) that were unable to
hash during an agent group chat. One could mark the model `frozen` so
that it is hashable; however, we should add better handling in our code,
in case a dev doesn't want to do that.

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->

### Description

This PR:
- checks if the function name in `on_function_result` in our process
framework is an enum, if so, convert to a string.
- improves the parsing of the `result` in `FunctionResultContent` so
that it can be included in the hash function during an agent group chat.
Adding tests to handle multiple different scenarios.
- Closes #10299 
- Closes #10305

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [X] The code builds clean without any errors or warnings
- [X] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [X] All unit tests pass, and I have added new tests where possible
- [X] I didn't break anyone 😄
 2025-01-28 21:35:37 +00:00
Evan Mattson 4d29656780 Python: Introduce the chat history reducer (#10190) 
### Motivation and Context

The SK Python framework has been missing the ability to configure a chat
history reducer of type `ChatHistoryTruncationReducer` and
`ChatHistorySummarizationReducer` which have existed in the .Net SK
Agent framework for some time.

The goal of this PR is to introduce the chat history reducers and allow
them for use for not only the agent framework, but also anything else
that uses a chat history (chat completion, for example). The
ChatHistoryReducer extends the ChatHistory class, and so it's simple to
include a reducer and logic to reduce messages as one manages the chat
history either in an agent framework setting or in a chat completion
setting.

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->

### Description

This PR:
- Introduces the chat history reducer functionality in Python -- both
the `ChatHistoryTruncationReducer` and
`ChatHistorySummarizationReducer`.
- Add unit tests for code coverage.
- Adds a sample `Chat Completion History Reducer` to show how to
configure both reducers and what each parameter does.
- Add chat completion samples showing how to manage a chat history
reducer, including how to specify that function call content and
function result content should be part of the summarization payload.
- Updates the Agent SelectionStrategy, KernelFunctionSelectionStrategy
and KernelFunctionTermination strategy to use the reducer.
- Additionally updates the classes above to use a new `select_agent`
abstract method so that one can define an initial agent to run in a
particular scenario.
- Removes the deprecated `FunctionCallBehavior` class, and removes some
nasty circular dependencies that we had lurking in the code base for
some time. This `FunctionCallBehavior` has been marked with a
deprecation warning for 6+ months now. All samples and docs have moved
over to use `FunctionChoiceBehavior` - developers using
`FunctionCallBehavior` should have had enough time to switch.
- Closes #7969
- Closes #10102

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [X] The code builds clean without any errors or warnings
- [X] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [X] All unit tests pass, and I have added new tests where possible
- [X] I didn't break anyone 😄
 2025-01-23 22:30:54 +00:00
Eduard van Valkenburg e44f70d9d6 Python: bug fixes from graphrag insights (#10270) 
### Motivation and Context

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->
Based on insights from the graphrag team, several bug fixes related to
data and model validators.

### Description

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
For all cases of model_validator in before mode, there is no guarantee
that the data passed is a dict,
especially when a model is used as a field in another model, then a
string is passed to the before validator.
So in those cases we check before doing logic, otherwise just pass it on
and let Pydantic handle the invalid data.

Some small logic fixes in the vector store model decorator:
- updated docstring with the right exceptions
- additional checks on attributes
- additional checks on VectorFields with tests

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄
 2025-01-23 20:22:29 +00:00
ふぁーTao ChenEvan Mattson
 		2e4455b202 Python: Feat/Add DEVELOPER role for openai o1 (#10033) 
### Motivation and Context

Currently, the OpenAI O1 model introduces a new role "developer".
However, passing the conventional "system" role results in an error. To
address this issue, the following changes are proposed.

### Description

1. **Add a Method to the `ChatHistory` Class**  
- Implement a method to handle role conversion or substitution logic for
compatibility with the O1 model.

2. **Expand the `AuthorRole` Enum**  
   - Add `"developer"` as a new value to the `AuthorRole` enum.

3. **Improve Developer Experience (UX)**  
- If the `"system"` role is mistakenly passed, the logic should
internally convert it to `"developer"` for better UX.
- However, since all models other than O1 still use `"system"`, simply
adding support for the `"developer"` role seems to be the most optimal
solution at this point.

- Closes: #10032 

### Background

- The O1 model no longer supports the `"system"` role, causing errors
when it is passed.
- The introduction of the `"developer"` role must be integrated in a way
that preserves compatibility with other models.

### Usage

```python
import asyncio

from semantic_kernel.connectors.ai.function_choice_behavior import FunctionChoiceBehavior
from semantic_kernel.connectors.ai.open_ai.prompt_execution_settings.azure_chat_prompt_execution_settings import (
    AzureChatPromptExecutionSettings,
)
from semantic_kernel.connectors.ai.open_ai.services.azure_chat_completion import AzureChatCompletion
from semantic_kernel.contents.chat_history import ChatHistory
from semantic_kernel.kernel import Kernel

async def main():
    kernel = Kernel()
    service_id = "azure-openai"
    kernel.add_service(
        service=AzureChatCompletion(
            service_id=service_id,
            api_key=*********,
            endpoint=*********,
            deployment_name="o1",
            api_version="2024-12-01-preview",
        )
    )

    settings = kernel.get_prompt_execution_settings_from_service_id(service_id=service_id)

    if isinstance(settings, AzureChatPromptExecutionSettings):
        settings.function_choice_behavior = FunctionChoiceBehavior.Auto(auto_invoke=True)

    service = kernel.get_service(service_id=service_id)

    if not isinstance(service, AzureChatCompletion):
        raise Exception("Invalid Value")

    history = ChatHistory()
    history.add_developer_message("You are a helpful assistant.")
    history.add_user_message("hello")

    result = await service.get_chat_message_contents(chat_history=history, settings=settings, kernel=kernel)

    if not result:
        raise Exception("result is None")

    print(result[0].content)

if __name__ == "__main__":
    asyncio.run(main())
```

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄

---------

Co-authored-by: Tao Chen <[email protected]>
Co-authored-by: Evan Mattson <[email protected]>
 2025-01-15 07:24:56 +00:00
Tao Chen 4882550306 Python: Fix function call content argument parsing (#10132) 
### Motivation and Context

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->
This PR includes changes to improve the handling of JSON parsing when it
fails to be parsed as it is in function call arguments by adding a
regular expression for preprocessing JSON strings to remove single
quotes only when they are not escaped.

### Description

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->

Enhancements to JSON parsing:

*
[`python/semantic_kernel/contents/function_call_content.py`](diffhunk://#diff-1f5f27eade7117045bc9e62d66ebef5c93047142cdb4fd0115b0ff8eadf2f69cL152-R164):
Modified `parse_arguments` method to preprocess JSON strings by
replacing single quotes with double quotes, except for escaped single
quotes. Added a debug log for invalid JSON.

Test file updates:

*
[`python/tests/unit/contents/test_function_call_content.py`](diffhunk://#diff-6e891b3044d5bf5caa7abc97a44ef11ce291aba01fa8032ab1104773f4545c9dR135-R148):
Added new tests for parsing single-quoted JSON strings. Renamed the test
file from `test_function_call.py` to `test_function_call_content.py`.

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄
 2025-01-09 10:47:09 +00:00
Evan Mattson 36701a2c73 Python: Include a function_invoke_attempt index with Streaming CMC (#10009) 
### Motivation and Context

During auto function calling, we're yielding all messages back without
any indication as to which invocation index they are related to. This
information could be helpful to the caller to understand in which order
message chunks were received during the auto function invocation loop.

Depending upon the behavior of auto function calling, the
`request_index` iterates up to the `maximum_auto_invoke_attempts`. The
caller doesn't know today which function auto invoke attempt they're
currently on -- so simply handing all yielded messages can be confusing.
In a new PR, we will handle adding the `request_index` (perhaps with a
different name) to make it easier to know which streaming message chunks
to concatenate, which should help reduce the confusion down the line.

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->

### Description

This PR adds:
- The `function_invoke_attempt` attribute to the
`StreamingChatMessageContent` class. This can help callers/users track
which streaming chat message chunks belong to which auto function
invocation attempt.
- A new keyword argument was added to the
`_inner_get_streaming_chat_message_contents` to allow the
`function_invoke_attempt` int to be passed through to the
`StreamingChatMessageContent` creation in each AI Service. This
**additive** keyword argument should not break existing.
- Updates unit tests
- Creates four new samples to showcase auto function calling: streaming
auto invoke / manual invoke (print tool calls), and non-streaming auto
invoke / manual invoke (print tool calls). These samples allow one to
specify the AI service that supports function calling, as listed in the
samples.
- Closes #10006 

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [X] The code builds clean without any errors or warnings
- [X] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [X] All unit tests pass, and I have added new tests where possible
- [X] I didn't break anyone 😄
 2024-12-19 08:13:00 +00:00
Evan Mattson 16690ed857 Python: Yield FunctionResultContent in streaming chat completion path. Update tests. (#9974) 
### Motivation and Context

Currently, if using SK's Python streaming chat completion path, we only
yield the following content types: `StreamingChatMessageContent` and
`FunctionCallContent`. We are not yielding `FunctionResultContent` which
is valuable for some use cases.

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->

### Description

This PR updates the code to yield `FunctionResultContent` if it exists
in the streaming chat completion path. When we merge the function call
result content together into a `StreamingChatMessageContent` type, we
check if that message has items (which are of type
`FunctionResultContent`) and if so, we yield them. The filter path still
works because once they're yielded, we break out of the function calling
loop.

We need to include the `ai_model_id` if it exists for the current
PromptExecutionSettings because if performing a `reduce` operation to
add two streaming chat message chunks together, the
`StreamingChatMessageContent` that has the function results will break
if the `ai_model_id` is not set (the error is thrown in the `__add__`
override for the `StreamingChatMessageContent`.

Some unit tests that cover function calling were also updated -- during
the test, the test JSON function args were breaking in the `json.loads`
call because they contained single quotes and not double. We're now
sanitizing the args, just in case, so we don't break there.

This PR fixes:
- #9408
- #9968 

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [X] The code builds clean without any errors or warnings
- [X] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [X] All unit tests pass, and I have added new tests where possible
- [X] I didn't break anyone 😄
 2024-12-18 01:09:25 +00:00
Evan MattsonandEduard van Valkenburg b42720890d Python: FunctionResultContent hash fix to handle lists/sets (#9978) 
### Motivation and Context

Currently, the hash function in `FunctionResultContent` does not
properly handle types of list or set. Adding the proper handling to turn
these into a tuple, and then the hash function works properly.

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->

### Description

This PR:
- Makes sure the FunctionResultContent `__hash__` method can properly
handle mutable types like lists or sets
- Adds tests to ensure the new behavior works as expected
- Ensures the create message content method can handle lists or sets
when generating the content and the model can only accept a string.
- Fixes #9977

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [X] The code builds clean without any errors or warnings
- [X] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [X] All unit tests pass, and I have added new tests where possible
- [X] I didn't break anyone 😄

---------

Co-authored-by: Eduard van Valkenburg <[email protected]>
 2024-12-16 22:14:22 +00:00
Eduard van Valkenburg 87967ac233 Python: enable ollama streaming tool calls (#9890) 
### Motivation and Context

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->

Ollama has enabled streaming tool calling, and so we add that too!
See release notes: https://github.com/ollama/ollama/releases/tag/v0.4.6

Also encountered a bug when passing generic PromptExecutionSettings to
all ChatCompletions with Function calling enabled, the
_verify_function_choice_settings often check if something is a type of
PES matching the service, that is not the case and throws even though it
should work (for instance when using a yaml based function), this fixes
that as well by casting in the public methods of chat completion base.

### Description

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄
 2024-12-09 15:58:57 +00:00
Eduard van Valkenburg e0042afa80 Python: graduate filters, add exception during addition and some cleanup (#9856) 
### Motivation and Context

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->
This PR graduates the filters.

This includes some updates to the docstrings.
And adds a specific Exception for errors during adding or removing of
filters.

Closes #9838 
Fixes #9641 

### Description

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄
 2024-12-03 17:40:01 +00:00
Eduard van Valkenburg d8496da277 Python: improve integration test runs (#9845) 
## Motivation and Context

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->
Split the integration tests into seperate ones for Memory and
Completions
Added code to allow the samples to run as well
Moved containers to service containers.

### Description

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄
 2024-11-29 11:31:12 +00:00
Evan MattsonandTao Chen 05f9a98121 Python: Bump Python version to 1.16.0 for a release. (#9781) 
### Motivation and Context

Bump Python version to 1.16.0 for a release.

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->

### Description

Bump Python version to 1.16.0 for a release. Includes feature work to
promote OpenAPI plugin to preview from experimental.

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [X] The code builds clean without any errors or warnings
- [X] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [X] All unit tests pass, and I have added new tests where possible
- [X] I didn't break anyone 😄

---------

Co-authored-by: Tao Chen <[email protected]>
 2024-11-21 22:20:57 +00:00
Tao Chen 63d1dc761f Python: Text to audio (#9625) 
### Motivation and Context

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->
Addresses: https://github.com/microsoft/semantic-kernel/issues/7433

### Description

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
Add text to audio interface to Python Semantic Kernel.

### Contribution Checklist
1. Text-to-audio client base.
2. OpenAI and Azure OpenAI implementations of the client.
3. Unit tests and integration tests.
4. Samples.

<!-- Before submitting this PR, please make sure: -->

- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄
 2024-11-12 22:07:34 +00:00
Evan Mattson 05f4589549 Python: Exclude the KernelContent's inner_content as it breaks serializing the model (#9626) 
### Motivation and Context

There's been an error for a while where serializing the chat history
leads to the following error:

```
PydanticSerializationError(Error serializing to JSON: TypeError: 'MockValSer' object 
cannot be converted to 'SchemaSerializer')}
```

This is due to the fact that we're trying to do a model dump of an
object that we don't have a Pydantic model for as the `inner_content`
type is `Any`. Most, if not all, of the `inner_content` info is already
included in our `ChatMessageContent`, therefore we're excluding this
attribute from serialization.

Note: if you require the information in this `inner_content` attribute,
you will be responsible for grabbing the content before serializing the
chat history, as it won't be included.

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->

### Description

This PR excludes the `inner_content` from the `model_dump()`,
`model_dump_json()`, or `serialize()` calls.
- Fixes #7903 
- Adds a concept sample showing how to serialize the `ChatHistory` after
invoking a prompt via the kernel.

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [X] The code builds clean without any errors or warnings
- [X] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [X] All unit tests pass, and I have added new tests where possible
- [X] I didn't break anyone 😄
 2024-11-08 20:23:53 +00:00
Tao Chen e9a755dfbc Python: Audio to text (#9505) 
### Motivation and Context

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->
Addresses: https://github.com/microsoft/semantic-kernel/issues/7434


### Description

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
Add audio to text to SK Python.

### Contribution Checklist
1. Add audio to text to SK Python.
2. Add text to image integration tests.
3. Refactor Azure AI connector AD auth.

<!-- Before submitting this PR, please make sure: -->

- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄
 2024-11-08 00:12:07 +00:00
Tao ChenandEvan Mattson c304c8beca Python: restructure integration tests (#9331) 
### Motivation and Context

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->
Our integration tests for the embedding services and memory are not set
up in a way that makes it easy for developers to add test cases.

### Description

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
1. Restructure the tests so that it's easier to add new tests when a new
service is created.
2. Reduce duplicate code.
3. Add Ollama embedding service tests.

> Note: Besides the new Ollama tests, no other tests are added or
dropped.

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄

---------

Co-authored-by: Evan Mattson <[email protected]>
 2024-10-23 22:08:44 +00:00
Evan Mattson 12a31daac8 Python: Return streaming tool call output to the caller (#9362) 
### Motivation and Context

The streaming function call content, function result content, or code
interpreter output was not being streamed back to the caller chunk by
chunk.

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->

### Description

This PR enabled yielding the proper chunks to the caller based on
handling the `thread.run.step.delta` event.
- Fixes #9352 
- Adds test coverage
- Fixes a separate mypy issue related to BinaryContent

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [X] The code builds clean without any errors or warnings
- [X] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [X] All unit tests pass, and I have added new tests where possible
- [X] I didn't break anyone 😄
 2024-10-22 13:20:27 -04:00
Tao Chen 481d16a749 Python: Bedrock connector (#9100) 
### Motivation and Context

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->
Add Amazon Bedrock AI to Semantic Kernel.

Amazon Bedrock is a platform on AWS that offers access to a wide range
to LLMs, including proprietary Amazon Titan models.

We are integrating it into Semantic Kernel as one of our AI connectors
to provide more options to developers.

### Description

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
1. All interfaces implemented: chat completion (streaming +
non-streaming), text completion (streaming + non-streaming), and text
embedding.
2. Supports for function calling.
3. Supports for image content.
4. Supports for all models available on Bedrock.

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄
 2024-10-10 22:33:42 +00:00
Nico MöllerTao ChenEduard van ValkenburgEvan Mattson
 		b9e1133091 Python: #6761 Onnx Connector (#8106) 
### Motivation and Context

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->

 1. Why is this changed required ?
To enable Onnx Models with Semantic Kernel, there was the issue #6761 in
the Backlog to add a Onnx Connector
2. What problem does it solve ?
It solves the problem, that semantic kernel is not yet integrated with
Onnx Gen Ai Runtime
3. What scenario does it contribute to?
The scenario is to use different connector than HF,OpenAI or
AzureOpenAI. When User's want to use Onnx they can easliy integrate it
now
4. If it fixes an open issue, please link to the issue here.
#6761

### Description

The changes made are designed by my own based on other connectors, i
tried to stay as close as possible to the structure.
For the integration i installed the mistral python package in the
repository.

I added the following Classes :

- OnnxCompletionBase --> Responsible to control the inference
- OnnxTextCompletion --> Inherits from OnnxCompletionBase 
    - Support for Text Completion with and without Images
    - Ready for Multimodal Inference
    - Ready for Text Only Inference
- Supports all Models on
[onnxruntime-genai](https://github.com/microsoft/onnxruntime-genai)
- OnnxChatCompletion -->Inherits from OnnxCompletionBase
    - Support for Text Completion with and without Images
- The user needs to provide the corresponding chat template to use this
class
    - Ready for Multimodal Inference
    - Ready for Text Only Inference
- Supports all Models on
[onnxruntime-genai](https://github.com/microsoft/onnxruntime-genai)


What is integrated yet :

- [X] OnnxCompletionBase Class
- [x]  OnnxChatCompletionBase Class with Dynamic Template Support
- [x]  OnnxTextCompletionBase Class
- [x] Sample Multimodal Inference with Phi3-Vision
- [x] Sample of OnnxChatCompletions with Phi3
- [x] Sample of OnnxTextCompletions with Phi3
- [x]  Integration Tests
- [x]  Unit Tests


### Some Notes



### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄

---------

Co-authored-by: Tao Chen <[email protected]>
Co-authored-by: Eduard van Valkenburg <[email protected]>
Co-authored-by: Evan Mattson <[email protected]>
 2024-10-10 17:41:21 +00:00
Evan Mattson f08cf3c778 Python: Add support for streaming OpenAI assistants (#9055) 
### Motivation and Context

OpenAI assistants were introduced several versions ago; however, there
was open work to support streaming for OpenAI assistants v2. Work to
support streaming assistants was also needed in Agent Group Chat
scenarios.

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->

### Description

This PR introduces:
- Support for streaming with (Azure) OpenAI assistants
- Support for Agent Chat Scenarios that can now leverage a streaming
invoke instead of just a sync invoke.
- Unit tests to increase the test coverage to near 100% for the newly
added code
- Adds a few more OpenAI assistant agent samples
- Closes #7267 

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [X] The code builds clean without any errors or warnings
- [X] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [X] All unit tests pass, and I have added new tests where possible
- [X] I didn't break anyone 😄
 2024-10-02 16:00:11 +00:00
Tao Chen cdb2d5930b Python: Improve exception log message in streaming content merge item list (#9052) 
### Motivation and Context

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->
Exceptions are expected to happen when merging the item lists from two
chat streaming contents. When all attempts fail, the new item will
simply be added to the end of the list.

The `WARN` log level is too high for this event.

### Description

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
1. Reduce the log level of the event to `DEBUG`.
2. Add an additional log message for the event when all attempts have
failed.

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄
 2024-10-01 17:17:27 +00:00
Tao ChenandBen Thomas 5c5d761c6f Python: Streaming content for token usage (#8902) 
### Motivation and Context

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->
OpenAI recently starts providing token usage information on their
streaming chat completion API. This ADR opens the discussion on how we
should consume that information within our `StreamingChatMessageContent`
data structure.

### Description

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->

1. An ADR documenting 4 approaches.
2. Implementation of the selected approach.
3. Fix issues where data in the streaming contents are modified after
concatenation (`__add__`).
4. Add streaming to model diagnostics operation names so that we can
distinguish streaming and non-streaming operations.

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄

---------

Co-authored-by: Ben Thomas <[email protected]>
 2024-09-26 00:03:42 +00:00
Eduard van Valkenburg 3bfee7b64b Python: fix discrimator field for CMC (#8417) 
### Motivation and Context

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->
Turns out we had a mistake in the way CMC discriminates content types,
this PR fixes that

### Description

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->
Adds discriminator field to the item in the list rather then the list
itself
Adds additional tests

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [x] I didn't break anyone 😄
 2024-08-29 16:01:53 +00:00
Eduard van Valkenburg f90538b9c0 Python: switch to uv (#8320) 
### Motivation and Context

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->
Switches from poetry to uv
much faster and easier lock management
easier maintenance of pyproject.toml

fixes: #6584
fixes: #8326

### Description

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [x] The code builds clean without any errors or warnings
- [x] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [x] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄
 2024-08-26 13:34:29 +02:00
Evan Mattson ee40e88c13 Python: Introduce Agent group chat functionality (#8184) 
### Motivation and Context

In the next phase of the agent work in Python, we are to introduce the
ability to have agents interact with each other. Agent group chat allows
developers to create richer and more dynamic agent interactions.

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->

### Description

This PR introduces the following:
- Agent group chat functionality via an `agent_chat` base class that
forms the foundation for group chats. An `agent_group_chat` class
extends that underlying functionality and provides an entry point to
kick off the group chat either via invoking the chat for the defined
agents, or specifying if a provided agent is to join the existing chat.
- The required channels for the chat completion agent and assistant
agent to properly chat together
- Adds more concept samples and getting started with agent samples to
enrich the onboarding experiencing for those wanting to see the
capability of agents and agent group chat.
- Adds unit test for near 100% coverage
- Updates some existing imports for agents, which try and make the
required import experience more intuitive.
- Introduces assistant agent abstractions for working with files and
vector stores so we don't directly expose the underlying client types.
- Closes #7966 
- Closes #7577

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [X] The code builds clean without any errors or warnings
- [X] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [X] All unit tests pass, and I have added new tests where possible
- [ ] I didn't break anyone 😄
 2024-08-19 18:04:44 +00:00