semantic-kernel/dotnet/samples/TelemetryExample

Semantic Kernel Telemetry Example

This example project shows how an application can be configured to send Semantic Kernel telemetry to Application Insights.

Note that it is also possible to use other Application Performance Management (APM) vendors. An example is Prometheus. Please refer to this link on how to do it.

For more information, please refer to the following articles:

1. Observability
2. OpenTelemetry
3. Enable Azure Monitor OpenTelemetry for .Net
4. Configure Azure Monitor OpenTelemetry for .Net
5. Add, modify, and filter Azure Monitor OpenTelemetry
6. Customizing OpenTelemetry .NET SDK for Metrics
7. Customizing OpenTelemetry .NET SDK for Logs

What to expect

In this example project, the Handlebars planner will be invoked to achieve a goal. The planner will request the model to create a plan, comprising three steps, with two of them being prompt-based kernel functions. The plan will be executed to produce the desired output, effectively fulfilling the goal.

The Semantic Kernel SDK is designed to efficiently generate comprehensive logs, traces, and metrics throughout the planner invocation, as well as during function and plan execution. This allows you to effectively monitor your AI application's performance and accurately track token consumption.

ActivitySource.StartActivity internally determines if there are any listeners recording the Activity. If there are no registered listeners or there are listeners that are not interested, StartActivity() will return null and avoid creating the Activity object. Read more here.

Configuration

Require resources

1. Application Insights
2. Azure OpenAI

Secrets

This example will require secrets and credentials to access your Application Insights instance and Azure OpenAI. We suggest using .NET Secret Manager to avoid the risk of leaking secrets into the repository, branches and pull requests. You can also use environment variables if you prefer.

To set your secrets with Secret Manager:

cd dotnet/samples/TelemetryExample

dotnet user-secrets set "AzureOpenAI:ChatDeploymentName" "..."
dotnet user-secrets set "AzureOpenAI:ChatModelId" "..."
dotnet user-secrets set "AzureOpenAI:Endpoint" "https://... .openai.azure.com/"
dotnet user-secrets set "AzureOpenAI:ApiKey" "..."

dotnet user-secrets set "ApplicationInsights:ConnectionString" "..."

Running the example

Simply run dotnet run under this directory if the command line interface is preferred. Otherwise, this example can also be run in Visual Studio.

This will output the Operation/Trace ID, which can be used later in Application Insights for searching the operation.

Application Insights/Azure Monitor

Logs and traces

Go to your Application Insights instance, click on Transaction search on the left menu. Use the operation id output by the program to search for the logs and traces associated with the operation. Click on any of the search result to view the end-to-end transaction details. Read more here.

Metrics

Running the application once will only generate one set of measurements (for each metrics). Run the application a couple times to generate more sets of measurements.

Note: Make sure not to run the program too frequently. Otherwise, you may get throttled.

Please refer to here on how to analyze metrics in Azure Monitor.

Log Analytics

It is also possible to use Log Analytics to query the telemetry items sent by the sample application. Please read more here.

For example, to create a pie chart to summarize the Handlebars planner status:

dependencies
| where name == "Microsoft.SemanticKernel.Planning.Handlebars.HandlebarsPlanner"
| extend status = iff(success == True, "Success", "Failure")
| summarize count() by status
| render piechart

Or to create a bar chart to summarize the Handlebars planner status by date:

dependencies
| where name == "Microsoft.SemanticKernel.Planning.Handlebars.HandlebarsPlanner"
| extend status = iff(success == True, "Success", "Failure"), day = bin(timestamp, 1d)
| project day, status
| summarize
    success = countif(status == "Success"),
    failure = countif(status == "Failure") by day
| extend day = format_datetime(day, "MM/dd/yy")
| order by day
| render barchart

Or to see status and performance of each planner run:

dependencies
| where name == "Microsoft.SemanticKernel.Planning.Handlebars.HandlebarsPlanner"
| extend status = iff(success == True, "Success", "Failure")
| project timestamp, id, status, performance = performanceBucket
| order by timestamp

It is also possible to summarize the total token usage:

customMetrics
| where name == "semantic_kernel.connectors.openai.tokens.total"
| project value
| summarize sum(value)
| project Total = sum_value

Or track token usage by functions:

customMetrics
| where name == "semantic_kernel.function.invocation.token_usage.prompt" and customDimensions has "semantic_kernel.function.name"
| project customDimensions, value
| extend function = tostring(customDimensions["semantic_kernel.function.name"])
| project function, value
| summarize sum(value) by function
| render piechart

Azure Dashboard

You can create an Azure Dashboard to visualize the custom telemetry items. You can read more here: Create a new dashboard.

More information

• Telemetry docs
• Planner telemetry improvement ADR