mirror of
https://github.com/microsoft/semantic-kernel.git
synced 2026-10-12 02:19:06 +00:00
Lee Miller
4a2cf70c9f
Implement MicrosoftML and DeepDev token counters in the TextChunker example. Update the project file with new package references and modify the RunExampleWithCustomTokenCounter method to support different token counter types. Inspired by #2809 Fixes #478 | Iteration | MicrosoftML (ms) | MicrosoftMLRoberta (ms) | SharpToken (ms) | DeepDev (ms) | |------------|-------------------|--------------------------|-----------------|--------------| | 1 | 38 | 10,189 | 14,305 | 16,701 | | 2 | 36 | 5,581 | 8,381 | 14,214 | | 3 | 13 | 5,354 | 7,955 | 13,630 | | 4 | 27 | 5,679 | 9,156 | 16,164 | | 5 | 16 | 5,158 | 8,657 | 17,276 | | Average | 26.0 | 7,512.2 | 9,710.8 | 15,597 | ### SharpToken <sup style="font-size: smaller;">(Avg. Execution Time: 9,710.8 ms)</sup> ``` The city of Venice, located in the northeastern part of Italy, is renowned for its unique geographical features. Built on more than 100 small islands in a lagoon in the Adriatic Sea, it has no roads, just canals including the Grand Canal thoroughfare lined with Renaissance and Gothic palaces. The central square, Piazza San Marco, contains St. Mark's Basilica, which is tiled with Byzantine mosaics, and the Campanile bell tower offering views of the city's red roofs. ------------------------ The Amazon Rainforest, also known as Amazonia, is a moist broadleaf tropical rainforest in the Amazon biome that covers most of the Amazon basin of South America. This basin encompasses 7 million square kilometers, of which 5.5 million square kilometers are covered by the rainforest. This region includes territory belonging to nine nations and 3.4 million square kilometers of uncontacted tribes. The Amazon represents over half of the planet's remaining rainforests and comprises the largest and most biodiverse tract of tropical rainforest in the world. ------------------------ The Great Barrier Reef is the world's largest coral reef system composed of over 2,900 individual reefs and 900 islands stretching for over 2,300 kilometers over an area of approximately 344,400 square kilometers. The reef is located in the Coral Sea, off the coast of Queensland, Australia. The Great Barrier Reef can be seen from outer space and is the world's biggest single structure made by living organisms. This reef structure is composed of and built by billions of tiny organisms, known as coral polyps. ``` ### MicrosoftML <sup style="font-size: smaller;">(Avg. Execution Time: 26.0 ms)</sup> ``` The city of Venice, located in the northeastern part of Italy, is renowned for its unique geographical features. Built on more than 100 small ------------------------ islands in a lagoon in the Adriatic Sea, it has no roads, just canals including the Grand Canal thoroughfare lined with Renaissance and ------------------------ Gothic palaces. The central square, Piazza San Marco, contains St. Mark's Basilica, which is tiled with Byzantine mosaics, ------------------------ and the Campanile bell tower offering views of the city's red roofs. The Amazon Rainforest, also known as Amazonia, ------------------------ is a moist broadleaf tropical rainforest in the Amazon biome that covers most of the Amazon basin of South America. This basin encompasses 7 ------------------------ million square kilometers, of which 5. 5 million square kilometers are covered by the rainforest. This region includes territory ------------------------ belonging to nine nations and 3. 4 million square kilometers of uncontacted tribes. The Amazon represents over ------------------------ half of the planet's remaining rainforests and comprises the largest and most biodiverse tract of tropical rainforest in the world. ------------------------ The Great Barrier Reef is the world's largest coral reef system composed of over 2, 900 individual reefs and 900 islands ------------------------ stretching for over 2, 300 kilometers over an area of approximately 344, 400 square kilometers. The reef is located in the ------------------------ Coral Sea, off the coast of Queensland, Australia. The Great Barrier Reef can be seen from outer space and is the world's ------------------------ biggest single structure made by living organisms. This reef structure is composed of and built by billions of tiny organisms, known as coral polyps. ``` ### MicrosoftMLRoberta <sup style="font-size: smaller;">(Avg. Execution Time: 7,512.2 ms)</sup> ``` The city of Venice, located in the northeastern part of Italy, is renowned for its unique geographical features. Built on more than 100 small islands in a lagoon in the Adriatic Sea, it has no roads, just canals including the Grand Canal thoroughfare lined with Renaissance and Gothic palaces. The central square, Piazza San Marco, contains St. Mark's Basilica, which is tiled with Byzantine mosaics, and the Campanile bell tower offering views of the city's red roofs. ------------------------ The Amazon Rainforest, also known as Amazonia, is a moist broadleaf tropical rainforest in the Amazon biome that covers most of the Amazon basin of South America. This basin encompasses 7 million square kilometers, of which 5.5 million square kilometers are covered by the rainforest. This region includes territory belonging to nine nations and 3.4 million square kilometers of uncontacted tribes. The Amazon represents over half of the planet's remaining rainforests and comprises the largest and most biodiverse tract of tropical rainforest in the world. ------------------------ The Great Barrier Reef is the world's largest coral reef system composed of over 2,900 individual reefs and 900 islands stretching for over 2,300 kilometers over an area of approximately 344,400 square kilometers. The reef is located in the Coral Sea, off the coast of Queensland, Australia. The Great Barrier Reef can be seen from outer space and is the world's biggest single structure made by living organisms. This reef structure is composed of and built by billions of tiny organisms, known as coral polyps. ``` ### DeepDev <sup style="font-size: smaller;">(Avg. Execution Time: 15,597 ms)</sup> ``` The city of Venice, located in the northeastern part of Italy, is renowned for its unique geographical features. Built on more than 100 small islands in a lagoon in the Adriatic Sea, it has no roads, just canals including the Grand Canal thoroughfare lined with Renaissance and Gothic palaces. The central square, Piazza San Marco, contains St. Mark's Basilica, which is tiled with Byzantine mosaics, and the Campanile bell tower offering views of the city's red roofs. ------------------------ The Amazon Rainforest, also known as Amazonia, is a moist broadleaf tropical rainforest in the Amazon biome that covers most of the Amazon basin of South America. This basin encompasses 7 million square kilometers, of which 5.5 million square kilometers are covered by the rainforest. This region includes territory belonging to nine nations and 3.4 million square kilometers of uncontacted tribes. The Amazon represents over half of the planet's remaining rainforests and comprises the largest and most biodiverse tract of tropical rainforest in the world. ------------------------ The Great Barrier Reef is the world's largest coral reef system composed of over 2,900 individual reefs and 900 islands stretching for over 2,300 kilometers over an area of approximately 344,400 square kilometers. The reef is located in the Coral Sea, off the coast of Queensland, Australia. The Great Barrier Reef can be seen from outer space and is the world's biggest single structure made by living organisms. This reef structure is composed of and built by billions of tiny organisms, known as coral polyps. ``` ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [ ] The code builds clean without any errors or warnings - [ ] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [ ] All unit tests pass, and I have added new tests where possible - [ ] I didn't break anyone 😄
72 lines
4.9 KiB
XML
72 lines
4.9 KiB
XML
<Project Sdk="Microsoft.NET.Sdk">
|
|
<PropertyGroup>
|
|
<UserSecretsId>5ee045b0-aea3-4f08-8d31-32d1a6f8fed0</UserSecretsId>
|
|
</PropertyGroup>
|
|
<PropertyGroup>
|
|
<RootNamespace></RootNamespace>
|
|
<TargetFramework>net6.0</TargetFramework>
|
|
<RollForward>LatestMajor</RollForward>
|
|
<OutputType>Exe</OutputType>
|
|
<IsPackable>false</IsPackable>
|
|
<!-- Suppress: "Declare types in namespaces", "Require ConfigureAwait" -->
|
|
<NoWarn>CA1050;CA1707;CA2007;VSTHRD111;CS1591</NoWarn>
|
|
</PropertyGroup>
|
|
<ItemGroup>
|
|
<None Remove="appsettings.json" />
|
|
</ItemGroup>
|
|
<ItemGroup>
|
|
<PackageReference Include="Azure.Identity" />
|
|
<PackageReference Include="Microsoft.Extensions.Configuration" />
|
|
<PackageReference Include="Microsoft.Extensions.Configuration.Binder" />
|
|
<PackageReference Include="Microsoft.Extensions.Configuration.EnvironmentVariables" />
|
|
<PackageReference Include="Microsoft.Extensions.Configuration.Json" />
|
|
<PackageReference Include="Microsoft.Extensions.Configuration.UserSecrets" />
|
|
<PackageReference Include="Microsoft.Extensions.DependencyInjection" />
|
|
<PackageReference Include="Microsoft.Extensions.Http" />
|
|
<PackageReference Include="Microsoft.Extensions.Logging.Console" />
|
|
<PackageReference Include="Newtonsoft.Json" />
|
|
<PackageReference Include="Polly" />
|
|
<PackageReference Include="SharpToken" />
|
|
<PackageReference Include="Microsoft.ML.Tokenizers" />
|
|
<PackageReference Include="Microsoft.DeepDev.TokenizerLib" />
|
|
<PackageReference Include="System.Linq.Async" />
|
|
</ItemGroup>
|
|
<ItemGroup>
|
|
<ProjectReference Include="..\..\src\Connectors\Connectors.AI.OpenAI\Connectors.AI.OpenAI.csproj" />
|
|
<ProjectReference Include="..\..\src\Connectors\Connectors.AI.HuggingFace\Connectors.AI.HuggingFace.csproj" />
|
|
<ProjectReference Include="..\..\src\Connectors\Connectors.Memory.AzureCognitiveSearch\Connectors.Memory.AzureCognitiveSearch.csproj" />
|
|
<ProjectReference Include="..\..\src\Connectors\Connectors.Memory.Chroma\Connectors.Memory.Chroma.csproj" />
|
|
<ProjectReference Include="..\..\src\Connectors\Connectors.Memory.Kusto\Connectors.Memory.Kusto.csproj" />
|
|
<ProjectReference Include="..\..\src\Connectors\Connectors.Memory.DuckDB\Connectors.Memory.DuckDB.csproj" />
|
|
<ProjectReference Include="..\..\src\Connectors\Connectors.Memory.Postgres\Connectors.Memory.Postgres.csproj" />
|
|
<ProjectReference Include="..\..\src\Connectors\Connectors.Memory.Sqlite\Connectors.Memory.Sqlite.csproj" />
|
|
<ProjectReference Include="..\..\src\Connectors\Connectors.Memory.Weaviate\Connectors.Memory.Weaviate.csproj" />
|
|
<ProjectReference Include="..\..\src\Connectors\Connectors.Memory.Redis\Connectors.Memory.Redis.csproj" />
|
|
<ProjectReference Include="..\..\src\Extensions\Planning.ActionPlanner\Planning.ActionPlanner.csproj" />
|
|
<ProjectReference Include="..\..\src\Extensions\Planning.SequentialPlanner\Planning.SequentialPlanner.csproj" />
|
|
<ProjectReference Include="..\..\src\Extensions\Planning.StepwisePlanner\Planning.StepwisePlanner.csproj" />
|
|
<ProjectReference Include="..\..\src\Connectors\Connectors.Memory.Pinecone\Connectors.Memory.Pinecone.csproj" />
|
|
<ProjectReference Include="..\..\src\Extensions\Reliability.Basic\Reliability.Basic.csproj" />
|
|
<ProjectReference Include="..\..\src\Extensions\TemplateEngine.PromptTemplateEngine\TemplateEngine.PromptTemplateEngine.csproj" />
|
|
<ProjectReference Include="..\..\src\Extensions\Reliability.Polly\Reliability.Polly.csproj" />
|
|
<ProjectReference Include="..\..\src\SemanticKernel.Abstractions\SemanticKernel.Abstractions.csproj" />
|
|
<ProjectReference Include="..\..\src\Plugins\Plugins.Core\Plugins.Core.csproj" />
|
|
<ProjectReference Include="..\..\src\Functions\Functions.OpenAPI\Functions.OpenAPI.csproj" />
|
|
<ProjectReference Include="..\..\src\Functions\Functions.Grpc\Functions.Grpc.csproj" />
|
|
<ProjectReference Include="..\..\src\Connectors\Connectors.Memory.Qdrant\Connectors.Memory.Qdrant.csproj" />
|
|
<ProjectReference Include="..\..\src\Plugins\Plugins.Web\Plugins.Web.csproj" />
|
|
<ProjectReference Include="..\..\src\SemanticKernel.Core\SemanticKernel.Core.csproj" />
|
|
<ProjectReference Include="..\NCalcPlugins\NCalcPlugins.csproj" />
|
|
<!-- Because some of the referenced projects have dependencies that themselves have System.Text.Json set with a minimum of 7.0. -->
|
|
<PackageReference Include="System.Text.Json" />
|
|
<PackageVersion Update="System.Text.Json" Version="7.0.3" />
|
|
</ItemGroup>
|
|
<ItemGroup>
|
|
<EmbeddedResource Include="Resources\30-user-prompt.txt" />
|
|
<EmbeddedResource Include="Resources\30-system-prompt.txt" />
|
|
<EmbeddedResource Include="Resources\30-user-context.txt" />
|
|
<EmbeddedResource Include="Resources\EnglishRoberta\dict.txt" />
|
|
<EmbeddedResource Include="Resources\EnglishRoberta\encoder.json" />
|
|
<EmbeddedResource Include="Resources\EnglishRoberta\vocab.bpe" />
|
|
</ItemGroup>
|
|
</Project> |