mirror of
https://github.com/microsoft/semantic-kernel.git
synced 2026-10-12 02:19:06 +00:00
8527c58cca
### Motivation and Context <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> addressing issue #1387 chunking text should allow use of an external tokenizer ### Description <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> added pass-through of an token counting function, defaulting to the existing _token_counter() method while fixing a type hint bug I got sucked into making a few changes to clean up the code. future work would be nice to add chunk overlap functionality similar to langchain's [TextSplitter]( https://github.com/hwchase17/langchain/blob/a6ebffb69504576a805f3b9f09732ad344751b89/langchain/text_splitter.py#L58) ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [X] The code builds clean without any errors or warnings - [X] The PR follows SK Contribution Guidelines (https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) - [X] The code follows the .NET coding conventions (https://learn.microsoft.com/dotnet/csharp/fundamentals/coding-style/coding-conventions) verified with `dotnet format` - [X] All unit tests pass, and I have added new tests where possible - [X] I didn't break anyone 😄 --------- Co-authored-by: Scott Graham <[email protected]> Co-authored-by: Lee Miller <[email protected]> Co-authored-by: Mark Karle <[email protected]> Co-authored-by: Shawn Callegari <[email protected]>