langgenius/dify-plugin-sdks
1
0
Fork 0
mirror of https://github.com/langgenius/dify-plugin-sdks.git synced 2026-07-22 02:15:22 -04:00
Code Issues 58 Packages Projects Releases 10 Wiki Activity

[PR #51] [CLOSED] feat: parameterize the default tokenizer to obtain the number of tokens #107

New Issue
Closed
opened 2026-02-15 21:15:47 -05:00 by yindo · 0 comments
yindo commented 2026-02-15 21:15:47 -05:00
Owner
Copy Link
Copy Source

📋 Pull Request Information

Original PR: https://github.com/langgenius/dify-plugin-sdks/pull/51
Author: @acelyc111
Created: 3/16/2025
Status: ❌ Closed

Base: main ← Head: selectable_tokenizer

📝 Commits (2)

  • d0e37f4 feat: parameterize the default tokenizer to obtain the number of tokens
  • ec16715 cm1

📊 Changes

6 files changed (+63 additions, -44 deletions)

View changed files

📝 python/.gitignore (+1 -0)
📝 python/dify_plugin/config/config.py (+10 -0)
📝 python/dify_plugin/interfaces/agent/__init__.py (+8 -6)
📝 python/dify_plugin/interfaces/model/ai_model.py (+15 -9)
📝 python/dify_plugin/interfaces/model/openai_compatible/llm.py (+27 -27)
📝 python/dify_plugin/interfaces/model/openai_compatible/text_embedding.py (+2 -2)

📄 Description

Especially for Chinese, it's recommended to use a more modern model for tokenizer to estimate the number of tokens.

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/langgenius/dify-plugin-sdks/pull/51 **Author:** [@acelyc111](https://github.com/acelyc111) **Created:** 3/16/2025 **Status:** ❌ Closed **Base:** `main` ← **Head:** `selectable_tokenizer` --- ### 📝 Commits (2) - [`d0e37f4`](https://github.com/langgenius/dify-plugin-sdks/commit/d0e37f4bf1054e7bc3c9d05258a5c9eb18b028ca) feat: parameterize the default tokenizer to obtain the number of tokens - [`ec16715`](https://github.com/langgenius/dify-plugin-sdks/commit/ec167158f686a86d3facff921e3d6744dfd1aa8f) cm1 ### 📊 Changes **6 files changed** (+63 additions, -44 deletions) <details> <summary>View changed files</summary> 📝 `python/.gitignore` (+1 -0) 📝 `python/dify_plugin/config/config.py` (+10 -0) 📝 `python/dify_plugin/interfaces/agent/__init__.py` (+8 -6) 📝 `python/dify_plugin/interfaces/model/ai_model.py` (+15 -9) 📝 `python/dify_plugin/interfaces/model/openai_compatible/llm.py` (+27 -27) 📝 `python/dify_plugin/interfaces/model/openai_compatible/text_embedding.py` (+2 -2) </details> ### 📄 Description Especially for Chinese, it's recommended to use a more modern model for tokenizer to estimate the number of tokens. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
yindo added the pull-request label 2026-02-15 21:15:47 -05:00
yindo closed this issue 2026-02-15 21:15:47 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
documentation
enhancement
pull-request
Mirrored from GitHub Pull Request
 question
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify-plugin-sdks#107
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?