[Plugin Request]: Tokenizer #56

New Issue

2026-02-22T17:16:59-05:00

yindo commented

2026-02-22 17:16:59 -05:00

Originally created by @DavideDelbianco on GitHub (May 30, 2025).

Plugin Name

Tokenizer

Function Description

The plugin should allow the selection of tokenization method (GPT, Tiktoken, etc..)
It should expose the ability to tokenize a text and/or return the number of tokens

It would allow to evaluate a received text to reject a query before sending it to LLM node
It would allow to create a workflow tool to put together with the use of Storage plugins a workflow tool that can count the number of used tokens by userid / appid / whatever.

Most of the code is probably already in the Dify repo because a similar functionality has been added as an ENV variable option.
It would be a nice alternative waiting for the team to expose as an output the LLM usage info from the LLM node

Official Website URL

No response

Originally created by @DavideDelbianco on GitHub (May 30, 2025). ### Plugin Name Tokenizer ### Function Description The plugin should allow the selection of tokenization method (GPT, Tiktoken, etc..) It should expose the ability to tokenize a text and/or return the number of tokens It would allow to evaluate a received text to reject a query before sending it to LLM node It would allow to create a workflow tool to put together with the use of Storage plugins a workflow tool that can count the number of used tokens by userid / appid / whatever. Most of the code is probably already in the Dify repo because a similar functionality has been added as an ENV variable option. It would be a nice alternative waiting for the team to expose as an output the LLM usage info from the LLM node ### Official Website URL _No response_

yindo added the Plugin Request label 2026-02-22 17:16:59 -05:00

yindo closed this issue

2026-02-22 17:16:59 -05:00

yindo commented

2026-02-22 17:17:00 -05:00

@jingfelix commented on GitHub (Jun 23, 2025):

https://github.com/jingfelix/dify-plugin-tokenizer

Hi, I've built a tokenizer plugin using transformers.AutoTokenizer. Does that meet your requirements? I'll add more details and get it listed later today.

Due to the limitation of cloud.dify.ai, you can only deploy it to your local environment (for now)

@jingfelix commented on GitHub (Jun 23, 2025): https://github.com/jingfelix/dify-plugin-tokenizer Hi, I've built a tokenizer plugin using transformers.AutoTokenizer. Does that meet your requirements? I'll add more details and get it listed later today. Due to the limitation of cloud.dify.ai, you can only deploy it to your local environment (for now)

yindo commented

2026-02-22 17:17:00 -05:00

@DavideDelbianco commented on GitHub (Jun 27, 2025):

This will probably help me a lot.
I would like to fork it to allow choosing either Transformers or TikToken because the latter is specialized in OpenAI models and is a lot faster

@DavideDelbianco commented on GitHub (Jun 27, 2025): This will probably help me a lot. I would like to fork it to allow choosing either Transformers or TikToken because the latter is specialized in OpenAI models and is a lot faster

yindo commented

2026-02-22 17:17:00 -05:00

@jingfelix commented on GitHub (Jun 27, 2025):

This will probably help me a lot. I would like to fork it to allow choosing either Transformers or TikToken because the latter is specialized in OpenAI models and is a lot faster

I considered using tiktoken directly, but it only supports OpenAI models, so I chose transformers to make it more versatile. I will consider adding tiktoken support in the future, although this may not be a priority.

If you have any updates or other needs on this issue, please feel free to contact me.

@jingfelix commented on GitHub (Jun 27, 2025): > This will probably help me a lot. I would like to fork it to allow choosing either Transformers or TikToken because the latter is specialized in OpenAI models and is a lot faster I considered using `tiktoken` directly, but it only supports OpenAI models, so I chose `transformers` to make it more versatile. I will consider adding `tiktoken` support in the future, although this may not be a priority. If you have any updates or other needs on this issue, please feel free to contact me.

yindo referenced this issue

2026-02-22 17:32:43 -05:00

[PR #56] [MERGED] Add WaterCrawl plugin version 0.3.1 #569

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: langgenius/dify-plugins#56