langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-22 04:25:29 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Prompt Tokens For Embedding Model exceed Document chunk max tokens? #13021

New Issue
Closed
opened 2026-02-21 19:10:11 -05:00 by yindo · 11 comments
yindo commented 2026-02-21 19:10:11 -05:00
Owner
Copy Link
Copy Source

Originally created by @Hollowmon111 on GitHub (Apr 16, 2025).

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:)
  • Please do not modify this template :) and fill in all the required fields.

Dify version

0.15.3

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

Upload document to knowledge base

Set max document chunk size =800

Embed document chunks

✔️ Expected Behavior

Set document chunk size = 800

Prompt tokens for embedding model = document chunk size, or atleast very close to it.

❌ Actual Behavior

Set max document chunk size =800

Embedding document chunks

Some chunks result in large prompt tokens in excess of 2000, despite a max document chunk setting of 800 tokens.

##embedding model log file excerpt:
0.00.481.257 I srv main: initializing server
0.00.481.258 I srv init: initializing slots, n_slots = 1
0.00.481.262 I slt init: id 00 | task -1 | new slot n_ctx_slot = 1024
0.00.481.350 I srv main: starting server
0.02.116.306 I srv log_server_request: rid 37870423 | GET /v1/models xxx
0.02.116.411 I srv log_server_response: rid 37870423 | GET /v1/models xxx | status 200 | cost 0.00s
0.02.237.445 I srv log_server_request: rid 37991561 | GET /v1/models xxx
0.02.237.552 I srv log_server_response: rid 37991561 | GET /v1/models x | status 200 | cost 0.00s
3.52.800.251 I srv log_server_request: rid 268561279 | POST /v1/embeddings xxx
3.52.801.201 I srv oaicompat_embeddings_requ: rid 268561279 | {"input":["A project is assigned responsibility \nthat encompasses one or more life cycle stages of the system-of-interest. ISO/IEC 15288:2008 is applicable \nto organizations and projects whether they act as the acquirer or the supplier of a system and whether the \nsystem is comprised of products, services, or a combination of both. \nThe context of ISO/IEC 15288:2008 is illustrated in Figure 1"],"model":"jina-embeddings-v2-base-en","encoding_format":"float"}
3.52.801.457 I srv log_server_request: rid 268562485 | POST /v1/embeddings xxx
3.52.802.385 I srv oaicompat_embeddings_requ: rid 268562485 | {"input":["15\r\n4.6.4 Hierarchy of projects ........................................................................................................................... 16\r\n4.7 Adaptation concepts ........................................................................................................................... 17\r\n4.7.1 General ................................................................................................................................................. 17\r\n4.7.2 Adaptation ............................................................................................................................................ 18\r\n4.7.3 Life cycle adaptation ........................................................................................................................... 18\r\n4.7.4 Adaptation for domains, disciplines and specialties ...................................................................... 18\r\n4.7.5 Tailoring ............................................................................................................................................... 19\r\n5 Applying ISO/IEC 15288:2008 ............................................................................................................. 19\r\n5.1 Overview ............................................................................................................................................... 19\r\n5.2 Application strategy ............................................................................................................................ 19\r\n5.2.1 Overview .............................................................................................................................................."],"model":"jina-embeddings-v2-base-en","encoding_format":"float"}
3.52.802.424 I slt launch_slot_with_task: rid 268561279 | id 00 | task 0 | processing task, max_tps = N/A
3.52.802.443 I slt update_slots: rid 268561279 | id 00 | task 0 | new prompt, n_ctx_slot = 1024, n_keep = 0, n_prompt_tokens = 83
3.52.802.452 I slt update_slots: rid 268561279 | id 00 | task 0 | prompt processing, n_past = 83, n_tokens = 83, n_prompt_tokens = 83, n_preprocessed_tokens = 0
3.52.802.465 I slt update_slots: rid 268561279 | id 00 | task 0 | prompt done, n_past = 83, n_tokens = 83
3.52.806.311 I srv log_server_request: rid 268567340 | POST /v1/embeddings 172.17.0.2:56582
3.52.808.645 I srv oaicompat_embeddings_requ: rid 268567340 | {"input":["2\r\n3.4 Comparison to prior version of ISO/IEC 15288:2008 ......................................................................... 4\r\n4 Application concepts ............................................................................................................................ 5\r\n4.1 Overview ................................................................................................................................................. 5\r\n4.2 System concepts ................................................................................................................................... 6\r\n4.3 Life cycle concepts ............................................................................................................................... 6\r\n4.4 Process concepts .................................................................................................................................. 6\r\n4.4.1 General ................................................................................................................................................... 6\r\n4.4.2 Process principles ................................................................................................................................. 8\r\n4.4.3 Process categories of ISO/IEC 15288:2008 ........................................................................................ 9\r\n4.4.4 Recursive/iterative application of processes ................................................................................... 10\r\n4.5 Organizational concepts ..................................................................................................................... 12\r\n4.5.1 General ................................................................................................................................................"],"model":"jina-embeddings-v2-base-en","encoding_format":"float"}
3.53.008.702 I slt release: rid 268561279 | id 00 | task 0 | stop processing: n_past = 83, truncated = false
3.53.008.832 I slt launch_slot_with_task: rid 268562485 | id 00 | task 1 | processing task, max_tps = N/A
3.53.008.862 I slt update_slots: rid 268562485 | id 00 | task 1 | new prompt, n_ctx_slot = 1024, n_keep = 0, n_prompt_tokens = 1476
3.53.008.863 I slt release: rid 268562485 | id 00 | task 1 | stop processing: n_past = 0, truncated = false
3.53.008.864 E srv send_error: task 1 | error = input is too large to process, please increase the physical batch size
3.53.008.884 I srv oaicompat_embeddings_resp: rid 268561279 | prompt_tokens: 83
3.53.008.950 I slt launch_slot_with_task: rid 268567340 | id 00 | task 2 | processing task, max_tps = N/A
3.53.008.951 I slt update_slots: rid 268567340 | id 00 | task 2 | new prompt, n_ctx_slot = 1024, n_keep = 0, n_prompt_tokens = 1545
3.53.008.952 I slt release: rid 268567340 | id 00 | task 2 | stop processing: n_past = 0, truncated = false
3.53.008.952 E srv send_error: task 2 | error = input is too large to process, please increase the physical batch size
3.53.009.094 I srv log_server_response: rid 268562485 | POST /v1/embeddings xxx | status 500 | cost 0.21s
3.53.009.166 I srv log_server_response: rid 268567340 | POST /v1/embeddings xx | status 500 | cost 0.20s
3.53.009.204 I srv log_server_response: rid 268561279 | POST /v1/embeddings xx | status 200 | cost 0.21s
3.53.016.593 I srv log_server_request: rid 268777622 | POST /v1/embeddings xx
3.53.016.638 I srv oaicompat_embeddings_requ: rid 268777622 | {"input":["Users are responsible for its correct application.\r\n© The British Standards Institution 2013\r\nPublished by BSI Standards Limited 2013\r\nISBN 978 0 580 73284 3\r\nICS 35.080\r\nCompliance with a British Standard cannot confer immunity from\r\nlegal obligations.\r\nThis Published Document was published under the authority of the\r\nStandards Policy and Strategy Committee on 30 April 2013.\r\nAmendments issued since publication\r\nAmd. No. Date Text affected\r\nPD ISO/IEC TR 24748-2:2011 PUBLISHED DOCUMENT"],"model":"jina-embeddings-v2-base-en","encoding_format":"float"}
3.53.016.801 I srv log_server_request: rid 268777830 | POST /v1/embeddings xxx
3.53.016.853 I slt launch_slot_with_task: rid 268777622 | id 00 | task 3 | processing task, max_tps = N/A
3.53.016.886 I slt update_slots: rid 268777622 | id 00 | task 3 | new prompt, n_ctx_slot = 1024, n_keep = 0, n_prompt_tokens = 95
3.53.016.920 I slt update_slots: rid 268777622 | id 00 | task 3 | prompt processing, n_past = 95, n_tokens = 95, n_prompt_tokens = 95, n_preprocessed_tokens = 0
3.53.016.963 I srv log_server_request: rid 268777992 | POST /v1/embeddings xxx
3.53.017.033 I slt update_slots: rid 268777622 | id 00 | task 3 | prompt done, n_past = 95, n_tokens = 95
3.53.017.053 I srv oaicompat_embeddings_requ: rid 268777830 | {"input":["National foreword\r\nThis Published Document is the UK implementation of ISO/IEC TR 24748-\r\n2:2011. It supersedes PD ISO/IEC TR 19760:2003 which is withdrawn.\r\nThe UK participation in its preparation was entrusted to Technical Committee\r\nIST/15, Software and systems engineering.\r\nA list of organizations represented on this committee can be obtained on\r\nrequest to its secretary.\r\nThis publication does not purport to include all the necessary provisions of a\r\ncontract"],"model":"jina-embeddings-v2-base-en","encoding_format":"float"}
3.53.017.062 I srv oaicompat_embeddings_requ: rid 268777992 | {"input":["9\r\n4.4.4 Recursive/iterative application of processes ................................................................................... 10\r\n4.5 Organizational concepts ..................................................................................................................... 12\r\n4.5.1 General ................................................................................................................................................. 12\r\n4.5.2 Responsibility ...................................................................................................................................... 12\r\n4.5.3 Organizational relationships .............................................................................................................. 12\r\n4.5.4 Project organizational structure ........................................................................................................ 13\r\n4.6 Project concepts .................................................................................................................................. 13\r\n4.6.1 General ................................................................................................................................................. 13\r\n4.6.2 Project relationships ........................................................................................................................... 14\r\n4.6.3 Enabling system relationships .......................................................................................................... 15\r\n4.6.4 Hierarchy of projects ........................................................................................................................... 16\r\n4.7 Adaptation concepts .........................................................................................................................."],"model":"jina-embeddings-v2-base-en","encoding_format":"float"}

Originally created by @Hollowmon111 on GitHub (Apr 16, 2025). ### Self Checks - [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [x] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:) - [x] Please do not modify this template :) and fill in all the required fields. ### Dify version 0.15.3 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce Upload document to knowledge base Set max document chunk size =800 Embed document chunks ### ✔️ Expected Behavior Set document chunk size = 800 Prompt tokens for embedding model = document chunk size, or atleast very close to it. ### ❌ Actual Behavior Set max document chunk size =800 Embedding document chunks Some chunks result in large prompt tokens in excess of 2000, despite a max document chunk setting of 800 tokens. ##embedding model log file excerpt: 0.00.481.257 I srv main: initializing server 0.00.481.258 I srv init: initializing slots, n_slots = 1 0.00.481.262 I slt init: id 00 | task -1 | new slot n_ctx_slot = 1024 0.00.481.350 I srv main: starting server 0.02.116.306 I srv log_server_request: rid 37870423 | GET /v1/models xxx 0.02.116.411 I srv log_server_response: rid 37870423 | GET /v1/models xxx | status 200 | cost 0.00s 0.02.237.445 I srv log_server_request: rid 37991561 | GET /v1/models xxx 0.02.237.552 I srv log_server_response: rid 37991561 | GET /v1/models x | status 200 | cost 0.00s 3.52.800.251 I srv log_server_request: rid 268561279 | POST /v1/embeddings xxx 3.52.801.201 I srv oaicompat_embeddings_requ: rid 268561279 | {"input":["A project is assigned responsibility \nthat encompasses one or more life cycle stages of the system-of-interest. ISO/IEC 15288:2008 is applicable \nto organizations and projects whether they act as the acquirer or the supplier of a system and whether the \nsystem is comprised of products, services, or a combination of both. \nThe context of ISO/IEC 15288:2008 is illustrated in Figure 1"],"model":"jina-embeddings-v2-base-en","encoding_format":"float"} 3.52.801.457 I srv log_server_request: rid 268562485 | POST /v1/embeddings xxx 3.52.802.385 I srv oaicompat_embeddings_requ: rid 268562485 | {"input":["15\r\n4.6.4 Hierarchy of projects ........................................................................................................................... 16\r\n4.7 Adaptation concepts ........................................................................................................................... 17\r\n4.7.1 General ................................................................................................................................................. 17\r\n4.7.2 Adaptation ............................................................................................................................................ 18\r\n4.7.3 Life cycle adaptation ........................................................................................................................... 18\r\n4.7.4 Adaptation for domains, disciplines and specialties ...................................................................... 18\r\n4.7.5 Tailoring ............................................................................................................................................... 19\r\n5 Applying ISO/IEC 15288:2008 ............................................................................................................. 19\r\n5.1 Overview ............................................................................................................................................... 19\r\n5.2 Application strategy ............................................................................................................................ 19\r\n5.2.1 Overview .............................................................................................................................................."],"model":"jina-embeddings-v2-base-en","encoding_format":"float"} 3.52.802.424 I slt launch_slot_with_task: rid 268561279 | id 00 | task 0 | processing task, max_tps = N/A 3.52.802.443 I slt update_slots: rid 268561279 | id 00 | task 0 | new prompt, n_ctx_slot = 1024, n_keep = 0, n_prompt_tokens = 83 3.52.802.452 I slt update_slots: rid 268561279 | id 00 | task 0 | prompt processing, n_past = 83, n_tokens = 83, n_prompt_tokens = 83, n_preprocessed_tokens = 0 3.52.802.465 I slt update_slots: rid 268561279 | id 00 | task 0 | prompt done, n_past = 83, n_tokens = 83 3.52.806.311 I srv log_server_request: rid 268567340 | POST /v1/embeddings 172.17.0.2:56582 3.52.808.645 I srv oaicompat_embeddings_requ: rid 268567340 | {"input":["2\r\n3.4 Comparison to prior version of ISO/IEC 15288:2008 ......................................................................... 4\r\n4 Application concepts ............................................................................................................................ 5\r\n4.1 Overview ................................................................................................................................................. 5\r\n4.2 System concepts ................................................................................................................................... 6\r\n4.3 Life cycle concepts ............................................................................................................................... 6\r\n4.4 Process concepts .................................................................................................................................. 6\r\n4.4.1 General ................................................................................................................................................... 6\r\n4.4.2 Process principles ................................................................................................................................. 8\r\n4.4.3 Process categories of ISO/IEC 15288:2008 ........................................................................................ 9\r\n4.4.4 Recursive/iterative application of processes ................................................................................... 10\r\n4.5 Organizational concepts ..................................................................................................................... 12\r\n4.5.1 General ................................................................................................................................................"],"model":"jina-embeddings-v2-base-en","encoding_format":"float"} 3.53.008.702 I slt release: rid 268561279 | id 00 | task 0 | stop processing: n_past = 83, truncated = false 3.53.008.832 I slt launch_slot_with_task: rid 268562485 | id 00 | task 1 | processing task, max_tps = N/A 3.53.008.862 I slt update_slots: rid 268562485 | id 00 | task 1 | new prompt, n_ctx_slot = 1024, n_keep = 0, n_prompt_tokens = 1476 3.53.008.863 I slt release: rid 268562485 | id 00 | task 1 | stop processing: n_past = 0, truncated = false 3.53.008.864 E srv send_error: task 1 | error = input is too large to process, please increase the physical batch size 3.53.008.884 I srv oaicompat_embeddings_resp: rid 268561279 | prompt_tokens: 83 3.53.008.950 I slt launch_slot_with_task: rid 268567340 | id 00 | task 2 | processing task, max_tps = N/A 3.53.008.951 I slt update_slots: rid 268567340 | id 00 | task 2 | new prompt, n_ctx_slot = 1024, n_keep = 0, n_prompt_tokens = 1545 3.53.008.952 I slt release: rid 268567340 | id 00 | task 2 | stop processing: n_past = 0, truncated = false 3.53.008.952 E srv send_error: task 2 | error = input is too large to process, please increase the physical batch size 3.53.009.094 I srv log_server_response: rid 268562485 | POST /v1/embeddings xxx | status 500 | cost 0.21s 3.53.009.166 I srv log_server_response: rid 268567340 | POST /v1/embeddings xx | status 500 | cost 0.20s 3.53.009.204 I srv log_server_response: rid 268561279 | POST /v1/embeddings xx | status 200 | cost 0.21s 3.53.016.593 I srv log_server_request: rid 268777622 | POST /v1/embeddings xx 3.53.016.638 I srv oaicompat_embeddings_requ: rid 268777622 | {"input":["Users are responsible for its correct application.\r\n© The British Standards Institution 2013\r\nPublished by BSI Standards Limited 2013\r\nISBN 978 0 580 73284 3\r\nICS 35.080\r\nCompliance with a British Standard cannot confer immunity from\r\nlegal obligations.\r\nThis Published Document was published under the authority of the\r\nStandards Policy and Strategy Committee on 30 April 2013.\r\nAmendments issued since publication\r\nAmd. No. Date Text affected\r\nPD ISO/IEC TR 24748-2:2011 PUBLISHED DOCUMENT"],"model":"jina-embeddings-v2-base-en","encoding_format":"float"} 3.53.016.801 I srv log_server_request: rid 268777830 | POST /v1/embeddings xxx 3.53.016.853 I slt launch_slot_with_task: rid 268777622 | id 00 | task 3 | processing task, max_tps = N/A 3.53.016.886 I slt update_slots: rid 268777622 | id 00 | task 3 | new prompt, n_ctx_slot = 1024, n_keep = 0, n_prompt_tokens = 95 3.53.016.920 I slt update_slots: rid 268777622 | id 00 | task 3 | prompt processing, n_past = 95, n_tokens = 95, n_prompt_tokens = 95, n_preprocessed_tokens = 0 3.53.016.963 I srv log_server_request: rid 268777992 | POST /v1/embeddings xxx 3.53.017.033 I slt update_slots: rid 268777622 | id 00 | task 3 | prompt done, n_past = 95, n_tokens = 95 3.53.017.053 I srv oaicompat_embeddings_requ: rid 268777830 | {"input":["National foreword\r\nThis Published Document is the UK implementation of ISO/IEC TR 24748-\r\n2:2011. It supersedes PD ISO/IEC TR 19760:2003 which is withdrawn.\r\nThe UK participation in its preparation was entrusted to Technical Committee\r\nIST/15, Software and systems engineering.\r\nA list of organizations represented on this committee can be obtained on\r\nrequest to its secretary.\r\nThis publication does not purport to include all the necessary provisions of a\r\ncontract"],"model":"jina-embeddings-v2-base-en","encoding_format":"float"} 3.53.017.062 I srv oaicompat_embeddings_requ: rid 268777992 | {"input":["9\r\n4.4.4 Recursive/iterative application of processes ................................................................................... 10\r\n4.5 Organizational concepts ..................................................................................................................... 12\r\n4.5.1 General ................................................................................................................................................. 12\r\n4.5.2 Responsibility ...................................................................................................................................... 12\r\n4.5.3 Organizational relationships .............................................................................................................. 12\r\n4.5.4 Project organizational structure ........................................................................................................ 13\r\n4.6 Project concepts .................................................................................................................................. 13\r\n4.6.1 General ................................................................................................................................................. 13\r\n4.6.2 Project relationships ........................................................................................................................... 14\r\n4.6.3 Enabling system relationships .......................................................................................................... 15\r\n4.6.4 Hierarchy of projects ........................................................................................................................... 16\r\n4.7 Adaptation concepts .........................................................................................................................."],"model":"jina-embeddings-v2-base-en","encoding_format":"float"}
yindo added the 🐞 bugstale labels 2026-02-21 19:10:11 -05:00
yindo closed this issue 2026-02-21 19:10:11 -05:00
yindo commented 2026-02-21 19:10:11 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Apr 16, 2025):

I found some similar threads that might be relevant to your issue:

  1. Maximum chunk length setting can't control the size of the chunk - This discussion is open and addresses issues with the maximum chunk length setting not effectively controlling the chunk size [1].

  2. Error when I modify the max_tokens of the custom Maximum chunk length from 1000 to 2000 - This discussion is open and deals with errors encountered when changing the maximum chunk length [2].

Regarding your issue, Dify calculates the number of tokens in a document chunk using the get_text_embedding_num_tokens method from the embedding_model. If the document's form is "qa_model", it calculates the tokens for the concatenated text of content and segment.answer. Otherwise, it calculates the tokens for content alone [3].

The mechanism for setting and enforcing the maximum document chunk size during the embedding process is controlled by the MAXIMUM_CHUNK_TOKEN_LENGTH configuration. In Dify 0.15.0, this setting is hardcoded to a maximum of 4000 tokens and cannot be modified through environment variables [4][5].

If you're experiencing issues with chunks exceeding the set maximum token limit, it might be related to how the tokens are calculated or processed. You may want to check if the document form or any preprocessing steps are affecting the token count.

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other

@dosubot[bot] commented on GitHub (Apr 16, 2025): <!-- Answer --> I found some similar threads that might be relevant to your issue: 1. [Maximum chunk length setting can't control the size of the chunk](https://github.com/langgenius/dify/discussions/16670) - This discussion is open and addresses issues with the maximum chunk length setting not effectively controlling the chunk size <sup>[[1]](https://github.com/langgenius/dify/discussions/16670)</sup>. 2. [Error when I modify the max_tokens of the custom Maximum chunk length from 1000 to 2000](https://github.com/langgenius/dify/discussions/4080) - This discussion is open and deals with errors encountered when changing the maximum chunk length <sup>[[2]](https://github.com/langgenius/dify/discussions/4080)</sup>. Regarding your issue, Dify calculates the number of tokens in a document chunk using the `get_text_embedding_num_tokens` method from the `embedding_model`. If the document's form is "qa_model", it calculates the tokens for the concatenated text of `content` and `segment.answer`. Otherwise, it calculates the tokens for `content` alone <sup>[[3]](https://github.com/langgenius/dify/blob/main/api/services/dataset_service.py)</sup>. The mechanism for setting and enforcing the maximum document chunk size during the embedding process is controlled by the `MAXIMUM_CHUNK_TOKEN_LENGTH` configuration. In Dify 0.15.0, this setting is hardcoded to a maximum of 4000 tokens and cannot be modified through environment variables <sup>[[4]](https://github.com/langgenius/dify-docs/blob/main/en/getting-started/install-self-hosted/environments.md)</sup><sup>[[5]](https://github.com/langgenius/dify/issues/12500)</sup>. If you're experiencing issues with chunks exceeding the set maximum token limit, it might be related to how the tokens are calculated or processed. You may want to check if the document form or any preprocessing steps are affecting the token count. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.* --- <sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/86398b69-7919-40c9-b8a0-a08eb6ce12c9?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/86398b69-7919-40c9-b8a0-a08eb6ce12c9?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/86398b69-7919-40c9-b8a0-a08eb6ce12c9?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/86398b69-7919-40c9-b8a0-a08eb6ce12c9?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/86398b69-7919-40c9-b8a0-a08eb6ce12c9?feedback_type=hallucination) | [Bug Report](https://app.dosu.dev/response-feedback/86398b69-7919-40c9-b8a0-a08eb6ce12c9?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/86398b69-7919-40c9-b8a0-a08eb6ce12c9?feedback_type=other)</sup>
yindo commented 2026-02-21 19:10:11 -05:00
Author
Owner
Copy Link
Copy Source

@Hollowmon111 commented on GitHub (Apr 16, 2025):

@dosu where can i find this file: get_text_embedding_num_tokens

@Hollowmon111 commented on GitHub (Apr 16, 2025): @dosu where can i find this file: get_text_embedding_num_tokens
yindo commented 2026-02-21 19:10:11 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Apr 16, 2025):

You can find the get_text_embedding_num_tokens method in the dataset_service.py file located in the api/services directory of this repository. This method is responsible for calculating the number of tokens in a document chunk.

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other

@dosubot[bot] commented on GitHub (Apr 16, 2025): <!-- Answer --> You can find the `get_text_embedding_num_tokens` method in the `dataset_service.py` file located in the `api/services` directory of this repository. This method is responsible for calculating the number of tokens in a document chunk. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.* --- <sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/ec01f39c-bfad-43e5-a4d4-c920abcd58b7?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/ec01f39c-bfad-43e5-a4d4-c920abcd58b7?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/ec01f39c-bfad-43e5-a4d4-c920abcd58b7?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/ec01f39c-bfad-43e5-a4d4-c920abcd58b7?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/ec01f39c-bfad-43e5-a4d4-c920abcd58b7?feedback_type=hallucination) | [Bug Report](https://app.dosu.dev/response-feedback/ec01f39c-bfad-43e5-a4d4-c920abcd58b7?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/ec01f39c-bfad-43e5-a4d4-c920abcd58b7?feedback_type=other)</sup>
yindo commented 2026-02-21 19:10:12 -05:00
Author
Owner
Copy Link
Copy Source

@Hollowmon111 commented on GitHub (Apr 16, 2025):

@dosu how does dify tokenise document chunks?

@Hollowmon111 commented on GitHub (Apr 16, 2025): @dosu how does dify tokenise document chunks?
yindo commented 2026-02-21 19:10:12 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Apr 16, 2025):

Dify tokenizes document chunks using two segmentation modes: "通用分段" (General Segmentation) and "父子分段" (Parent-Child Segmentation). In General Segmentation, content is split into independent segments based on user-defined rules, with the default segment identifier being \n for paragraph segmentation. Users can customize these rules using regular expressions. Dify also allows setting a maximum segment length (default 500 tokens, up to 4000 tokens) and a segment overlap length (recommended 10-25% of the segment length) to improve recall accuracy. Text preprocessing rules are applied to filter out meaningless content, such as replacing consecutive spaces and removing URLs and email addresses [1].

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other

@dosubot[bot] commented on GitHub (Apr 16, 2025): <!-- Answer --> Dify tokenizes document chunks using two segmentation modes: "通用分段" (General Segmentation) and "父子分段" (Parent-Child Segmentation). In General Segmentation, content is split into independent segments based on user-defined rules, with the default segment identifier being `\n` for paragraph segmentation. Users can customize these rules using regular expressions. Dify also allows setting a maximum segment length (default 500 tokens, up to 4000 tokens) and a segment overlap length (recommended 10-25% of the segment length) to improve recall accuracy. Text preprocessing rules are applied to filter out meaningless content, such as replacing consecutive spaces and removing URLs and email addresses <sup>[[1]](https://github.com/langgenius/dify-docs/blob/main/zh_CN/guides/knowledge-base/create-knowledge-and-upload-documents/chunking-and-cleaning-text.md)</sup>. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.* --- <sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/9b252693-4547-4598-bfa8-b6c160f3e18f?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/9b252693-4547-4598-bfa8-b6c160f3e18f?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/9b252693-4547-4598-bfa8-b6c160f3e18f?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/9b252693-4547-4598-bfa8-b6c160f3e18f?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/9b252693-4547-4598-bfa8-b6c160f3e18f?feedback_type=hallucination) | [Bug Report](https://app.dosu.dev/response-feedback/9b252693-4547-4598-bfa8-b6c160f3e18f?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/9b252693-4547-4598-bfa8-b6c160f3e18f?feedback_type=other)</sup>
yindo commented 2026-02-21 19:10:12 -05:00
Author
Owner
Copy Link
Copy Source

@Hollowmon111 commented on GitHub (Apr 16, 2025):

@dosu where can i find this file: embedding_model.get_text_embedding_num_tokens

@Hollowmon111 commented on GitHub (Apr 16, 2025): @dosu where can i find this file: embedding_model.get_text_embedding_num_tokens
yindo commented 2026-02-21 19:10:12 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Apr 16, 2025):

You can find the embedding_model.get_text_embedding_num_tokens method in the api/core/plugin/manager/model.py file of the Dify repository [1].

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other

@dosubot[bot] commented on GitHub (Apr 16, 2025): <!-- Answer --> You can find the `embedding_model.get_text_embedding_num_tokens` method in the `api/core/plugin/manager/model.py` file of the Dify repository <sup>[[1]](https://github.com/langgenius/dify/blob/main/api/core/plugin/manager/model.py)</sup>. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.* --- <sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/74edb6f7-d1fe-4534-9477-3bf308ffffe0?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/74edb6f7-d1fe-4534-9477-3bf308ffffe0?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/74edb6f7-d1fe-4534-9477-3bf308ffffe0?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/74edb6f7-d1fe-4534-9477-3bf308ffffe0?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/74edb6f7-d1fe-4534-9477-3bf308ffffe0?feedback_type=hallucination) | [Bug Report](https://app.dosu.dev/response-feedback/74edb6f7-d1fe-4534-9477-3bf308ffffe0?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/74edb6f7-d1fe-4534-9477-3bf308ffffe0?feedback_type=other)</sup>
yindo commented 2026-02-21 19:10:12 -05:00
Author
Owner
Copy Link
Copy Source

@Hollowmon111 commented on GitHub (Apr 16, 2025):

@dosu How do i force document chunks to not exceed a set value in tokens?

@Hollowmon111 commented on GitHub (Apr 16, 2025): @dosu How do i force document chunks to not exceed a set value in tokens?
yindo commented 2026-02-21 19:10:12 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Apr 16, 2025):

To ensure document chunks do not exceed a set value in tokens, you can configure the chunk_size parameter in the TextSplitter class. This parameter controls the maximum size of text segments when processing documents. You can set it to your desired token limit. Additionally, you can adjust the chunk_overlap parameter to specify the overlap in characters between chunks for smoother transitions.

If you're using a tokenizer-based approach, such as with the TokenTextSplitter class, you can specify tokens_per_chunk and chunk_overlap in the Tokenizer dataclass to control chunk size and overlap in terms of tokens. For more precise control, you can use specific tokenizers like HuggingFace or tiktoken with the from_huggingface_tokenizer or from_tiktoken_encoder methods to create a TextSplitter that counts length using these tokenizers [1][2][3].

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other

@dosubot[bot] commented on GitHub (Apr 16, 2025): <!-- Answer --> To ensure document chunks do not exceed a set value in tokens, you can configure the `chunk_size` parameter in the `TextSplitter` class. This parameter controls the maximum size of text segments when processing documents. You can set it to your desired token limit. Additionally, you can adjust the `chunk_overlap` parameter to specify the overlap in characters between chunks for smoother transitions. If you're using a tokenizer-based approach, such as with the `TokenTextSplitter` class, you can specify `tokens_per_chunk` and `chunk_overlap` in the `Tokenizer` dataclass to control chunk size and overlap in terms of tokens. For more precise control, you can use specific tokenizers like HuggingFace or tiktoken with the `from_huggingface_tokenizer` or `from_tiktoken_encoder` methods to create a `TextSplitter` that counts length using these tokenizers <sup>[[1]](https://github.com/langgenius/dify-docs/blob/main/en/getting-started/install-self-hosted/environments.md)</sup><sup>[[2]](https://github.com/langgenius/dify-docs/blob/main/en/guides/knowledge-base/knowledge-and-documents-maintenance/maintain-knowledge-documents.md)</sup><sup>[[3]](https://github.com/langgenius/dify-docs/blob/main/en/guides/knowledge-base/create-knowledge-and-upload-documents/chunking-and-cleaning-text.md)</sup>. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.* --- <sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/6f88b49e-e80e-4220-8a8c-36eed5ff95be?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/6f88b49e-e80e-4220-8a8c-36eed5ff95be?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/6f88b49e-e80e-4220-8a8c-36eed5ff95be?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/6f88b49e-e80e-4220-8a8c-36eed5ff95be?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/6f88b49e-e80e-4220-8a8c-36eed5ff95be?feedback_type=hallucination) | [Bug Report](https://app.dosu.dev/response-feedback/6f88b49e-e80e-4220-8a8c-36eed5ff95be?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/6f88b49e-e80e-4220-8a8c-36eed5ff95be?feedback_type=other)</sup>
yindo commented 2026-02-21 19:10:12 -05:00
Author
Owner
Copy Link
Copy Source

@Hollowmon111 commented on GitHub (Apr 16, 2025):

@dosu where can i find TokenTextSplitter class

@Hollowmon111 commented on GitHub (Apr 16, 2025): @dosu where can i find TokenTextSplitter class
yindo commented 2026-02-21 19:10:12 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Apr 16, 2025):

You can find the TokenTextSplitter class in the text_splitter.py file located in the api/core/plugin/manager directory of this repository.

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other

@dosubot[bot] commented on GitHub (Apr 16, 2025): <!-- Answer --> You can find the `TokenTextSplitter` class in the `text_splitter.py` file located in the `api/core/plugin/manager` directory of this repository. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.* --- <sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/4835ee3b-e7d7-439b-bb7f-0e7c9fa4ae6f?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/4835ee3b-e7d7-439b-bb7f-0e7c9fa4ae6f?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/4835ee3b-e7d7-439b-bb7f-0e7c9fa4ae6f?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/4835ee3b-e7d7-439b-bb7f-0e7c9fa4ae6f?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/4835ee3b-e7d7-439b-bb7f-0e7c9fa4ae6f?feedback_type=hallucination) | [Bug Report](https://app.dosu.dev/response-feedback/4835ee3b-e7d7-439b-bb7f-0e7c9fa4ae6f?feedback_type=bug_report) | [Other](https://app.dosu.dev/response-feedback/4835ee3b-e7d7-439b-bb7f-0e7c9fa4ae6f?feedback_type=other)</sup>
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label stale 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#13021
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?