langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-22 20:45:26 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

Unicode Escape Characters in LLM Prompt Increase Token Consumption #7973

New Issue
Closed
opened 2026-02-21 18:23:19 -05:00 by yindo · 10 comments
yindo commented 2026-02-21 18:23:19 -05:00
Owner
Copy Link
Copy Source

Originally created by @LinLin00000000 on GitHub (Feb 4, 2025).

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
  • Please do not modify this template :) and fill in all the required fields.

Dify version

v0.15.2

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

I followed the tutorial to set up a chatflow that supports document uploads. After uploading a file, the document parser correctly processed the Chinese text in the document, and the display was correct.

The test file was a DOCX document containing 540 Chinese characters and a total of 1213 characters.

test1.docx

Image

However, when passing the parsed result to the next node (LLM) in the system prompt, the parsed content was converted into Unicode escape characters.

Image

Although the LLM can understand these escape characters, this significantly increases the token consumption (2137 tokens).

I compared this with another AI workflow tool, fastgpt, which parsed the same document but sent the correct Chinese characters to the LLM node. Additionally, the token consumption was much lower (514 tokens, only 24% of Dify's consumption).

Image

Image

I also tested a PDF document with over 6000 Chinese characters and a total of over 16000 characters. Dify's Unicode escape character token consumption was 29364, while Fastgpt's token consumption was 7991 (27.2% of Dify's consumption).

I have not tested other non-ASCII languages and am unsure if there is a similar issue with other languages.

I browsed through other similar closed issues (#5010 #7040 #8056), which mention Unicode escape problems in other areas, but I'm unsure whether this issue will occur in other parts of the system.

✔️ Expected Behavior

Like fastgpt, the system should send the correct Chinese characters instead of Unicode escape characters to reduce token consumption.

❌ Actual Behavior

No response

Originally created by @LinLin00000000 on GitHub (Feb 4, 2025). ### Self Checks - [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [x] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [x] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:） - [x] Please do not modify this template :) and fill in all the required fields. ### Dify version v0.15.2 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce I followed the [tutorial](https://docs.dify.ai/zh-hans/workshop/intermediate/article-reader) to set up a chatflow that supports document uploads. After uploading a file, the document parser correctly processed the Chinese text in the document, and the display was correct. The test file was a DOCX document containing 540 Chinese characters and a total of 1213 characters. [test1.docx](https://github.com/user-attachments/files/18666238/test1.docx) ![Image](https://github.com/user-attachments/assets/ac39a46c-762b-44d9-9471-5c29a684eb90) However, when passing the parsed result to the next node (LLM) in the system prompt, the parsed content was converted into Unicode escape characters. ![Image](https://github.com/user-attachments/assets/285e049e-0a7e-4cb3-bdf7-4b950444ac9b) Although the LLM can understand these escape characters, this significantly increases the token consumption (2137 tokens). I compared this with another AI workflow tool, [fastgpt](https://github.com/labring/FastGPT), which parsed the same document but sent the correct Chinese characters to the LLM node. Additionally, the token consumption was much lower (514 tokens, only 24% of Dify's consumption). ![Image](https://github.com/user-attachments/assets/63af57fd-377a-4288-bb7f-059c43404937) ![Image](https://github.com/user-attachments/assets/15a7ff8a-7495-4943-8798-d5444f9f8651) I also tested a PDF document with over 6000 Chinese characters and a total of over 16000 characters. Dify's Unicode escape character token consumption was 29364, while Fastgpt's token consumption was 7991 (27.2% of Dify's consumption). I have not tested other non-ASCII languages and am unsure if there is a similar issue with other languages. I browsed through other similar closed issues (#5010 #7040 #8056), which mention Unicode escape problems in other areas, but I'm unsure whether this issue will occur in other parts of the system. ### ✔️ Expected Behavior Like fastgpt, the system should send the correct Chinese characters instead of Unicode escape characters to reduce token consumption. ### ❌ Actual Behavior _No response_
yindo added the 🐞 bug label 2026-02-21 18:23:19 -05:00
yindo closed this issue 2026-02-21 18:23:19 -05:00
yindo commented 2026-02-21 18:23:19 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Feb 4, 2025):

@LinLin00000000 Hello there! I'm here to help you with any bugs, questions, or contributions you have in mind. Let's work together to solve the issue you've encountered.

I found a similar open issue related to Chinese character encoding in Dify. The suggested solution is to ensure that JSON encoding and decoding operations handle non-ASCII characters correctly by using the ensure_ascii=False parameter in the json.dumps function. This parameter ensures that the JSON output includes non-ASCII characters as they are, rather than escaping them. Additionally, ensure that all text fields in configuration files support Chinese characters [1].

In your case, it seems that the conversion to Unicode escape characters is increasing token consumption significantly. You might want to check if the ensure_ascii parameter is being used in your setup and adjust it accordingly to prevent the conversion of Chinese characters to Unicode escape sequences. This could help reduce the token consumption when processing prompts with non-ASCII characters.

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other

@dosubot[bot] commented on GitHub (Feb 4, 2025): <!-- Greeting --> @LinLin00000000 Hello there! I'm here to help you with any bugs, questions, or contributions you have in mind. Let's work together to solve the issue you've encountered. <!-- Answer --> I found a similar open issue related to Chinese character encoding in Dify. The suggested solution is to ensure that JSON encoding and decoding operations handle non-ASCII characters correctly by using the `ensure_ascii=False` parameter in the `json.dumps` function. This parameter ensures that the JSON output includes non-ASCII characters as they are, rather than escaping them. Additionally, ensure that all text fields in configuration files support Chinese characters <sup>[[1]](https://github.com/langgenius/dify/issues/8021)</sup>. In your case, it seems that the conversion to Unicode escape characters is increasing token consumption significantly. You might want to check if the `ensure_ascii` parameter is being used in your setup and adjust it accordingly to prevent the conversion of Chinese characters to Unicode escape sequences. This could help reduce the token consumption when processing prompts with non-ASCII characters. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.* --- <sup>Help Dosu learn! Give it feedback: [Great Response](https://app.dosu.dev/response-feedback/84dcd046-7f62-4815-8e47-ed9313c07051?feedback_type=great_response) | [Irrelevant Answer](https://app.dosu.dev/response-feedback/84dcd046-7f62-4815-8e47-ed9313c07051?feedback_type=irrelevant_answer) | [Incorrect Sources](https://app.dosu.dev/response-feedback/84dcd046-7f62-4815-8e47-ed9313c07051?feedback_type=incorrect_sources) | [Too Verbose](https://app.dosu.dev/response-feedback/84dcd046-7f62-4815-8e47-ed9313c07051?feedback_type=too_verbose) | [Hallucination](https://app.dosu.dev/response-feedback/84dcd046-7f62-4815-8e47-ed9313c07051?feedback_type=hallucination) | [Other](https://app.dosu.dev/response-feedback/84dcd046-7f62-4815-8e47-ed9313c07051?feedback_type=other)</sup>
yindo commented 2026-02-21 18:23:19 -05:00
Author
Owner
Copy Link
Copy Source

@1999single commented on GitHub (Feb 9, 2025):

兄弟，你解决了吗

@1999single commented on GitHub (Feb 9, 2025): 兄弟，你解决了吗
yindo commented 2026-02-21 18:23:19 -05:00
Author
Owner
Copy Link
Copy Source

@LinLin00000000 commented on GitHub (Feb 9, 2025):

兄弟，你解决了吗

并没有，等待官方解决吧
No, wait for the official solution

@LinLin00000000 commented on GitHub (Feb 9, 2025): > 兄弟，你解决了吗 并没有，等待官方解决吧 No, wait for the official solution
yindo commented 2026-02-21 18:23:20 -05:00
Author
Owner
Copy Link
Copy Source

@allensavage commented on GitHub (Feb 10, 2025):

Not only increate the total input token, but also the process time of the LLM.
A rough estimate is that it will take 2-3 times longer than before.

@allensavage commented on GitHub (Feb 10, 2025): Not only increate the total input token, but also the process time of the LLM. A rough estimate is that it will take 2-3 times longer than before.
yindo commented 2026-02-21 18:23:20 -05:00
Author
Owner
Copy Link
Copy Source

@Zephyr69 commented on GitHub (Feb 26, 2025):

Not only that. Chinese characters are sometimes turned to unicode escape when being passed as a variable to HTTP request node. This makes stuff complicated in many situations,

@Zephyr69 commented on GitHub (Feb 26, 2025): Not only that. Chinese characters are sometimes turned to unicode escape when being passed as a variable to HTTP request node. This makes stuff complicated in many situations,
yindo commented 2026-02-21 18:23:20 -05:00
Author
Owner
Copy Link
Copy Source

@miRUAcleInjoker commented on GitHub (Mar 5, 2025):

快一个月过去了，解决了吗

@miRUAcleInjoker commented on GitHub (Mar 5, 2025): 快一个月过去了，解决了吗
yindo commented 2026-02-21 18:23:20 -05:00
Author
Owner
Copy Link
Copy Source

@allensavage commented on GitHub (Mar 5, 2025):

快一个月过去了，解决了吗

nope, I have migrated to v1.0.0, yet same old story...

@allensavage commented on GitHub (Mar 5, 2025): > 快一个月过去了，解决了吗 nope, I have migrated to v1.0.0, yet same old story...
yindo commented 2026-02-21 18:23:20 -05:00
Author
Owner
Copy Link
Copy Source

@LinLin00000000 commented on GitHub (Mar 5, 2025):

#13860
#14673

Two new issues have been raised regarding this problem. 🥲

@LinLin00000000 commented on GitHub (Mar 5, 2025): #13860 #14673 Two new issues have been raised regarding this problem. 🥲
yindo commented 2026-02-21 18:23:20 -05:00
Author
Owner
Copy Link
Copy Source

@allensavage commented on GitHub (Mar 12, 2025):

guys, thx to the dify team, they fix this problem, check this
https://github.com/langgenius/dify/pull/15318

@allensavage commented on GitHub (Mar 12, 2025): guys, thx to the dify team, they fix this problem, check this https://github.com/langgenius/dify/pull/15318
yindo commented 2026-02-21 18:23:20 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Apr 12, 2025):

Hi, @LinLin00000000. I'm Dosu, and I'm helping the Dify team manage their backlog and am marking this issue as stale.

Issue Summary

  • You reported an issue with Unicode escape characters in LLM prompts in Dify v0.15.2.
  • This led to increased token consumption and affected processing time and HTTP requests.
  • I suggested using ensure_ascii=False in JSON operations.
  • Related issues #13860 and #14673 were mentioned.
  • The issue has been resolved in a recent pull request, as confirmed by allensavage.

Next Steps

  • Please confirm if this issue is still relevant to the latest version of Dify. If so, you can keep the discussion open by commenting here.
  • Otherwise, this issue will be automatically closed in 15 days.

Thank you for your understanding and contribution!

@dosubot[bot] commented on GitHub (Apr 12, 2025): Hi, @LinLin00000000. I'm [Dosu](https://dosu.dev), and I'm helping the Dify team manage their backlog and am marking this issue as stale. **Issue Summary** - You reported an issue with Unicode escape characters in LLM prompts in Dify v0.15.2. - This led to increased token consumption and affected processing time and HTTP requests. - I suggested using `ensure_ascii=False` in JSON operations. - Related issues #13860 and #14673 were mentioned. - The issue has been resolved in a recent pull request, as confirmed by allensavage. **Next Steps** - Please confirm if this issue is still relevant to the latest version of Dify. If so, you can keep the discussion open by commenting here. - Otherwise, this issue will be automatically closed in 15 days. Thank you for your understanding and contribution!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#7973
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?