langgenius/dify
1
0
Fork 0
mirror of https://github.com/langgenius/dify.git synced 2026-07-21 12:05:30 -04:00
Code Issues 775 Packages Projects Releases 10 Wiki Activity

When I use a Japanese PDF as a knowledge, It garbled. #4467

New Issue
Closed
opened 2026-02-21 18:06:34 -05:00 by yindo · 3 comments
yindo commented 2026-02-21 18:06:34 -05:00
Owner
Copy Link
Copy Source

Originally created by @mihit on GitHub (Jul 5, 2024).

Originally assigned to: @JohnJyong on GitHub.

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
  • Please do not modify this template :) and fill in all the required fields.

Dify version

0.6.11

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

Prepare a Japanese PDF contains "の".
Use it as a knowledge.

✔️ Expected Behavior

Never garbled.

❌ Actual Behavior

Bah, it garbled as "䛾".
There appears to be more than this.

Originally created by @mihit on GitHub (Jul 5, 2024). Originally assigned to: @JohnJyong on GitHub. ### Self Checks - [X] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general). - [X] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones. - [X] I confirm that I am using English to submit this report (我已阅读并同意 [Language Policy](https://github.com/langgenius/dify/issues/1542)). - [X] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:） - [X] Please do not modify this template :) and fill in all the required fields. ### Dify version 0.6.11 ### Cloud or Self Hosted Self Hosted (Docker) ### Steps to reproduce Prepare a Japanese PDF contains "の". Use it as a knowledge. ### ✔️ Expected Behavior Never garbled. ### ❌ Actual Behavior Bah, it garbled as "䛾". There appears to be more than this.
yindo added the 🐞 bug label 2026-02-21 18:06:34 -05:00
yindo closed this issue 2026-02-21 18:06:34 -05:00
yindo commented 2026-02-21 18:06:34 -05:00
Author
Owner
Copy Link
Copy Source

@dosubot[bot] commented on GitHub (Jul 5, 2024):

To prevent text from becoming garbled when using a Japanese PDF as a knowledge base in Dify version 0.6.11, ensure that the text extraction process supports Japanese characters. Here are some steps to help:

  1. PDF Encoding: Ensure the PDF is properly encoded with Japanese text.
  2. Text Extraction Tool: Verify that the text extraction tool or library used by Dify supports Japanese characters.

Dify supports text extraction from various document formats, including PDFs. If issues persist, you might need to check the encoding settings or use a different text extraction tool that better supports Japanese text [1].

Additionally, you can adjust configuration settings within Dify to improve text extraction accuracy for Japanese PDFs. Specifically, you can use custom segmentation and preprocessing rules:

  • Segmentation Rules:

    • Set a segmentation identifier such as "\n".
    • Define the maximum segment length.
    • Set the number of overlapping characters between segments.

  • Preprocessing Rules:

    • Replace continuous spaces, newlines, and tabs.
    • Delete all URLs and email addresses [2].

For enhanced preprocessing capabilities, consider using optional ETL solutions like Dify ETL or Unstructured ETL [2].

To continue talking to Dosu, mention @dosu.

@dosubot[bot] commented on GitHub (Jul 5, 2024): <!-- Answer --> To prevent text from becoming garbled when using a Japanese PDF as a knowledge base in Dify version 0.6.11, ensure that the text extraction process supports Japanese characters. Here are some steps to help: 1. **PDF Encoding**: Ensure the PDF is properly encoded with Japanese text. 2. **Text Extraction Tool**: Verify that the text extraction tool or library used by Dify supports Japanese characters. Dify supports text extraction from various document formats, including PDFs. If issues persist, you might need to check the encoding settings or use a different text extraction tool that better supports Japanese text <sup>[[1]](https://github.com/langgenius/dify/blob/main/README_JA.md)</sup>. Additionally, you can adjust configuration settings within Dify to improve text extraction accuracy for Japanese PDFs. Specifically, you can use custom segmentation and preprocessing rules: - **Segmentation Rules**: - Set a segmentation identifier such as "\n". - Define the maximum segment length. - Set the number of overlapping characters between segments. - **Preprocessing Rules**: - Replace continuous spaces, newlines, and tabs. - Delete all URLs and email addresses <sup>[[2]](https://github.com/langgenius/dify-docs/blob/main/en/guides/knowledge-base/create-knowledge-and-upload-documents.md)</sup>. For enhanced preprocessing capabilities, consider using optional ETL solutions like **Dify ETL** or [**Unstructured ETL**](https://unstructured.io/) <sup>[[2]](https://github.com/langgenius/dify-docs/blob/main/en/guides/knowledge-base/create-knowledge-and-upload-documents.md)</sup>. <!-- Dosu Comment Footer --> *To continue talking to [Dosu](https://dosu.dev), mention @dosu.*
yindo commented 2026-02-21 18:06:34 -05:00
Author
Owner
Copy Link
Copy Source

@JohnJyong commented on GitHub (Jul 10, 2024):

hi @mihit ,after testing, some Japanese characters may be recognized incorrectly. If it is convenient, can you send us your document for testing? thanks in advace~

@JohnJyong commented on GitHub (Jul 10, 2024): hi @mihit ,after testing, some Japanese characters may be recognized incorrectly. If it is convenient, can you send us your document for testing? thanks in advace~
yindo commented 2026-02-21 18:06:34 -05:00
Author
Owner
Copy Link
Copy Source

@mihit commented on GitHub (Jul 11, 2024):

太宰治 走れメロス.pdf
@JohnJyong Thank you for reply. My file is private one so I can’t share it. But I send an other Japanese PDF of literary works in the public domain for testing. Hope this helps!

@mihit commented on GitHub (Jul 11, 2024): [太宰治 走れメロス.pdf](https://github.com/user-attachments/files/16185686/default.pdf) @JohnJyong Thank you for reply. My file is private one so I can’t share it. But I send an other Japanese PDF of literary works in the public domain for testing. Hope this helps!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Knowledge Pipeline
Proposal
WIP
cloud
dependencies
feat:webapp
good first issue
outdated
performance
pull-request
Mirrored from GitHub Pull Request
 python:uv
refactor
size:L
stale
status: accepting prs
status: blocked
status: in discussion
stt & tts
vulnerablity
web
wont-fix
🌊 feat:workflow
🌚 invalid
🐞 bug
👻 feat:rag
💪 enhancement
📚 documentation
🔨 feat:tools
🙋‍♂️ question
🙏 help wanted
🤔 cant-reproduce
🤖 feat:agent
No Label 🐞 bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
yindo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: langgenius/dify#4467
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?